Lightning Research

Automated knowledge discovery and web scraping. Build comprehensive knowledge bases for AI code intelligence through organic web crawling.

Development

Knowledge Architecture

                    ┌─────────────────────────────────────┐
                    │       Lightning Research            │
                    │     Web Scraping Orchestrator       │
                    └──────────────────┬──────────────────┘
                                       │
          ┌────────────────────────────┼────────────────────────────┐
          │                            │                            │
    ┌─────▼─────┐              ┌───────▼───────┐            ┌───────▼───────┐
    │  Basement │              │    Library    │            │  Specialists  │
    │    (PG)   │              │    (Redis)    │            │    (Redis)    │
    │  Raw Data │──────────────▶  Curated KB   │────────────▶ Domain Agents │
    └───────────┘              └───────────────┘            └───────────────┘
    
    Tier 1: Raw scraped data → Tier 2: Processed knowledge → Tier 3: Agent expertise

Organic Spawning

Discover-as-you-go: Research topics spawn organically from discovered references. No artificial limits - follow the knowledge wherever it leads.

# Start research on a topic
POST /api/research/spawn
{
  "seed_topic": "Python asyncio patterns",
  "max_depth": 4,
  "categories": ["programming", "python", "concurrency"],
  "organic_spawn": true
}

# System discovers related topics:
# asyncio → coroutines → event loops → uvloop → libuv
# Each spawns its own research branch

Research Categories

Category	Topics	Layer Depth
Python	stdlib, asyncio, typing, dataclasses	4
Web Frameworks	FastAPI, Flask, Django	3
Databases	Redis, PostgreSQL, SQLAlchemy	3
DevOps	Docker, Kubernetes, CI/CD	2
AI/ML	PyTorch, Transformers, LangChain	3

Powers Lightning Forge

Code Intelligence: Research populates the knowledge base that Lightning Forge uses to understand and fix code. Better knowledge = better fixes.