Skip to content

README.md

Latest commit

 

History

History
264 lines (208 loc) · 8.8 KB

README.md

File metadata and controls

264 lines (208 loc) · 8.8 KB
Raw

AgentIntel ⚡ Swarm

Website Intelligence Swarm for AEO/GEO and Competitive Intelligence

AI agents that analyze your documentation, compare competitors, score AI discoverability (AEO/GEO), and produce prioritized content recommendations — in ~41 seconds.

What it does

Paste a query like "What documentation should OpenAI improve to compete with LangChain?"
Seven coordinated AI agents answer it:

Step Agent Output
1 Planner Routes query, assigns focus to each agent
2+3 Website + Competitor (parallel) Scrapes OpenAI docs + LangChain + CrewAI
4 Knowledge NetworkX concept graph (16 concepts, ~17 edges)
5 GEO AEO/GEO scores per page — how likely AI assistants are to cite them
6 Gap Specific missing content with evidence from competitors
7 Recommendation Ranked action plan with % improvement estimate

Results appear live via SSE as each agent finishes. Full run: ~41 seconds.

Architecture

Architecture Diagram

Component Map

Demo Queries

What documentation should OpenAI improve to compete with LangChain?
Which framework has the best MCP documentation?
What content is OpenAI missing compared to competitors?
Which framework is easiest for beginners?
If you were OpenAI's docs team, what are the top 5 content improvements?

Quick Start

Prerequisites

Tool Version Install
Python 3.11+ winget install Python.Python.3.13
Node.js 18+ winget install OpenJS.NodeJS
Ollama any ollama.ai — pull qwen2.5:7b and llama3.2:3b
OpenAI key — Add to backend/.env as OPENAI_API_KEY (replaces Ollama, ~5× faster)
Actian VectorAI DB — actian.com/vectorai — install Windows service

1. Clone and configure

git clone <repo-url>
cd AgentIntel
copy backend\.env.example backend\.env
# Edit backend\.env — add OPENAI_API_KEY at minimum

2. Install dependencies

# Backend
cd backend
python -m venv .venv
.\.venv\Scripts\pip install -r requirements.txt

# Frontend
cd ..\frontend
npm install

3. Pre-populate cache (optional but recommended)

cd backend
.\.venv\Scripts\python.exe ingest.py
# Scrapes all 11 source URLs and caches them — first swarm run will be instant

4. Start everything

# From project root — starts Actian services, backend, and frontend
.\start_all.ps1

Or start individually:

# Terminal 1 — Backend (NO --reload — keeps swarms stable)
cd backend
.\.venv\Scripts\python.exe -m uvicorn main:app --host 0.0.0.0 --port 8000

# Terminal 2 — Frontend
cd frontend
npm run dev

5. Open the app

http://localhost:3000

Stop

# From project root — kills Python, Node, and stops Actian Windows services
.\shutdown.ps1

Configuration — backend/.env

# LLM — add OpenAI key to use gpt-4.1-nano (~5x faster than Ollama)
OPENAI_API_KEY=sk-...
OPENAI_ANALYSIS_MODEL=gpt-4.1-nano
OPENAI_ROUTING_MODEL=gpt-4.1-nano

# Ollama fallback (used when OPENAI_API_KEY is empty)
OLLAMA_BASE_URL=http://localhost:11434
OLLAMA_ANALYSIS_MODEL=qwen2.5:7b
OLLAMA_ROUTING_MODEL=llama3.2:3b

# Sponsor: Guild.ai (run tracking)
GUILD_API_KEY=your_guild_api_key_here   # run: guild auth login

# Sponsor: Band.ai (agent messaging)
BAND_API_KEY=band_u_...
BAND_BASE_URL=https://app.band.ai

# Sponsor: Actian VectorAI DB (vector persistence)
ACTIAN_HOST=localhost
ACTIAN_PORT=6573
ACTIAN_DB=agentintel

# Self-evolution
EVOLUTION_INTERVAL_SECONDS=300   # delay between continuous runs
MAX_ITERATIONS=0                  # 0 = infinite

API Reference

Method Endpoint Description
GET / Service info
GET /health Sponsor connectivity: ollama, band, guild, actian
POST /analyze Start swarm {"query": "...", "continuous": false}
GET /stream SSE — real-time agent step events
GET /result Latest complete swarm result (JSON)
POST /stop Stop a continuous loop
GET /status Loop state + quality history
GET /learnings Self-evolution learnings accumulated so far
GET /history Recent swarm results from Actian
GET /band/messages Band message log
GET /guild/runs Guild.ai run history

Project Structure

AgentIntel/
├── README.md
├── LEARNINGS.md              # Build notes and architecture decisions
├── cited.md                  # Published swarm output (auto-generated)
├── start_all.ps1             # Start everything (Actian + backend + frontend)
├── shutdown.ps1              # Safe shutdown including Actian services
│
├── backend/
│   ├── main.py               # FastAPI server, SSE, ThreadPoolExecutor
│   ├── config.py             # All environment variables and source URLs
│   ├── ingest.py             # Pre-populate scrape cache + Actian DB
│   ├── requirements.txt
│   ├── .env.example          # Copy to .env, fill in keys
│   │
│   ├── agents/
│   │   ├── base.py           # BaseAgent: LLM routing (OpenAI/Ollama), parse_json
│   │   ├── planner.py        # Routes query → per-agent focus instructions
│   │   ├── website.py        # Scrapes OpenAI docs, extracts concepts
│   │   ├── competitor.py     # Scrapes LangChain + CrewAI
│   │   ├── knowledge.py      # NetworkX concept graph builder
│   │   ├── geo.py            # AEO/GEO discoverability scorer
│   │   ├── gap.py            # Content gap finder
│   │   └── recommendation.py # Ranked action plan generator
│   │
│   ├── swarm/
│   │   ├── orchestrator.py   # Coordinates all 7 agents, parallel execution
│   │   └── loop.py           # Self-evolution: quality scoring, learnings
│   │
│   ├── tools/
│   │   ├── scraper.py        # HTTP scraper with 24h disk cache
│   │   ├── vectorstore.py    # 48-dim TF-IDF, numpy cosine similarity
│   │   └── publisher.py      # Writes results to cited.md
│   │
│   └── sponsors/
│       ├── band.py           # Band.ai agent messaging
│       ├── guild.py          # Guild.ai run tracking
│       └── actian.py         # Actian VectorAI DB (Qdrant REST)
│
├── frontend/
│   ├── app/
│   │   ├── page.tsx          # Main SPA: query, SSE stream, results tabs
│   │   └── layout.tsx        # Tailwind wrapper
│   └── lib/
│       └── api.ts            # Backend API client
│
└── docs/
    ├── architecture.svg      # Full system architecture diagram
    ├── components.svg        # File dependency map
    ├── agents/               # Per-agent logic documentation
    ├── swarm/                # Orchestrator and evolution loop docs
    ├── tools/                # Scraper, vectorstore, publisher docs
    └── sponsors/             # Band, Guild.ai, Actian integration docs

Sponsor Integrations

Sponsor Status Role
Actian VectorAI DB Connected (local) Stores doc chunks and swarm results as 48-dim vectors. Qdrant-compatible REST on port 6573.
Band.ai 401 (JWT auth required) Broadcasts agent events to jam rooms. Local log at GET /band/messages.
Guild.ai Needs CLI auth Tracks each swarm run with quality scores. Run guild auth login to activate.

Self-Evolution

Enable the Loop (self-evolve) checkbox in the UI to run continuously. After each iteration:

  1. Scores quality: recommendations × 0.8 + gaps × 0.3 + GEO/10 × 0.3 + summary + quick wins
  2. Extracts learnings: gap patterns, top recommendations, GEO trends
  3. Injects learnings into next iteration's Planner prompt
  4. Stores quality history in data/learnings.json and Actian

The swarm improves its focus over multiple iterations — if GEO scores are low, it will focus subsequent runs on FAQ coverage and direct answers.

Tech Stack

Layer Technology
Frontend Next.js 16, React 19, Tailwind CSS 4, TypeScript
Backend FastAPI (Python 3.13), uvicorn, ThreadPoolExecutor
LLM (fast) OpenAI gpt-4.1-nano — ~41s end-to-end
LLM (local) Ollama — qwen2.5:7b (analysis), llama3.2:3b (routing)
Vector Store Custom numpy TF-IDF (48-dim) + Actian VectorAI DB
Graph NetworkX DiGraph
Scraping httpx + BeautifulSoup4 with 24h disk cache
SSE sse-starlette, asyncio.Queue, call_soon_threadsafe

Built for the hackathon — AgentIntel · Website Intelligence Swarm for AEO/GEO and Competitive Intelligence