Autonomous Market Intelligence Agent

An agentic, Human-in-the-Loop market research system that plans web research, executes multi-source search and scraping, extracts structured competitive intelligence, performs iterative data completion loops, and generates executive-ready reports.

Why This Project Exists

Traditional market research is slow, expensive, and often hard to verify. Analysts spend significant time collecting fragmented data from search results, websites, discussion forums, and product pages before they can produce reliable insights.

This project solves that by automating the full intelligence pipeline with explicit safeguards:

Autonomous planning, search, scrape, extraction, validation, and reporting
Human approval gate before expensive scraping begins
Local memory patching pass before re-querying the web
Iterative loop for missing fields (with max-loop fail-safe)
Semantic caching to avoid repeating near-identical research jobs

What It Delivers

Decomposed targeted search queries from a single user prompt
Deduplicated and relevance-filtered URLs
Concurrent web scraping with content pruning
Structured entity extraction (tech stack, pricing, confidence, completeness, traceability)
Auto-generated executive summary in markdown
PDF report generation for stakeholder sharing
Streaming progress updates to frontend UI
Resume-capable Human-in-the-Loop (HITL) workflow

Real-World Use Cases

Venture Capital and PE scouting: track competitors, stack choices, pricing posture, and differentiation signals quickly.
Product and strategy teams: benchmark your product against alternatives with traceable sources.
Sales intelligence: enrich target accounts with public technical and commercial indicators.
Innovation and market mapping teams: repeatedly monitor emerging players without rerunning identical research.

Architecture Overview

End-to-End Flow

%%{init: {'theme': 'base', 'themeVariables': { 'primaryColor': '#ffffff', 'primaryBorderColor': '#333333', 'lineColor': '#666666', 'textColor': '#333333', 'fontFamily': 'arial'}}}%%
flowchart TD
    %% Define Styles
    classDef startEnd fill:#2563eb,stroke:#1e40af,stroke-width:2px,color:#ffffff,font-weight:bold,rx:10,ry:10;
    classDef agentNode fill:#f8fafc,stroke:#94a3b8,stroke-width:2px,color:#0f172a,rx:5,ry:5;
    classDef humanLoop fill:#fef08a,stroke:#ca8a04,stroke-width:2px,color:#854d0e,font-weight:bold,stroke-dasharray: 5 5;
    classDef localScan fill:#dcfce7,stroke:#16a34a,stroke-width:2px,color:#166534,rx:5,ry:5;
    classDef decision fill:#f3f4f6,stroke:#6b7280,stroke-width:2px,color:#374151,font-weight:bold;

    %% Nodes
    A([Start Query]):::startEnd --> B(Planner Node<br/><i>Decomposes Query</i>):::agentNode
    
    %% HITL Section
    B --> C{{Human-in-the-Loop<br/>Interrupt}}:::decision
    C -->|Wait for Approval| C1[Streamlit UI Pause]:::humanLoop
    C1 -.->|Human Edits & Approves| C
    
    C -->|Queries Approved| D(Searcher Node<br/><i>SearXNG</i>):::agentNode
    
    %% Extraction Pipeline
    D --> E(Filter Node<br/><i>Prunes URLs</i>):::agentNode
    E --> F(Scraper Node<br/><i>Crawl4AI</i>):::agentNode
    F --> G(Extractor Node<br/><i>Structured JSON</i>):::agentNode
    
    %% Local Memory Optimization
    G --> H(Local Patcher Node):::localScan
    H -.->|Zero-Cost Fallback| H1[(In-Memory<br/>Scraped Text)]
    H -.-> H2(Micro-Extract<br/>Missing Fields):::localScan
    H2 --> I(Merger Node<br/><i>Stitches Master List</i>):::agentNode
    
    %% Conditional Routing Loop
    I --> J{{Data Completeness<br/>Validator}}:::decision
    
    J -->|Missing Data &<br/>Loops < 2| K(Targeted Loop Setup<br/><i>Generates New Queries</i>):::agentNode
    K --> D
    
    J -->|Data Complete or<br/>Max Loops Reached| L(Reporter Node<br/><i>Generates Summary</i>):::agentNode
    
    %% Final Output
    L --> M([Final Payload & PDF]):::startEnd

Human-in-the-Loop Highlights

The graph pauses at approval using LangGraph interrupt semantics.
Streamlit renders editable query inputs and resumes execution with human-approved queries.
This creates a cost-aware and controllable research pipeline.

Local Memory Scanner Highlights

Before looping back to web search, local_patcher performs a zero-cost in-memory scan across already scraped text.
It targets only missing fields, reducing additional web/LLM calls where possible.

Tech Stack

Core Languages and Runtime

Python 3.13
Asyncio-based concurrent pipeline execution

Agentic and LLM Orchestration

LangGraph (stateful graph orchestration, interrupts, checkpointing)
LangChain ecosystem:
- langchain-core
- langchain
- langchain-classic
- langchain-community
- langchain-chroma
Model connectors:
- langchain-groq
- langchain-ollama

API and UI

FastAPI (streaming endpoint, HITL state routes, rate limiting)
Streamlit (interactive dashboard with progress logs, metrics, entity cards)
SlowAPI (IP-based rate limiting)

Search and Web Extraction

SearXNG (self-hosted metasearch aggregation)
Crawl4AI (async crawling + markdown extraction + content pruning)
HTTPX / Requests

Data, Cache, and Persistence

ChromaDB (semantic query vector cache)
SQLite + SQLAlchemy (payload storage and retrieval)
In-memory LangGraph checkpoint saver (thread-based run state)

Reporting and Artifacts

Markdown -> HTML conversion (markdown)
PDF generation (xhtml2pdf)
Generated PDF reports in reports/

Deployment and Infra

Docker + Docker Compose
Multi-service stack:
- backend (FastAPI + LangGraph)
- frontend (Streamlit)
- searxng
- n8n (automation/webhook integration)

Project Structure

backend/
	agent/
		graph.py                # LangGraph topology and routing
		state.py                # Shared graph state schema
		nodes/                  # Planner, Searcher, Filter, Scraper, Extractor, etc.
	api/
		main.py                 # Streaming API and orchestration bridge
		routes/hitl.py          # HITL pause-state inspection
	core/schema.py            # Pydantic output schemas
	tools/                    # Search + scraper tools
	utils/                    # Cache, DB, PDF generation
frontend/
	app.py                    # Streamlit interface
reports/                    # Generated PDF reports
searxng/
	settings.yml              # SearXNG configuration

API Contract

Stream Research

Endpoint: POST /api/v1/research/stream
Body:

{
	"query": "Compare top AI code assistants for enterprise teams in 2026",
	"thread_id": "<uuid>",
	"approved_queries": [
		"optional human-edited query 1",
		"optional human-edited query 2"
	]
}

Behavior:
- Streams progress events
- Pauses if HITL interrupt is reached
- Resumes when approved queries are submitted

HITL State Check

Endpoint: GET /hitl/agent-state/{thread_id}
Response contains pause status and pending approval queries.

Quick Start

1. Prerequisites

Docker and Docker Compose
Optional local Ollama runtime (if you want local model execution paths)
API keys depending on selected providers

2. Configure Environment

Create a .env file at project root. Typical keys used by this project include:

GROQ_API_KEY
OLLAMA_API_KEY (if required by your model endpoint setup)

You can also configure model-specific endpoints according to your environment.

3. Run with Docker Compose

docker compose up --build -d

4. Open Apps

Frontend: http://localhost:8501
Backend API docs: http://localhost:8000/docs
SearXNG: http://localhost:8080
n8n: http://localhost:5678

Running Without Docker (Optional)

uv sync
uv run fastapi run backend/api/main.py --host 0.0.0.0 --port 8000

In a second terminal:

cd frontend
uv run streamlit run app.py

Screenshots

1. Main Dashboard

2. HITL Approval Pause

3. Entity Cards and Metrics

4. Final Summary + Metadata

5. Architecture / Graph Visualization

Output Schema (High-Level)

Each extracted entity can include:

Identity: name, website, location, description
Commercial: pricing_model, funding_info
Technical: nested tech_stack (frontend/backend/infrastructure/ai_ml/other)
Quality: confidence_score, completeness_score
Traceability: source_links, source_attribution
Loop flags: missing_fields, requires_further_research

Notable Engineering Choices

Streaming-first UX: users see node-level progress, not a blank waiting screen.
HITL at the right moment: approval before expensive scraping/token usage.
Cost/latency optimization: local patch scan before another web loop.
Reliability guardrails: parser fallbacks, retries, timeout controls, max research loops.
Practical caching design: semantic query match in Chroma, heavy payloads in SQLite.

Testing and Utility Scripts

test_run.py for direct graph execution
test_scraper.py for isolated crawler validation
test_db.py for SQLite payload save/retrieve checks

Security Notes

Keep API keys in .env only.
Do not commit secrets, webhook tokens, or private tunneling credentials.

Roadmap Ideas

Add unit/integration tests for each node contract
Replace in-memory checkpointing with persistent production checkpointer
Add CI pipeline for lint, test, and container validation
Add role-based access and audit logs for HITL decisions

License

This project is released under the license defined in LICENSE.

Name		Name	Last commit message	Last commit date
assets		assets
backend		backend
frontend		frontend
.dockerignore		.dockerignore
.gitignore		.gitignore
.python-version		.python-version
LICENSE		LICENSE
README.md		README.md
docker-compose.yml		docker-compose.yml
graph.png		graph.png
main.py		main.py
pyproject.toml		pyproject.toml
test_db.py		test_db.py
test_run.py		test_run.py
test_scraper.py		test_scraper.py
uv.lock		uv.lock

Latest commit

History

Folders and files

Repository files navigation