ResearchForge is an autonomous multi-agent research platform built for structured investigation rather than one-shot summarization. Instead of returning a single answer, it decomposes a question into focused sub-queries, searches the web through SearXNG, crawls and normalizes source pages, extracts and challenges claims, debates competing interpretations, and assembles a fully referenced Markdown and PDF report with a confidence score.
ResearchForge runs as an event-driven pipeline built on Redis Streams. Each stage consumes a stream event, performs its work independently, and emits the next event when it completes. The system is designed to behave like a compact research organization rather than a single-step summarizer, with separate agents handling planning, retrieval, synthesis, critique, and final reporting, while persisting outputs to PostgreSQL and surfacing progress through the API.
The generated report includes:
- Executive summary
- Key findings
- Methodology
- Supporting evidence
- Counterarguments
- Final assessment
- Confidence score
- Autonomous multi-agent research pipeline
- Event-driven orchestration with Redis Streams
- Parallel web search and crawling across multiple workers
- Claim extraction with evidence-backed critique
- Multi-agent debate for balanced, adversarial analysis
- Markdown and PDF report generation with polished formatting
- Confidence scoring for final conclusions
- REST API and Streamlit interface for job submission and monitoring
- Live progress visibility through job and log endpoints
- Pluggable LLM backends through the existing core configuration layer
If Docker and uv are already available on your machine, the shortest path to a running system is:
- Start the infrastructure with
docker compose up -d. - Install dependencies with
uv sync. - Initialise the database with
uv run python init_db.py. - Launch the workers with
uv run python services_launcher.py. - Start the API with
uv run fastapi dev backend/api/main.py. - Start the frontend with
uv run streamlit run frontend/app.py.
flowchart LR
subgraph Frontend
UI[Streamlit UI]
end
subgraph Backend
API[FastAPI API]
RS[(Redis Streams)]
P[Planner Agent]
RW[Research Workers<br/>3x parallel]
C[Claim Extractor]
K[Critic Agent]
D[Debate Agent]
R[Report Writer]
DB[(PostgreSQL<br/>pgvector)]
end
subgraph External
SX[SearXNG<br/>self-hosted]
CA[Crawl4AI<br/>crawler]
EM[SentenceTransformers<br/>embeddings]
end
UI --> API
API -->|publish job| RS
RS -->|job_researched| C
RS -->|claims_completed| K
RS -->|critiques_completed| D
RS -->|debate_completed| R
RS -->|research_completed| P
P -->|publish tasks| RS
RS -->|research tasks| RW
RW -->|search| SX
RW -->|crawl| CA
RW -->|deduplicate / MMR| EM
RW -->|save findings| DB
C -->|save claims| DB
K -->|save critiques| DB
D -->|save arguments| DB
R -->|save report| DB
R -->|markdown| API
DB ---|read| UI
API ---|REST| UI
The FastAPI application is defined in backend/api/main.py, and the Streamlit frontend is in frontend/app.py.
- Backend: FastAPI, LangChain, LangGraph
- Messaging: Redis Streams via
redis-py - Database: PostgreSQL with pgvector
- Search: Self-hosted SearXNG
- Crawling: Crawl4AI
- LLMs: OpenAI-compatible clients, Groq, and Ollama integrations
- Embeddings: SentenceTransformers
- Frontend: Streamlit
- PDF generation: Markdown + WeasyPrint
- Package management: uv
- Python 3.11+
- Docker and Docker Compose
- Playwright browser binaries for Crawl4AI
- Service credentials or endpoints configured in backend/core/llm.py
- Start the infrastructure services.
docker compose up -dThis starts PostgreSQL with pgvector, Redis, and SearXNG.
- Install the Python environment.
uv syncRun the Crawl4ai Setup Command after installing the python environment
crawl4ai-setupIf Crawl4AI needs browser binaries in your environment, install them with:
uv run playwright install --with-deps- Configure environment variables.
Create a .env file in the project root and use the format below.
Use standard KEY=VALUE entries, one per line. Keep secrets out of version control and store them only in your local .env file or deployment environment.
Required variables:
DATABASE_URL=postgresql+asyncpg://forge:forgepass@localhost:5432/researchforge
REDIS_URL=redis://localhost:6379
SEARXNG_BASE_URL=http://localhost:8888Optional provider keys, depending on the LLM backend you enable:
OLLAMA_API_KEY=your-ollama-api-key
GROQ_API_KEY=your-groq-api-keyIf you are not using a provider, omit its key.
- Initialise the database.
uv run python init_db.pyStart the worker processes first:
uv run python services_launcher.pyThis launches the planner, research, claim, critic, debate, and report workers.
Start the API in a separate terminal:
uv run fastapi dev backend/api/main.pyStart the Streamlit frontend in another terminal:
uv run streamlit run frontend/app.pyThen open http://localhost:8501 in your browser.
You can also submit a research question from the terminal:
uv run python publish_tasks.py "Your research question here"This publishes the job to Redis, waits for the pipeline to finish, and writes the PDF report to data/report_<question>.pdf.
| Method | Endpoint | Description |
|---|---|---|
| GET | / |
Health check |
| POST | /research |
Create a new research job |
| GET | /job/{job_id}/status |
Check job status |
| GET | /job/{job_id}/report |
Get the structured report as JSON |
| GET | /job/{job_id}/report/markdown |
Get the cleaned Markdown report |
| GET | /job/{job_id}/report/pdf |
Download the rendered PDF report |
| GET | /job/{job_id}/logs |
Fetch live agent logs for a job |
backend/
api/ FastAPI application and routes
agents/ Planner, research, claim, critic, debate, gap, and report agents
core/ Configuration, database, Redis, and LLM setup
models/ SQLAlchemy models for jobs, claims, critiques, reports, and related data
services/ Search service wrappers
frontend/
app.py Streamlit dashboard
init_db.py Database bootstrap script
publish_tasks.py CLI helper for submitting a query from the terminal
services_launcher.py Worker launcher for the main agents
docker-compose.yml PostgreSQL, Redis, and SearXNG services
- Research workers use the SearXNG instance configured by
SEARXNG_BASE_URL. - The PDF export is generated from the Markdown report and includes the confidence score.
- The dashboard shows live agent logs while the job is running, then renders the final report and PDF download.
- CLI runs write PDFs into
data/using a sanitized version of the research question.
MIT