paper-comparison-engine

Ask a comparison question across a set of research papers and get back a table where every cell names the page it came from.

Compare these papers' approaches to X. What benchmarks were used,
what metrics were reported, and what limitations did the authors admit to?

Paper	Method	Benchmark	Metric	Result	Limitation	Source
A	...	...	...	...	...	p.6
B	...	...	...	...	...	p.4

The rule the system has to hold: no cell exists that isn't grounded in a retrieved chunk. If the corpus doesn't answer something, the cell reads "not reported" instead of a plausible guess.

Status

The pipeline runs end to end. 40 papers fetched, extracted, and chunked into 1098 chunks; both indexes built; retrieval, reranking, extraction, citation check and the Streamlit table are in place.

./venv/bin/python ingest/fetch.py      # arXiv -> data/pdfs + papers.json
./venv/bin/python ingest/chunk.py      # -> data/chunks.jsonl
./venv/bin/python index/search.py      # embed into Chroma (once, ~1 min)
./venv/bin/python eval/run.py          # scores
./venv/bin/streamlit run app.py

Measured so far, over two question sets. One describes papers roughly the way their titles do. The other asks about a detail from inside the paper, in words the title never uses.

recall@5	title-like (20 q)	hard (16 q)
embeddings only	100%	88%
BM25 only	90%	100%
both, fused	100%	100%

Neither index is redundant, and they fail on different questions. Dense retrieval misses when everything turns on a rare literal string, such as the name of a dataset used in one paper; BM25 misses when the question shares no vocabulary with the paper. Median retrieval latency is 21-31 ms.

Six questions the corpus cannot answer, asked of three papers each, produce 18 cells and no invented ones. The checker reads one field at a time against the excerpt that field cites, and both it and the extractor are shown the question, which is what keeps a true sentence about something else from passing as support.

That last figure is measured on gpt-oss-20b. The retrieval numbers above, and spot checks on individual cells, are on gpt-oss-120b, which is what the app uses. The free tier allows 200,000 tokens a day and one full pass spends most of it.

How it works

40 papers (PDF)
  -> PyMuPDF text extraction, page numbers preserved
  -> section-aware chunking (abstract / methods / results / discussion,
     ~300-500 tokens, 50-token overlap)
  -> two indexes: local embeddings in Chroma, plus BM25
  -> query -> hybrid retrieval, ~20-30 candidates, fused by reciprocal rank
  -> local cross-encoder rerank down to ~8
  -> Groq (gpt-oss-120b) extracts structured JSON from those chunks only
  -> citation check: does the cited chunk actually support the claim?
  -> rows aggregate into a comparison table
  -> Streamlit UI, filtered by year

BM25 sits alongside the embeddings because exact terms matter here. A query for "GSM8K" should find the papers that write "GSM8K", and a dense retriever will happily return something semantically adjacent instead.

Layout

ingest/    PDF extraction and chunking
index/     Chroma and BM25 setup
extract/   Groq extraction and the citation check
eval/      question set and scoring script
app.py     Streamlit UI
data/      raw PDFs and the vector store (gitignored)

Stack

Everything is either local or on a free tier, so a query costs nothing but latency.

Stage	Tool	Why
PDF extraction	PyMuPDF	Tracks page numbers reliably and survives two-column academic layout
Chunking	hand-rolled regex section detection	Small enough to understand completely, which matters when retrieval goes wrong
Embeddings	sentence-transformers, BAAI/bge-small-en-v1.5	Downloads once, runs on CPU, no API key
Vector store	Chroma	Local, persists to disk, filters on metadata
Keyword index	rank_bm25	In memory, fine at this corpus size
Reranker	BAAI/bge-reranker-base	Pretrained, inference only, no training run
Extraction	Groq, gpt-oss-120b in JSON mode	Fast, and holds a JSON schema better than a small local model
Citation check	same model	One narrow question: does this chunk support this claim?
UI	Streamlit	Shortest path to something demonstrable

Swapping Groq for a local model under Ollama makes the whole thing offline at some cost in JSON reliability.

Data

40 papers pulled from the arXiv API, all circling one narrow sub-question. Narrow enough that the papers are actually comparable, contested enough that they disagree about methods and results. Papers that agree produce a boring table.

Evaluation

eval/run.py scores retrieval against 36 questions whose answer is a known paper, hand-written and checked against the text. Twenty of them describe a paper roughly the way its title does. Sixteen ask about something inside the paper instead: the dataset it trains on, the model it uses to rerank, the language pair it has to bridge. Each set is scored on the fused retrieval and on each index alone, since a hybrid that scores the same as either half by itself is one index too many.

Six more questions have no answer anywhere in the corpus. Every cell they produce should read "not reported", so anything else is a hallucination the citation check failed to stop.

--retrieval-only skips the Groq calls, which is how the retrieval numbers stay runnable once the daily token allowance is gone. A query costs $0 either way, which is worth reading next to the latency.

Setup

python -m venv venv
source venv/bin/activate
pip install -r requirements.txt
cp .env.example .env    # then add your Groq key