diff --git a/AGENTS.md b/AGENTS.md index 800f0f9..845a6db 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -25,7 +25,8 @@ Treat the repo as a **workspace generator first, research workspace second**. If ```bash make init # refine intent → topic.json + agents/ + skills/ + TOPIC_AGENTS.md -make scout # OpenAlex + LLM ranking → data/candidates.json +make scout # OpenAlex + cheap prefilter + LLM rubric ranking → data/candidates.json +make eval # compare the cheap deterministic metric vs the LLM judge make review # print candidate review queue python3 scripts/accept_candidates.py ... # accept into corpus make corpus # notes + reports/research_report.md diff --git a/Makefile b/Makefile index e3f0944..b33b580 100644 --- a/Makefile +++ b/Makefile @@ -15,6 +15,7 @@ help: " make plan Print the generated multi-agent task plan" \ " python3 scripts/orchestrate.py emit --mode copilot|copilot-cli|microsoft-scouting" \ " make opportunities-check Validate LLM opportunity JSON" \ + " make eval Compare the cheap deterministic metric vs the LLM judge" \ " make test Run unit tests" init: @@ -51,5 +52,8 @@ plan: opportunities-check: $(PYTHON) scripts/validate_opportunities.py +eval: + $(PYTHON) scripts/eval_metric.py $(if $(INPUT),--input "$(INPUT)",) $(if $(REPORT),--report "$(REPORT)",) + test: $(PYTHON) -m unittest discover -s tests diff --git a/README.md b/README.md index b761d4c..cac0ec6 100644 --- a/README.md +++ b/README.md @@ -2,31 +2,197 @@ [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg?style=flat-square)](https://opensource.org/licenses/MIT) [![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg?style=flat-square)](https://www.python.org/) -[![OpenAlex](https://img.shields.io/badge/data-OpenAlex-green.svg?style=flat-square)](https://openalex.org/) +[![Data: OpenAlex](https://img.shields.io/badge/data-OpenAlex-green.svg?style=flat-square)](https://openalex.org/) +[![Agent-agnostic](https://img.shields.io/badge/agents-Claude%20%7C%20Codex%20%7C%20Copilot-purple.svg?style=flat-square)](AGENTS.md) -Turn a plain-language research intent into a self-updating literature workspace. Discover, rank, and maintain a living paper corpus for any AI research topic — driven by Codex, Claude Code, GitHub Copilot, Copilot CLI, or Microsoft scouting-style agents. +**Turn a plain-language research intent into a self-updating, beautifully formatted literature workspace — for _any_ topic.** + +Describe what you want to track ("AI in hiring", "AI for theorem proving", "agent governance"). Topic Scout refines that intent into a contract, discovers and ranks papers from [OpenAlex](https://openalex.org), curates a living corpus, and publishes a designed HTML dashboard, Markdown notes, and a research-gap analysis — in a **consistent house style you never have to design or pay tokens to produce.** + +> Topic Scout is the reusable, open-source engine generalized from [**EvaPaper**](https://github.com/ginaecho/EvaPaper), a living research archive on AI agent governance. EvaPaper is one topic; Topic Scout is the machine that produces an EvaPaper for whatever you care about. --- -## What it does +## 🤔 Why not just use Microsoft Scout (or another scouting tool)? + +Fair question — but it's partly a name collision. [**Microsoft Scout**](https://www.microsoft.com/en-us/microsoft-365/blog/2026/06/02/introducing-microsoft-scout-your-always-on-personal-agent/) (launched June 2026, built on OpenClaw — the same framework EvaPaper runs on) is an *always-on personal **work** agent* for Microsoft 365: it schedules meetings, preps materials, tracks deliverables, and flags stalled decisions across Teams, Outlook, and SharePoint. It is **not** a research-paper tool — it has no fixed research output format, no persistent topic corpus, and no per-topic analytics. Ask it to "scout papers" and it delegates to a general-purpose research sub-agent that hands back ad-hoc output. + +That's the gap Topic Scout fills. It's built for one job Microsoft Scout (and general chat assistants) don't do: **producing a consistent, ownable research product over time.** The difference matters once you scout the same topic more than once. + +**1. You get a house style for free — no tokens spent on formatting.** +Ask a general assistant to "scout papers and make a report," and *you* pay (in tokens, prompt engineering, and inconsistency) to specify the HTML layout, the Markdown structure, the summary format — every single run. Topic Scout ships a designed template. Every dashboard, every paper note, every synthesis report comes out in the same editorial style (warm broadsheet palette, serif display headings, an interactive wiki, a citation graph, a ranked opportunities column). You spend your tokens on **research judgment, not CSS.** + +**2. Consistent analytics of the topics you actually care about.** +Because the output format is fixed, the *analytics are comparable across runs and across topics*: corpus shape by taxonomy, discovery trend over time, scout token/cost accounting, citation-neighborhood graph, evidence-backed research gaps. A chat tool gives you a different-shaped answer every time; Topic Scout gives you the **same instrument pointed at different topics.** + +**3. The corpus is a durable artifact you own — not a chat transcript.** +Everything lives as plain files in your git repo: `topic.json` (the contract), `data/papers.json` (the corpus of record), Markdown notes, a self-contained HTML file. It's versioned, diffable, forkable, and self-hostable. No vendor lock-in, no re-running a prompt to recover last week's output. + +**4. It runs under the agent you already use.** +Not a closed hosted product — the same `make` command surface drives **Claude Code, Codex CLI, GitHub Copilot, Copilot CLI, Microsoft-scouting-style agents, or Claw/swarm workers.** Bring your own runtime. + +**5. It's opinionated about research hygiene.** +Discovery is not acceptance. Every paper keeps a stable identifier and source URL. Every gap states its evidence, inference, and uncertainty. Nothing gets fabricated. (See the hard rules in [AGENTS.md](AGENTS.md).) + +| | **AI Topic Scout** | Microsoft Scout / general work agents | +|---|---|---| +| Purpose | **Research-corpus engine** | General work autopilot (M365) | +| Output format | **Fixed house style**, zero setup | Ad-hoc, re-specified every run | +| Cross-run analytics | **Comparable** (same instrument) | None / varies per prompt | +| Persistent corpus | **Files in your git repo** | No research corpus | +| Runtime | **Any agent** (Claude/Codex/Copilot/…) | Vendor-hosted | +| Cost of formatting | **Free** (template) | Paid in tokens, every time | +| Self-host / fork | ✅ MIT | ❌ cloud-only | -1. **Intent → contract.** Refine natural language into `topic.json` (include/exclude rules, taxonomy, queries). -2. **Discover.** Query [OpenAlex](https://openalex.org) + expand citation neighborhoods. +**Bottom line:** use Microsoft Scout (or a chat assistant) to run your *workday*; use Topic Scout when you want a **consistent, analyzable, ownable research workspace that maintains itself** — and looks the same, good, every time. + +--- + +## ✨ What it does + +1. **Intent → contract.** Refine natural language into `topic.json` (include/exclude rules, taxonomy, dashboard sections, search queries). +2. **Discover.** Query [OpenAlex](https://openalex.org) and expand citation neighborhoods. 3. **Rank.** Score candidates with an LLM against the topic contract. -4. **Curate.** Approve into `data/papers.json`; auto-generate notes + synthesis report. -5. **Publish.** Interactive HTML dashboard, research gap analysis, agent task manifests. +4. **Curate.** Approve into `data/papers.json`; auto-generate paper notes + a synthesis report. +5. **Analyze.** Produce an evidence-backed research-gap analysis. +6. **Publish.** One self-contained HTML dashboard — metrics, corpus shape, discovery trend, interactive wiki, citation graph, ranked opportunities. + +```mermaid +flowchart LR + A["🗣️ Plain-language intent"] --> B["📋 topic.json contract"] + B --> C["🔍 Discover · OpenAlex + citation graph"] + C --> D["⚖️ Rank · LLM vs. contract"] + D --> E["✅ Curate · data/papers.json"] + E --> F["📝 Notes + synthesis report"] + E --> G["🧭 Research-gap analysis"] + F --> H["📊 Designed HTML dashboard"] + G --> H +``` --- -## Requirements +## ⚖️ How relevance is judged -- Python 3.9+ -- One of: [Codex CLI](https://github.com/openai/codex) (recommended, no key needed), or `OPENAI_API_KEY`, or `--offline` -- Optional: `git`, a browser for the dashboard +"Rank" is not a single opaque number from a model. The scout keeps **two independent signals** and combines them with transparent math you can tune: + +1. **Cheap keyword heuristic** (offline, from OpenAlex text) — a fast include/exclude filter. +2. **LLM judge** — scores each candidate *independently* on a small rubric, and is **deliberately blinded to the heuristic score** so the two signals stay independent: + - `topical_fit` — matches the include scope / research question + - `evidence_match` — is it the *kind* of evidence the contract wants (`evidence_types`)? + - `rigor` — venue / methodology / citation credibility + - `exclusion_hit` — a hard veto when excluded scope applies + +Those rubric dimensions are combined **deterministically in code** (not by the model) into a 0–10 score, weighted by your `topic.json`, with a gentle recency decay. The model judges criteria; the math does the ranking — so every score is reproducible and auditable. + +**The verdict is a two-threshold band, not one cutoff:** + +| Condition | Verdict | +|---|---| +| `exclusion_hit` | **reject** (veto) | +| score ≥ `accept_hi` **and** heuristic/LLM agree | **accept** (auto-curated when approval isn't required) | +| score ≤ `accept_lo` | **reject** | +| otherwise (mid-score, *or* the two signals disagree) | **uncertain** → stays in the review queue for a human | + +That uncertainty band is the key: the papers most likely to be misjudged — borderline scores, or where the keyword filter and the LLM disagree — are surfaced for review instead of being silently accepted or dropped. + +Tune any of it in `topic.json` (all optional, sensible defaults): + +```json +"judging": { + "weights": { "topical_fit": 0.5, "evidence_match": 0.3, "rigor": 0.2 }, + "recency": { "decay_per_year": 0.03, "floor": 0.75, "unknown_year": 0.9 }, + "accept_hi": 7.0, "accept_lo": 4.0, "min_confidence": 0.35 +} +``` + +Weights are normalized automatically; `accept_hi`/`accept_lo` set the band; `min_confidence` is how much the two signals must agree to auto-accept. Every candidate in `data/candidates.json` carries its `rubric`, `relevance_score`, `relevance_confidence`, and `relevance_verdict` for inspection. + +### Cheap metric vs. the LLM judge — cutting tokens ~90% + +LLM judging costs tokens, so the goal is to lean on the **free deterministic metric** and call the LLM as little as possible without losing quality. `make eval` measures exactly how well the cheap metric tracks the LLM judge (treating the LLM's score as ground truth) on any labeled `candidates.json`, using rank-based metrics (Spearman, ROC-AUC, precision@K, NDCG, top-K overlap): + +```bash +make eval # bundled 540-paper example +make eval REPORT=reports/metric_eval.md # write a findings report +python3 scripts/eval_metric.py --input data/candidates.json +``` + +On the bundled 540-paper corpus, two findings stand out (see [`docs/metric-vs-llm-eval.md`](docs/metric-vs-llm-eval.md)): + +- **The cheap metric already agrees with the LLM on what to *drop*.** Its "clearly off-topic" bucket held **485 of 540 candidates and 0 of the 8 relevant papers** — so auto-dropping it before the LLM runs saves **~90% of LLM calls with 100% of relevant papers retained**. A higher-AUC hybrid gate pushes that to **~94%**. This is the token lever: the LLM only judges the ambiguous survivors, not the obvious mass. +- **No single cheap metric matches the LLM's *fine* ranking** (7 vs 10 among relevant papers) — that still needs the model. Exact include-phrase matching is high-precision but coarse (few distinct values); TF-IDF/BM25 add resolution but rank noise; naive token-coverage is *worse than random* (it rewards topically-adjacent-but-wrong papers — you must IDF-weight). + +**Takeaway baked into the design:** use the cheap metric as a high-recall prefilter/gate to eliminate the obvious, and spend LLM tokens only on the uncertainty band. Because the right weights are corpus-dependent, `make eval` is meant to be **re-run per topic** to calibrate — the harness is the deliverable, not any single number. + +### The prefilter gate (this is where the tokens are actually saved) + +`scout` applies that finding automatically: before any LLM call, it scores candidates with the cheap metric and **drops the obvious-off-topic tail**, sending only the survivors to the judge. On the 540-paper example this sends **55 papers to the LLM instead of 540 — ~90% fewer calls, with all 8 relevant papers retained.** Dropped candidates stay in `data/candidates.json` marked `relevance_verdict: "prefiltered"` (nothing is silently lost), and a `keep_min` floor stops the gate from ever emptying a sparse topic. + +It's on by default (`current` scorer, threshold `0.0` = "no positive topical evidence"). Control it per run or in `topic.json`: + +```bash +make scout # prefilter on by default +python3 scripts/scout.py --no-prefilter # judge every candidate (old behavior) +python3 scripts/scout.py --prefilter-scorer hybrid --prefilter-score 3.2 +``` + +```json +"judging": { "prefilter": { "enabled": true, "scorer": "current", "threshold": 0.0, "keep_min": 5 } } +``` + +**Auto-derive the threshold from your own data.** After one full run (LLM judges everything once), let `make eval` compute the highest zero-recall-loss threshold and write it back into `topic.json`: + +```bash +python3 scripts/eval_metric.py --recommend --scorer hybrid --write-topic topic.json +``` + +Subsequent scouts then prefilter with a threshold calibrated to *your* corpus — turning the one-time labeling cost into a permanent ~90% token discount. + +--- + +## 🎨 The house style (what ships for free) + +The generated `topic-dashboard.html` is a single, dependency-free file styled as a **research broadsheet**: + +- **Metrics strip** — corpus size, categories, scout runs, tokens, spend. +- **Corpus shape** — papers per taxonomy category, as a bar chart. +- **Discovery trend** — papers accepted per scout run over time. +- **Interactive wiki** — one navigable page per paper, cross-linked by shared terms. +- **Citation graph** — a canvas force-graph of the corpus, colored by category. +- **Opportunities column** — ranked, evidence-backed research gaps. + +Alongside it: `reports/research_report.md` (synthesis) and `reports/papers/*.md` (one consistent note per paper). You do not design any of this. It is the same, every topic, every run. + +### Theming — make it yours without touching code + +The dashboard reads an optional `theme` block from `topic.json`, so a topic can carry its own palette and typography. Every key is optional and falls back to the default broadsheet look, so you can override just an accent color or go fully dark: + +```json +"theme": { + "palette": { + "ink": "#e8ecf8", "paper": "#0e1430", "panel": "#161d3d", + "line": "#2a355e", "muted": "#93a0c8", + "accent": "#ff2e88", "accent2": "#22d3ee" + }, + "fonts": { + "display": "\"Playfair Display\", Georgia, serif", + "body": "\"Inter\", system-ui, sans-serif" + }, + "category_colors": ["#ff2e88", "#22d3ee", "#f5c542", "#a78bfa"] +} +``` + +*(That example is a full dark theme; the snippet above renders as shown below.)* + +- **`palette`** — `ink` (**text color** + top rule), `paper` (page background), `panel` (cards / metric cells), `line` (borders, bar tracks, gridlines), `muted` (secondary text), `accent` / `accent2` (ranking highlights). For a dark theme, set `paper`/`panel`/`line` dark **and** `ink` light (it's the text). The citation-graph module stays a dark inset by design in every theme. +- **`fonts`** — `display` (headings) and `body` (everything else). Any CSS `font-family` stack; use web-safe fonts to keep the file self-contained. +- **`category_colors`** — the series palette used for taxonomy bars, the trend chart, and the citation graph. + +Every key is optional and falls back to the default; the default (broadsheet) look is unchanged if you omit the `theme` block entirely. Then just `make dashboard` — fonts and palette are pure CSS variables, so nothing else in your workflow changes. --- -## Quick start +## 🚀 Quick start ```bash git clone https://github.com/ginaecho/topic-scout.git @@ -37,46 +203,52 @@ make scout # OpenAlex + LLM ranking → data/candidates.json make review # inspect the review queue python3 scripts/accept_candidates.py openalex:W123 openalex:W456 make corpus # paper notes + reports/research_report.md -make opportunities # research gap analysis +make opportunities # research-gap analysis make dashboard # topic-dashboard.html ``` Open `topic-dashboard.html` in a browser when done. +### Requirements + +- Python 3.9+ +- One of: [Codex CLI](https://github.com/openai/codex) (recommended, no key needed), an `OPENAI_API_KEY`, or `--offline` mode +- Optional: `git`, and a browser for the dashboard + ### Provider options ```bash -make init # Codex CLI, interactive +make init # Codex CLI, interactive python3 scripts/init_topic.py --intent "your topic" --provider api # OpenAI API -python3 scripts/init_topic.py --offline # no LLM -python3 scripts/scout.py --accept-score 8.0 # auto-accept threshold -python3 scripts/scout.py --offline # OpenAlex only -QUERY="benchmark X" make scout # targeted supplemental query -make reset # wipe generated workspace +python3 scripts/init_topic.py --offline # deterministic, no LLM +python3 scripts/scout.py --accept-score 8.0 # auto-accept threshold +python3 scripts/scout.py --offline # OpenAlex-only heuristic ranking +QUERY="benchmark X" make scout # targeted supplemental query +make reset # wipe generated workspace ``` --- -## Starting an agent +## 🤖 Starting an agent -Every agent runs the same command surface — pick your runtime: +Every runtime drives the same command surface — pick yours: | Agent | Kickoff | |---|---| -| **Claude Code** | `cd` into the repo. Claude reads `AGENTS.md` automatically. Say: *"Scout papers on ."* | -| **Codex CLI** | `codex` in the repo root. Ask it to run `make init` then follow `AGENTS.md`. | +| **Claude Code** | `cd` into the repo. Claude reads `AGENTS.md` automatically. Say: *"Scout papers on <topic>."* | +| **Codex CLI** | `codex` in the repo root. Ask it to run `make init`, then follow `AGENTS.md`. | | **GitHub Copilot (GHCP)** | Open the repo, run `make init`, then `python3 scripts/orchestrate.py emit --mode copilot`. Copilot follows `data/copilot_tasks.json`. | | **Copilot CLI** | `gh copilot` in the repo. After `make init`, emit `--mode copilot-cli` and execute tasks in order. | | **Microsoft scouting-style** | Emit `--mode microsoft-scouting`; consume `data/microsoft-scouting_tasks.json`. | -| **Claw / Swarm** | Emit `--mode claw` or `--mode swarm`; coordinator dispatches roles under `agents/`. | +| **Claw / Swarm** | Emit `--mode claw` or `--mode swarm`; the coordinator dispatches roles under `agents/`. | Full per-agent instructions: **[AGENTS.md](AGENTS.md)**. Active topic contract (generated by `make init`): **`TOPIC_AGENTS.md`**. --- -## Scouting for papers or research topics +## 🔁 Keeping a topic fresh -The scout is topic-scoped. To change topics: +The scout is topic-scoped. To keep an existing corpus current, just re-run `make scout` — it deduplicates against `data/papers.json` and appends only new candidates. To switch topics: ```bash make reset # clear the generated workspace @@ -84,11 +256,9 @@ make init # define a new topic make scout # discover ``` -To keep the corpus fresh on an existing topic, just re-run `make scout` — it deduplicates against `data/papers.json` and appends candidates. - --- -## Outputs +## 📦 Outputs | Artifact | Purpose | |---|---| @@ -97,16 +267,30 @@ To keep the corpus fresh on an existing topic, just re-run `make scout` — it d | `agents/*.md` | Per-role briefs (coordinator, scout, reviewer, …) | | `data/candidates.json` | Ranked review queue | | `data/papers.json` | Accepted corpus + scout history | -| `reports/research_report.md` | Synthesis | -| `data/research_opportunities.json` | Evidence-backed gaps | -| `topic-dashboard.html` | Interactive dashboard | +| `reports/research_report.md` | Synthesis report | +| `reports/papers/*.md` | One consistent note per paper | +| `data/research_opportunities.json` | Evidence-backed research gaps | +| `topic-dashboard.html` | Self-contained interactive dashboard | | `data/{claw,swarm,copilot,copilot-cli,microsoft-scouting}_tasks.json` | Runtime manifests | -Example workspace: `examples/ai-in-hiring-processes/`. +Worked example: [`examples/ai-in-hiring-processes/`](examples/ai-in-hiring-processes/) — a full topic workspace, dashboard included. --- -## Testing +## 🗺️ Roadmap + +The template is the product, so the roadmap is mostly **more of it**: + +- **Theming knobs** in `topic.json` (palette, typography, category colors) so a topic can carry its own identity without touching code — ✅ **shipped** (see [Theming](#theming--make-it-yours-without-touching-code)). +- **A template gallery** — pick a *layout* at publish time (not just colors): the current broadsheet, a minimal-report theme, a slide-deck export (à la EvaPaper's PPTX), a print/PDF layout. +- **Scheduled scouting** — a cron cadence that keeps a corpus fresh and commits the diff, so the dashboard is always current. +- **More discovery signals** beyond OpenAlex (e.g. optional Semantic Scholar recommendations) behind the same contract. + +Ideas and template contributions welcome — open an issue or PR. + +--- + +## 🧪 Testing ```bash make test @@ -114,6 +298,10 @@ make test --- -## License +## 📄 License MIT — see [LICENSE](LICENSE). + +--- + +Keywords: research paper scout · literature review automation · OpenAlex · living literature review · AI research agent · topic monitoring · citation graph · research gap analysis · Claude Code · Codex CLI · GitHub Copilot · agent-agnostic · self-hosted research dashboard · alternative to Microsoft Scout. diff --git a/docs/metric-vs-llm-eval.md b/docs/metric-vs-llm-eval.md new file mode 100644 index 0000000..775cbb7 --- /dev/null +++ b/docs/metric-vs-llm-eval.md @@ -0,0 +1,39 @@ +# Deterministic metric vs. LLM judge + +- Source: `examples/ai-in-hiring-processes/data/candidates.json` +- Labeled candidates: **540** (**8** relevant at LLM score ≥ 7) +- Ground truth: the LLM judge's `relevance_score`. +- Metrics are rank-based / scale-free (higher is better; AUC 0.5 = random, 1.0 = perfect ranking). + +| scorer | spearman | auc | precision@10 | recall@10 | ndcg@20 | overlap@20 | distinct_values | +|---|---|---|---|---|---|---|---| +| `hybrid` | 0.4272 | 0.9843 | 0.5 | 0.625 | 0.5874 | 0.7 | 529 | +| `bm25` | 0.3127 | 0.9746 | 0.2 | 0.25 | 0.4879 | 0.4 | 511 | +| `tfidf_cosine` | 0.3063 | 0.9746 | 0.4 | 0.5 | 0.7487 | 0.55 | 509 | +| `current` | 0.8169 | 0.9585 | 0.7 | 0.875 | 0.8936 | 0.95 | 3 | +| `tfidf_cites` | 0.1791 | 0.8891 | 0.4 | 0.5 | 0.5361 | 0.35 | 529 | +| `idf_coverage` | 0.1039 | 0.8412 | 0.1 | 0.125 | 0.2941 | 0.3 | 425 | +| `lexical_v2` | -0.2388 | 0.2498 | 0.0 | 0.0 | 0.0047 | 0.0 | 331 | + +`distinct_values` = how many different scores the metric produces across the set — a proxy for discriminative power (1 = useless as a ranker). + +## Token-saving prefilter (current gate: `current`) + +- Auto-drop candidates with score ≤ `0.0` → removes **485/540** candidates. +- LLM judges only the **55** survivors (**10.2%** of candidates) — **~90% fewer LLM calls**. +- Relevant papers retained: **1.0** (8 of 8 kept). + +## Token-saving prefilter (best-AUC gate: `hybrid`) + +- Auto-drop candidates with score ≤ `3.2721118748416806` → removes **506/540** candidates. +- LLM judges only the **34** survivors (**6.3%** of candidates) — **~94% fewer LLM calls**. +- Relevant papers retained: **1.0** (8 of 8 kept). + +## How to read this / recommendations + +- **Two jobs, two metrics.** Dropping the obvious off-topic mass (a *routing* job, measured by AUC) and ranking the shortlist (a *precision* job, measured by precision@K / NDCG / overlap) are different. High-AUC metrics (`hybrid`, `tfidf`, `bm25`) route best; exact include-phrase matching (`current`) ranks the top best but is coarse (few `distinct_values`). +- **Biggest token win = prefilter routing.** Use a high-recall cheap gate to auto-drop candidates the metric is confident are off-topic, and only spend LLM tokens on the survivors + the uncertainty band. That is where the ~90% saving comes from. +- **Don't expect a cheap metric to reproduce fine LLM scores.** Ranking 7-vs-10 among relevant papers needs the model; keep the LLM for the shortlist and the ambiguous band only. +- **IDF-weight any lexical signal.** Naive token-coverage (`lexical_v2`) scores *worse than random* because generic tokens shared by adjacent-but-wrong papers dominate. + +> Numbers are for one topic/corpus and a small positive set; the LLM labels are themselves imperfect. Re-run this harness per corpus to calibrate — the framework is the deliverable, not any single number. diff --git a/scripts/build_dashboard.py b/scripts/build_dashboard.py index bfa81bc..4c09778 100644 --- a/scripts/build_dashboard.py +++ b/scripts/build_dashboard.py @@ -20,7 +20,57 @@ ) -COLORS = ["#e4572e", "#1d6f75", "#f3a712", "#963484", "#4267ac", "#5b8e3e", "#7b6045"] +DEFAULT_THEME = { + "palette": { + "ink": "#17201f", + "paper": "#f3efe4", + "panel": "#fffdf6", + "line": "#c9c1b1", + "muted": "#706c63", + "accent": "#e4572e", + "accent2": "#1d6f75", + }, + "fonts": { + "display": "Georgia, serif", + "body": '"Avenir Next", "Gill Sans", sans-serif', + }, + "category_colors": ["#e4572e", "#1d6f75", "#f3a712", "#963484", "#4267ac", "#5b8e3e", "#7b6045"], +} + +# Backwards-compatible default category palette. +COLORS = DEFAULT_THEME["category_colors"] + + +def resolve_theme(config: dict) -> dict: + """Merge an optional ``theme`` block from topic.json over the defaults. + + Unknown or malformed values fall back to the default, so a partial + ``theme`` (e.g. only ``palette.accent``) is always safe. + """ + theme = config.get("theme") or {} + palette = {**DEFAULT_THEME["palette"], **(theme.get("palette") or {})} + fonts = {**DEFAULT_THEME["fonts"], **(theme.get("fonts") or {})} + category_colors = theme.get("category_colors") + if not isinstance(category_colors, list) or not category_colors: + category_colors = DEFAULT_THEME["category_colors"] + return {"palette": palette, "fonts": fonts, "category_colors": list(category_colors)} + + +def root_css(theme: dict) -> str: + """Render the theme as a CSS custom-property block for :root.""" + palette, fonts = theme["palette"], theme["fonts"] + variables = [ + f"--ink:{palette['ink']}", + f"--paper:{palette['paper']}", + f"--panel:{palette['panel']}", + f"--line:{palette['line']}", + f"--muted:{palette['muted']}", + f"--accent:{palette['accent']}", + f"--accent2:{palette['accent2']}", + f"--font-display:{fonts['display']}", + f"--font-body:{fonts['body']}", + ] + return ":root{" + ";".join(variables) + "}" def terms(paper: dict) -> set[str]: @@ -167,6 +217,8 @@ def wiki_data(config: dict, papers: list[dict], graph: dict, candidates: list[di def build_data() -> dict: config = load_topic() + theme = resolve_theme(config) + category_colors = theme["category_colors"] corpus = load_json(PAPERS_PATH, {"papers": [], "scout_runs": []}) candidates_payload = load_json(CANDIDATES_PATH, {"candidates": [], "generated_at": None, "cost": {}}) papers = corpus["papers"] @@ -178,7 +230,7 @@ def build_data() -> dict: "label": category.title(), "count": 0, "ratio": 0, - "color": COLORS[index % len(COLORS)], + "color": category_colors[index % len(category_colors)], } for index, category in enumerate(config["taxonomy"]) ] @@ -276,25 +328,25 @@ def build_data() -> dict: __TITLE__ · AI Topic Scout
AI Topic Scout · Living research intelligence

@@ -319,7 +371,7 @@ def build_data() -> dict: categories.innerHTML=data.categories.map(x=>`
${x.label}${x.count} (${Math.round(x.ratio*100)}%)
`).join(""); const candidateSummaryText = data.candidate_count ? `Latest scout found ${data.candidate_count} candidates${data.candidate_generated_at ? ` at ${data.candidate_generated_at}` : ""}. Recorded scout usage: ${data.candidate_cost.token_count} tokens, $${data.candidate_cost.money_cost_usd.toFixed(2)} ${data.candidate_cost.currency} via ${data.candidate_cost.provider}${data.candidate_cost.model ? ` (${data.candidate_cost.model})` : ""}.${data.candidate_cost.note ? ` ${data.candidate_cost.note}` : ""}` : "No candidates recorded yet. Run `make scout` to populate this section."; document.getElementById("candidatesummary").textContent=candidateSummaryText; -const candidateItems=(data.candidates||[]).map((x,i)=>`
${String(i+1).padStart(2,"0")}
${x.relevance_score.toFixed(1)}${x.citation_count} cites

${x.title}

${x.year||""}

${x.id} · ${x.discovered_via?.[0]||"n/a"}

${x.relevance_reason||""}

`).join(""); +const candidateItems=(data.candidates||[]).map((x,i)=>`
${String(i+1).padStart(2,"0")}
${x.relevance_score.toFixed(1)}${x.relevance_verdict?`${x.relevance_verdict}`:""}${x.relevance_confidence!=null?`conf ${x.relevance_confidence.toFixed(2)}`:""}${x.citation_count} cites

${x.title}

${x.year||""}

${x.id} · ${x.discovered_via?.[0]||"n/a"}

${x.relevance_reason||""}

`).join(""); document.getElementById("candidates").innerHTML=candidateItems||"

No candidate rows loaded.

"; document.getElementById("trendlegend").innerHTML=data.categories.filter(x=>x.count||data.runs.some(r=>(r.cumulative_topics||{})[x.id])).map(x=>`${x.label}`).join(""); const ts=document.getElementById("trend"),runs=data.runs,W=820,H=330,pad={l:42,r:16,t:12,b:40},visibleTrendCategories=data.categories.filter(x=>x.count||runs.some(r=>(r.cumulative_topic_ratios||{})[x.id])),trendMax=1;let chart=``;const tx=i=>pad.l+(runs.length===1?((W-pad.l-pad.r)/2):i*(W-pad.l-pad.r)/Math.max(1,runs.length-1)),ty=v=>H-pad.b-v*(H-pad.t-pad.b)/trendMax,dotOffset=(i,v,catId)=>{const same=visibleTrendCategories.filter(cat=>Math.abs((((runs[i].cumulative_topic_ratios||{})[cat.id]||0)-v))<1e-9);if(same.length<=1)return 0;const idx=same.findIndex(cat=>cat.id===catId);return (idx-(same.length-1)/2)*10;};for(let i=0;i<=4;i++){const v=i/4;chart+=`${Math.round(v*100)}%`;}if(runs.length){runs.forEach((r,i)=>{chart+=`${r.date.slice(5)}`;});visibleTrendCategories.forEach(cat=>{const pts=runs.map((r,i)=>`${tx(i)},${ty((r.cumulative_topic_ratios||{})[cat.id]||0)}`).join(" ");if(runs.length>1)chart+=``;runs.forEach((r,i)=>{const value=(r.cumulative_topic_ratios||{})[cat.id]||0,y=ty(value),x=tx(i)+dotOffset(i,value,cat.id);chart+=``;});});}ts.innerHTML=chart; @@ -344,8 +396,11 @@ def main() -> int: DATA_DIR.mkdir(parents=True, exist_ok=True) write_json(DATA_DIR / "dashboard.json", payload) embedded = json.dumps(payload, separators=(",", ":"), ensure_ascii=True).replace(" list[str]: + return [ + token + for token in re.findall(r"[a-z0-9]+", (text or "").lower()) + if len(token) >= 3 and token not in STOPWORDS + ] + + +def paper_text(paper: dict) -> tuple[list[str], list[str]]: + """Return (title_tokens, body_tokens) for a candidate.""" + title = tokens(paper.get("title", "")) + body = tokens(paper.get("abstract", "")) + tokens(" ".join(paper.get("topics", []) or [])) + return title, body + + +def contract_terms(config: dict) -> list[str]: + """Flatten the topic contract into a weighted query token list.""" + parts: list[str] = [] + # Include phrases and the research question carry the most signal. + for phrase in config.get("include", []): + parts += tokens(phrase) * 3 + parts += tokens(config.get("topic", "")) * 2 + parts += tokens(config.get("research_question", "")) * 2 + for query in config.get("search_queries", []): + parts += tokens(query) + for term in config.get("taxonomy", []): + parts += tokens(term) + return parts + + +def exclude_terms(config: dict) -> set[str]: + out: set[str] = set() + for phrase in config.get("exclude", []): + out.update(tokens(phrase)) + return out + + +# --------------------------------------------------------------------------- # +# Deterministic scorers (all token-free / no LLM) +# --------------------------------------------------------------------------- # +def score_current(paper: dict, config: dict, ctx: dict) -> float: + candidate = Candidate( + paper.get("id", ""), paper.get("title", ""), paper.get("year"), + paper.get("url", ""), paper.get("doi"), paper.get("abstract", ""), + paper.get("citation_count", 0) or 0, paper.get("topics", []) or [], + paper.get("discovered_via", []) or [], + ) + score, _ = current_relevance(candidate, config) + return score + + +def score_lexical_v2(paper: dict, config: dict, ctx: dict) -> float: + """Token-level include coverage with a title boost and exclude penalty. + + Fixes the current metric's fatal flaw: multi-word include phrases almost + never appear verbatim, so exact-substring counting scores near-constant. + Here each include phrase earns partial credit for the fraction of its + content tokens present, weighted higher in the title. + """ + title, body = ctx["title"], ctx["body"] + title_set, body_set = set(title), set(body) + all_set = title_set | body_set + score = 0.0 + for phrase in config.get("include", []): + toks = [t for t in tokens(phrase)] + if not toks: + continue + covered = sum(t in all_set for t in toks) / len(toks) + in_title = any(t in title_set for t in toks) + score += covered * (1.6 if in_title else 1.0) + topic_toks = set(tokens(config.get("topic", ""))) + score += 0.4 * len(topic_toks & all_set) + score -= 2.0 * len(ctx["exclude"] & all_set) + return score + + +def _tfidf_vectors(config: dict, ctx: dict): + idf = ctx["idf"] + # Query vector from the contract terms. + qvec: dict[str, float] = {} + for token in ctx["query_terms"]: + qvec[token] = qvec.get(token, 0.0) + idf.get(token, ctx["idf_default"]) + return qvec + + +def score_tfidf_cosine(paper: dict, config: dict, ctx: dict) -> float: + idf = ctx["idf"] + qvec = ctx["qvec"] + title, body = ctx["title"], ctx["body"] + tf: dict[str, float] = {} + for token in body: + tf[token] = tf.get(token, 0.0) + 1.0 + for token in title: # title tokens weighted double + tf[token] = tf.get(token, 0.0) + 2.0 + dvec = {token: freq * idf.get(token, ctx["idf_default"]) for token, freq in tf.items()} + num = sum(dvec.get(token, 0.0) * weight for token, weight in qvec.items()) + dnorm = math.sqrt(sum(v * v for v in dvec.values())) + qnorm = math.sqrt(sum(v * v for v in qvec.values())) + if dnorm == 0 or qnorm == 0: + return 0.0 + return num / (dnorm * qnorm) + + +def score_bm25(paper: dict, config: dict, ctx: dict) -> float: + idf = ctx["idf"] + k1, b = 1.5, 0.75 + title, body = ctx["title"], ctx["body"] + doc = body + title + title # title tokens counted thrice overall + if not doc: + return 0.0 + tf: dict[str, float] = {} + for token in doc: + tf[token] = tf.get(token, 0.0) + 1.0 + dl = len(doc) + avgdl = ctx["avgdl"] + score = 0.0 + for token in set(ctx["query_terms"]): + if token not in tf: + continue + term_idf = idf.get(token, ctx["idf_default"]) + freq = tf[token] + score += term_idf * (freq * (k1 + 1)) / (freq + k1 * (1 - b + b * dl / avgdl)) + return score + + +def score_tfidf_cites(paper: dict, config: dict, ctx: dict) -> float: + """TF-IDF cosine blended with a small log-citation prior.""" + base = score_tfidf_cosine(paper, config, ctx) + cites = paper.get("citation_count", 0) or 0 + return base + 0.03 * math.log1p(cites) + + +def score_idf_coverage(paper: dict, config: dict, ctx: dict) -> float: + """IDF-weighted include coverage — the fix for naive lexical_v2. + + Only rare, discriminative include tokens earn credit, so generic tokens + shared by off-topic neighbours stop inflating the score. + """ + idf = ctx["idf"] + title, body = ctx["title"], ctx["body"] + title_set, all_set = set(title), set(title) | set(body) + score = 0.0 + for phrase in config.get("include", []): + toks = tokens(phrase) + if not toks: + continue + weight = sum(idf.get(t, ctx["idf_default"]) for t in toks if t in all_set) + weight += 0.5 * sum(idf.get(t, ctx["idf_default"]) for t in toks if t in title_set) + score += weight + score -= 2.0 * sum(idf.get(t, ctx["idf_default"]) for t in ctx["exclude"] if t in all_set) + return score + + +def score_hybrid(paper: dict, config: dict, ctx: dict) -> float: + """Coarse high-precision include gate + TF-IDF resolution + citation prior. + + The current metric's exact include-phrase match is the dominant term (it + separates on-topic from off-topic well); TF-IDF cosine breaks ties within a + bucket so the metric can actually rank the top papers, and a gentle log-cite + prior nudges credibility. + """ + gate = score_current(paper, config, ctx) + resolution = score_tfidf_cosine(paper, config, ctx) + cites = paper.get("citation_count", 0) or 0 + return 5.0 * gate + resolution + 0.05 * math.log1p(cites) + + +SCORERS = { + "current": score_current, + "lexical_v2": score_lexical_v2, + "idf_coverage": score_idf_coverage, + "tfidf_cosine": score_tfidf_cosine, + "bm25": score_bm25, + "tfidf_cites": score_tfidf_cites, + "hybrid": score_hybrid, +} + + +# --------------------------------------------------------------------------- # +# Rank-based metrics (pure Python, no deps) +# --------------------------------------------------------------------------- # +def _ranks(values: list[float]) -> list[float]: + """Average (fractional) ranks, ties shared.""" + order = sorted(range(len(values)), key=lambda i: values[i]) + ranks = [0.0] * len(values) + i = 0 + while i < len(order): + j = i + while j + 1 < len(order) and values[order[j + 1]] == values[order[i]]: + j += 1 + avg = (i + j) / 2.0 + 1.0 + for k in range(i, j + 1): + ranks[order[k]] = avg + i = j + 1 + return ranks + + +def pearson(xs: list[float], ys: list[float]) -> float: + n = len(xs) + if n == 0: + return 0.0 + mx, my = sum(xs) / n, sum(ys) / n + num = sum((x - mx) * (y - my) for x, y in zip(xs, ys)) + dx = math.sqrt(sum((x - mx) ** 2 for x in xs)) + dy = math.sqrt(sum((y - my) ** 2 for y in ys)) + if dx == 0 or dy == 0: + return 0.0 + return num / (dx * dy) + + +def spearman(xs: list[float], ys: list[float]) -> float: + return pearson(_ranks(xs), _ranks(ys)) + + +def roc_auc(scores: list[float], labels: list[int]) -> float: + """Mann-Whitney AUC with tie handling; labels are 0/1.""" + pos = sum(labels) + neg = len(labels) - pos + if pos == 0 or neg == 0: + return float("nan") + ranks = _ranks(scores) + rank_sum_pos = sum(r for r, l in zip(ranks, labels) if l == 1) + return (rank_sum_pos - pos * (pos + 1) / 2.0) / (pos * neg) + + +def precision_at_k(scores: list[float], labels: list[int], k: int) -> float: + order = sorted(range(len(scores)), key=lambda i: -scores[i])[:k] + return sum(labels[i] for i in order) / max(1, len(order)) + + +def recall_at_k(scores: list[float], labels: list[int], k: int) -> float: + total = sum(labels) + if total == 0: + return float("nan") + order = sorted(range(len(scores)), key=lambda i: -scores[i])[:k] + return sum(labels[i] for i in order) / total + + +def ndcg_at_k(scores: list[float], gains: list[float], k: int) -> float: + order = sorted(range(len(scores)), key=lambda i: -scores[i])[:k] + dcg = sum(gains[idx] / math.log2(rank + 2) for rank, idx in enumerate(order)) + ideal = sorted(gains, reverse=True)[:k] + idcg = sum(g / math.log2(rank + 2) for rank, g in enumerate(ideal)) + return dcg / idcg if idcg else 0.0 + + +def overlap_at_k(scores: list[float], gains: list[float], k: int) -> float: + top_scorer = set(sorted(range(len(scores)), key=lambda i: -scores[i])[:k]) + top_truth = set(sorted(range(len(gains)), key=lambda i: -gains[i])[:k]) + return len(top_scorer & top_truth) / max(1, k) + + +# --------------------------------------------------------------------------- # +# Harness +# --------------------------------------------------------------------------- # +def build_context(papers: list[dict], config: dict) -> dict: + """Precompute IDF and contract query terms shared across scorers.""" + docs_tokens = [] + for paper in papers: + title, body = paper_text(paper) + docs_tokens.append(set(title) | set(body)) + n = len(docs_tokens) + df: dict[str, int] = {} + for doc in docs_tokens: + for token in doc: + df[token] = df.get(token, 0) + 1 + idf = {token: math.log((n + 1) / (count + 1)) + 1.0 for token, count in df.items()} + idf_default = math.log((n + 1) / 1.0) + 1.0 + query_terms = contract_terms(config) + avgdl = sum(len(paper_text(p)[0]) * 2 + len(paper_text(p)[1]) for p in papers) / max(1, n) + ctx = { + "idf": idf, + "idf_default": idf_default, + "query_terms": query_terms, + "exclude": exclude_terms(config), + "avgdl": max(1.0, avgdl), + } + ctx["qvec"] = _tfidf_vectors(config, ctx) + return ctx + + +def cheap_scores(papers: list[dict], config: dict, scorer_name: str = "current") -> dict: + """Compute a token-free deterministic score per candidate id. + + ``current`` reuses the heuristic already attached during discovery; other + scorers (``hybrid``, ``tfidf_cosine``, ``bm25`` …) are computed on the fly + over the candidate set. Used by the scout prefilter gate. + """ + if scorer_name in (None, "heuristic"): + scorer_name = "current" + scorer = SCORERS.get(scorer_name) + if scorer is None: + raise ValueError(f"Unknown prefilter scorer: {scorer_name}") + # Recompute the deterministic score from raw fields (never trust a stored + # relevance_score, which may already hold an LLM verdict on a re-scored file). + ctx = build_context(papers, config) + out = {} + for paper in papers: + title, body = paper_text(paper) + out[paper["id"]] = scorer(paper, config, {**ctx, "title": title, "body": body}) + return out + + +def routing_analysis(labeled: list[dict], config: dict, scorer_name: str = "current") -> dict: + """How much LLM work a cheap prefilter can save at zero recall loss. + + Auto-drops candidates whose cheap score is at or below the highest + threshold that loses no relevant paper, and reports what fraction still + needs the LLM. This is the token-saving lever: the LLM only judges the + ambiguous survivors, not the obvious-off-topic mass. + """ + ctx = build_context(labeled, config) + scorer = SCORERS[scorer_name] + scores, labels = [], [] + for paper in labeled: + title, body = paper_text(paper) + scores.append(scorer(paper, config, {**ctx, "title": title, "body": body})) + labels.append(1 if float(paper["relevance_score"]) >= POSITIVE_THRESHOLD else 0) + total_pos = sum(labels) + # Highest auto-drop threshold that still loses zero relevant papers. + best_threshold, best_dropped = None, 0 + for threshold in sorted(set(scores)): + dropped = [i for i, s in enumerate(scores) if s <= threshold] + if sum(labels[i] for i in dropped) == 0 and len(dropped) > best_dropped: + best_threshold, best_dropped = threshold, len(dropped) + llm_calls = len(scores) - best_dropped + return { + "scorer": scorer_name, + "candidates": len(scores), + "relevant": total_pos, + "auto_drop_threshold": best_threshold, + "auto_dropped": best_dropped, + "llm_calls": llm_calls, + "llm_fraction": round(llm_calls / max(1, len(scores)), 4), + "token_saving": round(1 - llm_calls / max(1, len(scores)), 4), + "recall_retained": 1.0 if total_pos else float("nan"), + } + + +def evaluate(papers: list[dict], config: dict) -> dict: + labeled = [p for p in papers if p.get("relevance_score") is not None] + gains = [float(p["relevance_score"]) for p in labeled] + labels = [1 if g >= POSITIVE_THRESHOLD else 0 for g in gains] + ctx = build_context(labeled, config) + + results = {} + for name, scorer in SCORERS.items(): + per_paper_ctx = [] + scores = [] + for paper in labeled: + title, body = paper_text(paper) + pctx = {**ctx, "title": title, "body": body} + scores.append(scorer(paper, config, pctx)) + results[name] = { + "spearman": round(spearman(scores, gains), 4), + "pearson": round(pearson(scores, gains), 4), + "auc": round(roc_auc(scores, labels), 4), + "precision@10": round(precision_at_k(scores, labels, 10), 4), + "recall@10": round(recall_at_k(scores, labels, 10), 4), + "ndcg@20": round(ndcg_at_k(scores, gains, 20), 4), + "overlap@20": round(overlap_at_k(scores, gains, 20), 4), + "distinct_values": len(set(round(s, 6) for s in scores)), + } + # Rank scorers by AUC to pick the best coarse gate for routing. + def _auc(name): + auc = results[name]["auc"] + return auc if auc == auc else -1.0 + best_gate = max(results, key=_auc) + return { + "n": len(labeled), + "n_positive": sum(labels), + "positive_threshold": POSITIVE_THRESHOLD, + "metrics": results, + "routing": routing_analysis(labeled, config, "current"), + "routing_best_auc": routing_analysis(labeled, config, best_gate), + } + + +def render_report(summary: dict, source: str) -> str: + metrics = summary["metrics"] + order = ["spearman", "auc", "precision@10", "recall@10", "ndcg@20", "overlap@20", "distinct_values"] + header = "| scorer | " + " | ".join(order) + " |" + divider = "|" + "---|" * (len(order) + 1) + rows = [] + # Rank scorers by AUC (nan last), then Spearman. + def sort_key(item): + m = item[1] + auc = m["auc"] if m["auc"] == m["auc"] else -1 # nan guard + return (-auc, -m["spearman"]) + for name, m in sorted(metrics.items(), key=sort_key): + cells = [f"{m[k]}" for k in order] + rows.append(f"| `{name}` | " + " | ".join(cells) + " |") + lines = [ + "# Deterministic metric vs. LLM judge", + "", + f"- Source: `{source}`", + f"- Labeled candidates: **{summary['n']}** " + f"(**{summary['n_positive']}** relevant at LLM score ≥ {summary['positive_threshold']:.0f})", + "- Ground truth: the LLM judge's `relevance_score`.", + "- Metrics are rank-based / scale-free (higher is better; AUC 0.5 = random," + " 1.0 = perfect ranking).", + "", + header, + divider, + *rows, + "", + "`distinct_values` = how many different scores the metric produces across the" + " set — a proxy for discriminative power (1 = useless as a ranker).", + ] + + for label, route in (("current", summary.get("routing")), ("best-AUC", summary.get("routing_best_auc"))): + if not route: + continue + saving = route["token_saving"] * 100 + lines += [ + "", + f"## Token-saving prefilter ({label} gate: `{route['scorer']}`)", + "", + f"- Auto-drop candidates with score ≤ `{route['auto_drop_threshold']}` " + f"→ removes **{route['auto_dropped']}/{route['candidates']}** candidates.", + f"- LLM judges only the **{route['llm_calls']}** survivors " + f"(**{route['llm_fraction'] * 100:.1f}%** of candidates) — " + f"**~{saving:.0f}% fewer LLM calls**.", + f"- Relevant papers retained: **{route['recall_retained']}** " + f"({route['relevant']} of {route['relevant']} kept).", + ] + lines += [ + "", + "## How to read this / recommendations", + "", + "- **Two jobs, two metrics.** Dropping the obvious off-topic mass (a *routing* job," + " measured by AUC) and ranking the shortlist (a *precision* job, measured by" + " precision@K / NDCG / overlap) are different. High-AUC metrics (`hybrid`, `tfidf`," + " `bm25`) route best; exact include-phrase matching (`current`) ranks the top best but" + " is coarse (few `distinct_values`).", + "- **Biggest token win = prefilter routing.** Use a high-recall cheap gate to auto-drop" + " candidates the metric is confident are off-topic, and only spend LLM tokens on the" + " survivors + the uncertainty band. That is where the ~90% saving comes from.", + "- **Don't expect a cheap metric to reproduce fine LLM scores.** Ranking 7-vs-10 among" + " relevant papers needs the model; keep the LLM for the shortlist and the ambiguous" + " band only.", + "- **IDF-weight any lexical signal.** Naive token-coverage (`lexical_v2`) scores *worse" + " than random* because generic tokens shared by adjacent-but-wrong papers dominate.", + "", + "> Numbers are for one topic/corpus and a small positive set; the LLM labels are" + " themselves imperfect. Re-run this harness per corpus to calibrate — the framework" + " is the deliverable, not any single number.", + ] + return "\n".join(lines) + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--input", help="candidates.json with LLM relevance_score labels") + parser.add_argument("--topic", help="topic.json contract for the dataset") + parser.add_argument("--report", help="write the markdown report to this path") + parser.add_argument("--json", action="store_true", help="print the raw metrics as JSON") + parser.add_argument( + "--recommend", + action="store_true", + help="Print a judging.prefilter block with an auto-derived zero-recall-loss threshold", + ) + parser.add_argument("--scorer", help="Prefilter scorer to tune (default: highest AUC)") + parser.add_argument("--write-topic", help="Write the recommended prefilter block into this topic.json") + args = parser.parse_args() + + input_path = Path(args.input) if args.input else EXAMPLE_DIR / "data" / "candidates.json" + topic_path = Path(args.topic) if args.topic else input_path.parent.parent / "topic.json" + if not topic_path.exists(): + topic_path = EXAMPLE_DIR / "topic.json" + + papers = json.loads(input_path.read_text(encoding="utf-8")).get("candidates", []) + config = json.loads(topic_path.read_text(encoding="utf-8")) + summary = evaluate(papers, config) + + if args.json: + print(json.dumps(summary, indent=2)) + + if args.recommend or args.write_topic: + metrics = summary["metrics"] + scorer = args.scorer or max( + metrics, key=lambda n: metrics[n]["auc"] if metrics[n]["auc"] == metrics[n]["auc"] else -1.0 + ) + labeled = [p for p in papers if p.get("relevance_score") is not None] + route = routing_analysis(labeled, config, scorer) + threshold = route["auto_drop_threshold"] + block = { + "enabled": threshold is not None, + "scorer": scorer, + "threshold": threshold if threshold is not None else 0.0, + } + print("Recommended judging.prefilter (auto-derived, zero recall loss):") + print(json.dumps(block, indent=2)) + if threshold is not None: + print( + f"→ drops {route['auto_dropped']}/{route['candidates']} candidates before the LLM " + f"(~{route['token_saving'] * 100:.0f}% fewer LLM calls), keeping all " + f"{route['relevant']} relevant papers." + ) + else: + print("→ no zero-recall-loss threshold found for this scorer; prefilter left disabled.") + if args.write_topic: + topic_out = Path(args.write_topic) + topic_cfg = json.loads(topic_out.read_text(encoding="utf-8")) + topic_cfg.setdefault("judging", {})["prefilter"] = block + topic_out.write_text(json.dumps(topic_cfg, indent=2) + "\n", encoding="utf-8") + print(f"Wrote judging.prefilter into {topic_out}") + return 0 + + try: + source = str(input_path.resolve().relative_to(ROOT)) + except ValueError: + source = str(input_path) + report = render_report(summary, source) + print(report) + if args.report: + report_path = Path(args.report) + report_path.parent.mkdir(parents=True, exist_ok=True) + report_path.write_text(report + "\n", encoding="utf-8") + print(f"\nWrote {report_path}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/judging.py b/scripts/judging.py new file mode 100644 index 0000000..67acf0a --- /dev/null +++ b/scripts/judging.py @@ -0,0 +1,182 @@ +#!/usr/bin/env python3 +"""Deterministic relevance aggregation for scout judging. + +The LLM judge scores each candidate on a small rubric (0..1 per dimension); +this module turns that rubric into a single ranked score, a confidence signal, +and an accept / uncertain / reject verdict. Keeping the ranking math here (not +in the model) makes it transparent, reproducible, and tunable per topic via a +``judging`` block in ``topic.json``. +""" + +from __future__ import annotations + +# Rubric dimensions the judge scores independently, each in [0, 1]. +RUBRIC_DIMENSIONS = ("topical_fit", "evidence_match", "rigor") + +DEFAULT_JUDGING = { + # Weights applied to the rubric dimensions; normalized to sum to 1. + "weights": {"topical_fit": 0.5, "evidence_match": 0.3, "rigor": 0.2}, + # Gentle linear recency decay, expressed on the 0..1 multiplier scale. + "recency": {"decay_per_year": 0.03, "floor": 0.75, "unknown_year": 0.9}, + # Two-threshold band on the 0..10 relevance scale. + "accept_hi": 7.0, + "accept_lo": 4.0, + # Minimum heuristic/LLM concordance required to auto-accept. + "min_confidence": 0.35, + # Divisor that squashes the unbounded heuristic score into [0, 1]. + "heuristic_scale": 6.0, + # Cheap pre-LLM gate: drop candidates whose deterministic score is at or + # below `threshold` before spending any LLM tokens. `threshold: 0.0` on the + # default `current` scorer drops candidates with no positive topical + # evidence — empirically a zero-recall-loss cut (see docs/metric-vs-llm-eval.md). + "prefilter": { + "enabled": True, + "scorer": "current", + "threshold": 0.0, + "keep_min": 5, + }, +} + + +def _clamp(value: float, low: float = 0.0, high: float = 1.0) -> float: + return max(low, min(high, value)) + + +def resolve_judging(config: dict, *, accept_hi: float | None = None) -> dict: + """Merge an optional ``judging`` block from topic.json over the defaults. + + Partial or malformed values fall back to the default, so a topic that only + tweaks, say, ``weights.topical_fit`` is always safe. ``accept_hi`` (e.g. a + ``--accept-score`` CLI override) wins over both when provided. + """ + block = config.get("judging") or {} + + weights_in = block.get("weights") or {} + weights = {} + for dimension in RUBRIC_DIMENSIONS: + raw = weights_in.get(dimension, DEFAULT_JUDGING["weights"][dimension]) + try: + weights[dimension] = max(0.0, float(raw)) + except (TypeError, ValueError): + weights[dimension] = DEFAULT_JUDGING["weights"][dimension] + total = sum(weights.values()) + if total <= 0: + weights = dict(DEFAULT_JUDGING["weights"]) + total = sum(weights.values()) + weights = {dimension: value / total for dimension, value in weights.items()} + + recency_in = block.get("recency") or {} + recency = {**DEFAULT_JUDGING["recency"]} + for key in recency: + if key in recency_in: + try: + recency[key] = float(recency_in[key]) + except (TypeError, ValueError): + pass + + def _number(key: str) -> float: + try: + return float(block[key]) + except (KeyError, TypeError, ValueError): + return float(DEFAULT_JUDGING[key]) + + prefilter_in = block.get("prefilter") or {} + prefilter = {**DEFAULT_JUDGING["prefilter"]} + if "enabled" in prefilter_in: + prefilter["enabled"] = bool(prefilter_in["enabled"]) + if prefilter_in.get("scorer"): + prefilter["scorer"] = str(prefilter_in["scorer"]) + for key in ("threshold", "keep_min"): + if key in prefilter_in: + try: + prefilter[key] = float(prefilter_in[key]) + except (TypeError, ValueError): + pass + prefilter["keep_min"] = max(0, int(prefilter["keep_min"])) + + resolved = { + "weights": weights, + "recency": recency, + "accept_hi": _number("accept_hi"), + "accept_lo": _number("accept_lo"), + "min_confidence": _clamp(_number("min_confidence")), + "heuristic_scale": max(1e-6, _number("heuristic_scale")), + "prefilter": prefilter, + } + if accept_hi is not None: + resolved["accept_hi"] = float(accept_hi) + # Keep the band coherent even if a user inverts the thresholds. + resolved["accept_lo"] = min(resolved["accept_lo"], resolved["accept_hi"]) + return resolved + + +def recency_weight(year, reference_year, recency: dict) -> float: + """Gentle linear decay: recent papers keep full credit, older ones taper + down to a floor. Missing years get a neutral multiplier.""" + if not year: + return _clamp(recency.get("unknown_year", 0.9)) + age = max(0, int(reference_year) - int(year)) + weight = 1.0 - recency.get("decay_per_year", 0.0) * age + return _clamp(weight, recency.get("floor", 0.0), 1.0) + + +def normalize_heuristic(score: float, scale: float) -> float: + """Squash the unbounded keyword heuristic score into [0, 1].""" + try: + return _clamp(float(score) / scale) + except (TypeError, ValueError): + return 0.0 + + +def aggregate( + rubric: dict, + judging: dict, + *, + year=None, + reference_year: int, + heuristic_score: float = 0.0, +) -> dict: + """Combine a judged rubric into a ranked score, confidence, and verdict. + + Returns a dict with ``relevance_score`` (0..10), ``relevance_confidence`` + (0..1), ``relevance_verdict`` (accept | uncertain | reject), and the + normalized components used, so every number is auditable. + """ + weights = judging["weights"] + dims = { + dimension: _clamp(float(rubric.get(dimension, 0.0) or 0.0)) + for dimension in RUBRIC_DIMENSIONS + } + exclusion_hit = bool(rubric.get("exclusion_hit", False)) + + base = sum(weights[dimension] * dims[dimension] for dimension in RUBRIC_DIMENSIONS) + recency = recency_weight(year, reference_year, judging["recency"]) + combined = base * recency # 0..1 + score10 = round(combined * 10.0, 2) + + # Confidence = agreement between the independent cheap heuristic and the + # LLM rubric. Strong disagreement routes a candidate to human review. + heuristic_norm = normalize_heuristic(heuristic_score, judging["heuristic_scale"]) + confidence = round(_clamp(1.0 - abs(heuristic_norm - combined)), 3) + + if exclusion_hit: + verdict = "reject" + score10 = 0.0 + elif score10 >= judging["accept_hi"] and confidence >= judging["min_confidence"]: + verdict = "accept" + elif score10 <= judging["accept_lo"]: + verdict = "reject" + else: + verdict = "uncertain" + + return { + "relevance_score": score10, + "relevance_confidence": confidence, + "relevance_verdict": verdict, + "relevance_components": { + **dims, + "exclusion_hit": exclusion_hit, + "recency_weight": round(recency, 3), + "heuristic_norm": round(heuristic_norm, 3), + }, + } diff --git a/scripts/scout.py b/scripts/scout.py index d7b555d..9fc7e1c 100644 --- a/scripts/scout.py +++ b/scripts/scout.py @@ -9,6 +9,8 @@ from datetime import datetime, timezone from costs import zero_cost +from eval_metric import cheap_scores +from judging import aggregate, resolve_judging from paper_graph import discover from scout_llm import ScoutModelError, score_candidates from workspace import CANDIDATES_PATH, PAPERS_PATH, load_json, load_topic, write_json @@ -16,6 +18,31 @@ DEFAULT_ACCEPT_SCORE = 7.0 +def select_for_llm(candidates, config, prefilter, limit): + """Rank candidates by a cheap score and split into (pool, prefiltered). + + ``pool`` is sent to the LLM; ``prefiltered`` is the below-threshold, + obviously-off-topic tail that is dropped *before* any LLM call — the token + saving. Candidates above the threshold but beyond ``limit`` are left as-is + (heuristic only), matching the prior over-cap behaviour. A ``keep_min`` + floor guarantees the run never empties out on a sparse topic. + """ + scores = cheap_scores(candidates, config, prefilter.get("scorer", "current")) + ranked = sorted(candidates, key=lambda c: -scores.get(c["id"], 0.0)) + if not prefilter.get("enabled", True): + return ranked[:limit], [], scores + threshold = prefilter.get("threshold", 0.0) + above = [c for c in ranked if scores.get(c["id"], 0.0) > threshold] + below = [c for c in ranked if scores.get(c["id"], 0.0) <= threshold] + keep_min = int(prefilter.get("keep_min", 0) or 0) + if len(above) < keep_min: # rescue the best of the tail so the run isn't empty + rescued = ranked[:keep_min] + rescued_ids = {c["id"] for c in rescued} + above = rescued + below = [c for c in ranked if c["id"] not in rescued_ids] + return above[:limit], below, scores + + def main() -> int: parser = argparse.ArgumentParser() parser.add_argument("--query") @@ -44,6 +71,22 @@ def main() -> int: action="store_true", help="Skip model-backed scout scoring and keep OpenAlex-only heuristic ranking", ) + prefilter_group = parser.add_mutually_exclusive_group() + prefilter_group.add_argument( + "--prefilter", + dest="prefilter", + action="store_true", + default=None, + help="Drop obviously-off-topic candidates with a cheap score before LLM judging", + ) + prefilter_group.add_argument( + "--no-prefilter", + dest="prefilter", + action="store_false", + help="Send every discovered candidate to the LLM (disable the cheap gate)", + ) + parser.add_argument("--prefilter-score", type=float, help="Prefilter drop threshold override") + parser.add_argument("--prefilter-scorer", help="Prefilter scorer override (current, hybrid, tfidf_cosine, bm25)") args = parser.parse_args() config = load_topic() accept_score = args.accept_score @@ -102,7 +145,18 @@ def main() -> int: print("Scout cost: 0 tokens, $0.00 USD.") print("topic.json requested the legacy OpenAlex-only scout path; rerun with --provider codex or --provider api to require an LLM.") return 0 - llm_candidates = result["candidates"][: max(0, args.llm_candidates)] + judging = resolve_judging(config, accept_hi=args.accept_score) + prefilter = dict(judging["prefilter"]) + if args.prefilter is not None: + prefilter["enabled"] = args.prefilter + if args.prefilter_score is not None: + prefilter["threshold"] = args.prefilter_score + if args.prefilter_scorer: + prefilter["scorer"] = args.prefilter_scorer + llm_candidates, prefiltered, cheap = select_for_llm( + result["candidates"], config, prefilter, max(0, args.llm_candidates) + ) + result["prefiltered_count"] = len(prefiltered) try: updates, usage = score_candidates( config, @@ -112,14 +166,46 @@ def main() -> int: ) except ScoutModelError as exc: raise SystemExit(f"Scout scoring failed: {exc}") + reference_year = (config.get("years") or {}).get("to") or int(result["generated_at"][:4]) for candidate in result["candidates"]: - candidate["heuristic_relevance_score"] = candidate.get("relevance_score", 0) + # Preserve the independent keyword heuristic before the judge's + # rubric overwrites the ranked score. + heuristic_score = candidate.get("relevance_score", 0) + candidate["heuristic_relevance_score"] = heuristic_score candidate["heuristic_relevance_reason"] = candidate.get("relevance_reason", "") - update = updates.get(candidate["id"]) - if update: - candidate.update(update) + rubric = updates.get(candidate["id"]) + if rubric: + verdict = aggregate( + rubric, + judging, + year=candidate.get("year"), + reference_year=reference_year, + heuristic_score=heuristic_score, + ) + candidate.update(verdict) + candidate["relevance_reason"] = rubric["relevance_reason"] + candidate["rubric"] = { + key: rubric[key] + for key in ("topical_fit", "evidence_match", "rigor", "exclusion_hit") + } + # Mark the cheap-dropped tail (after heuristic fields are preserved) so + # nothing is silently lost and it never auto-accepts. + prefiltered_ids = {candidate["id"] for candidate in prefiltered} + for candidate in result["candidates"]: + if candidate["id"] in prefiltered_ids: + candidate["relevance_verdict"] = "prefiltered" + candidate["relevance_reason"] = ( + f"Cheap {prefilter.get('scorer', 'current')} score " + f"{cheap.get(candidate['id'], 0.0):.2f} <= prefilter threshold " + f"{prefilter.get('threshold', 0.0)}; not LLM-judged." + ) result["candidates"].sort( - key=lambda item: (-item.get("relevance_score", 0), -item.get("citation_count", 0), -(item.get("year") or 0)) + key=lambda item: ( + -item.get("relevance_score", 0), + -item.get("relevance_confidence", 0), + -item.get("citation_count", 0), + -(item.get("year") or 0), + ) ) result["cost"] = usage result["scout_provider"] = provider @@ -135,15 +221,24 @@ def main() -> int: "accepted_ids": [], "accepted_count": 0, "candidate_count": len(result["candidates"]), + "prefiltered_count": result.get("prefiltered_count", 0), "cost": result["cost"], } existing.setdefault("scout_runs", []).append(scout_run) + def _auto_acceptable(candidate: dict) -> bool: + if candidate["id"] in known or not candidate.get("abstract"): + return False + # When the judge produced a verdict, honor the accept/uncertain/reject + # band; the uncertainty band stays in the review queue for a human. + # Otherwise (offline/heuristic path) fall back to the raw threshold. + if "relevance_verdict" in candidate: + return candidate["relevance_verdict"] == "accept" + return candidate["relevance_score"] >= accept_score + if not config["approval_required"] and accept_score is not None: accepted = [ candidate for candidate in result["candidates"] - if candidate["id"] not in known - and candidate["relevance_score"] >= accept_score - and candidate["abstract"] + if _auto_acceptable(candidate) ] if accepted: for candidate in accepted: @@ -166,14 +261,32 @@ def main() -> int: f"Discovered {len(result['candidates'])} candidates " f"({result['new_candidate_count']} not in corpus)." ) + prefiltered_count = result.get("prefiltered_count", 0) + if prefiltered_count: + total = len(result["candidates"]) + judged = total - prefiltered_count + saving = 100 * prefiltered_count / total if total else 0 + print( + f"Prefilter dropped {prefiltered_count}/{total} obvious-off-topic candidates " + f"before LLM scoring ({judged} judged, ~{saving:.0f}% fewer LLM calls)." + ) if discovery_error: print(f"OpenAlex discovery failed; wrote an empty candidate queue. {discovery_error}") + uncertain = [ + candidate for candidate in result["candidates"] + if candidate.get("relevance_verdict") == "uncertain" and candidate["id"] not in known + ] if config["approval_required"]: print(f"Review {CANDIDATES_PATH}; acceptance requires a librarian or human.") elif accepted: - print(f"Accepted {len(accepted)} candidates at score >= {accept_score}.") + print(f"Accepted {len(accepted)} candidates (verdict=accept, score >= {accept_score}).") elif not config["approval_required"]: - print(f"No candidates met the auto-accept threshold of {accept_score}.") + print(f"No candidates met the auto-accept bar (verdict=accept, score >= {accept_score}).") + if uncertain: + print( + f"{len(uncertain)} candidate(s) landed in the uncertainty band; " + f"review {CANDIDATES_PATH} for verdict=uncertain." + ) print( f"Scout cost: {result['cost']['token_count']} tokens, " f"${result['cost']['money_cost_usd']:.2f} {result['cost']['currency']}." diff --git a/scripts/scout_llm.py b/scripts/scout_llm.py index 60a75ee..31ae81d 100644 --- a/scripts/scout_llm.py +++ b/scripts/scout_llm.py @@ -25,10 +25,20 @@ "type": "object", "properties": { "id": {"type": "string", "minLength": 3}, - "relevance_score": {"type": "number", "minimum": 0, "maximum": 10}, + "topical_fit": {"type": "number", "minimum": 0, "maximum": 1}, + "evidence_match": {"type": "number", "minimum": 0, "maximum": 1}, + "rigor": {"type": "number", "minimum": 0, "maximum": 1}, + "exclusion_hit": {"type": "boolean"}, "relevance_reason": {"type": "string", "minLength": 5}, }, - "required": ["id", "relevance_score", "relevance_reason"], + "required": [ + "id", + "topical_fit", + "evidence_match", + "rigor", + "exclusion_hit", + "relevance_reason", + ], "additionalProperties": False, }, } @@ -38,11 +48,18 @@ } SCOUT_INSTRUCTIONS = ( - "You are a research scout reviewer. Score candidate papers against the provided research " - "contract. Use the candidate title, abstract, topics, and provenance only. Be conservative. " - "Prefer low scores when relevance is ambiguous. Penalize items that match excluded scope. " - "Return every provided candidate id exactly once with a relevance_score from 0 to 10 and a " - "short concrete reason grounded in the candidate text." + "You are a research scout reviewer. Judge each candidate paper against the provided research " + "contract on a rubric, using the candidate title, abstract, topics, provenance, and citation " + "count only. Score every candidate independently on its own merits; do not rank candidates " + "relative to each other. For each candidate return four judgments:\n" + "- topical_fit (0..1): how well the work matches the include scope and research question.\n" + "- evidence_match (0..1): whether it is the KIND of evidence the contract wants (see " + "evidence_types/taxonomy): methods, benchmarks, systems, surveys, etc.\n" + "- rigor (0..1): credibility signal from venue, methodology, reproducibility, and citations.\n" + "- exclusion_hit (boolean): true if the work falls within the contract's excluded scope.\n" + "Be conservative: prefer low scores when relevance is ambiguous, and set exclusion_hit=true " + "whenever excluded scope clearly applies. Return every provided candidate id exactly once " + "with a short, concrete reason grounded in the candidate text." ) @@ -51,6 +68,9 @@ class ScoutModelError(RuntimeError): def _candidate_payload(config: dict, candidates: list[dict]) -> str: + # The judge is deliberately blinded to the keyword heuristic score so the + # two relevance signals stay independent (their agreement is used later as + # a confidence signal). Only raw candidate evidence is passed in. compact = [] for item in candidates: compact.append( @@ -62,8 +82,6 @@ def _candidate_payload(config: dict, candidates: list[dict]) -> str: "topics": item.get("topics", [])[:8], "discovered_via": item.get("discovered_via", [])[:4], "citation_count": item.get("citation_count", 0), - "heuristic_relevance_score": item.get("relevance_score", 0), - "heuristic_relevance_reason": item.get("relevance_reason", ""), } ) return json.dumps( @@ -73,6 +91,7 @@ def _candidate_payload(config: dict, candidates: list[dict]) -> str: "goal": config.get("goal"), "include": config.get("include", []), "exclude": config.get("exclude", []), + "evidence_types": config.get("evidence_types", []), "taxonomy": config.get("taxonomy", []), "candidates": compact, }, @@ -80,6 +99,17 @@ def _candidate_payload(config: dict, candidates: list[dict]) -> str: ) +def _rubric_from_row(row: dict) -> dict: + """Extract the judged rubric from one structured-output row.""" + return { + "topical_fit": float(row["topical_fit"]), + "evidence_match": float(row["evidence_match"]), + "rigor": float(row["rigor"]), + "exclusion_hit": bool(row["exclusion_hit"]), + "relevance_reason": row["relevance_reason"].strip(), + } + + def _usage_cost_from_responses(provider: str, model: str, body: dict) -> dict: usage = body.get("usage") or {} input_tokens = int(usage.get("input_tokens", 0) or 0) @@ -159,13 +189,7 @@ def score_candidates_api( parsed = json.loads(_output_text(body)) except json.JSONDecodeError as exc: raise ScoutModelError("OpenAI scout scoring returned invalid structured JSON") from exc - mapping = { - row["id"]: { - "relevance_score": float(row["relevance_score"]), - "relevance_reason": row["relevance_reason"].strip(), - } - for row in parsed.get("candidates", []) - } + mapping = {row["id"]: _rubric_from_row(row) for row in parsed.get("candidates", [])} return mapping, _usage_cost_from_responses("api", selected_model, body) @@ -243,13 +267,7 @@ def score_candidates_codex( continue if event.get("type") == "turn.completed": usage = event.get("usage") or {} - mapping = { - row["id"]: { - "relevance_score": float(row["relevance_score"]), - "relevance_reason": row["relevance_reason"].strip(), - } - for row in parsed.get("candidates", []) - } + mapping = {row["id"]: _rubric_from_row(row) for row in parsed.get("candidates", [])} selected_model = f"codex-cli:{model or 'configured-model'}" return mapping, usage_cost( provider="codex", diff --git a/tests/test_core.py b/tests/test_core.py index 890dae1..e2b4412 100644 --- a/tests/test_core.py +++ b/tests/test_core.py @@ -11,12 +11,21 @@ sys.path.insert(0, str(ROOT / "scripts")) from build_corpus import classify -from build_dashboard import graph_data, wiki_data +from build_dashboard import DEFAULT_THEME, graph_data, resolve_theme, root_css, wiki_data from costs import usage_cost, zero_cost +from judging import DEFAULT_JUDGING, aggregate, recency_weight, resolve_judging +from eval_metric import ( + evaluate as eval_metrics, + precision_at_k, + roc_auc, + routing_analysis, + spearman, +) from init_topic import build_queries, parse_years, slugify from intent_refiner import refine_intent, refine_intent_codex from paper_graph import Candidate, relevance import scout as scout_module +from scout import select_for_llm import analyze_research_gaps as gaps_module import orchestrate as orchestrate_module from scout_llm import score_candidates_api, score_candidates_codex @@ -68,6 +77,342 @@ def test_classification_graph_and_wiki(self): self.assertGreaterEqual(wiki["page_count"], 6) self.assertIn("overview", {page["id"] for page in wiki["pages"]}) + def test_prefilter_gate_splits_pool_and_dropped(self): + config = {"topic": "proof search", "include": ["proof search"], "exclude": []} + # `current` recomputes the heuristic from text, so on-topic abstracts + # score > 0 and off-topic ones score 0. + cands = [ + {"id": "a", "title": "Proof search A", "abstract": "formal proof search method", "topics": []}, + {"id": "b", "title": "Proof search B", "abstract": "proof search benchmark", "topics": []}, + {"id": "c", "title": "Cooking", "abstract": "bread recipe", "topics": []}, + {"id": "d", "title": "Gardening", "abstract": "plant tomatoes", "topics": []}, + ] + prefilter = {"enabled": True, "scorer": "current", "threshold": 0.0, "keep_min": 1} + pool, dropped, _ = select_for_llm(cands, config, prefilter, limit=10) + self.assertEqual({c["id"] for c in pool}, {"a", "b"}) + self.assertEqual({c["id"] for c in dropped}, {"c", "d"}) + # Disabled gate sends everything (up to the limit), nothing dropped. + off = select_for_llm(cands, config, {"enabled": False}, limit=10) + self.assertEqual(len(off[0]), 4) + self.assertEqual(off[1], []) + + def test_scout_prefilter_reduces_llm_pool(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + (root / "data").mkdir(parents=True, exist_ok=True) + papers_path = root / "data" / "papers.json" + candidates_path = root / "data" / "candidates.json" + papers_path.write_text(json.dumps({"papers": [], "scout_runs": []}), encoding="utf-8") + candidates_path.write_text(json.dumps({"candidates": []}), encoding="utf-8") + fake_topic = { + "topic": "AI theorem proving", "goal": "g", "audience": "a", + "include": ["proof search"], "exclude": [], "years": {"from": 2023, "to": 2026}, + "taxonomy": ["proof search"], "evidence_types": ["methods"], + "approval_required": False, "scout_provider": "api", + } + # 6 on-topic (heuristic > 0) + 6 off-topic (heuristic 0); keep_min=5 so + # the 6 survivors clear the floor and the 6 zeros are truly dropped. + cands = [] + for i in range(6): + cands.append({"id": f"y{i}", "title": "Proof search", "year": 2025, + "url": "u", "doi": None, "abstract": "proof search method", + "citation_count": 3, "topics": ["proof search"], + "discovered_via": ["q"], "relevance_score": 2.6, + "relevance_reason": "heuristic"}) + for i in range(6): + cands.append({"id": f"n{i}", "title": "Cooking", "year": 2020, + "url": "u", "doi": None, "abstract": "bread recipe", + "citation_count": 0, "topics": [], "discovered_via": ["q"], + "relevance_score": 0.0, "relevance_reason": "heuristic"}) + fake_result = {"topic": "AI theorem proving", "queries": ["q"], "edges": [], "candidates": cands} + seen = {} + + def fake_score(config, candidates, **kwargs): + seen["ids"] = [c["id"] for c in candidates] + return ({c["id"]: {"topical_fit": 0.9, "evidence_match": 0.9, "rigor": 0.8, + "exclusion_hit": False, "relevance_reason": "fit"} + for c in candidates}, zero_cost()) + + def fake_load_json(path, default): + return json.loads(path.read_text(encoding="utf-8")) if path.exists() else default + + def fake_write_json(path, payload): + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(json.dumps(payload, indent=2, ensure_ascii=True) + "\n", encoding="utf-8") + + with patch.object(scout_module, "discover", return_value=fake_result): + with patch.object(scout_module, "load_topic", return_value=fake_topic): + with patch.object(scout_module, "score_candidates", side_effect=fake_score): + with patch.object(scout_module, "load_json", side_effect=fake_load_json): + with patch.object(scout_module, "write_json", side_effect=fake_write_json): + with patch.object(scout_module, "PAPERS_PATH", papers_path): + with patch.object(scout_module, "CANDIDATES_PATH", candidates_path): + with patch.object(sys, "argv", ["scout.py"]): + self.assertEqual(scout_module.main(), 0) + + # The LLM only saw the 6 on-topic candidates; the 6 zeros were prefiltered. + self.assertEqual(sorted(seen["ids"]), sorted(f"y{i}" for i in range(6))) + queue = json.loads(candidates_path.read_text()) + verdicts = {c["id"]: c.get("relevance_verdict") for c in queue["candidates"]} + self.assertTrue(all(verdicts[f"n{i}"] == "prefiltered" for i in range(6))) + # The judged on-topic candidates carry a real verdict, not "prefiltered". + self.assertTrue(all(verdicts[f"y{i}"] in {"accept", "uncertain", "reject"} for i in range(6))) + self.assertEqual(json.loads(papers_path.read_text())["scout_runs"][0]["prefiltered_count"], 6) + + def test_eval_metric_rank_statistics(self): + self.assertAlmostEqual(spearman([1, 2, 3, 4], [1, 2, 3, 4]), 1.0) + self.assertAlmostEqual(spearman([1, 2, 3, 4], [4, 3, 2, 1]), -1.0) + # Positives ranked strictly above negatives -> AUC 1.0; reversed -> 0.0. + self.assertAlmostEqual(roc_auc([0.1, 0.2, 0.8, 0.9], [0, 0, 1, 1]), 1.0) + self.assertAlmostEqual(roc_auc([0.1, 0.2, 0.8, 0.9], [1, 1, 0, 0]), 0.0) + self.assertAlmostEqual(precision_at_k([0.9, 0.8, 0.1], [1, 1, 0], 2), 1.0) + + def test_eval_metric_routing_retains_recall_and_saves_calls(self): + config = {"topic": "proof search", "include": ["proof search"], "exclude": []} + papers = [ + {"id": "a", "title": "Proof search method", "abstract": "formal proof search", + "topics": [], "citation_count": 5, "year": 2025, "relevance_score": 9.0}, + {"id": "b", "title": "Neural proof search", "abstract": "proof search benchmark", + "topics": [], "citation_count": 2, "year": 2024, "relevance_score": 8.0}, + ] + [ + {"id": f"n{i}", "title": "Cooking recipe", "abstract": "how to bake bread", + "topics": [], "citation_count": 0, "year": 2020, "relevance_score": 0.0} + for i in range(8) + ] + route = routing_analysis(papers, config, "current") + # All 8 off-topic papers auto-dropped; both relevant papers survive to the LLM. + self.assertEqual(route["auto_dropped"], 8) + self.assertEqual(route["llm_calls"], 2) + self.assertEqual(route["recall_retained"], 1.0) + self.assertGreater(route["token_saving"], 0.5) + + def test_eval_metric_evaluate_covers_all_scorers(self): + config = {"topic": "proof search", "include": ["proof search"], "exclude": [], + "search_queries": ["proof search benchmark"], "taxonomy": ["benchmarks"]} + papers = [ + {"id": "a", "title": "Proof search", "abstract": "formal proof search method", + "topics": ["proof"], "citation_count": 5, "year": 2025, "relevance_score": 9.0}, + {"id": "n", "title": "Cooking", "abstract": "bread recipe", + "topics": [], "citation_count": 0, "year": 2020, "relevance_score": 0.0}, + ] + summary = eval_metrics(papers, config) + self.assertEqual(summary["n"], 2) + for name in ("current", "tfidf_cosine", "bm25", "hybrid"): + self.assertIn(name, summary["metrics"]) + self.assertIn("auc", summary["metrics"][name]) + self.assertEqual(summary["routing"]["recall_retained"], 1.0) + + def test_resolve_judging_merges_and_normalizes_weights(self): + judging = resolve_judging({"judging": {"weights": {"topical_fit": 1.0}}}) + # Weights always normalize to sum 1 after merging over defaults. + self.assertAlmostEqual(sum(judging["weights"].values()), 1.0) + self.assertGreater(judging["weights"]["topical_fit"], judging["weights"]["evidence_match"]) + # Empty/absent block reproduces the defaults. + self.assertEqual(resolve_judging({})["weights"], resolve_judging({"judging": {}})["weights"]) + # A malformed (all-zero) weight set falls back to defaults. + fallback = resolve_judging({"judging": {"weights": {"topical_fit": 0, "evidence_match": 0, "rigor": 0}}}) + self.assertAlmostEqual(fallback["weights"]["topical_fit"], DEFAULT_JUDGING["weights"]["topical_fit"]) + # A CLI accept-score override wins and keeps the band coherent. + overridden = resolve_judging({"judging": {"accept_lo": 9.0}}, accept_hi=6.0) + self.assertEqual(overridden["accept_hi"], 6.0) + self.assertLessEqual(overridden["accept_lo"], overridden["accept_hi"]) + + def test_recency_weight_decays_gently_to_a_floor(self): + recency = DEFAULT_JUDGING["recency"] + self.assertEqual(recency_weight(2026, 2026, recency), 1.0) + self.assertAlmostEqual(recency_weight(2021, 2026, recency), 0.85) + # Far-past papers clamp at the floor; missing years get the neutral value. + self.assertEqual(recency_weight(1990, 2026, recency), recency["floor"]) + self.assertEqual(recency_weight(None, 2026, recency), recency["unknown_year"]) + + def test_aggregate_produces_accept_reject_uncertain_and_veto(self): + judging = resolve_judging({}) + # Strong rubric + agreeing heuristic -> accept. + strong = aggregate( + {"topical_fit": 1.0, "evidence_match": 1.0, "rigor": 1.0, "exclusion_hit": False}, + judging, year=2026, reference_year=2026, heuristic_score=6.0, + ) + self.assertEqual(strong["relevance_score"], 10.0) + self.assertEqual(strong["relevance_verdict"], "accept") + # High rubric but the cheap heuristic strongly disagrees -> uncertain. + disputed = aggregate( + {"topical_fit": 0.9, "evidence_match": 0.9, "rigor": 0.9, "exclusion_hit": False}, + judging, year=2026, reference_year=2026, heuristic_score=0.0, + ) + self.assertLess(disputed["relevance_confidence"], judging["min_confidence"]) + self.assertEqual(disputed["relevance_verdict"], "uncertain") + # Low rubric -> reject. + weak = aggregate( + {"topical_fit": 0.2, "evidence_match": 0.2, "rigor": 0.2, "exclusion_hit": False}, + judging, year=2026, reference_year=2026, heuristic_score=1.2, + ) + self.assertEqual(weak["relevance_verdict"], "reject") + # Excluded scope is a hard veto regardless of the other scores. + vetoed = aggregate( + {"topical_fit": 1.0, "evidence_match": 1.0, "rigor": 1.0, "exclusion_hit": True}, + judging, year=2026, reference_year=2026, heuristic_score=6.0, + ) + self.assertEqual(vetoed["relevance_verdict"], "reject") + self.assertEqual(vetoed["relevance_score"], 0.0) + + def test_scout_uses_rubric_verdict_band_for_acceptance(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + (root / "data").mkdir(parents=True, exist_ok=True) + papers_path = root / "data" / "papers.json" + candidates_path = root / "data" / "candidates.json" + papers_path.write_text(json.dumps({"papers": [], "scout_runs": []}), encoding="utf-8") + candidates_path.write_text(json.dumps({"candidates": []}), encoding="utf-8") + + fake_topic = { + "topic": "AI theorem proving", + "goal": "Track proof systems", + "audience": "researchers", + "include": ["proof search"], + "exclude": [], + "years": {"from": 2023, "to": 2026}, + "taxonomy": ["proof search", "verification"], + "evidence_types": ["methods"], + "approval_required": False, + "scout_provider": "api", + } + fake_result = { + "topic": "AI theorem proving", + "queries": ["q"], + "edges": [], + "candidates": [ + { # heuristic agrees with a strong rubric -> accept + "id": "openalex:1", "title": "Accept Me", "year": 2025, + "url": "https://a", "doi": None, "abstract": "proof search method", + "citation_count": 12, "topics": ["proof search"], + "discovered_via": ["query:q"], "relevance_score": 6.0, + "relevance_reason": "heuristic", + }, + { # strong rubric but heuristic disagrees -> uncertain, not accepted + "id": "openalex:2", "title": "Unsure Me", "year": 2025, + "url": "https://b", "doi": None, "abstract": "proof search benchmark", + "citation_count": 3, "topics": ["proof search"], + "discovered_via": ["query:q"], "relevance_score": 0.0, + "relevance_reason": "heuristic", + }, + ], + } + rubrics = { + "openalex:1": {"topical_fit": 1.0, "evidence_match": 1.0, "rigor": 0.9, + "exclusion_hit": False, "relevance_reason": "clear fit"}, + "openalex:2": {"topical_fit": 0.9, "evidence_match": 0.9, "rigor": 0.9, + "exclusion_hit": False, "relevance_reason": "looks aligned"}, + } + + def fake_load_json(path, default): + if not path.exists(): + return default + return json.loads(path.read_text(encoding="utf-8")) + + def fake_write_json(path, payload): + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(json.dumps(payload, indent=2, ensure_ascii=True) + "\n", encoding="utf-8") + + with patch.object(scout_module, "discover", return_value=fake_result): + with patch.object(scout_module, "load_topic", return_value=fake_topic): + with patch.object(scout_module, "score_candidates", return_value=(rubrics, zero_cost())): + with patch.object(scout_module, "load_json", side_effect=fake_load_json): + with patch.object(scout_module, "write_json", side_effect=fake_write_json): + with patch.object(scout_module, "PAPERS_PATH", papers_path): + with patch.object(scout_module, "CANDIDATES_PATH", candidates_path): + with patch.object(sys, "argv", ["scout.py"]): + self.assertEqual(scout_module.main(), 0) + + corpus = json.loads(papers_path.read_text()) + queue = json.loads(candidates_path.read_text()) + verdicts = {c["id"]: c["relevance_verdict"] for c in queue["candidates"]} + # Only the agreed-strong candidate is auto-accepted. + self.assertEqual([p["id"] for p in corpus["papers"]], ["openalex:1"]) + self.assertEqual(verdicts["openalex:1"], "accept") + self.assertEqual(verdicts["openalex:2"], "uncertain") + # The judge's rubric is preserved on the candidate for auditing. + by_id = {c["id"]: c for c in queue["candidates"]} + self.assertIn("rubric", by_id["openalex:2"]) + self.assertEqual(by_id["openalex:1"]["relevance_reason"], "clear fit") + + def test_resolve_theme_merges_partial_override_over_defaults(self): + theme = resolve_theme( + { + "theme": { + "palette": {"accent": "#ff2e88"}, + "fonts": {"body": "Inter, sans-serif"}, + "category_colors": ["#ff2e88", "#22d3ee"], + } + } + ) + # Overridden values win. + self.assertEqual(theme["palette"]["accent"], "#ff2e88") + self.assertEqual(theme["fonts"]["body"], "Inter, sans-serif") + self.assertEqual(theme["category_colors"], ["#ff2e88", "#22d3ee"]) + # Unspecified values fall back to the defaults. + self.assertEqual(theme["palette"]["ink"], DEFAULT_THEME["palette"]["ink"]) + self.assertEqual(theme["fonts"]["display"], DEFAULT_THEME["fonts"]["display"]) + + def test_resolve_theme_defaults_when_absent_or_malformed(self): + self.assertEqual(resolve_theme({}), resolve_theme({"theme": {}})) + # A malformed category_colors value silently falls back. + theme = resolve_theme({"theme": {"category_colors": "not-a-list"}}) + self.assertEqual(theme["category_colors"], DEFAULT_THEME["category_colors"]) + root = root_css(theme) + self.assertTrue(root.startswith(":root{")) + self.assertIn("--accent:", root) + self.assertIn("--font-display:", root) + + def test_dashboard_applies_theme_palette_and_fonts(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + (root / "data").mkdir(parents=True, exist_ok=True) + (root / "reports").mkdir(parents=True, exist_ok=True) + (root / "topic.json").write_text( + json.dumps( + { + "topic": "Themed Topic", + "goal": "Verify theming", + "audience": "devs", + "taxonomy": ["alpha", "beta"], + "include": ["x"], + "exclude": [], + "years": {"from": 2023, "to": 2026}, + "theme": { + "palette": {"ink": "#0b1021", "accent": "#ff2e88"}, + "fonts": {"body": "Inter, system-ui, sans-serif"}, + "category_colors": ["#ff2e88", "#22d3ee"], + }, + } + ), + encoding="utf-8", + ) + (root / "data" / "papers.json").write_text( + json.dumps({"papers": [], "scout_runs": []}), encoding="utf-8" + ) + (root / "data" / "candidates.json").write_text( + json.dumps({"candidates": [], "generated_at": None, "cost": {}}), + encoding="utf-8", + ) + env = dict(os.environ, TOPIC_SCOUT_ROOT=directory) + subprocess.run( + [sys.executable, str(ROOT / "scripts" / "build_dashboard.py")], + check=True, + env=env, + capture_output=True, + text=True, + ) + html = (root / "topic-dashboard.html").read_text(encoding="utf-8") + self.assertIn("--ink:#0b1021", html) + self.assertIn("--accent:#ff2e88", html) + self.assertIn("--font-body:Inter, system-ui, sans-serif", html) + # Unspecified palette entries keep their defaults. + self.assertIn(f"--muted:{DEFAULT_THEME['palette']['muted']}", html) + # Category colors flow into the embedded payload. + payload = json.loads((root / "data" / "dashboard.json").read_text()) + self.assertEqual(payload["categories"][0]["color"], "#ff2e88") + self.assertEqual(payload["categories"][1]["color"], "#22d3ee") + def test_noninteractive_initialization_generates_agent_workspace(self): with tempfile.TemporaryDirectory() as directory: env = dict(os.environ, TOPIC_SCOUT_ROOT=directory) @@ -587,7 +932,10 @@ def test_scout_llm_api_returns_usage(self): "candidates": [ { "id": "openalex:1", - "relevance_score": 8.2, + "topical_fit": 0.9, + "evidence_match": 0.8, + "rigor": 0.7, + "exclusion_hit": False, "relevance_reason": "Matches proof search and verification scope.", } ] @@ -633,7 +981,9 @@ def fake_urlopen(request, timeout): model="test-model", urlopen=fake_urlopen, ) - self.assertEqual(scores["openalex:1"]["relevance_score"], 8.2) + self.assertEqual(scores["openalex:1"]["topical_fit"], 0.9) + self.assertEqual(scores["openalex:1"]["evidence_match"], 0.8) + self.assertFalse(scores["openalex:1"]["exclusion_hit"]) self.assertEqual(cost["token_count"], 154) self.assertEqual(cost["model"], "test-model") @@ -649,7 +999,10 @@ def fake_run(command, **kwargs): "candidates": [ { "id": "openalex:1", - "relevance_score": 7.5, + "topical_fit": 0.8, + "evidence_match": 0.7, + "rigor": 0.6, + "exclusion_hit": False, "relevance_reason": "Highly aligned with formal verification scope.", } ] @@ -684,7 +1037,8 @@ def fake_run(command, **kwargs): run=fake_run, ) self.assertIn("--json", captured["command"]) - self.assertEqual(scores["openalex:1"]["relevance_score"], 7.5) + self.assertEqual(scores["openalex:1"]["topical_fit"], 0.8) + self.assertEqual(scores["openalex:1"]["rigor"], 0.6) self.assertEqual(cost["token_count"], 230) def test_gap_analysis_skips_when_no_accepted_papers(self): diff --git a/topic.example.json b/topic.example.json index f9ed48c..db667dc 100644 --- a/topic.example.json +++ b/topic.example.json @@ -35,5 +35,42 @@ "LLM verifier guided proof generation", "neural theorem proving benchmark" ], + "judging": { + "weights": { + "topical_fit": 0.5, + "evidence_match": 0.3, + "rigor": 0.2 + }, + "recency": { + "decay_per_year": 0.03, + "floor": 0.75, + "unknown_year": 0.9 + }, + "accept_hi": 7.0, + "accept_lo": 4.0, + "min_confidence": 0.35, + "prefilter": { + "enabled": true, + "scorer": "current", + "threshold": 0.0, + "keep_min": 5 + } + }, + "theme": { + "palette": { + "ink": "#17201f", + "paper": "#f3efe4", + "panel": "#fffdf6", + "line": "#c9c1b1", + "muted": "#706c63", + "accent": "#e4572e", + "accent2": "#1d6f75" + }, + "fonts": { + "display": "Georgia, serif", + "body": "\"Avenir Next\", \"Gill Sans\", sans-serif" + }, + "category_colors": ["#e4572e", "#1d6f75", "#f3a712", "#963484", "#4267ac", "#5b8e3e", "#7b6045"] + }, "created_at": "2026-06-12" }