Pipeline B silently under-extracted compounds it had already read

[Airwhale]

Pipeline B silently under-extracted compounds it had already read

AI generated — not verified by a human yet.
Written by Claude from one session investigating why the tropoflavin dose/route
buckets had so few authors. Numbers come from queries and a two-arm re-run that are
reproducible from the paths at the bottom; the interpretation has not been reviewed.

Summary

For the tropoflavin study, Pipeline B produced no compound exposure row at all for 89 authors who provably discuss 7,8-DHF. Their posts are in the corpus, the compound is in the text, and the sentiment pipeline scored them — Pipeline B just returned nothing usable for them.

Re-running the same 89 authors surfaced six separate causes. The largest is not a bug in our code, which is why nothing here was ever going to show up in a test.

How this was found

comparators.db and nootropics_pipeline_a_b_linked.db share the author_hash / user_id namespace (13,545 of 13,568 linked authors resolve). That makes the sentiment pipeline an independent ground truth for "did this author discuss compound X" — which is exactly what Pipeline B's recall should be measured against, and never was.

The funnel for 7,8-DHF:

step	authors
discuss 7,8-DHF (≥1 sentiment report on drug_id=1)	279
Pipeline B produced a record	225
Pipeline B tagged a 7,8-DHF exposure	116
stated a dose or a route	63

The 89 authors with a record but zero compound exposure rows split into:

• 47 where the compound appears in medications only — a stack entry with no dose, route, or outcome attached
• 42 where the compound appears nowhere in the record, of which 36 had it inside the model's input window and 6 were truncated away

Cause 1 — the model is an unpinned moving target (largest single factor)

Re-running the unchanged August code over the same corpus today recovers 22 of these authors. August produced 0.

	original (Aug 27)	same code, re-run today
→ yields an exposure row	0	22
dose attached	0	8
route attached	0	7

(80 authors scorable. The re-run arm processed 88/89 but hit LLMTruncationError — response truncated at max_tokens — three times and exited without writing its CSV, leaving 80 records in the incremental save. See Cause 6.)

corpus/llm_provenance.json records "model_fast": "deepseek/deepseek-v4-flash" — a name, not a pinned version or snapshot — and no commit hash. So an extraction run is not reproducible, and a quality regression or improvement in the hosted model silently changes our data with nothing recording that it happened.

This also means the whole Pipeline B table is likely under-extracted, not just this tail. These 89 were selected because they failed; the same drift applied to all 752 records.

Cause 2 — HEALTH_SUBREDDITS contains no nootropics communities

collect_texts_from_user orders health-subreddit text first so that truncation keeps the good content. The set in variable_extraction/patientpunk/llm_extract.py is entirely long-COVID / ME-CFS communities.

On an r/Nootropics corpus every text falls into the other_texts bucket, so the prioritisation is inert and truncation keeps whatever happened to come first in scrape order. The mechanism intended to protect relevant content did nothing for this entire study.

Cause 3 — an 8,000-character input cap

MAX_TEXT_CHARS = 8_000, hardcoded at the commit that ran this study. Author histories in this cohort run to 145,000 characters. This truncated the compound away for 6 of the 42.

Small in isolation, and later made env-settable in 04376b1 — but that commit is not on the branch the study ran from (see Cause 5).

Cause 4 — the medications field description is shaped for a different population

medications: Prescription drugs and daily supplements (LDN, Paxlovid, gabapentin, magnesium, probiotics).

Every example is a long-COVID drug. Against a nootropics population, research chemicals named by abbreviation or chemical name get dropped — 36 of the 42 had the compound inside the input window and it was dropped anyway, so this is the dominant cause among the true misses, not truncation.

Broadening this rule to cover research chemicals and to require complete stack lists recovers 36 of those 42 (see "Caution" below — it is not free).

Cause 5 — a compound named only in medications produces nothing downstream, silently

pipeline_b_compound_exposures is built from dosage, administration_route, and treatment_outcome only. An author whose record says medications: 7,8-dhf and nothing else contributes zero rows — correct by design, but nothing warns that a recognised compound produced no usable output. That is the entire 47-author group.

Related: the run output carries an administration_route column that does not exist in the schema on main — it lives only on codex/pr140-comparator-analysis. The extraction code that produced the shipped study database is not on the mainline, which is how a stale medications description and the missing subreddit entries survived.

Cause 6 — a late failure discarded completed work (mostly covered by #138)

Re-running the unchanged code, 3 of 89 records failed with LLMTruncationError: response truncated at max_tokens. #138 already documents the truncation-handling asymmetry between
classify and extract, so this is not a new failure mode.

One detail there may be new: the run exited code 0 without writing records.csv,
discarding 8 completed extractions in flight past the last incremental save
(SAVE_EVERY_N = 10). A partial run reported success and lost work that was already done.

Worth noting against #138's framing: the patched arm, with a 5x larger input cap,
completed all 89 with zero failures — so this is not simply "long inputs break it".

Why no test caught any of this

• No recall metric. Coverage was only ever reported as absolute counts — "752 author histories", "202 compound exposures" — with no denominator. 202 out of what? Nothing compared extraction output against an independent list of authors known to discuss each compound, despite that list being available in comparators.db the whole time.
• No reproducibility check. No pinned model, no commit hash in provenance, so a re-run producing different output is invisible.
• Silent zero. In the August run, extraction "succeeded" for all 89 — valid JSON, no errors. The failure is entirely in what was absent, and nothing measures absence.

Proposed fixes

Roughly in order of value:

1. Record a recall metric per run. Keyword-scan the corpus for each target compound's aliases, then report extracted / present_in_text per compound. This one number would have caught causes 1-4 on day one.
2. Pin the model and record the commit hash in llm_provenance.json. Treat an unpinned model as a data-provenance defect.
3. Fail loudly and don't discard completed work — a run with failed records should exit non-zero and still flush what succeeded.
4. Warn on named-but-unusable compounds — a compound in medications producing no exposure row should be counted and surfaced in the run report.
5. Make HEALTH_SUBREDDITS schema-driven rather than a module constant, so each study declares its own relevant communities.
6. Domain-adapt the medications description per schema instead of shipping long-COVID examples to every population.
7. Re-run Pipeline B for all 752 authors, not just this tail — drift alone implies broad under-extraction.

Caution on fix 6

Broadening the medications rule raises recall sharply but costs attribution accuracy. In the patched re-run, auditing every extracted target dose/route against source text:

• numeric doses: 27 of 30 corroborated; 3 misattributed — a dose belonging to another compound in the same passage (100 mg was Phenyl Hydrazide, 10 mg was Kratom, <5 mg was tryptamines)
• routes: 16 of 20 supported; 4 unsupported, including a sublingual that appears nowhere in the source, and several oral values that look inferred — which the prompt's own rules explicitly forbid

≈85–90% precision. Any recall fix needs a paired precision guard — a corroboration check that the extracted value actually appears near a mention of the compound it was attributed to removes all three bad doses cleanly.

Reproduction

PatientPunk_data/studies/tropoflavin_nootropics/runs/2026-09-01-recall-repair/ contains both arms, the 89-author target list, per-arm caches, and a README with the full method and caveats. Worktrees are at commit 5f462e8, the last extraction commit before the 2026-08-27 run.

Combined effect of the three patches over the drift baseline (80 authors):

metric	original	re-run, same code	re-run, patched
named in medications	43	44	70
dose attached	0	8	22
route attached	0	7	15
outcome attached	0	16	43
→ yields an exposure row	0	22	48
→ usable by dose/route buckets	0	9	26

Applied to the study, with the corroboration filter on: 7,8-DHF authors with a dose 37 → 49, panel-wide route exposures 31 → 48.

Related issues

• DRAFT: run-breakage log — classify has no truncation fallback, LLM_MAX_TOKENS defeats extract's recovery, + 4 more #138 — truncation handling; overlaps Cause 6
• Phase 2 banner and llm_provenance.json can report different models for the same run #82 — Phase 2 banner and llm_provenance.json can disagree on model; adjacent to Cause 1
• Canonicalization is not reproducible: 8.3% merge agreement across identical runs #127 — canonicalization not reproducible across identical runs; same reproducibility theme as Cause 1
• Dose and drug are in different pipelines: extract dose per (post, drug) in Path A #128 — dose and drug in different pipelines; the downstream consumer of Cause 5
• run pipeline drops all comments: 72% of patients and 93% of text never extracted #87 — pipeline drops comments, 93% of text never extracted; same class of silent recall loss

[Airwhale]

Follow-up: the recall metric proposed as fix 1 now exists (studies/tropoflavin_nootropics/audit_recall.py, pushed to #140), and running it against the original 2026-08-27 extraction puts a number on the "likely under-extracted" inference above.

Across all 752 authors, not the 89-author tail this issue was written from:

authors whose text mentions a 7,8-DHF alias : 748
...named anywhere in their record           : 273   36.5%
...in a field that can reach an exposure row: 173   23.1%
...named in medications only                : 100   13.4%
...absent from the record entirely          : 475   63.5%

Please do not quote 63.5% as a defect rate. The denominator is "mentions the compound", and the prompt deliberately excludes a treatment someone only asked about, was offered and declined, or cited a study on. So this is an upper bound on recall loss, not a measured miss rate. The script prints that caveat in its own output so the number does not travel without it.

The 89-author figures elsewhere in this issue used a stronger ground truth — an author with a sentiment report on drug_id=1 has expressed an evaluative opinion, which implies use far more strongly than a bare mention. The true rate sits between the two. Getting a real number means intersecting the corpus with that stronger signal; treatment_reports shares the author_hash namespace and serves.

Discounted appropriately, it still supports fix 7: this is corpus-wide, not a tail.

Also on #140, and worth flagging against the fixes listed above:

• fix 5 (HEALTH_SUBREDDITS) landed as a schema key, priority_subreddits, rather than an edit to the shared constant. Schemas naming none keep the existing set, so long-COVID runs are byte-identical.
• fix 6 (medications) landed as an extension_fields override in nootropics_schema.json. build_field_descriptions already lets an extension override a base description, so no shared-prompt change was needed and no other study is affected.
• fix 2 landed as commit + extracted_at + max_text_chars in llm_provenance.json. Reading .git directly rather than shelling out, because TestPipelineNoSubprocess asserts the pipeline never calls subprocess.run. This makes drift detectable; it does not pin the model, which is a policy call and deliberately left alone.
• fix 3 is untouched — it overlaps DRAFT: run-breakage log — classify has no truncation fallback, LLM_MAX_TOKENS defeats extract's recovery, + 4 more #138's active work on truncation handling.

— Claude, via SG