feat(flowrite): a fact-check phase that gates the verdict on whether the page is true - #58
Merged
Merged
Conversation
`skipPhases` could only ever stop two phases. `review_page` and `fact_check_page` are gated in code and call `isPhaseSkipped`; research, design, write and integrate are prose-driven `task` delegations, and nothing put the skip list where the model could read it. So `skipPhases: ["research","design","write"]` researched, designed and wrote anyway, while the run directive it was meant to override went on naming the full flow. That contradicts the creation-data field's own description: "Skipping a head-phase prefix resumes a run whose artifacts already exist, e.g. ["research","design","write"] runs only the examples/integrate/review tail." It went unnoticed because nothing needed it. Once the other thirteen phase tools were deleted, the only phases anyone skipped were the two still gated in code, so the enforcement surface moved out from under this field. A phase gated in code refuses the call; a phase driven by prose needs the prose. `skippedPhases()` exposes the list and `useDocsWriter` states it last, so it qualifies the run directive rather than being qualified by it. Found while trying to run one phase in isolation — which the field promised and could not deliver. Unverified: no run has exercised it (see BACKLOG finding 7).
The review phase asks whether a page is good — structure, coverage, 28 style
rules. Nothing asked whether it is TRUE. Technical accuracy existed only as a
checklist line asking the reviewer to verify signatures against the research
file's citations, and BACKLOG finding 5 measures what that was worth: the
reviewer synthesises its own item list (46 items one round, 44 the next, 41 on a
module run), so accuracy was checked as often as it happened to invent an item
for it. Two real defects passed review because no item covered them.
Adds a `fact_checker` role and a `fact_check_page` phase tool. The role reads one
section of a finished page against the library's real sources and reports each
mismatch as a drift citing both sides — where the claim sits, and what the source
actually says. Drifts fold into `recordedVerdict()`, so a page stating something
the source contradicts can no longer be recorded as passing.
A second `harness: true` tool, against the grain of the thirteen that were
deleted, and for the one reason that saved review_page: a `task` delegation
returns final text and `defineSubagent` has no output schema, so a drift list
becomes data TypeScript holds only through `harness.prompt(..., { result })`. A
gate has to read evidence rather than ask the model how it did. agent.test.ts
pins the pair so a third cannot arrive quietly.
Design notes worth keeping:
- Source is the only authority. The research file may locate things faster but is
never trusted — it was written by another model, and a wrong citation there is
itself a drift.
- Both citations or nothing. A gate that flags correct prose gets switched off, so
when the checker is unsure it omits. A miss is recoverable; a correct page
failed is not.
- `incomplete` rather than a false all-clear. "Could not look" and "no drift" must
never arrive as the same answer, and `incomplete` fails the run.
- `high`/`medium` gate, `low` does not. Severity is model-authored and nothing
calibrates it yet.
- Its own round budget plus one conditional confirming round, mirroring review's.
Without the confirming round a gate can be failed and never cleared: the verdict
is the last recorded outcome, a refused call records none, so a run that
repaired every drift would file the pre-fix result — the write-module-ref-turn4
shape, where the anti-optimism mechanism produced the motive for optimism.
- One delegation per chunk of sections, serial. Parallel is not available: a
harness session runs one operation at a time and rejects a concurrent operation
with SessionBusyError. The chunk cap is therefore a wall-clock budget too.
- Split on `##` only. `###`/`####` are the per-capability subsections, and
splitting there would cut a method's prose from the plain ```scala signature it
describes — the one code on a finished page that mdoc never compiled, and so the
one place a wrong signature survives a verified page.
Adapted from the docs-gardener RFC's Signal 2: its findings shape, its
citation-pair rule, and fail-safe-not-silent. Its fingerprint/suggestedAction
fields serve cross-run dedup into a rolling issue, which this does not build.
Unverified against a live model — see BACKLOG finding 6 and
test-fixtures/fact-check/verify.sh. The 97 unit tests cover the budget
arithmetic, the verdict folding and the section splitting; none of them can say
whether the role finds a real drift or invents one.
The acceptance test for the gate, as a page whose every error is known: 5 planted drifts against tally/Ledger.scala and 4 correct claims that must not be flagged. Hand-planted rather than replayed from a real run because ground truth has to be exact. A page a real run produced has unknown accuracy to begin with, so "the checker found three problems" would be a judgement call; here the result is arithmetic. The false-positive half is the half that matters — a gate that fails correct pages gets switched off, and the real findings go unread with it. verify.sh plants the page, runs only the fact-check phase (every other phase skipped, so no research, no drafting, no sbt), prints what the check reported, and removes the page from a trap so an interrupted run cannot leave it behind. The fixture is a from-scratch baseline and must never keep it. The pass criterion lives in the script's header rather than in anyone's memory, which is the point of committing it.
archive-docs.sh writes to ../<fixture>-archive, so a tinyproject run lands in fixtures/tinyproject-archive/ — neither tracked nor ignored until now. That is exactly the `git add -A` hazard CLAUDE.md records having forced two history rewrites. The two entries beside it, tinyoptics-archive and tinytally-archive, are stale: there is one fixture directory now and it holds both modules. Left alone here — that cleanup spans CLAUDE.md and .env.testing.example too.
BACKLOG finding 6: the fact-check gate ships with no live measurement. The acceptance run died on turn 1 with tokens=0 — the API refused it on a usage limit until 2026-09-01 — so detection quality is entirely unmeasured, and the green unit tests must not be read as evidence about it. Names the three decisions waiting on that run: whether `low` should gate, the chunk budget, and whether a hierarchical module run needs more than one shared budget. BACKLOG finding 7: the skipPhases bug fixed in 3fc511d, with why it went unnoticed. The docs-gardener RFC moves from Proposed to partly implemented: Signal 2's detection engine now exists as a writer phase rather than a periodic auditor. Says what was taken from it, what is still unbuilt (the whole-repo sweep, the rolling issue, the cron, the read-only agent), and corrects two things the plan can no longer assume — the pre-Flue-2 file layout it names, and that a harness tool cannot fan out in parallel.
…ifts Carries zio#56's fix across to the other gate. This branch was cut before that landed, so the fact-check budget copied the single-grant design it replaced — and the defect it fixes is a property of the mechanism, not of reviews. The shape: `recordedVerdict()` reads the last outcome, a refused call records none, so a round that surfaces drifts the previous round missed spends the one grant and freezes the verdict on findings the run then repairs. Measured twice on the review side. Never measured here, because this phase has never run — but shipping two gates with divergent budgets would mean knowingly keeping the bug in one of them. A drift's identity across rounds is `kind|source`, not the whole finding. `claim` is page text and changes the moment the drift is fixed, `detail` and `fix` are model-authored prose, and `documented` is a line number that shifts when anything above it is edited. `source` is the declaration's own location, stable while the source is — and the run is forbidden to touch the source. So "the same problem, reported again" survives the page being rewritten around it. An incomplete round counts as a finding rather than a repeat: a round that could not look, then looked and found something, found something new by any reading that matters. Every write to `lastOutcome` now goes through `recordFactCheck`, so the previous round's findings cannot drift out of step with the current ones.
Member
Author
|
Verification tracked in #59 — to be run once the API key is renewed (2026-09-01, or sooner with a key that has headroom). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a
fact_checkerrole and afact_check_pagephase tool. The role reads a finished page section by section against the library's real sources and reports every drift — a claim the source contradicts, an API the library does not have, or a citation that no longer resolves. Drifts fold intorecordedVerdict(), so a page stating something the source contradicts can no longer be recorded as passing.Important
This ships unverified against a live model. The acceptance run died on turn 1 with
tokens=0— the API refused it on a usage limit until 2026-09-01. Local checks are green (tsc, 102 unit tests,vite build), but none of them can say whether the role finds a real drift or invents one on correct prose. Tracked in BACKLOG finding 7 and in the follow-up issue; merging is a deliberate call to land the code and measure it when the key works.Why
The review phase asks whether a page is good — structure, coverage, 28 style rules. Nothing asked whether it is true. Technical accuracy existed only as a checklist line asking the reviewer to verify signatures against the research file's citations, and BACKLOG finding 5 measures what that was worth: the reviewer synthesises its own item list (46 items one round, 44 the next, 41 on a module run), so accuracy got checked as often as it happened to invent an item for it. Two real defects passed review because no item covered either.
Meanwhile fabrication is the failure this repo keeps measuring — finding 1 (a run invented a Hugo shortcode and two stub pages),
SKILLS-ONLY-DESIGN.md:456(an invented API on a fixture with invented names is a revert trigger), and the wholetinytallyfixture, which exists because its names are made up.Design notes worth reviewing
A second
harness: truetool, against the grain of the thirteen that were deleted, for the one reason that savedreview_page: ataskdelegation returns final text anddefineSubagenthas no output schema, so a drift list becomes data TypeScript holds only throughharness.prompt(..., { result }). A gate has to read evidence rather than ask the model how it did.agent.test.tspins the pair so a third cannot arrive quietly.Source is the only authority. The research file may locate things faster but is never trusted — another model wrote it, and a wrong citation there is itself a drift.
Both citations or nothing. A gate that flags correct prose gets switched off, and the real findings go unread with it. When the checker is unsure it omits: a miss is recoverable, a correct page failed is not.
incompleterather than a false all-clear. "Could not look" and "no drift" must never arrive as the same answer, andincompletefails the run.high/mediumgate,lowdoes not. Severity is model-authored and nothing calibrates it yet.Serial fan-out, not parallel. One delegation per chunk of sections. Parallel is not available to a harness tool — a session runs one operation at a time and rejects a concurrent one with
SessionBusyError— so the chunk cap is a wall-clock budget as much as a token one.Split on
##only.###/####are the per-capability subsections; splitting there would cut a method's prose from the plain```scalasignature it describes. Those fences are the only code on a finished page that mdoc never compiled, and so the one place a wrong signature survives a verified page.Adapted from the
docs-gardenerRFC's Signal 2 — its findings shape, its citation-pair rule, fail-safe-not-silent. Itsfingerprint/suggestedActionfields serve cross-run dedup into a rolling issue, which this does not build; the RFC is updated to say what remains.Commits
fix(skip-phases)skipPhasesonly ever stopped the two code-gated tools; research/design/write/integrate are prose-driventaskdelegations and nothing put the skip list in front of the model — so skipping a head phase ran it anyway, contradicting the field's own description.feat(fact-check)test(fact-check)chore(gitignore)fixtures/tinyproject-archive/— the archive script writes there and it was neither tracked nor ignored.docsfix(fact-check)How to verify when the key works
Plants a page whose every error is known, runs only the fact-check phase, prints what it reported, and resets the fixture from a trap. Hand-planted rather than replayed from a real run because ground truth has to be exact — the result is arithmetic, not a judgement call.
Pass = all 5 planted drifts reported with both citations each, and none of the 4 correct claims flagged. The second half matters more.