Fabrication: Claude invents quotes and source citations not in retrieved chunks

[rob-9]

Priority: Critical

Accuracy is the weakest benchmark dimension at 1.24/3. In ~40% of test cases, Claude fabricates textual evidence — inventing quotes, source citation numbers, and plot details that don't appear in any retrieved chunk. This is the single biggest quality problem.

Observed patterns

• Cites [Source 4] when only 3 sources were returned
• Invents character dialogue and attributes it to a source
• Uses training-data knowledge of the book but presents it as if from retrieved passages
• More likely when retrieval returns few/no relevant chunks (fills gaps with model knowledge)

Affected test cases

theme-colonialism (acc=0), spoiler-goose-poisoning (acc=0), generation-bridge (acc=0), nonlinear-ewing-conclusion-ahead (acc=0), register-shift-ewing-formal (acc=0), theme-ocean-drops-ending (acc=0), gg-amy-person-25-vs-65 (acc=0), and 10+ more at acc=1.

Potential approaches

• Post-generation citation audit: verify every [Source N] references a real source number, strip or flag phantom citations
• Reduce max_tokens from 1024 to 300-400 to structurally constrain output length
• Add explicit source enumeration in the prompt: "You have N sources numbered 1 through N. Do NOT cite any source number outside this range."
• When retrieval returns 0 chunks, switch to a strict "no-context" prompt that forbids any book-specific claims
• Consider prefilling the assistant response with a grounding preamble

Benchmark baseline

• Overall: 1.81/3 (48 cases)
• Accuracy: 1.24/3

[rob-9]

Partially addressed via prompt changes:

• GROUNDING_RULE now explicitly instructs: "Do NOT state conclusions about character plans, hidden motives, or outcomes that the passages have not explicitly spelled out. Describe what the text shows, not what you infer it means."
• Added balanced-perspective instruction: "When passages show CONFLICTING or NUANCED perspectives, present the full complexity."

Benchmark accuracy improved from 1.24/3 → 2.67/3. Two cases remain at acc=1 (comparative-griffin-methods, meta-textual-frobisher-reads-ewing). Further improvement likely needs citation verification (check that [Source N] references real sources) and/or structured output constraints.