Relevance is not enough — reproduction package

Relevance is not enough: A Communication-Oriented Retrieval System for Consequential Scientific Question Answering

This repository contains the verification script, the evaluation artifacts every reported number is computed from, and the pipeline implementation used to produce those artifacts.

Not included: the knowledge base (build your own, see below) and the community survey/interview material behind Section 3, which is human-subject data and is not redistributed here.

---

Folder structure

.
├── requirements_full_repo.txt       pinned dependency list from the full project environment
│
├── analysis/                        everything needed to verify the paper's numbers   [Section 4]
│   ├── reproduce_paper_numbers.py   recomputes every reported number from the CSVs below
│   │
│   ├── abalation/                   per-answer scores for the four pipeline variants   [Section 4.1, Table 1]
│   │   ├── ctx_len.csv              retrieved context length in characters, per query x mode
│   │   ├── ctx_relevance.csv        context relevance in [0,1], plus the extracted sentences behind it
│   │   ├── answer_relevance.csv     answer relevance in [0,1], per query x mode x persona
│   │   ├── human_completeness.csv   the 648 human annotations: two annotators + adjudicated gold, 1-5 Likert
│   │   ├── llm_completeness.csv     default and fine-tuned LLM-judge completeness for the same answers
│   │   ├── reading_ease_score.csv   Flesch reading-ease score per answer
│   │   ├── emotional_reaction.csv   empathy labels (ER/IP/EX) per answer
│   │   └── readability_by_persona.csv  readability and emotional-reaction collapsed to mode x persona cells
│   │
│   └── data_summary/                per-query traces on the 84-query common set   [Section 4.2, Table 2]
│       ├── gap_trace.csv            evidential gaps identified in iteration 1 and remaining after iteration 2
│       ├── metrics.csv              answer relevance, completeness and context relevance keyed to those queries
│       └── context_strategy.csv     reasoning strategy, context size, and whether a 2nd retrieval pass fired
│
├── Fine-tune evaluator/             trains and evaluates the fine-tuned completeness judge   [Section 4]
│   ├── Prepare_data.py              builds the query/response/human-score JSONL and the 60:20:20 train/val/test split
│   ├── Train_model_LoRA_method.py   LoRA fine-tune of the open-source Qwen2.5-7B model on the 648 annotated pairs
│   ├── Inference_fine_tuned_model.py  predicts a 1-5 completeness score for a query-response pair
│   ├── evaluate_predictions.py      scores those predictions against the human annotations
│   ├── requirements.txt             pinned dependencies for the fine-tuning stack (torch, transformers, peft)
│   └── README.docx                  run order, commands, and the A100 (80GB) / 128GB RAM training setup
│
└── RAG/                             the pipeline implementation used for the experiments   [Section 3, Figure 1]
    ├── retriever.py                 flattens the knowledge base and retrieves by heading similarity (all-MiniLM-L6-v2, cosine > 0.51), falling back to full-text matching
    ├── prognostic_check.py          oracle LLM (gpt-4o-mini) scores each retrieved chunk 0-3 for relevance; >= 2 is kept
    ├── diagnostic_check.py          open-source diagnostic check: classifies the reasoning strategy, maps evidence to that strategy's requirements, flags evidential gaps, and writes targeted sub-queries
    ├── generator.py                 Generator LLM; turns relevant evidence and remaining gaps into structured Claim / Evidence / KnowledgeLimitations JSON
    ├── orchestrator.py              drives the full loop: retrieve -> prognostic -> diagnostic -> optional 2nd retrieval pass -> generate, over every query in queries.csv
    ├── baselineRAG.py               single-pass baseline: retrieve, then generate, with no prognostic or diagnostic check
    ├── articulator.py               Communicator LLM; rewrites the generator's claims into persona-adapted, accessible language
    ├── personas.py                  persona definitions (role + style) the articulator selects from
    ├── utility.py                   query simplification (tokenisation and stop-word removal) used by the retriever
    │
    ├── Closed_Source/               variants that route the diagnostic check to hosted models instead of a local one   [Section 4.1]
    │   ├── diagnostic_check.py      same diagnostic logic, dispatched to OpenAI / Anthropic / Gemini by model name
    │   └── orchestrator.py          orchestrator wired to the hosted diagnostic check
    │
    ├── queries.csv                  the 498-question probe set every pipeline variant was scored on
    ├── common_queries.csv           the frequent-community-question subset used by baselineRAG.py
    └── additional_queries.csv       held-out retrieval-only queries; scored by no reported number

Where each file appears in the paper

File	Paper
analysis/reproduce_paper_numbers.py	recomputes Tables 1 and 2 and every inline statistic in Section 4.2
analysis/abalation/ctx_len.csv	Table 1, Ctx Len; the context-size reduction reported in Section 4.2
analysis/abalation/ctx_relevance.csv	Table 1, Ctx Rel
analysis/abalation/answer_relevance.csv	Table 1, Ans Rel
analysis/abalation/human_completeness.csv	Table 1, Completeness; the 648 annotations and Cohen's kappa = 0.65 in Section 4
analysis/abalation/llm_completeness.csv	Section 4.2, Agreement between LLM-based and human evaluations (rho = 0.41 and 0.73, kappa = 0.54)
analysis/abalation/reading_ease_score.csv	Section 4.2, Effects of persona assignment on articulation; Figure 2
analysis/abalation/emotional_reaction.csv	Section 4.2, Effects of persona assignment on articulation; Figure 2
analysis/abalation/readability_by_persona.csv	Section 4.2 persona reading-ease and empathy means; Figure 2
analysis/data_summary/gap_trace.csv	Table 2; Section 4.2, Efficacy of diagnostic check in gap resolution
analysis/data_summary/metrics.csv	Section 4.2, completeness on the queries that triggered a second pass
analysis/data_summary/context_strategy.csv	Table 2, the reasoning-strategy rows; Section 3.3
RAG/retriever.py	Section 3.2, Retrieving content (the 0.51 heading-similarity threshold)
RAG/prognostic_check.py	Section 3.3, Prognostic check; prompt in Appendix A.2
RAG/diagnostic_check.py	Section 3.3, Diagnostic check; prompts in Appendix A.3, strategy templates in A.3.1
RAG/generator.py	Section 3.4, Generator LLM
RAG/articulator.py, RAG/personas.py	Section 3.4, Communicator LLM; Figure 2
RAG/orchestrator.py	Section 3.3, the iterative loop capped at two retrieval passes; Figure 1
RAG/baselineRAG.py	Section 4.1, the first configuration; prompt in Appendix A.1
RAG/utility.py	Section 3.2
RAG/Closed_Source/	Section 4.1, the heterogeneous open-source / closed-source model split
RAG/queries.csv	Section 3.1, the evaluation question set
RAG/common_queries.csv	Section 4.1, the 160 frequent community queries
Fine-tune evaluator/	Section 4, the fine-tuned LLM judge (LoRA on Qwen 2.5-7B, 60:20:20 split, ~35 min on one A100); its agreement with humans in Section 4.2 (rho = 0.73, kappa = 0.54)
KnowledgeBase/ (you build this)	Section 3.1, Constructing the knowledge base

---

Verifying the paper's numbers (Section 4)

pip install pandas numpy scipy statsmodels scikit-learn
python3 analysis/reproduce_paper_numbers.py            # add --csv out.csv to save the table

No API keys, no GPU and no knowledge base are needed for this — it reads only the CSVs in analysis/.

Running the pipeline: build your own knowledge base (Section 3.1)

The knowledge base is not distributed with this material. The corpus was scraped from EPA, CDC and USGS pages, and redistributing it here would mean shipping a stale copy of documents whose canonical versions are maintained and updated by those agencies. You must build your own before any of RAG/ will run.

Place it at:

KnowledgeBase/structured_knowledge_base.json     # sibling of RAG/, i.e. ../KnowledgeBase/ from RAG/

Expected structure

A JSON array of documents. Each document carries a source URL and a content list of arbitrarily nested sections. A section is either a container (heading + content) or a leaf (heading + text):

[
  {
    "source": "https://www.epa.gov/example-page",
    "content": [
      {
        "heading": "Basic Information on the CCL",
        "content": [
          {
            "heading": "Basic Information on the CCL > What is the drinking water CCL?",
            "text": "The drinking water CCL is a list of contaminants that are ..."
          },
          {
            "heading": "Basic Information on the CCL > What happens to contaminants on the CCL?",
            "text": "EPA uses the CCL to identify priority contaminants ..."
          }
        ]
      }
    ]
  }
]

---

Expected output of the verification script

reproduce_paper_numbers.py compares each reported number against the value it recomputes from the CSVs and prints a per-row MATCH / NEAR / MISMATCH status plus a summary count. The current package reproduces 170 values exactly; the remaining rows are rounding-level NEAR results and a handful of MISMATCH rows where the script's canonicalised merge differs from the as-run analysis notebook. The script labels each of these inline.

RelevanceIsNotEnough