corpus review: the instrument for one sitting of 150, prepared and empty - #98
Merged
Merged
Conversation
… 150
Makes step B1 of docs/design/REVIEW_THE_BULK_2026-08-19.md unconditional. The
design document offered the sample "on request" and the request never came;
the change that matters is that the sample no longer needs a ruling to exist.
Two scripts and one prepared pass.
prepare_corpus_review.py draws a seeded random 150 from the 1,129 arXiv
bulk records, writes frame.json BEFORE fetching
anything, then pulls the real abstracts into an
immutable raw dump.
serve_corpus_review.py one record per screen, four keys, and every
keypress appended and fsync'd to verdicts.jsonl
before the next card is drawn.
NO HUMAN FIELD IS SET BY THIS COMMIT. verdicts.jsonl does not exist yet, and
an absent verdicts.jsonl is the correct state before a sitting. The 150 rows
are Pip's to write.
Four answers, and the distinctions are the whole point: yes, no, unknown
("looked at it and could not tell"), skip ("passed over without judging").
Neither is ever folded into `no` and neither is ever folded into the other.
A record with no row is NOT YET REVIEWED, which is an absence and never a
value, because a value can be read as a judgement and an absence cannot.
`retracted` is an undo, not a fifth answer.
Reused rather than rebuilt: the interaction model is tools/review_queue.html
unchanged in substance -- note box focused at all times so dictation lands
somewhere, verdicts bound to punctuation so no letter fires one mid-sentence.
The append-before-state discipline is triage_watch.py and art_review's
serve_review.py. It is a local server rather than the static sheet the design
document proposed, on one requirement the document did not have: a judgement
must be on disk the moment it is made, and a file:// page cannot write to
disk. Closing the tab at 91 leaves 91 judgements.
Two things measured on the way, both external inputs per pdoom1#1075:
* The arXiv Atom API answers HTTP 429 "Rate exceeded" to every request from
this address, including a single id. OAI-PMH GetRecord works. Both paths
are in the script, Atom first.
* OAI-PMH <created> is NOT the v1 submission date -- for 1803.04585 it is
2019-02-24 against an identifier saying 2018-03, and for WebGPT it is
2022-06-01 against 2021-12. A first draft of this stored it as
`published`, and that field name alone produced a confident wrong finding
of 38 year-disagreements in 150. Against the identifier's own month, the
real figure is 9. The endpoint-specific names are now in the field names.
Nothing written to data/serveable/. No per-record field on any bulk record.
No widening of event_v1. No tag vocabulary and no rating scale.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…he keys The key mapping is only stated in two places -- the banner and the page footer. A banner lost to buffering is a reviewer guessing at the keys. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Branch left on the remote before the Singapore departure, never PR'd. Merged with current
main, no conflicts, all 28 gating checks pass.This is the review instrument for the bulk corpus. It is prepared and empty -- the sample frame exists, no verdict has been recorded, and merging it changes no served data.
One question, asked verbatim
It is the only question asked, and
REVIEW_THE_BULK_2026-08-19.md§1 works out why the other three candidates are not for a human: description quality is a defect in our own extractor, schema conformance is one key and one empty array, and whether the bulk belongs intimeline_eventsis one ruling rather than 1,166 judgements.The question is stored verbatim in
frame.jsonand again on every row ofverdicts.jsonl, because a verdict whose question must be reconstructed from context is not evidence of anything.The shape that makes it honest
yes/nounknownskipunknownandskipare never folded intono, into each other, or into absence -- a paper outside the reviewer's field is an honestunknown, and how many there are is a finding rather than a gap. Not-yet-reviewed is an absence, never a value, because a token can be read as a judgement and an absence cannot.retractedis an undo, not an answer: it returns a record to unreviewed while leaving both the judgement and the retraction in the log.The frame was written before anything was fetched and before any key was pressed -- seed, population, exclusions, n, and the drawn ids -- so it cannot have been chosen after seeing the results.
The .gitignore addition is not the #87 trap
It ignores exactly two things, and neither is judgement:
data/raw/arxiv_abstracts/dumps/*/data.jsonl-- a fetch thatprepare_corpus_review.pyreproduces from the seed in frame.json, and whose real abstracts carry non-ASCII author names and mathematics the ASCII gate would reject._metadata.jsonstays tracked -- that is the provenance.tools/corpus_review.html-- regenerated every time the server starts.verdicts.jsonlandframe.json, the parts that cannot be reconstructed, are tracked. That is the distinction #87 was about.