Repository navigation
fix: consume choice shuffle and count excluded items per bank - #24
Merged
Merged
Conversation
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
aharlap
marked this pull request as ready for review
October 20, 2026 01:38
aharlap
added a commit
to Praxity/rubato
that referenced
this pull request
Oct 20, 2026
The v2 eval fixed false-everywhere distractors but the masked gate
regressed to the no-skill level: writers attached the case's condition
to the key, so it was the longest option on 29 of 53 items, and the key
was usually the headline textbook answer.
- Options share one sentence shape; distractors carry their own case
clause. Key-length-rank check: key strictly longest on at most n/k
items, and keyLongest within longestExpected in Trace's spreads.
- Some stems make the neighbouring case the item, so the headline
answer is a distractor about one time in k. New Example 4.
- The masked solver is named in advance, at least the writer's model.
- Writer hand-in checklist: one (x) and {shuffle} per scored choice,
and a source count reconciled with Trace's included count.
- guessingGate takes exactly three seeded runs, pools masked.right with
exact tails, requires the same items and solver in each run, and stops
on any excludedCount above 0 or a missing excludedCount
(Praxity/praxity-trace#24).
- Key check plants a double-key positive control and scores distractor
notes blind before showing the writer's case.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
aharlap
added a commit
to Praxity/rubato
that referenced
this pull request
Oct 20, 2026
…te (#33) * Assessment skill v3: length parity, headline cue, pooled three-run gate The v2 eval fixed false-everywhere distractors but the masked gate regressed to the no-skill level: writers attached the case's condition to the key, so it was the longest option on 29 of 53 items, and the key was usually the headline textbook answer. - Options share one sentence shape; distractors carry their own case clause. Key-length-rank check: key strictly longest on at most n/k items, and keyLongest within longestExpected in Trace's spreads. - Some stems make the neighbouring case the item, so the headline answer is a distractor about one time in k. New Example 4. - The masked solver is named in advance, at least the writer's model. - Writer hand-in checklist: one (x) and {shuffle} per scored choice, and a source count reconciled with Trace's included count. - guessingGate takes exactly three seeded runs, pools masked.right with exact tails, requires the same items and solver in each run, and stops on any excludedCount above 0 or a missing excludedCount (Praxity/praxity-trace#24). - Key check plants a double-key positive control and scores distractor notes blind before showing the writer's case. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Assessment gate: declared counts and Trace #24 exclusion codes Studio shows a missing (x) the same way as a poll, so an exclusion alone can't tell them apart. The writer now declares scored choices and deliberate polls, and guessingGate(runs, { scoredChoices, polls }): - stops when Trace's included course count differs from the declared count; - stops on any excludedByReason code except multiple-select, and no-key-or-poll up to the declared polls; - stops when excludedByReason is missing (Trace before #24); - leaves a wholly excluded bank (count 0, no-included-items) out of the tails while its codes still count; - stops after the tails when any item's shuffle is unknown, since the position check skipped it; a fail still reports as fail. Each rule has a mutant test. The Trace pin is unchanged until #24 merges. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Assessment skill v3: review round 1 fixes - Option parity comes from each action's or claim's own detail, never the case's condition; no option restates or contradicts the stem. Example 4 rewritten; Examples 2 and 3 no longer echo stem words; no option ends with a period. Tests check every worked example for stem echoes and periods, and Example 4's stated length range. - keyLongestCue flags a bank only when the key is longest, ties as Trace counts them, above chance at one-sided 0.05. - Headline stems must test the same objective; the key-check judge, not the writer, counts headline keys. - The orchestrating step plants the double-key control under the reserved ref prefix 0., exempt from rewrites, with one judge rerun at most and a pre-ship check that no control remains. - Seeds 2 and 3 feed only the gate; their plain and cover answers are unread and unjudged. - guessingGate takes { seed, bundleId, courseHash, itemCues } per run: seeds 1, 2, 3, distinct bundles, one course; named solver on every row; course equals its banks item for item and hit for hit; full item metadata matches across runs. A clear statistical fail returns before any other stop. Polls match no-key-or-poll exclusions ref for ref. Exclusion accounting must be complete and reconcile. Missing accounting or unknown shuffle returns "tooling", and the skill says not to spend passes 2 and 3 on it. - Solver strength covers solvers and judges, not the explanations reviewer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Pin Trace 0.1.1-experimental.53ee25cc6 with exclusion accounting Trace #24 (53ee25c) adds per-row excludedByReason, maskedExcludedCount, coded excluded records and choice shuffle, which the assessment gate reads. Same route as #30: Trace prerelease built in hosted CI, rehosted byte for byte on Rubato's asset-only prerelease v0.1.1-tools.experimental.20261019.5. SHA-256 7596c94314fb587bbe414ed46daa5ed460af9aa009f282d6d17568a4def57aa2 matches the build log, Trace's asset digest, a local download, and Rubato's asset digest and an anonymous download. All 130 payload files match the inventory; licence, notices, skill and report reference match the source. The manifest test now refuses 97e6de3, which lacks the accounting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Assessment skill v3: review round 2 fixes - On "tooling", rewrites from pass 1 get one more pass: inspect, the three masked runs, and the key check and blind notes on the changed items. No rewrite ships unchecked. - Option detail describes the option itself, never why it suits a case. The key-check judge flags any option that points at a stem detail, including by paraphrase; the median example is the negative case. - guessingGate: excluded refs must be distinct and never also included. Exclusion records already in the report decide before a tooling limit, and accounting that is present but malformed is a stop, not "tooling". - Example 4 key no longer presumes coughing; back blows are "up to five". The numeric example's options are all falls of one length. - The headline count had no reader and is dropped. The control's ref follows the course's own pattern and lives in the orchestrator's notes; "the orchestrating step" is defined. - Seeds 2 and 3's plain and cover solvers are exempt in the solver rule and the done list names seed 1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Composer: declare polls and defer a tooling-limited bank assessment-writing v3's gate matches deliberate polls by ref against Trace's no-key-or-poll exclusions and can return "tooling". Composer now lists each poll by ref when it hands over the Items section, and treats a "tooling" bank like a non-English one: let the skill check pass 1's rewrites, mark the bank unverified, block under Draft, continue through steps 7.3 to 9 and ask at step 9. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Pin Studio 0.3.0-main.e1930acb with choice shuffle in inspect Studio's first admitted CLI candidate containing Praxity/praxity-studio#1816 (run 39361630685: admit, build with the CLI proof, and publish succeeded), rehosted byte for byte on Rubato's asset-only prerelease v0.1.1-tools.experimental.20261019.6. darwin-arm64 8acdb90fc11cb326e72e62ec74b36757d3fec510421c32691d9177d1d87b14a6 darwin-x64 01b821fbccc3758e397e5a9ea4f6fe08d9fbddd9b1823002e5b5ed87245f5981 Each matches the build job's recorded outputs, the candidate's asset digest and .sha256 file, a local download, and Rubato's asset digest and an anonymous download. verifyStudioArchive passes for both. A real inspect of a course with {shuffle} choices reports assessment.shuffle, and the assessment gate then returns pass or fail, not "tooling". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Trace reads and validates optional
assessment.shuffle, retaining unknown order for older Studio output. Existing spreads report fixed, shuffled and unknown counts.Per course and bank,
excludedCountretains structural exclusions,maskedExcludedCountcounts included choices withheld from masked preparation, andexcludedByReasoncounts stable structural and masked-only codes. The top-levelitemCues.excludedCountremains the structural course total. Every excluded record has a stablecode, a single counting ownerbankand group-membership provenance inbanks. Overlaps count once under the lesson; both course totals equal their respective bank sums.Wholly excluded banks report
no-included-itemswith their actual structural reasons. Legacy non-choice assessments no longer inflate exclusions. The report reference documents count reconciliation, missing-field stops, the ambiguity between a missing key and a poll, expected multiple-selection and declared-poll exclusions, and unknown-shuffle position coverage. This does not change Rubato's gate implementation.Checks: 109 targeted item-cue, schema-reader and skill checks passed; one existing opt-in skip. Typecheck and diff checks passed. New exclusion-contract tests and the legacy non-choice case failed before the fix. No configured lint script and no new dependency. Full suites and browsers left to CI under the assigned light-check limit.
Draft for maintainer review; no merge requested. Producer: https://github.com/Praxity/praxity-studio/pull/1816.
Review: 0/5/3, 0/0/2; fixes 1. Round 2 (Sol, Opus) found no P1 or P2; the P3 on report.md's per-bank 'must' wording is on the Trace backlog.