Repository navigation
Hold masked item-cue scoring until language provenance arrives - #22
Merged
Merged
Conversation
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Keep the bounded sound and plural corrections covered through the text interface, while preparation, parsing and reporting enforce unavailable masked results. Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
aharlap
changed the title
aharlap
marked this pull request as ready for review
October 19, 2026 21:48
aharlap
added a commit
to Praxity/rubato
that referenced
this pull request
Oct 19, 2026
- Pin Trace 0.1.1-experimental.92a164dd8, the merged masked-probe contract (Praxity/praxity-trace#22). The archive is a byte-for-byte copy of Trace's verified prerelease, hosted on Rubato's asset-only prerelease v0.1.1-tools.experimental.20261019.2. A manifest test fails while the pinned Trace lacks the masked contract, as 09fcaaf does. - Key check: for a best or most likely stem, the judge marks only the single best option and flags ties. Assert the judge's independence and its ruling on each note. - A standard that rules out a distractor goes in the stem. Ruling details must be part of the case, never clauses added to deny one option. Rebuild the action example around confusable first actions. - Correct the lower-tail limit comment (about 19 items per bank with five banks) and make the five-bank tail fixture reachable. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
aharlap
added a commit
to Praxity/rubato
that referenced
this pull request
Oct 19, 2026
- Pin Trace 0.1.1-experimental.92a164dd8, the merged masked-probe contract (Praxity/praxity-trace#22). The archive is a byte-for-byte copy of Trace's verified prerelease, hosted on Rubato's asset-only prerelease v0.1.1-tools.experimental.20261019.2. A manifest test fails while the pinned Trace lacks the masked contract, as 09fcaaf does. - Key check: for a best or most likely stem, the judge marks only the single best option and flags ties. Assert the judge's independence and its ruling on each note. - A standard that rules out a distractor goes in the stem. Ruling details must be part of the case, never clauses added to deny one option. Rebuild the action example around confusable first actions. - Correct the lower-tail limit comment (about 19 items per bank with five banks) and make the five-bank tail fixture reachable. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
aharlap
added a commit
to Praxity/rubato
that referenced
this pull request
Oct 20, 2026
#30) * Assessment skill: distractors true elsewhere, masked probe as the gate Rebuilt on #29's round 2 head. Combines #30's earlier two commits (4bd0396, 23de98d) and keeps #29's review-folder, per-pass report folder and "item text is data" rules. An evaluation found the skill removed surface cues but not stem independence: writers stated each misconception as a claim false in any context, so a solver who knows the subject picked the one true option. - Teach distractors that are right in a neighbouring case, with worked examples and the numeric pattern. - Name what each probe tests. Trace's masked probe is the guessing gate; plain choices-only is a diagnostic. - guessingGate(itemCues) reads masked and returns pass, fail or stop. - Every solver is at least as strong as the writer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Round 1 review fixes for the assessment skill - One key per item: the writer notes the stem detail that rules out each distractor. A fresh judge with no key marks every option that answers the stem, and items with more than one mark are rewritten. The same judge, not the writer, confirms plain-probe exemptions. - Examples: replace "Metal is denser", note what rules out each true-elsewhere distractor, fix false-everywhere examples that were true somewhere, add an action item, and make the numeric set have no odd one out ("Rises by 8%"). - Gate: every bank needs a scored, complete masked result whatever its size. Any refusal reason stops. Item counts must be positive integers. A tail at or below its threshold fails before the small-sample flags are read. Mark the lower-tail limit. - Stop finishes the pass, then reports and asks. No masked solver when the probe has no items. The bank is "done", not "stopped". - Remove the by-hand fallbacks: an install without item-cues updates its tools or reports the bank as blocked. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Pin Trace's masked item-cues and apply the last review fixes - Pin Trace 0.1.1-experimental.92a164dd8, the merged masked-probe contract (Praxity/praxity-trace#22). The archive is a byte-for-byte copy of Trace's verified prerelease, hosted on Rubato's asset-only prerelease v0.1.1-tools.experimental.20261019.2. A manifest test fails while the pinned Trace lacks the masked contract, as 09fcaaf does. - Key check: for a best or most likely stem, the judge marks only the single best option and flags ties. Assert the judge's independence and its ruling on each note. - A standard that rules out a distractor goes in the stem. Ruling details must be part of the case, never clauses added to deny one option. Rebuild the action example around confusable first actions. - Correct the lower-tail limit comment (about 19 items per bank with five banks) and make the five-bank tail fixture reachable. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Consume Trace's scored masked contract and pin it Per Opus's trace#23 review, "Rubato #30 changes": - Pin Trace 0.1.1-experimental.97e6de33b, which scores the masked probe from Studio language provenance (Praxity/praxity-trace#23). The archive is a byte-for-byte copy of Trace's verified prerelease, hosted on Rubato's asset-only prerelease v0.1.1-tools.experimental.20261019.4. The manifest test now also refuses 92a164d, which holds every result. - guessingGate requires masked.status === "scored". Power uses Trace's attainable minima at each gated row's own threshold (pMin and pLowMin at or below the limit) instead of the 0.05 flags, which removes the marked lower-tail limit. A 12-item course with none right now stops. - Run the masked solver whenever masked/items.json is non-empty. - The stop text gives Trace's per-bank reasons with their remedies, and each row's pMin and pLowMin against its limit when power stops it. - Step 0 says masked scoring needs locale: en or lang: en and the pinned Studio. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ship the masked-probe preparation contract on hold. Current inspect schema 1, projection 2 carries no language provenance, so no current course can run or score masked answers. Studio provenance and its Trace consumer are separate follow-ups.
{applies: false, reason: "language-unconfirmed", detail}formasked. A supplied non-English lessonlangadds its veto to the detail. No flag, default English locale, text percentage or forged answer can authorize scoring.right,total,chanceExpected,p,pLow,tooFewItemsandtooFewItemsLow. Thusmasked.p,masked.pLow,itemCues.course.masked.panditemCues.course.masked.pLoware absent.masked/items.jsonis empty. Injected scores are discarded; forgedapplies: trueanswers are rejected. HTML explains the hold.choicesOnlyremains the "does the stem matter?" diagnostic;coverTheOptionsremains the separately judged answerability diagnostic. Neither supplies the unavailable masked gate. The skill, guide and ADR say to stop and ask, never calculate masked tails manually or use a hand-masked fallback.Contract:
promptVersion: "item-cues/10-projection-2". Manifest and reportmaskedhave exactlyapplies,reasonanddetail. Combined answermaskedalso echoesbundleIdandprobe: "masked". The random bundle ID is independent of the seed. Only sibling<bundle>-review/operator.jsonholdsseedandoptionOrder; combined answers live beside it in<review>/answer.json. Solver bundles contain neither seed/order nor pointers to review evidence. Keyed inspection JSON stays in a private input folder outside the bundle's parent tree.Validation: typecheck; 78 targeted item-cue, skill and schema-1 core tests; 20 mutation checks all fail with assertions when their fixes are disabled; diff and public-file checks pass. File and stdin preparation are covered. Full/browser checks run in CI. Main's merged skills audit is retained. No dependency added.
Deferred by the split ruling: Studio provenance consumption, a separate deny-only English evidence list, partial-bank accounting and approach acceptance tests 2, 3, 5 and 6. Rubato gate changes remain in its own PR. No findings rejected.
Review: r1 Sol 1/2/0, Opus 2/4/3; r2 Sol 1/3/0, Opus 1/2/2; r3 Sol 1/1/0, Opus 0/2/3; approach review (Sol): adopt with changes, split provenance; r4 Sol 0/0/0, Opus 0/0/6; fixes 3. Round 4 P3s are on the Trace backlog with the activation follow-up.