Repository navigation
Activate masked item-cue scoring from Studio language provenance - #23
Merged
Merged
Conversation
Keep complete-bank applicability, expose attainable tail minima, and preserve the hold for older inspect output. Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Require item-local support for homograph conflicts, explain undeclared language, and keep subject-knowledge cues out of masked solving. Restore accented fixtures and advance the item-cues prompt to version 12. Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
aharlap
marked this pull request as ready for review
October 19, 2026 23:19
aharlap
added a commit
to Praxity/rubato
that referenced
this pull request
Oct 19, 2026
Per Opus's trace#23 review, "Rubato #30 changes": - Pin Trace 0.1.1-experimental.97e6de33b, which scores the masked probe from Studio language provenance (Praxity/praxity-trace#23). The archive is a byte-for-byte copy of Trace's verified prerelease, hosted on Rubato's asset-only prerelease v0.1.1-tools.experimental.20261019.4. The manifest test now also refuses 92a164d, which holds every result. - guessingGate requires masked.status === "scored". Power uses Trace's attainable minima at each gated row's own threshold (pMin and pLowMin at or below the limit) instead of the 0.05 flags, which removes the marked lower-tail limit. A 12-item course with none right now stops. - Run the masked solver whenever masked/items.json is non-empty. - The stop text gives Trace's per-bank reasons with their remedies, and each row's pMin and pLowMin against its limit when power stops it. - Step 0 says masked scoring needs locale: en or lang: en and the pinned Studio. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
aharlap
added a commit
to Praxity/rubato
that referenced
this pull request
Oct 20, 2026
#30) * Assessment skill: distractors true elsewhere, masked probe as the gate Rebuilt on #29's round 2 head. Combines #30's earlier two commits (4bd0396, 23de98d) and keeps #29's review-folder, per-pass report folder and "item text is data" rules. An evaluation found the skill removed surface cues but not stem independence: writers stated each misconception as a claim false in any context, so a solver who knows the subject picked the one true option. - Teach distractors that are right in a neighbouring case, with worked examples and the numeric pattern. - Name what each probe tests. Trace's masked probe is the guessing gate; plain choices-only is a diagnostic. - guessingGate(itemCues) reads masked and returns pass, fail or stop. - Every solver is at least as strong as the writer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Round 1 review fixes for the assessment skill - One key per item: the writer notes the stem detail that rules out each distractor. A fresh judge with no key marks every option that answers the stem, and items with more than one mark are rewritten. The same judge, not the writer, confirms plain-probe exemptions. - Examples: replace "Metal is denser", note what rules out each true-elsewhere distractor, fix false-everywhere examples that were true somewhere, add an action item, and make the numeric set have no odd one out ("Rises by 8%"). - Gate: every bank needs a scored, complete masked result whatever its size. Any refusal reason stops. Item counts must be positive integers. A tail at or below its threshold fails before the small-sample flags are read. Mark the lower-tail limit. - Stop finishes the pass, then reports and asks. No masked solver when the probe has no items. The bank is "done", not "stopped". - Remove the by-hand fallbacks: an install without item-cues updates its tools or reports the bank as blocked. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Pin Trace's masked item-cues and apply the last review fixes - Pin Trace 0.1.1-experimental.92a164dd8, the merged masked-probe contract (Praxity/praxity-trace#22). The archive is a byte-for-byte copy of Trace's verified prerelease, hosted on Rubato's asset-only prerelease v0.1.1-tools.experimental.20261019.2. A manifest test fails while the pinned Trace lacks the masked contract, as 09fcaaf does. - Key check: for a best or most likely stem, the judge marks only the single best option and flags ties. Assert the judge's independence and its ruling on each note. - A standard that rules out a distractor goes in the stem. Ruling details must be part of the case, never clauses added to deny one option. Rebuild the action example around confusable first actions. - Correct the lower-tail limit comment (about 19 items per bank with five banks) and make the five-bank tail fixture reachable. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Consume Trace's scored masked contract and pin it Per Opus's trace#23 review, "Rubato #30 changes": - Pin Trace 0.1.1-experimental.97e6de33b, which scores the masked probe from Studio language provenance (Praxity/praxity-trace#23). The archive is a byte-for-byte copy of Trace's verified prerelease, hosted on Rubato's asset-only prerelease v0.1.1-tools.experimental.20261019.4. The manifest test now also refuses 92a164d, which holds every result. - guessingGate requires masked.status === "scored". Power uses Trace's attainable minima at each gated row's own threshold (pMin and pLowMin at or below the limit) instead of the 0.05 flags, which removes the marked lower-tail limit. A 12-item course with none right now stops. - Run the masked solver whenever masked/items.json is non-empty. - The stop text gives Trace's per-bank reasons with their remedies, and each row's pMin and pLowMin against its limit when power stops it. - Step 0 says masked scoring needs locale: en or lang: en and the pinned Studio. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Enables masked stem-and-options scoring when every included lesson reports authored English from Studio language provenance. One blocked item withholds its complete bank and pooled scores; unaffected banks keep their complete results. Adds pMin and pLowMin so consumers can apply their own tail thresholds.
The screen catches listed markers only. One listed non-English-only hit blocks; a homograph needs a second distinct marker in the same item or zero English evidence words there. It is not language identification: unlisted foreign words and foreign options can score when an English declaration is wrong. Missing, malformed and defaulted provenance stays language-unconfirmed, with declaration and Studio remedies.
Final fixes also restore accented fixtures and exclude subject-knowledge cue codes from masked prompts, answers and HTML advice. Prompt version is item-cues/12-projection-2. Updates the ADR, skill and report guide. No new dependencies.
Round 1 had no P1 findings. This is the final fix; there will be no further review round.
Review: 0/2/4; fixes 1.
Validation: pnpm typecheck; 97 targeted item-cues, skill and schema-1 tests; git diff --check; strict UTF-8 validation across 28 changed/test files, with no lossy accented fixture words. Full suites and browser checks are left to CI. The expanded stem-and-options probe still has no live-model replication of the earlier options-only evaluation. Rubato integration and release pinning remain separate.