[aharlap]

Why

An evaluation of the assessment-writing skill found that it removed surface cues (a content-masked solver fell from 49% to 20%, against 25% chance) but not stem independence. A solver who knew the subject still picked the key from the options alone 89% of the time, with or without the skill. Writers stated each misconception as a general claim, which is false whatever the stem asks. The distractors that held up were true statements answering a neighbouring question, plus numeric ones.

What changed

• Trace pin. pack.json pins Trace 0.1.1-experimental.97e6de33b (source 97e6de33b3c30e67d44e7db6d0f1856bf358d3ef), which scores the masked probe from Studio's language provenance (Activate masked item-cue scoring from Studio language provenance praxity-trace#23). It follows Refresh Check and Trace pins to Rubato-hosted CLI prereleases #28's route.
  • Trace prerelease, built by this hosted run. Main CI on that commit: run.
  • Rehosted byte for byte on Rubato's asset-only prerelease v0.1.1-tools.experimental.20261019.4.
  • SHA-256 2204cbb49f7ed666cef941a83ff26d6a76611008fec6f10ca1fd5002820177fe. It matches the build log, Trace's asset digest, a local download, and both Rubato's asset digest and an anonymous download from the Rubato URL. All 130 payload files match the inventory, the source revision is clean, and the licence, notices, skill and report reference match the source.
  • The Studio pin (0.3.0-main.abaf1f3f, with inspect language provenance) comes from main unchanged. A manifest test fails while the pinned Trace lacks scored masked results, as 09fcaaf and 92a164d do.
• Distractors that need the stem. Distractors must be true or right in a neighbouring case, never false everywhere. For each distractor the writer notes the case it is right for and the stem detail that rules it out. That detail must belong to the case. A standard that rules out a distractor goes in the stem.
  • Worked examples cover an explanation, a classification, an action and a numeric item.
• Key check. A fresh judge that has never seen the key marks every option that answers each stem. For a "best" or "most likely" stem it marks only the single best option and flags ties. The same judge, not the writer, confirms the notes that would exempt a plain-probe hit from a rewrite.
• Probes. The masked probe is the guessing-cue gate; plain choices-only is a diagnostic. The masked solver runs whenever masked/items.json is non-empty. Masked scoring needs locale: en in course.yaml, or lang: en per lesson, plus the pinned Studio.
• Gate. guessingGate(itemCues) returns "pass", "fail" or "stop".
  • Every bank and the course need masked.applies === true, status === "scored" and total === count.
  • Counts must be positive integers.
  • A tail at or below its threshold fails.
  • Otherwise each gated row's pMin and pLowMin must reach its own threshold, or the gate stops.
• Stop path. On a stop, the agent finishes the other probes and rewrites, then reports everything. That report gives Trace's per-bank reason and its remedy (language-unconfirmed, language, mask-space), or the row's pMin and pLowMin against its limit. It marks the bank unverified and asks. No hand masking and no hand tails.
• Folders. Answers and judge notes go in <review>, Trace's <bundle>-review/ folder.

Tests

The shipped predicate runs against:

• each refusal reason, including an unknown one, with every scored field injected;
• "not run";
• a missing masked;
• incomplete small banks;
• string, fractional, infinite and null counts;
• masked against plain disagreement;
• eight of eight right on an underpowered course;
• 0 of 12 on a 12-item four-option course, which stops though Trace's 0.05 flags are false;
• a 15-item bank among five with none right;
• a missing minimum.

Each case also runs against a copy of the predicate with its rule removed, and the result must change. Prose rules are asserted by key phrase or structure, and removing a rule fails the test.

Local runs:

• node --test test/skills.test.mjs test/assessment-writing.test.mjs test/manifest.test.mjs: 121/121 pass.
• npm run validate and npm run validate:release: both pass.

Left

• Skipped P3s from round 2:
  • Example 1's expansion note names a metal-and-glass case.
  • The note and mark file formats for the key check.
  • Remaining exact-sentence prose pins.

Review: 4/8/4, 0/5/5; fixes 3.
Counts are distinct findings across Sol and Opus per round; overlapping findings count once. Round 2 found no P1s. Fix 3 consumes trace#23's contract, per Opus's trace#23 review ("Rubato #30 changes"), with no new review round.

🤖 Generated with Claude Code