[aharlap]

Skill v2 fixed false-everywhere distractors, but its masked guessing gate fell back to the no-skill level. Writers attached the case's condition to the key, so the key was the longest option on more than half the items. It was also usually the headline textbook answer, which any solver who knows the subject picks.

Pins. Praxity/praxity-studio#1816 and Praxity/praxity-trace#24 are both merged and pinned here. The Studio pin is the last commit (e50ac09). With both pins, a real course with {shuffle} choices gets a pass or fail verdict instead of "tooling" (see End-to-end check below).

What changed

• Option shape. Options match in shape and length through detail that describes the option itself, never why it suits a case or the case's condition. The key-check judge flags any option that points at a stem detail, including by paraphrase. No option may restate or contradict a stem condition. No option repeats a content word or phrase from the stem unless every option does. All four worked examples follow this, and none of their options ends with a period.
• Key length. keyLongestCue flags a bank only when the key is the longest option on more items than chance allows (one-sided binomial, 0.05). Ties count the way Trace counts them.
• Headline cue. Some stems use the neighbouring case, so the headline answer becomes a distractor. That case must test the same objective, and the item has a one-line check for it. Example 4 is a choking diner who can still answer and breathe, keyed "cough hard to bring the food back up".
• Solver strength. Solvers and judges must be at least as strong as the writer, and the masked solver is named before the first run. The explanations reviewer may be any model.
• Hand-in checklist. One (x) and {shuffle} per scored choice, a source count of scored choices, each deliberate poll by ref, and no stem echoes.
• Seeded runs. Three runs with seeds 1, 2 and 3, fixed before solving. Seeds 2 and 3 feed only the gate.
• Gate. guessingGate(runs, { scoredChoices, polls }). Each run is { seed, bundleId, courseHash, itemCues }.
  • Before the tails it stops when:
    • the seeds aren't exactly 1, 2 and 3, the bundles aren't distinct, or the runs aren't of one course;
    • the masked solver isn't named the same on every row;
    • the course isn't its banks' disjoint union, item for item and hit for hit;
    • item metadata differs across runs;
    • a result is unscored.
  • It pools masked.right with exact tails at 1/k per item per run.
  • A clear statistical fail returns fail before any other stop.
  • After the tails it stops when:
    • the course count differs from scoredChoices;
    • a row is too small to test;
    • exclusion accounting is incomplete or doesn't reconcile;
    • an exclusion code is anything but multiple-select and the declared polls, matched ref for ref;
    • an excluded ref repeats or is also included.
  • It returns "tooling" when the report has no exclusion accounting or any item's shuffle is unknown, unless an exclusion record already shows a fault in the bank. Accounting that is present but malformed is a stop. On "tooling" the writer finishes pass 1, runs one more pass only to check pass 1's rewrites, and tells the person.
• Key check.
  • The orchestrating step plants a double-key control, not the writer. If the orchestrating agent also wrote the items, a fresh subagent briefed only with the topic writes it. It uses a single-answer stem and a ref in the course's own pattern, recorded only in the orchestrator's notes.
  • The control is exempt from the rewrite rules, and the judge gets at most one rerun.
  • A pre-ship check confirms no control remains in the course.
  • Distractor notes are scored blind before the judge sees the writer's case.
• Composer. Composer (Add composer skill and selection fixtures #31) now lists each deliberate poll by ref when it hands items to assessment-writing. It treats a "tooling" bank like a non-English one: it lets the skill check pass 1's rewrites, marks the bank unverified, blocks under Draft, finishes the run and asks the person at step 9.
• Trace pin. pack.json pins Trace 0.1.1-experimental.53ee25cc6 (source 53ee25cc620fcbc669e0f00508962156f21bdce5, fix: consume choice shuffle and count excluded items per bank praxity-trace#24).
  • Trace prerelease, built by this hosted run. Main CI on that commit: run.
  • Rehosted byte for byte on Rubato's asset-only prerelease v0.1.1-tools.experimental.20261019.5.
  • SHA-256 7596c94314fb587bbe414ed46daa5ed460af9aa009f282d6d17568a4def57aa2 matches:
    • the build log;
    • Trace's asset digest;
    • a local download;
    • Rubato's asset digest and an anonymous download from the Rubato URL.
  • All 130 payload files match the inventory, and the source revision is clean. The licence, notices, skill and report reference match the source.
  • The manifest test now refuses 97e6de3, which lacks the exclusion accounting.
• Studio pin (last commit). pack.json pins Studio 0.3.0-main.e1930acb (source e1930acba2a684bbca976e4acef5607bc92328ae). It is Studio's first admitted CLI candidate containing Praxity/praxity-studio#1816: candidate run 39361630685, where admit, build (with the CLI proof) and publish succeeded.
  • Rehosted byte for byte on Rubato's asset-only prerelease v0.1.1-tools.experimental.20261019.6.
  • darwin-arm64 8acdb90fc11cb326e72e62ec74b36757d3fec510421c32691d9177d1d87b14a6, darwin-x64 01b821fbccc3758e397e5a9ea4f6fe08d9fbddd9b1823002e5b5ed87245f5981. Each matches:
    • the build job's recorded outputs (publish step EXPECTED_*_SHA256);
    • the candidate's GitHub asset digest and .sha256 file;
    • a local download;
    • Rubato's asset digest and an anonymous download from the Rubato URL.
  • Rubato's verifyStudioArchive passes for both: clean exact source revision, single root, no links, payload inventory, legal files, notices, the private-path scan and prax-format/SKILL.md.

End-to-end check

I built a 16-item course with locale: en: 15 single-answer choices with {shuffle} and one without, as a control. I ran a real praxity inspect --schema 1 with the pinned Studio build, then the pinned Trace, then three seeded prepare/report runs with scripted answers, and the shipped guessingGate on those reports.

• Inspect reports assessment.shuffle as true on the 15 shuffled choices and false on the control. Trace reads every item's shuffle as known, with no exclusions and a full excludedByReason map.
• Masked answers fixed on option 1 hit 8, 5 and 5 of 16, so guessingGate returns "pass". Answers that pick every key hit 16 of 16 in each run, so it returns "fail". Neither returns "tooling".

Tests

The shipped gate runs on report-shaped three-run data. Every rule has a mutant: the gate with that rule removed must give a different result. The rules covered:

• run identity;
• shape across runs;
• solver name;
• union and hit sums;
• mixed option counts (a fixed 0.25 chance fails where 1/k passes);
• pooling;
• applies and status;
• small banks;
• power order;
• declared count;
• polls by ref (a swapped poll and question pass under a count-only check);
• each unexpected code;
• empty banks;
• each accounting reconciliation;
• tooling versus stop;
• shuffle in every run;
• fail before tooling.

keyLongestCue is tested at 8 and 9 of 18, with tie and threshold mutants. Prose rules are tested by mutants. A behavioural check covers every worked example for stem echoes, periods and Example 4's stated length range.

Local: node --test test/skills.test.mjs test/assessment-writing.test.mjs test/manifest.test.mjs (127 pass), npm run validate (valid). CI runs the rest. With the Studio pin, test/local-install.test.mjs and test/update.test.mjs pass 45 of 45 under WSL, as in #29. On Windows, which Rubato doesn't support, three update tests failed after running for about a minute each.

Known

• Trace still requires plain and cover answers in every answer.json, so seeds 2 and 3 can't run the masked probe alone. Their plain and cover solvers still run, but any model may answer them, nothing reads them and they need no judge. A Trace change accepting not run for those probes would remove that cost.

Review: 2/12/2 0/4/5; fixes 2. Sol and Opus rounds 1 and 2; fixed the gate pooling and length-parity test.

🤖 Generated with Claude Code