Skip to content

docs: correct triggers.md and add the trigger-benchmark method (#5) - #26

Merged
lukemaj merged 2 commits into
mainfrom
docs/5-triggers-method
Sep 29, 2026
Merged

lukemaj merged 2 commits into
mainfrom
docs/5-triggers-method

Conversation

@lukemaj

@lukemaj lukemaj commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

What: triggers.md no longer credits karpathy-wiki's SKIP and anti-rationalization shape to superpowers. It now also covers when a description is not enough, how to word routing rows, and how to test a trigger. A new page, docs/06-testing/trigger-benchmarks.md, gives the full trigger-benchmark method.
Why: The repo's description reference got its main shape's source wrong, misstated length and over-firing, and had no valid method for testing a trigger. AgentsMD has since measured all of this.
So what: A reviewer checks the exact head, then the user merges.

Closes #5
Refs #7

Changes

  • docs/05-authoring/triggers.md: four corrections; new sections "When a description is not enough", "Routing rows" and "Test a trigger" (replaces the pressure-test loop); sources pinned.
  • docs/06-testing/trigger-benchmarks.md (new): decision, fixture, cases and moments, confinement, instrument validity, interleaved arms, records, instrument checks. Every rule cites the case study or an AgentsMD PR or results file.
  • docs/06-testing/unit-tests.md: trigger benchmarks are a fourth kind of test. Superpowers removed tests/skill-triggering/ in 8611a4e, and the page now says so. The "no CI" sentence is updated because Make the doc checks run: link check, llms.txt freshness and CI #10 merged: CI runs npm test, not the validator.
  • docs/06-testing/benchmark-integrity.md: one sentence linking the new page.
  • Navigation: docs.json, README Testing list, the loader Skill's map (new entry; the triggers entry was also updated because it described the old two-shape page). GLOSSARY.md gains Arm. public/llms.txt regenerated.

Superpowers lines re-checked (obra/superpowers 8ca22db, superpowers-evals e64684c)

Claim Line Result
No SKIP / anti-rationalization in superpowers repo-wide grep for "SKIP when", "Do NOT skip", "forbidden rationali" 0 hits
Before-clause anchor skills/writing-skills/SKILL.md:171, :546-552 confirmed
Under 500 characters :103 "Keep under 500 characters if possible" confirmed
Description lengths 15 skills/*/SKILL.md descriptions, parsed 79-375, not 92-392 as the Issue says; page uses the measured range
"an agent may follow" :152-154 confirmed
Section renamed to Skill Discovery Optimization :140, RELEASE-NOTES.md:264 confirmed
One reported case :154-156 confirmed (single code-review example)
Bootstrap AGENTS.md:76 confirmed
No-guidance control, 5+ reps :582-583 confirmed
skill-not-called, vacuous-pass -before verbs evals docs/scenario-authoring.md:520-524, :660-666 confirmed
Checkbox over-trigger ~35-40% on Opus 5 evals docs/experiments/2026-09-02-opus5-signature.md:311-320 2/5 on main, 7/20 earlier sweep; Opus 4.8 0/51
tests/skill-triggering/ superpowers 8611a4e removed, "covered by drill triggering-* scenarios"

AgentsMD numbers and sources (records at 06b7af6)

Number Source
Opus 0/40 naive, 15/40 negatives loaded operations docs/work/164-trigger-audit/results.md
Pointer 3/20 to 20/20, 417 characters docs/work/174-routing-injection/results.md
Verification 0/5 to 5/5 with the before-clause case study; #181 midtask.jsonl, #182 --summary
Bare conjunct: SKILL-MECHANICS.md 15/15, prose.md 4/15 (attempts), then 5/5 recomputed from 174-routing-injection/records-prefix.jsonl (Opus, arms A/B/C) and records.jsonl (B2, B3)
Diagnosis 8/10 to 5/5; fix1 verification 3/5 182-routing-sweep/records.jsonl --summary, regenerated
OpenCode question negative 0/5 to 5/5 clean case study (#182 results)
349 tests, 17/20 with 10 no-edit, 9/10 escaped runs, 12 verdicts rescored case study

The Issue's "prose.md 7/22 against 22/22" comes only from the non-public #174 analysis quoted in #6. The public records do not reproduce it, so the page uses the recomputable 4/15 against 15/15 and marks those runs as read attempts.

Elon record

Proof

  • npm ci && npm test: exit 0 (links, llms.txt freshness and coverage, absolute paths, bare-path gate).
  • npm run build:llms regenerated public/llms.txt.
  • Independent review: Luna (gpt-6-luna, max) REQUEST_CHANGES on 5d7e16f (qualify the #174 3/20 baseline), fixed in ccaa79f, then APPROVE on ccaa79f; review/independent is success.

🤖 Generated with Claude Code

Fix the four superpowers findings after re-checking obra/superpowers at
8ca22db and superpowers-evals at e64684c, add sections on when a
description is not enough, routing rows and testing a trigger, and add
docs/06-testing/trigger-benchmarks.md with the harness lessons from the
AgentsMD routing benchmarks.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@lukemaj

lukemaj commented Sep 29, 2026

Copy link
Copy Markdown
Contributor Author

REQUEST_CHANGES

  • P2 - Qualify the #174 baseline. docs/05-authoring/triggers.md:79 says the pointer took naive prompts from 3/20 to 20/20. The cited #174 results later explain that two base-arm test-design runs read the required file through a chained cat that exited non-zero; one read was before the first edit, so the corrected baseline may be 5/20. Please identify 3/20 as the originally scored value and include that caveat (or report the recomputed value).

npm test passes in the specified worktree. The remaining checked acceptance criteria, changed-file scope, internal links, glossary terms, and textual constraints pass.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@lukemaj

lukemaj commented Sep 29, 2026

Copy link
Copy Markdown
Contributor Author

APPROVE

The #174 baseline wording at docs/05-authoring/triggers.md:79 is fixed: 3/20 is explicitly the scored baseline, and the paragraph notes the possible higher count from the chained-cat reads with a link to the case-study caveat. I rechecked the exact head ccaa79f928064eeb05ec6610ad79c5e4dc8fa93c; I found no remaining actionable issues in the reviewed scope.

npm test passes in the specified worktree with Node 22.23.1. This repository's Mintlify link check does not support the environment's default Node 26.10.0.

@lukemaj
lukemaj merged commit 39f46b3 into main Sep 29, 2026
3 checks passed
@lukemaj
lukemaj deleted the docs/5-triggers-method branch September 29, 2026 21:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Correct triggers.md and add the trigger-benchmark method

1 participant