Repository navigation
Conversation
Empirical validation against
|
| Target | blis on examples/regression_workload_cache_warmup.yaml — a purpose-built routing regression workload with 3 SLO classes and documented known results (routing swings +242% sabotaged → −28% oracle), so the surface was known non-flat before either arm started |
| Objective | total goodput_rps summed across SLO classes |
| Factors | 6 two-level: ROUTE, SCHED, DISPATCH, PREEMPT, MAXRUN, TOKENS |
| Ground truth | all 64 configurations, computed exhaustively in 41 s |
| Observed range | 23.45 → 117.85 goodput (5.0× spread) — a live surface |
Both arms called one shared evaluator and nothing else, so any difference comes from search strategy rather than from how configurations were run or scored. Screen design verified balanced, orthogonal, and genuinely resolution IV (zero 2fi aliased on main effects) before any result was trusted.
Token budget — the currency that matters
BLIS runs are ~0.6 s and cost zero tokens. Costs below are measured from 69 real LLM calls across 4 corpus campaigns, not estimated:
| Arm | LLM calls | Cost | Best found |
|---|---|---|---|
| reflective, 3 iters | 20.4 | $31.84 | 117.85 (100%) |
| reflective, 4 iters | 27.2 | $42.45 | 117.85 (100%) |
| optimization | 6 | $9.38 | 117.85 (100%) |
3.4–4.5× cheaper in tokens for the identical answer. Measured reflective rate is 6.8 LLM calls and $10.61 per iteration — higher than the design spec assumed, because design-phase retries dominate (2–20 calls per campaign in the corpus).
The asymmetry is structural rather than tuned: orchestrator/optimize/ contains no dispatcher, no completion_fn, no sdk_runner anywhere, so screen and refine cannot make a model call.
Why quality tied — the honest caveat
Both arms reached the true optimum. Diagnosing why, rather than declaring a tie:
This surface is unimodal under single-factor moves. Greedy steepest-ascent from all 64 starting points reached the global optimum from every one. So sequential search cannot get trapped here, and the factorial design's structural advantage has nothing to exploit.
That is not because the target is trivial. It has three real interactions — MAXRUN×TOKENS −5.84, SCHED×MAXRUN −3.52, SCHED×TOKENS −3.12 — and MAXRUN shows the exact L5 sign-flip property the design spec is built around: −43.82 alone, +81.03 in the best context. Interactions exist; they simply do not create a local trap.
This is consistent with the guide's own framing, which already says the composition barrier is "an artifact of OFAT search, not an inherent property of the underlying system." This target is a system where the artifact does not bite.
One reflective ordering did finish at 90% of optimum (106.17), but tracing it showed a budget artefact of single-pass coordinate descent — MAXRUN came last, and TOKENS had been fixed at a level that was better before MAXRUN moved. A second pass recovered the optimum. Not a landscape trap.
What this means for the PR
The result argues for one doc addition rather than a code change, and it is being made in this PR: the guide explains why factorial beats OFAT but never tells an author how to tell which case they are in. On a unimodal surface the optimization kind still wins on tokens but its headline advantage is dormant, and an author deserves a cheap way to check — the 64-point greedy probe here cost 41 seconds and answered it definitively.
Also worth recording: the experiment exercised the harness end to end and found no defects in it, which is the first independent use of the feature outside its own test suite.
Reproduction
Scripts under /tmp/optexp/ (harness.py, arms.py, plus ground_truth.json and comparison.json). Not committed — they target a specific local inference-sim checkout and a built blis binary, so they would be dead weight in this repo. Happy to add them under examples/ if that is useful.
Caveat on scope
One target, one workload, one objective, 6 factors. It establishes the token claim and demonstrates the machinery works against a real system. It does not establish that optimization finds better optima than reflective in general — that needs a target whose surface genuinely traps greedy, which this one does not. Two candidates: a factor set including deliberately pathological levels (the always-busiest routing policy alone produced a 13× goodput cliff, 7.34 vs 98.66), or a 10–12 factor space where reflective's per-iteration cost compounds while optimization stays flat at ~6 calls.
Update — superseded by measured runs. The cost figures in this comment were computed arithmetically from corpus per-call averages, and both "experiments" here simulated the two search strategies as hand-written Python calling
blisdirectly. Neither invokednous. That is stated plainly at the end of the next comment, and it is why the headline "token claim confirmed" overstates what this establishes: it validates the search strategy comparison, not the two campaign kinds end to end.Superseded by real, measured campaigns:
- Experiment 3 — optimization complete at $0.082, reflective $5.451 and unfinished (66×), from
llm_metrics.jsonlrather than estimates.- Experiment 4 — both kinds author Go code first: optimization $2.65 complete, reflective $6.49 for ~1.3 iterations.
The measured per-phase split also gives a sharper account than the 3.4–4.5× here: authoring costs roughly the same in both kinds, and what optimization removes is the recurring per-iteration measurement cost ($4.77 vs $0 in experiment 4).
Experiment 2 (preemption starvation, discussion #521): mechanism reproduced, but the target cannot discriminateSecond attempt at a target that could test the quality half of the claim, since experiment 1's surface turned out unimodal. Grounded in inference-sim discussion #521, "Preemption starvation under sustained KV overload", whose Question 4 is an explicit invitation: "Is progress-aware eviction worth a hypothesis experiment? Test whether 'evict least progress' beats 'evict youngest' under sustained KV pressure." Isolation note: experiment 2 ran in its own directory with its own harness, cache, factor set, and workload. Nothing shared with experiment 1 except the read-only The mechanism reproduces, and it is severeThe original reproducer needs an HF-gated model (
Not the total starvation (0 completions) the discussion documents for llama-3.1-8b — different KV geometry — but unmistakably the same mechanism, and severe. Partial starvation is actually better for a comparison, since 0 completions everywhere would tie every configuration at zero. Only one factor is live, and it is not a mitigationTesting each of #521's five candidate mitigations in isolation against the baseline:
The only knob that moves the outcome is the pressure knob, not a mitigation. Why
Why this still cannot discriminate the two kindsA one-factor space is exactly where one-factor-at-a-time search is provably optimal. Different reason from experiment 1 (which was unimodal despite six live factors), same conclusion: no discriminating power on quality. Standing state of the two claims
Two independent failures to find a greedy trap is itself weak evidence that these surfaces are mostly benign. If that holds up, the honest pitch for Correction to experiment 1's framingOne thing I should state plainly, because it bears on how much weight to give the numbers above and in the previous comment: neither experiment invoked
|
Two REAL nous campaigns, head to head — and eight bugs that only a real run could findPrevious comments compared simulated search strategies (hand-written Python calling SetupIdentical for both arms: Independent ground truth: all 64 configurations swept exhaustively beforehand. True optimum 117.854. Result
~66× cheaper, and reflective had not finished. The cost driver was not what I expected: cache reads are 75% of reflective's bill ($4.08 of $5.45). That is the SDK agent re-reading repo context on every turn. Optimization's cache reads are zero, because For an external check, discussion #1388 documents a prior real nous reflective campaign on adjacent BLIS machinery at $5.26 / 43 minutes — close to the $5.45 measured here, which suggests this reflective figure is representative rather than anomalous. Quality: a tie, and honestly a tieReflective found 117.779 via seven hand-picked configurations ( This target cannot demonstrate a quality win, consistent with the earlier finding that its surface is unimodal — greedy steepest-ascent reaches the optimum from all 64 starting points. So the claim this experiment supports is same answer, ~66× cheaper, not "finds better answers". Credit where due: reflective's findings are richer prose. It identified The part that matters more than the numbers: eight real bugs
Bug 8 deserves emphasis: the campaign got the right answer and reported the wrong one, silently. That is precisely the confidently-wrong failure mode this feature exists to prevent, and only a real run surfaced it. Bug 3 and bug 6 are design defects rather than slips:
Also added:
|
Experiment 4 — both kinds must extend BLIS first, then optimizeEvery prior comparison here varied only existing CLI flags, so the
Isolation. Separate git worktrees on separate branches at identical HEADs, separate Ground truth (90-config exhaustive sweep, independent implementation): 182.2159 at Result
Both arms authored a correct mechanism. Three independent implementations — reflective's, optimization's, and my reference — agree to four decimals at the optimum and at exponential. That is the strongest evidence available that the mechanism is right, and it means this comparison measures search strategy, not implementation luck. Both answered #1367 honestly. Optimization's fitted effects put On cost: the advantage is real, but its source moved$2.65 vs $6.49 is 2.5× for a completed campaign against an unfinished one. That is far from experiment 3's 66×, and the per-phase breakdown explains why:
Both kinds pay a comparable one-time cost to author code — an agent working in a repo is expensive either way. What optimization eliminates is the recurring per-iteration cost: 33 real benchmark runs for zero tokens. The structural claim that survives contact with code-authoring campaigns is therefore narrower than "66× cheaper" and more durable: the authoring cost is unavoidable in both kinds; the per-iteration measurement cost is what this kind removes. The more iterations a question needs, the wider the gap. I am deliberately not claiming the ~7× that extrapolating reflective to 3 iterations would give. The run that would have tested it died for unrelated reasons. The 67.6% needs its caveatReflective's shortfall is not a failure of search. It pinned One genuine reflective advantage worth recording: its DESIGN phase caught an error in my mechanism spec. I had declared the metamorphic ordering as Four more defects this run surfaced, all fixed
Defect 9 is the one worth dwelling on: Defect 11 is the sharpest: the system diagnosed the problem, recorded the correct diagnosis, and then did the thing it had just warned against, because Defect 12 is an irony worth noting: Also in this batch
What is still open
|
`kind: optimization` could not extend a target. Every stage in orchestrator/optimize was pure Python with zero agent calls, so when the mechanism under study did not exist yet, `verify` ran the target's test command, found the declared native_test identifiers missing, and aborted fail-closed — correctly and uselessly, since nothing in the path was able to author them. The guide already claimed verify "writes mechanism code with native tests"; no code did that. Reflective never had this problem (its SDK agent edits the target as a matter of course), so the gap was specific to this kind, and it mattered: "extend the target, then optimize the extension" is the common shape of a real optimization request. `build` is an opt-in stage, absent from the default order, so campaigns written before it behave identically. It spends exactly ONE agent call to write the mechanism and its native tests, then returns to the tokenless machinery. It deliberately does not iterate to green, read measurements, or judge correctness: `verify` remains the gate, so the stage that writes the code is never the stage that certifies it. Cost goes from ~3 model calls to ~4 only for campaigns that need it. Two validator rules, both of which fire on real authoring mistakes: - rule 11 rejects `build` anywhere but position 1. Behind `verify` it guarantees an abort; behind `screen` it is worse than an abort, because the screen measured the OLD mechanism and the effect table then describes a system the campaign replaced. Both produce schema-valid artifacts, so neither is caught downstream. - rule 12 warns when declared native_test files are absent from the target and no build stage exists to author them — otherwise guaranteed to abort at verify, but only after a real run. Also: the test command no longer runs on a build iteration. It could only report the already-known fact that the identifiers are missing, and `go test -run <pattern>` exits 0 with "no tests to run" when the pattern matches nothing, so a pre-build run looks like a pass at the shell level. Tests inject at the sdk_runner seam per tests/CLAUDE.md — no live calls. Two are integration: BUILD authors a file in the target and CONTINUEs, then VERIFY independently aborts when the tests do not actually report. iteration.py remains 23 insertions / 0 deletions against main; the reflective path is untouched. 1673 passed, 1 skipped. Refs AI-native-Systems-Research#302 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…st found A fitted stationary point is an EXTRAPOLATION. When the surface is mis-specified — curvature the quadratic cannot express, an optimum outside the design hull, a categorical factor dragged into the fit — the solved point can land below a corner the screen already measured. confirm then replicated that inferior point, found its replicates agreed with each other, wrote status CONFIRMED, and the campaign reported an inferior configuration as its optimum. "The prediction reproduced" and "this is the best configuration found" are different claims, and only the second is what a reader of the report wants. _best_observed already existed but was used only to PICK the confirm point when no fit existed; it never CHECKED a fitted one. confirmation.json now carries best_observed, confirmed_is_best_observed, and (when they disagree) regression_vs_best_observed with the absolute and percentage gap, plus a warning naming the observed corner as the campaign's real answer. Respects response.primary.direction, so lower-is-better objectives are judged correctly rather than inverted. 1676 passed, 1 skipped. Refs AI-native-Systems-Research#302 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Found by running it: a build stage whose cwd was its own git worktree read and then EDITED the canonical checkout of the same project instead. Four tracked files in the wrong tree. Two supposedly independent campaign arms would have overwritten each other's mechanism, and every measurement after that would describe a system neither arm authored — silently, with no error anywhere, since both trees build and both test suites pass. Passing repo_path as `cwd` is not sufficient. When the campaign description mentions file paths and the agent already knows a checkout of that project, it resolves the paths against the checkout rather than against cwd. The prompt now states the absolute working root up front, says the path may be a worktree with a parallel experiment in a sibling checkout, and requires every path to be re-derived against that root. Prompt text alone is not a guarantee, so check_build_touched_repo() reports when a git target shows no local modifications after a "successful" build -- the signature of having edited somewhere else. Advisory, not fatal: verify remains the authority on whether the mechanism is present, a legitimate no-op build should not abort a campaign, and non-git targets are exempt. The warning also lands in build_warning.txt so it survives in the artifacts. 1680 passed, 1 skipped. Refs AI-native-Systems-Research#302 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…fy aborts Three diagnostics losses, all of which mattered most exactly when something had gone wrong: 1. run_test_command returned dict[str, bool] and discarded the output. The gate needs the booleans; a human fixing the failure needs the text — the failed assertion's expected-vs-actual, a panic and its stack, a compile error's file and line, a timeout's partial output naming the test that hung. Since a verify abort ends the campaign, that text was the only record of why, and re-running by hand may not reproduce a timeout or an ordering-dependent failure. Now written verbatim to runs/iter-N/test_output.log, including on timeout and exec failure. 2. A failed config run kept only the last 200 chars of stderr, in an exception message. The cause is usually at the HEAD of stderr (a usage error naming the rejected flag), so the tail dropped the useful part. Full stdout/stderr now land in runs/iter-N/failed_runs/failed_run_<i>.log with the row index and its levels; the exception points at the file. Same for output that parses to no JSON, which is the failure that NaN-poisons a fit. 3. The verify abort listed relation IDs without saying whether each test FAILED or was NEVER EXECUTED. Those need opposite fixes, and after a build stage the agent call is already spent, so a wrong diagnosis costs a whole second build. The message now splits them and names the fix for each, including the trap that makes "never executed" silent: go test -run with a pattern matching nothing exits 0. Logging is opt-in (log_path / log_dir default to None) and can never be the reason a campaign fails — every write is guarded and degrades to a warning. Guide: metamorphic relations now carry an explicit warning to do the algebra first and include a worked numeric example, with the real case from this campaign corpus where an author declared step <= linear <= exponential when the truth is step <= exponential <= linear (the exponential is concave in f). A metamorphic relation pointing the wrong way fails correct code or passes broken code, with the authority of a declared correctness relation. 1686 passed, 1 skipped. Refs AI-native-Systems-Research#302 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ults _format_results_summary walked only runs/iter-*/results/ — the reflective layout. `kind: optimization` stores measurements as runs.jsonl (one JSON object per configuration), so the helper emitted "results/ directory absent" for every iteration of a campaign that had measured everything it planned. Observed on a real completed campaign: the report announced a "storage/persistence gap in the apparatus", repeated it per iteration, and cited it as a reason every effect was low confidence -- while 33 rows of per-configuration telemetry sat in runs.jsonl beside it, with the top three measured values matching an independent ground-truth sweep to four decimals. That is precisely the search-orientation failure (AI-native-Systems-Research#214) this helper was added to prevent: a report dismissing real data as no data. It was reintroduced for a layout the helper did not know about. Now both layouts are enumerated, with the measured/complete counts and the neighbouring design_matrix / effects / confirmation / relations artifacts named so the extractor can read them. Malformed lines are counted rather than allowed to blank the iteration, and the reflective results/ path is untouched (test asserts it still wins). 1689 passed, 1 skipped. Refs AI-native-Systems-Research#302 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…hull decide_after_refine already detected this and said exactly the right thing -- "stationary point falls outside the declared design space on BANDCAP, THRESH; ranges were too narrow to contain the optimum -- escalate before confirming" -- but Trigger is documented as reported-not-acted-on, so confirm read the same solve off disk and replicated it anyway. The system diagnosed the problem, wrote it into findings, and then did the thing it had just warned against. Observed on a real completed campaign: goodput was monotone in both refined factors, so no interior optimum exists at all. The quadratic solve landed at coded BANDCAP=+1.62 / THRESH=-2.30, well outside the hull, and confirm replicated it for a mean of 112.4997 -- while the campaign had already MEASURED 182.2159 at a corner, which an independent ground-truth sweep confirms is the true optimum. 38% off, with replicate spread 0.0 so it looked perfectly reproducible. _read_confirm_at now returns None when any coded coordinate leaves [-1, 1], which routes confirm to the best OBSERVED configuration -- a weaker claim than reproducing a fitted optimum, but a true one. The warning names the offending factors and the fix (widen those ranges, re-run refine). Verified against the real campaign's own confirm_at.json: it now yields None, so that campaign would confirm 182.2159 instead of 112.4997. The boundary is inclusive: +/-1.0 is ON the hull, so a factorial corner remains a legitimate confirm target. This pairs with the confirmed_is_best_observed guard from 0e12a43. That guard reports the discrepancy after the fact; this prevents it happening. 1692 passed, 1 skipped. Refs AI-native-Systems-Research#302 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ssing The module docstring carried a TODO saying config_runner/integrity_check "production wiring ... is not built here", and the config_runner abort told the reader it was "a follow-up". Both were true when written and false since e410bbc wired them; four real campaigns have since completed through that path. A stale "not built" note is worse than no note, because it tells the next reader to build something that exists. The docstring now says what is actually true: the seams stay injected because that is what makes the module testable without subprocesses, and run_stage resolves both from the campaign when a caller supplies neither. It also records why the gap mattered — until the wiring existed every real campaign aborted at verify or screen while 1600+ tests stayed green, because they inject fakes at exactly the seams that were missing. The abort message now names the actual cause a reader can act on (no optimization.run_command, or no repo_path) instead of pointing at a non-existent follow-up. 1692 passed, 1 skipped. Refs AI-native-Systems-Research#302 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This kind makes only a handful of model calls -- one `build` (which authors the mechanism every later stage measures), one interpretation at the end, and a small gate summary per iteration -- while the tokenless screen/refine/confirm stages carry the bulk of the work. So the marginal cost of using the strongest model for all of them is small, and the downside of a weaker model on the build call is that every downstream measurement describes worse code. _resolve_model is now kind-aware: `kind: optimization` returns OPTIMIZATION_MODEL for every phase, reflective keeps its per-phase defaults.yaml entries untouched (design opus-4-6, execute_analyze/report sonnet-4-6), and an explicit campaign.models.<phase> still wins so an author can pin something cheaper. build.py previously hardcoded "claude-opus-4-6" in two places (the SDK call and the metrics row). It now resolves through the same precedence as the rest of Nous, so the metrics row names the model actually used and cost accounting stays honest. defaults.yaml also gains an explicit `build` entry for the same reason. Verified claude-opus-5 against the configured proxy: HTTP 200, and the usage block carries cache_creation_input_tokens / cache_read_input_tokens, so the existing cost accounting works unchanged. The proxy's /model/info is restricted, so per-token pricing is not discoverable at runtime -- costs continue to be computed from published rates. 1696 passed, 1 skipped. Refs AI-native-Systems-Research#302 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three build stages aborted without producing a mechanism, and the cause was a spec defect, not an agent failure. A campaign's mechanism spec described its baseline leg in terms of the deployed system's behaviour and quoted an exact figure for it. The figure was correct and reproducible; the LABEL was not -- one of the behaviours named in the label was not modelled by the target's evaluator at all, so it could not have contributed to the number. The build agent could not distinguish "the author mislabelled this leg" from "my implementation is wrong", so it did the diligent thing: twenty-plus shell probes grid-searching variants to close a sub-percentage-point gap that was unclosable by construction. It never reached the mechanism. That is the single call the campaign spends on the target, so the whole run produced nothing. Two fixes, at the two places the failure can start: - The build prompt gains a BUDGET DISCIPLINE section: reference numbers are a one-time sanity check, not a target to fit; a divergence should be recorded in the summary and the build continued; behaviour the spec asks for that does not exist in the target should be noted as absent rather than hunted for; probe scripts are spend that buys no measurement. The summary now explicitly asks which reference numbers failed to reproduce, so a mislabelled spec surfaces as a reported finding instead of a silent search. - The authoring guide gains "Reference numbers in a spec: verify them, or leave them out", with two rules: name a leg by the code path that produces it rather than by what the production system does, and state explicitly what to do on a mismatch. 1697 passed, 1 skipped. Refs AI-native-Systems-Research#302 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A campaign whose build stage did everything asked was aborted by verify. The
log line was the whole story:
test_command reported 88 test(s); 2 matched a declared native_test
All 88 tests passed. But pytest reports a parametrized test once per case
(`test_no_rebalancing[0.95-1.05]`), and match_declared_tests only did exact
trailing-identifier matching, so a declaration naming the bare function matched
nothing. Four of six declared identifiers were parametrized; those four
reconciled as "declared but not executed" and the fail-closed path killed the
run. Cost: one full build stage, ~$7, for correct and fully tested code the
matcher could not recognise.
The matcher now strips a pytest `[...]` parametrization suffix and a Go
`Test/subtest` path, so a bare declaration aggregates every case. Aggregation is
all(), not last-seen: one failing case fails the relation, which keeps
fail-closed intact rather than letting whichever case was iterated last decide.
A declaration with no matching result is still absent from the output, so a test
that was never written still fails.
This is the same class as the earlier `go test -run` finding: the test runner's
output format, not the test outcome, decided whether a relation could be
verified. Worth noting the guard behaved correctly at every step -- it refused to
fit a surface on an apparatus it could not confirm. The defect was that it could
not confirm an apparatus that was in fact sound.
Tests cover pytest parametrization, Go subtests, mixed pass/fail aggregation,
plain names (no regression), and the still-absent case. Verified against the
real campaign artifacts: 88 reported, 6/6 declared matched, 0 correctness
relations failing.
1702 passed, 1 skipped.
Refs AI-native-Systems-Research#302
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Six campaign runs aborted before producing a measurement. Static validation
passed cleanly before every one of them, because five of the six failures were
BEHAVIOURAL -- they live in the contract between the campaign and the target,
and nothing was executing that contract:
* a declared native_test the test runner never reports (fail-closed abort at
verify for tests that exist and pass);
* a run_command that cannot exec (an inline VAR=value prefix is parsed by
shlex as the binary name);
* an objective metric absent from the emitted JSON, so every run parses and
scores NaN;
* a manipulation predicate comparing a level (a string) against a value the
target emits as a bool -- 67 of 67 runs rejected while the target's own CLI
was entirely correct.
`--smoke` runs the test command and ONE configuration at every factor's first
level, and reports all four. It costs one test run plus one config; each of
those failures otherwise costs a full campaign.
The predicate check builds its scope the way run_stage does -- the target's own
`applied` echo wins over the requested levels. Using the requested levels would
make `applied.X == "{level}"` trivially true and hide the exact mismatch the
check exists to find. Verified by reintroducing the bug: exit 1, with the type
named as the suspect.
Also fixes refine dropping non-designed factors. refine builds a central
composite over refinable (numeric, >2 level) factors only, so a `choice` factor
contributed no level and therefore no `apply` fragment -- its CLI flag vanished
from the command line and a target that correctly requires all of its flags
failed every run on a usage error (observed: 48 of 48 refine and confirm runs).
Non-designed factors are now held at their first declared level, which for a
two-level ablation is the control, and the held value is recorded as
`held_fixed` in design_matrix.json so it is auditable rather than implicit.
Guide and CLAUDE.md document --smoke as the last step before launching.
1709 passed, 1 skipped.
Refs AI-native-Systems-Research#302
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Six aborted runs on a third target, and the validator passed cleanly before every oneExperiments 3 and 4 ran this kind against a Go target. This round ran it against a Python target with a genuinely expensive evaluator (a one-time ~110 s index build, then sub-second per configuration), authoring a new mechanism plus its native tests. Six runs aborted before producing a measurement, and the interesting part is why they were not caught earlier.
Two harness bugs, both found only by running1. Parametrized tests never matched their declaration ( pytest reports a parametrized test once per case ( This is the same shape as the earlier 2. The fix that matters most:
|
| Check | Failure it prevents |
|---|---|
declared native_test identifiers appear in the runner's output |
fail-closed abort for tests that exist and pass |
run_command execs and emits parseable JSON |
every run dies on a usage error |
| objective metric present in the output | every run parses and scores NaN, poisoning the fit |
| manipulation predicates hold at the first level | every run rejected while the target is fine |
One subtlety worth recording, because I got it wrong on the first attempt: the predicate check must build its scope the way run_stage does, with the target's own applied echo winning over the requested levels. My first version used the requested levels, which makes applied.X == "{level}" trivially true and hides the exact mismatch the check exists to find. Verified by reintroducing the bug — the corrected version exits 1 and names the type as the suspect:
x factor <F>: manipulation predicate fails at its first level
(applied.<f> = False == '0' -> FAIL). Every run would be rejected.
Check the observable's TYPE -- a level is a string, and a target that
emits a bool or int for it can never compare equal.
Cost: one test run plus one configuration. Each of those four failures otherwise costs a whole campaign.
The authoring lesson: an unverifiable claim in a spec is a bug
Three of the six aborts traced to the mechanism spec I wrote, and the most expensive was not a wrong number — it was a wrong label. The spec described a baseline leg in terms of the deployed system's behaviour and quoted an exact figure. The figure was correct and reproducible. But one behaviour named in the label was not modelled by the target's evaluator at all, so it could not have contributed.
The build agent could not distinguish "the author mislabelled this" from "my implementation is wrong", so it did the diligent thing: twenty-plus shell probes grid-searching variants to close a gap that was unclosable by construction. It never reached the mechanism. That is the one call the campaign spends on the target.
Two fixes, at both places the failure can start (3a42e86):
- The build prompt gained a
BUDGET DISCIPLINEsection: reference numbers are a one-time sanity check, not a target to fit; record a divergence in the summary and continue; note behaviour the spec asks for that does not exist rather than hunting for it; probe scripts are spend that buys no measurement. The summary now explicitly asks which reference numbers failed to reproduce, so a mislabelled spec returns as a reported finding instead of a silent search. - The guide gained "Reference numbers in a spec: verify them, or leave them out", with two rules: name a leg by the code path that produces it, not by what the production system does; and state explicitly what to do on a mismatch.
The next run wrote the mechanism at turn ~40 with zero probe scripts, versus never at turn 89. Turn budget went 200 → 120 so a probing spiral hits a wall and reports rather than consuming the whole allowance.
kind: optimization now runs on claude-opus-5 throughout (7d91866)
Every phase of an optimization campaign resolves to OPTIMIZATION_MODEL, rather than the per-phase defaults.yaml entries the reflective kind uses. The reasoning: this kind makes only a handful of model calls while the tokenless stages carry the bulk of the work, so the marginal cost of the strongest model is small — and a weaker model on the build call means every downstream measurement describes worse code. Reflective's defaults are untouched (test asserts it) and an explicit campaign.models.<phase> still wins.
While wiring this I found build.py had a model hardcoded in two places — the SDK call and the metrics row. So even with --model passed, the metrics row named the wrong model, quietly corrupting cost accounting. Both now use the resolved value.
Where this leaves the kind
The build stage does work, and well: on the successful run it authored a 27 KB mechanism and a 29 KB test file (68 tests, all passing), reproduced every reference value to full precision, and corrected a metamorphic relation I had specified wrongly — proving the invariant under a narrower condition and pinning a counterexample as an assertion. Marking that relation behavioral rather than correctness is what let it report the boundary instead of hard-failing on it.
What this round demonstrates is narrower than the earlier experiments and more useful: the tokenless stages are only as good as the campaign/target contract, and that contract needs executing, not just validating. Six aborts, two harness bugs, one new command. 1709 tests pass; iteration.py remains 23 insertions / 0 deletions against main.
Two defects raised in review, both real, both verified in code and by test. 1. confirm labelled the fitted stationary point `role="center"`, and matrix._decode_level treats that role as "ignore the coded coordinates, use the midpoint of every declared range". Correct for a genuine replicated centre point; catastrophic for the one point confirm exists to replay. A stationary point at coded +0.9 of [64, 256] ran level 160 -- the midpoint -- instead of 246, and the campaign then reported "the predicted optimum reproduced" about a configuration the fit never predicted. This also re-diagnoses an earlier finding. A real campaign's confirm reported a mean 38% below a corner its own screen had already measured. That was attributed at the time to the response surface extrapolating badly. The actual cause was this: confirm never ran the fitted point at all. The best-observed guard added in 0e12a43 caught the symptom; this is the cause. Fixed by using role="axial", which routes through decode_coded and so interpolates and grid-snaps any coded value -- including 0.0, so the no-refine case still yields the origin exactly as before. 2. The recorded run order was not the executed run order. matrix_payload generates a seeded permutation, records it as `run_order` in design_matrix.json, and expand's docstring tells callers to consult it. Nothing did. Every configuration still ran and the fit was unaffected, so this is a provenance defect rather than a numerical one -- but run-order randomization is what protects a factorial design against drift confounding (a warming cache, a throttling machine, a background job), and an artifact claiming a guarantee the run did not provide is worse than one that claims nothing. Fixed by permuting the rows handed to execute_design. This introduces a sharper hazard than the bug it fixes: _fitting_responses walks outcomes positionally and fit_effects pairs value i with design.points[i], so leaving outcomes in execution order would misalign every response with its design row and produce silently wrong coefficients. Outcomes are therefore restored to design order before anything reads them positionally, while runs.jsonl still records the true execution sequence because on_row fires per completion. Four tests: the fitted point is not the midpoint (verified to fail against the old role), the no-refine origin case is unchanged, execution follows the recorded permutation, and each row's response still matches its own levels -- the last one is the guard against the misalignment hazard above. 1713 passed, 1 skipped. Refs AI-native-Systems-Research#302 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two review findings, both real — and one of them re-diagnoses an earlier commentBoth claims raised in review are correct. I verified each in the code and by test rather than by reading, and one of them overturns a diagnosis I gave earlier in this PR. 1.
|
Five tests for `nous validate campaign --smoke`, one per failure it exists to catch: a manipulation predicate whose type can never match, an objective metric the target does not emit, a run_command that cannot exec, an unmatched native_test, and the no-false-positive case on a sound contract. Two tests for the refine fix: a non-designed `choice` factor still reaches the target on every row, and the held-fixed value is recorded as `held_fixed` in design_matrix.json so it is auditable rather than implicit. Also removes a duplicated `_smoke_campaign` helper. Two definitions of the same name existed in this file and the second shadowed the first, so edits to the first had no effect and the surviving fixture defaulted `test_cmd` to `true` -- a command that reports no per-test results at all, which made the sound-contract test fail for a reason unrelated to what it was testing. Worth noting the smoke check was right every time: it correctly reported that a declared native_test never appeared in the output of `true`. 1713 passed, 1 skipped. Refs AI-native-Systems-Research#302 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…oned the fit
Two latent defects, both verified on the branch before the fix and both now
pinned by tests that fail when the fix is reverted.
D1 -- every tabulated resolution-IV screen aborted at fit. fit_effects added one
column per two-factor interaction, but at resolution IV aliased interactions have
IDENTICAL +/-1 columns, so X^T X was singular and _solve_normal_equations raised
"design matrix is singular". Measured before: k=5,6,7,8 all raised; resolution V
was fine. So `resolution: 4` was documented, permitted by the validator (rule 7
only warns), and guaranteed to abort every campaign that declared it.
Interaction columns are now collapsed into ALIAS CLASSES: one estimable
coefficient per class, with the confounding recorded rather than hidden. A
fractional design cannot separate aliased terms -- no arithmetic can -- but it can
say what each coefficient is confounded with, which is what makes aliasing a
resource question ("resolve it only if it could change the decision") instead of
a crash. All of k=4..8 at resolution IV now fit with every main effect estimable,
and resolution V is unchanged (15 effects, no aliases).
D2 -- a single infeasible row silently NaN-poisoned every coefficient.
_fitting_responses carries NaN for any non-complete row, and the abort guard
deliberately exempts `infeasible`/`rejected` because a constrained design
routinely has inadmissible corners and aborting on one would make constraints
unusable. But those NaNs flowed into fit_effects, where one NaN makes every
coefficient NaN while still returning a schema-valid Fit. Verified:
[0.1875, -0.5625, -0.0625, 0.1875] became [nan, nan, nan, nan] with no error
raised and no warning logged.
The comment above that guard already stated the intended behaviour -- "refit on
the completed rows and report the reduced resolution honestly" -- but nothing
implemented it. run_stage now fits on the complete-row subset, logs which rows
were excluded, and writes fit_exclusions.json so the reduced resolution is
visible rather than implied. Fitting aborts only when fewer than two rows remain.
Existing coverage missed D2 because it asserted the NaN was *carried*, not what
the fit did with it.
Also lands the design spec and implementation plan for the paper's decision layer
(compiled policy, recommend(), residual-regret certificate, terminal
discrimination, foldover, epoch semantics): 16 tasks across 6 phases, oracle-first,
with these two defects folded in as the first work items of Tasks 10 and 7.
1718 passed, 1 skipped. iteration.py remains 23 insertions / 0 deletions.
Refs AI-native-Systems-Research#302
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Documents what Tasks 1-15 built, for the audience that has to read the artifacts rather than the code. `docs/optimization-campaign-guide.md` gains an unnumbered "The compiled policy" section (placed after §2 so the existing §3-§6 cross-references stay valid): the `policy` / `known_valid_baseline` / `workload` blocks and `design.confirm.shortlist_size` field-by-field, the states-and-registered- branches table, how to read `report.json`'s six-valued fallback ladder and its two separately-reported bounds, the full artifact inventory, what a semantic exception is, and the four steps that start epoch 2. It ships one complete example campaign declaring every field in the section, so the existing extraction tests hold it to the real validator. `docs/data-model.md` §7 grows from four artifacts to the complete, code-verified set: `policy.json`/`policy.sha256`, `transitions.jsonl`, `recommendation.json`, `fit_exclusions.json`, `confirmation.json`, `shortlist.json`, `report.json`, `epoch_end-<epoch>.json`, and the three build oracles' records — each with its location, the function that writes it, and whether it is schema-governed. `CLAUDE.md`'s optimization section gains the artifact tables, "never edit policy.json" with the epoch-boundary recompilation that IS the legitimate way to change a design, the two-bounds rule, and the `basis` ladder. Two places where the spec's artifact table and the code disagree are documented as such rather than papered over, with the code as the authority on names: spec §3.9's `epoch.json` is really `epoch_end-<epoch>.json`, and its `alias_map.json` does not exist — that content is `effects.json`'s `aliases` plus `recommendation.json`'s `alias_consequential`. `docs/superpowers/specs/2026-08-13-optimization-campaign-kind-design.md` gets a superseded-in-part banner plus in-place notes at §5.5 and §6.3. Its §6.3 "escalation triggers that *do* re-consult the model" and §5.5's "the model authors prose at verify and confirm" are both false of the shipped code; a reader landing mid-file now finds that out there. `docs/targets.md` (Task 15) had zero inbound links; it is now in the "See also" list of both the guide and `CLAUDE.md`. Also corrects three stale claims in `docs/data-model.md` §7c found while verifying against the code: `effects.json` is written at three fitting states (`foldover` too, not just screen/refine), its `aliases` are the fit's own alias classes rather than a carry-over from `design_matrix.json`, and the file's header no longer claims a fixed artifact count. Tests: four new checks in `tests/test_optimize_guide_examples.py` — the guide names the compiled-policy vocabulary, its example declares `policy`/`workload`/`known_valid_baseline` and validates as a real campaign, `docs/data-model.md` names the artifacts the code actually writes, and `docs/targets.md` is reachable from both entry points. 2003 -> 2007 passed, 0 xfail. No production code changed; zero model calls anywhere in the diff. Refs AI-native-Systems-Research#302 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…floor `docs/targets.md` §5 said "set epsilon above your noise floor or the campaign cannot certify anything." That rule of thumb is wrong in both directions, and all three shipped example campaigns contradict it — vllm pairs a 2% epsilon with a 4% noise estimate, qdrant 2% with 3%, knative 3% with 8%. What epsilon actually has to clear is the *terminal Bonferroni-t bound at the planned replicate count*, roughly `t · σ · √(2(1−ρ)/n)` over `M = shortlist_size − 1` challengers. Measured against the shipped `certificate.terminal_regret_bound` (not a reimplementation of it): at vllm's 4 paired replicates with ρ≈0.9, a winner ~5% clear of the field certifies ~100% of the time and a near-tie never does. So the examples are correctly configured and the documented rule was the defect — which is why this fixes the claim rather than churning three already-reviewed YAMLs. §5 now states the arithmetic, and the three consequences that do not follow from "above the noise floor": replicates buy √n so widening epsilon is usually cheaper than buying the bound down; the `(1−ρ)` factor is the entire payoff of `workload.seed_env`, so §3's seed check gates the budget arithmetic; and an epsilon under the noise floor is not automatically wrong because the bound floors at zero once the winner's margin exceeds it. A near-tie returning `terminal_best` is the instrument working — three indistinguishable configurations is a real result, and certifying one of them would be the failure. vllm-batching.yaml and qdrant-hnsw.yaml each state their own arithmetic and their own null-result condition in `guidance.interpretation`, matching what knative-autoscale.yaml already did. All three still validate statically with no new errors. Refs AI-native-Systems-Research#302 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
Three commits pushed from a field test of this kind against a real target (BLIS /
|
Field test:
|
| substantive calls | billed | |
|---|---|---|
| optimization, clean campaign (mechanism exists) | 1 | $1.68 |
| optimization, first campaign (authors the mechanism) | 1 | $6.57 |
| optimization, entire run incl. 2 defect-forced restarts | 3 | $10.21 |
| reflective (1 of 3 iterations completed) | 10 | $53.09 |
5.2×–31.7× depending on which comparison is fair. All three are reported; none is the "real" one alone.
The mechanism is not per-turn efficiency — measured, reflective's execute-analyze ran at $0.102/turn, cheaper than build's $0.131. What differs is structural:
- optimization pays one substantive call, then measures tokenlessly. Marginal model cost per benchmark run: $0.
- reflective pays a
design+execute_analyzepair per iteration, and those phases can fail expensively: a $9.25 design superseded by a retry, a $13.85execute_analyzethat hiterror_max_turnsat 121 turns. ~49% of its spend produced no usable result. - reflective absorbed 10 API-boundary failures (3 SDK-silence aborts, 6
ConnectionRefused, 1 max-turns). The optimization arm had zero — a compiled epoch makes no API calls, so there is nothing to fail. That asymmetry is architectural, not luck.
Axis 2 — correctness: a tie
Both arms independently authored working ARC (Megiddo & Modha four-list, T1/T2 + B1/B2 ghosts). 154/154 and 153/153 of BLIS's own sim/kv tests pass.
The designed discriminator was a stale guard test asserting mustPanic("arc") — implementing ARC makes that assertion false, so an honest arm must update it while a test-defeating one deletes it. Both updated it honestly. Both also wrote an unrequested scan-resistance test that fails if ARC were a no-op wrapper over LRU.
Structural difference only: optimization isolated the mechanism in a new file (357 lines, 178 insertions total); reflective integrated it in place (848 insertions). The trap did not discriminate. With the same strong model on both sides, the kinds were equal here — worth reporting plainly rather than spinning.
Axis 3 — optimality: optimization found a 52% better configuration
| reflective | optimization | |
|---|---|---|
| valid objective measurements | 4 | 11 (clean screen) / 20 (pooled) |
| factors varied | eviction_policy × tier-presence |
eviction_policy × offload_prompt_only × cpu_bytes × device_class |
| best λ* found | 1.3125 | 2.0 |
| its conclusion | "ARC ≡ LRU" | device_class dominates; ARC unresolved |
Both arms are correct, and the optimization arm found more. Reflective's h_main: REFUTED (ARC ≡ LRU) holds — the optimization data agrees ARC is unresolved (|t| ≈ 0.9). But reflective held offload_prompt_only and cpu_bytes fixed at baseline and never varied device_class at all, so the factor with the largest observed effect was outside its design by construction.
The largest effect is counter-intuitive and has a traced mechanism: sata_ssd beats nvme_gen4 (+0.36 sessions/s, ~28%). cascade() pins each CPU block for its write-through duration, so a slower secondary tier keeps blocks pinned longer, preventing CPU-tier eviction and retaining the hot working set. The slow device acts as an accidental retention policy.
Honest limits: on the clean single-estimator screen (n=11, 4 factors, no replication) no individual factor reaches significance — effect directions are stable but attribution is not established. An earlier |t| = 2.30 for device_class came from pooling rows across two search methods, which inflates n without adding independent information. This screen ranks configurations; confirm is what would certify an effect with a residual-regret bound.
So: optimization wins the deliverable by ~52%, because its factorial design included the factor that mattered — not because it searched better within the same space.
What the run found in this PR's machinery
Seven nous_asks, six fixed in code across the commits on this branch (2140 → passing). Three produced plausible wrong results rather than errors, which is the dangerous class:
apply.kind: config_patchwas schema-documented and applied nothing. Every row of such a campaign ran the baseline while the design matrix, runs, and fitted surface all looked real.validatepassed it 0/0;--smokepassed it too, because the run succeeds. Only the run-timemanipulationpredicate noticed — on row 1 of 18, after a fullbuildstage.design_matrix.jsonnever validated against its own schema. Nine undeclared fields includingkind: "foldover"— an entire design family the schema rejected. Every campaign reachingconfirmemitted a pre-registration record that failed its own schema; the existing test validated a freshly-built skeleton, never the enriched document on disk.- Rule 12 silently skipped 2 of 3
native_testlocator styles, including the bare identifier the Go result parser actually matches on — so it was inert for the style that works.
Plus run_timeout_sec (the hardcoded 600 s ceiling blocked any objective whose evaluation is itself a search), max_parallel (bounded concurrency for confirm replicate blocks only, with screen rows structurally forbidden to opt in), parallel_arms.run_units honoring the max_parallel it previously validated-then-ignored, and --liveness (verify factors matter, and catch a declared level that aborts the target — 3 of 8 candidate factors here were unusable).
The finding I'd most want a reader to take away
My target-side adapter had more defects than Nous did — seven — and they were more expensive. The worst: I edited its output schema three times mid-epoch, so rows measured before each edit carried null for the new keys, and a None reaching float(raw) killed an entire iteration at fit time after ~2 hours of measurement.
Nous already enforces instrument stability for the campaign side: policy.json is content-hashed and _load_or_compile_policy hard-aborts on mismatch. There is no equivalent guard on the adapter — yet the adapter produces every number the pre-registration is about. That asymmetry let me repair the instrument while it ran, three times, and destroy an epoch's comparability with nothing firing.
Guards for this are in progress: an adapter contract hash (fingerprint the response keys+types on the first successful row, hard-abort on drift), output freshness (reject a response byte-identical to the previous row's when levels differ — the defect where a panicking level read as "identical to baseline"), and a declared response.self_check so an author states the invariant their objective must satisfy and Nous enforces it per row. One line —
self_check:
- {metric: backlog_slope, op: "<=", value: 0.060}— would have caught 8 of 12 bad rows at the moment each was measured instead of after the epoch. docs/optimization-campaign-guide.md §7 now carries the pre-flight checklist these lessons produced, written target-agnostically.
Correction to the field-test comment above — Axis 3 (optimality) restated, not withdrawnI re-read the reflective arm's The two factual errors1. "reflective … never varied 2. "optimization wins the deliverable by ~52%" — not a like-for-like ratio.
No shared level, a 2.5× gap between the ranges. A single ratio across that gap The objective both arms maximizeSame quantity, same instrument, both arms: λ* — the highest session-arrival Held identical across arms: BLIS at How the two arms are faring
Each did what its kind is for, and the two outputs are different in kind:
Whether 2.25 reflects better navigation or a friendlier starting neighborhood is Honest limits on the optimization number. Each of the 15 complete screen rows What is unchanged
Field-test caveats worth recordingThe CPU-level mismatch was my authoring choice, not a property of either |
`kind: optimization` could not extend a target. Every stage in orchestrator/optimize was pure Python with zero agent calls, so when the mechanism under study did not exist yet, `verify` ran the target's test command, found the declared native_test identifiers missing, and aborted fail-closed — correctly and uselessly, since nothing in the path was able to author them. The guide already claimed verify "writes mechanism code with native tests"; no code did that. Reflective never had this problem (its SDK agent edits the target as a matter of course), so the gap was specific to this kind, and it mattered: "extend the target, then optimize the extension" is the common shape of a real optimization request. `build` is an opt-in stage, absent from the default order, so campaigns written before it behave identically. It spends exactly ONE agent call to write the mechanism and its native tests, then returns to the tokenless machinery. It deliberately does not iterate to green, read measurements, or judge correctness: `verify` remains the gate, so the stage that writes the code is never the stage that certifies it. Cost goes from ~3 model calls to ~4 only for campaigns that need it. Two validator rules, both of which fire on real authoring mistakes: - rule 11 rejects `build` anywhere but position 1. Behind `verify` it guarantees an abort; behind `screen` it is worse than an abort, because the screen measured the OLD mechanism and the effect table then describes a system the campaign replaced. Both produce schema-valid artifacts, so neither is caught downstream. - rule 12 warns when declared native_test files are absent from the target and no build stage exists to author them — otherwise guaranteed to abort at verify, but only after a real run. Also: the test command no longer runs on a build iteration. It could only report the already-known fact that the identifiers are missing, and `go test -run <pattern>` exits 0 with "no tests to run" when the pattern matches nothing, so a pre-build run looks like a pass at the shell level. Tests inject at the sdk_runner seam per tests/CLAUDE.md — no live calls. Two are integration: BUILD authors a file in the target and CONTINUEs, then VERIFY independently aborts when the tests do not actually report. iteration.py remains 23 insertions / 0 deletions against main; the reflective path is untouched. 1673 passed, 1 skipped. Refs AI-native-Systems-Research#302 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…st found A fitted stationary point is an EXTRAPOLATION. When the surface is mis-specified — curvature the quadratic cannot express, an optimum outside the design hull, a categorical factor dragged into the fit — the solved point can land below a corner the screen already measured. confirm then replicated that inferior point, found its replicates agreed with each other, wrote status CONFIRMED, and the campaign reported an inferior configuration as its optimum. "The prediction reproduced" and "this is the best configuration found" are different claims, and only the second is what a reader of the report wants. _best_observed already existed but was used only to PICK the confirm point when no fit existed; it never CHECKED a fitted one. confirmation.json now carries best_observed, confirmed_is_best_observed, and (when they disagree) regression_vs_best_observed with the absolute and percentage gap, plus a warning naming the observed corner as the campaign's real answer. Respects response.primary.direction, so lower-is-better objectives are judged correctly rather than inverted. 1676 passed, 1 skipped. Refs AI-native-Systems-Research#302 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Found by running it: a build stage whose cwd was its own git worktree read and then EDITED the canonical checkout of the same project instead. Four tracked files in the wrong tree. Two supposedly independent campaign arms would have overwritten each other's mechanism, and every measurement after that would describe a system neither arm authored — silently, with no error anywhere, since both trees build and both test suites pass. Passing repo_path as `cwd` is not sufficient. When the campaign description mentions file paths and the agent already knows a checkout of that project, it resolves the paths against the checkout rather than against cwd. The prompt now states the absolute working root up front, says the path may be a worktree with a parallel experiment in a sibling checkout, and requires every path to be re-derived against that root. Prompt text alone is not a guarantee, so check_build_touched_repo() reports when a git target shows no local modifications after a "successful" build -- the signature of having edited somewhere else. Advisory, not fatal: verify remains the authority on whether the mechanism is present, a legitimate no-op build should not abort a campaign, and non-git targets are exempt. The warning also lands in build_warning.txt so it survives in the artifacts. 1680 passed, 1 skipped. Refs AI-native-Systems-Research#302 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…fy aborts Three diagnostics losses, all of which mattered most exactly when something had gone wrong: 1. run_test_command returned dict[str, bool] and discarded the output. The gate needs the booleans; a human fixing the failure needs the text — the failed assertion's expected-vs-actual, a panic and its stack, a compile error's file and line, a timeout's partial output naming the test that hung. Since a verify abort ends the campaign, that text was the only record of why, and re-running by hand may not reproduce a timeout or an ordering-dependent failure. Now written verbatim to runs/iter-N/test_output.log, including on timeout and exec failure. 2. A failed config run kept only the last 200 chars of stderr, in an exception message. The cause is usually at the HEAD of stderr (a usage error naming the rejected flag), so the tail dropped the useful part. Full stdout/stderr now land in runs/iter-N/failed_runs/failed_run_<i>.log with the row index and its levels; the exception points at the file. Same for output that parses to no JSON, which is the failure that NaN-poisons a fit. 3. The verify abort listed relation IDs without saying whether each test FAILED or was NEVER EXECUTED. Those need opposite fixes, and after a build stage the agent call is already spent, so a wrong diagnosis costs a whole second build. The message now splits them and names the fix for each, including the trap that makes "never executed" silent: go test -run with a pattern matching nothing exits 0. Logging is opt-in (log_path / log_dir default to None) and can never be the reason a campaign fails — every write is guarded and degrades to a warning. Guide: metamorphic relations now carry an explicit warning to do the algebra first and include a worked numeric example, with the real case from this campaign corpus where an author declared step <= linear <= exponential when the truth is step <= exponential <= linear (the exponential is concave in f). A metamorphic relation pointing the wrong way fails correct code or passes broken code, with the authority of a declared correctness relation. 1686 passed, 1 skipped. Refs AI-native-Systems-Research#302 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… it (P8 audit) Names the current bound methods and the full union bound (Pr(certified and wrong) <= delta_s + delta_t [+ delta_constraint + delta_feasibility]), matching data-model.md and the guide. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… P8 audit); take nousko's expert audit wholesale, dropping the P7 (e') hunk (P7 review H5, M3) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ice (two_peaks optimum (14,3) was off the 9-point grid); one shared measured-row counter; best_so_far strikes rejected; prefix GP uses the run's seed; peak RSS in meta; previous run's head-to-heads and baseline values in previous.json; tests (P7 review H3, M4, M5, LOWs, WIP) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…aces; peak RSS and wall time in every generated header (P7 review M4, LOW) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…, the H3 surface audit, the one measured-row counter, the strike rule and the Wilson scope (P7 review H3, M5, LOWs, WIP) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…chmark 47bc42e50f66cb60 (lattice truth); 50 certified of 336, 0 false (P7 review, WIP) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s ff14a3afe6e0fdb1 / benchmark 47bc42e50f66cb60 (lattice truth); 0 false certificates in every arm (P7 review, WIP) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…st-selection block through an undeclared confirm count; the charged budget withholds when it cannot see the whole spend; plan-next validates at the next epoch's floor; confirm.replicates is optional H1: rule 31 counted 3 confirm replicates when undeclared, the compile uses the A5 floor (6), so validate/inertness could re-measure on confirm's draws; rule 31 now counts the compiled default and _assign_workload_seeds refuses to wrap a post-selection block. H2: campaign_delta_exceeded withholds when the cumulative is a lower bound or an earlier epoch's declared total is unreadable. M1: plan-next validates proposals (and accept re-validates) at epoch e+1, so a proposal below that epoch's raised floor is refused. M2: design.confirm.replicates is no longer schema-required (the docs and validator advise omitting it). Each new test fails on 05884f0 and passes here. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…us recommends the measured centre: fails_at_one_level CPU=24; infeasible_corner_interior truth 14.4 -> 14.7 at BS=20); test that every committed Nous recommendation lies in the shared domain; profile.py makes the CPU profile reproducible (P7 review H3, M4) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ext epoch's floor instead of proposing only moves that cannot certify af81f6d made plan-next refuse a proposal below epoch e+1's raised floor; under a charged campaign_delta with a declared replicate count that refused every move. Proposals now raise the count to that floor in place and record the change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…onfigurations Nous's design code runs (stage_runner._build_design + matrix.expand), never a per-factor centre, which had over-stated it (infeasible_corner_interior truth 14.7 -> 14.4, 6 spurious false certificates); tests (a) truth in domain on every surface, (b) every committed recommendation in domain, (c) infeasible_corner_interior 14.4 (P7 review H3) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… the design's centre configurations, from matrix.expand), why (certificates are claims over the declared candidate space), and the per-factor-centre false start (P7 review H3, WIP) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…chmark 659d385ed99dfafa; 50 certified of 336, 0 false (P7 review, WIP) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…59d385ed99dfafa (nousko 86cbec9); every prose number from that run's JSON with its source tagged; H1 many-knob 37; H2 anytime wording from the data; H4 drift certificates within seed noise; M1 two/three-way delta split; M2 no detectable SO allocate change; BO-ablation and post-hoc t-interval sentences; profile JSON; 0 false certificates in every arm (P7 review) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…native-Systems-Research#321 benchmark completeness, AI-native-Systems-Research#320 D4) Anytime curves, seeded CIs and Wilson intervals with a pre-declared primary, multi-objective HV regret vs RS/LHS/ParEGO, D4 ablations, CPU profile; truth and every baseline over the declared lattice plus Nous's own centre configurations. 0 false certificates in every arm. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… P8 audit) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e5d / bench 659d385ed99dfafa) Q6 cited the superseded nk-p7-bench@0c57b40 run (digest 44d94bc5ee2a41b2), before 01eded8 and the lattice-truth fix. Every number is replaced from the merged benchmark's JSON; the primaries, post-hoc splits, per-arm certificate table, run totals and CPU medians were re-derived independently. - non-quadratic now inconclusive, -0.020 [-0.063, +0.037]; +0.046 without two_peaks - shipped -0.057 rests on stochastic_constraint_boundary (post-hoc, t interval caveat) - many-knob a small GP-BO advantage; ParEGO ahead on fronts; GP-BO ahead at 40% of N - conservatism cost in runs and certificates; CPU profile - new paragraph on the ground-truth correction (lattice/domain rule, two_peaks 0.364 -> 1.263, the per-factor-centre false start and its 6 spurious false certificates, the domain tests) - header, Q1, Q4, section 12 and section 13 updated; "many-knob 47 -> 0" was wrong (the previous run had 37) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…survivor (AI-native-Systems-Research#322) The post-batch survivor check ran while SIGKILLed multi-GB processes were tearing down: kill(pid,0) succeeded, getpgid raised ESRCH, killpg(pgid,0) answered, so every dying pid was 'escaped' and a 6.5 h screen failed. The check now waits a bounded settle period (SETTLE_SECONDS) for the group and tracked descendants to exit, treats ESRCH from getpgid as gone, and ignores a group whose remaining members are zombies. A genuinely escaped process is still caught and killed. The guide documents the settle and exactly what nous resume reuses after a gate halt (complete/infeasible/rejected reused, failed incl. timeouts re-run). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…me (AI-native-Systems-Research#323) Each run/resume is one entry in wall_clock.json; the budget is checked against the sum of their spans (time between invocations not charged), so a resume continues the budget instead of getting a fresh one. progress.json reports wall_clock_hours, invocations and the declared budget (it read null after a halt); report.json records provenance.wall_clock (hours, invocations, resumes). Reanalysis ignores the ledger (a fact about the original run). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… rows per level, and gives a hard bound (AI-native-Systems-Research#325) A 262-row screen read 93 s at 1 row and took 6.5 h: row cost depended on the levels, and early rows were the cheap ones. progress.json now records eta_confidence (low until enough fresh rows, or while durations vary widely), prices pending rows at their own level of the factor that explains the duration spread (unseen levels at the slowest seen), and adds eta_upper_bound_seconds = pending x run_timeout_sec / slots. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…irm rounds and compare the front with it (AI-native-Systems-Research#324) Pareto campaigns never seated the baseline, so a deployment-tuning campaign had to measure the deployed configuration outside nous. Confirm now seats it in every round (outside shortlist_size, CRN-paired), as single-objective does; the preview and the next-round cost count the seat. confirmation.json and report.json gain front_vs_baseline: per admissible finalist, the gain over the baseline on each objective with two-sided intervals simultaneous over every (finalist, objective) at delta_terminal, and a verdict (dominates_baseline / dominated_by_baseline / trade_off / undecided). A separate claim: front statuses and certificates are unchanged. Measured (8 Pareto surfaces x 8 seeds): false certificates 0 with and without the seat (30 vs 32 certified); interval coverage 936/938 against a nominal >= 0.95. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…so it is a bound (AI-native-Systems-Research#325) A manipulation transient can run a row twice (execute_design max_retries=1), so pending x run_timeout_sec could be exceeded. ROW_ATTEMPTS_MAX is now the single constant both the runner default and the bound use. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
-AI-native-Systems-Research#325 fixes) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds
kind: optimization: a campaign kind that finds the best configuration of a system's knobs, or the trade-off front across several objectives, by estimating, not searching. It works on one repo or on a harness that composes several. Each epoch is a pre-registered, content-hashed experimental policy (policy.json) that pure Python interprets with zero model calls. The model is used only for the opt-inplanandbuildstages, which design and author a mechanism that does not exist yet. A campaign without them makes no LLM calls at all.Refs #120. Closes #264, #281, #303, #304, #315, #316, #317, #318, #319, #320, #321, #322, #323, #324, #325. Every issue found or filed during the nousko work is resolved here; the per-issue evidence is in §0 below. Specs:
docs/superpowers/specs/2026-08-16-compiled-policy-design.md,2026-10-15-pareto-front-design.md,2026-10-16-mixed-level-factors-design.md,2026-10-17-model-adequacy-design.mdand2026-10-17-certification-soundness.md. Start here:docs/optimization-authoring-quickstart.md, thendocs/optimization-campaign-guide.md. For an expert reviewer:docs/optimization-expert-audit.mdanswers eleven expert questions (error control, robustness, misspecification, design, winner's curse, benchmarks, reproducibility, measurement validity, failures, AI-native risks, allocation), each with evidence, the tests that pin it, what the user is told, and the remaining limits.0. Issues closed by this PR
ProcessErrorexit 143 (gateway stream drop) is transient; every other non-zero exit stays permanent. The plan/build calls share the classifier. PR #316 is merged in (784eb0c).TestSDKTransientClassification,test_a_permanent_error_escapes_unretrieddispatch()). (2) Each SDK turn runs in its own loop, which reaps its children and their process groups. (3) A per-phase wall-clock ceiling turns a hang into a countedwall_clockretry.TestLiveThresholdRegression303,TestAbortedTurnsReapChildren(pre-fix: hung),test_a_tool_process_the_child_started_is_reaped_with_it,TestPhaseWallClockResolutionnous resume --silence-threshold.TestWatchdogIgnoresBundleOverride,test_resume_silence_threshold_overrides_every_phase--agent sdkthe reflective report goes through the SDK: final answer only, run in work_dir, at most 10 turns, best-effort. A Claude model on an OpenAI client with noOPENAI_BASE_URLandANTHROPIC_BASE_URLset fails loudly with the fix. Limit: gate summaries keep LLMDispatcher (they get the loud error).TestReportPhaseRouting304,TestReportPhaseRoutingReviewFixesdesign.confirm.allocateround. D2:nous plan-next. D3: the optional Optuna proposer (nous[bo]). D4: oracles and benchmark arms. D5: audit Q11.test_optimize_allocate*.py,test_optimize_planner.py,test_optimize_bo_proposer.pynous resumereuses after a halt.test_optimize_survivor_race.pymax_wall_clock_hoursis campaign-wide: each run/resume is a ledger entry and the budget counts their sum;progress.jsonandreport.jsonprovenance.wall_clockshow it.test_a_resume_continues_the_wall_clock_budget_instead_of_resetting_it(throughrun_campaign),test_the_wall_clock_budget_continues_across_a_resumeknown_valid_baselineevery round (CRN-paired) and reportsfront_vs_baseline: per front point, the gain on each objective with simultaneous intervals and a verdict. A separate claim; front certificates unchanged (0 false with and without; coverage 936/938).test_optimize_mo_baseline.pyeta_confidence, prices pending rows at their own level of the cost-driving factor (unseen levels at the slowest), and giveseta_upper_bound_seconds(pending x max attempts x run_timeout_sec / slots).test_an_early_eta_is_marked_low_confidence_and_bounded,test_the_eta_prices_pending_rows_at_their_own_levelreuse_screenacross choice slices rejected (bias). See the issue comment for item-by-item evidence.1. Objectives → mechanisms → evidence
orchestrator/optimize/synthetic.py), each named for a real past failure:saddle,interaction_only,choice_x_numeric,bowl_out_of_hull, …candidate_spacein recommendation.json and noted in report.md. The--livenesspreview names the design, compares it with the smallest known, and prices a resolution-IV screen.response.objectives(≥ 2, eachmaximize/minimize, ε per objective). Every objective is fitted on the same design; each fitting stage writes a predicted front. Confirm seats the measured and predicted extremes, the preferred point and spread points, then a registeredconfirm → confirmgap-fill branch (Bonferroni over rounds) fills the widest resolvable gap. The measured front carries a status per point (confidently_nondominated/possibly_dominated), claimed simultaneously at δ_t (α = δ/(N(N−1)K)).preference(knee / weights / lexicographic) picks one point from the whole measured front; it is certified only if it is confidently nondominated and its scalarised terminal regret is ≤ ε. Hypervolume is exact for K = 2 and K = 3.pareto_convex_min(a minimised cost),pareto_concave,pareto_choice_split,pareto_noisy(noise ≈ ε) andpareto_three_noisy: 0 confident points off the true front, 0 false certificates, with stated hypervolume floors. A 20-mutation sweep (sign, weights, Bonferroni levels, refill, report) now kills 19; the survivor is an equivalent mutant. Single-objective campaigns: 288 runs byte-identical before and after.policy.out_of_hull: confirmconfirms the best configuration inside the ranges.report.jsonrecordsboundary_optimum(the axes, pluscertificate_scope: declared_ranges), and the next-epoch advice gives runnable levels: on the grid, inside the domain, and deferring to measured marginal means when they peak inside the range. It says to pin a knob at its edge when the edge is a hard limit.end_epochkeeps the strict exception.bowl_out_of_hull(peak A=30, declared [2, 16]) now confirms A=16 on every seed, instead of ending with one unreplicated row. Same on the real gopack campaign.response.constraintsmark rows infeasible. Constraint metrics are fitted on the same design, and only confidently infeasible candidates (one-sided bound at δ/C) leave the ranking. Infeasible rows still inform the primary fit. Feasibility is judged per configuration, not per row.infeasible_corner_interior/stochastic_constraint_boundarysurfaces. On the constrained bowl (16 seeds): aborts 4→0, certified 0→3. A real campaign seated 0 infeasible finalists after the fix vs 2 of 3 before.levels_onlylattices. They are tested against the model's predictions with a joint predictive F. Each adequacy test is a per-test veto (a Holm family over them is recorded as information). Adequacy is sticky for the epoch, and an UNKNOWN verdict is never promoted to adequate. Screen → refine routes whenever a refinable factor exists, and a screen-only global claim with two or more refinable factors withholdsmodel_class_unchecked. D-, G- and I-efficiency are reported.levels_onlynumeric knobs run only at their declared levels. Centre points cycle every categorical level, so curvature is tested under each. A level lost to failed runs is refitted around and never recommended.2026-10-16-mixed-level-factors-design.md.scale: logcodes geometric knobs (sizes, batches, intervals) in log space; rule 25 warns when levels look geometric. Aliases are collapsed per alias class. Lack of fit is tested against pure error. Curvature detected by the centre points routes the screen to refine even when no main effect survives, and no certificate is issued on a rejected model.additive12/12→1/12,choice_x_numeric6/6→0/6. Ongeometric_knob, linear coding certified an answer 26% off on 8 of 8 seeds; now it never certifies it, andscale: logfinds the exact optimum. Certificates that were wrong (true gap >2%) across all 15 surfaces × 8 seeds: 16 → 0. Cost: about α of correctly modelled campaigns lose a certificate.docs/optimization-expert-audit.md.shortlist_refilled), and a certificate needs a measured feasible rival (no_feasible_rival).sla, seeds 1–8: from one feasible finalist to at least two; the answer moves from 1.2 to 0.4 below the constrained optimum, with 0 false certificates. Measured on 5 constraint surfaces: the alternative rule (caveat instead of veto) certified truly inadmissible configurations, so it was rejected and pinned by a test.report.json:certified → terminal_best → model → measured → baseline → none. The rung is recorded asrecommendation.basisand never hidden.report.md, gate summaries, and the deployment verdict inmeta_findings.jsonare rendered fromreport.jsonwith zero tokens, so the three cannot disagree.response.held_outmetrics are split off the response structurally, never fitted, ranked or certified on, and reported per recommended configuration and finalist.locked_parametersthroughlocked_observables(an unenforceable lock is an error unless acknowledged); drift of the policy hash, the mechanism patch and the adapter contract; design fidelity; held-out leakage; build oracles. A failed gate halts the campaign; a pre-epoch stage is never skipped.tests/test_optimize_campaign_halts_on_gates.pydrives the realrun_campaignagainst a real subprocess target. 82 registered invariants (docs/optimization-invariants.md)..git. Gate records are HMAC-sealed outside the repo. Every target command runs in its own session and is reaped (setsid descendants included). Authored tests must discriminate the mechanism (a revert mutant and a stub-body mutant). The import probe refuses an interpreter that imports from the agent's repo, with exec-exact resolution. Preconditions are checked before any token is spent..pyc, symlink copy-back,env PATH,pytest -s) has a test that fails without the fix. An honest end-to-end build certifies the true optimum. The guide's security claims are pinned by a test that catches 9/9 drifts.report.jsonrecords provenance: Nous version, source digest, every input hash, analysis constants, waivers, the served model ID, and an external anchor of the pre-registration, which is create-only and refuses re-anchoring after data. It also records a cross-epoch error budget. The run order is concealed by a recorded salt.nous reanalyzere-derives every fitted artifact from the recorded rows and diffs them. A matched-budget benchmark compares Nous with RS, LHS, a fitted GP-BO and ParEGO.plan-nextand a second epoch. Benchmark (docs/benchmarks/optimization-benchmark.md, final code, metric pre-declared): 0 false certificates in every arm. Against GP-BO: shipped family −0.057 [−0.066, −0.039] (resting on one surface, labelled post hoc); non-quadratic family inconclusive; GP-BO slightly ahead on many-knob. ParEGO beats Nous on multi-objective fronts. The stricter certification costs runs and certificates against the previous run; see the doc.planandbuildrun under a bounded retry budget with a live silence watchdog, and every attempt's cost is logged.nous create-campaign --kind optimization(with--adapter-stub), a short quickstart, and validation that reports every problem in one pass, ending in a singlenext:line.--smokeruns the real contract and understands tests the build will author.--livenessmeasures the noise floor and prints a design preview: runs per stage, ETA, and ε vs the screen's MDE.tests/test_optimization_authoring.pyruns the real CLI against a temp target. The preview's run counts are cross-checked against what a campaign actually runs.2. Field evidence (real targets, real
nous run)planstage)plan+builda newaccess_update_intervalsetting (sampled LRU/LFU access-metadata writes), then tune, then two narrowing epochsreplacedirectives; GOGC/GOMAXPROCS as env vars)BATCH=1581, GOGC=1600, PROCS=7= 134.1 at 46.0 MiB (−8% throughput for −28% memory). With about 40 measured configurations and 4 replicates, no point reachedconfidently_nondominatedunder the simultaneous correction; the report says so and says how to resolve it. 0 tokens3. What this update changed
This round: closing every nousko-discovered issue (#264 #281 #303 #304 #315 #316 #317 #318 #319 #320 #321)
Eight packages, each with its own
/pr-review-toolkit:review-prgate, one fix round, and an expert self-audit on questions chosen per stage. A final 11-question audit and a whole-PR review followed. Every review found real defects; each was fixed with a behavioral test that fails on the pre-fix commit:nous resume --silence-threshold, and Iteration #3 wrongly routed to OpenAI's endpoint #304 report routing.delta_constraint, opt-in post-selection validation (fresh draws even with pinned seeds), a chargedcampaign_deltathat binds, TOST inertness, and baseline-seat accounting (which fixed amax_runsoverspend).delta_feasibility, a CRN probe, and oracle 2(c) as a TOST.model_class_unchecked. This closed a measured screen-only false-certificate hole on a masked saddle. Also the outer validation ring, opt-in screen validation, per-test vetoes, G/I efficiency, and rejection of partial reuse across choice slices.nous plan-next(a true lower-bound utility, a sealed decision, an accept that refuses edits or stale inputs), the Optuna proposer, and the opt-in allocate round (digest-anchored seats).Earlier round: expert-audit hardening
Each package went through
/pr-review-toolkit:review-prpasses and expert self-audits (docs/optimization-expert-audit.md) before merging. Every finding that touched a core claim was fixed with a test that fails without the fix.levels_only, centre cycling, lost-level handling, and epoch-scoped reports.nous reanalyze, and the benchmark.Latest round: multi-objective fronts, multi-repo field campaigns, Claude-first authoring
Pareto front (spec
docs/superpowers/specs/2026-10-15-pareto-front-design.md, with its Decisions section). New modulesorchestrator/optimize/pareto.py(pure Python, no numpy, enforced by an import-scan test) andpareto_stage.py. Schema, validator, liveness and design preview work per objective. Three independent reviews (correctness, silent failures, test quality) were run, and every finding was fixed with a regression test. Among them:epsilon_unknown.Defects found by running real multi-repo campaigns (F25–F39):
nous docsnow serves them. Progress could not be looked up from another shell; a work-dir registry fixes that. Validate's next step now saysnous resumewhen the campaign already has a work_dir.campaign.yamlwent stale across epochs; each epoch now writes its own copy. A policy schema failure was not a gate; now it is. A gate halt left the next resume appending duplicate rows; rows are now reused. A stalehalt.jsonread HALTED on a completed campaign; it is now archived tohalt_history.jsonl.Claude-first authoring.
.claude-plugin/marketplace.json→plugin/nous, andclaude plugin validatepasses). A newnous-optimizeskill covers the full loop: understand the target, write the adapter, scaffold, validate until clean, run in the background, monitor, read the basis, and iterate.nous docs quickstart|guide|targets|invariantsserves the reference docs from any install.Any number of knobs. The screen used textbook generators for 5–8 knobs and fell back to the 2^k full factorial everywhere else. The candidate search also enumerated 9^k points, so 7 knobs did not finish. Three changes:
design.generators_forbuilds a Res III, IV or V fraction for any k, guaranteed by construction and checked off the design matrix.Validator rule 8 now compares the exact run count with
max_runsand names which lower resolutions would fit. The synthetic oracle gainedSurface.active, so many-knob truth stays exact without enumerating 9^k points.Earlier on this branch
Main is merged in (conflicts resolved in
campaign.schema.yaml,pyproject.toml,README.md).Issues
test_command. A killed command is now named as a timeout (with the partial output saved), separately from unmatched output.optimization.test_timeout_secis added. Declared-vs-observed identifiers are diffed. Rule 23 rejects shell syntax in the argv-style commands.report.mdare rendered with zero tokens; nothing routes throughOPENAI_*.planandbuildnow retry transient failures and overload/context-exhaustion results underoptimization.agent_retries(default 2) and log every attempt. A live watchdog defaults tomax(600, test_timeout_sec + 120)forbuild. The exit-143 classifier itself is fix(sdk): retry ProcessError exit-143 (gateway stream-drop) instead of failing permanently #316 against main, which merges cleanly on top.Defects found by running real campaigns. Most of these were invisible to the existing test suite. Each fix has a behavioral regression test.
nous run(the most serious finding). A failed verify recorded one failed iteration and advanced to screen; a rejected plan advanced to a planless build. Fixed withOptimizationGateFailedplus halting on any pre-epoch failure.currency: 'ops_per_s'was "too short". A rejected plan is now repaired once with the rejection as feedback, and then stops.scale: logis added.20.0;NOUS_RUN_DIRwas created underfailed_runs/or inside the target's tree; the target'sgit statuswas polluted by.nous/; the deploy verdict contradicted the report; held-out values were never surfaced; warning noise; messages pointed atnous run --resume, which does not exist.Authoring. Top-level help names both kinds. Scaffold, quickstart, design preview, and the one-pass-then-
next:validation flow are added. Stale claims in the schema and docs about model calls that no longer happen are corrected.4. Honest limits
One build call.
buildis one call, not a build-until-green loop. A sound-but-slow mechanism can be revised only by a new epoch.planmitigates this by pricing alternatives first.Deterministic targets with CRN. When paired differences have exactly zero variance, the terminal bound is reported as not estimable rather than as an exact certificate. This is deliberate.
Wall-clock objectives on a shared machine are noisy (measured CV 0.5%→12% under co-scheduled load). The quickstart advises CPU time for CPU-bound in-process targets; Nous measures and reports the noise and the MDE but cannot remove it.
Certification costs power and runs. Every conservatism added by optimization: deferred items from the #302 expert audit (tracking) #321 is a veto: the replicate floor, the skewness pretest (a deliberate ~26% Gaussian withhold, measured necessary for validity), the Brown–Forsythe check, per-test adequacy vetoes, and
model_class_unchecked. Measured against the previous benchmark run:model_bound_above_epsilon).See
docs/benchmarks/optimization-benchmark.md.The certificate is conditional on the response class and the noise model. The adequacy checks have local power (validation points at ±0.5 and ±0.8). A narrow peak between points, or a misfit below the noise, can still pass. Under strong heteroscedasticity (h ≥ 8) the model half's conditional miss is 0.317, against 0 false certificates end to end.
semantics: meanis approximate under skew (about 2× nominal at lognormal σ = 0.5); usechance:or declaresupport:for an exact bound.Nous is not the best optimizer everywhere. GP-BO leads early on the anytime curve (at 40% of N) and on several non-quadratic and constrained surfaces. ParEGO beats Nous's multi-objective fronts on hypervolume. Nous's strength is a sound, pre-registered certificate and a zero-token epoch, not raw regret.
Allocation and the BO proposer are opt-in and modest. The allocate round improves fronts slightly but costs certificates in single-objective campaigns (its extra look is charged whether it fires or not). The Optuna proposer shows no measurable benefit in the benchmark.
Build gates assume a same-user agent. The seal key is reachable by same-user code, in-process monkeypatching by in-scope mechanism code is invisible on disk, and under
--auto-approvenobody reviews the mechanism. These limits are stated in the guide.guidance.interpretationis not read by any stage; the schema says so.Simultaneous front statuses are conservative. They are claimed at α = δ/(N(N−1)K) over every measured configuration. On gopack-front (about 40 configurations, 4 replicates) no point reached
confidently_nondominated, so the hypervolume is reported as undefined. That is honest, and report.md says to add replicates or widen ε. Restricting the family to confirm-measured points would be more powerful and is a candidate follow-up.Above 50,000 candidates the search is not exhaustive. The recommendation, the predicted front and the alias-consequence check are only as strong as the seeded search. On planted optima it matched exhaustive enumeration in at least 97% of 180 random quadratics, and hit the optimum exactly at 12, 16 and 24 knobs. Every such campaign records
candidate_spaceand says so in report.md.Resolution V above 30 knobs. The construction can use one more doubling than the smallest known regular design up to 64 knobs (e.g. 2048 runs for 31, against a known 1024), and two at 100. The preview states the gap.
Resuming across this upgrade. Combinations outside the textbook table (e.g. resolution IV at 9 knobs) and composite designs with ≥ 5 factors now compile smaller designs. A campaign resumed mid-epoch across the upgrade builds its next stage with the new design. The pre-registered rule is the same; the run count is smaller.
The knee depends on normalisation. Objectives are scaled to [0, 1] by their measured ranges, so a narrow absolute range (rendersvc's RSS, about 5 MiB) counts as much as a wide one. Use
preference: {method: weights}when the units matter.5. How to review
docs/optimization-authoring-quickstart.md(5 min). Then runnous create-campaign --kind optimization --to /tmp/c.yamlandnous validate campaign /tmp/c.yamlto see the authoring loop.orchestrator/optimize/policy.py(the compiled state machine),orchestrator/optimize/certificate.py(the bounds), andorchestrator/optimize/pareto.pyplus the Pareto spec's Decisions (the front).orchestrator/optimize/stage_runner.py: start atrun_stage, then_finish_confirmand_run_report. It is still the module most worth a human read.docs/optimization-invariants.mdalongside the oracle tests (tests/test_optimize_harness.py,tests/test_optimize_contract_chain.py).docs/optimization-expert-audit.mdwith the two 2026-10-17 specs.Full suite: 4694 passed, 11 skipped on Python 3.13 locally and on 3.11 and 3.12 in CI,
pytest -ra --strict-markers, with zero live LLM calls (enforced bytests/conftest.py).🤖 Generated with Claude Code