kb: address e2e records by release version, apply what is recalled, and report both

[yueliu14]

Three halves of one problem: a warm start that silently finds nothing, one that finds
something and silently fails to apply it, and a report that cannot tell you either happened.

1. Addressing — kb/identity.py

e2e_identity keyed framework_version verbatim. Serving stacks are installed from git in these images, so sglang.__version__ carries a PEP 440 dev/local suffix — 0.5.15.post1.dev20260723+g6c9fd0adc5 — that changes on every rebuild of the same release. A long opaque string is also one that gets hand-transcribed with a segment missing.

Either way the run opens a fresh set of pages and every earlier result becomes unreachable. Not degraded — invisible: the e2e store is an exact lookup with no search, so a miss and a never-recorded page are the same plain 404. And framework_version sits in the ladder's base, so no rung drops it:

...:sglang:0.5.15.post1.dev20260723+g6c9fd0adc5:mxfp4:tp_8:isl_8192:osl_1024:conc_64
...:sglang:0.5.15.post1.dev20260723+g6c9fd0adc5:mxfp4:tp_8
...:sglang:0.5.15.post1.dev20260723+g6c9fd0adc5:mxfp4

One changed character misses all three at once.

_release_version cuts the address to <major>.<minor>.<patch>:

0.5.15.post1.dev20260723+g6c9fd0adc5  ->  sglang:0.5.15
0.5.15.post1.dev20260723              ->  sglang:0.5.15
v0.5.15                               ->  sglang:0.5.15
0.26.0.dev0+g1234abc                  ->  vllm:0.26.0

Three components, not two. 0.5.15 and 0.5.17 are different releases with different kernels; cutting to 0.5 would merge results that genuinely do not transfer. The exact build string still travels in the record's value, so only the address is coarse — the same bargain as kernel_identity's ROCm cut, one component wider.

Reader and writer share this single choke point, so they cannot drift apart.

2. Applying — e2e_workflow.js, roles/config_tuner.md, kb/attest.py

Finding the record is half of it. A stored e2e record holds a whole launch configuration, not a delta — the only form that can be replayed on a box whose baseline nobody wrote down. Replaying it therefore means combining it with the baseline this run was handed, and that combination was a string concatenation performed by an agent (roles/config_tuner.md: EXTRA_SERVER_ARGS="<current flags + this flag>").

adapters/sglang.sh expands $EXTRA_SERVER_ARGS straight into argparse and $EXTRA_ENV straight into env, and both resolve a repeated key as last-wins. So when the baseline and the record each pin --context-length, which value applies is decided by the order the agent happened to write them in. Lose that coin flip and the server runs the baseline configuration twice: ~0% delta, nothing wrong in any log — the flag was honoured, just not with the recorded value — and the record is filed as rejected. A record that wins gets recorded as a loss, which is the one outcome the warm start exists to prevent.

Two more failures sit behind the same seam:

• A flag deleted upstream makes argparse exit before the model loads. One shot spent, measured=0, not_reproduced filed, no repair.
• not_reproduced meant both "the flag is gone upstream" (the record is stale) and "it collides with a baseline this run did not choose" (the record is fine). kb/attest.py:retire_hint retires on two of them, so a good record could be retired for being read on the wrong machines.

This is the default path, not an edge case, wherever a caller seeds the run with a full flag string it produced elsewhere and disables the exploratory sweep. The warm-start replay itself is not gated by CONFIG_TUNE_ENABLED; that flag gates only the exploratory ConfigSweep, so there is then no later phase that could correct any of it.

Merge key by key, in the orchestrator, and report what it did. mergeFlags / mergeEnv combine the recovered configuration with the run's baseline by flag name / env key. The recorded value wins every collision in place, so the baseline's ordering survives and the output names each key exactly once — last-wins never gets a vote. This generalizes the guard the launcher already applies by hand to one flag (adapters/sglang.sh: don't add --watchdog-timeout if the caller set one) and moves it to where the string is built instead of where it is used. Handles --flag value, --flag=value, bare flags, and multi-valued flags (kept whole, so a flag is overridden as a unit or not at all); a small explicit allow-list keeps genuinely repeatable flags (--lora-path) appended rather than deduped. What was overridden is reported, not just resolved — MERGE_OVERRIDES reaches the tuner, the run log, measured_on_this_box.md, the attestation note, and §2c below. A record the baseline already carries in full is now skipped instead of benched against itself.

One repair attempt, and only when nothing came back. When the merged configuration produces no measurement at all, the launch failure goes back to config_tuner, which classifies each recorded knob as upstream_gone / conflicts_with_baseline / env_unsupported, drops or adapts the minimum needed, and benches once more. Exactly once — a repair loop against a configuration that cannot run here spends server launches to learn nothing — and through the same accept gate, with no allowance made for having been repaired. If knobs had to be dropped, the verdict records applied_partial and which ones. SWEEP_SCHEMA gains an optional repair object, out of required, so every ordinary sweep keeps validating unchanged.

kb/attest.py learns inapplicable. A fourth outcome for "could not be applied to THIS box's baseline". It counts toward recalls — the invariant that recalls is the sum of its buckets is what makes a ledger checkable, and hiding an attempt would make a record that keeps failing to fit look untouched — but it is subtracted from the denominator retire_hint judges on, because it is a verdict on the pairing and not on the record. The bucket list is now a single BUCKETS constant, since four call sites have to agree on it and three fail silently when they don't (a key missing from attestations_of reads as 0 forever; one missing from carry_attestations's emptiness test drops a whole ledger on the next rewrite). Surfaced through e2e_store.py's _view() and track-record line so a reader sees "N did not fit that box's baseline (no verdict on the record)" rather than N failures.

Also fixed here: the sweep result was read as trial.notes while the role's return schema spells it note, so the per-trial text never reached the "never took effect" check at all.

3. Reporting — e2e_workflow.js, roles/system_architect.md

KB_RECALL collects, on both planes, what was asked and how it fared:

• the exact canonical ids tried, the read reason, match tier, plane, candidate count — recorded before anything is benched, so a run that dies mid-warm-start still reports the ask;
• each recalled config re-measured against this box's own baseline (stored claim / measured here / Δ / parity / outcome), plus what the merge and the repair did to it (overrides, applied_partial, dropped_flags) — a Δ measured from a partly-applied configuration is not a measurement of the record, and the row now says so;
• each kernel lane's warm_start block — adopted vs incremental speedup, rounds committed, no_rounds_after_adopt. This previously died inside the lane, so an e2e report could describe a kernel as authored from scratch when the lane had in fact adopted a stored patch.

It is threaded into the Report role's inputs and emitted as kb_recall in the workflow return, independently of kb_warm_start — that object is null whenever KB dims were never established, but the kernel lanes can still have recalled by then, and those are exactly the runs where knowing what was recalled matters most.

system_architect.md gains a mandatory 2c. Knowledge-base recall section, written even when nothing came back. On an exact-lookup store the addresses tried are the finding: without them a reader cannot tell "no prior art" from "prior art one segment away". The section refuses to collapse the outcomes into each other — rejected (ran and lost), not_reproduced (never took effect), inapplicable (never fit this box) say different things about the record, and only the middle one is evidence it is stale.

Compatibility

• Records written before inapplicable existed read it as 0 and produce exactly the same retire hint as before. No migration.
• repair is optional in the sweep schema; existing sweeps are unaffected.
• interface/run_e2e.py, adapters/sglang.sh and bench_e2e.sh are unchanged.

Verification

• identity normalization over dev / local / v-prefixed spellings of one release, and across sglang + vllm
• new e2e_workflow/scripts/test_warm_start_merge.js — 26 assertions, with the merge functions extracted from the shipped source rather than reimplemented, so it cannot pass while e2e_workflow.js drifts. Covers the collision case, both flag spellings, bare and multi-valued flags, agreement-is-not-an-override, repeatable flags, and that the orchestrator actually hands the merged strings to the tuner.
• new kb/tests/test_attest.py cases — the fourth outcome, the retire arithmetic (many inapplicable reads never retire; a real pattern still does), carry-forward of an inapplicable-only ledger, pre-existing documents.
• kb/tests + e2e_workflow/scripts/tests/test_e2e_store.py — 127 passed, 1 skipped
• e2e_workflow/scripts/tests — 1211 passed, 5 skipped, 77 subtests
• kernel_workflow/scripts/tests/test_experience_store.py — 83 passed
• all six e2e_workflow/scripts/test_*.js, and node --check e2e_workflow/e2e_workflow.js

Not yet exercised by a live run: the repair pass needs a real launch failure to fire.

Note for reviewers

main already has a 2b (tuning skillset). The recall section is numbered 2c and the PHASE=report Inputs line carries both KB_RECALL (§2c) and TUNING_RESULT (§2b).

🤖 Generated with Claude Code