Merge/warmup plus tuning ci fixed - #431
Merged
Merged
Conversation
…e store
Close the read/write loop on the machine-produced, code-carrying KB so a
kernel lane can start from its own best historical patch instead of cold.
- scripts/experience_store.py: the store itself (stdlib + PyYAML, GPU-free).
`write` stores one measured win under
kb_artifacts/<gfx>/<kernel_class>/<slug>/<exp_id>/ (meta.yaml + patch.diff +
report.md) behind its own gate (missing_arch / no_improvement / empty_diff);
`resolve` enumerates a slug's solutions, keeps the SAME gfx only, ranks by
speedup and mirrors every candidate's prose into <eval>/kb_references so a
rejected warm start is still auditable. Neither subcommand ever raises — a
store failure prints {"written": false, ...} and exits 0.
- kernel_lane.js: new WarmStart phase between Profile and Optimize. Reads the
top-3 same-arch patches, validates EACH through the same verify_engineer
gate as a round winner, and adopts the first that passes; the recorded
speedup only ranks, adoption is decided by a fresh on-box measurement. After
Validate the run writes its own win back. The return value splits
total_speedup from incremental_speedup so a KB-derived gain is never
reported as this run's own work.
- kernel_workflow.js: thread warm_start / kb_artifacts_dir down to each
bake-off lane (the lane invocation spreads specific keys, not ...A).
- warm_start=off is a cold start, byte-identical to pre-feature behavior.
- kb_artifacts/ is gitignored: runtime-accumulated and unbounded.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…gnment Condense the verbose comment blocks in experience_store.py / kernel_lane.js / kernel_workflow.js (remove internal plan/KernelForge references and restated prose) and remove the dead `warm_start.total_speedup` write — the return value reports total from finalPrimary, never from that field. No behavior change; syntax + a write/resolve round-trip smoke test pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
KernelForge already ships the two-plane design this needs: one RewriteRecordStore protocol with LocalRewriteRecords on disk and KBStoreRewriteRecords over HTTP, selected by KNOWLEDGE_STORE_MODE. kb_store_local.py is that same on-disk shape, so the whole read/apply/optimize/write-back loop can be proven offline and moving to the service later is a change of backend, not of behaviour. Three properties are copied from upstream deliberately and must not drift: ranking is `speedup` descending and nothing else (the store does not know what a bench key is — comparability is the caller's job); candidates() reads knowledge documents only, so a 240KB patch is not paid for until it is selected; and every mutation lands by atomic rename, with a repeated session id meaning overwrite, because session ids are content-addressed and one port must stay one candidate. The id and path regexes are copied rather than widened: a record this plane accepts and the service rejects is exactly the failure it exists to catch early. kb_remote_upload.py gains --local DIR, which takes the SAME records the service path takes, byte for byte. That is the only thing supporting "proven locally = correct remotely", and it needs no KB_STORE_URL or token. Cross-checked against upstream's own reader: LocalRewriteRecords lists, ranks and materializes this tree identically (skipped when no KernelForge checkout is importable, so it catches drift without adding a dependency). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The store was lossless and unranked: every recorded win was offered, including retired duplicates and near-ties, and nothing said which measurements were even comparable. This adds the curation the read path needs and the export/resolve/ write path that reaches the same experience through a KB Store key. Curation (the directory plane): a `retained:false` gate, one rank per optimization `direction` with runners-up riding along as `alternates` rather than costing a second verify slot, a `--min-speedup` floor, and bench-key comparability — an imported `b:` measurement and an on-box `b2:` one are ranked together but never claimed to be comparable, because they are not. Re-recording code the store already holds counts a reproduction instead of importing our own output as a fresh win. tech_lead now emits closed directions as a machine-readable block, so the next run does not spend a round re-funding a dead end that has evidence against it. The store plane: `export-remote` maps an entry onto the record shape the service uses, under the seven-segment key `kernel:geak:<name>:rocm:<major.minor>:<triton|hip|ck>:mi355x` — framework stays `rocm` for all three languages because one container image supplies them, and the language is the `backend` dimension. `resolve-remote` and `write-remote` read and write through that key while printing the same JSON as `resolve`/`write`, so the lane needs no branch. What lands under a key is decided by the patch, not the caller: new code appends a session, the same code remeasured replaces its own. `--framework-version` exists because a box with no /opt/rocm measures no stack, and every record would then file under `unspecified` — splitting one kernel's history across two keys. It overrides the key segment only, never the recorded stack. On the read path a store root that is not there is a hard miss rather than an empty store: a typo must not quietly cold-start a run with experience waiting. Verified offline on the real 248-entry store: 80 records over 20 keys, all valid against upstream's own identity regexes and record_id; read a key, land its top patch on a workspace whose layout it was never recorded against, optimize on top, write back, and read the improvement out again. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`--kb_mode local|store` picks which plane warm start reads and writes. Only the two command strings change: the phases, the schemas, the verify gate, the remap and the adopt decision are untouched, because the store subcommands were built to print the same JSON as the directory ones. `local` stays the default, so an unparameterized run is unchanged. Writing in store mode records BOTH planes in one call. The directory tree stays the source of truth a curation pass edits and the KB record is derived from it, so the two cannot drift into disagreeing about what was measured. In store mode the lane logs the canonical id it read from and, on write, which of the two outcomes it got — appended a candidate under the key, or updated this patch's own. Both fields are declared in the schemas rather than left to additionalProperties, so the agent relaying the JSON has no reason to drop them. kernel_workflow and e2e forward the knobs the same way they forward the rest, so every recursive lane in a run uses the plane the run was launched with. The CI job that runs these tests needs pyyaml and the three new test files added to its explicit file list; that edit touches .github/workflows and is left out of this commit because pushing it needs a token with `workflow` scope. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`resolve` and `resolve-remote` had grown the same ~90 lines twice: collapse to one rank per direction, render reference_NN.md, build the candidate dict, write index.md. Two copies of prose that is supposed to read identically whichever plane served it is a slow drift, not a saving. Extracted `_collapse_by_direction`, `_render_references` and `_candidate`. Only the genuinely per-plane bits stay behind: the address in the page header (slug vs canonical id), the origin line (source eval dir vs session id + champion flag), and the extra candidate keys the store plane carries. Also drops three LocalKBStore members nothing calls (`configured`, `Candidate.as_dict`, `read_bytes`); the one test that used `read_bytes` now checks the same thing through `materialize`, which is the path production actually takes. No behaviour change: 1105 passed, and a read of the real probe store returns the same two ranks, the same alternates and the same index text as before. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Wires the e2e KB into e2e_workflow.js as a front read+validate module and a final write module, adds the remote plane to both lanes, and adds the one thing a store with no DELETE needs: a way to take a record back. Retraction is a rewrite to a tombstone (mode="replace" + deterministic session ids), and it does three things at once because doing two is worse than doing none: mark the document, zero the ranking scalars, re-point the champion. Shared by both lanes in kb_retract.py. 54 tests pass. Validated end-to-end against the real service on a scratch canonical id. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- e2e_store.py build_record: carry a `comparability` block (schema v2) so a stored speedup travels with the basis it was measured on (client, workload points, measured-on config) instead of being rediscovered per run. - e2e_workflow.js: template-literal → string-concat polish in the KB warm-start config path (no behavior change). - knowledge/learned: record the 2026-08-19 real-run confirms (gpt-oss-120b mxfp4 grouped-MoE +26.9% byte-exact after corrective re-author; Qwen3-14B-FP8 a8w8 swap-only 1.513× serving-wtd with prefill-regression note; new moe-fp8-blockscale-tune-gfx950 lever) and a roofline-prior calibration line for the launch-overhead-invisible failure mode. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…f kernel_workflow Both the kernel lane and the e2e serving lane warm-start from the same machine-produced KB, but the shared machinery lived under kernel_workflow/scripts/ with each lane carrying its own near-duplicate of open_plane / collapse-by-direction / ladder-publish. Hoist that into a single kb/ package both lanes import: kb/plane.py open_plane per-metric (primary, mirror, why) kb/curate.py collapse_by_direction one rung per idea, alternates ride along kb/ladder.py publish write one rung all-or-none, champion promote kb/identity.py kb/retract.py kb/store_local.py kb/store_remote.py kb/store_client.py kb/remote_upload.py (moved from kernel_workflow/scripts) e2e_store.py and its test move from kernel_workflow/scripts/ to e2e_workflow/scripts/ — it is the e2e lane's CLI, used only by e2e_workflow.js, so the reference is now same-subtree instead of reaching across into the kernel dir. e2e_store.py and experience_store.py both drop their private copies and call the kb/ helpers. Fix: retract --result recompute crashed (AttributeError) because the retract subparser has no --file; build_record now reads it via getattr. The move pulls e2e_store.py (~313 stmts) into the coverage tree; new test_e2e_store.py brings it to 99.68% and both e2e_store tests are added to ci-l0-checks.yml. .gitignore ignores the on-disk store root (kb_store_local/). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ible records Four gaps in what the KB gave a reader back, and one new module both lanes share. Recall ordering. Only the finest e2e rung ranked on throughput; the two coarse rungs — the ones a reader on a different workload point actually lands on — ranked on speedup, so "the offer is ordered by throughput" was true on a third of the pages. The read path now opens every rung on throughput (--sort-by restores the old order). This had to move to the store metric rather than a client-side re-sort: RemoteKBStore.candidates() pages sessions/top?metric=self.metric, so re-sorting a speedup-ranked sample ranks a biased sample. Writes still crown per-rung; the two metrics are reported as separate fields rather than collapsed into one word. Attestation (kb/attest.py). A record's `validated` flag is a judgement its own writer made about its own measurement, once. It cannot answer the question that decides whether the record is worth keeping: has anyone since pulled it out, run it, and had it work. The new value.attestations ledger counts that — recalls / validations / failures / not_reproduced plus a bounded history — in one vocabulary both lanes use. `recalls` counts attempts ON HARDWARE, not reads, or the only ratio a retire pass can act on would decay for records nobody ever doubted. Unlike retraction it moves no ranking scalar and re-points no champion: one failure on one box is evidence, not a verdict, and collapsing the two would make the command too dangerous to run automatically, which would mean it never ran. retire_hint() is advisory, a string naming which pattern fired, and nothing filters on it. Writes carry the ledger forward, because session ids are content-addressed off the config and exclude the measurement — re-benching one config lands on the SAME session and would otherwise silently reset its whole history while looking well-formed. Reproducibility. An e2e record could be recalled and consist of nothing you could run. Now value.repro is structured, launch.sh is synthesized against bench_e2e.sh's env contract when none was captured (and says plainly that it was), kernels without their patch are counted rather than omitted — three kernels and two patches otherwise reads either as a no-op or as lost bytes, and those point opposite ways — and --kernel-store fetches patches from the kernel lane, whose scratch is usually gone by the time an e2e run finalizes. A result with no script, no flags, no env, no patch and no overlay is refused: this store has no delete. Not reproduced != rejected. The warm-start verdict was binary, so "would not run" and "ran but did not win" were the same word despite meaning opposite things to a retire pass. It is three-way now, benched candidates attest back (non-fatal), and the ones that did not reproduce get their own REFERENCE ONLY section carrying the launch script and patch paths, as leads for the optimization flow rather than as discards. Verification: 190 passed, 1 skipped over the CI set; node --check clean. Nine failures in test_experience_store.py's export-remote cluster are pre-existing drift, confirmed by stashing this work and re-running. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…rust The KB_ENV_PRELUDE no longer probes for /shared_nfs/hyperloom/ca and exports SSL_CERT_FILE/REQUESTS_CA_BUNDLE/CURL_CA_BUNDLE/NODE_EXTRA_CA_CERTS itself. Container TLS trust (AMD CA + DNS) is a launch-time concern, injected by the run harness via `docker run --add-host` + a read-only CA mount and the four CA env vars (warmstart_run/node_docker.sh -> kb_net_docker_args). Keeping a copy here baked a /shared_nfs host path into the repo for a job the launcher already does, so remove it. The prelude is back to just KB_STORE_URL + KB_STORE_TOKEN. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Most callers run the e2e/kernel workflows OUTSIDE the warm-start launcher (node_docker.sh), which is the only place that injects AMD-CA trust at `docker run`. GEAK's own CI (ci/node/run_geak_e2e.sh) is one such caller: it drives run_e2e.py with the default kb_mode=both (remote KB) and sets no CA of its own, so without an in-workflow fallback its warm-start silently degrades to a cold start on any node where the gateway's internal AMD CA is untrusted. So KB_ENV_PRELUDE again DETECTS then heals: only when SSL_CERT_FILE is unset does it point urllib/requests/curl/node at the first readable AMD-root bundle (KB_CA_BUNDLE override, else the shared Hyperloom bundle). It is a strict no-op when the caller already set SSL_CERT_FILE (warm-start lane) or no bundle is readable (CI / already-trusting images stay byte-identical). Path-only, no CA content or secret in the repo. Supersedes d6fd2ee. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
When the e2e workflow crashes at the wall-clock before writing
director_e2e_validation.json, _recover_workflow_return rebuilds the
return from on-disk artifacts. It previously discarded any adopted
serving config from the sweep, so warm-start recall gains never reached
the reported/written-back result and got flattened to no_gain.
Recover the adopted config from config/sweep_results.json in three tiers:
1. best accepted intermediate kernel win (now folds the config in, and
restacks its baseline to the true default when the kernel A/B ref
leg ran on the config-applied server),
2. adopted serving config only (new tier),
3. true no_gain (last).
The kernel-win restack uses a midpoint test (ref leg on the
config-applied side of the baseline->config gap) so it is robust to ~1%
measurement drift but refuses to double-count a kernel measured against
the raw baseline.
Verified against the 20260820 crash artifacts: deepseek 1.05x->1.20x
(config+kernel), mixtral/qwen27b/qwen14b recovered from no_gain to
1.07x/1.18x/1.65x; gptoss and qwen122b unchanged (correct).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two holes, both found by auditing which measured gains actually reached the deployment KB. The write lived only at the tail of e2e_workflow.js (Module B). When that process died, run_e2e.py's disk recovery still rebuilt a complete result and wrote result.json, but nothing wrote to the KB -- the 08-20 e2eall batch lost a x1.86 gpt-oss-120b and a x1.05 DeepSeek-V4-Pro that way, with a healthy warm-start read on both. run_e2e.py now writes the record itself after the result lands, driven by kb_identity.json (dropped by the resolve step, raw argv so the flags hand straight back) and gated on kb_write.json so it never duplicates Module B's own write. The other direction: a same-session ratio of 1.02 that the Director already declared validated_no_win was still being written, and publish() promotes on score alone, so box drift took the champion slot on seven deployment pages. win_gate() in e2e_store.py is now the one implementation of "may this be recorded", shared by both writers, and publish(promote=False) keeps a salvaged provisional number from displacing a validated champion. kb_env.sh exists because two programs now issue KB commands; a second copy of the token path and CA fallback list in Python would drift silently, and the only symptom would be a run whose result never reached the KB. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…mmits The rebase onto main replayed only the branch's non-merge commits, so three hand-resolutions that lived exclusively in the discarded merge commits were lost. Restored verbatim: - ci-l0-checks.yml: kb/tests/test_attest.py was no longer in the L0 test list, so the attestation suite existed in the tree but never ran. - ci-l0-checks.yml: the pyyaml rationale for the experience-store meta.yaml tests, re-applied on top of main's reworded comment. - kernel_lane.js: the WarmStart phase detail had reverted to the local-only wording, which understates the remote-first behaviour the lane now has. No functional KB code changed; kb/, experience_store.py and e2e_store.py are byte-identical to the pre-rebase branch tip.
`final_overlay` names a directory, always -- the mechanism is a sitecustomize.py plus the modules it swaps in, and no single file carries it. _artifact_files() took only os.path.isfile, so it dropped that directory without a word and no e2e record has ever carried an overlay. The runs that cleared _repro()'s reproducibility gate were the ones that also happened to emit a kernel patch; a run whose whole win lived in the overlay could not be recorded at all, which is what blocked backfilling the salvaged x1.86 gpt-oss-120b. Pack the mechanism only -- manifest, sitecustomize.py, _patched/ -- under a single `overlay/` top level. The accepted-candidate directory also holds the A/B evidence it was judged on (cand/, ref/, server logs, profiles): that is how the number was arrived at, not how it is reproduced, and it dwarfs what a reader needs. No manifest means it is not an overlay directory and nothing is packed, so the gate still refuses rather than promise something nobody can install. The synthesized launch.sh untars it before pointing OVERLAY_PYTHONPATH at it, guarded so a reader who already unpacked their copy does not lose edits. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both branches fork from the same main (f1aa29b) and add disjoint features: the KB warm-start plane and the standalone vendored tuning skillset. Three textual conflicts, all "both sides appended at the same spot": .gitignore - keep both blocks (kb_artifacts/kb_store_local from warm-start, the tuning_skillset re-include whitelist from tuning). Neither rule reaches the other's tree; tuning_skillset_sync.py --verify still sees all 337 files, so the whitelist survived the new ignores. e2e_workflow.js @schemas - keep KB_RESOLVE_SCHEMA and TUNING_SCHEMA. e2e_workflow.js @return - keep kbWarmStart and tuningReturn(). The fourth site needed an actual merge rather than keep-both: warm-start had refactored the deep-head integrator's inline input object into a `deepInputs` const, while tuning had spliced `...tuningIntegrateInputs()` into that same literal. Taking either side alone loses the other's change, and this call site invokes safeAgent directly instead of going through runIntegrateBothLegs, so it does NOT pick up the central tuning injection at runIntegrateBothLegs. The spread is therefore carried into deepInputs explicitly. Verified: node --check; all four e2e_workflow/scripts/test_*.js; the vendored skillset manifest. test_expert_skills_off_identical.js fails on the warm-start branch on its own (pre-existing); fixed in the follow-up commit.
The guard pinned roleAgent's return to the literal `return base + expertSkillsBlock(role);`. warmStartBlock(role) was appended after the guard was written, so the branch reds an L0 check on a line that is still perfectly additive - the assertion was matching the block list, not the invariant. The invariant was never "expert skills is the only injector": it is that base is never rewritten and every block hanging off it is inert when its own gate is off. So match the tail of the return generically and hold each extra `<name>Block(role)` found there to the SAME off-is-empty proof already applied to expertSkillsBlock - rebuilt from real source with its SCREAMING_CASE module deps neutralised. warmStartBlock is now covered, and the next injector is covered the day it is added with no edit here. Mutation-checked: deleting warmStartBlock's `if (!KB_REF_DIR || ...) return ''` turns the new assertion red.
…cts' into merge/warmstart-plus-tuning
…1202 round 2) The 20260823 Qwen3.5-27B-FP8 / gpt-oss-120b / DeepSeek-V4-Pro sessions each ran the full budget and shipped NO final report, with the 50min reserve configured and never overridden. Three defects, all on the same path: 1. ELAPSED_MS under-counted. The clock was a self-rearming 60s setTimeout chain, so each rung's scheduler lateness was added to the next rung's start and the error compounded. Under-counting is the dangerous direction: remainingMs() reports time the run does not have. Replaced with an absolute ladder armed at t0 (bounded to 2048 rungs), where a late rung only delays itself. 2. The Finalize-gate was wall-clock-unbounded, and it is the dominant cause. TIME_DEADLINE_HIT only stops STARTING new head/milestone work; the pendingIntegrations drain loop after it boots a server and benches two legs per iteration with no deadline check. All three runs were killed in there after 3.4-4.4h of zero artifact writes. New TIME_FINAL_DEADLINE_HIT (single absolute timer at TIME_BUDGET_EFFECTIVE_MS) skips the gate wholesale if the reserve has already begun, and breaks the loop per-iteration otherwise. Unfinished A/Bs stay in pendingIntegrations and reach the caller via pending_integrations: deferred, not discarded. 3. result.json advertised paths that did not exist. report_path fell back to <eval_dir>/final_report.md and final_launch_script to final/final_launch.sh unconditionally, so a killed run looked like it had produced both. They now resolve to "" when absent, and _emit synthesizes a final_report.md from the recovered numbers (LLM-free, atomic, never overwrites a real report) with an explicit SYNTHESIZED banner plus final_report_synthesized=true, so nobody promotes an unvalidated headline off it. Tests: new e2e_workflow/scripts/test_time_budget_reserve.js runs the clock extracted from the shipped source against a virtual scheduler that injects lateness (ladder 5s worst-case under-count vs 55min for the chain) and pins the gate's deadline checks; 7 new cases in test_run_e2e_recovery.py cover the report fallback and the phantom paths. Wired into CI L0.
…xemption 2f30ee9's Option A capped every optimization agent at max(2min, remainingMs - FINAL_RESERVE_MS) and exempted the final phase, keyed on the phase LABEL via FINAL_PHASE_LABELS. That is correct everywhere except the one place it needed to hold: the Finalize-GATE (the pendingIntegrations drain that runs BEFORE phase('Finalize')) tagged BOTH its agents 'Finalize' too. So pure measured optimization work — a server boot plus two benches, retried AB_FINISH_RETRIES times, up to 4 x AGENT_TIMEOUT_MS = 8h — took an exemption written for Finalize/Report/Validate and ran with no budget bound at all. That is why the 20260823 sessions still burned 3.4-4.4h past the deadline with the cap in place. The gate now runs under its own 'Finalize-gate' phase, which is deliberately not in FINAL_PHASE_LABELS, so it takes the cap like every other optimization agent (and shows up as its own progress group, which would have made the stall visible in the log). The real Finalize/Report/Validate keep their exemption. test_time_budget_reserve.js lifts agentTimeoutFor out of the shipped source and asserts the verdicts directly: gate agent capped, Finalize agent exempt, 'Finalize-gate' never in FINAL_PHASE_LABELS, and both gate call sites passing it.
Replaces the FINAL_PHASE_LABELS list with a single FINAL_PHASE_STARTED flag set
at the phase('Finalize') call site. agentTimeoutFor() now takes no arguments.
The list was a self-reported claim, and it failed OPEN: name your phase
'Finalize' and you got unlimited time, silently. That is exactly how the
Finalize-GATE (which runs BEFORE phase('Finalize') and tags its integrator
agents 'Finalize') took an exemption written for the final phase and ran
unbounded. The flag is a fact about where execution actually is, so there is
nothing to pass and nothing to keep in sync: no agent added anywhere before that
line can opt itself out, and the previous commit's 'Finalize-gate' relabel is
demoted to display grouping only.
Agents in flight when the flag flips keep the cap they were armed with (the
timeout is fixed at agent start), so optimization work that began before the
final phase stays bounded by the reserve. No call site changes; no-budget path
still byte-identical.
50min was p75 of 85 historical final phases -- fine for the median run and exactly wrong for the runs that need it. Big models (DeepSeek-V4-Pro, gpt-oss-120b, Kimi) spend most of the final phase waiting on server boots, and it was those runs, not the median ones, that were SIGKILLed with no report at all. Note this is a revert of the direction 2f30ee9 took (3600 -> 3000). The 20% cap is unchanged, so it now binds below a 5h budget rather than below ~4h10m; a budget that short cannot afford an hour anyway. Pinned by three new assertions in the reserve regression: the 60min default at 12h, the cap taking over at 3h, and GEAK_FINAL_RESERVE_S still widening.
feat/tuning-skillset branched before the held-node work and still carries only the sbatch dispatch, which pends indefinitely under the current QOS limits. This takes ci/ wholesale from ci/update_ci_models (that branch is the only one that touched ci/ since the merge base, so there is nothing to reconcile) so the tuning-skillset optimizer can be exercised by the same runner as the previous matrix. Local-only branch for the 13-node run; not for upstream.
Nine findings, all in vendored tuning_skillset/: `generic-api-key` wants a keyword near a high-entropy string, and an adjacent `KeyError:` or `token_to_kv_pool` supplies the keyword while the dtype literal supplies the entropy. Allowlisted on the secret rather than the path, so this excuses exactly three literals and leaves every file fully armed — tuning-kb/*/artifacts/ carries real sglang source, and a path-shaped exclusion there would be a place to hide a real key.
The check reported 14 undeclared constants, every one a false positive: prose and shell words
(NOT, THIS, SSL_CERT_FILE, KB_STORE_TOKEN) lifted out of strings the stripper had lost track of.
The cause is the cascade itself. Each regex re-scans the whole file, so a construct holding a
delimiter belonging to a later pass shifts every boundary after it and the rest of the file is
read against the wrong state. Replaced with a single left-to-right scanner that closes strings,
templates, comments and regex literals in the order they actually appear, and counts brace depth
inside ${...} so an object literal's `}` no longer ends the substitution early.
Still catches both real cases: a constant declared only in kernel_workflow.js, and one declared
nowhere. That mattered — a check that cries wolf 14 times is one nobody reads the 15th time.
12 failures, all stale expectations rather than broken behaviour: the ladder redesign publishes each measurement at both an exact and a coarse address, and the suites still counted raw records and spelled canonical ids in the old vendor-first order. - count measurements, not rungs: an exact() helper filters to rung 0, so "how many measurements" stops meaning "how many rungs those measurements were published at" - canonical ids respell to scheme:kernel:gpu:name:backend:framework[:version] - champions are one per rung, and a version-only mismatch now reports any_version, not other_version - the stale fake in test_e2e_store grew the `promote` argument the real publish() takes, so it fails on a signature change instead of passing against a function that no longer exists Also covers the identity handoff and the overlay packer in the same file, both untested: the handoff is what lets a writer address the page a reader found, and the packer is what a pure-overlay win travels as. Pinned that .pyc files stay out of the bundle — shipping bytecode built for another interpreter is how an overlay installs cleanly and runs the code it replaced.
With the unit tests passing, the coverage gate stopped being masked and failed at 95.13% against its 97% ratchet. Two gaps, both of them code this branch added without tests. _kb_write_back was the largest single uncovered block in the tree. It is the path that makes a crashed run worth something to the next one, and nothing exercised it: not the address it sends, not the refusal to file an unaddressable record, and not the promise that a KB failure stays a missed record rather than becoming a failed run. Its direction string is pinned too, since e2e_workflow.js spells the same key and a second spelling splits one deployment's history. Four suites existed in the tree but were named nowhere in the workflow, so they never ran — test_effective_config and test_bench_replica_lifecycle arrived with the Hyperloom alignment, test_roofline_skill and test_vllm_adapter_profiler_config predate it. Because the file list is explicit, an unnamed test file is a test file that does not exist, which is the same trap the contract checks were added to close. They also measure the modules that were dragging the gate down. Added the parser edges those suites left open (effective_config is now at 100%), where the contract is that ambiguous text is canonicalised or rejected by name, never silently dropped. Coverage 95.13% -> 97.44%.
… points Removes the 12 model directories under tuning_skillset/tuning-kb (295 files, ~12M of captured artifacts, patches and evidence logs). README.md and ENTRY_TEMPLATE.md stay, which is the whole of what anything actually references: test_tuning_skillset_sync and test_tuning_skillset_phase both assert on tuning-kb/README.md and nothing else, and no code names an individual model. Manifest re-recorded (337 -> 42 files), since the integrity job hashes the vendored tree against it and would otherwise read 295 deletions as drift. The gitleaks dtype allowlist stays: four of its nine findings are in tuning-aiter/SKILL.md, tuning-ck/SKILL.md and docs/coverage_gfx950.md, which are outside tuning-kb and unaffected here.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
All three are non-security uses that the queries cannot tell apart from the real thing, so the fix is to stop tripping them rather than to suppress them. - `_render_reference` / `_render_references`: the hash is a write-time uniquifier for a reference page's filename, never read back (every consumer globs `e2e_reference_*.md` or walks `sets/`), so moving it from SHA-1 to SHA-256 changes nothing but the query's verdict. The `b2:` bench key and the `exp_id` stamp are left alone — those values ARE persisted, and rehashing them would invalidate existing entries. - The kernel verdict table's `why` cell escaped `|` but not `\`, so a backslash in the text could carry the escape and break the cell. Escape both, and slice before escaping so the 160-char cut cannot split a `\|` pair in half. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Collaborator
|
With suggestions on the current design:
|
Addresses the review comment that tuning knowledge should land in the shared KB instead of a parallel store. This is the plumbing half; the directory reorganization is left to a follow-up. A tuned table is applied through an env var and deployed INTO the installed package, not into the framework's git tree, so it is structurally absent from final.patch — no amount of diffing the source picks it up. The KB record therefore carried the launch command but not the thing that made it fast. DeepSeek-V4-Pro 20260823T124728Z is the worked example: the tuning phase banked a 56-row a8w8 blockscale table measured at 3.29x isolated, the run wrote a validated_win at 1.043x, and its record's files were [final.patch, launch.sh, report.md]. accepted_kernels was []. The lever stayed on the box, so the next run at that canonical id recalls a configuration it cannot reproduce. Two changes: - e2e_workflow.js banks each accepted tuned op into acceptedKernels as kind=env, deterministically at the phase's own accept gate rather than leaving it to Finalize to remember — the run above proves the role returns []. Gated on tuneOk, so an unproven claim banks nothing, just as it folds nothing into curEnv. With several ops the per-op e2e_delta_pct is left at 0: the phase measured one A/B covering all of them, and stamping the whole delta onto each would double-count it. - e2e_store.py carries tuning_skillset.artifacts and .live_tree_files into the record's file set under tuning/. Accepted tunings only, so a rejected search residue cannot be mistaken for a banked lever. Files are index-prefixed because the deploy bundle's copy and the installed copy share a basename, and anything missing, unreadable, oversized or over the count cap is reported on stderr rather than dropped quietly. Both are inert without an accepted tuning, so a run with the phase off is byte-identical. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Warm-start KB credentials were read only from KB_STORE_URL / KB_STORE_TOKEN, which collide with Hyperloom's own KB_STORE_* on a shared host. Prefer the GEAK_-prefixed names everywhere the credentials enter, falling back to the bare names so existing setups are byte-for-byte unaffected. - kb/store_client.py: new kb_store_url()/kb_store_token() helpers; from_env uses them. - kb/store_remote.py, kb/remote_upload.py: guards use the same precedence (remote_upload normalises into the bare names so the upstream client works too). - kb_env.sh, kernel_lane.js KB_ENV_PRELUDE, ci/node/run_local.sh: same precedence, re-exported under the bare names every downstream reader branches on (token stays name-only in docker -e, never in argv). - docs/help text + kb/tests/test_store_env.py (precedence, fallback, strip, from_env). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
yueliu14
approved these changes
Aug 25, 2026
Umangatamd
approved these changes
Aug 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
Merge of two feature branches —
feat/kernel-warm-start-kb-artifactsandfeat/tuning-skillset— plus the Hyperloom measurement alignment that both depend on. 34 commits, ~26k insertions.The two features answer the same complaint from different directions: GEAK re-derived the same tuning from scratch on every run, and had no vocabulary for "tune the existing kernel" as distinct from "author a new one." The KB gives a run access to what previous runs found; the skillset gives it a procedure to follow when there is nothing to recall.
Features
1. Warm-start knowledge base (
kb/,*_store.py)A run can now open with what earlier runs on the same workload already established, instead of an empty context.
kb/package —store_local.py/store_remote.py/store_client.pybehind one interface, so a lane reads the same whether the entry came from a directory or from behind a KB Store key. Plusidentity.py(content signatures),attest.py,curate.py,ladder.py,retract.py.kernel_workflow/scripts/experience_store.pyande2e_workflow/scripts/e2e_store.py— write/read/rank the entries, and render each offered candidate into prose the Director actually reads (KB_REFERENCE_DIR/e2e_reference_*.md).kb/retract.py) — an entry later shown to be wrong can be pulled without rewriting history.2. Tuning skillset (
tuning_skillset/)A vendored, versioned skillset that gives the tuning path an actual procedure, and a new
tuning_specialistrole plus a dedicated tuning phase ine2e_workflow.jsto drive it.tuning-aiter,tuning-ck,tuning-hipblaslt,tuning-triton,tuning-hip,tuning-flydsl, and framework-sidetuning-in-vllm/tuning-in-sglang.tuning-core/carries the parts that are easy to get wrong and expensive to get wrong:measurement.md,engagement_verification.md,correctness_gates.md,graph_captured_benchmarking.md,search_strategy.md,arch_migration.md,clocks_and_power.md.tuning-kb/— per-model tuning entries the skillset can recall.validate/claims.py— every performance claim in the skillset is machine-checkable against a report;report_{vllm,sglang}[_gfx950].jsonare the checked-in fixtures.e2e_workflow/knowledge/tuning_skillset.manifest.sha256and enforced in CI, so the vendored tree cannot drift from its source.3. Measurement alignment with Hyperloom
The reason earlier numbers were hard to compare across harnesses.
interface/effective_config.py— one place that resolves what config a run actually ran with, instead of each consumer reconstructing it.e2e_workflow/scripts/bench_replica.sh+ cache-cold replica alignment — isolated replicas now measure on the same basis as the e2e leg.bench_e2e.shreworked; recovery, dispatch, and tuning-result handling inrun_e2e.pyhardened, each with tests.4. CI / dispatch
ci/dispatch/reorganized into held-node dispatch (held_nodes.sh,held_submit.sh,held_job.sh) with the old sbatch path kept underlegacy_sbatch/.ci-l0-checks.ymlgains the manifest check, the tuning-phase regression, the time-budget reserve test, and therun_e2edry-run mapping check.5. Tests
~4.5k lines of new test code:
kb/tests/,test_experience_store.py,test_e2e_store.py,test_kb_loop_offline.py,test_kb_retract.py,test_effective_config.py,test_bench_replica_lifecycle.py,test_tuning_skillset_sync.py, plus Node-sidetest_tuning_skillset_phase.jsandtest_time_budget_reserve.js.Test results
9 models, two waves, uniform workload ISL/OSL 8192/1024, concurrency 64, 24h budget. Wave 1 (0822) ran warm-start KB only; wave 2 (0823) added the tuning skillset.
Reported below is GEAK's own final report —
geak/e2e_cycle*/workflow_return.json, which is a same-session tight A/B (base vs final, server flags held fixed). This is the isolated figure and it is independent of how the optimization was applied. Runs are counted as a gain only onvalidation_status == validated_winandoutput_parity == pass.Wins
gemm_a8w8_blockscale_bpreshuffle_ck, all 4 dense-linear shapes — isolated 1.694x, 28.7% of GPU timefrom_knowledge_base: true)gate: accepted,mode: kb_assisted)Both wins are the features working as designed, and each exercises a different one:
kb_session_id: geak-qwen3-14b-fp8-7aa985936b3b-…) and applied it askind: env— no patch, no re-tuning. This is the warm-start case: the run did not rediscover the lever, it looked it up. Both cycles independently landedvalidated_win.tuning-kb/deepseek-v4-pro,tuning-aiter,tuning-core,tuning-in-vllm) and produced a new 56-row tuned CSV, then verified engagement (544 → 0 fallbacks).Independent reconciliation against a separate 4-leg interleaved A/B (
geak_benefit_0821.csv) puts these at +30.3% and +5.05% hot-to-hot, the latter with fully disjoint distributions (gap 2.86%, noise floor 1.90%).Not counted
Reported for completeness rather than filtered out:
validated_win1.057x / parity pass, but a clean 34-minute revalidation only reproduced +2.4%, and the skillset's own gate on that run wasno_win. Graded do-not-cite; excluded.recovered_intermediatewith the head already consumed by a priorintegrate_patch— not GEAK's final. Excluded.validated_no_winin the 0.999–1.037 range. The tuning skillset ran on 5 of the wave-2 runs and gatedacceptedon 2,no_winon 3 — on the small dense models it correctly found no headroom rather than manufacturing a number.So: 2 of 9 models produced a defensible GEAK-attributable gain. That ratio is the honest read. The e2e campaign totals are considerably larger (+51% on Qwen3-14B-FP8-sglang, for instance) but most of that is framework/explore levers — server flags, attention backend,
kv-cache-dtype— not GEAK kernels, and it is not claimed here.