Repository navigation
fix: fail-closed Crave wrapper (status, log, pull) + audit answers to the lock - #7
BangerSoul wants to merge 8 commits into
Conversation
revision_audit compared the checkout against `repo manifest -r` output - the witness of itself - so disk-vs-disk could never disagree, and a tree with 190 drifted projects passed as "OK all 1175 active projects" (job 303324). The lock is now the sole yardstick: SHA pins are compared directly against it, tag pins are resolved in-project, an unresolvable pin fails closed as MISMATCH, and `repo manifest -r` is demoted to corroboration with per-class counters - its disagreements become note lines, so it can neither rescue a bad tree nor convict a good one. Scenario 9 of test-revision-audit.sh is rewritten around that contract (unresolvable pin, witness cannot rescue it, clean corroboration, the 303324 regression, a contradicting witness), and a negative control via EMBERBIRD_RECIPE confirms the old logic goes red over a drifted tree with the exact false OK line 303324 printed. 🤖 Generated with Codebuff Co-Authored-By: Codebuff <noreply@codebuff.com>
status ran an unpinned getlog, looked for a "Your jobs" header the client
has never printed, and exited 0 even while the client answered with
"Error: could not get matching git url at: ..." on the same stream as its
data (observed live against job 303324). A status command that cannot
fail converts "I could not look" into "there is nothing wrong" - the one
confusion this runbook cannot afford.
It now pins --projectID/--jobID from $JOB or the run record and refuses
to guess when there is neither; parses the real section tables
("Your active jobs:" / "Job History:") with any other header ending the
section, so a project row can never be read as a job row; distinguishes
"left the queue" from "the client drew no table at all" (UNKNOWN, not a
guess); and judges every probe on both its exit status and its text,
exiting nonzero and naming the failing probe.
Pinned offline by the new tools/checks/test-crave-remote-build.sh: a stub
client on CRAVE_SHIM records the argv it was called with, and 15
assertions drive the real script through the pins, the table parsing, the
refusal to guess, and both failure modes. Wired into
run-offline-checks.sh; a control run against the pre-fix script goes red.
🤖 Generated with Codebuff
Co-Authored-By: Codebuff <noreply@codebuff.com>
Same treatment status got: both refuse without a job id instead of falling back to whichever job the cwd resolves to, both judge the client on its exit status and its text, and both exit nonzero with the reason named. `log` still streams whatever it received - a client error message is the most useful thing to show an operator who asked for a log - it just can no longer exit 0 while doing it. `pull` dies before the copy + verify steps, which would otherwise hash whatever the PREVIOUS pull left in the ticket and present it as this job's record; pull now also clears the ticket's eb/image/out before fetching, so a failed or empty pull reports "produced no directory" instead of quietly verifying leftovers. The stub client gains a pull mode; the offline test pins all three subcommands (33 assertions), including a full happy path through copy + x2 provenance verification with a stale leftover proving it never lands. A control run against the pre-fix revision goes red on the new assertions, among them the old pull copying a previous job's stale .iso into image/out. 🤖 Generated with Codebuff Co-Authored-By: Codebuff <noreply@codebuff.com>
ECC Tools / Security EvidenceCommit: Security evidence gate passed (success) No security-sensitive scanner-evidence gap detected. Mode: enforce Scanned 7 changed file(s). No missing scanner-evidence signal was detected. Check publication was denied or unavailable. An app owner must enable Checks: read and write, and the installation owner must approve the updated permission. |
ECC Tools / PR Risk TaxonomyCommit: PR taxonomy clear (success) Scanned 7 changed file(s). No taxonomy bucket signals were detected. Scanned 7 changed file(s). No PR taxonomy bucket signals were detected. Check publication was denied or unavailable. An app owner must enable Checks: read and write, and the installation owner must approve the updated permission. |
ECC Tools / Reference Set ReadinessCommit: Reference set readiness gaps detected (neutral) Reference evidence present for 0/7 areas (0%) across 7 changed file(s). This check is based on files changed in this PR. Repository-level readiness is still reported by
Check publication was denied or unavailable. An app owner must enable Checks: read and write, and the installation owner must approve the updated permission. |
ECC Tools / Hosted Promotion ReadinessCommit: Hosted promotion readiness passed (success) No hosted promotion evidence gaps detected across 7 changed file(s); 0 corpus scenarios had matching evidence. This check compares PR file changes against the evaluator/RAG promotion corpus in No evaluator corpus scenarios matched this PR. Check publication was denied or unavailable. An app owner must enable Checks: read and write, and the installation owner must approve the updated permission. |
Submission note for the resubmission from the PR branch tip (0e89318): run pinned the recipe only after the push, per the origin-reachability guard the 302852 lesson produced, and the live `status` dogfood shows the queued row alongside this machine's interleaved client error while exiting 1 - "answered partially" is now distinguishable from "answered". The known foreign-base seed risk (303324's ~8-minute failure mode) stays documented as the outstanding gap until inspect_seed refuses the node. 🤖 Generated with Codebuff Co-Authored-By: Codebuff <noreply@codebuff.com>
ECC Tools / Security EvidenceCommit: Security evidence gate passed (success) No security-sensitive scanner-evidence gap detected. Mode: enforce Scanned 7 changed file(s). No missing scanner-evidence signal was detected. Check publication was denied or unavailable. An app owner must enable Checks: read and write, and the installation owner must approve the updated permission. |
ECC Tools / PR Risk TaxonomyCommit: PR taxonomy clear (success) Scanned 7 changed file(s). No taxonomy bucket signals were detected. Scanned 7 changed file(s). No PR taxonomy bucket signals were detected. Check publication was denied or unavailable. An app owner must enable Checks: read and write, and the installation owner must approve the updated permission. |
ECC Tools / Reference Set ReadinessCommit: Reference set readiness gaps detected (neutral) Reference evidence present for 0/7 areas (0%) across 7 changed file(s). This check is based on files changed in this PR. Repository-level readiness is still reported by
Check publication was denied or unavailable. An app owner must enable Checks: read and write, and the installation owner must approve the updated permission. |
ECC Tools / Hosted Promotion ReadinessCommit: Hosted promotion readiness passed (success) No hosted promotion evidence gaps detected across 7 changed file(s); 0 corpus scenarios had matching evidence. This check compares PR file changes against the evaluator/RAG promotion corpus in No evaluator corpus scenarios matched this PR. Check publication was denied or unavailable. An app owner must enable Checks: read and write, and the installation owner must approve the updated permission. |
… can vanish Job 303483 exposed a blind spot the 2026-10-03 hardening could not close. The client's human job tables stopped being drawn for this account entirely - `list` answered with Configured Projects / Configured Platforms only, hour after hour, ~10 calls, while the job had in fact left the queue - so the table parse could only ever report UNKNOWN, and the job's real outcome had to be recovered by hand. The same client answers `list --jobID <id> --json` with the record that matters: status, exitCode, startTime, endTime, job_url. status now asks for it first, pinned to the job id, and keeps the tables as fallback and as corroboration, so a client that cannot answer json behaves exactly as before. The three cases are kept distinct: json locates the job (state, exit code, start -> end, the duration between them, the job url); json cannot be parsed or lacks the job (the table verdict stands); the two disagree about whether the job exists (reported as DISPUTED, exit nonzero - neither probe wins by default, because quietly preferring the convenient answer is how a dead job gets reported as healthy). The error-on-the-data-stream rule is unchanged and applies here too: verified live, the json arrives behind "Error: could not get matching git url" with the client exiting 0, so the state behind it is still printed and the command still exits nonzero. Against 303483 this prints `finished`, status 'done', exit code 130 and 2h18m where the previous revision printed UNKNOWN. Scenario 9 pins it (18 new assertions, 56 overall); the control run against 540c549 goes red on 14 of the 18 - the rest assert the table fallback it already had. The runbook records the 303483 post-mortem: the job ran 2h18m, so it was not the ~8-minute foreign-base seed class, and the platform retained no stdout, no diagnostics and no pullable artifacts, which leaves exit 130 unattributable from the client. PR #7 stays unmerged for that reason. 🤖 Generated with Codebuff Co-Authored-By: Codebuff <noreply@codebuff.com>
ECC Tools / Security EvidenceCommit: Security evidence gate passed (success) No security-sensitive scanner-evidence gap detected. Mode: enforce Scanned 7 changed file(s). No missing scanner-evidence signal was detected. Check publication was denied or unavailable. An app owner must enable Checks: read and write, and the installation owner must approve the updated permission. |
ECC Tools / PR Risk TaxonomyCommit: PR taxonomy clear (success) Scanned 7 changed file(s). No taxonomy bucket signals were detected. Scanned 7 changed file(s). No PR taxonomy bucket signals were detected. Check publication was denied or unavailable. An app owner must enable Checks: read and write, and the installation owner must approve the updated permission. |
ECC Tools / Reference Set ReadinessCommit: Reference set readiness gaps detected (neutral) Reference evidence present for 0/7 areas (0%) across 7 changed file(s). This check is based on files changed in this PR. Repository-level readiness is still reported by
Check publication was denied or unavailable. An app owner must enable Checks: read and write, and the installation owner must approve the updated permission. |
ECC Tools / Hosted Promotion ReadinessCommit: Hosted promotion readiness passed (success) No hosted promotion evidence gaps detected across 7 changed file(s); 0 corpus scenarios had matching evidence. This check compares PR file changes against the evaluator/RAG promotion corpus in No evaluator corpus scenarios matched this PR. Check publication was denied or unavailable. An app owner must enable Checks: read and write, and the installation owner must approve the updated permission. |
Two numbers in M2-CRAVE-BUILD.md were wrong, and a runbook that argues for measured claims should not carry unmeasured ones. "then 15 assertions" for the 2026-10-03 status fix is 17 - measured by running 51177d8's own test against 51177d8's own runner (17 pass, 0 fail). The 33 attributed to the log/pull fix is 38 (scenarios 1-8 today, none of which scenario 9 touched), and the current total is 56 with 18 of them new. The quoted `client error:` line is also brought in line with the 2026-10-05 attribution fix, which names the probe that printed it. Docs only: the offline suite is re-run and still reports 13 passed, 0 failed.
ECC Tools / Security EvidenceCommit: Security evidence gate passed (success) No security-sensitive scanner-evidence gap detected. Mode: enforce Scanned 7 changed file(s). No missing scanner-evidence signal was detected. Check publication was denied or unavailable. An app owner must enable Checks: read and write, and the installation owner must approve the updated permission. |
ECC Tools / PR Risk TaxonomyCommit: PR taxonomy clear (success) Scanned 7 changed file(s). No taxonomy bucket signals were detected. Scanned 7 changed file(s). No PR taxonomy bucket signals were detected. Check publication was denied or unavailable. An app owner must enable Checks: read and write, and the installation owner must approve the updated permission. |
ECC Tools / Reference Set ReadinessCommit: Reference set readiness gaps detected (neutral) Reference evidence present for 0/7 areas (0%) across 7 changed file(s). This check is based on files changed in this PR. Repository-level readiness is still reported by
Check publication was denied or unavailable. An app owner must enable Checks: read and write, and the installation owner must approve the updated permission. |
ECC Tools / Hosted Promotion ReadinessCommit: Hosted promotion readiness passed (success) No hosted promotion evidence gaps detected across 7 changed file(s); 0 corpus scenarios had matching evidence. This check compares PR file changes against the evaluator/RAG promotion corpus in No evaluator corpus scenarios matched this PR. Check publication was denied or unavailable. An app owner must enable Checks: read and write, and the installation owner must approve the updated permission. |
`status` already asked the client for `list --jobID <id> --json` after the human tables stopped being drawn for this account (job 303483, 2026-10-05), but `watch` still decided completion from that table alone - so the same blind spot was fatal there rather than merely inconvenient: with no table drawn, a table-only watcher could see the job leave the queue only by concluding it had not, and its miss counter turned the rest into "the client answered nothing 10 times in a row" while the real outcome sat one query away. The probe is now a shared function, asked on every poll, and `watch` follows the precedence `status` uses: json first, table as fallback and corroboration, and a disagreement between them reported and waited on - never resolved into a completion in either direction, because the one failure mode this must not have is "watch declared the build over while it was still running". A record with no such job at all is its own state (unseen), waited on and reported as not a completion. The give-up guard is kept for genuine silence and is now per- condition and bounded by WATCH_MAX_MISSES. WATCH_ONCE=1 runs a single poll, which is how the offline suite drives `watch` with no queue behind it. Also fixed on the way: `building`/`syncing` read as TERMINAL before, so a job in either state had its log captured mid-build; both are in the one shared still-active list the two probes are compared against. The negative control is also documented correctly now - a control runner written to /tmp dies at 128 on every assertion because the script derives REPO_ROOT from its own location, so the recipe writes it inside the tree instead. Scenario 10 adds 33 assertions (89 overall). Against `babd63e` it goes red on exactly those 21 that describe the new behaviour and on nothing in scenarios 1-9, which is what a correct control looks like. Live, against job 303483: the state is read out of the json behind the client's diagnostic (state=done), and the watch reports the real verdict - exit code 130 after 2h18m - instead of timing out blind.
ECC Tools / Security EvidenceCommit: Security evidence gate passed (success) No security-sensitive scanner-evidence gap detected. Mode: enforce Scanned 7 changed file(s). No missing scanner-evidence signal was detected. Check publication was denied or unavailable. An app owner must enable Checks: read and write, and the installation owner must approve the updated permission. |
ECC Tools / PR Risk TaxonomyCommit: PR taxonomy clear (success) Scanned 7 changed file(s). No taxonomy bucket signals were detected. Scanned 7 changed file(s). No PR taxonomy bucket signals were detected. Check publication was denied or unavailable. An app owner must enable Checks: read and write, and the installation owner must approve the updated permission. |
ECC Tools / Reference Set ReadinessCommit: Reference set readiness gaps detected (neutral) Reference evidence present for 0/7 areas (0%) across 7 changed file(s). This check is based on files changed in this PR. Repository-level readiness is still reported by
Check publication was denied or unavailable. An app owner must enable Checks: read and write, and the installation owner must approve the updated permission. |
ECC Tools / Hosted Promotion ReadinessCommit: Hosted promotion readiness passed (success) No hosted promotion evidence gaps detected across 7 changed file(s); 0 corpus scenarios had matching evidence. This check compares PR file changes against the evaluator/RAG promotion corpus in No evaluator corpus scenarios matched this PR. Check publication was denied or unavailable. An app owner must enable Checks: read and write, and the installation owner must approve the updated permission. |
…ader The suite's header explained why the json probe exists for `status` and stopped there, so a reader had no way to tell that scenario 10 drives `watch` through the same probe - which is the half that decides completion rather than reporting it. Header only; no assertion changes (still 89).
ECC Tools / Security EvidenceCommit: Security evidence gate passed (success) No security-sensitive scanner-evidence gap detected. Mode: enforce Scanned 7 changed file(s). No missing scanner-evidence signal was detected. Check publication was denied or unavailable. An app owner must enable Checks: read and write, and the installation owner must approve the updated permission. |
ECC Tools / PR Risk TaxonomyCommit: PR taxonomy clear (success) Scanned 7 changed file(s). No taxonomy bucket signals were detected. Scanned 7 changed file(s). No PR taxonomy bucket signals were detected. Check publication was denied or unavailable. An app owner must enable Checks: read and write, and the installation owner must approve the updated permission. |
ECC Tools / Reference Set ReadinessCommit: Reference set readiness gaps detected (neutral) Reference evidence present for 0/7 areas (0%) across 7 changed file(s). This check is based on files changed in this PR. Repository-level readiness is still reported by
Check publication was denied or unavailable. An app owner must enable Checks: read and write, and the installation owner must approve the updated permission. |
ECC Tools / Hosted Promotion ReadinessCommit: Hosted promotion readiness passed (success) No hosted promotion evidence gaps detected across 7 changed file(s); 0 corpus scenarios had matching evidence. This check compares PR file changes against the evaluator/RAG promotion corpus in No evaluator corpus scenarios matched this PR. Check publication was denied or unavailable. An app owner must enable Checks: read and write, and the installation owner must approve the updated permission. |
1 participant
Why
Three defects, all found the same way - by asking a check to fail and watching it refuse to:
revision_auditwas a tautology. It compared the checkout againstrepo manifest -routput - the witness of itself - so disk-vs-disk could never disagree. Job 303324 proved it: 190 SHA-pinned projects drifted, and the audit printedOK all 1175 active projectsover the same tree the canonical gate rejected.statuscould not fail. It looked for aYour jobsheader the client has never printed, rangetlogunpinned, and exited 0 through the client's ownError:lines (observed live:Error: could not get matching git url at: ...with exit 0).logandpullstill had the unpinned fallbacks, andpullwould copy + verify whatever the previous pull had left in the ticket's staging dir.What changed
066deec): the lock is the sole yardstick. SHA pins compared directly against it; tag pins resolved in-project; an unresolvable pin fails closed as MISMATCH;repo manifest -rdemoted to corroboration with per-class counters - its disagreements become note lines, so the witness can neither rescue a bad tree nor convict a good one.51177d8): pins--projectID/--jobID(refuses to guess when there is neither), parses the real section tables with any other header ending the section, distinguishes "left the queue" from "the client drew no table at all" (UNKNOWN), judges every probe on exit status AND text, exits nonzero naming the failing probe.0e89318): same treatment - refuse without a job id, pinned calls, fail closed on client errors.pullalso clears the ticket'seb/image/out/before fetching, so a failed pull can never leave the previous job's artifacts to be hashed and presented as this job's record.27373b0):statusnow reads the client's machine-readablelist --jobID <id> --jsonrecord first - status, exit code, start/end and the duration between them, job url - because the human tables stopped being drawn for this account entirely on 2026-10-05 (job 303483:listanswered with projects/platforms only for hours, so the table parse could only say UNKNOWN). The tables stay as fallback and as corroboration; the two probes disagreeing is reported as DISPUTED rather than resolved in favour of one.6869b28):watchdecides completion, so the table-only read was the dangerous half of that blind spot - with no table drawn, it could see the job leave the queue only by concluding it had not left, and its miss counter turned the rest into "the client answered nothing 10 times in a row". The probe is now a shared function asked on every poll, withstatus's precedence: json first, table as fallback and corroboration, and a disagreement reported as DISPUTED and waited on - never resolved into a completion in either direction. A record with no such job is its own state (unseen), waited on and explicitly not a completion. The give-up guard is kept for genuine silence and is now per-condition and bounded byWATCH_MAX_MISSES;WATCH_ONCE=1runs a single poll. Also fixed:building/syncingused to read as TERMINAL, so a job in either state had its log captured mid-build.Tests
test-revision-audit.shscenario 9 rewritten around the new contract + a negative control throughEMBERBIRD_RECIPE: the old logic goes red over a drifted tree, reproducing the exact false OK line 303324 printed.test-crave-remote-build.sh(new): a stub client onCRAVE_SHIMrecords argv; 89 assertions drive the real script through the pins, the table parsing, the refusals, both failure modes, the json probe,watchone poll at a time (WATCH_ONCE=1), and a full pull happy path (copy + x2 provenance verification) in which a deliberately stale leftover never reachesimage/out/.51177d8), the new assertions go red - including the oldpullcopying a stale.isointoimage/out/.540c549): red on 14 of its 18 assertions - the probe is never issued, so its state is neither reported nor judged - and green on the four that assert the table fallback the old revision already had. Verified live: the newstatusagainst job 303483 printsfinished, status 'done',exit code 130,2h18mand the job url, where the previous revision printed UNKNOWN.watch(babd63e): red on exactly the 21 assertions scenario 10 adds, green on all of scenarios 1-9 - the new behaviour is missing and nothing else moved. Two of the 21 are the old loop polling a stub forever (exit 124under the suite's owntimeoutbound), because it has noWATCH_ONCE. The control recipe itself was wrong until now too: a control runner written to/tmpdies at 128/127 on every assertion, because the script derives REPO_ROOT from its own location - so it is written inside the tree. Verified live:watchagainst job 303483 reads the state out of the json behind the client'sError:line (state=done) and reports the real verdict -exit code 130 (2h18m)- where the old loop would have burned its miss counter and given up blind.Risk / known gap
The foreign-base
.reposeed is still warning-only (inspect_seedreports, does not refuse). Job 303324 failed on exactly that, so the next submission may fail the same way (~8 min) until seed validation is escalated to fatal.Resubmission
Submitted 2026-10-04 as job 303483, pinned to this branch's head (
540c549tree, recipe recorded at0e89318), into the free slot: https://foss.crave.io/app/#/build/info/303483?team=14. Outcome (2026-10-05, vialist --jobID 303483 --json): the job is finished and did NOT succeed - statusdone, exit code 130 (interrupted), after 2h18m (2026-10-04T02:29:20Z -> 04:47:38Z). The recipe demonstrably fetched the pinned commit0e89318and ranbuild-from-manifest.sh; the ~8-minute foreign-base seed failure class is ruled out by the duration. The platform retained no telemetry -getlogsays "Could not find job stdout",fetchdiagnosticssays "not available",pullsays "no running jobs on this workspace" - so the in-job cause is not determinable from the client; the web UI build page is the only place the full log may still exist. The hardened wrapper behaved as designed throughout:statusreported UNKNOWN rather than guessing while the tables were absent, andpullfailed closed naming the probe. After the fact,watchrunning against this same job now reports the real outcome -state=done, exit code 130 after 2h18m - from the json probe, where the table-only loop it replaces would have hit its 10-empty-poll give-up. PR left open, not merged.