Skip to content

F2 receipt key uses CARGO_PKG_VERSION, so rc.1/rc.2/final (and every dev build) of one version share receipts: a forward-path change skips the CPU reference check #4290

Closed as not planned

Description

@noahgift

Found from the cop's gx10 finding, 2026-09-24. Read at release/0.69.3.

How the guard decides. f2_receipt::decide (f2_receipt.rs:161) skips the CPU reference forward when a stored receipt's key equals the current key. The key is {model_sha256, apr_version, device}. apr_version() (f2_receipt.rs:313) is env!("CARGO_PKG_VERSION"), the RUNNING binary's version.

What the gx10 line means. The line read 'F2 guard: receipt matches (… apr 0.69.1 …) [source=receipt]'. It prints the running binary's own version, so it came from a 0.69.1 binary, not from rc.1 (7ff50ec is 0.69.3). A 0.69.3 binary never accepts a 0.69.1 receipt: that is AprVersionMismatch.

The real defect. Every build that shares a version string shares receipts: v0.69.3-rc.1, rc.2, the final tag, and every dev build on release/0.69.3. RC2 rewrites the generation loop (one-engine, #4263), yet it would skip F2 on a receipt that rc.1 wrote.

Fix. Key the receipt on the build identity:

  • the git sha when one is embedded;
  • otherwise a hash of the executable (the rc assets report '+no-git');
  • never the version string alone.

Case table: same version with a different sha → Validate; a +no-git build with a different exe hash → Validate; identical build → Skip. Include a planted mutant that reverts to the version-only key.

0.69.3 mitigation (captain ruling). Every G1–G3 cell on RC2 runs with --revalidate, or with an empty receipt dir, and its log must show F2 source=fresh, not receipt.

Filed by aprender-36.

Activity

  1. added this to the 0.69.4 milestone on Sep 24, 2026
  2. noahgift commented on Sep 24, 2026

    @noahgift
    MemberAuthor

    Claimed by aprender-59; branch fix/4290-f2-receipt-build-identity.

  3. noahgift commented on Sep 25, 2026

    @noahgift
    MemberAuthor

    Verified on batch/0.69.4-epic-4249: the fix is 0590d50 + 9e72256 (the F2 receipt key includes the executable's sha256). cargo test -p aprender-serve --lib f2_receipt: 21 passed.

  4. noahgift commented on Sep 25, 2026

    @noahgift
    MemberAuthor

    Related cost, measured 2026-09-25 on lambda/4090 while A/B-ing two binaries against the same model: the F2 receipt ping-pongs between versions.

    • The 0.69.3 build (7ff50ec) accepts the receipt.
    • The 0.69.4 build (da38177) then logs F2 guard: validating on this run (receipt schema 2, this build writes 3) and rewrites it.
    • The next 0.69.3 start revalidates again.

    Each alternation costs 78–206 s of first-request TTFT (F2 guard: passed in 206403 ms on the first run). Any A/B harness, and any fleet host that runs two versions side by side, pays this on every switch. Keying receipts by schema (or by build) so that versions coexist would remove it.

  5. modified the milestones: 0.69.4, 0.69.5 on Sep 25, 2026
  6. added
    P0Critical priority
    owner:la-0.71Owned by the 0.71 look-ahead slot
    on Oct 4, 2026
  7. modified the milestones: 0.69.5, 0.71.0 on Oct 4, 2026
  8. noahgift commented on Oct 12, 2026

    @noahgift
    MemberAuthor

    Evidence (E3 look-ahead lane), read on main @ 70f8c754d3.

    • The fix is on main. apr_version() (crates/aprender-serve/src/gguf/inference/forward/f2_receipt.rs:321) returns build_identity(CARGO_PKG_VERSION, <the running executable's bytes>), so the key is the build and not the version string.
    • The case table is on main. f2_receipt_key_is_the_build_not_the_version in f2_receipt_tests.rs checks three cases through decide:
      • same version, different exe → Validate (AprVersionMismatch);
      • identical build → Skip;
      • unreadable exe → never matches.
    • The planted mutant is covered by that test. A key that reverts to the version alone makes rc1 == rc2, and the first assert fails.
    • Still open: the ping-pong cost in the comment above. receipt_path(dir, model_sha256) gives one file per model, so two builds on one host overwrite each other's receipt, and each switch pays F2 again. That is a separate defect from the key.
  9. added
    P1High priority
    and removed
    P0Critical priority
    on Oct 14, 2026
  10. noahgift commented on Oct 14, 2026

    @noahgift
    MemberAuthor

    P0 to P1 (2026-10-14 P0 triage): P0 now means it blocks the next release or carries the operator's own words on it; this issue does neither, so it stays on the 0.71.0 milestone at P1.

  11. noahgift commented on Oct 14, 2026

    @noahgift
    MemberAuthor

    merged into #5312 (master ticket for epic #3994; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.

  12. noahgift commented on Oct 14, 2026

    @noahgift
    MemberAuthor

    Moved: this sub-ticket is now a row in #5332 (Release receipts: every verb receipted, version-exact receipt keys, a working install.sh), split out of #5312 by theme.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High priorityowner:la-0.71Owned by the 0.71 look-ahead slotsize:MK̂ 3-4 h

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions