Skip to content

memguardcheck (B6b) is red on Linux only, across unrelated branches — and the arm cannot say why #462

Closed

Description

@pt-act

memguardcheck (B6b) is red on Linux and green on macOS — and it is failing across unrelated branches

Summary

test/memguardcheck.sh arm (B6b) fails only on Linux CI, on many unrelated branches and on
main itself, always with exactly one failure-shaped line:

=== memguardcheck.sh (rc=1, 14.4s) ===
    what failed (1 failure-shaped line(s)):
      L50:   FAIL  (B6b) id=3 _memory_stop=''

(B6) — the first answer under the crawl stop — passes. Only the second answer in the same
session loses the note. Both macOS shard-4/4 jobs pass the identical arm in the same run.

Where it is red

run commit context result
40639559139 aba4d28e (train 39) main plain, gcc shard 4/4 red
40671724898 12056c16 (train 40) main Release, clang shard 4/4 red
40693614920 0875bfcd lane/defs-lookup 2 Linux shard-4/4 jobs red
40720209993 72af5aab lane/git-porcelain-framing-v2 (PR #433) 4 of 4 Linux shard-4/4 jobs red, both macOS shard-4/4 green

Two facts worth leading with:

  • It is not deterministic per commit. aba4d28e failed at 05:42Z and passed at 07:23Z
    (run 40649757344) with no code change in between.
  • It is not confined to one branch. Four unrelated branches hit the identical arm today, which is
    what makes this look like a repo-level problem rather than anyone's change.

In every failing job the shard's own summary is gates=185 pass=182 skip=2 fail=1 — i.e. this arm is
the only red in the job.

What I ruled out (so nobody has to repeat it)

Measured on macOS 14.8.7, Intel Core i5-8257U (x86_64, 8 logical cores), plain build, on
72af5aab. The architecture is worth noting: this is x86_64, the same as the Linux CI runner, so the
negative results below are not explained away by "a different CPU" — only by the OS and the runner's
core count.

  1. Not the lane that reported it. PR gitmine: a committed filename can no longer be read as commit metadata (git log framing) #433 touches src/gitmine.h, src/infra/stdinline.h and
    test/doc files only; git log b7c56cf5..cab28607 -- src/memguard.h src/mcp.h test/memguardcheck.sh
    is empty.
  2. Not the recent ci-flakes work. 360e895e and 4662d6c3 change only lines ≥ 740 of the gate
    (B12's new $EG12 fixture). (B6) is at line 375 and its $FX fixture is untouched.
  3. Not the kqueue/no-watcher asymmetry — surprisingly. src/mcpindex.h:45 documents that
    -DRW_OS_HAS_KQUEUE=0 compiles the Linux (no-watcher) path on a Mac, so I built exactly that and
    ran the real gate against it. (B6)/(B6b)/(B6c) all PASS. So "Linux has no watcher and
    therefore always sweeps" is not sufficient to trigger it. (The build did reproduce the known
    macOS-host-trait B8/B8b failures, so the binary was genuinely the fallback path.)
  4. Not simple rebuild-between-requests. Driving the two requests by hand with the real $FX
    fixture and touching a file between them does change _index on the second answer
    (hash=8d545c53 → aa5ef2b0, _reingest=0) — and _memory_stop is still present, because the
    re-ingest trips the crawl:30 seam again (the seam counts per ingest).
  5. Not CPU load. Six concurrent memguardcheck.sh runs on 8 logical cores: (B6b) PASS 6/6.
    That is 6 workers against 8 cores; the CI leg is -j 3 on 4 vCPU, so this is at least as contended.
  6. Not the background prefetch. maybePrefetchHeadSnapshot returns early for this fixture —
    fileCount (72) is under mcpPrefetchMinFiles(), and $FX is not a git repo, so
    gitHeadMoveToken is 0.

So on this side I could not reproduce it at all: the arm passes locally 3/3 idle, 6/6 under load, on
both the watcher and the no-watcher build. That is itself the useful signal — the trigger is
something about the Linux runner that I cannot reproduce, and the arm currently reports only the
symptom.

Relevant code

  • src/mcp.h:1636 — the note is derived from the resident index:
    memoryStopNote = mix.ing.memoryStop.isSet() ? memguard::softStopLine( mix.ing ) : std::string();
  • src/mcp.h:1657 — the fallback only fires when a stop was recorded during this call
    (stopCounts().recorded > stopsAtEntry). If the index was replaced by one built earlier, the
    fallback is blind: the stop was counted before stopsAtEntry was read.
  • src/memguard.h:53-60 — crawl:N "counts per INGEST (each ingest's own Watch)", so a plain
    re-crawl re-trips and keeps the note (confirmed in experiment 4).
  • src/mcpindex.h:257 — on a no-watcher build getIndex() always sweeps; mcpStale() compares
    dir mtimes plus per-file mtime/size/ctime.

Given (4), the note can only go missing if the answering index was produced by an ingest that did
not reach crawl entry 30 — e.g. one served from the ingest cache, or a rebuild over a smaller
crawl set. I have not proven which.

Proposed fix

Two parts, in the order I'd take them.

1. Make the arm report the distinction it currently hides (this is the bug in the gate)

(B6b) is named "the next answer from the same partial index still carries it" — but it never
checks that the index is the same. It compares a string against a wildcard and prints only
_memory_stop='', which cannot distinguish "the note was lost" (a real defect) from "the index was
replaced between the two requests" (legitimate, and outside the arm's stated premise). The envelope
already carries everything needed — _index is a sibling of content on both answers.

Minimal, in the house style of B12's attempt loop:

# (B6b) the next answer from the same partial index still carries it — the premise is checked, not assumed
b6b="" b6bix=""
for attempt in 1 2 3 4 5 6; do
    ix2="$( mcp_field "$TMP/b6.out" 2 _index )"; ix3="$( mcp_field "$TMP/b6.out" 3 _index )"
    ms3="$( mcp_field "$TMP/b6.out" 3 _memory_stop )"
    case "$ms3" in *"memory guard"*) b6b="ok"; break;; esac
    if [ "$ix2" != "$ix3" ]; then
        # the second answer did not come from the same index: re-drive, the premise is what failed
        b6bix="index moved between the answers ($ix2 -> $ix3)"
        <re-send the two requests into $TMP/b6.out>
        continue
    fi
    b6b="lost"; break;      # same index, no note: a REAL defect, and now it says so
done
case "$b6b" in
    ok)  ok "(B6b) the next answer from the same partial index still carries it";;
    lost) no "(B6b) id=3 _memory_stop='' with an UNCHANGED index $ix2 — the note was dropped";;
    *)   no "(B6b) never observed two answers from one partial index: ${b6bix:-no note in 6 attempts}";;
esac

This turns a one-line symptom into a triage decision, and if the answer really is "the index moved",
that is a different issue (why does a request-time rebuild drop the crawl stop on Linux?) with a
reproduction attached.

2. Decide which invariant the server owes

The honest-output rule in AGENTS.md says every truncation is disclosed. Stated as a property of the
answer, independent of index identity:

If the index that answered was cut by the memory guard, the envelope must carry _memory_stop.

If (1) shows the note survives on an unchanged index, then the arm is merely over-specified and (1) is
the whole fix. If (1) shows the index moves and the replacement is itself partial while carrying no
note, that is a real disclosure bug and the fix belongs in src/mcp.h — key the note off the index
being partial rather than only off a stored memoryStop, so a partial index can never answer without
saying so.

I have deliberately not sent a PR: the right patch depends on which of those two the diagnostic shows,
and I cannot see it from a macOS box. If you can re-run shard 4/4 with the diagnostic above, the next
step is decided by its output.

Why this matters beyond one arm

memguardcheck is in shard 4/4, so this single arm is currently able to redden an otherwise-clean PR
on every Linux leg — 4 of 4 Linux jobs on 72af5aab, while the same code is green on both macOS legs.
That is a lot of CI signal destroyed by one non-reproducible assertion.

Activity

  1. pt-act commented on Oct 29, 2026

    @pt-act
    ContributorAuthor

    Root cause found and fixed upstream — closing as completed, with thanks.

    The mechanism is the gate's own fixture meeting filesystem readdir order, not index identity:

    • RIPWIRE_TEST_MEMGUARD=crawl:30 stops the crawl at entry 30 before the sort — recursive_directory_iterator order is the filesystem's readdir order, so which fixture files survive the cut varies by filesystem. APFS keeps a/f02.c; ext4's hashed order can drop it (the simulation in the fixing commit keeps it ~37% of random orders).
    • When a/f02.c is outside the kept set, this arm's second call — find_symbol a_02 — returns a JSON-RPC not-found ERROR, which has no result._memory_stop at all, so the arm read ''.
    • That explains everything this issue observed from the outside: Linux-only, plausibly the same-sha red-then-green, and never reproducible on an APFS host.

    The fix (aa13b254, landed in train-41 #463, with ba155d13 hardening B21 in the same sweep):

    • (B6b)'s second call is now grep — a result on any partial index, an empty match included — so the premise "a later answer from the same partial index keeps the disclosure" is now checked on every filesystem, not just the ones whose readdir order cooperates.
    • find_symbol a_02 stays as a third probe, asserted on the shape it actually returned, with a NOTE line when it fell outside the kept files — the arm can now say why, which is the gap this issue reported.
    • pargatescheck's (I) control (0.5 s → 3.0 s) and memguardcheck (B21) (a seconds cap → asking whether the stream finished) were load-hardened in the same commit.

    For the record: aa13b254's author date is 2026-10-28 18:19Z, about two hours before this issue was filed (20:35Z) — the diagnosis converged independently on both sides. The index-identity analysis in this issue's mechanism section (mcp.h:1636, a whole-index replacement between requests) is superseded by the above. The six negative results stand, and the APFS-keeps-the-file fact is why the arm never reproduced on the machine that produced them.

    The disclosure property this issue proposed as the tie-breaker can now be stated precisely, and it holds without a code change: a result from a guard-cut index carries _memory_stop — pinned by (B6)/(B6b) in their grep form — while a not-found error carries no envelope and is not the shape the arms pin. No gap remains.

    Housekeeping note for readers arriving from PR #433: its Linux shard-4/4 reds are this same arm on the pre-fix gate (head 72af5aab, base train-40) and clear when that lane absorbs train-41.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions