Repository navigation
memguardcheck (B6b) is red on Linux only, across unrelated branches — and the arm cannot say why #462
Description
memguardcheck (B6b) is red on Linux and green on macOS — and it is failing across unrelated branches
Summary
test/memguardcheck.sh arm (B6b) fails only on Linux CI, on many unrelated branches and on
main itself, always with exactly one failure-shaped line:
=== memguardcheck.sh (rc=1, 14.4s) ===
what failed (1 failure-shaped line(s)):
L50: FAIL (B6b) id=3 _memory_stop=''
(B6) — the first answer under the crawl stop — passes. Only the second answer in the same
session loses the note. Both macOS shard-4/4 jobs pass the identical arm in the same run.
Where it is red
| run | commit | context | result |
|---|---|---|---|
| 40639559139 | aba4d28e (train 39) |
main |
plain, gcc shard 4/4 red |
| 40671724898 | 12056c16 (train 40) |
main |
Release, clang shard 4/4 red |
| 40693614920 | 0875bfcd |
lane/defs-lookup |
2 Linux shard-4/4 jobs red |
| 40720209993 | 72af5aab |
lane/git-porcelain-framing-v2 (PR #433) |
4 of 4 Linux shard-4/4 jobs red, both macOS shard-4/4 green |
Two facts worth leading with:
- It is not deterministic per commit.
aba4d28efailed at 05:42Z and passed at 07:23Z
(run40649757344) with no code change in between. - It is not confined to one branch. Four unrelated branches hit the identical arm today, which is
what makes this look like a repo-level problem rather than anyone's change.
In every failing job the shard's own summary is gates=185 pass=182 skip=2 fail=1 — i.e. this arm is
the only red in the job.
What I ruled out (so nobody has to repeat it)
Measured on macOS 14.8.7, Intel Core i5-8257U (x86_64, 8 logical cores), plain build, on
72af5aab. The architecture is worth noting: this is x86_64, the same as the Linux CI runner, so the
negative results below are not explained away by "a different CPU" — only by the OS and the runner's
core count.
- Not the lane that reported it. PR gitmine: a committed filename can no longer be read as commit metadata (
git logframing) #433 touchessrc/gitmine.h,src/infra/stdinline.hand
test/doc files only;git log b7c56cf5..cab28607 -- src/memguard.h src/mcp.h test/memguardcheck.sh
is empty. - Not the recent
ci-flakeswork.360e895eand4662d6c3change only lines ≥ 740 of the gate
(B12's new$EG12fixture).(B6)is at line 375 and its$FXfixture is untouched. - Not the kqueue/no-watcher asymmetry — surprisingly.
src/mcpindex.h:45documents that
-DRW_OS_HAS_KQUEUE=0compiles the Linux (no-watcher) path on a Mac, so I built exactly that and
ran the real gate against it.(B6)/(B6b)/(B6c)all PASS. So "Linux has no watcher and
therefore always sweeps" is not sufficient to trigger it. (The build did reproduce the known
macOS-host-traitB8/B8bfailures, so the binary was genuinely the fallback path.) - Not simple rebuild-between-requests. Driving the two requests by hand with the real
$FX
fixture andtouching a file between them does change_indexon the second answer
(hash=8d545c53→aa5ef2b0,_reingest=0) — and_memory_stopis still present, because the
re-ingest trips thecrawl:30seam again (the seam counts per ingest). - Not CPU load. Six concurrent
memguardcheck.shruns on 8 logical cores:(B6b)PASS 6/6.
That is 6 workers against 8 cores; the CI leg is-j 3on 4 vCPU, so this is at least as contended. - Not the background prefetch.
maybePrefetchHeadSnapshotreturns early for this fixture —
fileCount(72) is undermcpPrefetchMinFiles(), and$FXis not a git repo, so
gitHeadMoveTokenis 0.
So on this side I could not reproduce it at all: the arm passes locally 3/3 idle, 6/6 under load, on
both the watcher and the no-watcher build. That is itself the useful signal — the trigger is
something about the Linux runner that I cannot reproduce, and the arm currently reports only the
symptom.
Relevant code
src/mcp.h:1636— the note is derived from the resident index:
memoryStopNote = mix.ing.memoryStop.isSet() ? memguard::softStopLine( mix.ing ) : std::string();src/mcp.h:1657— the fallback only fires when a stop was recorded during this call
(stopCounts().recorded > stopsAtEntry). If the index was replaced by one built earlier, the
fallback is blind: the stop was counted beforestopsAtEntrywas read.src/memguard.h:53-60—crawl:N"counts per INGEST (each ingest's own Watch)", so a plain
re-crawl re-trips and keeps the note (confirmed in experiment 4).src/mcpindex.h:257— on a no-watcher buildgetIndex()always sweeps;mcpStale()compares
dir mtimes plus per-file mtime/size/ctime.
Given (4), the note can only go missing if the answering index was produced by an ingest that did
not reach crawl entry 30 — e.g. one served from the ingest cache, or a rebuild over a smaller
crawl set. I have not proven which.
Proposed fix
Two parts, in the order I'd take them.
1. Make the arm report the distinction it currently hides (this is the bug in the gate)
(B6b) is named "the next answer from the same partial index still carries it" — but it never
checks that the index is the same. It compares a string against a wildcard and prints only
_memory_stop='', which cannot distinguish "the note was lost" (a real defect) from "the index was
replaced between the two requests" (legitimate, and outside the arm's stated premise). The envelope
already carries everything needed — _index is a sibling of content on both answers.
Minimal, in the house style of B12's attempt loop:
# (B6b) the next answer from the same partial index still carries it — the premise is checked, not assumed
b6b="" b6bix=""
for attempt in 1 2 3 4 5 6; do
ix2="$( mcp_field "$TMP/b6.out" 2 _index )"; ix3="$( mcp_field "$TMP/b6.out" 3 _index )"
ms3="$( mcp_field "$TMP/b6.out" 3 _memory_stop )"
case "$ms3" in *"memory guard"*) b6b="ok"; break;; esac
if [ "$ix2" != "$ix3" ]; then
# the second answer did not come from the same index: re-drive, the premise is what failed
b6bix="index moved between the answers ($ix2 -> $ix3)"
<re-send the two requests into $TMP/b6.out>
continue
fi
b6b="lost"; break; # same index, no note: a REAL defect, and now it says so
done
case "$b6b" in
ok) ok "(B6b) the next answer from the same partial index still carries it";;
lost) no "(B6b) id=3 _memory_stop='' with an UNCHANGED index $ix2 — the note was dropped";;
*) no "(B6b) never observed two answers from one partial index: ${b6bix:-no note in 6 attempts}";;
esacThis turns a one-line symptom into a triage decision, and if the answer really is "the index moved",
that is a different issue (why does a request-time rebuild drop the crawl stop on Linux?) with a
reproduction attached.
2. Decide which invariant the server owes
The honest-output rule in AGENTS.md says every truncation is disclosed. Stated as a property of the
answer, independent of index identity:
If the index that answered was cut by the memory guard, the envelope must carry
_memory_stop.
If (1) shows the note survives on an unchanged index, then the arm is merely over-specified and (1) is
the whole fix. If (1) shows the index moves and the replacement is itself partial while carrying no
note, that is a real disclosure bug and the fix belongs in src/mcp.h — key the note off the index
being partial rather than only off a stored memoryStop, so a partial index can never answer without
saying so.
I have deliberately not sent a PR: the right patch depends on which of those two the diagnostic shows,
and I cannot see it from a macOS box. If you can re-run shard 4/4 with the diagnostic above, the next
step is decided by its output.
Why this matters beyond one arm
memguardcheck is in shard 4/4, so this single arm is currently able to redden an otherwise-clean PR
on every Linux leg — 4 of 4 Linux jobs on 72af5aab, while the same code is green on both macOS legs.
That is a lot of CI signal destroyed by one non-reproducible assertion.
Activity
pt-act commentedon Oct 29, 2026
Root cause found and fixed upstream — closing as completed, with thanks.
The mechanism is the gate's own fixture meeting filesystem readdir order, not index identity:
RIPWIRE_TEST_MEMGUARD=crawl:30stops the crawl at entry 30 before the sort —recursive_directory_iteratororder is the filesystem'sreaddirorder, so which fixture files survive the cut varies by filesystem. APFS keepsa/f02.c; ext4's hashed order can drop it (the simulation in the fixing commit keeps it ~37% of random orders).a/f02.cis outside the kept set, this arm's second call —find_symbol a_02— returns a JSON-RPC not-found ERROR, which has noresult._memory_stopat all, so the arm read''.The fix (
aa13b254, landed in train-41 #463, withba155d13hardening B21 in the same sweep):grep— a result on any partial index, an empty match included — so the premise "a later answer from the same partial index keeps the disclosure" is now checked on every filesystem, not just the ones whose readdir order cooperates.find_symbol a_02stays as a third probe, asserted on the shape it actually returned, with a NOTE line when it fell outside the kept files — the arm can now say why, which is the gap this issue reported.pargatescheck's (I) control (0.5 s → 3.0 s) andmemguardcheck(B21) (a seconds cap → asking whether the stream finished) were load-hardened in the same commit.For the record:
aa13b254's author date is 2026-10-28 18:19Z, about two hours before this issue was filed (20:35Z) — the diagnosis converged independently on both sides. The index-identity analysis in this issue's mechanism section (mcp.h:1636, a whole-index replacement between requests) is superseded by the above. The six negative results stand, and the APFS-keeps-the-file fact is why the arm never reproduced on the machine that produced them.The disclosure property this issue proposed as the tie-breaker can now be stated precisely, and it holds without a code change: a result from a guard-cut index carries
_memory_stop— pinned by (B6)/(B6b) in their grep form — while a not-found error carries no envelope and is not the shape the arms pin. No gap remains.Housekeeping note for readers arriving from PR #433: its Linux shard-4/4 reds are this same arm on the pre-fix gate (head
72af5aab, base train-40) and clear when that lane absorbs train-41.