Repository navigation
fix: a flickering read-back no longer convicts a write that landed - #20
Merged
Merged
Conversation
Three real write runs on an X570 board died reporting "read-back mismatch (chip 0x00, image XX)" at blocks 40, 28 and 2. The chip was fine and the blocks had been programmed correctly. The read-back lied. Always chip 0x00, never 0xff, which is the whole diagnosis: erased flash reads 0xff and a failed program leaves 0xff, so 0x00 was never the chip's content. It was the data lines with nothing driving them. ChunkTrust::LivenessOnly catches a dropout still present when the chunk ends, because the chip cannot answer RDID through a dead bus. It is blind to a flicker that starts and ends inside the chunk: the data lines go 0x00, the probe settles, the post-chunk id answers healthy, and the zeros are trusted. The write then compared those zeros to the image and called a good block a failed program, spending one of only WRITE_BLOCK_ATTEMPTS tries. Three unlucky flickers killed a write whose every block had landed, and the worse contact got the earlier it died: 40 -> 28 -> 2 was not a degrading chip, it was a read losing a coin-flip sooner. LivenessOnly's doc claimed a second read "would only buy a nicer error" because the bytes get compared against a known image anyway. That reasoning is what cost the board three dead runs. A mismatch is equally evidence of a bad read and of a bad write, and one look cannot tell them apart. So try_write_block now re-reads with DoubleRead before believing a mismatch, demanding the same evidence a backup does: two reads that agree. The happy path pays nothing. The fake could not model this dropout, which is why a mutation run blessed deleting the retry wait. dead_from latches contact_lost so the id goes silent too, which is precisely the case liveness DOES catch; every dropout the fake could express was one the code already handled. Adds flickering_zeros_at: zeros on the data lines with RDID answering normally. Also stops the write's read-back reporting progress. read_range reports against the length it was handed, so a 64 KB read-back printed "write: 100% (65536/65536 bytes)" once per block and buried the write's real 0->100% under 512 lines all claiming 100%. Reported from the bench as "confusing saying it is already 100%". Test asserts erase COUNT, since that is what separates the fix from the bug: both leave the right bytes on the chip, but only the broken one gets there by believing a lying read and re-erasing a correct block. Mutating the confirm read out gives left: 2, right: 1.
The confirm read added in the previous commit used ChunkTrust::DoubleRead, which demands two reads that agree WITH EACH OTHER. Through a flickering probe they never do, and read_chunk_patient's retry loop is silent while the chip still answers RDID, so it spun the entire patience budget with nothing on screen. With --patience 600 and three attempts that is a 30-minute silent hang per block. Reported from the bench as "stuck": the process was alive and burning 0.04s of CPU per 12 seconds of wall clock. A wrong-and-fast failure had been traded for a right-and-never. DoubleRead was reached for out of habit from the backup path, and it was never needed here. A backup demands two agreeing reads because it has nothing to check against and gets trusted forever after. A read-back has the image: 64 KB matching the target byte-for-byte cannot be a flicker, so matching is self-validating and a third read adds no evidence. One more look is the whole fix. Also give the retry loop a voice. It can retry for the whole patience budget without wait_for_contact ever printing, because that only speaks when the chip stops answering, and this branch is the chip answering perfectly while the chunk comes back wrong. Minutes of silence from a live process reads as a hang, which is exactly how it was reported.
A hard three-strike limit is the wrong shape for a marginal probe, and it outlived every other fix in this branch: real runs kept dying around block 30 whatever else was corrected (40, 28, 2, 29). The arithmetic is unforgiving. If a glitch costs one attempt in three, a block fails ~12% of the time, and 512 blocks at 12% is not a risk, it is a certainty. The read path learned this long ago and says so in its own comment: "Contact is the scarce resource; spend it on chunks, not on failing." It spends the whole --patience budget riding out flickers. The write never got the memo, so --patience 600 bought a block 60 milliseconds of retries. Retries are now bounded by the patience budget. WRITE_BLOCK_ATTEMPTS stays as the floor, so a caller asking for no patience still gets three tries. Safe to repeat for reasons already established: each attempt re-erases before re-programming, so a half-programmed block cannot AND stale zero bits into the result. Also give the retry a voice, and the loop yields its reason rather than carrying a mutable Option that clippy correctly flagged as never read. Test drives a block that needs five attempts (eight flickered read passes, two spent per attempt). Mutating the bound back to three strikes reproduces the exact bench failure: "3 attempts over 5s failed: read-back mismatch, still wrong on a second read".
jackulau
force-pushed
the
fix/readback-flicker-false-write-failure
branch
from
July 17, 2026 08:10
fbe51f6 to
12960dc
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three fixes, found by a real bricked X570 board that this branch's tool could not write. Each was uncovered only when the one before it stopped hiding it.
1. A flickering read-back convicted writes that had landed
Runs died reporting:
Always
chip 0x00, never0xff. That is the whole diagnosis: erased flash reads0xffand a failed program LEAVES0xff, so0x00was never the chip's content. It was the data lines with nothing driving them. The blocks had been programmed correctly and the read lied.ChunkTrust::LivenessOnlycatches a dropout STILL PRESENT when the chunk ends, because the chip cannot answer RDID through a dead bus. It is blind to a flicker that starts and ends INSIDE the chunk: the data lines go0x00, the probe settles, the post-chunk id answers healthy, and the zeros are trusted. Its doc claimed a second read "would only buy a nicer error" since the bytes get compared against a known image anyway. The error is not nicer, it is FALSE, and believing it spends one of only three tries.try_write_blocknow looks again before believing a mismatch.2. The confirm read used DoubleRead and hung
Fix 1 shipped with
ChunkTrust::DoubleRead, which demands two reads that agree WITH EACH OTHER. Through a flickering probe they never do, andread_chunk_patient's retry loop is silent while the chip still answers RDID, so it spun the entire patience budget with nothing on screen. With--patience 600and three attempts that is a 30-minute silent hang per block. Reported from the bench as "stuck": alive, burning 0.04s of CPU per 12 seconds of wall clock.A wrong-and-fast failure had been traded for a right-and-never.
DoubleRead was never needed. A backup demands two agreeing reads because it has nothing to check against and gets trusted forever. A read-back has the image: 64 KB matching the target byte-for-byte cannot be a flicker, so matching is self-validating. One more look is the whole fix.
Also gives that retry loop a voice. It could retry for the whole budget without
wait_for_contactever printing, because that only speaks when the chip stops answering, and this branch is the chip answering perfectly while the chunk comes back wrong.3. Three fixed strikes was the wrong shape
This one outlived the other two. Runs kept dying around block 30 whatever else was corrected: 40, 28, 2, 29. The arithmetic is unforgiving. If a glitch costs one attempt in three, a block fails ~12% of the time, and 512 blocks at 12% is not a risk, it is a certainty.
The read path learned this long ago and says so in its own comment:
It spends the whole
--patiencebudget riding out flickers. The write never got the memo, so--patience 600bought a block about 60 milliseconds of retries. Retries are now bounded by the patience budget, withWRITE_BLOCK_ATTEMPTSas the floor rather than the ceiling. Safe to repeat for reasons already established: each attempt re-erases before re-programming, so a half-programmed block cannot AND stale zero bits into the result.Why the suite blessed all of this
The fake could not model the dropout that mattered, which is why a mutation run had previously endorsed deleting the retry wait.
dead_fromLATCHEScontact_lost, so the id goes silent too, which is precisely the case liveness DOES catch. Every dropout the fake could express was one the code already handled.Adds
flickering_zeros_at: zeros on the data lines with RDID answering normally. Third time in this project that a fake kinder than physics hid a bug physics causes.Also
The write's read-back no longer reports progress.
read_rangereports against the length it was handed, so a 64 KB read-back printedwrite: 100% (65536/65536 bytes)once per block and buried the write's real 0 to 100% under 512 lines all claiming 100%. Reported from the bench as "confusing saying it is already 100%".Tests
Both assert through mutation, not vibes:
a_flickering_readback_must_not_convict_a_write_that_landedasserts erase COUNT, because that is what separates the fix from the bug: both leave the right bytes on the chip, but only the broken one gets there by believing a lying read and re-erasing a correct block. Mutating the confirm read out:left: 2, right: 1.a_block_that_needs_more_than_three_tries_still_lands_while_patience_remainsdrives a block needing five attempts. Mutating the bound back to three strikes reproduces the bench failure verbatim:3 attempts over 5s failed: read-back mismatch, still wrong on a second read.Outcome
The board this came from now writes clean:
cargo test --workspace: exit 0, 517 + 16 + 10 + 5 passed, 0 failedcargo clippy --workspace --all-targets -- -D warnings: exit 0