Skip to content

fix: a flickering read-back no longer convicts a write that landed - #20

Merged
jackulau merged 3 commits into
mainfrom
fix/readback-flicker-false-write-failure
Jul 17, 2026
Merged

jackulau merged 3 commits into
mainfrom
fix/readback-flicker-false-write-failure

Conversation

@jackulau

@jackulau jackulau commented Jul 17, 2026 •

Copy link
Copy Markdown
Owner

Three fixes, found by a real bricked X570 board that this branch's tool could not write. Each was uncovered only when the one before it stopped hiding it.

1. A flickering read-back convicted writes that had landed

Runs died reporting:

read-back mismatch at 0x280000 (chip 0x00, image 0xef)
read-back mismatch at 0x1c0000 (chip 0x00, image 0x40)
read-back mismatch at 0x20000  (chip 0x00, image 0xaa)

Always chip 0x00, never 0xff. That is the whole diagnosis: erased flash reads 0xff and a failed program LEAVES 0xff, so 0x00 was never the chip's content. It was the data lines with nothing driving them. The blocks had been programmed correctly and the read lied.

ChunkTrust::LivenessOnly catches a dropout STILL PRESENT when the chunk ends, because the chip cannot answer RDID through a dead bus. It is blind to a flicker that starts and ends INSIDE the chunk: the data lines go 0x00, the probe settles, the post-chunk id answers healthy, and the zeros are trusted. Its doc claimed a second read "would only buy a nicer error" since the bytes get compared against a known image anyway. The error is not nicer, it is FALSE, and believing it spends one of only three tries.

try_write_block now looks again before believing a mismatch.

2. The confirm read used DoubleRead and hung

Fix 1 shipped with ChunkTrust::DoubleRead, which demands two reads that agree WITH EACH OTHER. Through a flickering probe they never do, and read_chunk_patient's retry loop is silent while the chip still answers RDID, so it spun the entire patience budget with nothing on screen. With --patience 600 and three attempts that is a 30-minute silent hang per block. Reported from the bench as "stuck": alive, burning 0.04s of CPU per 12 seconds of wall clock.

A wrong-and-fast failure had been traded for a right-and-never.

DoubleRead was never needed. A backup demands two agreeing reads because it has nothing to check against and gets trusted forever. A read-back has the image: 64 KB matching the target byte-for-byte cannot be a flicker, so matching is self-validating. One more look is the whole fix.

Also gives that retry loop a voice. It could retry for the whole budget without wait_for_contact ever printing, because that only speaks when the chip stops answering, and this branch is the chip answering perfectly while the chunk comes back wrong.

3. Three fixed strikes was the wrong shape

This one outlived the other two. Runs kept dying around block 30 whatever else was corrected: 40, 28, 2, 29. The arithmetic is unforgiving. If a glitch costs one attempt in three, a block fails ~12% of the time, and 512 blocks at 12% is not a risk, it is a certainty.

The read path learned this long ago and says so in its own comment:

"Contact is the scarce resource; spend it on chunks, not on failing."

It spends the whole --patience budget riding out flickers. The write never got the memo, so --patience 600 bought a block about 60 milliseconds of retries. Retries are now bounded by the patience budget, with WRITE_BLOCK_ATTEMPTS as the floor rather than the ceiling. Safe to repeat for reasons already established: each attempt re-erases before re-programming, so a half-programmed block cannot AND stale zero bits into the result.

Why the suite blessed all of this

The fake could not model the dropout that mattered, which is why a mutation run had previously endorsed deleting the retry wait. dead_from LATCHES contact_lost, so the id goes silent too, which is precisely the case liveness DOES catch. Every dropout the fake could express was one the code already handled.

Adds flickering_zeros_at: zeros on the data lines with RDID answering normally. Third time in this project that a fake kinder than physics hid a bug physics causes.

Also

The write's read-back no longer reports progress. read_range reports against the length it was handed, so a 64 KB read-back printed write: 100% (65536/65536 bytes) once per block and buried the write's real 0 to 100% under 512 lines all claiming 100%. Reported from the bench as "confusing saying it is already 100%".

Tests

Both assert through mutation, not vibes:

  • a_flickering_readback_must_not_convict_a_write_that_landed asserts erase COUNT, because that is what separates the fix from the bug: both leave the right bytes on the chip, but only the broken one gets there by believing a lying read and re-erasing a correct block. Mutating the confirm read out: left: 2, right: 1.
  • a_block_that_needs_more_than_three_tries_still_lands_while_patience_remains drives a block needing five attempts. Mutating the bound back to three strikes reproduces the bench failure verbatim: 3 attempts over 5s failed: read-back mismatch, still wrong on a second read.

Outcome

The board this came from now writes clean:

write: 100% (33554432/33554432 bytes, 31 KiB/s)
write success=true verified=true backup=None backend=ch341a
  • cargo test --workspace: exit 0, 517 + 16 + 10 + 5 passed, 0 failed
  • cargo clippy --workspace --all-targets -- -D warnings: exit 0

jackulau added 3 commits July 17, 2026 03:09
Three real write runs on an X570 board died reporting "read-back mismatch
(chip 0x00, image XX)" at blocks 40, 28 and 2. The chip was fine and the
blocks had been programmed correctly. The read-back lied.

Always chip 0x00, never 0xff, which is the whole diagnosis: erased flash
reads 0xff and a failed program leaves 0xff, so 0x00 was never the chip's
content. It was the data lines with nothing driving them.

ChunkTrust::LivenessOnly catches a dropout still present when the chunk
ends, because the chip cannot answer RDID through a dead bus. It is blind
to a flicker that starts and ends inside the chunk: the data lines go 0x00,
the probe settles, the post-chunk id answers healthy, and the zeros are
trusted. The write then compared those zeros to the image and called a good
block a failed program, spending one of only WRITE_BLOCK_ATTEMPTS tries.
Three unlucky flickers killed a write whose every block had landed, and the
worse contact got the earlier it died: 40 -> 28 -> 2 was not a degrading
chip, it was a read losing a coin-flip sooner.

LivenessOnly's doc claimed a second read "would only buy a nicer error"
because the bytes get compared against a known image anyway. That reasoning
is what cost the board three dead runs. A mismatch is equally evidence of a
bad read and of a bad write, and one look cannot tell them apart. So
try_write_block now re-reads with DoubleRead before believing a mismatch,
demanding the same evidence a backup does: two reads that agree. The happy
path pays nothing.

The fake could not model this dropout, which is why a mutation run blessed
deleting the retry wait. dead_from latches contact_lost so the id goes
silent too, which is precisely the case liveness DOES catch; every dropout
the fake could express was one the code already handled. Adds
flickering_zeros_at: zeros on the data lines with RDID answering normally.

Also stops the write's read-back reporting progress. read_range reports
against the length it was handed, so a 64 KB read-back printed
"write: 100% (65536/65536 bytes)" once per block and buried the write's
real 0->100% under 512 lines all claiming 100%. Reported from the bench as
"confusing saying it is already 100%".

Test asserts erase COUNT, since that is what separates the fix from the
bug: both leave the right bytes on the chip, but only the broken one gets
there by believing a lying read and re-erasing a correct block. Mutating
the confirm read out gives left: 2, right: 1.
The confirm read added in the previous commit used ChunkTrust::DoubleRead,
which demands two reads that agree WITH EACH OTHER. Through a flickering
probe they never do, and read_chunk_patient's retry loop is silent while
the chip still answers RDID, so it spun the entire patience budget with
nothing on screen. With --patience 600 and three attempts that is a
30-minute silent hang per block. Reported from the bench as "stuck": the
process was alive and burning 0.04s of CPU per 12 seconds of wall clock.

A wrong-and-fast failure had been traded for a right-and-never.

DoubleRead was reached for out of habit from the backup path, and it was
never needed here. A backup demands two agreeing reads because it has
nothing to check against and gets trusted forever after. A read-back has
the image: 64 KB matching the target byte-for-byte cannot be a flicker, so
matching is self-validating and a third read adds no evidence. One more
look is the whole fix.

Also give the retry loop a voice. It can retry for the whole patience
budget without wait_for_contact ever printing, because that only speaks
when the chip stops answering, and this branch is the chip answering
perfectly while the chunk comes back wrong. Minutes of silence from a live
process reads as a hang, which is exactly how it was reported.
A hard three-strike limit is the wrong shape for a marginal probe, and it
outlived every other fix in this branch: real runs kept dying around block
30 whatever else was corrected (40, 28, 2, 29). The arithmetic is
unforgiving. If a glitch costs one attempt in three, a block fails ~12% of
the time, and 512 blocks at 12% is not a risk, it is a certainty.

The read path learned this long ago and says so in its own comment:
"Contact is the scarce resource; spend it on chunks, not on failing." It
spends the whole --patience budget riding out flickers. The write never got
the memo, so --patience 600 bought a block 60 milliseconds of retries.

Retries are now bounded by the patience budget. WRITE_BLOCK_ATTEMPTS stays
as the floor, so a caller asking for no patience still gets three tries.
Safe to repeat for reasons already established: each attempt re-erases
before re-programming, so a half-programmed block cannot AND stale zero bits
into the result.

Also give the retry a voice, and the loop yields its reason rather than
carrying a mutable Option that clippy correctly flagged as never read.

Test drives a block that needs five attempts (eight flickered read passes,
two spent per attempt). Mutating the bound back to three strikes reproduces
the exact bench failure: "3 attempts over 5s failed: read-back mismatch,
still wrong on a second read".
@jackulau
jackulau force-pushed the fix/readback-flicker-false-write-failure branch from fbe51f6 to 12960dc Compare July 17, 2026 08:10
@jackulau
jackulau merged commit 8bdfebd into main Jul 17, 2026
1 check passed
@jackulau
jackulau deleted the fix/readback-flicker-false-write-failure branch July 17, 2026 08:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant