Repository navigation
fix: reads ride out a flickering probe instead of dying on it - #16
Merged
Merged
Conversation
A marginal probe on a leadless package does not fail cleanly, it flickers: contact drops and comes back seconds later. Reads treated the first flicker as fatal (3 attempts ~20ms apart, then give up), so a 32MB dump died at 7% and the user still has no backup. Measured contact on the real board is 1-in-32 samples, so "wait for minutes of unbroken contact" is not a plan. Wait for contact instead of failing on it. When the probe is not touching, RDID is a SUCCESSFUL USB transfer reporting 000000: a reading, not an error. Nothing retries, nothing times out, no transfer is left in flight to wedge the host controller. So polling until the chip answers costs patience and nothing else, and a read now grinds forward through whatever contact windows it gets. Budget is per chunk, default 120s, `ratchet read --patience N`. Also read every backup chunk twice and require the two to agree. The liveness check alone cannot see a probe that corrupts data mid-chunk while the chip keeps answering RDID: the id is healthy before and after, so the chunk banks as done and the garbage reaches disk inside a backup the user trusts. This is the same evidence as the project's two-full-reads gate, every byte read twice and matched, but per 64KB instead of per 32MB: matching two whole dumps needs minutes of unbroken contact twice over, matching one chunk needs a second. Verify and write read-back stay single-read (they compare against a known image, so a bad read there already fails closed as a mismatch). Persist each chunk as it lands, not only when the read fails. A 9 minute read on a bad probe looks exactly like a hang, users kill it, and there is no error path to hang the save off. Bytes are written before the sidecar records them, so a kill in the gap costs a re-read of one chunk rather than leaving a sidecar entry for filler that a resume would skip. Fixes a hot-spin found while reviewing this: a chunk can fail while RDID answers perfectly (two reads disagreeing is exactly that), and wait_for_contact then returns instantly, so the retry loop re-read at full tilt for the whole patience budget. Hammering a marginal bus is the one response guaranteed not to help. Every retry now paces itself. Tests, each mutation-checked to confirm it discriminates: - read_waits_out_a_flickering_probe_and_still_gets_the_whole_chip - read_rejects_a_chunk_that_reads_differently_twice (with liveness-only trust this passes a corrupt dump to disk and reports success: backup-1's failure) - read_persists_each_chunk_as_it_lands_not_only_when_it_fails Unit tests default to a near-zero patience: a fake bus has no probe that might flicker back, so waiting is dead test time, and four dead-bus tests were each sitting through the full production budget (module went 120s to 7s).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
A marginal probe on a leadless package does not fail cleanly, it flickers: contact drops and comes back seconds later. Reads treated the first flicker as fatal (3 attempts ~20ms apart, then give up), so a 32MB dump died at 7% and there is still no backup of the board.
Measured contact on the real hardware tonight was 1 good sample in 32. Against that, "hold still for 4.5 minutes" is not a plan.
Wait for contact instead of failing on it
When the probe is not touching, RDID is a successful USB transfer that reports
000000. It is a reading, not an error: nothing retries, nothing times out, no transfer is left in flight to wedge the host controller. So polling until the chip answers costs patience and nothing else, and patience is the one resource we have plenty of.A read now grinds forward through whatever contact windows it gets. Budget is per chunk, default 120s,
ratchet read --patience Nto change it.Read each backup chunk twice
The liveness check alone cannot see a probe that corrupts data mid-chunk while the chip keeps answering RDID: the id is healthy before and after, so the chunk banks as done and the garbage reaches disk inside a backup the user trusts.
Two reads that must agree is the same evidence as this project's two-full-reads gate, every byte read twice and matched, but per 64KB instead of per 32MB. Matching two whole dumps needs minutes of unbroken contact twice over; matching one chunk needs about a second. Same guarantee, reachable with the grip that actually exists.
Verify and write read-back stay single-read: they compare against a known image, so a bad read there already fails closed as a mismatch.
This does not replace the two-matching-backups gate. It makes one read trustworthy; two still adds independent confirmation, and resume makes the second one cheap.
Persist each chunk as it lands
A 9 minute read on a bad probe looks exactly like a hang, so users kill it, and there was no error path to hang the save off. Bytes are written before the sidecar records them: a kill in the gap costs a re-read of one chunk, rather than leaving a sidecar entry for filler that a resume would skip and hand back inside a "complete" backup.
A bug found while reviewing this
A chunk can fail while RDID answers perfectly (two reads disagreeing is exactly that).
wait_for_contactthen returns instantly, so the retry loop re-read at full tilt for the entire patience budget. Hammering a marginal bus is the one response guaranteed not to help. Every retry now paces itself.Verification
cargo test --workspace500 pass,clippy --workspace --all-targets -D warningsclean,fmt --checkclean.Each new test was mutation-checked to confirm it discriminates:
DoubleReadtoLivenessOnly)set_chunk_patienceKnown gap, stated rather than papered over: the write-then-record ordering is load-bearing but not test-enforced, because catching it needs the process killed between two syscalls. The reason is documented at the call site.
Unit tests now default to a near-zero patience: a fake bus has no probe that might flicker back, so waiting is dead test time. Four dead-bus tests were each sitting through the full production budget; the module went from 120s to 7s.