Skip to content

fix: reads ride out a flickering probe instead of dying on it - #16

Merged
jackulau merged 1 commit into
mainfrom
read-rides-contact-windows
Jul 17, 2026
Merged

jackulau merged 1 commit into
mainfrom
read-rides-contact-windows

Conversation

@jackulau

Copy link
Copy Markdown
Owner

The problem

A marginal probe on a leadless package does not fail cleanly, it flickers: contact drops and comes back seconds later. Reads treated the first flicker as fatal (3 attempts ~20ms apart, then give up), so a 32MB dump died at 7% and there is still no backup of the board.

Measured contact on the real hardware tonight was 1 good sample in 32. Against that, "hold still for 4.5 minutes" is not a plan.

Wait for contact instead of failing on it

When the probe is not touching, RDID is a successful USB transfer that reports 000000. It is a reading, not an error: nothing retries, nothing times out, no transfer is left in flight to wedge the host controller. So polling until the chip answers costs patience and nothing else, and patience is the one resource we have plenty of.

A read now grinds forward through whatever contact windows it gets. Budget is per chunk, default 120s, ratchet read --patience N to change it.

Read each backup chunk twice

The liveness check alone cannot see a probe that corrupts data mid-chunk while the chip keeps answering RDID: the id is healthy before and after, so the chunk banks as done and the garbage reaches disk inside a backup the user trusts.

Two reads that must agree is the same evidence as this project's two-full-reads gate, every byte read twice and matched, but per 64KB instead of per 32MB. Matching two whole dumps needs minutes of unbroken contact twice over; matching one chunk needs about a second. Same guarantee, reachable with the grip that actually exists.

Verify and write read-back stay single-read: they compare against a known image, so a bad read there already fails closed as a mismatch.

This does not replace the two-matching-backups gate. It makes one read trustworthy; two still adds independent confirmation, and resume makes the second one cheap.

Persist each chunk as it lands

A 9 minute read on a bad probe looks exactly like a hang, so users kill it, and there was no error path to hang the save off. Bytes are written before the sidecar records them: a kill in the gap costs a re-read of one chunk, rather than leaving a sidecar entry for filler that a resume would skip and hand back inside a "complete" backup.

A bug found while reviewing this

A chunk can fail while RDID answers perfectly (two reads disagreeing is exactly that). wait_for_contact then returns instantly, so the retry loop re-read at full tilt for the entire patience budget. Hammering a marginal bus is the one response guaranteed not to help. Every retry now paces itself.

Verification

cargo test --workspace 500 pass, clippy --workspace --all-targets -D warnings clean, fmt --check clean.

Each new test was mutation-checked to confirm it discriminates:

Mutation Result
backup read trusts a single pass (DoubleRead to LivenessOnly) corrupt-chunk test FAILS (and the corrupt dump reaches disk reporting success: backup-1's exact failure)
never wait for contact to return flicker test FAILS
save only on failure persistence test FAILS
drop set_chunk_patience flicker test FAILS
mark chunk done before the bytes land passed, not covered (see below)

Known gap, stated rather than papered over: the write-then-record ordering is load-bearing but not test-enforced, because catching it needs the process killed between two syscalls. The reason is documented at the call site.

Unit tests now default to a near-zero patience: a fake bus has no probe that might flicker back, so waiting is dead test time. Four dead-bus tests were each sitting through the full production budget; the module went from 120s to 7s.

A marginal probe on a leadless package does not fail cleanly, it flickers:
contact drops and comes back seconds later. Reads treated the first flicker
as fatal (3 attempts ~20ms apart, then give up), so a 32MB dump died at 7%
and the user still has no backup. Measured contact on the real board is
1-in-32 samples, so "wait for minutes of unbroken contact" is not a plan.

Wait for contact instead of failing on it. When the probe is not touching,
RDID is a SUCCESSFUL USB transfer reporting 000000: a reading, not an error.
Nothing retries, nothing times out, no transfer is left in flight to wedge
the host controller. So polling until the chip answers costs patience and
nothing else, and a read now grinds forward through whatever contact windows
it gets. Budget is per chunk, default 120s, `ratchet read --patience N`.

Also read every backup chunk twice and require the two to agree. The liveness
check alone cannot see a probe that corrupts data mid-chunk while the chip
keeps answering RDID: the id is healthy before and after, so the chunk banks
as done and the garbage reaches disk inside a backup the user trusts. This is
the same evidence as the project's two-full-reads gate, every byte read twice
and matched, but per 64KB instead of per 32MB: matching two whole dumps needs
minutes of unbroken contact twice over, matching one chunk needs a second.
Verify and write read-back stay single-read (they compare against a known
image, so a bad read there already fails closed as a mismatch).

Persist each chunk as it lands, not only when the read fails. A 9 minute read
on a bad probe looks exactly like a hang, users kill it, and there is no error
path to hang the save off. Bytes are written before the sidecar records them,
so a kill in the gap costs a re-read of one chunk rather than leaving a
sidecar entry for filler that a resume would skip.

Fixes a hot-spin found while reviewing this: a chunk can fail while RDID
answers perfectly (two reads disagreeing is exactly that), and wait_for_contact
then returns instantly, so the retry loop re-read at full tilt for the whole
patience budget. Hammering a marginal bus is the one response guaranteed not
to help. Every retry now paces itself.

Tests, each mutation-checked to confirm it discriminates:
- read_waits_out_a_flickering_probe_and_still_gets_the_whole_chip
- read_rejects_a_chunk_that_reads_differently_twice (with liveness-only trust
  this passes a corrupt dump to disk and reports success: backup-1's failure)
- read_persists_each_chunk_as_it_lands_not_only_when_it_fails

Unit tests default to a near-zero patience: a fake bus has no probe that might
flicker back, so waiting is dead test time, and four dead-bus tests were each
sitting through the full production budget (module went 120s to 7s).
@jackulau
jackulau merged commit dcbd617 into main Jul 17, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant