feat: retry failed targets and classify transient failures as user errors

[notheotherben]

Overview

Two related improvements to how the backup engine handles transient, user-facing failures (dropped/reset connections, timeouts, and git pack errors) which were previously surfaced as System errors and reported as bugs.

1. Retries in the policy system

• New optional retries policy property (default 1), read via BackupPolicy::retries().
• Pairing::run_all_backups retries each target backup up to retries + 1 times before reporting the failure, skipping further attempts if the backup is cancelled.
• Retries are applied per-target, so a transient failure writing to one destination does not affect the others, and they compose with the existing git auto-recovery ladder (each attempt runs its own recovery).
• Invalid retries values are rejected as a User error.

backups:
  - kind: github/repo
    from: "repos/my-org/repo"
    to: /backups/work
    properties:
      retries: "3"

2. Reclassify transient failures as User errors

Previously these tripped error.is(Kind::System) in main.rs, firing record_custom_error telemetry and telling users to report a bug. They are now User errors:

• src/errors.rs (reqwest::Error): is_timeout() and is_request() (e.g. connection reset / dropped send) now map to wrap_user instead of wrap_system. This covers every reqwest call site (GitHub client, Forgejo client, HTTP file downloads). is_decode and genuinely unknown errors remain System.
• src/engines/git.rs: prepare_clone and fetch_only (the "Failed to consume the pack sent by the remote" case) changed from wrap_system_err → wrap_user_err, matching the existing fetch path. Local disk/config writes remain System.

Examples now classified as User errors

error sending request for url (...): client error (SendRequest): connection error: connection reset
Unable to clone remote repository '...': Failed to consume the pack sent by the remote

Tests

• policy.rs: retries default / configured / zero (disabled) / invalid.
• pairing.rs: recover-after-retry, give-up-after-budget, retries disabled, invalid-property reporting (via a flaky mock engine).
• errors.rs: timeout → User (wiremock delayed response + short client timeout) and connect-failure → User.

All 152 tests pass; cargo fmt --check and cargo clippy --all-targets are clean.

Docs

• New docs/advanced/retries.md, registered in the VuePress navbar + sidebar.

Note for reviewers

retries is implemented via the existing properties map (like recovery/refspecs) and defaults to 1 retry globally. Happy to change it to opt-in (default 0) or promote it to a first-class BackupPolicy field if preferred.

[codecov]

Codecov Report

❌ Patch coverage is 96.00000% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 76.83%. Comparing base (4f34aaf) to head (7667d16).

Files with missing lines	Patch %	Lines
src/errors.rs	82.97%	8 Missing ⚠️
src/pairing.rs	99.19%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #622      +/-   ##
==========================================
+ Coverage   74.43%   76.83%   +2.40%     
==========================================
  Files          31       31              
  Lines        2394     2612     +218     
==========================================
+ Hits         1782     2007     +225     
+ Misses        612      605       -7     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.