Skip to content

PRE-PUSH-LATENCY: guard-cargo 6223 s cold in pre_push_local.sh — keyed shared target dir + overlap guard-tree #5983

Closed

Description

@noahgift

Five-whys, from aprender-pace measurements (inbox 19:04Z) and scripts/pre_push_local.sh on branch pace/pre-push-local @c910afba93:

  1. Why did the pre-push take >100 min? guard-cargo alone ran 6223 s cold (guard-tree ~11 min, quick tier 166 s).
  2. Why cold? step_guard_cargo unsets CARGO_TARGET_DIR on purpose (match the runner env), so cargo builds in the worktree-local ./target; every fresh worktree starts empty.
  3. Why also slow to start? On lambda ~/.cargo/registry is a symlink, so CARGO_HOME falls back to /cargo-home and the first run downloads crates.
  4. Why 6223 s and not ~1/3 of that? Load average ~43 (74 sessions) on lambda; cargo competes with the fleet. Not measured: per-phase split (compile vs download vs wait).
  5. Why serial? preflight, guard-tree, guard-cargo, quick tier run one after another; guard-tree (no cargo) could overlap guard-cargo.
    Root cause: no reuse across pushes. The one step that must see the runner env has no stable, keyed build cache.

Countermeasure (one): give guard-cargo a stable CARGO_TARGET_DIR + CARGO_HOME keyed by sha(Cargo.lock + rust-toolchain.toml), e.g. /mnt/nvme-raid0/targets/pre-push-guard-cargo-, and start guard-tree in parallel with it. Acceptance: warm run <= 15 min, AND the host rebuilds <= the CI unit count after the container build (the 157-vs-12 check), measured and posted; cold run unchanged.
Owner: aprender-pace. Parent: #5923. Requested by aprender-57.

Activity

  1. noahgift commented on Oct 24, 2026

    @noahgift
    MemberAuthor

    Closing under EMR-001 P6 step 1: the fix landed on main as f56911b (commit trailer PMAT-Ticket: PMAT-5983, in v0.71.4..main). Reopen if a part is still missing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions