Skip to content

Build anatomy audit: cargo --timings critical path, dup deps, test-binary count (read-only) #5841

Closed as not planned
Closed as not planned
Build anatomy audit: cargo --timings critical path, dup deps, test-binary count (read-only)#5841
Milestone
0.72.0

Description

@noahgift
noahgift
opened on Oct 21, 2026

Build anatomy audit (aprender), read-only, no ci.yml change

Owner: aprender-flow (audit). Operator, verbatim, 08:53Z: "Build anatomy audit, one worker per repo, pmat ticket first, read-only, no ci.yml change. Report by 12:00Z." Goal: warm-cache build share of shard time 78% -> <50%.

Method (on origin/main, idle non-merge-queue host, warm and cold cache):

  1. cargo build --timings (workspace, test profile): wall time, top-5 critical-path crates with secs, % of time cores idle, top-10 crates by time; timings HTML attached.
  2. cargo tree -d, cargo machete, cargo nextest list (test-binary count), build.rs / proc-macros on the critical path, cargo llvm-lines on the top 3 crates.
  3. Each number cited with its command and UTC time.
  4. Rank fixes by build-minutes saved: split hub crates, merge integration-test binaries, dedupe deps, unify features (hakari), drop unused deps, cut dev-dep edges (like perf(ci): FLOW-010 cut 1 — drop aprender-compute dev-dep on aprender-core (Refs #5803) #5804).

Baseline: pooled over 3 MG runs / 20 shard jobs, build 78% vs nextest run 22% (see #5839, duplicate of #5812).

Activity

  1. noahgift
    added this to the 0.73.0 milestone on Oct 21, 2026

  2. noahgift commented on Oct 21, 2026

    @noahgift
    noahgift
    on Oct 21, 2026
    MemberAuthor

    Added at aprender-57's request (not yet measured by me, figures are theirs): in pace's batch gate, check_pv_bin_resolution.sh ran 26 min (09:38:43–10:05:01Z) and corpus-ingest 9 min. Both will be ranked with the build-anatomy fixes by minutes saved in the 12:00Z report. Owner: aprender-flow.

  3. noahgift commented on Oct 21, 2026

    @noahgift
    noahgift
    on Oct 21, 2026
    MemberAuthor

    #5841 build anatomy audit — aprender @ b9cb67a (origin/main, shallow clone)

    Host lambda-vector (24 cpu visible to cargo, mold 1.10.1 + clang), nice 19, CARGO_BUILD_JOBS=8, fresh target dir. UTC 2026-10-21.
    Command: cargo build --timings --workspace --tests --exclude aprender-gpu --exclude aprender-cuda-edge --exclude aprender-compute (profile dev, unoptimized+debuginfo; the 3 excluded crates are CI's own exclusions)

    Wall (cargo "Finished", cold.log/warm.log; ended 10:32:49Z / 10:42:46Z)

    • COLD 367.4 s (6m07), 1883 units, cpu-sum 2844 s. Not counted: ~15 min queued in the cargo-memcap wrapper before cargo started.
    • "WARM" 210.3 s (3m30): touch crates/aprender-core/src/lib.rs then rebuild, so it is the dependents-of-core rebuild, not a no-op. 338 units, cpu-sum 1348 s.

    Idle (units from the timings UNIT_DATA, 1 s samples)

    • COLD: <8 units running 27 s of 368 s (7%); <4 running 14 s. WARM: <8 running 59 s of 211 s (28%); <4 running 44 s. Slots are jobs=8, not the 24 cpus; per-core idle not measured.

    Top 10 units by time (COLD) aprender-core lib(test) 153 s · aprender-serve lib(test) 131 · aprender-train lib(test) 114 · aprender-test-lib lib(test) 85 · apr-cli lib(test) 80 · aprender-orchestrate lib(test) 72 · orchestrate bin(test) 50 · aprender-compute 43 · apr-cli 39 · chromiumoxide_cdp 38

    Top crates by summed unit time (COLD) serve 366 · core 293 · orchestrate 196 · apr-cli 188 · train 171 · test-lib 117 · profile 107 · contracts 75

    Longest chain at the tail (from unblocked_units; rough, ignores rmeta pipelining) COLD: aprender-test-lib 26 s -> aprender-serve lib(test) 131 s. WARM: aprender-train 15 s -> aprender-core lib(test) 85 s.

    Build scripts / proc-macros: 61 s cpu of 2844 s (2%); top zstd-sys 9.3 s, libsqlite3-sys 6.7 s, ring 3.8 s. Not on the critical path.

    Other numbers

    • cargo tree -d: 335 crates with >=2 versions (earlier run today).
    • cargo machete: 58 crates, 145 deps flagged (machete.txt; false positives possible, e.g. workspace aliases; --with-metadata not used).
    • cargo nextest list --workspace --lib (CI scope): 79 test binaries, 85,483 tests. Integration test files: 641 in 49 crates (CI runs --lib only, so merging them does not shorten CI today).
    • cargo llvm-lines --lib: core 1.47M lines/51k copies, serve 1.75M/60k, train 1.40M/42k. Top fn is under 4%; no single dominant instantiation (core: slice::sort::stable_partition 226 copies, 3.9%).

    Ranked by minutes (estimates marked ~; only the first two rows are measured elsewhere)

    1. check_pv_bin_resolution.sh 26 min + corpus-ingest 9 min in pace's batch gate (aprender-57's figures, 09:38-10:05Z) = 35 min per batch, more than every compile item below.
    2. Build once, then shard from a nextest archive (ci(FLOW-010 P1): nextest archive — build the tree-reader set once, shards run from it #5812/CI: build once + nextest archive to shards (80-93% of shard build-min is duplicate) #5839): every shard recompiles; saves ~(N-1) x one build per run: ~3 x 6 min cpu per 4-shard MG at jobs=8, ~11 x for 12 PR shards. Wall saving per shard ~ its build share (pooled 78%, n=3).
    3. Split aprender-core (single 153 s cold / 85 s warm test unit; every core-touching PR rebuilds ~210 s with 28% slots idle): 1-1.5 min wall per core-touching PR if the unit halves ().
    4. Same for serve/train/apr-cli/orchestrate lib(test) units (80-131 s each): 1 min each ().
    5. Dedupe deps (335 dup crates) and hakari feature unification: not measured; compile share unknown.
    6. Drop machete deps (145): ~<1 min; cuts dev-dep edges, small.
    7. Build scripts / proc-macros: <=2% of cpu; skip.
      Not measured: link time split, per-core idle (24 cpus), CI-shard build share on this commit; 78% -> <50% target cannot be checked from a single-host run.
  4. noahgift
    modified the milestones: 0.73.0, 0.72.0 on Oct 22, 2026
  5. noahgift
    added a parent issue on Oct 22, 2026

  6. noahgift commented on Oct 22, 2026

    @noahgift
    noahgift
    on Oct 22, 2026
    MemberAuthor

    Tracked in #5510 (triage 2026-10-22).

  7. noahgift commented on Oct 22, 2026

    @noahgift
    noahgift
    on Oct 22, 2026
    MemberAuthor

    tracked in #5962

  8. noahgift
    closed this as not plannedon Oct 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions