Skip to content

Pin inscount backend (--pin) for deterministic instruction counts - #5

Open
m-hu wants to merge 5 commits into
masterfrom
pin-inscount
Open

m-hu wants to merge 5 commits into
masterfrom
pin-inscount

Conversation

@m-hu

@m-hu m-hu commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Port of hhh-pa/clojure-code-snippet#52 to the Python harness, then switched from valgrind/callgrind to Intel Pin 4.4, then to a pin-only single-phase sweep with an allocation signal.

Harness (tools/measure.py):

  • Native --pin mode (--pin-bin/--pin-tool/--pin-repeat/--pin-timeout): each case runs under Intel Pin with an inscount tool, recording pin_instructions min/median/max plus pin_version provenance. Pin timeouts degrade to a pin-skipped marker (timed data, if any, stays valid); crashes and output mismatches fail the case. No JVM signal flags: the Python harness has no JVM leg.
  • Pin-only mode (--repeat 0, needs --pin; cf. clojure-code-snippet#52 commit 4fe8d86): skips the native timed phase entirely, so each case is measured ONCE under Pin; correctness still gates on the first instrumented run's stdout. Timed signals stay null; the console/summary renderers print "-" and BMF omits the rows instead of emitting zeros (an all-skipped snippet stays out of BMF entirely).
  • Memory signal: the same Pin run preloads an LD_PRELOAD malloc/calloc/realloc-counting shim (scripts/ci/instrumentation/allocshim.c, ported from the clojure-code-snippet embedded shim, built on demand with cc), whose destructor reports snip_alloc_total_bytes=N on the child's stderr; recorded as allocated_bytes min/median/max alongside pin_instructions in results + BMF + summary. A missing shim total after a successful exit fails the case (harness broken, not the snippet).
  • Locally, --repeat N (N>=1) still runs the native timed phase first (or under Callgrind when valgrind happens to be installed); the Callgrind code paths stay for local use. TOOL_VERSION 1->4.

Memory-signal semantics (boundary):

  • allocated_bytes is cumulative libc allocation for the whole process run (freed bytes never subtracted). It INCLUDES Pin's own JIT/VM allocations, so like pin_instructions it is only comparable within one Pin version; do not compare against plain (no-Pin) shim totals.
  • Peak RSS under Pin is NOT meaningful (Pin's own footprint pollutes it): maxrss is never read from instrumented runs and stays null in pin-only sweeps; allocated_bytes is the memory signal.
  • Hang behavior change: with no native phase, a genuine hang surfaces as a pin timeout -> pin-skipped (no BMF rows) rather than a failure; hangs still fail in the test-shard jobs, which run natively with --timeout 300.

CI (.github/workflows/test.yml):

  • RENAMED perf-callgrind -> perf-instrumentation, report -> report-instrumentation (no behavior change in that step).
  • Then SWITCHED the backend: the Install Valgrind step is gone; the job now caches ~/instrumentation (actions/cache, keyed on the setup script + tool source), provisions Pin 4.4 via scripts/ci/setup-instrumentation.sh (kit download sha256-pinned, idempotent; fails fast when allocshim.c is missing), builds the vendored inscount tool (scripts/ci/instrumentation/inscount.cpp, own minimal TRACE/BBL implementation printing one "Count N" line — no Intel example sources), and sweeps pin-only with --repeat 0 --pin --pin-repeat 2 --pin-timeout 600.

Bencher continuity tradeoff (by design):

  • The callgrind instructions (Ir) history ENDS: CI no longer installs valgrind, so that measure stops being reported.
  • The cpu_time history also ENDS in CI: pin-only sweeps have no timed phase, so no cpu_time row is emitted (never a zero one).
  • Tracking continues on the new pin_instructions measure (fresh history; Pin counts are a different signal than Ir and must not share the axis) plus the new allocated_bytes measure (fresh history; includes Pin overhead, within-version comparisons only). Default-branch runs gate on --threshold-measure pin_instructions.

Validation on workstation.hhh.pa:

  • setup-instrumentation.sh from scratch into a fresh dir: kit download OK (sha256 match), inscount tool builds, pin -version reports pin-4.4-99977-54fbb8814; rerun is a no-op re-export.
  • --self-test passes (local mac + workstation Linux; Linux additionally builds the shim with cc and smoke-tests LD_PRELOAD counting). allocshim.c also compiles warning-free under -Wall -Wextra.
  • Real pin-only sweep (--repeat 0 --pin --pin-repeat 2) over synthetic bubble_sort + insertion_sort cases (n=400): single-phase shape confirmed (runs=0, cpu/wall/maxrss/instructions null), alloc_shim_used with pin_version recorded, pin_instructions agree within ~0.2% across repeats (101140446/101347041/101553637), allocated_bytes bit-identical across repeats (2385533, 2377222); BMF holds only pin_instructions + allocated_bytes (no cpu_time/instructions rows); summary shows the Allocated (B) column with "-" for Ir/CPU; no failures.
  • Plain-shim sanity (same scripts, LD_PRELOAD only, no Pin): 2377509/2369195, bit-identical across repeats, stdout matches expected; pin+shim totals are +8024/+8027 B (+0.34%) over plain — sane deterministic Pin overhead.
  • Timed+pin regression (--repeat 1 --pin; valgrind present on workstation): all four measures flow (pin_instructions, allocated_bytes, instructions Ir, cpu_time); alloc totals identical to the pin-only run.
  • Sharded pin-only (0/2, 1/2) --merge output equals the plain shard-BMF union (union check in Python; no jq on workstation).

m-hu added 5 commits November 5, 2026 11:35
Replace the valgrind/callgrind backend with Intel Pin 4.4:
provision the kit (sha256-pinned, cached) and build the new
vendored inscount tool (scripts/ci/instrumentation), then sweep
with --pin. The harness keeps its Callgrind code paths for local
use but CI no longer installs valgrind, so the deterministic
signal is now the pin_instructions BMF measure; cpu_time
continues, callgrind 'instructions' history ends by design.
--repeat 0 skips the native timed phase (needs --pin), so each case
is measured once under Pin; correctness still gates on the first
instrumented run's stdout. The same Pin run now also preloads the
LD_PRELOAD alloc shim (scripts/ci/instrumentation/allocshim.c,
built on demand with cc, total parsed from the child's stderr),
recording allocated_bytes alongside pin_instructions. Renderers
print '-' / omit BMF rows instead of zeros when timed signals are
absent; maxrss is never read from instrumented runs. TOOL_VERSION
3->4. Callgrind code paths untouched.
Measure step runs --repeat 0 (no native timed phase, no --timeout);
each case measured once under Pin with the alloc shim in the same
run. setup-instrumentation.sh fails fast when allocshim.c is
missing.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant