Skip to content

feat(chartstats): cache compiled measurement programs - #44

Merged
nichinichisou0609 merged 1 commit into
mainfrom
feat/chart-stats-program-cache
Oct 10, 2026
Merged

nichinichisou0609 merged 1 commit into
mainfrom
feat/chart-stats-program-cache

Conversation

@nichinichisou0609

@nichinichisou0609 nichinichisou0609 commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Problem

A whole-chart cache loses useful work when unrelated master metadata or a skill catalog changes. It also cannot retain an expensive completed subprogram if a later measurement fails.

Changes

  • Add source-versioned, content-addressed caches for compiled expectations, deterministic runs and lottery propagation curves.
  • Identify actual initialized interpreter inputs and complete play/rank schedules. Normalize naming-only skill/effect IDs while preserving alias relationships, resolved predicates, native execution order and lottery proof dependencies.
  • Keep current input validation, metadata, kind/shape catalogs, check decks and final statistics assembly outside the cache. New kinds, new shapes and catalog index shifts can reuse existing programs.
  • Persist raw integer results and binary64 endpoint bits with schema/key/source checks and a payload checksum. Commit each completed program atomically; repair corrupt records on demand and retry temporary-file collisions after an interrupted process.
  • Expose ChartStatsCache, chart_stats_with_cache, document_with_cache and --stats-cache, with work counters on stderr. Existing uncached APIs and chart-stats/3 output remain unchanged.
  • Start a new cache namespace without importing older whole-chart caches.

Validation

  • cargo fmt --all -- --check
  • Both CI clippy variants, including native-fixtures, with -D warnings
  • 10 program identity tests, 6 storage/recovery tests and all 22 chartstats integration tests
  • Four native incremental cases: metadata/growth changes, a new ordinary kind, changed real skill values, and a new member shape shifting support shape IDs
  • Three real charts covering Just, Combo and Luck: uncached, cold and warm output is byte-identical to each other and to the pre-change main baseline. The warm run records 1,023 hits, zero computations and zero writes.

Full regression and snapshot validation

Full workspace cargo test --release: 813 passed, zero failed, 5 ignored. The ignored tests are the repository's existing manual fixture/corpus exports. The validation build used a temporary single-codegen-unit setting; repository build profiles are unchanged.

A complete real snapshot update grows from 348 charts and 37 aptitude shapes to 352 charts and 38 shapes, including a changed shape and shifted catalog indices:

Run Cache hits Computed units Writes
Old snapshot, cold cache 14,690 96,562 96,562
New snapshot, reuse old cache 95,940 6,644 6,644
New snapshot, warm cache 100,676 0 0

The incremental, warm and uncached new-snapshot outputs are byte-identical: 12,668,775 bytes, SHA-256 7b48dda69ee9e828ac78bdc383f466fa572c733de3d4a1908d86c4d0dd266448. Counters represent operations across the three cache tiers, not chart counts; an expectation hit avoids lower-tier requests.

The actual Python extension in StarMoe-org/nnnotes#21 also passed warm reuse, seed-prefix/provenance-only changes, invalid-original-input rejection, interrupted-build recovery and one-record corruption recovery.

Integration

The coordinated producer is StarMoe-org/nnnotes#21. Nominal expectation consumers are empty-sekai/ournotes-player#18 and nichinichisou0609/moenotes#7. Production replay/recommendation bundles require a release with matching model commit and source identities; this PR does not publish a release.

See docs/chart-stats-cache.md for the identity boundary, APIs, counter meanings and storage lifecycle. The cache is disposable and has no automatic garbage collection; obsolete source namespaces can be removed independently.

@nichinichisou0609
nichinichisou0609 force-pushed the feat/chart-stats-program-cache branch from df86cc7 to 14c8ded Compare October 10, 2026 04:39
@nichinichisou0609
nichinichisou0609 marked this pull request as ready for review October 10, 2026 04:48
@nichinichisou0609
nichinichisou0609 merged commit 293d81c into main Oct 10, 2026
5 checks passed
@nichinichisou0609
nichinichisou0609 deleted the feat/chart-stats-program-cache branch October 10, 2026 08:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant