Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 21 additions & 28 deletions .github/MUSIC_DATA.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ checks the affected enabled regions; an older dispatch without a region list che
one region. The matrix has `fail-fast: false`, so an unavailable JP source does not cancel TW. Each region calls
`music-data-region.yml` with its package, catalog, client and output prefix: TW `music-data`, JP `jp/music-data`.
The previous global `MUSIC_DATA_MASTERDATA_REGION` no longer selects which regions run; it only preserves ownership
of a legacy `MUSIC_DATA_S3_PREFIX` (JP-owned prefixes remain JP). Both normal and prebuilt reject equal effective
of a legacy `MUSIC_DATA_S3_PREFIX` (JP-owned prefixes remain JP). Publication rejects equal effective
TW/JP prefixes before publication, including slash-normalized duplicates. See [Japanese release](../docs/jp.md)
for setup and validation limits; enabling the matrix does not certify a new client model or guarantee JP availability.

Expand Down Expand Up @@ -83,19 +83,19 @@ extend the existing linear plain-skill UI domain or derive source rank threshold

1. **plan** (seconds): the inputs of a build, from `index.json` of moenotes-masterdata-sync (the snapshot's master
data version, resource version, client version, manifest hash and decoded file-inventory hash of the selected region)
and the checkout (the
ournotes-deck commit `rust/Cargo.lock` pins for the `ournotes-sim` package, the last nnnotes commit that changed `src/`, `rust/` or
`pyproject.toml`, consumer repository/ref and `RECIPE` of `music_data.py`), against `inputs` of the published `build.json`. The same
and the installed nnnotes release of `.github/nnnotes.txt` (its version, the ournotes-deck commit of the
`ournotes-sim` release its extra `deck` pins), consumer repository/ref and `RECIPE` of `music_data.py`, against
`inputs` of the published `build.json`. The same
inputs (and a published `music-data.json`): the run ends here. `force` builds anyway.
2. **build**:
- the chart data page's modules (`examples/songs` of `MUSIC_DATA_PLAYER_REF`, not built), nnnotes with its deck
model (the install builds `nnnotes._deck`), the gate self-test, the APK (playfetch with `PLAYFETCH_CREDENTIALS`:
- the chart data page's modules (`examples/songs` of `MUSIC_DATA_PLAYER_REF`, not built), the nnnotes release of
`.github/nnnotes.txt` with its deck model (the package `ournotes-sim`), the gate self-test, the APK (playfetch with `PLAYFETCH_CREDENTIALS`:
`provenance.client`; nnnotes also reads the bundles the APK carries, as in the story site's builds), the decoded
master data of moenotes-masterdata-sync (every file SHA-256 checked against `index.json`, `MasterManifest.json`
included);
- the replay and recommendation engines: `ournotes-replay-wasm-v<version>.tar.gz` and
`ournotes-recommend-wasm-v<version>.tar.gz` of the ournotes-deck release `v<version>`, the version of the
`ournotes-sim` package `rust/Cargo.lock` pins, each checked against the release's `SHA256SUMS`; nnnotes takes
installed `ournotes-sim`, each checked against the release's `SHA256SUMS`; nnnotes takes
their web packages after checking each `build-info.json` (module, commit, file SHA-256) against its deck model;
- `nnnotes music-data --decoded-master --jackets jackets --replay-dir replay --replay-engine ... --recommend-engine
... --stats-cache $MUSIC_DATA_STATS_CACHE -o music-data.json`: restores the region's native measurement
Expand Down Expand Up @@ -137,10 +137,9 @@ missed, and `workflow_dispatch`:
| `dry_run` | build and check, then list what would be uploaded instead of uploading |
| `reencode_only` | verify and gzip the currently published snapshot without compiling Rust or recomputing statistics |

Normal and prebuilt writers share `music-data-<region>` concurrency with cancellation disabled. TW and JP have
Runs of one region share `music-data-<region>` concurrency with cancellation disabled. TW and JP have
separate output prefixes and can proceed independently. Prefix overrides must remain distinct between regions.
The two dedicated `songs/` page writers also share `songs-page-publication`; prebuilt acquires that global lock
before its data-region lock. Story publication does not write the dedicated `songs/` closure.
The dedicated `songs/` page writer holds `songs-page-publication`. Story publication does not write the dedicated `songs/` closure.

`reencode_only=true` requires every current gate attestation, the unchanged source identity and build marker, all
13 same-source labels, and every decoded replay/archive/engine hash. It first writes a content-addressed gzip probe
Expand Down Expand Up @@ -173,8 +172,8 @@ Every one must pass, else nothing is published. Warnings go to the job summary a
| Gate | Checks |
|---|---|
| (build) | nnnotes' own checks: every table, chart and cue sheet read, every chart measured, the deck statistics cross-checked against the chart facts and the master data (the command writes no file otherwise) |
| `schema` | the file against `docs/schema/music-data.schema.json` of the checkout (JSON Schema 2020-12), requiring `nnnotes.music-data/2` and `ournotes-deck.chart-stats/3` statistics |
| `provenance` | `format`; `region` `tw`; `master.source` `api`; `master.version` equal to the snapshot's and its `MasterManifest.json`'s; every table's SHA-256 the manifest's, every decoded table read the one `index.json` lists; the song tables and the deck model's present; `deck.commit` the one `rust/Cargo.lock` pins; `exporter.version` the installed nnnotes; an APK version; a catalog SHA-256 (warning: the APK is another client version than the snapshot's) |
| `schema` | the file against the music data schema the installed nnnotes ships (`nnnotes.schemas`, JSON Schema 2020-12), requiring `nnnotes.music-data/2` and `ournotes-deck.chart-stats/3` statistics |
| `provenance` | `format`; `region` `tw`; `master.source` `api`; `master.version` equal to the snapshot's and its `MasterManifest.json`'s; every table's SHA-256 the manifest's, every decoded table read the one `index.json` lists; the song tables and the deck model's present; `deck.commit` the installed `ournotes-sim` release's; `exporter.version` the installed nnnotes; an APK version; a catalog SHA-256 (warning: the APK is another client version than the snapshot's) |
| `counts` | no fewer songs and charts than the published file (warning: ids no longer in it) |
| `deck` | native kinds, positive power, events and positions matching chart facts; explicit nominal `expectation` unless unplayable; replay seeds equal `[0]` without luck or the published prefix shared by luck charts; finite `[center, outward radius]` estimates, valid dimensions and deterministic counters; lottery estimates enclose zero outside luck ranges; complete expectation check decks within their bounds |
| `scenarios` | exactly one deterministic Free Live `offSeeds` entry (seed 0, scalar score/weights and bounded check); five integer rank percentages per range; explicit best/Perfect scores and `rankBonusPerfect`; rank-weight dimensions and rank checks match the supported linear domain; nulls warn about unavailable rank or Free Live weights rather than becoming zero gains |
Expand All @@ -195,7 +194,7 @@ legitimate drop (a song the game removed) stops the run: a person checks it, the
The self-test runs in each build and locally in seconds, without the network:

```
python -m pytest -q -p no:cacheprovider .github/scripts/test_music_data.py .github/scripts/test_jp_workflow.py .github/scripts/test_music_data_refresh.py .github/scripts/test_music_data_compression.py .github/scripts/test_prebuilt_source_identity.py
python -m pytest -q -p no:cacheprovider .github/scripts/test_music_data.py .github/scripts/test_jp_workflow.py .github/scripts/test_music_data_refresh.py .github/scripts/test_music_data_compression.py
```

with, optionally, `MUSIC_DATA_SCHEMA` (a schema file when the checkout has none), `MUSIC_DATA_PAGE` (an
Expand Down Expand Up @@ -240,30 +239,24 @@ Repository variables:

## Before the first run

- **Native model and engines.** The native program cache is supplied by
[ournotes-deck#44](https://github.com/empty-sekai/ournotes-deck/pull/44). The exact source revision in
`rust/Cargo.toml` and `rust/Cargo.lock` must also be available as a model release with both replay and
recommendation WASM packages. Building this source pin locally is supported before that release exists;
production publication still requires matching engine commit and source identities. If the release uses a
different commit after merging, update the Cargo revision and lockfile package version and rebuild the extension.
The publisher selects `v<version>` from that package version, so creating a new tag alone is insufficient.
Older engines, including an older release with the
same package version, must not be substituted or admitted by weakening the identity gate.
- **Native model and engines.** The deck model is the `ournotes-sim` release that the installed nnnotes pins; the
publisher downloads the replay and recommendation WASM packages of that same release (`v<version>`), and nnnotes
rejects engines of any other release.
- **Consumers.** Deploy readers for `nnnotes.music-data/2` and `ournotes-deck.chart-stats/3` before publishing new
data. The default `MUSIC_DATA_PLAYER_REF` already selects the nominal-expectation consumer in
[ournotes-player#18](https://github.com/empty-sekai/ournotes-player/pull/18); an explicit repository variable
overrides that default and must pass the same smoke test. The corresponding site changes are
[moenotes#7](https://github.com/nichinichisou0609/moenotes/pull/7), stacked on the deck-goal repair in
[moenotes#6](https://github.com/nichinichisou0609/moenotes/pull/6).
- **Producer and publication.** This checkout publishes only the new nominal format and starts a fresh native
cache namespace; it does not migrate old Python cache records. Run the normal or prebuilt publication gates
cache namespace; it does not migrate old Python cache records. Run the publication gates
with the matching engines and consumers. Set `MUSIC_DATA_PUBLISH` to `true` after the dry run passes.

## Notes

- **Versions.** A new ournotes-deck pin (`rust/Cargo.toml`, `rust/Cargo.lock`) or a new nnnotes commit in `src/`,
`rust/` or `pyproject.toml` reaches this fork with a sync, and the next run builds a new file. The deck
statistics may then differ: `provenance.deck.commit` and `build.json` name the commit.
- **Versions.** The workflows install the nnnotes release named in `.github/nnnotes.txt`. Raising it (a new
nnnotes release, which may pin a new ournotes-deck release) changes the build's inputs, and the next run builds a
new file. The deck statistics may then differ: `provenance.deck.commit` and `build.json` name the commit.
- **The page and the data.** The page's modules are pinned by `MUSIC_DATA_PLAYER_REF`: changing that pin or its
repository is now a build input and starts a new check. Use an immutable commit to keep the consumer identity stable.
- **Byte identity.** A build from the same inputs gives the same bytes (the file is canonical); the jackets'
Expand All @@ -272,8 +265,8 @@ Repository variables:
uploaded as an artifact.
# HTTP compression

Normal and validated-prebuilt publication uploads every JSON object at its existing `.json` URL with `Content-Type: application/json` and `Content-Encoding: gzip`. Compression uses gzip level 6 and `mtime=0`; engine JavaScript, WASM and images remain unchanged. Manifest SHA-256 values and byte counts always describe the decoded payload consumed by browser `fetch`. S3 metadata records `decoded-sha256`, `decoded-bytes`, `encoded-sha256` and `encoded-bytes` separately. Read-back gates verify both stored identity and decoded identity before advancing pointers. Python public-source reads explicitly decode the HTTP content encoding.
Publication uploads every JSON object at its existing `.json` URL with `Content-Type: application/json` and `Content-Encoding: gzip`. Compression uses gzip level 6 and `mtime=0`; engine JavaScript, WASM and images remain unchanged. Manifest SHA-256 values and byte counts always describe the decoded payload consumed by browser `fetch`. S3 metadata records `decoded-sha256`, `decoded-bytes`, `encoded-sha256` and `encoded-bytes` separately. Read-back gates verify both stored identity and decoded identity before advancing pointers. Python public-source reads explicitly decode the HTTP content encoding.

To re-encode an already published, verified snapshot without running Rust or generating a replacement model, dispatch `music-data.yml` with `reencode_only=true`, the target `region`, and `dry_run=false`. This path shares the normal/prebuilt region lock. It requires the current source identity, an unchanged published build marker, all gate attestations, the complete SHA-bound replay runtime and all 13 Snap label tables. It checks source and marker again before each read/write, first verifies the store/CDN using a fresh gzip probe, then re-encodes the JSON payloads and externally verifies headers, stored identity and decoded identity. Manifest, music-data and build-marker decoded bytes never change. The encoding report records every object and total encoded/decoded size; it contains no game payloads or credentials.
To re-encode an already published, verified snapshot without running Rust or generating a replacement model, dispatch `music-data.yml` with `reencode_only=true`, the target `region`, and `dry_run=false`. This path shares the region lock. It requires the current source identity, an unchanged published build marker, all gate attestations, the complete SHA-bound replay runtime and all 13 Snap label tables. It checks source and marker again before each read/write, first verifies the store/CDN using a fresh gzip probe, then re-encodes the JSON payloads and externally verifies headers, stored identity and decoded identity. Manifest, music-data and build-marker decoded bytes never change. The encoding report records every object and total encoded/decoded size; it contains no game payloads or credentials.

JP builds perform an actual CDN-authentication preflight before Rust/WASM allocation. A refused `Version` call remains a failed JP update; it cannot substitute another region or bypass authenticated snapshot checks. Report artifacts use canonical workspace paths so `upload-artifact@v7` can retain diagnostics.
9 changes: 5 additions & 4 deletions .github/STORY_SITE.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ lacks with this repository's `nnnotes web --story` and uploads them to the bucke
published internationally and only builds stories missing from both sites. MoeNotes merges the two indexes and
plays shared advIds from the international site, preserving all its text languages. It only adds: a run
never deletes anything from the bucket. Its helper steps are in
`.github/scripts/`; nothing outside `.github/` differs from upstream, so the fork syncs with it as before.
`.github/scripts/`; it installs the nnnotes release named in `.github/nnnotes.txt`.

## A run

Expand Down Expand Up @@ -95,11 +95,12 @@ Repository variables (optional; the defaults are the StarMoe site): `STORY_S3_EN
CDN authentication. Generated VPN profiles use connection fields and certificates, discarding supplied scripts
and routes. The helpers are `jp_vpngate.py` and `story_site.py jp-check`.
- **The player version.** The site's player pages come from `STORY_PLAYER_REF`, and its data must be what that player
reads. After a player release that changes the data format, sync this fork with upstream nnnotes and set
`STORY_PLAYER_REF` to the matching player; new stories are then written for it. Stories already published are
reads. After a player release that changes the data format, raise `.github/nnnotes.txt` to the nnnotes release
that writes it and set `STORY_PLAYER_REF` to the matching player; new stories are then written for it. Stories already published are
not rebuilt (run with `stories` and `force` for that).
- **The APK.** Google Play serves the current game version. When nnnotes' type trees do not match it, the build stops
naming the class and the Unity version: sync the fork with upstream once nnnotes supports that version.
naming the class and the Unity version: raise `.github/nnnotes.txt` once an nnnotes release supports that
version.
- **Bytes.** A rebuild on the runner is not byte-identical to a build elsewhere: the AAC encoder (ffmpeg) and the PNG
encoder (zlib) differ between machines. The pixels and every JSON value are the same, so new stories and the
published ones fit together; only a forced rebuild re-uploads such files.
Expand Down
3 changes: 3 additions & 0 deletions .github/nnnotes.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# The nnnotes release the publishing workflows install: music-data with its deck model (the extra deck, which
# pins one ournotes-deck release) and the story site with the open fonts (the extra fonts).
nnnotes[deck,fonts]==0.2.1
Loading
Loading