diff --git a/.github/MUSIC_DATA.md b/.github/MUSIC_DATA.md index 365c5a3..3382d4e 100644 --- a/.github/MUSIC_DATA.md +++ b/.github/MUSIC_DATA.md @@ -5,7 +5,7 @@ checks the affected enabled regions; an older dispatch without a region list che one region. The matrix has `fail-fast: false`, so an unavailable JP source does not cancel TW. Each region calls `music-data-region.yml` with its package, catalog, client and output prefix: TW `music-data`, JP `jp/music-data`. The previous global `MUSIC_DATA_MASTERDATA_REGION` no longer selects which regions run; it only preserves ownership -of a legacy `MUSIC_DATA_S3_PREFIX` (JP-owned prefixes remain JP). Both normal and prebuilt reject equal effective +of a legacy `MUSIC_DATA_S3_PREFIX` (JP-owned prefixes remain JP). Publication rejects equal effective TW/JP prefixes before publication, including slash-normalized duplicates. See [Japanese release](../docs/jp.md) for setup and validation limits; enabling the matrix does not certify a new client model or guarantee JP availability. @@ -83,19 +83,19 @@ extend the existing linear plain-skill UI domain or derive source rank threshold 1. **plan** (seconds): the inputs of a build, from `index.json` of moenotes-masterdata-sync (the snapshot's master data version, resource version, client version, manifest hash and decoded file-inventory hash of the selected region) - and the checkout (the - ournotes-deck commit `rust/Cargo.lock` pins for the `ournotes-sim` package, the last nnnotes commit that changed `src/`, `rust/` or - `pyproject.toml`, consumer repository/ref and `RECIPE` of `music_data.py`), against `inputs` of the published `build.json`. The same + and the installed nnnotes release of `.github/nnnotes.txt` (its version, the ournotes-deck commit of the + `ournotes-sim` release its extra `deck` pins), consumer repository/ref and `RECIPE` of `music_data.py`, against + `inputs` of the published `build.json`. The same inputs (and a published `music-data.json`): the run ends here. `force` builds anyway. 2. **build**: - - the chart data page's modules (`examples/songs` of `MUSIC_DATA_PLAYER_REF`, not built), nnnotes with its deck - model (the install builds `nnnotes._deck`), the gate self-test, the APK (playfetch with `PLAYFETCH_CREDENTIALS`: + - the chart data page's modules (`examples/songs` of `MUSIC_DATA_PLAYER_REF`, not built), the nnnotes release of + `.github/nnnotes.txt` with its deck model (the package `ournotes-sim`), the gate self-test, the APK (playfetch with `PLAYFETCH_CREDENTIALS`: `provenance.client`; nnnotes also reads the bundles the APK carries, as in the story site's builds), the decoded master data of moenotes-masterdata-sync (every file SHA-256 checked against `index.json`, `MasterManifest.json` included); - the replay and recommendation engines: `ournotes-replay-wasm-v.tar.gz` and `ournotes-recommend-wasm-v.tar.gz` of the ournotes-deck release `v`, the version of the - `ournotes-sim` package `rust/Cargo.lock` pins, each checked against the release's `SHA256SUMS`; nnnotes takes + installed `ournotes-sim`, each checked against the release's `SHA256SUMS`; nnnotes takes their web packages after checking each `build-info.json` (module, commit, file SHA-256) against its deck model; - `nnnotes music-data --decoded-master --jackets jackets --replay-dir replay --replay-engine ... --recommend-engine ... --stats-cache $MUSIC_DATA_STATS_CACHE -o music-data.json`: restores the region's native measurement @@ -137,10 +137,9 @@ missed, and `workflow_dispatch`: | `dry_run` | build and check, then list what would be uploaded instead of uploading | | `reencode_only` | verify and gzip the currently published snapshot without compiling Rust or recomputing statistics | -Normal and prebuilt writers share `music-data-` concurrency with cancellation disabled. TW and JP have +Runs of one region share `music-data-` concurrency with cancellation disabled. TW and JP have separate output prefixes and can proceed independently. Prefix overrides must remain distinct between regions. -The two dedicated `songs/` page writers also share `songs-page-publication`; prebuilt acquires that global lock -before its data-region lock. Story publication does not write the dedicated `songs/` closure. +The dedicated `songs/` page writer holds `songs-page-publication`. Story publication does not write the dedicated `songs/` closure. `reencode_only=true` requires every current gate attestation, the unchanged source identity and build marker, all 13 same-source labels, and every decoded replay/archive/engine hash. It first writes a content-addressed gzip probe @@ -173,8 +172,8 @@ Every one must pass, else nothing is published. Warnings go to the job summary a | Gate | Checks | |---|---| | (build) | nnnotes' own checks: every table, chart and cue sheet read, every chart measured, the deck statistics cross-checked against the chart facts and the master data (the command writes no file otherwise) | -| `schema` | the file against `docs/schema/music-data.schema.json` of the checkout (JSON Schema 2020-12), requiring `nnnotes.music-data/2` and `ournotes-deck.chart-stats/3` statistics | -| `provenance` | `format`; `region` `tw`; `master.source` `api`; `master.version` equal to the snapshot's and its `MasterManifest.json`'s; every table's SHA-256 the manifest's, every decoded table read the one `index.json` lists; the song tables and the deck model's present; `deck.commit` the one `rust/Cargo.lock` pins; `exporter.version` the installed nnnotes; an APK version; a catalog SHA-256 (warning: the APK is another client version than the snapshot's) | +| `schema` | the file against the music data schema the installed nnnotes ships (`nnnotes.schemas`, JSON Schema 2020-12), requiring `nnnotes.music-data/2` and `ournotes-deck.chart-stats/3` statistics | +| `provenance` | `format`; `region` `tw`; `master.source` `api`; `master.version` equal to the snapshot's and its `MasterManifest.json`'s; every table's SHA-256 the manifest's, every decoded table read the one `index.json` lists; the song tables and the deck model's present; `deck.commit` the installed `ournotes-sim` release's; `exporter.version` the installed nnnotes; an APK version; a catalog SHA-256 (warning: the APK is another client version than the snapshot's) | | `counts` | no fewer songs and charts than the published file (warning: ids no longer in it) | | `deck` | native kinds, positive power, events and positions matching chart facts; explicit nominal `expectation` unless unplayable; replay seeds equal `[0]` without luck or the published prefix shared by luck charts; finite `[center, outward radius]` estimates, valid dimensions and deterministic counters; lottery estimates enclose zero outside luck ranges; complete expectation check decks within their bounds | | `scenarios` | exactly one deterministic Free Live `offSeeds` entry (seed 0, scalar score/weights and bounded check); five integer rank percentages per range; explicit best/Perfect scores and `rankBonusPerfect`; rank-weight dimensions and rank checks match the supported linear domain; nulls warn about unavailable rank or Free Live weights rather than becoming zero gains | @@ -195,7 +194,7 @@ legitimate drop (a song the game removed) stops the run: a person checks it, the The self-test runs in each build and locally in seconds, without the network: ``` -python -m pytest -q -p no:cacheprovider .github/scripts/test_music_data.py .github/scripts/test_jp_workflow.py .github/scripts/test_music_data_refresh.py .github/scripts/test_music_data_compression.py .github/scripts/test_prebuilt_source_identity.py +python -m pytest -q -p no:cacheprovider .github/scripts/test_music_data.py .github/scripts/test_jp_workflow.py .github/scripts/test_music_data_refresh.py .github/scripts/test_music_data_compression.py ``` with, optionally, `MUSIC_DATA_SCHEMA` (a schema file when the checkout has none), `MUSIC_DATA_PAGE` (an @@ -240,15 +239,9 @@ Repository variables: ## Before the first run -- **Native model and engines.** The native program cache is supplied by - [ournotes-deck#44](https://github.com/empty-sekai/ournotes-deck/pull/44). The exact source revision in - `rust/Cargo.toml` and `rust/Cargo.lock` must also be available as a model release with both replay and - recommendation WASM packages. Building this source pin locally is supported before that release exists; - production publication still requires matching engine commit and source identities. If the release uses a - different commit after merging, update the Cargo revision and lockfile package version and rebuild the extension. - The publisher selects `v` from that package version, so creating a new tag alone is insufficient. - Older engines, including an older release with the - same package version, must not be substituted or admitted by weakening the identity gate. +- **Native model and engines.** The deck model is the `ournotes-sim` release that the installed nnnotes pins; the + publisher downloads the replay and recommendation WASM packages of that same release (`v`), and nnnotes + rejects engines of any other release. - **Consumers.** Deploy readers for `nnnotes.music-data/2` and `ournotes-deck.chart-stats/3` before publishing new data. The default `MUSIC_DATA_PLAYER_REF` already selects the nominal-expectation consumer in [ournotes-player#18](https://github.com/empty-sekai/ournotes-player/pull/18); an explicit repository variable @@ -256,14 +249,14 @@ Repository variables: [moenotes#7](https://github.com/nichinichisou0609/moenotes/pull/7), stacked on the deck-goal repair in [moenotes#6](https://github.com/nichinichisou0609/moenotes/pull/6). - **Producer and publication.** This checkout publishes only the new nominal format and starts a fresh native - cache namespace; it does not migrate old Python cache records. Run the normal or prebuilt publication gates + cache namespace; it does not migrate old Python cache records. Run the publication gates with the matching engines and consumers. Set `MUSIC_DATA_PUBLISH` to `true` after the dry run passes. ## Notes -- **Versions.** A new ournotes-deck pin (`rust/Cargo.toml`, `rust/Cargo.lock`) or a new nnnotes commit in `src/`, - `rust/` or `pyproject.toml` reaches this fork with a sync, and the next run builds a new file. The deck - statistics may then differ: `provenance.deck.commit` and `build.json` name the commit. +- **Versions.** The workflows install the nnnotes release named in `.github/nnnotes.txt`. Raising it (a new + nnnotes release, which may pin a new ournotes-deck release) changes the build's inputs, and the next run builds a + new file. The deck statistics may then differ: `provenance.deck.commit` and `build.json` name the commit. - **The page and the data.** The page's modules are pinned by `MUSIC_DATA_PLAYER_REF`: changing that pin or its repository is now a build input and starts a new check. Use an immutable commit to keep the consumer identity stable. - **Byte identity.** A build from the same inputs gives the same bytes (the file is canonical); the jackets' @@ -272,8 +265,8 @@ Repository variables: uploaded as an artifact. # HTTP compression -Normal and validated-prebuilt publication uploads every JSON object at its existing `.json` URL with `Content-Type: application/json` and `Content-Encoding: gzip`. Compression uses gzip level 6 and `mtime=0`; engine JavaScript, WASM and images remain unchanged. Manifest SHA-256 values and byte counts always describe the decoded payload consumed by browser `fetch`. S3 metadata records `decoded-sha256`, `decoded-bytes`, `encoded-sha256` and `encoded-bytes` separately. Read-back gates verify both stored identity and decoded identity before advancing pointers. Python public-source reads explicitly decode the HTTP content encoding. +Publication uploads every JSON object at its existing `.json` URL with `Content-Type: application/json` and `Content-Encoding: gzip`. Compression uses gzip level 6 and `mtime=0`; engine JavaScript, WASM and images remain unchanged. Manifest SHA-256 values and byte counts always describe the decoded payload consumed by browser `fetch`. S3 metadata records `decoded-sha256`, `decoded-bytes`, `encoded-sha256` and `encoded-bytes` separately. Read-back gates verify both stored identity and decoded identity before advancing pointers. Python public-source reads explicitly decode the HTTP content encoding. -To re-encode an already published, verified snapshot without running Rust or generating a replacement model, dispatch `music-data.yml` with `reencode_only=true`, the target `region`, and `dry_run=false`. This path shares the normal/prebuilt region lock. It requires the current source identity, an unchanged published build marker, all gate attestations, the complete SHA-bound replay runtime and all 13 Snap label tables. It checks source and marker again before each read/write, first verifies the store/CDN using a fresh gzip probe, then re-encodes the JSON payloads and externally verifies headers, stored identity and decoded identity. Manifest, music-data and build-marker decoded bytes never change. The encoding report records every object and total encoded/decoded size; it contains no game payloads or credentials. +To re-encode an already published, verified snapshot without running Rust or generating a replacement model, dispatch `music-data.yml` with `reencode_only=true`, the target `region`, and `dry_run=false`. This path shares the region lock. It requires the current source identity, an unchanged published build marker, all gate attestations, the complete SHA-bound replay runtime and all 13 Snap label tables. It checks source and marker again before each read/write, first verifies the store/CDN using a fresh gzip probe, then re-encodes the JSON payloads and externally verifies headers, stored identity and decoded identity. Manifest, music-data and build-marker decoded bytes never change. The encoding report records every object and total encoded/decoded size; it contains no game payloads or credentials. JP builds perform an actual CDN-authentication preflight before Rust/WASM allocation. A refused `Version` call remains a failed JP update; it cannot substitute another region or bypass authenticated snapshot checks. Report artifacts use canonical workspace paths so `upload-artifact@v7` can retain diagnostics. diff --git a/.github/STORY_SITE.md b/.github/STORY_SITE.md index d399636..51ec8d1 100644 --- a/.github/STORY_SITE.md +++ b/.github/STORY_SITE.md @@ -8,7 +8,7 @@ lacks with this repository's `nnnotes web --story` and uploads them to the bucke published internationally and only builds stories missing from both sites. MoeNotes merges the two indexes and plays shared advIds from the international site, preserving all its text languages. It only adds: a run never deletes anything from the bucket. Its helper steps are in -`.github/scripts/`; nothing outside `.github/` differs from upstream, so the fork syncs with it as before. +`.github/scripts/`; it installs the nnnotes release named in `.github/nnnotes.txt`. ## A run @@ -95,11 +95,12 @@ Repository variables (optional; the defaults are the StarMoe site): `STORY_S3_EN CDN authentication. Generated VPN profiles use connection fields and certificates, discarding supplied scripts and routes. The helpers are `jp_vpngate.py` and `story_site.py jp-check`. - **The player version.** The site's player pages come from `STORY_PLAYER_REF`, and its data must be what that player - reads. After a player release that changes the data format, sync this fork with upstream nnnotes and set - `STORY_PLAYER_REF` to the matching player; new stories are then written for it. Stories already published are + reads. After a player release that changes the data format, raise `.github/nnnotes.txt` to the nnnotes release + that writes it and set `STORY_PLAYER_REF` to the matching player; new stories are then written for it. Stories already published are not rebuilt (run with `stories` and `force` for that). - **The APK.** Google Play serves the current game version. When nnnotes' type trees do not match it, the build stops - naming the class and the Unity version: sync the fork with upstream once nnnotes supports that version. + naming the class and the Unity version: raise `.github/nnnotes.txt` once an nnnotes release supports that + version. - **Bytes.** A rebuild on the runner is not byte-identical to a build elsewhere: the AAC encoder (ffmpeg) and the PNG encoder (zlib) differ between machines. The pixels and every JSON value are the same, so new stories and the published ones fit together; only a forced rebuild re-uploads such files. diff --git a/.github/nnnotes.txt b/.github/nnnotes.txt new file mode 100644 index 0000000..53fa9c1 --- /dev/null +++ b/.github/nnnotes.txt @@ -0,0 +1,3 @@ +# The nnnotes release the publishing workflows install: music-data with its deck model (the extra deck, which +# pins one ournotes-deck release) and the story site with the open fonts (the extra fonts). +nnnotes[deck,fonts]==0.2.1 diff --git a/.github/scripts/music_data.py b/.github/scripts/music_data.py index 889cda8..a0d8a20 100755 --- a/.github/scripts/music_data.py +++ b/.github/scripts/music_data.py @@ -4,8 +4,8 @@ (music-data.json, jackets/, archive/, build.json) for the chart data page of ournotes-player (examples/songs). plan the inputs of a build (the master data snapshot of $MASTERDATA_REGION in index.json, the - deck commit rust/Cargo.lock pins, the last nnnotes commit of src/, rust/ and pyproject.toml, - RECIPE) against those of the published build.json; GitHub output `build`: true when they + installed nnnotes version and the ournotes-deck release of its deck model, RECIPE) against + those of the published build.json; GitHub output `build`: true when they differ, nothing is published or $FORCE is true master OUT every file of the snapshot of $MASTERDATA_REGION into OUT (SHA-256 checked against index.json, MasterManifest.json included) and its index entry as OUT.snapshot.json @@ -57,8 +57,6 @@ RECIPE = 8 FILE, FILE_BR, MARKER, JACKETS, ARCHIVE = "music-data.json", "music-data.json.br", "build.json", "jackets/", "archive/" MANIFEST = "MasterManifest.json" -SOURCE_PATHS = ("src", "rust", "pyproject.toml") # nnnotes' code: the commit that last changed one of them -SCHEMA = Path("docs/schema/music-data.schema.json") SMOKE = Path(__file__).resolve().parent / "music_data_smoke.mjs" ARCHIVE_CACHE = story_site.ASSET_CACHE # archive//.json: content-addressed RUNTIME_CACHE = story_site.ASSET_CACHE # replay//...: content-addressed @@ -89,8 +87,8 @@ REPLAY_LABEL_TABLES = ("MasterSupportSkill", "MasterSupportSkillEffect", "MasterGekisouSupportSkill", "MasterGekisouSupportSkillEffect", "MasterText", "MasterSkillConditionSet", "MasterSkillCondition", "MasterSkillCumulativeCondition", "MasterSkillTarget", "MasterCharacter", "MasterMemberCard", "MasterSupportCard", "MasterBand") -# the web WASM packages (replay, recommendation) of the ournotes-deck release of the pinned deck model: the release -# tagged v of the ournotes-sim package in rust/Cargo.lock +# the web WASM packages (replay, recommendation) of the ournotes-deck release of the installed deck model: the release +# tagged v of the installed ournotes-sim DECK_RELEASES = "https://github.com/empty-sekai/ournotes-deck/releases/download" ENGINES = ("replay", "recommend") SHA256 = re.compile(r"[0-9a-f]{64}") @@ -156,44 +154,34 @@ def exists(key: str) -> bool: # ---------------------------------------------------------------- the inputs of a build -def deck_package(root: Path = Path(".")) -> tuple[str, str]: - """The ournotes-sim package nnnotes builds its deck model with, in rust/Cargo.lock: its version and its - ournotes-deck commit.""" - lock = root / "rust" / "Cargo.lock" - if not lock.is_file(): - fail("no rust/Cargo.lock: this nnnotes has no music-data command with the deck model (sync the fork with " - "upstream, .github/MUSIC_DATA.md)") - for block in lock.read_text(encoding="utf-8").split("[[package]]"): - if re.search(r'^name = "ournotes-sim"$', block, re.M): - version = re.search(r'^version = "([^"]+)"$', block, re.M) - m = re.search(r'^source = "git\+[^"#]*#([0-9a-f]{40})"$', block, re.M) - if version and m: - return version.group(1), m.group(1) - fail("rust/Cargo.lock has no ournotes-sim git commit") - - -def deck_commit(root: Path = Path(".")) -> str: - """The ournotes-deck commit nnnotes builds its deck model with.""" - return deck_package(root)[1] - - -def nnnotes_commit(root: Path = Path(".")) -> str: - """The commit that last changed nnnotes' code (SOURCE_PATHS); the checkout needs its history.""" - def git(*args): - return subprocess.run(["git", "-C", str(root), *args], capture_output=True, text=True) - if git("rev-parse", "--is-shallow-repository").stdout.strip() != "false": - fail("the checkout is shallow: the nnnotes commit needs the history (actions/checkout fetch-depth: 0)") - commit = git("log", "-1", "--format=%H", "--", *SOURCE_PATHS).stdout.strip() - if not COMMIT.fullmatch(commit): - fail("no commit changed src/, rust/ or pyproject.toml") - return commit - - -def require_decoded_master(root: Path = Path(".")) -> None: - cli = root / "src" / "nnnotes" / "cli.py" - if "--decoded-master" not in cli.read_text(encoding="utf-8"): - fail("this nnnotes has no `music-data --decoded-master` (MetaSekaiLab/nnnotes#6): sync the fork with " - "upstream (.github/MUSIC_DATA.md)") +def deck_package() -> tuple[str, str]: + """The ournotes-deck release of the installed deck model (the package ournotes-sim, nnnotes' extra deck): its + version and its commit.""" + try: + import ournotes_sim + except ImportError: + fail("the deck model is not installed: install nnnotes with its extra deck (.github/nnnotes.txt)") + info = ournotes_sim.info() + if not COMMIT.fullmatch(str(info.get("commit"))): + fail(f"ournotes-sim {info.get('version')} is not a release build: it names no commit") + return info["version"], info["commit"] + + +def deck_commit() -> str: + """The ournotes-deck commit of the installed deck model.""" + return deck_package()[1] + + +def nnnotes_version() -> str: + """The version of the installed nnnotes.""" + import nnnotes + return nnnotes.__version__ + + +def schema() -> dict: + """The JSON Schema of the music data, as the installed nnnotes ships it.""" + from nnnotes import schemas + return schemas.load("music-data") def snapshot_identity(region: str, entry: dict, files: dict) -> dict: @@ -211,13 +199,13 @@ def snapshot_identity(region: str, entry: dict, files: dict) -> dict: "decodedFilesSha256": sha256(json.dumps(files, sort_keys=True, separators=(",", ":")).encode())} -def inputs(entry: dict, root: Path = Path("."), *, files: dict | None = None) -> dict: +def inputs(entry: dict, *, files: dict | None = None) -> dict: """What a build is made of: the master data snapshot, the deck model, nnnotes and this script.""" identity = snapshot_identity(env("MASTERDATA_REGION"), entry, files) if files is not None else { "masterRegion": env("MASTERDATA_REGION"), "masterVersion": entry.get("version"), "resourceVersion": entry.get("resource_version"), "clientVersion": entry.get("client_version"), "resourceHash": entry.get("resource_hash"), "manifestSha256": entry.get("manifest_sha256")} - return {**identity, "deckCommit": deck_commit(root), "nnnotesCommit": nnnotes_commit(root), + return {**identity, "deckCommit": deck_commit(), "nnnotesVersion": nnnotes_version(), "playerRepository": os.environ.get("PLAYER_REPOSITORY"), "playerRef": os.environ.get("PLAYER_REF"), "recipe": RECIPE} @@ -286,7 +274,6 @@ def short(v) -> str: def cmd_plan() -> None: require_distinct_region_prefixes() _, region = story_site.master_index() - require_decoded_master() now = inputs(region.get("entry") or {}, files=region["files"]) raw = published(MARKER) marker = json.loads(raw) if raw else None @@ -298,9 +285,9 @@ def cmd_plan() -> None: build = force or bool(changed) summary(f"### Music data\n\n- master: {now['masterVersion']} of {now['masterRegion']} (resource " f"{now['resourceVersion']}, client {now['clientVersion']}); deck {short(now['deckCommit'])}; nnnotes " - f"{short(now['nnnotesCommit'])}; recipe {RECIPE}\n- published: " + f"{now['nnnotesVersion']}; recipe {RECIPE}\n- published: " + (f"master {have.get('masterVersion')}, deck {short(have.get('deckCommit'))}, nnnotes " - f"{short(have.get('nnnotesCommit'))}, built {marker.get('builtAt')}" if isinstance(have, dict) + f"{have.get('nnnotesVersion')}, built {marker.get('builtAt')}" if isinstance(have, dict) else "nothing") + "\n- this run: " + ("builds" + (" (force)" if force else "") + (f", changed: {', '.join(changed)}" if changed and have else "") @@ -345,8 +332,8 @@ def cmd_build(out: str) -> None: usage = subprocess.run(nnnotes + ["music-data", "--help"], capture_output=True, text=True).stdout if any(flag not in usage for flag in ("--decoded-master", "--replay-dir", "--replay-engine", "--recommend-engine", "--stats-cache")): - fail("the installed nnnotes lacks decoded-master/replay/recommend export or the statistics cache (sync the " - "fork with upstream)") + fail("the installed nnnotes lacks decoded-master/replay/recommend export or the statistics cache (raise the " + "version in .github/nnnotes.txt)") engine, recommend = release_engines(o) if env("MASTERDATA_REGION") == "jp": import jp_vpngate @@ -372,9 +359,9 @@ def cmd_build(out: str) -> None: def release_engines(out: Path) -> tuple[Path, ...]: - """Download the web WASM packages (ENGINES) of the ournotes-deck release of the pinned deck model into OUT, each - checked against the release's SHA256SUMS; returns their paths. nnnotes checks each package's build-info.json - against the deck model it is built with.""" + """Download the web WASM packages (ENGINES) of the ournotes-deck release of the installed deck model into OUT, + each checked against the release's SHA256SUMS; returns their paths. nnnotes checks each package's build-info.json + against the installed deck model.""" version, _ = deck_package() base = f"{DECK_RELEASES}/v{version}" sums = {} @@ -480,7 +467,7 @@ class Context: deck_commit: str | None = None nnnotes_version: str | None = None jackets: Path | None = None - schema: Path | None = None + schema: dict | None = None # the music data's JSON Schema published: bytes | None = None # the published music-data.json (None: nothing is published) page: Path | None = None # examples/songs of ournotes-player file: Path | None = None # the file on disk, for the smoke test @@ -548,15 +535,11 @@ def gate_schema(doc, ctx: Context, g: Gate): if ctx.schema is None: g.note = "skipped" return - if not ctx.schema.is_file(): - g.fail(f"no JSON Schema {ctx.schema}") - return import jsonschema - schema = json.loads(ctx.schema.read_text(encoding="utf-8")) - validator = jsonschema.Draft202012Validator(schema) + validator = jsonschema.Draft202012Validator(ctx.schema) for e in sorted(validator.iter_errors(doc), key=lambda e: list(map(str, e.absolute_path))): g.fail(f"{'/'.join(map(str, e.absolute_path)) or '(root)'}: {e.message[:160]}") - g.note = ctx.schema.as_posix() + g.note = ctx.schema.get("$id", "") def gate_provenance(doc, ctx: Context, g: Gate): @@ -603,7 +586,7 @@ def gate_provenance(doc, ctx: Context, g: Gate): if deck.get("name") != "ournotes-deck" or not COMMIT.fullmatch(str(deck.get("commit"))): g.fail("provenance.deck names no ournotes-deck commit") elif ctx.deck_commit is not None and deck["commit"] != ctx.deck_commit: - g.fail(f"deck commit {deck['commit'][:12]}, rust/Cargo.lock pins {ctx.deck_commit[:12]}") + g.fail(f"deck commit {deck['commit'][:12]}, the installed deck model is {ctx.deck_commit[:12]}") ex = p.get("exporter") or {} if ex.get("name") != "nnnotes": g.fail(f"exporter {ex.get('name')!r}") @@ -1175,7 +1158,7 @@ def cmd_check(out: str, master: str, page: str) -> None: snapshot = json.loads(snapshot_file(m).read_text(encoding="utf-8")) ctx = Context(region=env("NNNOTES_CATALOG_REGION"), language=os.environ.get("NNNOTES_CATALOG_LANGUAGE"), master=m, snapshot=snapshot, deck_commit=deck_commit(), nnnotes_version=nnnotes.__version__, - jackets=o / "jackets", schema=SCHEMA, published=published(FILE), page=Path(page), file=o / FILE) + jackets=o / "jackets", schema=schema(), published=published(FILE), page=Path(page), file=o / FILE) report = gates(raw, ctx) replay_gate = {"gate":"replay", "passed":True,"failures":[],"failureCount":0, "warnings":[],"warningCount":0,"note":""} diff --git a/.github/scripts/prebuilt_publish.py b/.github/scripts/prebuilt_publish.py deleted file mode 100644 index 6ec8054..0000000 --- a/.github/scripts/prebuilt_publish.py +++ /dev/null @@ -1,277 +0,0 @@ -#!/usr/bin/env python3 -"""DRAFT: package/unpack a hash-anchored final export; never builds or publishes. - -Install beside music_data.py only after review. Existing music_data.py remains the -authority for master/check/publish. No credentials are read by pack/unpack/verify. -""" -from __future__ import annotations - -import argparse -import hashlib -import importlib.util -import json -import os -import re -import stat -import subprocess -import sys -import zipfile -from pathlib import Path, PurePosixPath - -FORMAT = "ournotes.prebuilt-music-data/1" -SHA = re.compile(r"[0-9a-f]{64}") -COMMIT = re.compile(r"[0-9a-f]{40}") -REPOSITORY = re.compile(r"[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+") -MAX_ARCHIVE = 512 * 1024 * 1024 -MAX_EXPANDED = 1024 * 1024 * 1024 - - -def require(condition, message): - if not condition: - raise ValueError(message) - - -def digest(path): - h = hashlib.sha256() - with open(path, "rb") as stream: - for chunk in iter(lambda: stream.read(1024 * 1024), b""): - h.update(chunk) - return h.hexdigest() - - -def git(root, *args): - return subprocess.check_output(["git", "-C", str(root), *args], text=True).strip() - - -def publisher(path): - spec = importlib.util.spec_from_file_location("music_data", path) - module = importlib.util.module_from_spec(spec) - sys.modules[spec.name] = module - spec.loader.exec_module(module) - return module - - -def safe_name(name): - require(isinstance(name, str) and name and not any(c in name for c in ("\\", ":", "\x00")), - "invalid archive path") - path = PurePosixPath(name) - require(not path.is_absolute() and all(p not in ("", ".", "..") for p in name.split("/")), - "archive path escapes extraction root") - require(name == "prebuilt.json" or name.startswith("data/"), "unexpected archive root") - return path - - -def inventory_entry(path): - return {"sha256": digest(path), "bytes": path.stat().st_size} - - -def validate_exporter_identity(root, proof, verify_raw_checkout=False): - """Prove committed blobs and explicitly reconstruct producer byte identity in memory. - - Linux can use canonical bytes; a Windows checkout may declare observed - LF-to-CRLF conversions. CI reconstructs only those conversions in memory. - """ - identity = proof.get("exporterSourceIdentity", {}) - require(identity.get("format") == "ournotes.exporter-source-identity/1", "missing exporter source identity") - paths = git(root, "ls-tree", "-r", "--name-only", "HEAD", "--", "src/nnnotes").splitlines() - paths = sorted(p for p in paths if p.endswith(".py")) - require(paths and len(paths) == identity.get("fileCount"), "exporter Python source count mismatch") - blobs = {p: subprocess.check_output(["git", "-C", str(root), "show", "HEAD:" + p]) for p in paths} - def tree_hash(items): - values = {p: hashlib.sha256(raw).hexdigest() for p, raw in items.items()} - return hashlib.sha256(json.dumps(values, sort_keys=True, separators=(",", ":")).encode()).hexdigest() - canonical = tree_hash(blobs) - require(canonical == identity.get("canonicalGitTreeSha256"), "committed exporter source tree mismatch") - normalized = {p: hashlib.sha256(raw.replace(b"\r\n", b"\n")).hexdigest() for p, raw in blobs.items()} - normalized_tree = hashlib.sha256(json.dumps(normalized, sort_keys=True, separators=(",", ":")).encode()).hexdigest() - transforms = identity.get("checkoutTransforms") - require(isinstance(transforms, list), "missing producer checkout transforms") - seen = set() - for transform in transforms: - require(isinstance(transform, dict) and set(transform) == {"path", "operation"} - and transform["operation"] == "lf-to-crlf" and transform["path"] in blobs - and transform["path"] not in seen, "unexpected producer checkout transform") - path = transform["path"] - seen.add(path) - source = blobs[path] - require(b"\r" not in source, "canonical transformed source is not LF-only") - blobs[path] = source.replace(b"\n", b"\r\n") - raw_tree = tree_hash(blobs) - require(raw_tree == identity.get("rawTreeSha256") == proof.get("exporterSourceTreeSha256"), - "reconstructed producer byte tree mismatch") - require(proof.get("exporterCommit") == git(root, "rev-parse", "HEAD"), "generation exporter commit mismatch") - if verify_raw_checkout: - actual = {p: (root / p).read_bytes() for p in paths} - require(tree_hash(actual) == raw_tree, "actual pack producer byte tree differs from generation") - return {"format": "ournotes.exporter-normalized-source/1", "scope": "committed Python files; CRLF-to-LF only", - "fileCount": len(paths), "lfNormalizedTreeSha256": normalized_tree, "lfNormalizedFiles": normalized} - - -def validate_generation(proof, doc, main_sha): - require(proof.get("format") == "ournotes.final-generation/1", "unknown generation proof") - require(proof.get("sourceStableDuringRun") is True and proof.get("exporterStableDuringRun") is True, - "producer source changed during generation") - require(proof.get("unmetTargets") == 0, "generation contains unmet precision targets") - require(proof.get("musicDataSha256") == main_sha, "generation/main SHA mismatch") - require(proof.get("modelCommit") == doc["provenance"]["deck"]["commit"], "generation/model mismatch") - require(proof.get("manifestSha256") == doc["replay"]["sha256"], "generation/runtime mismatch") - require(proof.get("songs") == len(doc["songs"]), "generation/song count mismatch") - require(proof.get("charts") == sum(len(s["charts"]) for s in doc["songs"]), "generation/chart count mismatch") - require(proof.get("engine", {}).get("workingTreeDirty") is False, "generation engine is not explicitly clean") - - -def pack(args): - require(REPOSITORY.fullmatch(args.producer_repository), "invalid producer repository") - require(REPOSITORY.fullmatch(args.player_repository), "invalid player repository") - require(COMMIT.fullmatch(args.player_commit), "player must be pinned to a full commit") - root, data, jackets = args.producer.resolve(), args.data.resolve(), args.jackets.resolve() - require(git(root, "rev-parse", "--is-shallow-repository") == "false", "producer needs full history") - require(not git(root, "status", "--porcelain", "--untracked-files=all"), "producer checkout is dirty") - producer_commit = git(root, "rev-parse", "HEAD") - require(COMMIT.fullmatch(producer_commit), "invalid producer commit") - module = publisher(args.publisher.resolve()) - require(module.nnnotes_commit(root) == producer_commit, "producer HEAD must be its latest source commit") - doc = json.loads((data / "music-data.json").read_bytes()) - main_sha = digest(data / "music-data.json") - require(module.deck_commit(root) == doc["provenance"]["deck"]["commit"], "producer pins another model") - proof = json.loads(args.generation.read_bytes()) - validate_generation(proof, doc, main_sha) - normalized_source = validate_exporter_identity(root, proof, verify_raw_checkout=True) - resources = module.replay_resources(data, doc) - require(resources, "runtime resources are mandatory") - files = {"data/music-data.json": data / "music-data.json"} - for path in resources: - files["data/" + path.relative_to(data).as_posix()] = path - for song in doc["songs"]: - jacket = song["jacket"] - require(isinstance(jacket, str) and re.fullmatch(r"[A-Za-z0-9_.-]+", jacket), "invalid jacket name") - path = jackets / f"{jacket}.webp" - require(path.is_file() and not path.is_symlink() and path.stat().st_size > 0, f"missing jacket {jacket}") - files[f"data/jackets/{jacket}.webp"] = path - for name, path in files.items(): - safe_name(name) - require(path.is_file() and not path.is_symlink(), f"payload is not a regular file: {name}") - require("data/check.json" not in files and "data/build.json" not in files, "fresh CI check/marker required") - metadata = { - "format": FORMAT, - "producer": {"repository": args.producer_repository, "commit": producer_commit, - "normalizedSourceIdentity": normalized_source}, - "player": {"repository": args.player_repository, "commit": args.player_commit}, - "deckCommit": module.deck_commit(root), - "masterVersion": doc["provenance"]["master"]["version"], - "musicDataSha256": main_sha, - "replayManifestSha256": doc["replay"]["sha256"], - "generation": proof, - "files": {name: inventory_entry(path) for name, path in sorted(files.items())}, - } - require(sum(item["bytes"] for item in metadata["files"].values()) < MAX_EXPANDED, "bundle too large") - args.output.parent.mkdir(parents=True, exist_ok=True) - require(not args.output.exists(), "refusing to replace an existing bundle") - with zipfile.ZipFile(args.output, "x", compression=zipfile.ZIP_DEFLATED, compresslevel=6) as archive: - archive.writestr("prebuilt.json", json.dumps(metadata, indent=2) + "\n") - for name, path in sorted(files.items()): - archive.write(path, name) - require(args.output.stat().st_size <= MAX_ARCHIVE, "bundle archive too large") - checksum = digest(args.output) - args.output.with_suffix(args.output.suffix + ".sha256").write_text(f"{checksum} {args.output.name}\n", encoding="utf8") - print(json.dumps({"bundle": args.output.name, "sha256": checksum, "bytes": args.output.stat().st_size, - "payloadFiles": len(files), "producerCommit": producer_commit, "playerCommit": args.player_commit})) - - -def unpack(args): - require(SHA.fullmatch(args.sha256), "expected bundle SHA must be lowercase 64 hex") - require(args.archive.stat().st_size <= MAX_ARCHIVE and digest(args.archive) == args.sha256, - "bundle SHA/size mismatch") - require(not args.out.exists(), "extraction destination must be new") - with zipfile.ZipFile(args.archive) as archive: - entries = archive.infolist() - names = [i.filename for i in entries] - require(len(names) == len(set(names)), "duplicate archive entry") - require(len(entries) <= 2000 and sum(i.file_size for i in entries) <= MAX_EXPANDED, "archive exceeds bounds") - for entry in entries: - safe_name(entry.filename) - require(not entry.is_dir() and not stat.S_ISLNK(entry.external_attr >> 16), "non-file archive entry") - require("prebuilt.json" in names, "bundle has no identity metadata") - require(archive.getinfo("prebuilt.json").file_size <= 1024 * 1024, "metadata too large") - meta = json.loads(archive.read("prebuilt.json")) - require(meta.get("format") == FORMAT, "unknown bundle format") - require(meta["producer"]["repository"] == args.producer_repository, "unexpected producer repository") - for key in ("producer", "player"): - require(REPOSITORY.fullmatch(meta[key]["repository"]) and COMMIT.fullmatch(meta[key]["commit"]), - f"invalid {key} identity") - expected = set(meta["files"]) | {"prebuilt.json"} - require(set(names) == expected, "archive entries differ from payload inventory") - require("data/check.json" not in names and "data/build.json" not in names, - "bundle must not supply CI gate results or outer build marker") - # All identities, names and sizes are validated before creating any files. - for name, identity in meta["files"].items(): - safe_name(name) - require(SHA.fullmatch(identity["sha256"]) and type(identity["bytes"]) is int - and identity["bytes"] == archive.getinfo(name).file_size, "invalid payload inventory") - args.out.mkdir(parents=True) - for entry in entries: - dest = args.out / entry.filename - dest.parent.mkdir(parents=True, exist_ok=True) - dest.write_bytes(archive.read(entry)) - if entry.filename != "prebuilt.json": - require(inventory_entry(dest) == meta["files"][entry.filename], "payload hash mismatch") - if os.environ.get("GITHUB_OUTPUT"): - with open(os.environ["GITHUB_OUTPUT"], "a", encoding="utf8") as stream: - for key in ("producer", "player"): - stream.write(f"{key}_repository={meta[key]['repository']}\n{key}_commit={meta[key]['commit']}\n") - print(json.dumps({"unpacked": True, "payloadFiles": len(meta["files"]), "sha256": args.sha256})) - - -def verify(args): - meta = json.loads((args.bundle / "prebuilt.json").read_bytes()) - root = args.producer.resolve() - require(meta["format"] == FORMAT, "unknown identity metadata") - require(git(root, "rev-parse", "HEAD") == meta["producer"]["commit"], "producer checkout mismatch") - require(not git(root, "status", "--porcelain", "--untracked-files=all"), "producer checkout is dirty") - module = publisher(args.publisher.resolve()) - require(module.nnnotes_commit(root) == meta["producer"]["commit"], "latest NN source commit mismatch") - require(module.deck_commit(root) == meta["deckCommit"], "Cargo.lock model mismatch") - sys.path.insert(0, str(root / "src")) - import nnnotes - require(Path(nnnotes.__file__).resolve() == root / "src/nnnotes/__init__.py", "nnnotes import is not genuine producer source") - doc = json.loads((args.bundle / "data/music-data.json").read_bytes()) - require(nnnotes.__version__ == doc["provenance"]["exporter"]["version"], "exporter version mismatch") - for name, identity in meta["files"].items(): - require(inventory_entry(args.bundle / name) == identity, f"changed payload {name}") - validate_generation(meta["generation"], doc, digest(args.bundle / "data/music-data.json")) - normalized_source = validate_exporter_identity(root, meta["generation"]) - require(normalized_source == meta["producer"].get("normalizedSourceIdentity"), "per-file LF-normalized source differs") - require(doc["provenance"]["master"]["version"] == meta["masterVersion"], "master identity mismatch") - require(module.replay_resources(args.bundle / "data", doc), "missing runtime") - print(json.dumps({"verified": True, "producerCommit": meta["producer"]["commit"], "deckCommit": meta["deckCommit"]})) - - -def main(): - parser = argparse.ArgumentParser(description=__doc__) - sub = parser.add_subparsers(dest="cmd", required=True) - p = sub.add_parser("pack") - for flag in ("producer", "data", "jackets", "generation", "publisher", "output"): - p.add_argument("--" + flag, type=Path, required=True) - p.add_argument("--producer-repository", default="MetaSekaiLab/nnnotes") - p.add_argument("--player-repository", default="empty-sekai/ournotes-player") - p.add_argument("--player-commit", required=True) - p.set_defaults(func=pack) - p = sub.add_parser("unpack") - p.add_argument("--archive", type=Path, required=True) - p.add_argument("--out", type=Path, required=True) - p.add_argument("--sha256", required=True) - p.add_argument("--producer-repository", default="MetaSekaiLab/nnnotes") - p.set_defaults(func=unpack) - p = sub.add_parser("verify") - for flag in ("bundle", "producer", "publisher"): - p.add_argument("--" + flag, type=Path, required=True) - p.set_defaults(func=verify) - args = parser.parse_args() - try: - args.func(args) - except (ValueError, KeyError, OSError, zipfile.BadZipFile) as error: - parser.exit(1, f"prebuilt_publish: {error}\n") - - -if __name__ == "__main__": - main() diff --git a/.github/scripts/prebuilt_publish_download.py b/.github/scripts/prebuilt_publish_download.py deleted file mode 100644 index 6ee1477..0000000 --- a/.github/scripts/prebuilt_publish_download.py +++ /dev/null @@ -1,57 +0,0 @@ -#!/usr/bin/env python3 -"""DRAFT: read a same-repository draft Release asset with GH_TOKEN via gh. - -The expected SHA is a dispatch input approved independently of release metadata. -Tokens and temporary redirect URLs are never printed or written by this script. -""" -import argparse -import hashlib -import json -import re -import subprocess -from pathlib import Path - - -def api(endpoint, output=None): - command = ["gh", "api", endpoint] - if output is None: - return json.loads(subprocess.check_output(command)) - with output.open("xb") as stream: - subprocess.run(command + ["-H", "Accept: application/octet-stream"], stdout=stream, check=True) - - -def main(): - p = argparse.ArgumentParser(description=__doc__) - p.add_argument("--repository", required=True) - p.add_argument("--release-id", required=True) - p.add_argument("--asset-name", required=True) - p.add_argument("--sha256", required=True) - p.add_argument("--out", type=Path, required=True) - a = p.parse_args() - if not (re.fullmatch(r"[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+", a.repository) - and re.fullmatch(r"[0-9]+", a.release_id) - and re.fullmatch(r"[A-Za-z0-9_.-]+\.zip", a.asset_name) - and re.fullmatch(r"[0-9a-f]{64}", a.sha256)): - p.error("invalid repository, release ID, asset name or SHA") - release = api(f"repos/{a.repository}/releases/{a.release_id}") - if release.get("draft") is not True: - p.error("asset must belong to a draft release") - assets = [v for v in release["assets"] if v["name"] == a.asset_name] - checksums = [v for v in release["assets"] if v["name"] == a.asset_name + ".sha256"] - if len(assets) != 1 or len(checksums) != 1 or assets[0].get("state") != "uploaded": - p.error("exactly one completed ZIP and checksum asset are required") - if not (0 < assets[0]["size"] <= 512 * 1024 * 1024 and checksums[0]["size"] < 1024): - p.error("unexpected release asset size") - a.out.parent.mkdir(parents=True, exist_ok=True) - api(f"repos/{a.repository}/releases/assets/{assets[0]['id']}", a.out) - checksum_path = a.out.with_suffix(a.out.suffix + ".sha256") - api(f"repos/{a.repository}/releases/assets/{checksums[0]['id']}", checksum_path) - actual = hashlib.sha256(a.out.read_bytes()).hexdigest() - checksum = checksum_path.read_text(encoding="utf8").strip() - if actual != a.sha256 or checksum != f"{a.sha256} {a.asset_name}" or a.out.stat().st_size != assets[0]["size"]: - p.error("release asset/checksum differ from the independently pinned bundle SHA") - print(json.dumps({"draftRelease": True, "asset": a.asset_name, "sha256": actual, "bytes": assets[0]["size"]})) - - -if __name__ == "__main__": - main() diff --git a/.github/scripts/test_jp_workflow.py b/.github/scripts/test_jp_workflow.py index 67edd52..b955a83 100644 --- a/.github/scripts/test_jp_workflow.py +++ b/.github/scripts/test_jp_workflow.py @@ -90,8 +90,8 @@ def test_reject_mixed_master_and_catalog(monkeypatch): def test_asset_hash_changes_build_inputs(monkeypatch): monkeypatch.setenv('MASTERDATA_REGION', 'jp') - monkeypatch.setattr(music_data, 'deck_commit', lambda root: 'a' * 40) - monkeypatch.setattr(music_data, 'nnnotes_commit', lambda root: 'b' * 40) + monkeypatch.setattr(music_data, 'deck_commit', lambda: 'a' * 40) + monkeypatch.setattr(music_data, 'nnnotes_version', lambda: '0.2.0') a = music_data.inputs({'version': 'v', 'resource_hash': 'a' * 32}) b = music_data.inputs({'version': 'v', 'resource_hash': 'b' * 32}) assert a != b and a['resourceHash'] == 'a' * 32 diff --git a/.github/scripts/test_music_data.py b/.github/scripts/test_music_data.py index e231c62..bc46c99 100644 --- a/.github/scripts/test_music_data.py +++ b/.github/scripts/test_music_data.py @@ -3,7 +3,7 @@ python -m pytest -q -p no:cacheprovider .github/scripts/test_music_data.py -The JSON Schema gate uses docs/schema/music-data.schema.json (or $MUSIC_DATA_SCHEMA) when the checkout has it; the +The JSON Schema gate uses the schema the installed nnnotes ships (nnnotes.schemas) when jsonschema is installed; the page smoke test runs when $MUSIC_DATA_PAGE names ournotes-player's examples/songs; a configured page requires its modules and Node.js. Real files, when named: $MUSIC_DATA_SAMPLE (a current nnnotes.music-data/2 file: the content gates pass) and $MUSIC_DATA_OLD_SAMPLE (a legacy sampled file: the deck gate rejects its missing nominal expectations). @@ -33,11 +33,9 @@ CONTENT = ("deck", "scenarios", "finite", "references", "bgm") -def schema_path(): - p = Path(os.environ.get("MUSIC_DATA_SCHEMA") or ROOT / music_data.SCHEMA) - if not p.is_file(): - return None - return p if importlib.util.find_spec("jsonschema") else None +def schema(): + """The music data's JSON Schema, or None without jsonschema.""" + return music_data.schema() if importlib.util.find_spec("jsonschema") else None def page_path(): @@ -233,7 +231,7 @@ def context(tmp_path: Path, doc: dict, **kw) -> tuple[bytes, Context]: (tmp_path / "music-data.json").write_bytes(raw) ctx = Context(region="tw", language="zh-Hant", master=master, snapshot={"entry": {"version": "v-test", "client_version": "1.0.1"}, "files": files}, - deck_commit=DECK, nnnotes_version="0.1.2", jackets=jackets, schema=schema_path(), published=None, + deck_commit=DECK, nnnotes_version="0.1.2", jackets=jackets, schema=schema(), published=None, page=None, file=tmp_path / "music-data.json") for k, v in kw.items(): setattr(ctx, k, v) @@ -289,8 +287,8 @@ def test_the_sample_passes_every_gate(tmp_path): def test_schema(tmp_path): - if schema_path() is None: - pytest.skip("no docs/schema/music-data.schema.json (a fork not synced with upstream) or no jsonschema") + if schema() is None: + pytest.skip("no jsonschema") doc = sample() doc["songs"][0]["charts"][0]["level"] = "10" g = gate(run(tmp_path, doc), "schema") @@ -419,7 +417,7 @@ def unplayable(doc): (lambda d: d["provenance"]["master"]["tables"]["MasterText"].update(sha256="00" * 32), "provenance", "MasterText.sha256 is not"), (lambda d: d["provenance"]["master"]["tables"].pop("MasterBand"), "provenance", "lacks MasterBand"), - (lambda d: d["provenance"]["deck"].update(commit="ee" * 20), "provenance", "rust/Cargo.lock pins"), + (lambda d: d["provenance"]["deck"].update(commit="ee" * 20), "provenance", "the installed deck model is"), (lambda d: d["provenance"]["exporter"].update(version="0.0.9"), "provenance", "installed nnnotes"), (lambda d: d["provenance"]["client"].update(versionName=None), "provenance", "no APK version"), ]) @@ -534,16 +532,23 @@ def test_report_markdown(tmp_path): assert "- scenarios failure: chart 10 (100001 easy): offSeeds missing or not exactly one entry" in text_ -def test_deck_package_from_cargo_lock(tmp_path): - lock = tmp_path / "rust" / "Cargo.lock" - lock.parent.mkdir() - lock.write_text('[[package]]\nname = "pyo3"\nversion = "0.29.0"\n\n[[package]]\nname = "ournotes-sim"\n' - 'version = "0.0.1"\nsource = "git+https://github.com/empty-sekai/ournotes-deck?rev=' + "a" * 40 - + "#" + "b" * 40 + '"\n', encoding="utf-8") - assert music_data.deck_package(tmp_path) == ("0.0.1", "b" * 40) - assert music_data.deck_commit(tmp_path) == "b" * 40 - with pytest.raises(SystemExit, match="sync the fork"): - music_data.deck_commit(tmp_path / "nowhere") +@pytest.mark.parametrize("commit", ["b" * 40, None]) +def test_deck_package_is_the_installed_release(monkeypatch, commit): + module = type(sys)("ournotes_sim") + module.info = lambda: {"name": "ournotes-deck", "version": "0.0.1", "commit": commit} + monkeypatch.setitem(sys.modules, "ournotes_sim", module) + if commit is None: + with pytest.raises(SystemExit, match="not a release build"): + music_data.deck_package() + return + assert music_data.deck_package() == ("0.0.1", "b" * 40) + assert music_data.deck_commit() == "b" * 40 + + +def test_deck_package_needs_the_deck_model(monkeypatch): + monkeypatch.setitem(sys.modules, "ournotes_sim", None) + with pytest.raises(SystemExit, match="extra deck"): + music_data.deck_package() # ---------------------------------------------------------------- real files (when named) @@ -871,14 +876,11 @@ def test_runtime_payload_cannot_escape_to_a_shared_mutable_key(tmp_path, monkeyp assert not s3.log -def test_both_shared_songs_publishers_hold_one_global_lock_then_prebuilt_holds_region_lock(): +def test_the_songs_page_publisher_holds_the_global_lock(): workflows = Path(__file__).resolve().parent.parent / "workflows" - prebuilt = (workflows / "music-data-prebuilt.yml").read_text() page = (workflows / "songs-page.yml").read_text() - for workflow in (prebuilt, page): - assert "\nconcurrency:\n" in workflow - assert "\n group: songs-page-publication\n cancel-in-progress: false" in workflow - assert "\n concurrency:\n group: music-data-${{ inputs.region }}\n cancel-in-progress: false" in prebuilt + assert "\nconcurrency:\n" in page + assert "\n group: songs-page-publication\n cancel-in-progress: false" in page @pytest.mark.parametrize("defect", [None, "bytes", "region", "table_hash"]) diff --git a/.github/scripts/test_music_data_refresh.py b/.github/scripts/test_music_data_refresh.py index 36ddd27..33b8fb9 100644 --- a/.github/scripts/test_music_data_refresh.py +++ b/.github/scripts/test_music_data_refresh.py @@ -11,8 +11,8 @@ def test_same_version_table_change_rebuilds_but_check_timestamp_does_not(monkeypatch): monkeypatch.setenv("MASTERDATA_REGION", "jp") - monkeypatch.setattr(music_data, "deck_commit", lambda root: "a" * 40) - monkeypatch.setattr(music_data, "nnnotes_commit", lambda root: "b" * 40) + monkeypatch.setattr(music_data, "deck_commit", lambda: "a" * 40) + monkeypatch.setattr(music_data, "nnnotes_version", lambda: "0.2.0") entry = {"version": "same", "verified_at": "yesterday", "manifest_sha256": "c" * 64} files = {"MasterLiveScoreRank.json": "d" * 64, "MasterManifest.json": "c" * 64} before = music_data.inputs(entry, files=files) @@ -42,8 +42,8 @@ def test_snapshot_identity_requires_a_consistent_manifest(files, entry): def test_changed_consumer_pin_rebuilds(monkeypatch): monkeypatch.setenv("MASTERDATA_REGION", "jp") - monkeypatch.setattr(music_data, "deck_commit", lambda root: "a" * 40) - monkeypatch.setattr(music_data, "nnnotes_commit", lambda root: "b" * 40) + monkeypatch.setattr(music_data, "deck_commit", lambda: "a" * 40) + monkeypatch.setattr(music_data, "nnnotes_version", lambda: "0.2.0") monkeypatch.setenv("PLAYER_REPOSITORY", "example/player") monkeypatch.setenv("PLAYER_REF", "first") before = music_data.inputs({}) @@ -129,18 +129,15 @@ def test_workflows_isolate_regions_and_share_publication_lock(): root = Path(__file__).resolve().parents[1] / "workflows" wrapper = (root / "music-data.yml").read_text() normal = (root / "music-data-region.yml").read_text() - prebuilt = (root / "music-data-prebuilt.yml").read_text() assert "fail-fast: false" in wrapper and "'hk-tw-mo jp'" in wrapper assert "group: music-data-${{ matrix.region }}" in wrapper - assert "group: music-data-${{ inputs.region }}" in prebuilt - for workflow in (normal, prebuilt): - assert "MUSIC_DATA_MASTERDATA_REGION == 'jp' && vars.MUSIC_DATA_S3_PREFIX" in workflow - assert "MUSIC_DATA_MASTERDATA_REGION != 'jp' && vars.MUSIC_DATA_S3_PREFIX" in workflow - assert "MASTERDATA_REGION: ${{ inputs.region }}" in workflow - assert "MUSIC_DATA_TW_PREFIX_CHECK:" in workflow - assert "MUSIC_DATA_JP_PREFIX_CHECK:" in workflow - test_step = workflow[workflow.index("Gate self-test"):] - assert "test_music_data_refresh.py" in test_step.split("- name:", 1)[0] + assert "MUSIC_DATA_MASTERDATA_REGION == 'jp' && vars.MUSIC_DATA_S3_PREFIX" in normal + assert "MUSIC_DATA_MASTERDATA_REGION != 'jp' && vars.MUSIC_DATA_S3_PREFIX" in normal + assert "MASTERDATA_REGION: ${{ inputs.region }}" in normal + assert "MUSIC_DATA_TW_PREFIX_CHECK:" in normal + assert "MUSIC_DATA_JP_PREFIX_CHECK:" in normal + test_step = normal[normal.index("Gate self-test"):] + assert "test_music_data_refresh.py" in test_step.split("- name:", 1)[0] @pytest.mark.parametrize("tw,jp", [("jp/music-data", "jp/music-data"), diff --git a/.github/scripts/test_prebuilt_source_identity.py b/.github/scripts/test_prebuilt_source_identity.py deleted file mode 100644 index 4fbb010..0000000 --- a/.github/scripts/test_prebuilt_source_identity.py +++ /dev/null @@ -1,73 +0,0 @@ -"""Observe real committed/checkout bytes; no Windows-only identity is assumed.""" -import hashlib -import json -import subprocess - -import pytest - -import prebuilt_publish as prebuilt - - -def tree_hash(files): - entries = {path: hashlib.sha256(raw).hexdigest() for path, raw in files.items()} - return hashlib.sha256(json.dumps(entries, sort_keys=True, separators=(",", ":")).encode()).hexdigest() - - -@pytest.fixture -def producer(tmp_path): - files = {"src/nnnotes/a.py": b"VALUE = 1\n", "src/nnnotes/b.py": b"VALUE = 2\n"} - for path, raw in files.items(): - file = tmp_path / path; file.parent.mkdir(parents=True, exist_ok=True); file.write_bytes(raw) - for args in [("init", "-q"), ("config", "user.name", "Fixture"), ("config", "user.email", "fixture@example.test"), - ("config", "core.autocrlf", "false"), ("add", "src"), ("commit", "-qm", "fixture")]: - subprocess.run(["git", "-C", str(tmp_path), *args], check=True) - return tmp_path, files - - -def proof(root, files, transforms): - raw_files = dict(files) - for transform in transforms: - if transform.get("path") in raw_files: - raw_files[transform["path"]] = raw_files[transform["path"]].replace(b"\n", b"\r\n") - raw_sha = tree_hash(raw_files) - return {"exporterCommit": prebuilt.git(root, "rev-parse", "HEAD"), "exporterSourceTreeSha256": raw_sha, - "exporterSourceIdentity": {"format": "ournotes.exporter-source-identity/1", "fileCount": len(files), - "canonicalGitTreeSha256": tree_hash(files), "rawTreeSha256": raw_sha, "checkoutTransforms": transforms}} - - -def test_canonical_linux_checkout_has_no_transforms_and_actual_file_count(producer): - root, files = producer - normalized = prebuilt.validate_exporter_identity(root, proof(root, files, []), verify_raw_checkout=True) - assert normalized["fileCount"] == 2 - assert normalized["lfNormalizedTreeSha256"] == tree_hash(files) - - -def test_explicit_windows_line_endings_are_reconstructed_and_actual_bytes_verified(producer): - root, files = producer - changes = [{"path": "src/nnnotes/b.py", "operation": "lf-to-crlf"}] - evidence = proof(root, files, changes) - (root / "src/nnnotes/b.py").write_bytes(files["src/nnnotes/b.py"].replace(b"\n", b"\r\n")) - assert prebuilt.validate_exporter_identity(root, evidence, verify_raw_checkout=True)["lfNormalizedTreeSha256"] == tree_hash(files) - (root / "src/nnnotes/b.py").write_bytes(b"MODIFIED = 3\r\n") - with pytest.raises(ValueError, match="actual pack producer byte tree differs"): - prebuilt.validate_exporter_identity(root, evidence, verify_raw_checkout=True) - - -@pytest.mark.parametrize("changes", [[{"path": "src/nnnotes/missing.py", "operation": "lf-to-crlf"}], - [{"path": "src/nnnotes/a.py", "operation": "edit"}], [{"path": "src/nnnotes/a.py", "operation": "lf-to-crlf"}] * 2]) -def test_unknown_or_duplicate_transform_is_rejected(producer, changes): - root, files = producer - with pytest.raises(ValueError, match="unexpected producer checkout transform"): - prebuilt.validate_exporter_identity(root, proof(root, files, changes)) - - -def test_unobserved_transform_or_changed_committed_count_cannot_fake_identity(producer): - root, files = producer - evidence = proof(root, files, []) - evidence["exporterSourceIdentity"]["fileCount"] += 1 - with pytest.raises(ValueError, match="source count mismatch"): - prebuilt.validate_exporter_identity(root, evidence) - evidence = proof(root, files, []) - evidence["exporterSourceIdentity"]["rawTreeSha256"] = "a" * 64 - with pytest.raises(ValueError, match="producer byte tree mismatch"): - prebuilt.validate_exporter_identity(root, evidence) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 24bfa33..ac32216 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -43,19 +43,31 @@ jobs: run: python -m nnnotes --help - name: Test run: python -m pytest + + workflows: + # the publishing scripts with the nnnotes release they install (.github/nnnotes.txt) + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v7 + - uses: actions/setup-python@v7 + with: + python-version: "3.13" + - uses: astral-sh/setup-uv@v7 + with: + enable-cache: true + - name: Install nnnotes + run: uv pip install --system -r .github/nnnotes.txt boto3 pytest jsonschema - uses: actions/setup-node@v7 - if: matrix.python-version == '3.13' with: node-version: "22" - name: Chart data page - if: matrix.python-version == '3.13' env: PLAYER_REPOSITORY: ${{ vars.MUSIC_DATA_PLAYER_REPOSITORY || 'empty-sekai/ournotes-player' }} PLAYER_REF: ${{ vars.MUSIC_DATA_PLAYER_REF }} run: .github/scripts/songs_page.sh "$RUNNER_TEMP/music-data-player" - name: Workflow tests env: - MUSIC_DATA_PAGE: ${{ matrix.python-version == '3.13' && format('{0}/music-data-player/examples/songs', runner.temp) || '' }} + MUSIC_DATA_PAGE: ${{ format('{0}/music-data-player/examples/songs', runner.temp) }} run: python -m pytest -q .github/scripts goldens: diff --git a/.github/workflows/music-data-prebuilt.yml b/.github/workflows/music-data-prebuilt.yml deleted file mode 100644 index 4847b77..0000000 --- a/.github/workflows/music-data-prebuilt.yml +++ /dev/null @@ -1,165 +0,0 @@ -# DRAFT: install as .github/workflows/music-data-prebuilt.yml only after review. -# Copy the prebuilt_publish*.py/mjs helpers into .github/scripts too. -# This path skips only the already completed build and uses the same region lock. -name: Music data (validated prebuilt) - -on: - workflow_dispatch: - inputs: - region: - description: Region of the verified bundle (never substitute another region) - type: choice - options: [hk-tw-mo, jp] - default: hk-tw-mo - release_id: - description: Same-repository draft release ID containing the final bundle and .sha256 - type: string - required: true - asset_name: - description: Final bundle ZIP asset name - type: string - required: true - bundle_sha256: - description: Independently verified SHA-256 of the final bundle ZIP - type: string - required: true - dry_run: - description: Check and list uploads without writing objects - type: boolean - default: true - -permissions: - # GitHub restricts private draft-release downloads to callers with push access. - contents: write - -concurrency: - # Both regions and songs-page.yml publish the same ten-file songs/ closure. - group: songs-page-publication - cancel-in-progress: false - -env: - STORY_S3_ENDPOINT: ${{ vars.STORY_S3_ENDPOINT || 'https://storage.bdon.moe' }} - STORY_S3_BUCKET: ${{ vars.STORY_S3_BUCKET || 'moenotes' }} - # The legacy region determines only ownership of its old prefix, never which matrix regions run. - MUSIC_DATA_S3_PREFIX: ${{ inputs.region == 'jp' && (vars.MUSIC_DATA_JP_S3_PREFIX || (vars.MUSIC_DATA_MASTERDATA_REGION == 'jp' && vars.MUSIC_DATA_S3_PREFIX) || 'jp/music-data') || (vars.MUSIC_DATA_TW_S3_PREFIX || (vars.MUSIC_DATA_MASTERDATA_REGION != 'jp' && vars.MUSIC_DATA_S3_PREFIX) || 'music-data') }} - MUSIC_DATA_TW_PREFIX_CHECK: ${{ vars.MUSIC_DATA_TW_S3_PREFIX || (vars.MUSIC_DATA_MASTERDATA_REGION != 'jp' && vars.MUSIC_DATA_S3_PREFIX) || 'music-data' }} - MUSIC_DATA_JP_PREFIX_CHECK: ${{ vars.MUSIC_DATA_JP_S3_PREFIX || (vars.MUSIC_DATA_MASTERDATA_REGION == 'jp' && vars.MUSIC_DATA_S3_PREFIX) || 'jp/music-data' }} - MUSIC_DATA_PUBLISH: ${{ vars.MUSIC_DATA_PUBLISH || 'false' }} - MASTERDATA_BASE_URL: ${{ vars.MASTERDATA_BASE_URL || 'https://metadata.bdon.moe' }} - MASTERDATA_REGION: ${{ inputs.region }} - NNNOTES_CATALOG_REGION: ${{ inputs.region == 'jp' && 'jp' || 'tw' }} - NNNOTES_CATALOG_LANGUAGE: ${{ inputs.region == 'jp' && 'ja' || 'zh-Hant' }} - WORK: ${{ github.workspace }}/prebuilt-work - -jobs: - check-and-publish: - # Acquire the shared page lock first, then the selected data region lock. - concurrency: - group: music-data-${{ inputs.region }} - cancel-in-progress: false - runs-on: ubuntu-latest - timeout-minutes: 30 - steps: - - uses: actions/checkout@v7 - with: - path: publisher - fetch-depth: 0 - - uses: actions/setup-python@v7 - with: - python-version: '3.13' - - uses: actions/setup-node@v7 - with: - node-version: '22' - - name: Check-only dependencies (no Rust extension build) - run: python -m pip install boto3 jsonschema pytest - - name: Download the hash-anchored same-repository draft asset - env: - GH_TOKEN: ${{ github.token }} - RELEASE_ID: ${{ inputs.release_id }} - ASSET_NAME: ${{ inputs.asset_name }} - BUNDLE_SHA256: ${{ inputs.bundle_sha256 }} - run: | - python publisher/.github/scripts/prebuilt_publish_download.py \ - --repository "$GITHUB_REPOSITORY" --release-id "$RELEASE_ID" \ - --asset-name "$ASSET_NAME" --sha256 "$BUNDLE_SHA256" --out "$WORK/final.zip" - - name: Validate inventory and extract a fresh output directory - id: bundle - env: - BUNDLE_SHA256: ${{ inputs.bundle_sha256 }} - run: | - python publisher/.github/scripts/prebuilt_publish.py unpack \ - --archive "$WORK/final.zip" --sha256 "$BUNDLE_SHA256" --out "$WORK/bundle" \ - --producer-repository "$GITHUB_REPOSITORY" - - name: The authentic NN producer source (full history) - uses: actions/checkout@v7 - with: - repository: ${{ steps.bundle.outputs.producer_repository }} - ref: ${{ steps.bundle.outputs.producer_commit }} - path: producer - fetch-depth: 0 - - name: Verify authentic producer identity and every runtime resource - run: | - python publisher/.github/scripts/prebuilt_publish.py verify \ - --bundle "$WORK/bundle" --producer "$GITHUB_WORKSPACE/producer" \ - --publisher "$GITHUB_WORKSPACE/publisher/.github/scripts/music_data.py" - - name: Producer's declared runtime dependencies without rebuilding the extension - working-directory: producer - run: | - python - <<'PY' - import pathlib, subprocess, sys, tomllib - deps = tomllib.loads(pathlib.Path('pyproject.toml').read_text())['project']['dependencies'] - subprocess.run([sys.executable, '-m', 'pip', 'install', '--no-cache-dir', *deps], check=True) - PY - - name: The final pinned chart data page - env: - PLAYER_REPOSITORY: ${{ steps.bundle.outputs.player_repository }} - PLAYER_REF: ${{ steps.bundle.outputs.player_commit }} - run: publisher/.github/scripts/songs_page.sh "$WORK/page" - - name: Gate self-tests (unchanged) - env: - MUSIC_DATA_PAGE: ${{ env.WORK }}/page/examples/songs - working-directory: producer - run: python -m pytest -q -p no:cacheprovider "$GITHUB_WORKSPACE/publisher/.github/scripts/test_music_data.py" "$GITHUB_WORKSPACE/publisher/.github/scripts/test_jp_workflow.py" "$GITHUB_WORKSPACE/publisher/.github/scripts/test_music_data_refresh.py" "$GITHUB_WORKSPACE/publisher/.github/scripts/test_music_data_compression.py" "$GITHUB_WORKSPACE/publisher/.github/scripts/test_prebuilt_source_identity.py" - - name: Genuine current API master snapshot (unchanged) - working-directory: producer - run: python "$GITHUB_WORKSPACE/publisher/.github/scripts/music_data.py" master "$WORK/master" - - name: Full original gates and runtime inventory (unchanged) - env: - PYTHONPATH: ${{ github.workspace }}/producer/src - PLAYER_REPOSITORY: ${{ steps.bundle.outputs.player_repository }} - PLAYER_REF: ${{ steps.bundle.outputs.player_commit }} - working-directory: producer - run: | - python "$GITHUB_WORKSPACE/publisher/.github/scripts/music_data.py" check \ - "$WORK/bundle/data" "$WORK/master" "$WORK/page/examples/songs" - - name: Final consumer modules and actual WASM replay smoke - run: | - node publisher/.github/scripts/prebuilt_publish_smoke.mjs \ - "$WORK/page/examples/songs" "$WORK/bundle/data" - - name: Pinned standalone songs page before the music-data pointer - env: - STORY_S3_ACCESS_KEY: ${{ !inputs.dry_run && vars.MUSIC_DATA_PUBLISH == 'true' && secrets.STORY_S3_ACCESS_KEY || '' }} - STORY_S3_SECRET_KEY: ${{ !inputs.dry_run && vars.MUSIC_DATA_PUBLISH == 'true' && secrets.STORY_S3_SECRET_KEY || '' }} - run: | - python publisher/.github/scripts/prebuilt_publish_page.py \ - --publisher "$GITHUB_WORKSPACE/publisher/.github/scripts/music_data.py" \ - --page "$WORK/page/examples/songs" --bundle "$WORK/bundle" \ - ${{ (inputs.dry_run || vars.MUSIC_DATA_PUBLISH != 'true') && '--dry-run' || '' }} - - name: Publish with existing object ordering and read-back checks - env: - STORY_S3_ACCESS_KEY: ${{ !inputs.dry_run && vars.MUSIC_DATA_PUBLISH == 'true' && secrets.STORY_S3_ACCESS_KEY || '' }} - STORY_S3_SECRET_KEY: ${{ !inputs.dry_run && vars.MUSIC_DATA_PUBLISH == 'true' && secrets.STORY_S3_SECRET_KEY || '' }} - working-directory: producer - run: | - python "$GITHUB_WORKSPACE/publisher/.github/scripts/music_data.py" publish \ - "$WORK/bundle/data" ${{ (inputs.dry_run || vars.MUSIC_DATA_PUBLISH != 'true') && '--dry-run' || '' }} - - name: Preserve gate reports (not payloads or secrets) - if: always() - uses: actions/upload-artifact@v7 - with: - name: prebuilt-publish-reports - if-no-files-found: warn - path: | - ${{ env.WORK }}/bundle/prebuilt.json - ${{ env.WORK }}/bundle/data/check.json - ${{ env.WORK }}/bundle/data/build.json diff --git a/.github/workflows/music-data-region.yml b/.github/workflows/music-data-region.yml index e065bac..8de5e85 100644 --- a/.github/workflows/music-data-region.yml +++ b/.github/workflows/music-data-region.yml @@ -36,7 +36,6 @@ env: PLAYFETCH_VERSION: ${{ vars.PLAYFETCH_VERSION || 'v0.92' }} APK_PACKAGE: ${{ inputs.region == 'jp' && 'com.bushiroad.sirius' || vars.STORY_APK_PACKAGE || 'com.bilibili.sirius' }} PYTHON_VERSION: "3.13" - PYTHONPATH: src WORK: ${{ github.workspace }}/music-data-work # Completed native programs, under the model source identity and compiled semantic inputs. A change to a # chart, kind or shape only recomputes affected programs; current metadata and checks are assembled each run. @@ -58,11 +57,15 @@ jobs: build: ${{ steps.plan.outputs.build }} steps: - uses: actions/checkout@v7 - with: - fetch-depth: 0 # the nnnotes commit: the last one that changed src/, rust/ or pyproject.toml - uses: actions/setup-python@v7 with: python-version: ${{ env.PYTHON_VERSION }} + - uses: astral-sh/setup-uv@v7 + with: + enable-cache: true + - name: Install nnnotes + # the build's inputs name the installed nnnotes and the ournotes-deck release of its deck model + run: uv pip install --system -r .github/nnnotes.txt boto3 - name: Inputs against the published build id: plan env: @@ -76,16 +79,14 @@ jobs: timeout-minutes: 120 steps: - uses: actions/checkout@v7 - with: - fetch-depth: 0 - uses: actions/setup-python@v7 with: python-version: ${{ env.PYTHON_VERSION }} - cache: pip - cache-dependency-path: pyproject.toml - uses: astral-sh/setup-uv@v7 with: enable-cache: true + - name: Install nnnotes + run: uv pip install --system -r .github/nnnotes.txt boto3 pytest jsonschema - name: APK env: PLAYFETCH_CREDENTIALS_JSON: ${{ secrets.PLAYFETCH_CREDENTIALS }} @@ -96,7 +97,6 @@ jobs: NNNOTES_PATHS_APK: ${{ env.WORK }}/apk/base.apk run: | sudo apt-get update -qq && sudo apt-get install -y -qq openvpn > /dev/null - uv pip install --system --only-binary=:all: UnityPy numpy grpcio pycryptodome brotli python .github/scripts/jp_vpngate.py watch > "$RUNNER_TEMP/jp-vpngate-watch.log" 2>&1 & echo $! > "$RUNNER_TEMP/jp-vpngate-watch-shell.pid" for i in $(seq 1 60); do @@ -116,13 +116,6 @@ jobs: - name: The chart data page run: .github/scripts/songs_page.sh "$WORK/page" - - uses: Swatinem/rust-cache@v2 - with: - workspaces: rust - - name: Install nnnotes - # builds the extension module nnnotes._deck: the deck model at the commit rust/Cargo.toml pins - run: uv pip install --system -e ".[test]" boto3 - - name: Gate self-test env: MUSIC_DATA_PAGE: ${{ env.WORK }}/page/examples/songs diff --git a/.github/workflows/music-data.yml b/.github/workflows/music-data.yml index 5115821..5a2a431 100644 --- a/.github/workflows/music-data.yml +++ b/.github/workflows/music-data.yml @@ -54,7 +54,7 @@ jobs: fail-fast: false matrix: region: ${{ fromJSON(needs.regions.outputs.regions) }} - # The normal and prebuilt workflows serialize the same region's publication. + # Runs of the same region serialize their publication. concurrency: group: music-data-${{ matrix.region }} cancel-in-progress: false diff --git a/.github/workflows/songs-page.yml b/.github/workflows/songs-page.yml index 68856d3..78547b0 100644 --- a/.github/workflows/songs-page.yml +++ b/.github/workflows/songs-page.yml @@ -16,7 +16,7 @@ permissions: contents: read concurrency: - # Full prebuilt publication also writes songs/. Keep the two releases ordered. + # One songs/ page publication at a time. group: songs-page-publication cancel-in-progress: false diff --git a/.github/workflows/story-site-region.yml b/.github/workflows/story-site-region.yml index 58f8757..371c2e7 100644 --- a/.github/workflows/story-site-region.yml +++ b/.github/workflows/story-site-region.yml @@ -100,16 +100,11 @@ jobs: # The runner's preinstalled SDKs take most of its disk; a story build keeps its bundles and work files. run: sudo rm -rf /usr/local/lib/android /usr/share/dotnet /opt/ghc /usr/local/.ghcup /opt/hostedtoolcache/CodeQL && df -h "$GITHUB_WORKSPACE" - - uses: Swatinem/rust-cache@v2 # the install builds the extension module nnnotes._deck (rust/) - with: - workspaces: rust - uses: astral-sh/setup-uv@v7 with: enable-cache: true - name: Install nnnotes - # UnityPy 1.25.x requires etcpak, which has no manylinux x86_64 wheel on PyPI (only PyPy / cp37), so the - # runner's cp313 environment cannot resolve it. pyproject.toml already pins it to the 1.24 line. - run: uv pip install --system -e ".[fonts]" boto3 + run: uv pip install --system -r .github/nnnotes.txt boto3 - name: Tools run: |