ci: wait for exported artwork and retry downloads

.github/RECOGNITION.md

@@ -104,8 +104,8 @@ A failed run leaves the pointer on the previous bundle; the objects it already c
 and unreferenced), and the next run resumes: matching objects are kept, only missing ones are uploaded.
 
 Triggers: `repository_dispatch` `masterdata-updated` (moenotes-masterdata-sync's `dispatch_repositories` already names
-this repository: the story site and music data workflows run on it too), a daily schedule (03:53 UTC) in case a dispatch
-was missed, and `workflow_dispatch`:
+this repository: the story site and music data workflows run on it too), a schedule every six hours (at minute 53, UTC)
+in case a dispatch was missed, and `workflow_dispatch`:
 
 | Input | Meaning |
 |---|---|
@@ -114,6 +114,13 @@ was missed, and `workflow_dispatch`:
 
 Runs do not overlap (`concurrency: recognition-gallery`).
 
+**Waiting for the artwork.** A new master version is served before the asset service has exported its files, so the
+artwork of new cards is missing at first. The catalog is read only once the asset service's `/versions/current_version.json`
+lists, for every region, a release of the same master version in state `succeeded` or `partial`. Until then the plan job
+reads the master data and that file again every `RECOGNITION_EXPORT_POLL` seconds (60), for up to
+`RECOGNITION_EXPORT_WAIT` seconds (3600), and then stops with the pointer unchanged. The build job does not wait: a
+master version that moved after the plan stops it, and that version's own dispatch builds it.
+
 **Publishing switch.** Nothing is uploaded unless the repository variable `RECOGNITION_PUBLISH` is `true` (unset: off).
 Off, every run is a dry run, whatever its trigger: the runtime and publish steps get no bucket key and only read. Turn it
 on with `gh variable set RECOGNITION_PUBLISH --body true`.

.github/scripts/apk.sh

@@ -10,8 +10,8 @@ if [ -z "${PLAYFETCH_CREDENTIALS_JSON:-}" ]; then
   exit 1
 fi
 bin="$RUNNER_TEMP/playfetch"
-curl -fsSL -o "$bin" "https://github.com/Exmeaning/playfetch/releases/download/$PLAYFETCH_VERSION/playfetch-$PLAYFETCH_VERSION-linux-amd64"
-curl -fsSL -o "$RUNNER_TEMP/SHA256SUMS" "https://github.com/Exmeaning/playfetch/releases/download/$PLAYFETCH_VERSION/SHA256SUMS"
+curl -fsSL --retry 5 --retry-delay 10 --retry-all-errors -o "$bin" "https://github.com/Exmeaning/playfetch/releases/download/$PLAYFETCH_VERSION/playfetch-$PLAYFETCH_VERSION-linux-amd64"
+curl -fsSL --retry 5 --retry-delay 10 --retry-all-errors -o "$RUNNER_TEMP/SHA256SUMS" "https://github.com/Exmeaning/playfetch/releases/download/$PLAYFETCH_VERSION/SHA256SUMS"
 (cd "$RUNNER_TEMP" && grep " playfetch-$PLAYFETCH_VERSION-linux-amd64\$" SHA256SUMS | sed "s| playfetch-$PLAYFETCH_VERSION-linux-amd64| playfetch|" | sha256sum -c -)
 chmod 755 "$bin"
 export PLAYFETCH_CREDENTIALS="$RUNNER_TEMP/playfetch-credentials.json"

.github/scripts/fonts.sh

@@ -9,7 +9,7 @@ cd "$dir"
 cjk=https://github.com/notofonts/noto-cjk/raw/165c01b46ea533872e002e0785ff17e44f6d97d8/Sans/OTF
 fetch() {  # fetch FILE URL SHA256
   if [ ! -f "$1" ] || ! echo "$3  $1" | sha256sum -c --quiet - 2>/dev/null; then
-    curl -fsSL -o "$1" "$2"
+    curl -fsSL --retry 5 --retry-delay 10 --retry-all-errors -o "$1" "$2"
     echo "$3  $1" | sha256sum -c -
   fi
 }
@@ -18,7 +18,7 @@ fetch NotoSansCJKtc-Regular.otf "$cjk/TraditionalChinese/NotoSansCJKtc-Regular.o
 fetch NotoSansCJKsc-Regular.otf "$cjk/SimplifiedChinese/NotoSansCJKsc-Regular.otf" 2c76254f6fc379fddfce0a7e84fb5385bb135d3e399294f6eeb6680d0365b74b
 fetch NotoColorEmoji.ttf https://github.com/googlefonts/noto-emoji/raw/v2.051/fonts/NotoColorEmoji.ttf 72a635cb3d2f3524c51620cdde406b217204e8a6a06c6a096ff8ed4b5fd6e27b
 if [ ! -f Pretendard-SemiBold.otf ] || ! echo "c89bc43027dc7cde5726e96223376f8eec09302b2fc1f8147fd5b57cfc376118  Pretendard-SemiBold.otf" | sha256sum -c --quiet - 2>/dev/null; then
-  curl -fsSL -o pretendard.zip https://github.com/orioncactus/pretendard/releases/download/v1.3.9/Pretendard-1.3.9.zip
+  curl -fsSL --retry 5 --retry-delay 10 --retry-all-errors -o pretendard.zip https://github.com/orioncactus/pretendard/releases/download/v1.3.9/Pretendard-1.3.9.zip
   echo "04be351a74d6bf7d60c480a3087e51d185485d35a52023142af1df19eb8c428a  pretendard.zip" | sha256sum -c -
   unzip -q -o -j pretendard.zip public/static/Pretendard-SemiBold.otf
   rm pretendard.zip

.github/scripts/recognition.py

@@ -24,7 +24,9 @@
 
 Bucket: $STORY_S3_ENDPOINT, $STORY_S3_BUCKET, $RECOGNITION_S3_PREFIX (a key prefix, may be empty), credentials
 $STORY_S3_ACCESS_KEY / $STORY_S3_SECRET_KEY (publish and runtime only). Sources: $MASTERDATA_BASE_URL (decoded master
-data, index.json with every file's SHA-256), $RECOGNITION_REGIONS, $RECOGNITION_ASSET_API (the asset service).
+data, index.json with every file's SHA-256), $RECOGNITION_REGIONS, $RECOGNITION_ASSET_API (the asset service). plan and
+build read the catalog once the asset service has exported every region's master version, waiting up to
+$RECOGNITION_EXPORT_WAIT seconds (3600) and checking every $RECOGNITION_EXPORT_POLL seconds (60).
 """
 from __future__ import annotations
 
@@ -73,6 +75,8 @@
          "snap": {"table": "MasterSupportCard", "directory": "SupportCard", "label": "snap_thumbnail", "file": "snap_thumbnail.webp",
                   "levels": "MasterSupportCardLevel", "limits": "MasterSupportCardRank"}}
 GROWTH_TABLES = tuple(info[key] for info in KINDS.values() for key in ("levels", "limits"))
+# Asset service release states whose files are published (/versions/current_version.json).
+EXPORT_STATES = ("succeeded", "partial")
 RUNTIME_ROLES = ("glue", "module", "wasm")
 STAGES = ("encoders", "fields", "ranks")
 HEX64 = re.compile(r"[a-f0-9]{64}")
@@ -414,6 +418,38 @@ def master_sources(names: list[str]) -> list[dict]:
     return sources
 
 
+def export_lag(api: str, sources: list[dict]) -> list[str]:
+    """The regions whose latest asset service release (/versions/current_version.json) is not at the master version the
+    catalog was read from: their new cards' artwork is not published yet."""
+    status = json.loads(fetch(f"{api}/versions/current_version.json").body)
+    releases = {entry.get("metadata_region"): entry for entry in (status.get("regions") or {}).values()
+                if isinstance(entry, dict)}
+    lag = []
+    for source in sources:
+        release = releases.get(source["region"]) or {}
+        if release.get("master_version") != source["masterVersion"] or release.get("state") not in EXPORT_STATES:
+            lag.append(f"{source['region']} (master {source['masterVersion']}, asset release "
+                       f"{release.get('master_version')} {release.get('state')})")
+    return lag
+
+
+def exported_sources(names: list[str]) -> list[dict]:
+    """master_sources once the asset service has exported each region's master version. Master data moves first, so
+    both are read again every $RECOGNITION_EXPORT_POLL seconds for up to $RECOGNITION_EXPORT_WAIT seconds."""
+    api = env("RECOGNITION_ASSET_API", "https://assets.bdon.moe").rstrip("/")
+    poll = max(1, int(env("RECOGNITION_EXPORT_POLL", "60")))
+    attempts = max(0, int(env("RECOGNITION_EXPORT_WAIT", "3600"))) // poll + 1
+    for attempt in range(attempts):
+        sources = master_sources(names)
+        lag = export_lag(api, sources)
+        if not lag:
+            return sources
+        if attempt + 1 < attempts:
+            print(f"waiting for the asset service to export {', '.join(lag)}", flush=True)
+            time.sleep(poll)
+    raise Failure(f"the asset service has not exported {', '.join(lag)}; the pointer stays on the published bundle")
+
+
 def catalog(sources: list[dict]) -> tuple[list[dict], list[str]]:
     """The union of every region's cards; a card's identity and level limit are the first region's (in
     RECOGNITION_REGIONS order)."""
@@ -526,7 +562,7 @@ def superset_problem(state: dict | None, keys: set[str]) -> str | None:
 # ---------------------------------------------------------------- plan
 def cmd_plan() -> None:
     spec = runtime_spec()
-    sources = master_sources(regions())
+    sources = exported_sources(regions())
     cards, warnings = catalog(sources)
     arts = resolve_art(cards)
     inputs = inputs_identity(spec, cards, arts)
@@ -732,7 +768,7 @@ def cmd_build(out: Path, models_dir: Path, runtime_dir: Path | None) -> None:
     models = {name: pinned_file(models_dir, name, spec["files"][name]) for name in builder_inputs(spec)}
     site = out / "site"
     (site / "assets").mkdir(parents=True)
-    sources = master_sources(regions())
+    sources = exported_sources(regions())
     cards, warnings = catalog(sources)
     arts = resolve_art(cards)
     state = published_state()

.github/scripts/test_recognition.py

@@ -62,6 +62,7 @@ def __init__(self, tmp_path):
         self.store = {}
         self.puts = []
         self.corrupt_downloads = set()
+        self.exports = {}  # region: (master version, state) of its asset release; default: exported
         self.runtime = {}
         spec = json.loads(r.RUNTIME_SPEC.read_text(encoding="utf-8"))
         for index, (name, record) in enumerate(spec["files"].items()):
@@ -111,6 +112,12 @@ def respond(body, headers=None):
                 for table in self.tables(region):
                     if url == f"{MASTER}/{region.split('-')[0]}/master/{table}.json":
                         return respond(self.table(region, table))
+        if url == f"{ASSETS}/versions/current_version.json":
+            regions = {}
+            for region in self.masters:
+                version, state = self.exports.get(region, (f"v-{region}", "succeeded"))
+                regions[region.split("-")[0]] = {"metadata_region": region, "master_version": version, "state": state}
+            return respond(json.dumps({"schema_version": 1, "regions": regions}).encode())
         if url.startswith(ASSETS + "/files/"):
             name = url.removeprefix(ASSETS + "/files/")
             kind, asset = ("member" if name[0] == "m" else "snap"), int(name[1:])
@@ -152,7 +159,7 @@ def world(tmp_path, monkeypatch):
                         "RECOGNITION_REGIONS": "hk-tw-mo jp", "RECOGNITION_ASSET_API": ASSETS,
                         "MASTERDATA_BASE_URL": MASTER, "GITHUB_REPOSITORY": "example/nnnotes"}.items():
         monkeypatch.setenv(name, value)
-    for name in ("GITHUB_OUTPUT", "GITHUB_STEP_SUMMARY", "FORCE"):
+    for name in ("GITHUB_OUTPUT", "GITHUB_STEP_SUMMARY", "FORCE", "RECOGNITION_EXPORT_WAIT", "RECOGNITION_EXPORT_POLL"):
         monkeypatch.delenv(name, raising=False)
     monkeypatch.setattr(r, "RUNTIME_SPEC", w.spec_path)
     monkeypatch.setattr(r, "fetch", w.fetch)
@@ -456,6 +463,32 @@ def test_plan_builds_only_when_inputs_change(world, monkeypatch):
     assert outputs["build"] == "true"
 
 
+def test_plan_waits_for_the_asset_release_of_the_master_version(world, monkeypatch):
+    outputs, sleeps = {}, []
+    monkeypatch.setattr(r, "output", outputs.__setitem__)
+    world.exports["jp"] = ("v-previous", "succeeded")
+
+    def sleep(seconds):
+        sleeps.append(seconds)
+        world.exports.pop("jp")
+    monkeypatch.setattr(r.time, "sleep", sleep)
+    r.cmd_plan()
+    assert sleeps == [60]
+    assert outputs == {"build": "true", "runtime": "false"}
+
+
+@pytest.mark.parametrize("export", [("v-previous", "succeeded"), ("v-hk-tw-mo", "running")])
+def test_plan_stops_when_the_asset_release_stays_behind(world, monkeypatch, export):
+    sleeps = []
+    monkeypatch.setattr(r.time, "sleep", sleeps.append)
+    monkeypatch.setenv("RECOGNITION_EXPORT_WAIT", "120")
+    world.exports["hk-tw-mo"] = export
+    with pytest.raises(r.Failure, match="has not exported hk-tw-mo"):
+        r.cmd_plan()
+    assert sleeps == [60, 60]
+    assert not world.puts
+
+
 def test_a_threshold_or_level_limit_change_is_a_new_input(world, monkeypatch):
     outputs = {}
     monkeypatch.setattr(r, "output", outputs.__setitem__)

.github/scripts/tools.sh

@@ -6,7 +6,7 @@ version=r2117
 digest=2f98c77f756079f63fbd119939067f1ed461d77e70993bc4cc372736d859c84a
 mkdir -p "$dir"
 if [ ! -x "$dir/vgmstream-cli" ]; then
-  curl -fsSL -o "$dir/vgmstream.zip" "https://github.com/vgmstream/vgmstream/releases/download/$version/vgmstream-linux.zip"
+  curl -fsSL --retry 5 --retry-delay 10 --retry-all-errors -o "$dir/vgmstream.zip" "https://github.com/vgmstream/vgmstream/releases/download/$version/vgmstream-linux.zip"
   echo "$digest  $dir/vgmstream.zip" | sha256sum -c -
   unzip -q -o "$dir/vgmstream.zip" vgmstream-cli -d "$dir"
   rm "$dir/vgmstream.zip"

.github/workflows/recognition.yml

@@ -9,9 +9,10 @@ name: Recognition gallery
 # them all, and last the small pointer recognition/current.json that names the bundle. Nothing is deleted or
 # overwritten; a catalog that lacks a published card, or any artwork that fails its check, leaves the pointer where it is.
 #
-# Triggers: moenotes-masterdata-sync's repository_dispatch `masterdata-updated`, a daily schedule in case a dispatch was
-# missed, and workflow_dispatch (force: build although the inputs are unchanged; dry_run). Publishing switch: the
-# repository variable RECOGNITION_PUBLISH (unset: every run is a dry run). docs: .github/RECOGNITION.md
+# Triggers: moenotes-masterdata-sync's repository_dispatch `masterdata-updated`, a schedule every six hours in case a
+# dispatch was missed, and workflow_dispatch (force: build although the inputs are unchanged; dry_run). Master data moves
+# before the asset service exports its artwork: the plan job waits for that export. Publishing switch: the repository
+# variable RECOGNITION_PUBLISH (unset: every run is a dry run). docs: .github/RECOGNITION.md
 
 on:
   repository_dispatch:
@@ -27,7 +28,7 @@ on:
         type: boolean
         default: false
   schedule:
-    - cron: "53 3 * * *"
+    - cron: "53 */6 * * *"
 
 permissions:
   contents: read
@@ -52,7 +53,8 @@ env:
 jobs:
   plan:
     runs-on: ubuntu-latest
-    timeout-minutes: 10
+    # up to an hour waiting for the asset service to export a new master version
+    timeout-minutes: 75
     outputs:
       build: ${{ steps.plan.outputs.build }}
       runtime: ${{ steps.plan.outputs.runtime }}
@@ -130,6 +132,8 @@ jobs:
           name: recognition-encoder-models
           path: ${{ env.WORK }}/models
       - name: Build
+        env:
+          RECOGNITION_EXPORT_WAIT: "0"   # the plan job waited; a master version newer than its catalog stops here
         run: python .github/scripts/recognition.py build "$WORK/out" --models "$WORK/models"
       - name: Publish
         # Read-only preflight first; a dry run (the dry_run input, or RECOGNITION_PUBLISH off) stops after it.