From 13dfcc190a6e0040cb480e04349ccc3f2a3aa9ac Mon Sep 17 00:00:00 2001 From: nichinichisou Date: Thu, 15 Oct 2026 12:55:44 +0800 Subject: [PATCH 01/18] feat(live2d)!: read models from product manifests and their object tables AssetStore reads the `objects` of a manifest, the object table of an nnnotes.product/1 manifest, and gives objects by id: their class, their files by role (json, image, index, program:/, blob:), the objects their `$ref` values name, and typed readers for prefabs, materials, clips, MonoBehaviours and textures. The model viewer and the story player load a model from a product manifest of live2d.model. Its model.json (format 3) names the prefab, the CubismMoc (the moc3 is its blob:_bytes), the shaders and the Cubism mask materials by object id, and lists the clip bindings resolved against the model's hierarchy. Atlas pages are Texture2D objects with their sampler settings and mip count; shader programs come from the GLES3 programs of the shader objects. A story store adds the object tables of the story's model manifests to one table, and refuses manifests that list an object differently. `validate-data` checks a model manifest object by object: the classes model.json names, the moc3 header, the prefab, fades, clips and bindings, expressions, materials, textures and the shader programs the viewer and the story renderer draw with. schema/product.schema.json and schema/model-json.schema.json describe the formats, and docs/data-format.md the model objects. BREAKING CHANGE: a model manifest is a product manifest of live2d.model with a model.json of format 3, and a story manifest has the format ournotes.story-manifest/2. Co-Authored-By: Claude Opus 5.5 --- README.en.md | 3 +- README.md | 2 +- docs/data-format.md | 170 ++++++---- docs/story-data-format.md | 77 ++--- schema/model-json.schema.json | 90 +++-- schema/model.schema.json | 103 ------ schema/product.schema.json | 65 ++++ schema/story-manifest.schema.json | 15 +- schema/story.schema.json | 17 +- scripts/lib/headless.mjs | 1 + scripts/lib/story-author/script.js | 22 +- scripts/lib/story-author/site.js | 20 +- scripts/validate-data.mjs | 453 ++++++++++++++++---------- src/data/assets.js | 173 +++++++++- src/engine/glsl.js | 46 ++- src/engine/prefab.js | 9 +- src/engine/texture.js | 10 +- src/live2d/character.js | 75 +++-- src/live2d/drawing.js | 33 +- src/live2d/model.js | 62 ++-- src/live2d/motion.js | 37 ++- src/live2d/session.js | 11 +- src/story/assets.js | 79 +++-- src/story/models.js | 30 +- src/story/renderer.js | 8 +- src/story/session.js | 8 +- src/story/simple/render.js | 7 +- src/story/simple/session.js | 6 +- tests/data/story-data.test.mjs | 4 +- tests/types/live2d.ts | 5 +- tests/unit/live2d-adv.test.mjs | 83 +++-- tests/unit/live2d-model.test.mjs | 123 +++---- tests/unit/live2d-objects.mjs | 115 +++++++ tests/unit/live2d-session.test.mjs | 141 ++++---- tests/unit/live2d.test.mjs | 23 +- tests/unit/site-fixture.mjs | 119 +++---- tests/unit/story-assets.test.mjs | 158 ++++----- tests/unit/story-author.test.mjs | 27 +- tests/unit/story-schema.test.mjs | 94 ++---- tests/unit/validate-manifest.test.mjs | 111 +++---- types/index.d.ts | 51 ++- types/live2d.d.ts | 24 +- types/story.d.ts | 17 +- 43 files changed, 1560 insertions(+), 1167 deletions(-) delete mode 100644 schema/model.schema.json create mode 100644 schema/product.schema.json create mode 100644 tests/unit/live2d-objects.mjs diff --git a/README.en.md b/README.en.md index fba517f..8e9974c 100644 --- a/README.en.md +++ b/README.en.md @@ -197,8 +197,7 @@ nnnotes builds such a site from the user's own game files, for example: nnnotes web out/site --all --all-live2d --all-stories --player ``` -- The player also reads data in the earlier formats: assets stored as they are, and story manifests that hold the - models' files inside each story (`ournotes.story-manifest/1`). +- The player also reads assets stored as they are (not encoded). - Hosting (paths, CORS, encoded assets, caching): [docs/embedding.md](docs/embedding.md#hosting-the-data). A site encoded with brotli plays in Chromium-based browsers only when its `.br` assets are served with `Content-Encoding: br`. diff --git a/README.md b/README.md index a8ef75c..507fba9 100644 --- a/README.md +++ b/README.md @@ -152,7 +152,7 @@ nnnotes 从使用者自己的游戏文件生成这样的站点，例如： nnnotes web out/site --all --all-live2d --all-stories --player ``` -- 播放器也读取较早格式的数据：未编码的资源，以及把模型文件放在每集剧情之内的剧情清单（`ournotes.story-manifest/1`）。 +- 播放器也读取未编码存放的资源。 - 托管方式（路径、CORS、编码资源、缓存）：[docs/embedding.md](docs/embedding.md#hosting-the-data)。以 brotli 编码的站点要在 Chromium 系浏览器中播放，需以 `Content-Encoding: br` 提供 `.br` 资源。 - 校验：`npm run validate-data -- <站点目录>` 按数据格式与 `schema/` 检查谱面、模型与剧情，包括剧情与其模型之间的对应。 diff --git a/docs/data-format.md b/docs/data-format.md index ef46dd1..98d9ae8 100644 --- a/docs/data-format.md +++ b/docs/data-format.md @@ -481,8 +481,8 @@ characters of a story ([story-data-format.md](story-data-format.md#live2d-models ``` / models.json model index (for listings; the viewer does not read it) - models/.json model manifest, one per model - assets/. file contents (or .gz / .br encoded), shared with the charts and stories + models/.json model manifest, one per model: a product manifest of live2d.model + assets/. file contents (or .gz / .br encoded), shared with the charts, stories and other models ``` `` is the model's name, `[a-z0-9_]+` (e.g. `adv_live2d_rana_003_casual_spring_01`). The viewer resolves the asset @@ -496,80 +496,108 @@ paths of a manifest against `models/..`, the site root, as for charts. { "format": 2, "models": [ { "id": "adv_live2d_rana_003_casual_spring_01", "manifest": "models/adv_live2d_rana_003_casual_spring_01.json", "key": "Character/Live2D/003_adv/…", "group": "003_adv", "bytes": 10655335, - "files": 12, "textures": 2, "canvas": { "pixelsPerUnit": 6000, "…": "…" } } ] } + "files": 210, "textures": 2, "canvas": { "pixelsPerUnit": 6000, "…": "…" } } ] } ``` | Key | Meaning | |---|---| | `format` | `2`, the version of this layout. | | `models[].id`, `manifest` | The model and the path of its manifest, relative to the site root. | -| `models[].key` | The game's asset key of the model prefab. | -| `models[].group`, `label` | Optional: a group for listings (the directory of the key) and a display text (the character's name in the site's language, when the producer knows it). | +| `models[].key` | The model's key (the manifest's `subject`). | +| `models[].group`, `label` | Optional: a group for listings (the `` of the key) and a display text (the character's name in the site's language, when the producer knows it). | | `models[].character`, `names` | Optional: the character's id in the game's master data, and its name per language (`{ "": "…" }`, only languages with a name). | -| `models[].bytes`, `files` | Optional: the sum of `size` over the manifest's `files`, and their count. | -| `models[].textures`, `canvas` | Optional: atlas page count, and the moc3 canvas (as the prefab's `canvas`). | +| `models[].bytes`, `files` | Optional: the sum of `size` over the files of the manifest and of its objects, and their count. | +| `models[].textures`, `canvas` | Optional: the atlas page count and the moc3 canvas (as `model.json` has them). | ### Model manifest -`models/.json` ([schema](../schema/model.schema.json)): -`{ "format": 3, "id": "…", "key": "…", "model": { … }, "files": { … } }`. `format` 3 is the current version (its -assets may be encoded, its `model.json` has format 2); the viewer and the story player also read format 2, the earlier -version. `id` is the name of the manifest (`models/.json`). `files` maps logical paths to file entries exactly as -in a [chart manifest](#file-entries) (whole files, encoded or not, and split JSON objects, with the same checks). The -viewer reads `files`; `id`, `key` and `model` are handed to the page (`ModelPlayer.info`) as they are. `model` holds -facts about the model, all optional: `group` (as in models.json), `canvas` (the moc3 canvas, as in models.json), -`textures` (the atlas page count), `nodes` (the node count of the prefab), and `character`, `names`, `label` (as in -models.json); producers may add their own. Where a models.json entry has `key`, `group`, `canvas` or `textures`, they -equal the manifest's `key` and `model` values; `character`, `names` and `label` are in both or in neither, with equal -values. +`models/.json` is a product manifest ([schema](../schema/product.schema.json)) of the product `live2d.model`: -The logical files, all read when the model loads: +```json +{ "schema": "nnnotes.product/1", "product": "live2d.model", + "subject": "Character/Live2D/003_adv/adv_live2d_rana_003_casual_spring_01/model/adv_live2d_rana_003_casual_spring_01", + "files": { "model.json": { "asset": "assets/.json", "size": 2048 } }, + "objects": { + "CAB-…:-4153…": { "class": "Texture2D", "name": "texture_00", + "files": { "json": { "asset": "assets/.json", "size": 512 }, + "image": { "asset": "assets/.png", "size": 1048576 } } }, + "…": "…" } } +``` -| Logical file | Content | +| Key | Meaning | +|---|---| +| `schema`, `product` | `"nnnotes.product/1"` and `"live2d.model"`; the viewer and the story player refuse others. | +| `subject` | The model's key, `Character/Live2D///model/`; `` is the manifest's ``. | +| `files` | The product's own files: `model.json` alone. A [file entry](#file-entries), whole or encoded. | +| `objects` | The objects the model uses: object id → its `class`, `name` and `files` by role, each a file entry, whole or encoded. | + +An object id is `:` (the path id as decimal digits, possibly negative). An object's JSON +names another object by a reference `{ "$ref": { "file": "", "pathId": "" } }`, the object +`:`; `null` names none. The roles of an object's files: + +| Role | File | |---|---| -| `model.json` | Index (below). | -| `.moc3` | The Cubism moc3 (`CubismMoc` bytes). | -| `.prefab.json` | The model prefab (below). | -| `textures/-.png` | The atlas pages the drawables use, as named by their texture descriptors (relative to the prefab's directory). | -| `shaders/shaders.json`, `shaders/…` | The two Live2D shaders, in the layout of the chart's [shader directories](#shaders). | +| `json` | The object as JSON: every object has it (below). | +| `image` | A Texture2D's level 0 as a PNG. | +| `index` | A Shader's variant index: `{ name, variants: [{ platform, type, subShader, pass, keywords, role, … }] }`. | +| `program:/` | A Shader's program of the variant whose `role` it is: GLSL ES 3.00 for `gles3` (`#ifdef VERTEX` / `#ifdef FRAGMENT` blocks, as in [Shaders](#shaders)). | +| `blob:` | The bytes of a byte-array field: `blob:_bytes` of the CubismMoc is the moc3. | + +The viewer fetches every file of the manifest and of its objects when the model loads. The objects of a model, by +class and by what reaches them: + +| Object | `json` | Other files | +|---|---|---| +| The model's GameObject (`model.json` `prefab`) | the [model prefab](#the-model-prefab) | | +| The CubismMoc (`moc`), a MonoBehaviour | `{ "$script": "…\|CubismMoc", … }` | `blob:_bytes` | +| The atlas pages (`textures`), Texture2D | Unity's fields without the pixels: `m_Width`, `m_Height`, `m_MipCount`, `m_ColorSpace`, `m_TextureSettings` (filter and wrap modes) | `image` | +| The drawables' materials and the mask materials (`resources`), Material | `{ material, shader (a reference), keywords, floats, colors, … }` | | +| Their shaders (`shaders`), Shader | the parsed shader, as a shader directory's parsed file | `index`, the `program:` files below | +| The motions, AnimationClip | the clip: `name`, `stopTime`, `loopTime`, `startTime`, `cycleOffset`, `events`, `bindings` (one per curve, as stored: `typeID`, `class`, `scriptClass` for a MonoBehaviour, `pptr`, …), `streamed` (`curveCount`, `frames`), `dense`, `constant` | | +| The fade motion list, the fade motions, the expression list and the expressions, MonoBehaviour | their fields and `$script` (the script class, `"\|\|"`) | | -A manifest lists exactly these files: every texture a drawable uses, and only the shader variants below (those the -viewer draws with and those the story renderer draws with). +A manifest lists exactly these objects and files: every texture a drawable uses, and of a shader only the programs +below (those the viewer draws with and those the story renderer draws with). ### model.json ([schema](../schema/model-json.schema.json)) ```json -{ "format": 2, "name": "adv_live2d_rana_003_casual_spring_01", "key": "Character/Live2D/003_adv/…", - "moc3": "adv_live2d_rana_003_casual_spring_01.moc3", "prefab": "adv_live2d_rana_003_casual_spring_01.prefab.json", - "textures": ["textures/texture_00-abf131f7.png", "textures/texture_01-259653a7.png"], - "shaders": "shaders/shaders.json", "motionSync": true, - "resources": { "cubismMask": { "material": "Mask", "shader": { "shader": "Live2D Cubism/Mask" }, "floats": { "_Cull": 0 }, "…": "…" }, - "cubismMaskCulling": { "material": "MaskCulling", "shader": { "shader": "Live2D Cubism/Mask" }, "floats": { "_Cull": 1 }, "…": "…" } } } +{ "format": 3, "name": "adv_live2d_rana_003_casual_spring_01", "key": "Character/Live2D/003_adv/…", + "prefab": "CAB-…:-4153…", "moc": "CAB-…:2207…", + "canvas": { "pixelsPerUnit": 6000, "originX": 3000, "originY": 4500, "width": 6000, "height": 9000, "mocVersion": 5 }, + "textures": ["CAB-…:-7312…", "CAB-…:5521…"], + "shaders": { "Live2D Cubism/Lit-URP-ADV-optimize": "CAB-…:…", "Live2D Cubism/Mask": "unity_builtin_extra:…" }, + "resources": { "cubismMask": "resources.assets:…", "cubismMaskCulling": "resources.assets:…" }, + "motionSync": true, + "bindings": { "CAB-…:-118…": [ { "path": "Parameters/ParamAngleX", "attribute": "Value" }, "…" ] } } ``` | Key | Read | Meaning | |---|---|---| -| `format` | yes | `2`, the version of this file (`1`, the earlier version, has no `motionSync`); the viewer and the story player refuse other values. | -| `moc3`, `prefab` | yes | Paths of the moc3 and the prefab. | -| `shaders` | yes | Path of the shader index, a file named `shaders.json`; the paths inside it are relative to its directory. | -| `resources.cubismMask`, `resources.cubismMaskCulling` | yes | The Cubism mask materials (`Live2D/Cubism/Materials/Mask` and `MaskCulling` of the game's resources), inline materials as in [node lists](#node-lists); their shader is "Live2D Cubism/Mask". The viewer and the story player read `shader`, `floats` and `colors`. | -| `motionSync` | no | Format 2: `true` when the prefab's root has a `CubismMotionSyncController` with a `Live2DMotionSyncCriAudioInput` (a model whose mouth follows a voice through MotionSync). A fact for pages; the players take the controller from the prefab. | -| `name`, `key`, `textures` | no | The model's name, the asset key, the atlas pages. | +| `format` | yes | `3`, the version of this file; the viewer and the story player refuse other values. | +| `prefab` | yes | The model's GameObject; its `json` is the model prefab. | +| `moc` | yes | The CubismMoc (the root's `CubismModel._moc`); the moc3 is its file `blob:_bytes`. | +| `shaders` | yes | Unity shader name → Shader object: "Live2D Cubism/Lit-URP-ADV-optimize" (the drawables' materials) and "Live2D Cubism/Mask" (the mask materials). | +| `resources.cubismMask`, `resources.cubismMaskCulling` | yes | The Cubism mask Materials (`Live2D/Cubism/Materials/Mask` and `MaskCulling` of the game's resources); their shader is "Live2D Cubism/Mask". The viewer and the story player read `shader`, `floats` and `colors`. | +| `bindings` | yes | AnimationClip object → per binding of the clip, `{path, attribute}`: the path below the model's root (`""` for the root) and the attribute it animates in the model's hierarchy; `path` null where the hierarchy has no single node of the binding's path (it animates nothing). Every clip of `_motionList` has its entry. | +| `motionSync` | no | `true` when the prefab's root has a `CubismMotionSyncController` with a `Live2DMotionSyncCriAudioInput` (a model whose mouth follows a voice through MotionSync). A fact for pages; the players take the controller from the prefab. | +| `name`, `key`, `canvas`, `textures` | no | The model's name and key, the moc3 canvas, the atlas pages. | ### The model prefab -The model prefab as a [node list](#node-lists) `{ key, nodes, canvas }`: the model root, its `Parameters/`, -`Parts/` and `Drawables/` nodes and the rest of the hierarchy, with every component. Unity's references and -inline assets follow the conventions of the scene files; the clips, the fade motion list and the expressions are -inline. The viewer reads: +The prefab's `json` is the hierarchy of the model as a [node list](#node-lists) `{ name, nodes }`: the model root, its +`Parameters/`, `Parts/` and `Drawables/` nodes and the rest of the hierarchy. A component is +`{ type, pathId, script?, fields }`: a MonoBehaviour's `script` is its script class (`"||"`; +the class names the component below), `fields` its serialized fields, with references to other objects as `$ref`. The +viewer reads: | Where | Component | Fields | |---|---|---| -| root | `Live2DCharacter` | `DefaultMotionName`, `DefaultExpressionName`, `BasePosition`, `BaseScale`, `_motionList` (the clips), `_expressionList` (expression names, in `ExpressionsList` order) | -| root | `CubismFadeController` | `CubismFadeMotionList`: `MotionInstanceIds`, `CubismFadeMotionObjects` (`MotionName`, `FadeInTime`, `FadeOutTime`, `MotionLength`, `ParameterIds`, `ParameterCurves[].m_Curve`, `ParameterFadeInTimes`, `ParameterFadeOutTimes`) | -| root | `CubismExpressionController` | `UseLegacyBlendCalculation` (must be 0), `CurrentExpressionIndex`, `CurrentFadeInTime`, `ExpressionsList.CubismExpressionObjects` (`name`, `FadeInTime`, `FadeOutTime`, `Parameters[]`: `Id`, `Value`, `Blend`) | +| root | `Live2DCharacter` | `DefaultMotionName`, `DefaultExpressionName`, `BasePosition`, `BaseScale`, `_motionList` (references to the AnimationClips), `_expressionList` (expression names, in `ExpressionsList` order) | +| root | `CubismFadeController` | `CubismFadeMotionList`: the fade motion list (`MotionInstanceIds`, `CubismFadeMotionObjects`: references to the CubismFadeMotionData, whose fields are `MotionName`, `FadeInTime`, `FadeOutTime`, `MotionLength`, `ParameterIds`, `ParameterCurves[].m_Curve`, `ParameterFadeInTimes`, `ParameterFadeOutTimes`) | +| root | `CubismExpressionController` | `UseLegacyBlendCalculation` (must be 0), `CurrentExpressionIndex`, `CurrentFadeInTime`, `ExpressionsList`: the expression list (`CubismExpressionObjects`: references to the CubismExpressionData: `m_Name`, `FadeInTime`, `FadeOutTime`, `Parameters[]`: `Id`, `Value`, `Blend`), or null for a model without expressions | | root | `CubismAutoEyeBlinkInput`, `CubismEyeBlinkController` | `Mean`, `MaximumDeviation`, `Timescale`; `BlendMode` (must be 2), `EyeOpening` | | root | `CubismMouthController` | `BlendMode` (must be 0), `MouthOpening` | | root | `CubismHarmonicMotionController` | `BlendMode` (must be 1), `ChannelTimescales` | @@ -577,31 +605,29 @@ inline. The viewer reads: | root | `CubismRenderController` | `_sortingOrder`, `Opacity`, `_lastOpacity` | | `Parameters/` | `CubismEyeBlinkParameter`, `CubismHarmonicMotionParameter` | the parameters the eye blink and the breath drive (`Channel`, `Direction` (must be 2), `NormalizedOrigin`, `NormalizedRange`, `Duration`) | | `Drawables/` | `CubismDrawable` | `_unmanagedIndex`: the Core drawable index (the node's name is the Core drawable id) | -| `Drawables/` | `CubismRenderer` | `_mainTexture` (a [texture descriptor](#texture-descriptors), relative to the prefab's directory; unlike chart textures it may be mipmapped: the PNG is level 0 and the viewer generates the other `mipCount - 1` levels), `_color`, `_localSortingOrder` | -| `Drawables/` | `MeshRenderer` | `m_Materials`: one material of "Live2D Cubism/Lit-URP-ADV-optimize" (`keywords`, `floats`, `colors`) | - -A clip of `_motionList` is a [Mecanim clip](#note-assets-livenotesnotesjson) (`clip` is its name) with streamed and -constant curves only (no dense curves, `startTime` and `cycleOffset` 0) and an `InstanceId` animation event whose -`intParameter` names its entry of the fade motion list. Each binding is a `CubismParameter` `Value` at -`Parameters/`, a field of a root controller (path `""`: `CubismEyeBlinkController.EyeOpening`, -`CubismMouthController.MouthOpening`, `CubismRenderController.Opacity`), or unresolved (`path` null: it animates -nothing). Expression parameters whose `Id` the moc3 lacks are skipped, as in the game. -Models with `CubismPosePart` components, other blend modes or more than 36 mask groups are not supported; loading them -fails with an error that names the feature. The prefab's `canvas` is not read (the viewer reads the canvas from the -moc3). +| `Drawables/` | `CubismRenderer` | `_mainTexture` (the atlas page; unlike chart textures it may be mipmapped: the PNG is level 0 and the viewer generates the other `m_MipCount - 1` levels), `_color`, `_localSortingOrder` | +| `Drawables/` | `MeshRenderer` | `m_Materials`: one Material of "Live2D Cubism/Lit-URP-ADV-optimize" (`keywords`, `floats`, `colors`) | + +A clip of `_motionList` has streamed and constant curves only (no dense curves, `startTime` and `cycleOffset` 0), one +float curve per binding (the streamed curves, then the constant ones), and an `InstanceId` animation event whose +`intParameter` names its entry of the fade motion list. Through `model.json` `bindings`, each binding is a +`CubismParameter` `Value` at `Parameters/`, a field of a root controller (path `""`: +`CubismEyeBlinkController.EyeOpening`, `CubismMouthController.MouthOpening`, `CubismRenderController.Opacity`), or +unresolved (`path` null). Expression parameters whose `Id` the moc3 lacks are skipped, as in the game. Models with +`CubismPosePart` components, other blend modes or more than 36 mask groups are not supported; loading them fails +with an error that names the feature. ### Model shaders -`shaders/shaders.json` lists "Live2D Cubism/Lit-URP-ADV-optimize" and, for a model with masked drawables, -"Live2D Cubism/Mask", with only the GLES3 variants the players run (subshader 0, pass 0): +Of the shaders of `model.json` `shaders`, the manifest lists the parsed form (`json`), the variant index (`index`) and +only the GLES3 programs (platform `gles3`, type `GLES3`, subshader 0, pass 0) the players run: - "Live2D Cubism/Lit-URP-ADV-optimize": for each distinct keyword set of the drawables' materials (restricted to the - keywords its variants use), the variant with exactly that set, and the variant with that set plus + keywords its variants use), the program with exactly that set, and the program with that set plus `_ADDITIONAL_LIGHTS_VERTEX`. The viewer draws with the first; the story renderer adds `_ADDITIONAL_LIGHTS_VERTEX` at quality 4 (Best: per-vertex additional lights) and draws with the second there, with the first at the other - qualities. A model whose `model.json` has format 1 may lack the second; the viewer does not need it, a story does. - No player enables `_ADDITIONAL_LIGHTS`, so the variants with it are not read. -- "Live2D Cubism/Mask": its variant without keywords, when some drawable is masked (a material with `CUBISM_MASK_ON`). + qualities. No player enables `_ADDITIONAL_LIGHTS`, so the programs with it are not read. +- "Live2D Cubism/Mask": its program without keywords, when some drawable is masked (a material with `CUBISM_MASK_ON`). ## Conventions @@ -644,13 +670,15 @@ failures and a summary. A site without `charts.json` is validated from the manif - textures: the lane skin, background, jacket and film grain PNGs are present, and every PNG a descriptor refers to has the described size and `mipCount` 1. -Per model: the manifest (schema, agreement with `models.json`: `id`, `key`, the model facts, `bytes`, `files`; without -`models.json`, `id` equal to the manifest's name), every asset as for charts, `model.json` (with format 2, its -`motionSync` as the prefab gives it), the moc3 header, the prefab (the components and fields above, clip and fade -references, one Lit material and a texture descriptor per drawable), the drawables' PNGs (present, described size, -`mipCount` at most a full chain), the shader index and programs with the variants above (those with -`_ADDITIONAL_LIGHTS_VERTEX` required from `model.json` format 2), and that the manifest lists exactly the files the -viewer and the story player read. +Per model: the manifest (schema, `live2d.model`, a subject of the manifest's name, agreement with `models.json`: `id`, +`key` and `group` of the subject, `canvas` and `textures` of `model.json`, `bytes`, `files`), every asset of its files +and of its objects' files as for charts (an object file's extension by its role: `json` and `index` JSON, `image` +PNG, a program GLSL), `model.json` (schema, the class of every object it names, its `motionSync` as the prefab gives +it, the names of every motion's bindings), the moc (a CubismMoc whose `blob:_bytes` has the moc3 header), the prefab +(the components above), the motions with a fade motion for each `InstanceId`, the expressions, one Lit material and +one atlas page per drawable (the PNG of the size the texture's JSON gives, `m_MipCount` at most a full chain), the +mask materials, every shader's parsed form and variant index and the programs above, and that the manifest lists +exactly the files the viewer and the story player read. It needs Node.js 22 or later and no dependencies. The opt-in data tests (`OURNOTES_DATA= npm run test:data`) go further and run charts through the player in Node (see [CONTRIBUTING.md](../CONTRIBUTING.md)). diff --git a/docs/story-data-format.md b/docs/story-data-format.md index 7d18f45..8058b78 100644 --- a/docs/story-data-format.md +++ b/docs/story-data-format.md @@ -54,9 +54,8 @@ Contents: and an asset may be stored gzip- or brotli-encoded. Stories share the shaders, stage textures, cue sheets, UI textures and glyph pages they have in common, and the model manifests of the models they use. - Asset and model manifest paths in a story manifest are relative to the site root. A player that loads a manifest by - URL finds the site root through the manifest's `root` (format `/2`: `../` for `stories/.json`, `../../` for - `stories//.json`); for a manifest of format `/1` it takes the directory above the manifest's - directory. + URL finds the site root through the manifest's `root` (`../` for `stories/.json`, `../../` for + `stories//.json`). - `` is the story's episode id (a decimal integer, the game's `MasterAdv` id). ## stories.json @@ -94,7 +93,7 @@ An entry holds the manifest's [story facts](#story-facts) and: |---|---|---| | `id` | string | The `advId` as a decimal string. | | `manifest` | string | Path of the story manifest, relative to the site root. | -| `size` | object | `common`: the sum of `size` over the manifest's `files`; `models` (a manifest of format `/2`): the sum of `size` over the `files` of the model manifests it lists; `languages`: per language, the sum over that language group's `files` (decoded sizes, before shared assets are deduplicated). | +| `size` | object | `common`: the sum of `size` over the manifest's `files`; `models`: the sum of `size` over the files of the model manifests it lists and of their objects (an object several of them list counted once); `languages`: per language, the sum over that language group's `files` (decoded sizes, before shared assets are deduplicated). | | `regions` | string[] | Optional. The regions the manifest serves (the manifest's `regions`). An entry without it serves every region. | | `audio`, `audioFormat`, `fonts` | | Copies of the manifest keys of the same name. | @@ -171,9 +170,9 @@ the Live2D models the story uses. | Key | Type | Read by the player | Meaning | |---|---|---|---| -| `format` | `"ournotes.story-manifest/2"` \| `"ournotes.story-manifest/1"` | yes | Version of this document. `/2`: the Live2D models are model manifests of the site (`models`); `/1`, the earlier version: the models' files are among the story's common files (below). The player reads both and refuses other values. | +| `format` | `"ournotes.story-manifest/2"` | yes | Version of this document; the player refuses other values. | | `advId` | integer | no | The episode id. | -| `root` | string | yes | Format `/2`: the site root relative to the manifest, `"../"` for `stories/.json`, `"../../"` for `stories//.json`. | +| `root` | string | yes | The site root relative to the manifest, `"../"` for `stories/.json`, `"../../"` for `stories//.json`. | | `story` | object | no | The [story facts](#story-facts), handed to the page as they are. | | `regions` | string[] | no | Optional. The regions this manifest serves. | | `language` | string | yes | The default language: the language the player loads when the page names none. One of the keys of `languages`. | @@ -181,8 +180,8 @@ the Live2D models the story uses. | `audioFormat` | `"aac"` \| `"flac"` \| `null` | no | Format of every waveform file: AAC-LC in an MP4 container (`.m4a`) or FLAC; `null` without audio. | | `fonts` | `"open"` \| `"game"` | no | Where the glyphs of the [font assets](#fonts-uifontsjson) come from: font files of the producer's choice (`open`) or the game's own font assets (`game`). The format is the same; the player does not distinguish them. | | `requires` | object | yes | What the player needs to play the story: `commands` (equal to the facts' `commands`; a player that lacks one of them refuses the story before loading its files, naming them), `cubismCore` (`true`: the page must provide Live2D Cubism Core), `motionSync` (`true`: a Talk row maps a voice onto a character's lip sync, which needs Live2D's MotionSync Core with voices on). | -| `files` | object | yes | The common files: logical path → [file entry](data-format.md#file-entries). With format `/2` none of them is under `live2d/`. | -| `models` | object | yes | Format `/2`: model id → the path of its [model manifest](data-format.md#model-manifest) relative to the site root (`models/.json`), keys sorted: the models of the story's Character rows (the ids of `story.json` `models`). | +| `files` | object | yes | The common files: logical path → [file entry](data-format.md#file-entries). None of them is under `live2d/`. | +| `models` | object | yes | Model id → the path of its [model manifest](data-format.md#model-manifest) relative to the site root (`models/.json`), keys sorted: the models of the story's Character rows (the ids of `story.json` `models`). | | `host` | object | no | Optional. An Overlay episode (`playbackMode` 1) with open fonts: the screen the game plays it over and the simple talk window ([story-simple.md](story-simple.md)): `kind` (`home`: a home spot talk; `afterlive`: the reward phase of a live result), `doc` (`host/host.json`), `ui` (`ui/simple/ui.json`). The player finds these files by path. Absent for a Normal episode and for an Overlay episode with game fonts, which then has no host data and no simple talk window (the simple player refuses it). | | `languages` | object | yes | Language → `{ files }`: the files of that language (below). At least one language. | @@ -212,15 +211,18 @@ A player plays a story from one language at a time. Its file set is the union of - `files`, - `languages[].files`, -- with format `/2`, for every model of `models`, the `files` of its model manifest under `live2d//` (the model's - `model.json` is `live2d//model.json`). +- for every model of `models`, the `files` of its model manifest under `live2d//` (the model's `model.json` is + `live2d//model.json`), + +and its object table the union of the `objects` of the model manifests: an object several models use is one object, +listed alike in each (a store refuses manifests that differ on it). The logical paths are the same in every language, so the reader code does not change with the language. The model manifests are fetched together once the story manifest is read; their paths, like the asset paths, are relative to the -site root (`root`). Loading with the chart loader works on that union: `AssetStore.fromManifest` reads the `files` of -the manifest it fetches, so a story loader passes it the manifest with `files` replaced by the union (for example -through its `fetch` option), or fetches the entries itself; the entry forms and checks are the same. A language switch -loads the other language's group and keeps the common files and the models. +site root (`root`). Loading with the chart loader works on that union: `AssetStore.fromManifest` reads the `files` and +`objects` of the manifest it fetches, so a story loader passes it the manifest with both replaced by the unions (for +example through its `fetch` option), or fetches the entries itself; the entry forms and checks are the same. A +language switch loads the other language's group and keeps the common files and the models. Every listed file is fetched before the story starts. A manifest lists the files the player's data (this document) names: shader directories keep only the GLES3 programs (GLSL ES 3.00) of the shaders they list; SPIR-V containers, @@ -233,9 +235,9 @@ GLSL ES 3.10 programs and the `streams.json` of the sound directories are left o | `story.json` | Index: the paths of the files below. | common | | `episode.json` | The episode: command rows, texts, sounds, cue sheets, videos, master data row, title. | common | | `scene.json` | Player graphics, cameras, ADV fields, volumes, player settings, stages. | common | -| `shaders/shaders.json`, `shaders/…` | Shaders of the scene, the stages and the media files (format `/1`: also of the models) ([Shaders](data-format.md#shaders)). | common | +| `shaders/shaders.json`, `shaders/…` | Shaders of the scene, the stages and the media files ([Shaders](data-format.md#shaders)). | common | | `textures/*.png` | Textures of the scene and the media files ([Textures](data-format.md#textures)); paths relative to the story root. | common | -| `live2d//…` | The Live2D models ([Live2D models](#live2d-models)): format `/2`, the files of the model manifests; format `/1`, `live2d//…` among the common files. | model manifests (`/2`), common (`/1`) | +| `live2d//model.json` | The Live2D models ([Live2D models](#live2d-models)): the `model.json` of each model manifest; their objects are in the object table. | model manifests | | `audio//cues.json`, `audio//.m4a` or `.flac` | Sounds per cue sheet ([Sounds](#sounds)). | common | | `ui/ui.json` | The story UI: front canvas nodes, sprites, materials, clips, controllers, transitions, text records ([Story UI](#story-ui-uiuijson)). | language (common when equal) | | `ui/textures/*.png`, `ui/shaders/…` | Packed UI textures and the UI shaders, including the TextMeshPro distance-field shader. | common | @@ -248,8 +250,7 @@ GLSL ES 3.10 programs and the `streams.json` of the sound directories are left o | `ui/simple/ui.json`, `ui/simple/fonts.json`, `ui/simple/fonts/*.png`, `ui/simple/shaders/…` | With a `host`: the simple talk window in the story UI record format, its fonts and glyph pages in the format of `ui/fonts.json` and its text shader (paths relative to `ui/simple/`). | language (`ui.json` and the shaders common when equal) | The directory names are fixed: `ui/fonts.json` and `ui/languages.json` are found by path, the UI's texture and -shader paths are relative to `ui/`, a model's paths to its directory (`live2d//`, or the `dir` of a format `/1` -model), the scene's and the media files' texture paths to the story root. +shader paths are relative to `ui/`, the scene's and the media files' texture paths to the story root. ## story.json @@ -267,7 +268,7 @@ model), the scene's and the media files' texture paths to the story root. |---|---| | `advId` | The episode id. | | `episode`, `scene`, `ui` | Paths of [episode.json](#episodejson), [scene.json](#scenejson) and [ui/ui.json](#story-ui-uiuijson). | -| `models` | Model key (the address the episode's Character rows name, `Character/Live2D///model/`) → the model. Format `/2`: its model id (the files under `live2d//`, [Live2D models](#live2d-models)). Format `/1`: `{ dir, moc3, prefab }`, the model directory among the common files and the file names of the moc3 and the prefab in it. | +| `models` | Model key (the address the episode's Character rows name, `Character/Live2D///model/`) → the model id (its `model.json` under `live2d//`, [Live2D models](#live2d-models)). | | `modelsDir` | Only in a story directory (a story's logical files as files on disk, outside a site): the directory holding its models, `//model.json` and the files it names, relative to the story directory with `/`. A reader of the directory maps `live2d//…` there. Not in a site. | | `audio` | Cue sheet name → its directory. Empty when the manifest's `audio` is `false`. | | `frames`, `effects`, `postEffects`, `stills`, `talkWindows`, `chat` | Path of the media file of that kind, `null` when the episode does not use it. | @@ -309,7 +310,6 @@ scene, [Live scene](data-format.md#live-scene-livescenescenejson)). Keys read: | `settings.playerSettings` | `AdvPlayerSettings`: `_initializeEpisodes`, `_finalizeEpisodes` (rows with `Command` values, run before and after the episode), `_focusDataSettingsMap`, `_defaultFocusDataSettingsKey`, `_defaultPanV2FocusSlideRate`, `_defaultTransitionAssetAddress`, `_waitTalkTextUnitTime`, `_minTalkDisplayTime`, `_waitAfterVoiceTime`, `_targetNameSplitKey`, the `_allow*` flags, wait and shake values. | | `settings.masterIdSettings` | `AdvMasterIdSettings`: `_unknownCharacterNameTextId`, `_splitCharacterNameTextId`. | | `stages` | Stage key (the address after `Adv/Stage/`) → the stage prefab (node list) of every stage the episode uses. | -| `resources` | Format `/1`: `cubismMask`, `cubismMaskCulling`, the Cubism mask materials of the story's models (as in [model.json](data-format.md#modeljson)). Not read with format `/2`: each model has its own in its `model.json`. | | `postTextures` | Renderer name → post-processing textures (`filmGrainTex`, …) as texture descriptors. | | `shaders.cameraRenderers` | The shader names each camera renderer draws with. | @@ -340,21 +340,15 @@ decoded PCM of the voice file; with AAC data its samples differ from the game's ## Live2D models -Format `/2`: each model is a model of the site ([Live2D models](data-format.md#live2d-models)): its manifest -`models/.json`, listed in `models.json`, and its files, which the player loads under `live2d//`. The story -player reads them as the model viewer does: `model.json` names the moc3, the prefab (texture paths relative to the -prefab's directory) and the shader index; a character draws with its model's own shaders and with the Cubism mask -materials of its `model.json` `resources`. The story renderer adds `_ADDITIONAL_LIGHTS_VERTEX` to every character draw -at quality 4, so each model's shaders hold, for every keyword set its drawables use, the set and the set with -`_ADDITIONAL_LIGHTS_VERTEX` ([Model shaders](data-format.md#model-shaders)). A model the stories share is one -manifest. - -Format `/1`: `live2d//` among the story's common files holds per model the moc3, the prefab -(`.prefab.json`) and the atlas pages under `textures/`, in the form of a model site's files (texture paths -relative to the model directory, atlas pages may be mipmapped). The shaders the drawables use are in the story's -`shaders/`, the mask materials in `scene.json` `resources`. - -In both formats a character's MotionSync controller comes from its prefab. +Each model is a model of the site ([Live2D models](data-format.md#live2d-models)): its manifest `models/.json`, +listed in `models.json`, whose `model.json` the player loads as `live2d//model.json` and whose objects it adds to +the store's object table. The story player reads them as the model viewer does: `model.json` names the prefab, the +moc, the shaders and the Cubism mask materials by object id; a character draws with its model's own shaders and with +the mask materials of its `model.json` `resources`. The story renderer adds `_ADDITIONAL_LIGHTS_VERTEX` to every +character draw at quality 4, so each model's shaders hold, for every keyword set its drawables use, the program of the +set and of the set with `_ADDITIONAL_LIGHTS_VERTEX` ([Model shaders](data-format.md#model-shaders)). A model the +stories share is one manifest, and an object the models share (a mask material, a shader) is one object. A +character's MotionSync controller comes from its prefab. ## Story UI (`ui/ui.json`) @@ -684,7 +678,7 @@ Besides the charts and models, validates `stories.json` and every story (or the `advId`, or `/` for a region's own manifest). Per story: - the manifest (schema, the language group rules above, agreement with `stories.json`: facts, sizes, `regions`; - with format `/2`, `root` as the manifest's place in the site gives it and no file under `live2d/`); + `root` as the manifest's place in the site gives it and no file under `live2d/`); - every asset of `files` and of every language group, as for charts (present, an encoded asset's stored length and decoding, byte size, SHA-256 of the decoded bytes equal to its name, extension matching the logical file, split JSON rebuilt to its `size`, JSON parses); @@ -693,13 +687,10 @@ Besides the charts and models, validates `stories.json` and every story (or the present with the manifest's `audioFormat`, FLAC `STREAMINFO` or MP4 header agreeing with the cue; without audio no cue sheet and no waveform file); `story.json` `crilips`, when set, only with audio, both files common, the descriptor float32 little-endian with every block inside `crilips.bin`; -- the models. Format `/2`: `models` keys sorted, each path `models/.json`, the same ids as `story.json` `models` - (which names them by id), each Character row's model among them, each model manifest present (in `models.json` when - the site has one) and valid as a model ([Validation](data-format.md#validation)), and its shaders holding the - variants the story renderer draws with at every quality (each keyword set, and with `_ADDITIONAL_LIGHTS_VERTEX`; the - mask shader's without keywords). Format `/1`: `story.json` `models` in the `{ dir, moc3, prefab }` form, each - model's moc3 header, its prefab as the model viewer reads it, its atlas pages, and a `shaders/` variant for every - drawable material; +- the models: `models` keys sorted, each path `models/.json`, the same ids as `story.json` `models`, each + Character row's model among them, each model manifest present (in `models.json` when the site has one) and valid as + a model ([Validation](data-format.md#validation)), which includes the programs the story renderer draws with at + every quality; - `requires.commands` equal to the facts' `commands` and to the command names of the episode's rows without `IgnoreData` together with the player settings' initialize and finalize rows; `requires.motionSync` as the episode's rows give it; diff --git a/schema/model-json.schema.json b/schema/model-json.schema.json index ee13a81..55e0493 100644 --- a/schema/model-json.schema.json +++ b/schema/model-json.schema.json @@ -1,50 +1,74 @@ { "$schema": "https://json-schema.org/draft/2020-12/schema", "$id": "https://raw.githubusercontent.com/empty-sekai/ournotes-player/main/schema/model-json.schema.json", - "title": "Model index file (model.json)", - "description": "The logical file model.json of a Live2D model: the paths of its moc3, prefab and shader index, and the Cubism mask materials; format 2 adds motionSync. See docs/data-format.md.", + "title": "Model document (model.json)", + "description": "The file model.json of a Live2D model's product (live2d.model): the objects of its object table the player reads, by object id, and facts about the model. See docs/data-format.md.", "type": "object", - "required": ["format", "moc3", "prefab", "shaders", "resources"], - "anyOf": [ - { "properties": { "format": { "const": 1 } } }, - { "properties": { "format": { "const": 2 } }, "required": ["motionSync"] } - ], + "required": ["format", "name", "key", "prefab", "moc", "canvas", "textures", "shaders", "resources", "motionSync", "bindings"], + "additionalProperties": false, "properties": { - "format": { "enum": [1, 2] }, - "motionSync": { - "description": "Format 2: the prefab's root has a CubismMotionSyncController with a Live2DMotionSyncCriAudioInput.", - "type": "boolean" + "format": { "const": 3 }, + "name": { "description": "The model's id: the last part of its key.", "type": "string", "pattern": "^[A-Za-z0-9][A-Za-z0-9_.-]*$" }, + "key": { "description": "Character/Live2D///model/.", "type": "string" }, + "prefab": { "description": "The model's root GameObject; its json is the hierarchy (prefab.json).", "$ref": "#/$defs/objectId" }, + "moc": { "description": "The CubismMoc MonoBehaviour; the moc3 is its file blob:_bytes.", "$ref": "#/$defs/objectId" }, + "canvas": { + "description": "The moc3's canvas.", + "type": "object", + "required": ["pixelsPerUnit", "originX", "originY", "width", "height", "mocVersion"], + "additionalProperties": false, + "properties": { + "pixelsPerUnit": { "type": "number" }, + "originX": { "type": "number" }, + "originY": { "type": "number" }, + "width": { "type": "number" }, + "height": { "type": "number" }, + "mocVersion": { "type": "integer" } + } + }, + "textures": { + "description": "The atlas pages (Texture2D) the drawables draw with, by name.", + "type": "array", + "items": { "$ref": "#/$defs/objectId" } + }, + "shaders": { + "description": "Unity shader name -> Shader object: the shaders of the drawables' materials and the Cubism mask materials.", + "type": "object", + "additionalProperties": { "$ref": "#/$defs/objectId" } }, - "name": { "type": "string" }, - "key": { "type": "string" }, - "moc3": { "type": "string", "pattern": "\\.moc3$" }, - "prefab": { "type": "string", "pattern": "\\.json$" }, - "textures": { "type": "array", "items": { "type": "string", "pattern": "\\.png$" } }, - "shaders": { "type": "string", "pattern": "(^|/)shaders\\.json$" }, "resources": { + "description": "The Material objects the Cubism mask pass draws with (Resources/Live2D/Cubism/Materials).", "type": "object", "required": ["cubismMask", "cubismMaskCulling"], + "additionalProperties": false, "properties": { - "cubismMask": { "$ref": "#/$defs/maskMaterial" }, - "cubismMaskCulling": { "$ref": "#/$defs/maskMaterial" } + "cubismMask": { "$ref": "#/$defs/objectId" }, + "cubismMaskCulling": { "$ref": "#/$defs/objectId" } } - } - }, - "$defs": { - "maskMaterial": { + }, + "motionSync": { + "description": "The prefab's root has a CubismMotionSyncController with a Live2DMotionSyncCriAudioInput.", + "type": "boolean" + }, + "bindings": { + "description": "AnimationClip object -> per binding of the clip, the path below the model's root (\"\" for the root) and the attribute it animates; both null where the hierarchy names no single one (pathCandidates: the paths that fit).", "type": "object", - "required": ["shader", "floats"], - "properties": { - "material": { "type": "string" }, - "shader": { + "additionalProperties": { + "type": "array", + "items": { "type": "object", - "required": ["shader"], - "properties": { "shader": { "const": "Live2D Cubism/Mask" } } - }, - "keywords": { "type": "array", "items": { "type": "string" } }, - "floats": { "type": "object", "required": ["_Cull"], "properties": { "_Cull": { "enum": [0, 1, 2] } } }, - "colors": { "type": "object" } + "required": ["path", "attribute"], + "additionalProperties": false, + "properties": { + "path": { "type": ["string", "null"] }, + "attribute": { "type": ["string", "null"] }, + "pathCandidates": { "type": "array", "items": { "type": "string" } } + } + } } } + }, + "$defs": { + "objectId": { "description": "An object of the product's object table: :.", "type": "string", "pattern": "^.+:-?[0-9]+$" } } } diff --git a/schema/model.schema.json b/schema/model.schema.json deleted file mode 100644 index 5ac8ee8..0000000 --- a/schema/model.schema.json +++ /dev/null @@ -1,103 +0,0 @@ -{ - "$schema": "https://json-schema.org/draft/2020-12/schema", - "$id": "https://raw.githubusercontent.com/empty-sekai/ournotes-player/main/schema/model.schema.json", - "title": "Model manifest (models/.json)", - "description": "The logical files of one Live2D model and where their bytes are; the file entries are those of a chart manifest. See docs/data-format.md.", - "type": "object", - "required": ["format", "id", "files"], - "properties": { - "format": { "enum": [2, 3], "description": "3: the current form (files may be encoded assets; model.json of format 2); 2: the older form." }, - "id": { "type": "string", "pattern": "^[a-z0-9_]+$" }, - "key": { "type": "string" }, - "model": { - "type": "object", - "properties": { - "group": { "type": "string" }, - "canvas": { "$ref": "#/$defs/canvas" }, - "textures": { "type": "integer", "minimum": 0 }, - "nodes": { "type": "integer", "minimum": 0 }, - "character": { "type": "integer" }, - "names": { "$ref": "#/$defs/names" }, - "label": { "type": "string" } - } - }, - "files": { - "type": "object", - "propertyNames": { "pattern": "^[A-Za-z0-9_][^\\\\]*$" }, - "additionalProperties": { - "anyOf": [{ "$ref": "#/$defs/whole" }, { "$ref": "#/$defs/encoded" }, { "$ref": "#/$defs/parts" }] - } - } - }, - "$defs": { - "names": { - "type": "object", - "propertyNames": { "minLength": 1 }, - "additionalProperties": { "type": "string", "minLength": 1 } - }, - "canvas": { - "type": "object", - "properties": { - "pixelsPerUnit": { "type": "number" }, - "originX": { "type": "number" }, - "originY": { "type": "number" }, - "width": { "type": "number" }, - "height": { "type": "number" }, - "mocVersion": { "type": "integer" } - } - }, - "asset": { "type": "string", "pattern": "^assets/[0-9a-f]{64}\\.[a-z0-9]+$" }, - "encodedAsset": { - "description": "An asset stored gzip- or brotli-encoded: the SHA-256 of its decoded bytes, the logical file's extension, then .gz or .br.", - "type": "string", - "pattern": "^assets/[0-9a-f]{64}\\.[a-z0-9]+\\.(gz|br)$" - }, - "size": { "description": "The byte count of the file (decoded).", "type": "integer", "minimum": 0 }, - "stored": { "description": "The byte count of an encoded asset as stored (less than size).", "type": "integer", "minimum": 0 }, - "whole": { - "type": "object", - "required": ["asset", "size"], - "additionalProperties": false, - "properties": { "asset": { "$ref": "#/$defs/asset" }, "size": { "$ref": "#/$defs/size" } } - }, - "encoded": { - "type": "object", - "required": ["asset", "size", "stored"], - "additionalProperties": false, - "properties": { - "asset": { "$ref": "#/$defs/encodedAsset" }, - "size": { "$ref": "#/$defs/size" }, - "stored": { "$ref": "#/$defs/stored" } - } - }, - "parts": { - "type": "object", - "required": ["parts", "size"], - "additionalProperties": false, - "properties": { - "parts": { - "type": "array", - "minItems": 1, - "items": { - "anyOf": [ - { - "type": "array", - "prefixItems": [{ "type": "string" }, { "$ref": "#/$defs/asset" }, { "$ref": "#/$defs/size" }], - "minItems": 3, - "maxItems": 3 - }, - { - "type": "array", - "prefixItems": [{ "type": "string" }, { "$ref": "#/$defs/encodedAsset" }, { "$ref": "#/$defs/size" }, - { "$ref": "#/$defs/stored" }], - "minItems": 4, - "maxItems": 4 - } - ] - } - }, - "size": { "$ref": "#/$defs/size" } - } - } - } -} diff --git a/schema/product.schema.json b/schema/product.schema.json new file mode 100644 index 0000000..ea36a38 --- /dev/null +++ b/schema/product.schema.json @@ -0,0 +1,65 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://raw.githubusercontent.com/empty-sekai/ournotes-player/main/schema/product.schema.json", + "title": "Product manifest (nnnotes.product/1)", + "description": "A product's files and the files of the objects it uses (its object table), each a content-addressed asset relative to the site root; a Live2D model's manifest models/.json is one, of product live2d.model. See docs/data-format.md.", + "type": "object", + "required": ["schema", "product", "subject", "files", "objects"], + "additionalProperties": false, + "properties": { + "schema": { "const": "nnnotes.product/1" }, + "product": { "type": "string", "pattern": "^[a-z][a-z0-9]*(\\.[a-z][a-z0-9-]*)+$" }, + "subject": { "type": "string", "minLength": 1 }, + "files": { + "description": "The product's own files by logical path.", + "type": "object", + "propertyNames": { "pattern": "^[A-Za-z0-9_][^\\\\]*$" }, + "additionalProperties": { "$ref": "#/$defs/file" } + }, + "objects": { + "description": "Object id (:) -> the object's class, name and files by role (json, image, index, program:/, blob:).", + "type": "object", + "propertyNames": { "pattern": "^.+:-?[0-9]+$" }, + "additionalProperties": { + "type": "object", + "required": ["class", "files"], + "additionalProperties": false, + "properties": { + "class": { "type": "string", "minLength": 1 }, + "name": { "type": "string" }, + "files": { + "type": "object", + "additionalProperties": { "$ref": "#/$defs/file" } + } + } + } + } + }, + "$defs": { + "file": { "anyOf": [{ "$ref": "#/$defs/whole" }, { "$ref": "#/$defs/encoded" }] }, + "asset": { "type": "string", "pattern": "^assets/[0-9a-f]{64}\\.[a-z0-9]+$" }, + "encodedAsset": { + "description": "An asset stored gzip- or brotli-encoded: the SHA-256 of its decoded bytes, the file's extension, then .gz or .br.", + "type": "string", + "pattern": "^assets/[0-9a-f]{64}\\.[a-z0-9]+\\.(gz|br)$" + }, + "size": { "description": "The byte count of the file (decoded).", "type": "integer", "minimum": 0 }, + "stored": { "description": "The byte count of an encoded asset as stored (less than size).", "type": "integer", "minimum": 0 }, + "whole": { + "type": "object", + "required": ["asset", "size"], + "additionalProperties": false, + "properties": { "asset": { "$ref": "#/$defs/asset" }, "size": { "$ref": "#/$defs/size" } } + }, + "encoded": { + "type": "object", + "required": ["asset", "size", "stored"], + "additionalProperties": false, + "properties": { + "asset": { "$ref": "#/$defs/encodedAsset" }, + "size": { "$ref": "#/$defs/size" }, + "stored": { "$ref": "#/$defs/stored" } + } + } + } +} diff --git a/schema/story-manifest.schema.json b/schema/story-manifest.schema.json index 4507d6a..884fe6c 100644 --- a/schema/story-manifest.schema.json +++ b/schema/story-manifest.schema.json @@ -4,24 +4,17 @@ "title": "Story manifest (stories/.json, or stories//.json)", "description": "The logical files of one story (common files and one group per language) and where their bytes are; the file entries are those of a chart manifest. See docs/story-data-format.md.", "type": "object", - "required": ["format", "advId", "story", "language", "audio", "audioFormat", "fonts", "requires", "files", "languages"], - "anyOf": [ - { "properties": { "format": { "const": "ournotes.story-manifest/1" } } }, - { "properties": { "format": { "const": "ournotes.story-manifest/2" } }, "required": ["root", "models"] } - ], + "required": ["format", "advId", "root", "models", "story", "language", "audio", "audioFormat", "fonts", "requires", "files", "languages"], "properties": { - "format": { - "description": "ournotes.story-manifest/2: the Live2D models are model manifests of the site (models); /1: the models' files are among the story's files.", - "enum": ["ournotes.story-manifest/1", "ournotes.story-manifest/2"] - }, + "format": { "const": "ournotes.story-manifest/2" }, "advId": { "type": "integer", "minimum": 1 }, "root": { - "description": "Format /2: the site root relative to the manifest: \"../\" for stories/.json, \"../../\" for stories//.json.", + "description": "The site root relative to the manifest: \"../\" for stories/.json, \"../../\" for stories//.json.", "type": "string", "pattern": "^(\\.\\./)+$" }, "models": { - "description": "Format /2: model id -> the path of its model manifest relative to the site root (models/.json), keys sorted: the Live2D models the story's Character rows use.", + "description": "Model id -> the path of its model manifest (a product manifest of live2d.model) relative to the site root (models/.json), keys sorted: the Live2D models the story's Character rows use.", "type": "object", "propertyNames": { "pattern": "^[a-z0-9_]+$" }, "additionalProperties": { "type": "string", "pattern": "^models/[a-z0-9_]+\\.json$" } diff --git a/schema/story.schema.json b/schema/story.schema.json index 4a84dbb..355bf22 100644 --- a/schema/story.schema.json +++ b/schema/story.schema.json @@ -11,23 +11,10 @@ "scene": { "$ref": "#/$defs/path" }, "ui": { "$ref": "#/$defs/path" }, "models": { - "description": "Model address (the address the Character rows name) -> the model: its model id (story manifest /2; the model's files are under live2d// of the story's file set), or {dir, moc3, prefab} (story manifest /1; the model's files are in dir among the story's files).", + "description": "Model address (the address the Character rows name) -> the model's id: its model.json is under live2d// of the story's file set, its objects in the store's object table.", "type": "object", "propertyNames": { "pattern": "^Character/Live2D/[^/]+/[^/]+/model/[^/]+$" }, - "additionalProperties": { - "anyOf": [ - { "type": "string", "pattern": "^[a-z0-9_]+$" }, - { - "type": "object", - "required": ["dir", "moc3", "prefab"], - "properties": { - "dir": { "$ref": "#/$defs/path" }, - "moc3": { "type": "string", "pattern": "^[^/\\\\]+\\.moc3$" }, - "prefab": { "type": "string", "pattern": "^[^/\\\\]+\\.json$" } - } - } - ] - } + "additionalProperties": { "type": "string", "pattern": "^[a-z0-9_]+$" } }, "modelsDir": { "description": "A story directory only (not a site): the directory holding the models (//model.json, ...), relative to the story directory, \"/\"-separated.", diff --git a/scripts/lib/headless.mjs b/scripts/lib/headless.mjs index c49da49..75a857a 100644 --- a/scripts/lib/headless.mjs +++ b/scripts/lib/headless.mjs @@ -149,6 +149,7 @@ const pngSize = (b) => { // images decode to their size only ({width, height, close}): enough for texture uploads to the headless context export const headlessImages = (store) => { store.image = async (p) => ({ ...pngSize(Buffer.from(store.bytes(p))), close() {} }); + store.objectImage = async (id, role = "image") => ({ ...pngSize(Buffer.from(store.objectBytes(id, role))), close() {} }); return store; }; diff --git a/scripts/lib/story-author/script.js b/scripts/lib/story-author/script.js index ef4ce03..6909e63 100644 --- a/scripts/lib/story-author/script.js +++ b/scripts/lib/story-author/script.js @@ -15,6 +15,7 @@ // window). compileScript turns a script without errors into the episode's command rows (episode.json) and the story // index (story.json). +import { componentClass } from "../../../src/engine/prefab.js"; import { ADV_COMMAND } from "../../../src/story/interfaces.js"; import { STORY_LANGUAGES } from "../../../src/story/params.js"; @@ -306,14 +307,17 @@ export const stageBackground = (prefab) => { // ------------------------------------------------------------------------------------------------ models -// The motions and expressions of a model, from its prefab (the Live2DCharacter component of the root node): -// {motions: clip names, expressions, defaultMotion, defaultExpression}. -export const modelActions = (prefab) => { - const root = ((prefab && prefab.nodes) || [])[0]; - const ch = root && (root.components || []).find((c) => c && c.class === "Live2DCharacter"); +// The motions and expressions of a model (`model`: its model.json; `assets`: the AssetStore of its objects), from the +// Live2DCharacter component of its prefab's root: {motions: the names of its clips, expressions, defaultMotion, +// defaultExpression}. +export const modelActions = (assets, model) => { + const root = assets.prefab(model.prefab).nodes[0]; + const ch = root && root.components.find((c) => componentClass(c) === "Live2DCharacter"); if (!ch) throw new Error("the model prefab has no Live2DCharacter"); - return { motions: (ch._motionList || []).map((m) => m.clip).filter(Boolean), expressions: [...(ch._expressionList || [])], - defaultMotion: ch.DefaultMotionName || "", defaultExpression: ch.DefaultExpressionName || "" }; + const f = ch.fields; + return { motions: (f._motionList || []).map((r) => assets.clip(assets.ref(r, "AnimationClip")).name), + expressions: [...(f._expressionList || [])], + defaultMotion: f.DefaultMotionName || "", defaultExpression: f.DefaultExpressionName || "" }; }; // "mtn_smile02_L" -> {stem: "mtn_smile02", side: "L"}; a name without a side is its own stem @@ -504,7 +508,7 @@ export const parseScript = (doc) => { // The names of a script against its site. `site`: // base {languages, language, stages: stageChoices(...), music: musicChoices(...)} of the script's base story // models the model ids of the site's models.json (an array or a Set) -// actions model id -> modelActions(prefab), for the models of the cast that exist +// actions model id -> modelActions, for the models of the cast that exist // -> {language, errors, warnings}: language, the language the story plays in; error codes unknownLanguage, // unknownModel, unknownCast, unknownStage, unknownMusic, unknownMotion, unknownExpression, beforeScene, // positionTaken, offStage; warning codes speakerOffStage (a line of a character not on stage), unusedCast. @@ -580,7 +584,7 @@ export const checkScript = (script, site) => { // base {story, episode, stages, music} of the base story (its story.json and episode.json; stageChoices, // musicChoices) // models model id -> model key (Character/Live2D///model/) of the site's models.json -// actions model id -> modelActions(prefab) +// actions model id -> modelActions // language: the story language. -> {episode, story, lines: [{line, beat, path, id, speaker, name, text}], scenes} export const compileScript = (script, site, language) => { const field = STORY_LANGUAGES[language]; diff --git a/scripts/lib/story-author/site.js b/scripts/lib/story-author/site.js index 0657768..2542b25 100644 --- a/scripts/lib/story-author/site.js +++ b/scripts/lib/story-author/site.js @@ -6,7 +6,7 @@ import fs from "node:fs"; import path from "node:path"; import { pathToFileURL } from "node:url"; import { AssetStore } from "../../../src/data/assets.js"; -import { MODEL_MANIFEST_FORMATS, STORY_MANIFEST_FORMAT } from "../../../src/story/assets.js"; +import { STORY_MANIFEST_FORMAT, checkModelManifest } from "../../../src/story/assets.js"; import { StoryPlayerCore } from "../../../src/story/player-core.js"; import "../../../src/story/features/index.js"; import { fileFetch, nodeDecode } from "../headless.mjs"; @@ -56,9 +56,10 @@ export class Site { return doc.models; } - // An AssetStore of `files` (logical path -> file entry, asset paths relative to the site root) - store(files) { - const url = pathToFileURL(path.join(this.dir, ".story-author-files.json")).href, body = { files }; + // An AssetStore of `files` (logical path -> file entry, asset paths relative to the site root) and `objects` (an + // object table of a product manifest) + store(files, objects = {}) { + const url = pathToFileURL(path.join(this.dir, ".story-author-files.json")).href, body = { files, objects }; const fetch = (u) => (String(u) === url ? Promise.resolve({ ok: true, status: 200, json: async () => body }) : fileFetch(u)); return AssetStore.fromManifest(url, { fetch, base: pathToFileURL(this.dir + path.sep).href, decode: nodeDecode }); } @@ -108,11 +109,12 @@ export class Site { const entry = this.models().find((m) => m.id === id); if (!entry) throw new SiteError(`model ${id} is not in models.json`); const manifest = this.json(entry.manifest); - if (!manifest || !MODEL_MANIFEST_FORMATS.includes(manifest.format) || !manifest.files) - throw new SiteError(`${entry.manifest}: not a model manifest`); - const index = (await this.store({ "model.json": manifest.files["model.json"] })).json("model.json"); - const prefab = (await this.store({ [index.prefab]: manifest.files[index.prefab] })).json(index.prefab); - return { entry, manifest, actions: modelActions(prefab) }; + try { checkModelManifest(manifest, id, entry.manifest); } catch (e) { throw new SiteError(e.message); } + // model.json and the objects modelActions reads: the prefab and the clips + const { prefab } = (await this.store({ "model.json": manifest.files["model.json"] })).json("model.json"); + const objects = Object.fromEntries(Object.entries(manifest.objects).filter(([oid, o]) => oid === prefab || o.class === "AnimationClip")); + const store = await this.store({ "model.json": manifest.files["model.json"] }, objects); + return { entry, manifest, actions: modelActions(store, store.json("model.json")) }; } // the bytes of logical files of a base story (path -> Buffer), from its common files diff --git a/scripts/validate-data.mjs b/scripts/validate-data.mjs index 24aa36b..0489a21 100644 --- a/scripts/validate-data.mjs +++ b/scripts/validate-data.mjs @@ -9,23 +9,24 @@ // id references, waveform files and their FLAC / MP4 headers), livescene/scene.json and livenotes/notes.json (the // structures the player reads, animation references, node order), both shader directories (index, parsed shader data, // GLSL ES 3.00 stage blocks) and the texture descriptors (PNG present where required, size as described). -// Live2D models (models.json, models/.json): the manifest (schema, agreement with models.json), every asset, -// model.json (its motionSync against the prefab), the moc3 header, the prefab (the components the model viewer reads, -// clip and fade references, drawable materials and textures), the shader index and programs (with model.json format -// 2 also the variants the story renderer adds a keyword to), and that the manifest lists exactly the files the viewer -// and the story player read. +// Live2D models (models.json, models/.json, product manifests of live2d.model): the manifest (schema, subject, +// agreement with models.json), every asset of its files and of its objects' files, model.json (the classes of the +// objects it names, its motionSync against the prefab, the names of every motion's bindings), the moc (a CubismMoc, +// its moc3 header), the prefab (the components the model viewer reads), the motions and their fade motions, the +// expressions, the drawables' materials and textures, the mask materials, the shaders' parsed forms, variant indexes +// and GLES3 programs (also the variants the story renderer adds a keyword to), and that the manifest lists exactly the +// files the viewer and the story player read. // charts.json of a site of several regions: an id at most once per region (an entry without `regions` serves every // region, so its id only once), every entry's regions among the index's `regions`, text languages among `languages`. -// Without charts.json the manifests are charts/*.json and charts//*.json. A models.json entry's key and model -// facts (group, canvas, textures) must equal its manifest's `key` and `model` where both have them, and its character, -// names and label where either has them. +// Without charts.json the manifests are charts/*.json and charts//*.json. A models.json entry's id, key and +// group must be those of its manifest's subject, its canvas and textures those of model.json, its bytes and files those +// of the manifest's files and its objects' files. // Stories (stories.json, stories/.json, stories//.json; docs/story-data-format.md): the manifest // (schema, root, language groups, agreement with stories.json), every asset of the common files and of each language // group, story.json, episode.json, the required commands against the episode's rows and the player settings, the cue -// sheets (cues.json, waveform files and their FLAC / MP4 headers), the Live2D models (manifest format /2: the model -// manifests the manifest lists, each valid as a model, one per model story.json names, with the shader variants the -// story renderer draws at every quality, no story file under live2d/; format /1: each model's moc3 header, prefab, -// textures and the story's shader variants), the Animator controller and clip references of the media files, both +// sheets (cues.json, waveform files and their FLAC / MP4 headers), the Live2D models (the model manifests the manifest +// lists, each valid as a model, one per model story.json names; no story file under live2d/), the Animator controller +// and clip references of the media files, both // shader directories, texture // descriptors, and per language ui/ui.json, ui/languages.json and ui/fonts.json (text bindings, the dialog, chat // window and frame bindings, font assets with their fallbacks and missing glyph, fallback materials, sprite assets, @@ -47,13 +48,12 @@ const here = path.dirname(fileURLToPath(import.meta.url)); const schema = (name) => compile(JSON.parse(fs.readFileSync(path.join(here, "..", "schema", `${name}.schema.json`), "utf8"))); const S = { charts: schema("charts"), manifest: schema("manifest"), live: schema("live"), audio: schema("live-audio"), - score: schema("score"), models: schema("models"), model: schema("model"), modelJson: schema("model-json"), + score: schema("score"), models: schema("models"), product: schema("product"), modelJson: schema("model-json"), stories: schema("stories"), storyManifest: schema("story-manifest"), story: schema("story"), episode: schema("episode"), storyFonts: schema("story-fonts"), storyLanguage: schema("story-language"), }; const STORY_HOST_FORMAT = "ournotes.story-host/1"; -const STORY_MANIFEST_SITED = "ournotes.story-manifest/2"; const extOf = (p) => (p.match(/\.([A-Za-z0-9]+)$/) || [])[1]?.toLowerCase() ?? ""; // encoded assets (assets/..gz / .br): the encoding, the logical file's extension, the extensions that may // be encoded (every other file is stored as it is) @@ -160,7 +160,7 @@ class Site { let r, model = null; try { model = new Model(this, id, JSON.parse(fs.readFileSync(this.file(manifest), "utf8")), manifest); - r = { model, errs: model.run(null), missing: false }; + r = { model, errs: model.run(), missing: false }; } catch (e) { r = { model, errs: [e.code === "ENOENT" ? "missing" : e.message], missing: e.code === "ENOENT" }; } this.models.set(manifest, r); } @@ -247,6 +247,9 @@ class Chart { if (e.id !== `${m.musicId}_${m.difficulty}`) this.err("charts.json", `id ${e.id} does not match the manifest`); } + // the extension the asset of logical file `p` must have (null: any) + extOfFile(p) { return extOf(p); } + checkAssets() { for (const [p, f] of Object.entries(this.files)) { if (f.parts) { @@ -263,7 +266,8 @@ class Chart { } if (total !== f.size) this.err(p, `rebuilt text is ${total} bytes, manifest ${f.size}`); } else { - if (assetExt(f.asset) !== extOf(p)) this.err(p, `asset extension .${assetExt(f.asset)} differs from the file's`); + const want = this.extOfFile(p); + if (want !== null && assetExt(f.asset) !== want) this.err(p, `asset extension .${assetExt(f.asset)}, expected .${want}`); const e = this.site.checkAsset(f.asset, f.size, f.stored); if (e) this.err(p, e); } @@ -413,13 +417,7 @@ class Chart { if (!r || r.errs.length) return; for (const rec of r.list) { const pp = `${dir}/${rec.parsed}`; - this.json("shaderinfo", pp, (s) => { - const errs = []; - if (!Array.isArray(s.subShaders) || !s.subShaders.every((x) => Array.isArray(x.passes) && x.passes.every((y) => isObj(y.state)))) - errs.push("subShaders[].passes[].state missing"); - if (s.properties !== undefined && !Array.isArray(s.properties)) errs.push("properties is not an array"); - return { errs }; - }); + this.json("shaderinfo", pp, (s) => ({ errs: parsedShaderErrors(s) })); for (const f of rec.files) { const fp = `${dir}/${f}`; if (!this.has(fp)) { this.err(idx, `${rec.name}: ${fp} not in the manifest`); continue; } @@ -463,6 +461,8 @@ const isDescriptor = (o) => isObj(o) && typeof o.texture === "string" && /\.png$ Number.isInteger(o.width) && Number.isInteger(o.height); const desc = (o) => ({ texture: o.texture, width: o.width, height: o.height, mipCount: o.mipCount ?? 1, name: o.name }); const dedupe = (ds) => [...new Map(ds.map((d) => [`${d.texture}|${d.width}|${d.height}|${d.mipCount}`, d])).values()]; +const dirOf = (p) => (p.includes("/") ? p.slice(0, p.lastIndexOf("/")) : ""); +const inDir = (dir, p) => (dir ? `${dir}/${p}` : p); // every object inside v (skipping the top-level keys in `skip`) function walk(v, fn, skip = []) { @@ -482,142 +482,281 @@ function walk(v, fn, skip = []) { const LIT_SHADER = "Live2D Cubism/Lit-URP-ADV-optimize", MASK_SHADER = "Live2D Cubism/Mask"; const ROOT_COMPONENTS = ["Live2DCharacter", "CubismFadeController", "CubismExpressionController", "CubismRenderController", "CubismAutoEyeBlinkInput", "CubismEyeBlinkController", "CubismMouthController", "CubismHarmonicMotionController"]; -const dirOf = (p) => (p.includes("/") ? p.slice(0, p.lastIndexOf("/")) : ""); -const inDir = (dir, p) => (dir ? `${dir}/${p}` : p); - -// the prefab as the model viewer reads it: {errs, textures (descriptors), keywordSets, masked} +const MODEL_PRODUCT = "live2d.model"; +const MODEL_KEY = /^Character\/Live2D\/([^/]+)\/([^/]+)\/model\/\2$/; +// the programs the player draws with (ShaderLib.fromObjects) +const SHADER_PLATFORM = "gles3", SHADER_PROGRAM_TYPE = "GLES3"; +// a prefab component's class: a MonoBehaviour's script class, else its type +const classOf = (c) => (typeof c.script === "string" ? c.script.slice(c.script.lastIndexOf("|") + 1) : c.type); +// the script class of a MonoBehaviour's document (mono.json $script) +const scriptOf = (v) => (isObj(v) && typeof v.$script === "string" ? v.$script.slice(v.$script.lastIndexOf("|") + 1) : null); +// the object id a reference names ({"$ref": {file, pathId}}): null for null, undefined for anything else +const refOf = (v) => (v === null ? null + : isObj(v) && isObj(v.$ref) && typeof v.$ref.file === "string" && typeof v.$ref.pathId === "string" ? `${v.$ref.file}:${v.$ref.pathId}` + : undefined); + +// the prefab as the model viewer reads it: {errs, ch (the Live2DCharacter's fields), clips, fadeList, expressionList +// (object ids; expressionList null without one), drawables: [{path, texture, material}], motionSync} function modelPrefab(pf) { const errs = []; if (!isObj(pf)) return { errs: ["not an object"] }; nodeList(pf, "prefab", errs); if (errs.length) return { errs }; - const root = pf.nodes[0], comp = (cls) => root.components.filter((c) => c.class === cls); + const root = pf.nodes[0], comp = (cls) => root.components.filter((c) => classOf(c) === cls); for (const cls of ROOT_COMPONENTS) if (comp(cls).length !== 1) errs.push(`root ${root.path}: ${comp(cls).length} ${cls}`); if (comp("CubismPhysicsController").length > 1) errs.push(`root ${root.path}: more than one CubismPhysicsController`); if (errs.length) return { errs }; - const ch = comp("Live2DCharacter")[0], fl = comp("CubismFadeController")[0].CubismFadeMotionList; - const clips = Array.isArray(ch._motionList) ? ch._motionList : []; + const fields = (cls) => (isObj(comp(cls)[0].fields) ? comp(cls)[0].fields : {}); + const ch = fields("Live2DCharacter"); + const clips = (Array.isArray(ch._motionList) ? ch._motionList : []).map(refOf); if (!clips.length) errs.push("Live2DCharacter._motionList is empty"); - const fadeIds = new Set(isObj(fl) && Array.isArray(fl.MotionInstanceIds) ? fl.MotionInstanceIds : []); - for (const c of clips) { - const ev = (c.events || []).filter((e) => e.functionName === "InstanceId"); - if (!ev.length || !fadeIds.has(ev[ev.length - 1].intParameter)) errs.push(`clip ${c.clip}: no fade motion for its InstanceId`); - } - if (!clips.some((c) => c.clip === ch.DefaultMotionName)) errs.push(`default motion ${ch.DefaultMotionName} is not a clip`); - const ex = comp("CubismExpressionController")[0]; - const exprs = isObj(ex.ExpressionsList) && Array.isArray(ex.ExpressionsList.CubismExpressionObjects) - ? ex.ExpressionsList.CubismExpressionObjects.map((e) => String(e.name).replace(/\.exp3$/, "")) : []; + if (clips.some((c) => !c)) errs.push("Live2DCharacter._motionList: an entry is not a reference"); + const fadeList = refOf(fields("CubismFadeController").CubismFadeMotionList); + if (!fadeList) errs.push("CubismFadeController.CubismFadeMotionList is not a reference"); + const ex = fields("CubismExpressionController"); if (ex.UseLegacyBlendCalculation) errs.push("legacy expression blend is not supported"); - if (ch.DefaultExpressionName && !exprs.includes(ch.DefaultExpressionName)) errs.push(`default expression ${ch.DefaultExpressionName} is not in ExpressionsList`); - const textures = [], kw = new Map(); - let drawables = 0, masked = false; + const expressionList = refOf(ex.ExpressionsList); + if (expressionList === undefined) errs.push("CubismExpressionController.ExpressionsList is not a reference"); + const drawables = []; + let nodes = 0; for (const n of pf.nodes) { - if (!n.components.some((c) => c.class === "CubismDrawable")) continue; - drawables++; - const r = n.components.find((c) => c.class === "CubismRenderer"), mr = n.components.find((c) => c.type === "MeshRenderer"); - if (!r || !isDescriptor(r._mainTexture)) { errs.push(`${n.path}: CubismRenderer._mainTexture is not a texture descriptor`); continue; } - textures.push(desc(r._mainTexture)); - const mats = mr && Array.isArray(mr.m_Materials) ? mr.m_Materials : []; - if (mats.length !== 1 || !isObj(mats[0].shader) || mats[0].shader.shader !== LIT_SHADER) { - errs.push(`${n.path}: needs one material of ${LIT_SHADER}`); continue; - } - const k = Array.isArray(mats[0].keywords) ? [...mats[0].keywords].sort() : []; - kw.set(k.join(" "), k); - if (k.includes("CUBISM_MASK_ON")) masked = true; + if (!n.components.some((c) => classOf(c) === "CubismDrawable")) continue; + nodes++; + const r = n.components.find((c) => classOf(c) === "CubismRenderer"), mr = n.components.find((c) => c.type === "MeshRenderer"); + const texture = r && isObj(r.fields) ? refOf(r.fields._mainTexture) : undefined; + const mats = mr && isObj(mr.fields) && Array.isArray(mr.fields.m_Materials) ? mr.fields.m_Materials : []; + if (!texture) errs.push(`${n.path}: CubismRenderer._mainTexture is not a reference`); + else if (mats.length !== 1 || !refOf(mats[0])) errs.push(`${n.path}: needs one material`); + else drawables.push({ path: n.path, texture, material: refOf(mats[0]) }); } - if (!drawables) errs.push("no CubismDrawable nodes"); - if (pf.nodes.some((n) => n.components.some((c) => c.class === "CubismPosePart"))) errs.push("pose parts are not supported"); + if (!nodes) errs.push("no CubismDrawable nodes"); + if (pf.nodes.some((n) => n.components.some((c) => classOf(c) === "CubismPosePart"))) errs.push("pose parts are not supported"); // a MotionSync controller with a CRI audio input on the root (CubismMotionSyncController.fromPrefab's test) const motionSync = comp("CubismMotionSyncController").length > 0 && comp("Live2DMotionSyncCriAudioInput").length > 0; - return { errs, textures: dedupe(textures), keywordSets: [...kw.values()], masked, motionSync }; + return { errs, ch, clips, fadeList, expressionList, drawables, motionSync }; } +// a parsed shader (a Shader object's json, the parsed file of a shader directory): the problems found +const parsedShaderErrors = (s) => { + if (!isObj(s) || !Array.isArray(s.subShaders) || !s.subShaders.every((x) => Array.isArray(x.passes) && x.passes.every((y) => isObj(y.state)))) + return ["subShaders[].passes[].state missing"]; + return s.properties !== undefined && !Array.isArray(s.properties) ? ["properties is not an array"] : []; +}; + // the keyword the story renderer adds to every character draw at Best (StoryRenderer.characterKeywords); a model's -// Lit variants hold each material keyword set with and without it +// Lit programs hold each material keyword set with and without it const STORY_LIGHTS = "_ADDITIONAL_LIGHTS_VERTEX"; +// A model's product manifest (models/.json). Its files are model.json and, as "#", the files of +// its objects (the AssetStore's names for them). class Model extends Chart { + constructor(site, id, man, where = null) { + super(site, id, man, where); + this.objects = isObj(man.objects) ? man.objects : {}; + this.files = { ...(isObj(man.files) ? man.files : {}) }; + for (const [oid, o] of Object.entries(this.objects)) + if (isObj(o) && isObj(o.files)) for (const [role, f] of Object.entries(o.files)) this.files[`${oid}#${role}`] = f; + this.index = null; // model.json, once the model is valid + this.read = new Set(); // the files the model viewer and the story player read + } + + // an object's file by role: json and index JSON, image PNG, a program GLSL, a blob any + extOfFile(p) { + const cut = p.indexOf("#"); + if (cut < 0) return extOf(p); + const role = p.slice(cut + 1); + return role === "json" || role === "index" ? "json" : role === "image" ? "png" : role.startsWith("program:") ? "glsl" : null; + } + // a models.json entry against the manifest: the problems found (not added to the model's own) entryErrors(entry) { - const m = this.man, saved = this.errs; - this.errs = []; - try { - if (entry.id !== m.id) this.err("models.json", `id ${entry.id}, manifest id ${m.id}`); - if (has(entry, "key") && has(m, "key") && entry.key !== m.key) this.err("models.json", "key differs from the manifest"); - if (isObj(m.model)) - for (const k of ["group", "canvas", "textures"]) - if (has(entry, k) && has(m.model, k) && JSON.stringify(entry[k]) !== JSON.stringify(m.model[k])) - this.err("models.json", `${k} differs from the manifest's model.${k}`); - const facts = isObj(m.model) ? m.model : {}; - for (const k of ["character", "names", "label"]) - if ((has(entry, k) || has(facts, k)) && JSON.stringify(entry[k]) !== JSON.stringify(facts[k])) - this.err("models.json", `${k} differs from the manifest's model.${k}`); - const total = Object.values(this.files).reduce((n, f) => n + (f.size || 0), 0); - if (has(entry, "bytes") && entry.bytes !== total) this.err("models.json", `bytes ${entry.bytes}, manifest files total ${total}`); - if (has(entry, "files") && entry.files !== Object.keys(this.files).length) this.err("models.json", `files ${entry.files}, manifest ${Object.keys(this.files).length}`); - return this.errs; - } finally { this.errs = saved; } + const m = this.man, out = [], key = MODEL_KEY.exec(String(m.subject)), idx = this.index; + const e = (msg) => out.push(`models.json: ${msg}`); + if (key && entry.id !== key[2]) e(`id ${entry.id}, the manifest's model ${key[2]}`); + if (has(entry, "key") && entry.key !== m.subject) e(`key ${entry.key}, manifest subject ${m.subject}`); + if (has(entry, "group") && key && entry.group !== key[1]) e(`group ${entry.group}, the manifest subject's ${key[1]}`); + if (idx && has(entry, "canvas") && !sameJson(entry.canvas, idx.canvas)) e("canvas differs from model.json canvas"); + if (idx && has(entry, "textures") && entry.textures !== idx.textures.length) e(`textures ${entry.textures}, model.json ${idx.textures.length}`); + const total = sizeOf(this.files), count = Object.keys(this.files).length; + if (has(entry, "bytes") && entry.bytes !== total) e(`bytes ${entry.bytes}, manifest files total ${total}`); + if (has(entry, "files") && entry.files !== count) e(`files ${entry.files}, manifest ${count}`); + return out; } - // the model's own checks (entry: its models.json entry, checked too) - run(entry) { + // the object `id` is in the manifest, of class `cls` (`where`: the reference's place, for errors) + object(id, cls, where) { + const o = this.objects[id]; + if (!isObj(o)) { this.err(where, `object ${id} not in the manifest`); return false; } + if (o.class !== cls) { this.err(where, `${id}: a ${o.class}, not a ${cls}`); return false; } + return true; + } + + // the file of `role` of object `id`, read: its name, null when the object has none + role(id, role, where) { + const p = `${id}#${role}`; + if (!this.has(p)) { this.err(where, `${id}: no ${role} file`); return null; } + this.read.add(p); + return p; + } + + // the parsed JSON of the file of `role` of object `id`, read and checked by `check` (cached per asset and `kind`), + // or null + doc(kind, id, role, where, check = () => []) { + const p = this.role(id, role, where); + if (!p) return null; + const r = this.json(kind, p, (v) => { const errs = check(v); return { errs, v: errs.length ? null : v }; }); + return r ? r.v : null; + } + + run() { const m = this.man; - for (const e of S.model(m)) this.err("manifest", e); - if (!isObj(m.files)) return this.errs; + for (const e of S.product(m)) this.err("manifest", e); + if (this.errs.length) return this.errs; + if (m.product !== MODEL_PRODUCT) this.err("manifest", `product ${m.product}, a model is ${MODEL_PRODUCT}`); + const key = MODEL_KEY.exec(m.subject); + if (!key) this.err("manifest", `subject ${m.subject} is not a model key (Character/Live2D///model/)`); const name = this.where ? path.posix.basename(this.where, ".json") : null; - if (name !== null && has(m, "id") && m.id !== name) this.err("manifest", `id ${m.id}, expected ${name} (the manifest's name)`); - if (entry) this.errs.push(...this.entryErrors(entry)); + if (key && name !== null && key[2] !== name) this.err("manifest", `subject: model ${key[2]}, expected ${name} (the manifest's name)`); + if (Object.keys(m.files).join("\n") !== "model.json") this.err("manifest", "files: model.json is the one file of a model"); this.checkAssets(); if (this.errs.length) return this.errs; const mj = this.json("modeljson", "model.json", (v) => ({ errs: S.modelJson(v), v: S.modelJson(v).length ? null : v })); const idx = mj && mj.v; if (!idx) return this.errs; - for (const k of ["moc3", "prefab", "shaders"]) if (!this.has(idx[k])) this.err("model.json", `${k}: ${idx[k]} not in the manifest`); + this.read.add("model.json"); + if (idx.key !== m.subject) this.err("model.json", `key ${idx.key}, manifest subject ${m.subject}`); + if (key && idx.name !== key[2]) this.err("model.json", `name ${idx.name}, the key's model ${key[2]}`); + const named = [[idx.prefab, "GameObject", "prefab"], [idx.moc, "MonoBehaviour", "moc"], + ...idx.textures.map((id) => [id, "Texture2D", "textures"]), + ...Object.entries(idx.shaders).map(([n, id]) => [id, "Shader", `shaders.${n}`]), + ...Object.entries(idx.resources).map(([n, id]) => [id, "Material", `resources.${n}`]), + ...Object.keys(idx.bindings).map((id) => [id, "AnimationClip", "bindings"])]; + for (const [id, cls, what] of named) this.object(id, cls, `model.json ${what}`); if (this.errs.length) return this.errs; - const moc = this.site.bytes(this.files[idx.moc3].asset); - if (moc.length < 8 || moc.toString("latin1", 0, 4) !== "MOC3") this.err(idx.moc3, "not a moc3 file"); - const pf = this.json("prefab", idx.prefab, modelPrefab); + + // the moc: a CubismMoc whose blob:_bytes is a moc3 + this.doc("mocjson", idx.moc, "json", "model.json moc", (v) => (scriptOf(v) === "CubismMoc" ? [] : ["not a CubismMoc"])); + const mp = this.role(idx.moc, "blob:_bytes", "model.json moc"); + const moc3 = (asset) => { const b = this.site.bytes(asset); return b.length >= 8 && b.toString("latin1", 0, 4) === "MOC3"; }; + if (mp && !this.site.once("moc3", this.files[mp].asset, () => moc3(this.files[mp].asset))) this.err(mp, "not a moc3 file"); + const pp = this.role(idx.prefab, "json", "model.json prefab"); + const pf = pp && this.json("prefab", pp, modelPrefab); if (!pf || pf.errs.length) return this.errs; - if (idx.format >= 2 && idx.motionSync !== pf.motionSync) + if (idx.motionSync !== pf.motionSync) this.err("model.json", `motionSync ${idx.motionSync}, the prefab's root ${pf.motionSync ? "has" : "has no"} MotionSync controller with a CRI audio input`); - const read = new Set(["model.json", idx.moc3, idx.prefab]); - const dir = dirOf(idx.prefab); - for (const d of pf.textures) { - const p = inDir(dir, d.texture); - read.add(p); - if (!this.has(p)) { this.err(p, "texture of a drawable not in the manifest"); continue; } + + // motions: each clip of the motion list with the names of its bindings and a fade motion for its InstanceId + const fadeIds = new Set(); + const fl = this.object(pf.fadeList, "MonoBehaviour", "CubismFadeController.CubismFadeMotionList") + ? this.doc("fadelist", pf.fadeList, "json", "fade motion list", (v) => (Array.isArray(v.MotionInstanceIds) && Array.isArray(v.CubismFadeMotionObjects) + && v.MotionInstanceIds.length === v.CubismFadeMotionObjects.length ? [] : ["MotionInstanceIds and CubismFadeMotionObjects are not lists of one length"])) + : null; + for (const [i, r] of (fl ? fl.CubismFadeMotionObjects : []).entries()) { + const id = refOf(r); + if (!id) { this.err("fade motion list", `CubismFadeMotionObjects[${i}] is not a reference`); continue; } + if (!this.object(id, "MonoBehaviour", "fade motion list")) continue; + if (this.doc("fade", id, "json", "fade motion list", (v) => (scriptOf(v) === "CubismFadeMotionData" ? [] : ["not a CubismFadeMotionData"]))) + fadeIds.add(fl.MotionInstanceIds[i]); + } + const clipNames = []; + for (const id of pf.clips) { + if (!this.object(id, "AnimationClip", "Live2DCharacter._motionList")) continue; + const c = this.doc("clip", id, "json", "Live2DCharacter._motionList", + (v) => (isObj(v) && typeof v.name === "string" && Array.isArray(v.bindings) && Array.isArray(v.events) ? [] : ["not a clip (name, bindings, events)"])); + if (!c) continue; + clipNames.push(c.name); + const names = idx.bindings[id]; + if (!Array.isArray(names)) this.err("model.json", `bindings: clip ${c.name} (${id}) missing`); + else if (names.length !== c.bindings.length) this.err("model.json", `bindings.${id}: ${names.length} names, clip ${c.name} has ${c.bindings.length} bindings`); + const ev = c.events.filter((e) => e.functionName === "InstanceId"); + if (!ev.length || !fadeIds.has(ev[ev.length - 1].intParameter)) this.err(`clip ${c.name}`, "no fade motion for its InstanceId"); + } + for (const id of Object.keys(idx.bindings)) if (!pf.clips.includes(id)) this.err("model.json", `bindings.${id}: not a clip of Live2DCharacter._motionList`); + if (!clipNames.includes(pf.ch.DefaultMotionName)) this.err("prefab", `default motion ${pf.ch.DefaultMotionName} is not a clip`); + + // expressions + const exprNames = []; + if (pf.expressionList !== null && this.object(pf.expressionList, "MonoBehaviour", "CubismExpressionController.ExpressionsList")) { + const el = this.doc("exprlist", pf.expressionList, "json", "expression list", + (v) => (Array.isArray(v.CubismExpressionObjects) ? [] : ["CubismExpressionObjects is not a list"])); + for (const [i, r] of (el ? el.CubismExpressionObjects : []).entries()) { + const id = refOf(r); + if (!id) { this.err("expression list", `CubismExpressionObjects[${i}] is not a reference`); continue; } + if (!this.object(id, "MonoBehaviour", "expression list")) continue; + const e = this.doc("expr", id, "json", "expression list", + (v) => (scriptOf(v) === "CubismExpressionData" && typeof v.m_Name === "string" ? [] : ["not a CubismExpressionData"])); + if (e) exprNames.push(e.m_Name.replace(/\.exp3$/, "")); + } + } + if (pf.ch.DefaultExpressionName && !exprNames.includes(pf.ch.DefaultExpressionName)) + this.err("prefab", `default expression ${pf.ch.DefaultExpressionName} is not in ExpressionsList`); + + // materials: the drawables' draw with the Lit shader (their keyword sets), the mask materials with the mask shader + const material = (id, where) => (this.object(id, "Material", where) + ? this.doc("material", id, "json", where, (v) => (isObj(v) && Array.isArray(v.keywords) && refOf(v.shader) ? [] : ["not a material (shader, keywords)"])) + : null); + const kw = new Map(); + let masked = false; + for (const d of pf.drawables) { + const mat = material(d.material, d.path); + if (!mat) continue; + if (refOf(mat.shader) !== idx.shaders[LIT_SHADER]) { this.err(d.path, `material ${mat.material} is not drawn with ${LIT_SHADER} (model.json shaders)`); continue; } + const k = [...mat.keywords].sort(); + kw.set(k.join(" "), k); + if (k.includes("CUBISM_MASK_ON")) masked = true; + } + for (const [n, id] of Object.entries(idx.resources)) { + const mat = material(id, `model.json resources.${n}`); + if (mat && refOf(mat.shader) !== idx.shaders[MASK_SHADER]) this.err("model.json", `resources.${n}: material ${mat.material} is not drawn with ${MASK_SHADER}`); + } + + // textures: model.json's are the drawables' main textures, each a PNG of its size + const pages = [...new Set(pf.drawables.map((d) => d.texture))].sort(); + if (!sameJson([...idx.textures].sort(), pages)) this.err("model.json", "textures differ from the drawables' main textures"); + for (const id of pages) { + if (!this.object(id, "Texture2D", "a drawable's main texture")) continue; + const t = this.doc("texture", id, "json", `texture ${id}`, + (v) => (["m_Width", "m_Height", "m_MipCount"].every((k) => Number.isInteger(v[k])) ? [] : ["m_Width, m_Height, m_MipCount missing"])); + const p = this.role(id, "image", `texture ${id}`); + if (!t || !p) continue; const asset = this.files[p].asset; + if (assetExt(asset) !== "png") { this.err(p, "not a .png asset"); continue; } const info = this.site.once("png", asset, () => { try { return pngInfo(this.site.bytes(asset)); } catch (e) { return { error: e.message }; } }); if (info.error) this.err(p, info.error); - else if (info.width !== d.width || info.height !== d.height) this.err(p, `PNG is ${info.width}x${info.height}, descriptor says ${d.width}x${d.height}`); - const full = Math.floor(Math.log2(Math.max(d.width, d.height))) + 1; - if (!(Number.isInteger(d.mipCount) && d.mipCount >= 1 && d.mipCount <= full)) this.err(p, `mipCount ${d.mipCount} (1 to ${full})`); + else if (info.width !== t.m_Width || info.height !== t.m_Height) this.err(p, `PNG is ${info.width}x${info.height}, the texture ${t.m_Width}x${t.m_Height}`); + const full = Math.floor(Math.log2(Math.max(t.m_Width, t.m_Height))) + 1; + if (!(t.m_MipCount >= 1 && t.m_MipCount <= full)) this.err(p, `m_MipCount ${t.m_MipCount} (1 to ${full})`); } - const sdir = dirOf(idx.shaders), before = this.errs.length; - this.checkShaders(sdir); - if (this.errs.length > before) return this.errs; - read.add(idx.shaders); - const list = JSON.parse(this.text(idx.shaders)); - // the variants read: the model viewer's (each material keyword set of the Lit shader, the mask shader's without - // keywords) and the story renderer's at Best (each set with STORY_LIGHTS: required from model.json format 2) - const need = [[LIT_SHADER, pf.keywordSets], ...(pf.masked ? [[MASK_SHADER, [[]]]] : [])]; - for (const [name, sets] of need) { - const rec = list.find((r) => r.name === name); - if (!rec) { this.err(idx.shaders, `${name} not in the index`); continue; } - read.add(inDir(sdir, rec.parsed)); + + // shaders: of each, the parsed form and the variant index (ShaderLib reads both), and the GLES3 programs read: the + // model viewer's (each material keyword set of the Lit shader, the mask shader's without keywords when a drawable is + // masked) and the story renderer's at Best (each set with STORY_LIGHTS) + const list = []; + for (const [n, id] of Object.entries(idx.shaders)) { + this.doc("shaderinfo", id, "json", `shader ${n}`, parsedShaderErrors); + const index = this.doc("shaderindex", id, "index", `shader ${n}`, + (v) => (isObj(v) && v.name === n && Array.isArray(v.variants) ? [] : [`not the variant index of ${n}`])); + if (index) list.push({ name: n, id, variants: index.variants.filter((v) => v.platform === SHADER_PLATFORM && v.type === SHADER_PROGRAM_TYPE) }); + } + const program = (rec, v) => { + const p = this.role(rec.id, v.role, `shader ${rec.name}`); + if (p) for (const e of this.site.once("glsl", this.files[p].asset, () => glslStages(this.text(p)))) this.err(p, e); + }; + for (const [n, sets] of [[LIT_SHADER, [...kw.values()]], ...(masked ? [[MASK_SHADER, [[]]]] : [])]) { + const rec = list.find((r) => r.name === n); + if (!rec) { this.err("model.json", `shaders: ${n} missing`); continue; } for (const set of sets) { - const v = variantFor(list, name, set); - if (v.error) this.err(idx.shaders, v.error); - else read.add(inDir(sdir, v.variant.file)); - if (name !== LIT_SHADER) continue; - const s = storyVariant(list, set); - if (!s.error) read.add(inDir(sdir, s.variant.file)); - else if (idx.format >= 2) this.err(idx.shaders, s.error); + const v = variantFor(list, n, set); + if (v.error) this.err(`shader ${n}`, v.error); + else program(rec, v.variant); + if (n !== LIT_SHADER) continue; + const sv = storyVariant(list, set); + if (sv.error) this.err(`shader ${n}`, sv.error); + else program(rec, sv.variant); } } - for (const p of Object.keys(this.files)) if (!read.has(p)) this.err(p, "listed but read by neither the model viewer nor the story player"); - // what a story using the model checks (Story.checkModels) - this.index = idx; this.prefabInfo = pf; this.shaderList = list; this.shaderDir = sdir; + for (const p of Object.keys(this.files)) if (!this.read.has(p)) this.err(p, "listed but read by neither the model viewer nor the story player"); + if (!this.errs.length) this.index = idx; return this.errs; } } @@ -699,8 +838,6 @@ function storyVariant(list, keywords) { class Story extends Chart { constructor(site, id, man, where = null) { super(site, id, man, where); - // manifest format /2: the models in model manifests of the site (format /1: their files among the story's) - this.sited = man.format === STORY_MANIFEST_SITED; this.common = this.files; this.groups = isObj(man.languages) ? man.languages : {}; this.lang = null; // the language group being checked (error prefix) @@ -780,17 +917,15 @@ class Story extends Chart { checkGroups() { const m = this.man; - if (this.sited) { - // root: the site root seen from the manifest (stories/.json, stories//.json) - if (this.where) { - const root = "../".repeat(this.where.split("/").length - 1); - if (m.root !== root) this.err("manifest", `root ${JSON.stringify(m.root)}, the site root from ${this.where} is ${JSON.stringify(root)}`); - } - // the store the player builds: the common files, one group and the models' files under live2d// - for (const [where, files] of [["files", m.files], ...Object.entries(m.languages).map(([l, g]) => [`languages.${l}`, isObj(g) ? g.files : null])]) - for (const p of Object.keys(isObj(files) ? files : {})) - if (p.startsWith("live2d/")) this.err("manifest", `${where}: ${p}: live2d/ holds the files of the models (manifest models)`); + // root: the site root seen from the manifest (stories/.json, stories//.json) + if (this.where) { + const root = "../".repeat(this.where.split("/").length - 1); + if (m.root !== root) this.err("manifest", `root ${JSON.stringify(m.root)}, the site root from ${this.where} is ${JSON.stringify(root)}`); } + // the store the player builds: the common files, one group and each model's model.json under live2d// + for (const [where, files] of [["files", m.files], ...Object.entries(m.languages).map(([l, g]) => [`languages.${l}`, isObj(g) ? g.files : null])]) + for (const p of Object.keys(isObj(files) ? files : {})) + if (p.startsWith("live2d/")) this.err("manifest", `${where}: ${p}: live2d/ holds the files of the models (manifest models)`); if (!has(m.languages, m.language)) this.err("manifest", `language ${m.language} is not a key of languages`); for (const [lang, g] of Object.entries(m.languages)) { if (!isObj(g) || !isObj(g.files)) continue; @@ -812,20 +947,21 @@ class Story extends Chart { for (const k of ["audio", "audioFormat", "fonts"]) if (e[k] !== m[k]) this.err("stories.json", `${k} ${JSON.stringify(e[k])}, manifest ${JSON.stringify(m[k])}`); if (isObj(m.story)) for (const [k, v] of Object.entries(m.story)) if (!sameJson(e[k], v)) this.err("stories.json", `${k} differs from the manifest's story.${k}`); if ((has(e, "regions") || has(m, "regions")) && !sameJson(e.regions, m.regions)) this.err("stories.json", "regions differ from the manifest"); - const size = { common: sizeOf(m.files), ...(this.sited ? { models: this.modelsSize() } : {}), + const size = { common: sizeOf(m.files), models: this.modelsSize(), languages: Object.fromEntries(Object.entries(this.groups).map(([l, g]) => [l, sizeOf(g.files || {})])) }; if (size.models !== null && !sameJson(e.size, size)) this.err("stories.json", `size ${JSON.stringify(e.size)}, manifest files ${JSON.stringify(size)}`); } - // the sum of size over the files of the listed model manifests (null while one of them is missing or unreadable) + // the sum of size over the files of the listed model manifests and of their objects, an object several of them list + // counted once (null while one of them is missing or unreadable) modelsSize() { - let n = 0; + const files = {}; for (const [id, p] of Object.entries(isObj(this.man.models) ? this.man.models : {})) { const r = p === `models/${id}.json` ? this.site.model(id, p) : null; if (!r || !r.model) return null; - n += sizeOf(isObj(r.model.man.files) ? r.model.man.files : {}); + for (const [k, f] of Object.entries(r.model.files)) files[k.includes("#") ? k : `${id}/${k}`] = f; } - return n; + return sizeOf(files); } // the files every language shares: story.json, episode.json, scene.json, commands, sounds, models, shaders, textures @@ -855,21 +991,9 @@ class Story extends Chart { if (m.requires.motionSync !== lipSync) this.err("manifest", `requires.motionSync ${m.requires.motionSync}, the episode ${lipSync ? "has" : "has no"} lip-synced Talk rows`); this.checkSounds(story, episode); - const models = this.sited ? this.checkModels(story, episode) - : Object.entries(story.models).map(([key, md]) => this.checkEmbeddedModel(key, md)).filter(Boolean); + const models = this.checkModels(story, episode); if (this.has("shaders/shaders.json")) this.checkShaders("shaders"); else if (models.length) this.err("shaders/shaders.json", "not in the manifest"); - // format /1: the story's shaders draw its models - if (!this.sited && models.length && this.has("shaders/shaders.json")) { - const list = JSON.parse(this.text("shaders/shaders.json")); - const sets = new Map(); - for (const r of models) for (const k of r.keywordSets) sets.set(k.join(" "), k); - const need = [[LIT_SHADER, [...sets.values()]], ...(models.some((r) => r.masked) ? [[MASK_SHADER, [[]]]] : [])]; - for (const [name, ks] of need) for (const k of ks) { - const v = variantFor(list, name, k); - if (v.error) this.err("shaders/shaders.json", v.error); - } - } this.descriptors(scene, ""); this.frameTexts = {}; for (const k of STORY_MEDIA) if (story[k] !== null && k !== "videos") { @@ -963,15 +1087,14 @@ class Story extends Chart { // the Live2D models (story.json models: address -> id; the manifest's models: id -> models/.json): the same ids // in both, keys sorted, each Character row's model among them, each model manifest present (listed in models.json - // when the site has one) and valid as a model, with the variants the story renderer draws its materials with at - // every quality (the set; the set with STORY_LIGHTS at Best; the mask shader's without keywords). Returns the valid - // models. + // when the site has one) and valid as a model (with the programs the story renderer draws its materials with at + // every quality). Returns the valid models. checkModels(story, episode) { const m = this.man, listed = isObj(m.models) ? m.models : {}, out = []; const ids = Object.keys(listed); if (ids.some((k, i) => i > 0 && k <= ids[i - 1])) this.err("manifest", "models: keys not sorted"); for (const [address, id] of Object.entries(story.models)) - if (typeof id !== "string") this.err("story.json", `models.${address}: not a model id (a story of ${STORY_MANIFEST_SITED} names its models by id)`); + if (typeof id !== "string") this.err("story.json", `models.${address}: not a model id`); else if (!has(listed, id)) this.err("story.json", `models.${address}: model ${id} is not in the manifest's models`); const named = new Set(Object.values(story.models)); for (const id of ids) if (!named.has(id)) this.err("manifest", `models.${id}: no model address of story.json names it`); @@ -991,35 +1114,11 @@ class Story extends Chart { this.err(p, `not a valid model (${r.errs.length} problem${r.errs.length > 1 ? "s" : ""}): ${r.errs.slice(0, 3).join("; ")}`); continue; } - const md = r.model, list = md.shaderList, where = `${p}: ${inDir(md.shaderDir, "shaders.json")}`; - for (const set of md.prefabInfo.keywordSets) { - for (const v of [variantFor(list, LIT_SHADER, set), storyVariant(list, set)]) if (v.error) this.err(where, v.error); - } - if (md.prefabInfo.masked) { const v = variantFor(list, MASK_SHADER, []); if (v.error) this.err(where, v.error); } - out.push(md); + out.push(r.model); } return out; } - // a Live2D model of a story of manifest format /1 (story.json models: address -> {dir, moc3, prefab}, its files - // among the story's common files): moc3 header, prefab (as the model viewer reads it), atlas pages - checkEmbeddedModel(key, md) { - if (!isObj(md)) { this.err("story.json", `models.${key}: not {dir, moc3, prefab} (a story of format /1 holds its models' files)`); return null; } - const moc = inDir(md.dir, md.moc3), pf = inDir(md.dir, md.prefab); - for (const p of [moc, pf]) if (!has(this.common, p)) { this.err("story.json", `models.${key}: ${p} is not a common file`); return null; } - const asset = this.files[moc].asset; - if (!this.site.once("moc3", asset, () => { const b = this.site.bytes(asset); return b.length >= 8 && b.toString("latin1", 0, 4) === "MOC3"; })) - this.err(moc, "not a moc3 file"); - const r = this.json("prefab", pf, modelPrefab); - if (!r || r.errs.length) return null; - for (const d of r.textures) { - const p = inDir(md.dir, d.texture); - if (!this.has(p)) this.err(p, "texture of a drawable not in the manifest"); - else this.png(p, d, false); - } - return r; - } - // one language: ui/ui.json, ui/languages.json, ui/fonts.json checkLanguage(lang, story) { const m = this.man; diff --git a/src/data/assets.js b/src/data/assets.js index aa11efe..dd821bf 100644 --- a/src/data/assets.js +++ b/src/data/assets.js @@ -1,20 +1,43 @@ -// AssetStore: the files of one chart (path -> text or bytes), read synchronously by the session once loaded. +// AssetStore: the files of one chart or product (path -> text or bytes) and the objects it uses (object id -> class, +// name and files by role), read synchronously by the session once loaded. // // Sources: -// AssetStore.fromManifest(url) a chart manifest ({files: {path: {asset, size}}, ...}) and the content-addressed -// assets it lists, all fetched before the session starts. A large JSON file is listed -// per top-level key ({parts: [[key, asset, size], ...], size}); its text is rebuilt as +// AssetStore.fromManifest(url) a manifest ({files: {path: {asset, size}}, ...}) and the content-addressed assets +// it lists, all fetched before the session starts. A large JSON file is listed per +// top-level key ({parts: [[key, asset, size], ...], size}); its text is rebuilt as // {"k1":t1,"k2":t2,...} from the parts' raw texts (numbers stay exactly as stored). // An encoded asset (assets/..gz or .br) has its stored byte count too // ({asset, size, stored}, [key, asset, size, stored]); it is decoded here unless the -// server already did (Content-Encoding). -// new AssetStore({ text, bytes, info }) in memory: path -> string and path -> Uint8Array / ArrayBuffer. -// Text files are the .json and .glsl files; every other file is binary. +// server already did (Content-Encoding). A product manifest (nnnotes.product/1) also +// lists its objects ({objects: {id: {class, name, files: {role: {asset, size}}}}}), +// fetched the same way. +// new AssetStore({ text, bytes, objects, info }) in memory: path -> string, path -> Uint8Array / ArrayBuffer, and +// object id -> {class, name, files: {role: string | Uint8Array | ArrayBuffer}}. +// Text files are the .json and .glsl files; every other file is binary. An object's file is text when its asset is a +// .json or .glsl asset (in memory: when it is given as a string). +// +// Objects (docs/data-format.md "Objects"): an object id is ":", the path id in decimal. Every +// object has a `json` file; a reference in it, {"$ref": {"file", "pathId"}}, names the object ":" +// (refId). The typed readers (prefab, material, clip, mono, texture) check the object's class. + +// the schema of a product manifest +export const PRODUCT_MANIFEST = "nnnotes.product/1"; export const TEXT_FILE = /\.(json|glsl)$/; +// the assets whose bytes are text: a .json or .glsl asset, encoded or not +export const TEXT_ASSET = /\.(json|glsl)(\.(gz|br))?$/; const entries = (m) => (m instanceof Map ? [...m.entries()] : Object.entries(m || {})); +// The object id a reference names: {"$ref": {file, pathId}} -> ":"; null for a null reference. +export const refId = (v) => { + if (v === null || v === undefined) return null; + const r = v.$ref; + if (!r || typeof r.file !== "string" || typeof r.pathId !== "string") + throw new Error(`not a reference to an object of a known file: ${JSON.stringify(v)}`); + return `${r.file}:${r.pathId}`; +}; + // the content encoding of an asset by its name: "gzip" (.gz), "br" (.br), or null (stored as it is) export const assetEncoding = (asset) => { const m = /\.(gz|br)$/.exec(String(asset)); @@ -40,11 +63,22 @@ export const decodeAsset = async (bytes, encoding) => { const u8 = (b) => (b instanceof Uint8Array ? new Uint8Array(b.buffer, b.byteOffset, b.byteLength) : new Uint8Array(b)); export class AssetStore { - // text: path -> string; bytes: path -> Uint8Array | ArrayBuffer; info: the chart manifest without `files` (or null) - constructor({ text = {}, bytes = {}, info = null } = {}) { + // text: path -> string; bytes: path -> Uint8Array | ArrayBuffer; objects: object id -> {class, name, files: role -> + // string | Uint8Array | ArrayBuffer}; info: the manifest without `files` and `objects` (or null) + constructor({ text = {}, bytes = {}, objects = {}, info = null } = {}) { this._text = new Map(entries(text)); this._bin = new Map(); for (const [p, b] of entries(bytes)) this._bin.set(p, b instanceof Uint8Array ? b : new Uint8Array(b)); + this._objects = new Map(); + for (const [id, o] of entries(objects)) { + if (!o || typeof o.class !== "string") throw new Error(`object ${id}: no class`); + const files = new Map(); + for (const [role, f] of entries(o.files)) + files.set(role, typeof f === "string" || f instanceof Uint8Array ? f : new Uint8Array(f)); + if (!files.has("json")) throw new Error(`object ${id}: no json file`); + this._objects.set(id, { class: o.class, name: o.name ?? null, files }); + } + this._parsed = new Map(); // "#" -> the parsed JSON of an object's file this.info = info; } @@ -83,9 +117,15 @@ export class AssetStore { }; const man = await (await get(manifestUrl.href)).json(); requestSignal.throwIfAborted(); - if (!man || typeof man.files !== "object") throw new Error(`${manifestUrl.href}: not a chart manifest`); + if (!man || typeof man.files !== "object") throw new Error(`${manifestUrl.href}: not a manifest`); + if (man.objects !== undefined && (!man.objects || typeof man.objects !== "object")) + throw new Error(`${manifestUrl.href}: objects is not an object table`); + // every file to fetch: [where, entry]; an object's file is "#" + const slots = Object.entries(man.files); + for (const [id, o] of Object.entries(man.objects || {})) + for (const [role, f] of Object.entries((o && o.files) || {})) slots.push([`${id}#${role}`, f]); const jobs = new Map(); // asset -> {size, stored, path}: each asset fetched once - for (const [path, f] of Object.entries(man.files)) { + for (const [path, f] of slots) { const list = f.parts ? f.parts.map((p) => [p[1], p[2], p[3]]) : [[f.asset, f.size, f.stored]]; for (const [asset, size, stored] of list) jobs.set(asset, { size, stored, path }); } @@ -117,17 +157,30 @@ export class AssetStore { }; await Promise.all(Array.from({ length: Math.max(1, Math.min(concurrency, list.length)) }, worker)); requestSignal.throwIfAborted(); - const text = new Map(), bin = new Map(), enc = new TextEncoder(); - for (const [path, f] of Object.entries(man.files)) { + const enc = new TextEncoder(); + // a file's text (`asText`, a rebuilt file always) or bytes + const content = (path, f, asText) => { if (f.parts) { const t = `{${f.parts.map(([k, asset]) => `${JSON.stringify(k)}:${dec.decode(got.get(asset))}`).join(",")}}`; if (enc.encode(t).byteLength !== f.size) throw new Error(`${path}: rebuilt text is not ${f.size} bytes`); - text.set(path, t); - } else if (TEXT_FILE.test(path)) text.set(path, dec.decode(got.get(f.asset))); - else bin.set(path, got.get(f.asset)); + return t; + } + return asText ? dec.decode(got.get(f.asset)) : got.get(f.asset); + }; + const text = new Map(), bin = new Map(); + for (const [path, f] of Object.entries(man.files)) { + const c = content(path, f, TEXT_FILE.test(path)); + if (typeof c === "string") text.set(path, c); else bin.set(path, c); } - const { files, ...info } = man; - return new AssetStore({ text, bytes: bin, info }); + const objects = {}; + for (const [id, o] of Object.entries(man.objects || {})) { + const files = {}; + for (const [role, f] of Object.entries(o.files || {})) + files[role] = content(`${id}#${role}`, f, TEXT_ASSET.test(f.asset ?? "")); + objects[id] = { class: o.class, name: o.name, files }; + } + const { files: _files, objects: _objects, ...info } = man; + return new AssetStore({ text, bytes: bin, objects, info }); } catch (e) { controller.abort(e); throw requestSignal.reason; @@ -172,6 +225,90 @@ export class AssetStore { list(prefix = "") { return [...this._text.keys(), ...this._bin.keys()].filter((k) => k.startsWith(prefix)); } + + // ---- objects + + hasObject(id) { return this._objects.has(id); } + + // {id, class, name} of an object; `cls`: the class it must have + object(id, cls = null) { + const o = this._objects.get(id); + if (!o) throw new Error(`object not found: ${id}`); + if (cls !== null && o.class !== cls) throw new Error(`${id}: a ${o.class}, not a ${cls}`); + return { id, class: o.class, name: o.name }; + } + + // the roles of an object's files (json, image, index, program:..., blob:...) + roles(id) { + const o = this._objects.get(id); + if (!o) throw new Error(`object not found: ${id}`); + return [...o.files.keys()]; + } + + _file(id, role) { + const o = this._objects.get(id); + if (!o) throw new Error(`object not found: ${id}`); + const f = o.files.get(role); + if (f === undefined) throw new Error(`${id} (${o.class}${o.name ? ` ${o.name}` : ""}): no ${role} file`); + return f; + } + + objectText(id, role) { + const f = this._file(id, role); + return typeof f === "string" ? f : new TextDecoder("utf-8").decode(f); + } + + // the parsed JSON of an object's file, parsed once and shared: callers do not modify it + objectJson(id, role = "json") { + const k = `${id}#${role}`; + let v = this._parsed.get(k); + if (v === undefined) { v = JSON.parse(this.objectText(id, role)); this._parsed.set(k, v); } + return v; + } + + // a copy (callers may transfer or detach it) + objectBytes(id, role) { + const f = this._file(id, role); + return typeof f === "string" ? new TextEncoder().encode(f) : f.slice(); + } + + // an image file of an object, decoded as image() decodes a PNG + async objectImage(id, role = "image") { + const blob = new Blob([this.objectBytes(id, role)], { type: "image/png" }); + return createImageBitmap(blob, { premultiplyAlpha: "none", colorSpaceConversion: "none", imageOrientation: "flipY" }); + } + + // the object a reference value names (refId), checked to be in the store and of class `cls` when given; null for a + // null reference + ref(v, cls = null) { + const id = refId(v); + if (id !== null) this.object(id, cls); + return id; + } + + // ---- typed readers: an object's JSON, its class checked (shared: callers do not modify it) + + // a root GameObject's prefab.json: {name, nodes: [{path, name, pathId, active, layer, localPosition, localRotation, + // localScale, components: [{type, pathId, script, fields}]}]} + prefab(id) { this.object(id, "GameObject"); return this.objectJson(id); } + + // material.json: {material, shader (a reference), keywords, floats, colors, textures, ...} + material(id) { this.object(id, "Material"); return this.objectJson(id); } + + // clip.json: the Mecanim clip data and its bindings as stored + clip(id) { this.object(id, "AnimationClip"); return this.objectJson(id); } + + // mono.json: a MonoBehaviour outside a hierarchy: {$script, m_Name, ...its fields} + mono(id) { this.object(id, "MonoBehaviour"); return this.objectJson(id); } + + // a Texture2D: {id, name, width, height, mipCount, colorSpace, settings} from its typetree; its pixels are + // objectImage(id) + texture(id) { + this.object(id, "Texture2D"); + const t = this.objectJson(id); + return { id, name: t.m_Name, width: t.m_Width, height: t.m_Height, mipCount: t.m_MipCount, + colorSpace: t.m_ColorSpace, settings: t.m_TextureSettings }; + } } // The store of the session that draws into a WebGL context. Shader libraries and texture loads reach the files through diff --git a/src/engine/glsl.js b/src/engine/glsl.js index 9d7d3a4..d409ba2 100644 --- a/src/engine/glsl.js +++ b/src/engine/glsl.js @@ -311,26 +311,54 @@ export const applyState = (gl, s) => { } }; -// Shader index + variant selection + program cache. `base` is a packed shader -// directory (shaders.json + /gles3/*.glsl + .json). +// The GPU programs a ShaderLib draws with: the GLES3 sub-programs (GLSL ES 3.00) of the shaders' gles3 platform. +export const SHADER_PLATFORM = "gles3", SHADER_PROGRAM_TYPE = "GLES3"; +const RECORDS = Symbol("shader records"); + +// Shader index + variant selection + program cache. The shaders are named by their Unity names: +// new ShaderLib(gl, base, assets) a packed shader directory `base` (shaders.json + +// /gles3/*.glsl + .json) +// ShaderLib.fromObjects(gl, shaders, assets) the Shader objects of a shader table {name: object id}: each +// object's `json` (the parsed form), `index` (its variants) and +// `program:/` files (docs/data-format.md "Shaders") export class ShaderLib { constructor(gl, base = "shaders", assets = assetsOf(gl)) { this.gl = gl; - this.base = base; this.assets = assets; GL.init(gl); - this.index = new Map(assets.json(`${base}/shaders.json`).map((r) => [r.name, r])); + // name -> {variants: [{subShader, pass, keywords, key}], info(): the parsed form, text(variant): its GLSL} + this.index = base && base[RECORDS] ? base[RECORDS] : new Map(assets.json(`${base}/shaders.json`).map((r) => [r.name, { + variants: r.variants.map((v) => ({ ...v, key: `${base}/${v.file}` })), + info: () => assets.json(`${base}/${r.parsed}`), + text: (v) => assets.text(v.key), + }])); this.parsed = new Map(); - this.cache = new Map(); // variant file -> UnityProgram + this.cache = new Map(); // variant key -> UnityProgram this.variants = new Map(); // "name|subShader|pass|keywords" -> UnityProgram (the variant search, once) this.states = new Map(); // "name|subShader|pass" -> {state, names, byValues}: passState per value set } + // The shaders of a shader table {Unity name: Shader object id}; a name must be the one its object's index states. + static fromObjects(gl, shaders, assets = assetsOf(gl)) { + const records = new Map(Object.entries(shaders).map(([name, id]) => { + assets.object(id, "Shader"); + const index = assets.objectJson(id, "index"); + if (index.name !== name) throw new Error(`shader ${id} is ${index.name}, the table names it ${name}`); + return [name, { + variants: index.variants.filter((v) => v.platform === SHADER_PLATFORM && v.type === SHADER_PROGRAM_TYPE) + .map((v) => ({ ...v, key: `${id}#${v.role}` })), + info: () => assets.objectJson(id), + text: (v) => assets.objectText(id, v.role), + }]; + })); + return new ShaderLib(gl, { [RECORDS]: records }, assets); + } + info(name) { if (!this.parsed.has(name)) { const rec = this.index.get(name); if (!rec) throw new Error(`shader not packed: ${name}`); - this.parsed.set(name, this.assets.json(`${this.base}/${rec.parsed}`)); + this.parsed.set(name, rec.info()); } return this.parsed.get(name); } @@ -353,11 +381,11 @@ export class ShaderLib { const want = [...new Set(keywords.filter((k) => known.has(k)))].sort(); const v = vs.find((x) => x.keywords.length === want.length && [...x.keywords].sort().every((k, i) => k === want[i])); if (!v) throw new Error(`${name} pass ${pass}: no variant for [${want.join(" ")}]`); - if (!this.cache.has(v.file)) { + if (!this.cache.has(v.key)) { const label = `${name}#${subShader}.${pass}[${want.join(" ")}]`; - this.cache.set(v.file, new UnityProgram(this.gl, label, this.assets.text(`${this.base}/${v.file}`))); + this.cache.set(v.key, new UnityProgram(this.gl, label, rec.text(v))); } - return this.cache.get(v.file); + return this.cache.get(v.key); } // passState of the pass against the material floats. The result depends on the floats the state names only: it is diff --git a/src/engine/prefab.js b/src/engine/prefab.js index 93f9ce8..9ad1755 100644 --- a/src/engine/prefab.js +++ b/src/engine/prefab.js @@ -3,6 +3,11 @@ import { Transform } from "./math.js"; // Prefab: Transform hierarchy and component lookup over an exported prefab node list // (hierarchy order, local TRS, components in GameObject order). +// A component's class: a MonoBehaviour's script class (the last part of a prefab.json component's `script`, +// "assembly|namespace|class"; the `class` of a component in a chart or story file), else the component's type. +export const componentClass = (c) => + (typeof c.script === "string" ? c.script.slice(c.script.lastIndexOf("|") + 1) : (c.class || c.type)); + export class Prefab { constructor(exported) { this.key = exported.key; @@ -37,7 +42,7 @@ export class Prefab { components(path, cls) { const e = this.nodes.get(path); if (!e) throw new Error(`${this.key}: no node ${path}`); - return e.node.components.filter((c) => (cls ? (c.class || c.type) === cls : true)); + return e.node.components.filter((c) => (cls ? componentClass(c) === cls : true)); } component(path, cls) { @@ -50,7 +55,7 @@ export class Prefab { find(cls) { const hits = []; for (const [path, e] of this.nodes) - for (const c of e.node.components) if ((c.class || c.type) === cls) hits.push({ path, c }); + for (const c of e.node.components) if (componentClass(c) === cls) hits.push({ path, c }); if (hits.length !== 1) throw new Error(`${this.key}: ${hits.length} ${cls}`); return hits[0]; } diff --git a/src/engine/texture.js b/src/engine/texture.js index 3975de3..eb69834 100644 --- a/src/engine/texture.js +++ b/src/engine/texture.js @@ -32,8 +32,16 @@ export class GLTex { // ENGINE: a mipmapped texture samples the mip chain stored in the asset; the data holds its base level only, so the // other levels are generated from it (glGenerateMipmap) and filtered as the texture's FilterMode asks. static async load(gl, base, desc, assets = assetsOf(gl)) { + return GLTex._upload(gl, await assets.image(join(base, desc.texture)), desc); + } + + // A Texture2D object of the store (AssetStore.texture: size, mip count, settings) and its image. + static async loadObject(gl, id, assets = assetsOf(gl)) { + return GLTex._upload(gl, await assets.objectImage(id), assets.texture(id)); + } + + static _upload(gl, bmp, desc) { const mips = desc.mipCount > 1 ? desc.mipCount : 1; - const bmp = await assets.image(join(base, desc.texture)); if (bmp.width !== desc.width || bmp.height !== desc.height) throw new Error(`${desc.name}: size mismatch`); const t = gl.createTexture(); gl.bindTexture(gl.TEXTURE_2D, t); diff --git a/src/live2d/character.js b/src/live2d/character.js index 47872df..75b412b 100644 --- a/src/live2d/character.js +++ b/src/live2d/character.js @@ -1,6 +1,6 @@ import { F } from "../engine/core.js"; import { quat } from "../engine/math.js"; -import { Prefab } from "../engine/prefab.js"; +import { Prefab, componentClass } from "../engine/prefab.js"; import { UnityRandom } from "../engine/random.js"; import { EASE } from "../engine/tween.js"; import { CUBISM, CubismModel } from "./cubism.js"; @@ -34,25 +34,29 @@ import { Live2DPhysics } from "./physics.js"; // The managed math is float32 without FMA; F (Math.fround) is applied in source order. export class Live2DCharacter { - // prefab: the exported model prefab; mocBytes: the moc3 (ArrayBuffer); loop: the PlayerLoop that drives the model. - // opts.random: the UnityRandom stream (UnityEngine.Random) of the auto eye blink and the pseudo lip sync (default: - // a stream seeded from the clock). opts.motionSync: the MotionSync Core (motionsync.js motionSyncCore()) for voice - // lip sync, or null. - constructor(prefab, mocBytes, loop, { random = null, motionSync = null } = {}) { + // assets: the AssetStore holding the model's objects; model: its model.json (model.js modelIndex); loop: the + // PlayerLoop that drives the model. opts.random: the UnityRandom stream (UnityEngine.Random) of the auto eye blink and + // the pseudo lip sync (default: a stream seeded from the clock). opts.motionSync: the MotionSync Core (motionsync.js + // motionSyncCore()) for voice lip sync, or null. + constructor(assets, model, loop, { random = null, motionSync = null } = {}) { this.loop = loop; this.random = random || new UnityRandom(Date.now() >>> 0); this.motionSyncCore = motionSync; + const prefab = assets.prefab(model.prefab); this.name = prefab.nodes[0].name; - this.core = new CubismModel(mocBytes); + this.core = new CubismModel(assets.objectBytes(model.moc, "blob:_bytes").buffer); try { - this._build(prefab); + this._build(assets, model, prefab); } catch (e) { this.core.release(); throw e; } } - _build(prefab) { + // assets.mono() of the MonoBehaviours a list of references names + static _monos(assets, refs) { return refs.map((r) => assets.mono(assets.ref(r, "MonoBehaviour"))); } + + _build(assets, model, prefab) { this.core.update(); this.params = new Live2DParameters(this.core); this.parts = Float32Array.from(this.core.parts.opacities); @@ -60,7 +64,8 @@ export class Live2DCharacter { this.prefab = new Prefab(prefab); this.root = this.prefab.root; const rootPath = this.root.name; - const comp = (cls) => this.prefab.component(rootPath, cls); + // the fields of the root's component of a class + const comp = (cls) => this.prefab.component(rootPath, cls).fields; const ch = comp("Live2DCharacter"); this.defaultMotionName = ch.DefaultMotionName; this.defaultExpressionName = ch.DefaultExpressionName; @@ -68,10 +73,13 @@ export class Live2DCharacter { this.baseScale = ch.BaseScale; // motions: clips by name; fade data by instance id - this.clips = new Map(ch._motionList.map((c) => [c.clip, new Live2DClip(c, this.params)])); - const fl = comp("CubismFadeController").CubismFadeMotionList; - this.fadeById = new Map(fl.MotionInstanceIds.map((id, k) => - [id, new Live2DFadeMotion(fl.CubismFadeMotionObjects[k], this.params)])); + this.clips = new Map(ch._motionList.map((r) => { + const id = assets.ref(r, "AnimationClip"), c = new Live2DClip(assets.clip(id), model.bindings[id], this.params); + return [c.name, c]; + })); + const fl = assets.mono(assets.ref(comp("CubismFadeController").CubismFadeMotionList, "MonoBehaviour")); + const fades = Live2DCharacter._monos(assets, fl.CubismFadeMotionObjects); + this.fadeById = new Map(fl.MotionInstanceIds.map((id, k) => [id, new Live2DFadeMotion(fades[k], this.params)])); for (const c of this.clips.values()) if (!this.fadeById.has(c.instanceId)) throw new Error(`${c.name}: no fade motion for id ${c.instanceId}`); @@ -79,9 +87,10 @@ export class Live2DCharacter { const el = comp("CubismExpressionController"); if (el.UseLegacyBlendCalculation) throw new Error("legacy expression blend not implemented"); // a model without expressions has no ExpressionsList (null); its CurrentExpressionIndex stays -1 - const eo = (el.ExpressionsList && el.ExpressionsList.CubismExpressionObjects) || []; + const list = assets.ref(el.ExpressionsList, "MonoBehaviour"); + const eo = list === null ? [] : Live2DCharacter._monos(assets, assets.mono(list).CubismExpressionObjects); this.expressions = eo.map((e) => ({ - name: e.name.replace(/\.exp3$/, ""), fadeIn: e.FadeInTime, fadeOut: e.FadeOutTime, + name: e.m_Name.replace(/\.exp3$/, ""), fadeIn: e.FadeInTime, fadeOut: e.FadeOutTime, // CubismPlayingExpression.Create: FindById is null for an id the model lacks; UpdateExpression and // ResetExpressionParametersToDefault skip such a destination dest: e.Parameters.filter((p) => this.params.index.has(p.Id)) @@ -95,8 +104,8 @@ export class Live2DCharacter { // tagged parameters const tagged = (cls) => [...this.prefab.nodes.values()] - .filter((e) => e.node.components.some((c) => c.class === cls)) - .map((e) => ({ id: e.node.name, c: e.node.components.find((c) => c.class === cls) })); + .filter((e) => e.node.components.some((c) => componentClass(c) === cls)) + .map((e) => ({ id: e.node.name, c: e.node.components.find((c) => componentClass(c) === cls).fields })); this.eyeBlinkParams = tagged("CubismEyeBlinkParameter").map((x) => this.params.idx(x.id)); this.harmonic = tagged("CubismHarmonicMotionParameter").map((x) => { if (x.c.Direction !== 2) throw new Error("harmonic direction other than Centric not implemented"); @@ -124,13 +133,13 @@ export class Live2DCharacter { // live stage's background characters only.) this.mouthDestinations = tagged("CubismMouthParameter").filter((x) => x.id === "ParamMouthOpenY") .map((x) => this.params.idx(x.id)); - if ([...this.prefab.nodes.values()].some((e) => e.node.components.some((c) => c.class === "CubismPosePart"))) + if ([...this.prefab.nodes.values()].some((e) => e.node.components.some((c) => componentClass(c) === "CubismPosePart"))) throw new Error("pose parts not implemented"); // a model without CubismPhysicsController has no physics (the update controller's chain has no order 800) const pc = this.prefab.components(rootPath, "CubismPhysicsController"); if (pc.length > 1) throw new Error(`${pc.length} CubismPhysicsController components`); - this.physics = pc.length ? new Live2DPhysics(pc[0]._rig, this.params) : null; + this.physics = pc.length ? new Live2DPhysics(pc[0].fields._rig, this.params) : null; // state this.showing = false; this.pausing = false; this.ignoreAllUpdate = false; @@ -152,15 +161,15 @@ export class Live2DCharacter { frequency: 0.5 }; this.model = { ignore: false, wasJustEnabled: false, lastTick: -1, didExecute: false, dyn: null }; - this._initRender(comp("CubismRenderController")); + this._initRender(assets, comp("CubismRenderController")); this._initMasks(); - this._initGameLayer(rootPath, tagged, fl, ch); + this._initGameLayer(rootPath, tagged, fades, ch); } // The rest of Live2DCharacter.Init: the override parameters (a parameter the model lacks is skipped wherever it // would be written), the lip sync controller over the model's MotionSync controller, the eye-blink stop targets, // the parameter loop controller, the rim light and shadow values and the state of the game layer's overrides. - _initGameLayer(rootPath, tagged, fadeList, ch) { + _initGameLayer(rootPath, tagged, fades, ch) { const opt = (id) => (this.params.index.has(id) ? this.params.idx(id) : -1); this.mouthForm = opt("ParamMouthForm"); this.angleXParam = opt("ParamAngleX"); @@ -171,7 +180,8 @@ export class Live2DCharacter { const head = `${rootPath}/Anchors/Head`; this.headAnchor = this.prefab.nodes.has(head) ? this.prefab.transform(head) : null; this.mouthControllerEnabled = true; // CubismMouthController.enabled - this.motionSync = CubismMotionSyncController.fromPrefab((cls) => this.prefab.components(rootPath, cls), this.params, + this.motionSync = CubismMotionSyncController.fromPrefab((cls) => this.prefab.components(rootPath, cls).map((c) => c.fields), + this.params, this.loop, this.motionSyncCore); // ResolveAdvLipSyncPresentationProfile counts the CubismMouthParameter components before // RemoveMouthParametersExceptMouthOpenY keeps the one on ParamMouthOpenY @@ -181,7 +191,7 @@ export class Live2DCharacter { const eb = [...this.eyeBlinkParams, opt("ParamEyeLOpen"), opt("ParamEyeROpen")].filter((i) => i >= 0); this.eyeBlinkStop = { stopped: false, wasAuto: false, start: 0, elapsed: 0, duration: 0, targets: [...new Set(eb)] }; this.defaultFadeMotionName = `${this.defaultMotionName}.fade`; - this.paramLoop = new Live2DParameterLoopController(fadeList.CubismFadeMotionObjects, this.params, this.store); + this.paramLoop = new Live2DParameterLoopController(fades, this.params, this.store); this.breathEnabled = true; // _playingHarmonicMotion // Live2DCharacter _isOverrideRotate, _isAdditiveOverrideRotate, _angleX, _bodyAngleX and the additive amounts // applied last (_appliedAdditiveAngleX, _appliedAdditiveBodyAngleX); the controller's angle tweens @@ -260,19 +270,22 @@ export class Live2DCharacter { // CubismRenderer.TryInitialize per drawable: two meshes (the displayed one and the one written next), vertex // colours, the material of its MeshRenderer, sorting mode 2 (BackToFrontOrder, set by Live2DCharacter.Init: every // drawable at z = 0, sortingOrder = controller order + local order + Core render order). - _initRender(rc) { + _initRender(assets, rc) { const d = this.core.drawables, n = d.count; const byIndex = new Array(n); + const of = (e, cls) => { const c = e.node.components.find((x) => componentClass(x) === cls); return c ? c.fields : null; }; for (const e of this.prefab.nodes.values()) { - const dc = e.node.components.find((c) => c.class === "CubismDrawable"); + const dc = of(e, "CubismDrawable"); if (!dc) continue; if (d.ids[dc._unmanagedIndex] !== e.node.name) throw new Error(`drawable ${e.node.name} index mismatch`); e.transform.localPosition = { x: 0, y: 0, z: 0 }; - const r = e.node.components.find((c) => c.class === "CubismRenderer"); - const mr = e.node.components.find((c) => c.type === "MeshRenderer"); + const r = of(e, "CubismRenderer"), mr = of(e, "MeshRenderer"); if (!r || !mr) throw new Error(`${e.node.name}: no CubismRenderer / MeshRenderer`); if (mr.m_Materials.length !== 1) throw new Error(`${e.node.name}: ${mr.m_Materials.length} materials`); - byIndex[dc._unmanagedIndex] = { node: e.node, transform: e.transform, cr: r, material: mr.m_Materials[0] }; + const texture = assets.ref(r._mainTexture, "Texture2D"); + if (texture === null) throw new Error(`${e.node.name}: no main texture`); + byIndex[dc._unmanagedIndex] = { node: e.node, transform: e.transform, cr: r, texture, + material: assets.material(assets.ref(mr.m_Materials[0], "Material")) }; } // renderOpacity: CubismRenderController.Opacity (clips may animate it); rc.opacity: the value last passed to the // renderers (_lastOpacity, cubism_ModelOpacity) @@ -285,7 +298,7 @@ export class Live2DCharacter { const mc = d.multiplyColors, sc = d.screenColors; const r = { index: i, id: d.ids[i], name: x.node.name, transform: x.transform, material: x.material, - textureDesc: x.cr._mainTexture, localSortingOrder: x.cr._localSortingOrder, + texture: x.texture, localSortingOrder: x.cr._localSortingOrder, color: { ...x.cr._color }, opacity: d.opacities[i], renderOrder: d.renderOrders[i], meshes: [{ pos, color: null, bounds: null }, { pos: Float32Array.from(pos), color: null, bounds: null }], front: 0, thisSwap: {}, lastSwap: {}, enabled: visible, isMaskSource: false, diff --git a/src/live2d/drawing.js b/src/live2d/drawing.js index aae88b5..c58a1a2 100644 --- a/src/live2d/drawing.js +++ b/src/live2d/drawing.js @@ -15,19 +15,18 @@ export const LIT_SHADER = "Live2D Cubism/Lit-URP-ADV-optimize"; export const MASK_SHADER = "Live2D Cubism/Mask"; export const MASK_SIZE = 1024; -// A drawable's main texture (a mipmapped one as GLTex.load describes). -const loadTexture = (gl, dir, d, assets) => GLTex.load(gl, dir, d, assets); - export class Live2DDrawing { - // lib: the ShaderLib of the model's shaders; resources: {cubismMask, cubismMaskCulling} materials; - // dir: directory of the prefab (texture paths are relative to it); white: Texture2D.whiteTexture - constructor(gl, lib, character, { dir = "", resources, white, assets }) { - this.gl = gl; this.lib = lib; this.ch = character; this.dir = dir; this.white = white; this.assets = assets; - if (!resources || !resources.cubismMask || !resources.cubismMaskCulling) - throw new Error("model.json: resources.cubismMask / cubismMaskCulling missing"); - this.maskMats = { 0: resources.cubismMask, 1: resources.cubismMaskCulling }; + // lib: the ShaderLib of the model's shaders (ShaderLib.fromObjects of model.json shaders); model: its model.json + // (the Cubism mask materials of `resources` and the shader table); assets: the AssetStore of its objects (the + // drawables' main textures); white: Texture2D.whiteTexture + constructor(gl, lib, character, { model, white, assets }) { + this.gl = gl; this.lib = lib; this.ch = character; this.white = white; this.assets = assets; + this.maskMats = { 0: assets.material(model.resources.cubismMask), 1: assets.material(model.resources.cubismMaskCulling) }; for (const m of Object.values(this.maskMats)) - if (!m.shader || m.shader.shader !== MASK_SHADER) throw new Error(`mask material shader ${m.shader && m.shader.shader}`); + if (assets.ref(m.shader, "Shader") !== model.shaders[MASK_SHADER]) throw new Error(`mask material ${m.material}: not drawn with ${MASK_SHADER}`); + for (const r of character.renderers) + if (assets.ref(r.material.shader, "Shader") !== model.shaders[LIT_SHADER]) + throw new Error(`${r.name}: material ${r.material.material} is not drawn with ${LIT_SHADER}`); // CubismMaskTexture: new RenderTexture(1024, 1024, 0, ARGB32), RGBA8 in the gamma-space project // ENGINE: the mask RenderTexture's filter and wrap modes are never set (engine defaults); bilinear and clamped here. this.maskRT = new GLTarget(gl, MASK_SIZE, MASK_SIZE, { label: `${character.name} mask` }); @@ -63,12 +62,10 @@ export class Live2DDrawing { gl.bindVertexArray(null); } + // the drawables' main textures (Texture2D objects; a mipmapped one as GLTex.load describes) async loadTextures() { - for (const r of this.ch.renderers) { - const d = r.textureDesc; - if (!d || !d.texture) throw new Error(`${r.name}: no main texture`); - if (!this.textures.has(d.texture)) this.textures.set(d.texture, await loadTexture(this.gl, this.dir, d, this.assets)); - } + for (const r of this.ch.renderers) + if (!this.textures.has(r.texture)) this.textures.set(r.texture, await GLTex.loadObject(this.gl, r.texture, this.assets)); } // compiles every program the model draws with (the Lit variants of its materials, the mask program when it has @@ -171,7 +168,7 @@ export class Live2DDrawing { for (const mi of j.masks) { const r = ch.renderers[mi]; const mat = this.maskMats[r.doubleSided ? 0 : 1]; - const tex = this.textures.get(r.textureDesc.texture); + const tex = this.textures.get(r.texture); // ENGINE: _ProjectionParams while a command buffer draws into a render texture is set natively; x = 1 here. this._drawSlot(prog, [{ _ProjectionParams: [1, 0.3, 1000, 1 / 1000], cubism_MaskTile: j.tile, cubism_MaskTransform: j.transform, _MainTex: tex }, mat.floats, mat.colors], @@ -186,7 +183,7 @@ export class Live2DDrawing { _mpb(r) { const ch = this.ch, mt = ch.multiplyTexture; const sheet = { - _MainTex: this.textures.get(r.textureDesc.texture), + _MainTex: this.textures.get(r.texture), cubism_ModelOpacity: ch.rc.opacity, cubism_MultiplyColor: r.multiplyColor, cubism_ScreenColor: r.screenColor, diff --git a/src/live2d/model.js b/src/live2d/model.js index 3360aa7..acc4e5a 100644 --- a/src/live2d/model.js +++ b/src/live2d/model.js @@ -1,29 +1,41 @@ -// model.json: the index of one Live2D model's files (docs/data-format.md "model.json"). The model viewer reads it at the -// root of its store; a story reads the model of each character from live2d// of the story's store. +// model.json: one Live2D model of a product (docs/data-format.md "model.json"). The model viewer reads it at the root +// of its store; a story reads the model of each character from live2d// of the story's store. Its objects +// are in the store's object table. -export const MODEL_JSON_FORMATS = Object.freeze([1, 2]); +export const MODEL_JSON_FORMAT = 3; -const dirname = (p) => { const i = p.lastIndexOf("/"); return i < 0 ? "" : p.slice(0, i); }; - -// The model whose model.json is at `dir`/model.json of `assets` (dir "" for the store's root): -// index model.json as it is -// moc3, prefab, shaders the store paths of the moc3, the prefab and the shader index -// textureDir the prefab's directory (the drawables' texture paths are relative to it) -// shaderDir the shader index's directory (the ShaderLib base of the model's shaders) -// resources the Cubism mask materials (resources.cubismMask / cubismMaskCulling) -// motionSync format 2: the prefab's root has a MotionSync controller with a CRI audio input; format 1: null (not -// stated). The characters take their MotionSync controller from the prefab either way. +// The model.json at `dir`/model.json of `assets` (dir "" for the store's root), its objects checked: +// prefab the model's root GameObject (its prefab.json) +// moc the CubismMoc MonoBehaviour (its json names the CubismMoc script); the moc3 is its blob:_bytes file +// shaders {Unity shader name: Shader object}: the shaders its materials and the Cubism mask materials name +// resources {cubismMask, cubismMaskCulling}: the Cubism mask Materials (Resources/Live2D/Cubism/Materials) +// bindings {AnimationClip object: [{path, attribute}]}: per binding of the clip, the path below the model's root +// ("" for the root) and the attribute it animates, null where the hierarchy names none +// motionSync the root has a MotionSync controller with a CRI audio input (the characters take the controller +// from the prefab either way) +// name, key, canvas, textures facts about the model export const modelIndex = (assets, dir = "") => { - const at = (p) => (dir ? `${dir}/${p}` : p); - const where = at("model.json"); - const index = assets.json(where); - if (!index || !MODEL_JSON_FORMATS.includes(index.format)) - throw new Error(`${where}: format ${index && index.format} is not supported (${MODEL_JSON_FORMATS.join(" or ")} expected)`); - for (const k of ["moc3", "prefab", "shaders"]) - if (typeof index[k] !== "string" || !index[k]) throw new Error(`${where}: "${k}" missing`); - if (!/(^|\/)shaders\.json$/.test(index.shaders)) throw new Error(`${where}: "shaders" must name a shaders.json`); - if (index.format >= 2 && typeof index.motionSync !== "boolean") throw new Error(`${where}: "motionSync" missing`); - const moc3 = at(index.moc3), prefab = at(index.prefab), shaders = at(index.shaders); - return { index, moc3, prefab, shaders, textureDir: dirname(prefab), shaderDir: dirname(shaders), - resources: index.resources, motionSync: index.format >= 2 ? index.motionSync : null }; + const where = dir ? `${dir}/model.json` : "model.json"; + const m = assets.json(where); + if (!m || m.format !== MODEL_JSON_FORMAT) + throw new Error(`${where}: format ${m && m.format} is not supported (${MODEL_JSON_FORMAT} expected)`); + const table = (k) => { + if (!m[k] || typeof m[k] !== "object" || Array.isArray(m[k])) throw new Error(`${where}: "${k}" missing`); + return m[k]; + }; + try { + assets.object(m.prefab, "GameObject"); + assets.object(m.moc, "MonoBehaviour"); + const script = assets.objectJson(m.moc).$script; + if (typeof script !== "string" || script.slice(script.lastIndexOf("|") + 1) !== "CubismMoc") + throw new Error(`${m.moc}: not a CubismMoc`); + for (const id of Object.values(table("shaders"))) assets.object(id, "Shader"); + const r = table("resources"); + for (const k of ["cubismMask", "cubismMaskCulling"]) assets.object(r[k], "Material"); + for (const id of Object.keys(table("bindings"))) assets.object(id, "AnimationClip"); + } catch (e) { + throw new Error(`${where}: ${e.message}`, { cause: e }); + } + if (typeof m.motionSync !== "boolean") throw new Error(`${where}: "motionSync" missing`); + return m; }; diff --git a/src/live2d/motion.js b/src/live2d/motion.js index 4314a0a..3ac5578 100644 --- a/src/live2d/motion.js +++ b/src/live2d/motion.js @@ -14,36 +14,47 @@ const sample = (s, t) => { return F(F(F(F(F(F(s.k[k] * x) + s.k[k + 1]) * x) + s.k[k + 2]) * x) + s.k[k + 3]); }; +// The class a clip binding (clip.json) animates: a MonoBehaviour's script class (the last part of `scriptClass`), +// else the class of its type id (null when unknown). +const bindingClass = (b) => (b.typeID === 114 ? (b.scriptClass ? b.scriptClass.slice(b.scriptClass.lastIndexOf("|") + 1) : null) + : b.class ?? null); + // One AnimationClip as the Animator samples it into raw CubismParameter.Value (no clamp) and the model curves' -// controller fields: the streamed cubic keys and the constant curves of the exported clip data. +// controller fields: the streamed cubic keys and the constant curves of the clip (clip.json). `names`: per binding of +// the clip, the path and the attribute it animates in the model's hierarchy (model.json bindings). // ENGINE: Mecanim samples streamed clips natively; this is the cubic form of the stored keys, float32 in source order. export class Live2DClip { - constructor(clip, params) { - this.name = clip.clip; + constructor(clip, names, params) { + this.name = clip.name; this.length = clip.stopTime; this.loopTime = clip.loopTime; if (clip.dense.curveCount) throw new Error(`${this.name}: dense clip curves not implemented`); if (clip.startTime !== 0 || clip.cycleOffset !== 0) throw new Error(`${this.name}: clip start / cycle offset not implemented`); + const nS = clip.streamed.curveCount; + // every binding animates one float curve: the streamed ones, then the constant ones + if (!names || names.length !== clip.bindings.length) throw new Error(`${this.name}: no names for its bindings`); + if (clip.bindings.length !== nS + clip.constant.length || clip.bindings.some((b) => b.pptr || (b.curves ?? 1) !== 1)) + throw new Error(`${this.name}: bindings other than one float curve each not implemented`); const ev = clip.events.filter((e) => e.functionName === "InstanceId"); this.instanceId = ev.length ? ev[ev.length - 1].intParameter : -1; - // A binding whose path is not in the model's hierarchy (path null in the export) animates nothing. + // A binding whose path is not in the model's hierarchy (path null) animates nothing. // ENGINE: the Animator leaves a binding without a transform at its path unbound. - const target = (b) => { - if (b.path === null) return -1; - if (b.path === "" && FIELDS[`${b.class}.${b.attribute}`]) return FIELDS[`${b.class}.${b.attribute}`]; - if (b.class !== "CubismParameter" || b.attribute !== "Value" || !b.path.startsWith("Parameters/")) - throw new Error(`${this.name}: binding ${JSON.stringify(b)} not implemented`); - return params.idx(b.path.slice("Parameters/".length)); + const target = (i) => { + const { path, attribute } = names[i], cls = bindingClass(clip.bindings[i]); + if (path === null) return -1; + if (path === "" && FIELDS[`${cls}.${attribute}`]) return FIELDS[`${cls}.${attribute}`]; + if (cls !== "CubismParameter" || attribute !== "Value" || !path.startsWith("Parameters/")) + throw new Error(`${this.name}: binding of ${cls} ${JSON.stringify(path)} ${attribute} not implemented`); + return params.idx(path.slice("Parameters/".length)); }; - const nS = clip.streamed.curveCount; this.streamed = []; - for (let c = 0; c < nS; c++) this.streamed.push({ param: target(clip.bindings[c]), t: [], k: [] }); + for (let c = 0; c < nS; c++) this.streamed.push({ param: target(c), t: [], k: [] }); for (const [time, keys] of clip.streamed.frames) for (const [c, c0, c1, c2, c3] of keys) { const s = this.streamed[c]; s.t.push(time); s.k.push(F(c0), F(c1), F(c2), F(c3)); } - const constants = clip.constant.map((v, j) => ({ param: target(clip.bindings[nS + j]), v: F(v) })); + const constants = clip.constant.map((v, j) => ({ param: target(nS + j), v: F(v) })); const field = (c) => typeof c.param === "string"; this.fieldCurves = this.streamed.filter(field); this.fieldConstants = constants.filter(field); diff --git a/src/live2d/session.js b/src/live2d/session.js index b38f8f5..12a4126 100644 --- a/src/live2d/session.js +++ b/src/live2d/session.js @@ -60,7 +60,7 @@ export class ModelSession { // opts: // gl WebGL2RenderingContext (required); used by this session alone while it lives - // assets AssetStore of the model (required): model.json and the files it names + // assets AssetStore of the model (required): model.json and the objects it names // motion, expression shown first (default: the model's default motion / expression); loop: replay that motion // physics, breath CubismPhysicsController and the harmonic breath on (default true) // seed seed of the auto eye blink's random intervals (default: from the clock) @@ -87,21 +87,18 @@ export class ModelSession { this.gl = gl; this.assets = assets; this._gl = trackGL(gl); const model = modelIndex(assets); - const prefab = assets.json(model.prefab); this.info = assets.info; this.seed = seed ?? (Date.now() >>> 0); const loop = this.loop = new PlayerLoop(MODEL_FRAME_RATE); - const ch = this.character = new Live2DCharacter(prefab, assets.arrayBuffer(model.moc3), loop, - { random: new UnityRandom(this.seed) }); + const ch = this.character = new Live2DCharacter(assets, model, loop, { random: new UnityRandom(this.seed) }); this.motions = [...ch.clips.keys()]; this.expressions = ch.expressions.map((e) => e.name); for (const [name, list] of [[motion, this.motions], [expression, this.expressions]]) if (name && !list.includes(name)) throw new Error(`${ch.name}: no ${list === this.motions ? "motion" : "expression"} "${name}"`); - const lib = new ShaderLib(gl, model.shaderDir, assets); + const lib = ShaderLib.fromObjects(gl, model.shaders, assets); gl.frontFace(gl.CW); // Unity's front-face convention with its world-to-camera matrix this.white = GLTex.solid(gl, [255, 255, 255, 255], "white"); - const drawing = this.drawing = new Live2DDrawing(gl, lib, ch, { dir: model.textureDir, resources: model.resources, - white: this.white, assets }); + const drawing = this.drawing = new Live2DDrawing(gl, lib, ch, { model, white: this.white, assets }); await drawing.loadTextures(); drawing.prepare(); const c = gl.canvas; diff --git a/src/story/assets.js b/src/story/assets.js index a5ba06e..4059a0f 100644 --- a/src/story/assets.js +++ b/src/story/assets.js @@ -1,17 +1,26 @@ -import { AssetStore, decodeAsset } from "../data/assets.js"; +import { AssetStore, PRODUCT_MANIFEST, decodeAsset } from "../data/assets.js"; import { ADV_PLAYBACK_MODE, StoryCommandError } from "./interfaces.js"; import { StoryPlayerCore } from "./player-core.js"; import { simpleUnsupportedCommands } from "./simple/validator.js"; -// Loading a story from a site: the story manifest (stories/.json, stories//.json) and the files -// of one language; with manifest format /2 also the manifests of the Live2D models it uses (models/.json). +// Loading a story from a site: the story manifest (stories/.json, stories//.json), the files of +// one language and the manifests of the Live2D models it uses (models/.json, products of live2d.model). -// the current story manifest format, and the formats this player reads: /2 (the models in model manifests of the -// site) and /1 (the models' files among the story's own) +// the story manifest format, the formats this player reads export const STORY_MANIFEST_FORMAT = "ournotes.story-manifest/2"; -export const STORY_MANIFEST_FORMATS = Object.freeze(["ournotes.story-manifest/1", STORY_MANIFEST_FORMAT]); -// the model manifest formats this player reads -export const MODEL_MANIFEST_FORMATS = Object.freeze([2, 3]); +export const STORY_MANIFEST_FORMATS = Object.freeze([STORY_MANIFEST_FORMAT]); +// the product of a model manifest (a product manifest, nnnotes.product/1) +export const MODEL_PRODUCT = "live2d.model"; + +// Checks that `mm` (fetched from `where`) is the model manifest of model `id`: a product manifest of live2d.model whose +// subject is a model key Character/Live2D///model/. +export const checkModelManifest = (mm, id, where) => { + if (!mm || mm.schema !== PRODUCT_MANIFEST || mm.product !== MODEL_PRODUCT || !mm.files || typeof mm.files !== "object" + || !mm.objects || typeof mm.objects !== "object") + throw new Error(`${where}: not a model manifest (${PRODUCT_MANIFEST} of ${MODEL_PRODUCT})`); + const name = String(mm.subject).split("/").pop(); + if (name !== id) throw new Error(`${where}: model ${name}, the story names it ${id}`); +}; const here = () => (globalThis.document ? globalThis.document.baseURI : globalThis.location ? globalThis.location.href : undefined); @@ -40,42 +49,44 @@ export const fetchStoryManifest = async (url, { fetch = globalThis.fetch, signal return { url: href, manifest: man }; }; -// The AssetStore of one language of a story: the manifest's common files and the language group's files; with format -// /2 also, under live2d//, the files of every model the manifest lists (their manifests fetched in parallel). -// `lang` defaults to the manifest's language. The store's `info` is the manifest without its file lists, plus -// `loadedLanguage`. -// `base`: the site root the asset and model manifest paths are relative to (default: format /2, the manifest's -// `root`, relative to the manifest; format /1, the directory above the manifest's directory). `decode`, -// `concurrency`: as AssetStore.fromManifest. +// The AssetStore of one language of a story: the manifest's common files and the language group's files; under +// live2d// the files of every model the manifest lists (their manifests fetched in parallel), and the objects of +// those models (an object several models use is one object of the store). `lang` defaults to the manifest's +// language. The store's `info` is the manifest without its file lists, plus `loadedLanguage`. +// `base`: the site root the asset and model manifest paths are relative to (default: the manifest's `root`, relative +// to the manifest). `decode`, `concurrency`: as AssetStore.fromManifest. export const loadStoryStore = async (url, { lang = null, fetch = globalThis.fetch, signal = null, onProgress = null, manifest = null, base = null, decode = decodeAsset, concurrency } = {}) => { const m = manifest || await fetchStoryManifest(url, { fetch, signal }); const man = m.manifest, language = lang || man.language; checkFormat(man, m.url); - const sited = man.format === STORY_MANIFEST_FORMAT; // the models in model manifests of the site - if (sited && (typeof man.root !== "string" || !man.models || typeof man.models !== "object")) + if (typeof man.root !== "string" || !man.models || typeof man.models !== "object") throw new Error(`${m.url}: a story manifest without root / models`); if (!man.languages || !man.languages[language]) throw new Error(`story ${man.advId}: no language ${language} (${Object.keys(man.languages || {}).join(", ")})`); - const root = base ? new URL(String(base), here()) : new URL(sited ? man.root : "../", m.url); + const root = base ? new URL(String(base), here()) : new URL(man.root, m.url); const files = { ...man.files, ...man.languages[language].files }; - if (sited) { - const models = await Promise.all(Object.entries(man.models).map(async ([id, p]) => { - const u = new URL(p, root).href; - if (signal) signal.throwIfAborted(); - const r = await fetch(u, signal ? { signal } : undefined); - if (!r.ok) throw new Error(`${u}: HTTP ${r.status}`); - const mm = await r.json(); - if (!mm || !MODEL_MANIFEST_FORMATS.includes(mm.format) || !mm.files || typeof mm.files !== "object") - throw new Error(`${u}: not a model manifest (format ${MODEL_MANIFEST_FORMATS.join(" or ")})`); - if (mm.id !== undefined && mm.id !== id) throw new Error(`${u}: model ${mm.id}, the story names it ${id}`); - return [id, mm.files]; - })); - for (const p of Object.keys(files)) - if (p.startsWith("live2d/")) throw new Error(`story ${man.advId}: ${p}: live2d/ holds the models' files`); - for (const [id, mf] of models) for (const [p, f] of Object.entries(mf)) files[`live2d/${id}/${p}`] = f; + const models = await Promise.all(Object.entries(man.models).map(async ([id, p]) => { + const u = new URL(p, root).href; + if (signal) signal.throwIfAborted(); + const r = await fetch(u, signal ? { signal } : undefined); + if (!r.ok) throw new Error(`${u}: HTTP ${r.status}`); + const mm = await r.json(); + checkModelManifest(mm, id, u); + return [id, mm]; + })); + for (const p of Object.keys(files)) + if (p.startsWith("live2d/")) throw new Error(`story ${man.advId}: ${p}: live2d/ holds the models' files`); + const objects = {}; + for (const [id, mm] of models) { + for (const [p, f] of Object.entries(mm.files)) files[`live2d/${id}/${p}`] = f; + for (const [oid, o] of Object.entries(mm.objects)) { + if (oid in objects && JSON.stringify(objects[oid]) !== JSON.stringify(o)) + throw new Error(`story ${man.advId}: object ${oid} differs between the manifests of its models`); + objects[oid] = o; + } } - const merged = { ...man, loadedLanguage: language, files }; + const merged = { ...man, loadedLanguage: language, files, objects }; delete merged.languages; merged.languages = Object.fromEntries(Object.keys(man.languages).map((k) => [k, {}])); // the codes only const inner = (u, init) => (String(u) === m.url diff --git a/src/story/models.js b/src/story/models.js index cc1e12b..d7a1045 100644 --- a/src/story/models.js +++ b/src/story/models.js @@ -1,29 +1,19 @@ import { ShaderLib } from "../engine/glsl.js"; import { modelIndex } from "../live2d/model.js"; -const join = (...parts) => parts.filter(Boolean).join("/").replace(/\/+/g, "/"); - // The models of a story's Character rows. story.json `models` maps a model address (Character/Live2D/// -// model/) to -// a model id (story manifest /2): the store holds the model under live2d// (model.json and the files it names); -// the character draws with the model's own shaders and Cubism mask materials (model.json resources); -// {dir, moc3, prefab} (story manifest /1): the model's files in `dir` among the story's files; the character draws -// with the story's shaders (`lib`) and scene.json resources (`resources`). -// Returns model(address) -> {id (null for /1), model: {prefab, moc3 (store paths), textureDir, resources}, lib (the -// ShaderLib to draw with: one per model id; null without gl)}; `what` names the story in errors. -export const storyModels = (gl, store, story, what, { lib = null, resources = null } = {}) => { +// model/) to a model id: the store holds the model's model.json under live2d// and its objects in the +// store's object table (loadStoryStore). Returns model(address) -> {id, model (its model.json, model.js modelIndex), +// lib (the ShaderLib of its shaders: one per model id; null without gl)}; `what` names the story in errors. +export const storyModels = (gl, store, story, what) => { const byId = new Map(); return (address) => { - const m = story.models[address]; - if (typeof m === "string" && m) { - if (!byId.has(m)) { - const model = modelIndex(store, `live2d/${m}`); - byId.set(m, { id: m, model, lib: gl ? new ShaderLib(gl, model.shaderDir, store) : null }); - } - return byId.get(m); + const id = story.models[address]; + if (typeof id !== "string" || !id) throw new Error(`${what}: model ${address} is not in the story`); + if (!byId.has(id)) { + const model = modelIndex(store, `live2d/${id}`); + byId.set(id, { id, model, lib: gl ? ShaderLib.fromObjects(gl, model.shaders, store) : null }); } - if (m && typeof m === "object" && typeof m.dir === "string" && typeof m.moc3 === "string" && typeof m.prefab === "string") - return { id: null, lib, model: { prefab: join(m.dir, m.prefab), moc3: join(m.dir, m.moc3), textureDir: m.dir, resources } }; - throw new Error(`${what}: model ${address} is not in the story`); + return byId.get(id); }; }; diff --git a/src/story/renderer.js b/src/story/renderer.js index 1fb984c..a34b126 100644 --- a/src/story/renderer.js +++ b/src/story/renderer.js @@ -87,11 +87,9 @@ export class StoryRenderer { } // a character's drawing (its buffers now, its textures and programs in load()): `lib` the ShaderLib of its model's - // shaders, `dir` the directory its texture paths are relative to, `resources` its model's Cubism mask materials - // (model.json resources) - addCharacter(ch, { lib, dir, resources }) { - const g = new Live2DDrawing(this.gl, lib, ch, { dir, resources, white: this.tex ? this.tex.white : null, - assets: this.assets }); + // shaders, `model` its model.json (models.js) + addCharacter(ch, { lib, model }) { + const g = new Live2DDrawing(this.gl, lib, ch, { model, white: this.tex ? this.tex.white : null, assets: this.assets }); this.characters.push({ ch, gl: g }); return g; } diff --git a/src/story/session.js b/src/story/session.js index b61541c..b801b46 100644 --- a/src/story/session.js +++ b/src/story/session.js @@ -214,20 +214,20 @@ export class StorySession { // renderer and characters: the Character rows' preload, keyed TargetName-TargetAssetIndex (the first model of a // key is kept); IgnoreData rows are not preloaded. The story's shaders draw the stage and the post-processing, - // each model's own shaders its character (the story's in a story of manifest format /1, models.js). + // each model's own shaders its character (models.js). const storyLib = gl ? new ShaderLib(gl, "shaders", store) : null; const renderer = this.renderer = gl ? new StoryRenderer(gl, storyLib, sc, quality, loop, { assets: store }) : null; if (renderer) renderer.filmGrain = opts.filmGrain ?? 0; - const modelOf = storyModels(gl, store, story, what, { lib: storyLib, resources: scene.resources }); + const modelOf = storyModels(gl, store, story, what); for (const c of episode.commands) { if (c.cmd !== "Character" || c.IgnoreData) continue; if (characters.has(c.TargetName, c.TargetAssetIndex || 0)) continue; const { model, lib } = modelOf(`Character/Live2D/${c.TargetAssetName}`); - const ch = new Live2DCharacter(store.json(model.prefab), store.arrayBuffer(model.moc3), loop, { random, motionSync }); + const ch = new Live2DCharacter(store, model, loop, { random, motionSync }); ch.setParent(sc.field.poolRoot); sc.field.characterRoots.add(ch.root); characters.add(c.TargetName, c.TargetAssetIndex || 0, ch); - if (renderer) renderer.addCharacter(ch, { lib, dir: model.textureDir, resources: model.resources }); + if (renderer) renderer.addCharacter(ch, { lib, model }); } if (renderer) await renderer.load(""); await ui.load(); diff --git a/src/story/simple/render.js b/src/story/simple/render.js index f754045..db1bf01 100644 --- a/src/story/simple/render.js +++ b/src/story/simple/render.js @@ -65,10 +65,9 @@ export class SimpleCaptureRenderer { this.white = null; } - // a character's drawing with its model's shaders (`lib`), texture directory (`dir`) and Cubism mask materials - // (`resources`, model.json resources), as StoryRenderer.addCharacter - addCharacter(ch, { lib, dir, resources }) { - const g = new Live2DDrawing(this.gl, lib, ch, { dir, resources, white: this.white, assets: this.assets }); + // a character's drawing with its model's shaders (`lib`) and model.json (`model`), as StoryRenderer.addCharacter + addCharacter(ch, { lib, model }) { + const g = new Live2DDrawing(this.gl, lib, ch, { model, white: this.white, assets: this.assets }); this.drawings.set(ch, g); return g; } diff --git a/src/story/simple/session.js b/src/story/simple/session.js index 0cad176..f7210cd 100644 --- a/src/story/simple/session.js +++ b/src/story/simple/session.js @@ -158,15 +158,15 @@ export class SimpleStorySession { const storyLib = gl ? new ShaderLib(gl, "shaders", store) : null; // the story's shaders (URP post) const renderer = this.renderer = gl ? new SimpleCaptureRenderer(gl, loop, { assets: store }) : null; // each character's model and the shaders it draws with (models.js) - const modelOf = storyModels(gl, store, story, what, { lib: storyLib, resources: scene.resources }); + const modelOf = storyModels(gl, store, story, what); for (const c of episode.commands) { if (c.cmd !== "Character" || c.IgnoreData) continue; if (characters.has(c.TargetName, c.TargetAssetIndex || 0)) continue; const { model, lib } = modelOf(`Character/Live2D/${c.TargetAssetName}`); - const ch = new Live2DCharacter(store.json(model.prefab), store.arrayBuffer(model.moc3), loop, { random, motionSync }); + const ch = new Live2DCharacter(store, model, loop, { random, motionSync }); ch.setParent(pool); characters.add(c.TargetName, c.TargetAssetIndex || 0, ch); - if (renderer) renderer.addCharacter(ch, { lib, dir: model.textureDir, resources: model.resources }); + if (renderer) renderer.addCharacter(ch, { lib, model }); } // one CameraTargetRenderer per slot under the runtime parent (the host manager's transform) const desc = cameraTargetDesc(host); diff --git a/tests/data/story-data.test.mjs b/tests/data/story-data.test.mjs index 03b1e28..fedacb2 100644 --- a/tests/data/story-data.test.mjs +++ b/tests/data/story-data.test.mjs @@ -76,8 +76,8 @@ for (const e of chosen()) { const story = store.json("story.json"); for (const p of [story.episode, story.scene, story.ui]) assert.ok(store.has(p), `${lang}: ${p}`); for (const [key, id] of Object.entries(story.models)) { - const m = modelIndex(store, `live2d/${id}`); - for (const f of [m.moc3, m.prefab, m.shaders]) assert.ok(store.has(f), `${lang}: ${key}: ${f}`); + const m = modelIndex(store, `live2d/${id}`); // its objects checked by class + assert.ok(store.roles(m.moc).includes("blob:_bytes"), `${lang}: ${key}: the moc's bytes`); } for (const dir of Object.values(story.audio)) assert.ok(store.has(`${dir}/cues.json`), `${lang}: ${dir}/cues.json`); assert.equal(store.json("ui/fonts.json").language, lang); diff --git a/tests/types/live2d.ts b/tests/types/live2d.ts index 6b7d65b..85a06ab 100644 --- a/tests/types/live2d.ts +++ b/tests/types/live2d.ts @@ -24,9 +24,8 @@ export async function useModelPlayer(host: HTMLElement): Promise { player.pause(); player.play(); const info: ModelInfo | null = player.info; - const character: number | undefined = info?.model?.character; - const shown: string = info?.model?.names?.["zh-Hant"] ?? info?.model?.label ?? String(character ?? ""); - const names: string[] = [player.name, player.motion, player.expression, String(player.motionPlaying), String(info && info.id), shown]; + const subject: string | undefined = info?.subject; + const names: string[] = [player.name, player.motion, player.expression, String(player.motionPlaying), subject ?? ""]; const root: HTMLDivElement = player.root, canvas: HTMLCanvasElement = player.canvas; console.log(root.tagName, canvas.width, player.session && player.session.time, player.disposed); await player.dispose(); diff --git a/tests/unit/live2d-adv.test.mjs b/tests/unit/live2d-adv.test.mjs index 0be5174..45f9ca7 100644 --- a/tests/unit/live2d-adv.test.mjs +++ b/tests/unit/live2d-adv.test.mjs @@ -1,9 +1,10 @@ -// Live2DCharacter's story (ADV) calls over a synthetic model: a stand-in for Live2D Cubism Core and a prefab with the -// components the runtime reads. Pause / resume with requests made while paused, motion speed, seeking the playing -// motion, the parameter loop, the eye-blink stop, the angle and look overrides, brightness, sorting, layer, parent -// and the lip-sync entry points. Synthetic inputs only. +// Live2DCharacter's story (ADV) calls over a synthetic model: a stand-in for Live2D Cubism Core and the model's objects +// (live2d-objects.mjs), with the components the runtime reads. Pause / resume with requests made while paused, motion +// speed, seeking the playing motion, the parameter loop, the eye-blink stop, the angle and look overrides, brightness, +// sorting, layer, parent and the lip-sync entry points. Synthetic inputs only. import assert from "node:assert/strict"; import { test } from "node:test"; +import { ModelObjects, component, mb, node, ref } from "./live2d-objects.mjs"; import { Transform } from "../../src/engine/math.js"; import { PlayerLoop } from "../../src/engine/loop.js"; import { EASE } from "../../src/engine/tween.js"; @@ -50,41 +51,48 @@ const fadeData = (name, len, ids, curves, pfit = ids.map(() => -1), pfot = ids.m m_Name: `${name}.fade`, MotionName: name, FadeInTime: 0.5, FadeOutTime: 0.5, MotionLength: len, ParameterFadeInTimes: pfit, ParameterFadeOutTimes: pfot, ParameterIds: ids, ParameterCurves: curves, }); -const clip = (name, id, len, value) => ({ - clip: name, stopTime: len, loopTime: true, startTime: 0, cycleOffset: 0, - events: [{ functionName: "InstanceId", intParameter: id }], dense: { curveCount: 0 }, - bindings: [{ path: "Parameters/ParamAngleX", class: "CubismParameter", attribute: "Value" }], - streamed: { curveCount: 0, frames: [] }, constant: [value], +const clip = (o, name, id, len, value) => o.clip({ + name, stopTime: len, loopTime: true, events: [{ functionName: "InstanceId", intParameter: id }], + bindings: [{ cls: "CubismParameter", path: "Parameters/ParamAngleX", attribute: "Value" }], constant: [value], }); // ParamAngleX from 0 to `to` over the clip's length (one streamed cubic segment, linear) -const ramp = (name, id, len, to) => ({ - ...clip(name, id, len, 0), constant: [], +const ramp = (o, name, id, len, to) => o.clip({ + name, stopTime: len, loopTime: true, events: [{ functionName: "InstanceId", intParameter: id }], + bindings: [{ cls: "CubismParameter", path: "Parameters/ParamAngleX", attribute: "Value" }], constant: [], streamed: { curveCount: 1, frames: [[0, [[0, 0, 0, F(to / len), 0]]], [len, [[0, 0, 0, 0, to]]]] }, }); -const prefab = ({ mouthTags = ["ParamMouthOpenY"] } = {}) => { +// the model's objects and model.json: an AssetStore (store) and the model.json document (model) +const modelFiles = ({ mouthTags = ["ParamMouthOpenY"] } = {}) => { + const o = new ModelObjects(); + const lit = o.shader("Live2D Cubism/Lit", { properties: [], subShaders: [] }, []); + const materialId = o.material("Lit", { shader: lit }); + const textureId = o.texture("texture_00"); + const fades = o.mono("CubismFadeMotionList", "common.fadeMotionList", { + MotionInstanceIds: [-1, -2, -3, -4], + CubismFadeMotionObjects: [ + o.mono("CubismFadeMotionData", "mtn_idle.fade", fadeData("mtn_idle", 2, ["ParamAngleX", "ParamEyeLOpen"], + [curve(2, 0, 0), curve(2, 1, 1)])), + o.mono("CubismFadeMotionData", "mtn_turn.fade", fadeData("mtn_turn", 1, ["ParamAngleX"], [curve(1, 30, 30)])), + o.mono("CubismFadeMotionData", "misc_sway.fade", fadeData("misc_sway", 2, ["ParamLoopA", "ParamMissing"], + [curve(2, 0, 1), curve(2, 0, 1)])), + o.mono("CubismFadeMotionData", "mtn_nod.fade", fadeData("mtn_nod", 1, ["ParamAngleX"], [curve(1, 0, 20)])), + ].map(ref), + }); + const expressions = o.mono("CubismExpressionList", "common.expressionList", { CubismExpressionObjects: [ + o.mono("CubismExpressionData", "exp_idle.exp3", { FadeInTime: 0.5, FadeOutTime: 0.5, Parameters: [] }), + o.mono("CubismExpressionData", "exp_closed.exp3", { FadeInTime: 0.5, FadeOutTime: 0.5, + Parameters: [{ Id: "ParamEyeLOpen", Value: 0, Blend: 2 }] }), + ].map(ref) }); + const idle = clip(o, "mtn_idle", -1, 2, 0), turn = clip(o, "mtn_turn", -2, 1, 30), nod = ramp(o, "mtn_nod", -4, 1, 20); const root = "model"; - const node = (path, components = []) => ({ path, name: path.split("/").pop(), active: true, layer: 0, - localPosition: { x: 0, y: 0, z: 0 }, localRotation: { x: 0, y: 0, z: 0, w: 1 }, localScale: { x: 1, y: 1, z: 1 }, components }); - const mb = (cls, fields) => ({ type: "MonoBehaviour", class: cls, ...fields }); - const head = node(`${root}/Anchors/Head`); - head.localPosition = { x: 0.25, y: 1.5, z: 0 }; - return { key: "model", canvas: {}, nodes: [ + const prefabId = o.prefab([ node(root, [ mb("Live2DCharacter", { DefaultMotionName: "mtn_idle", DefaultExpressionName: "exp_idle", BasePosition: { x: 0, y: 0, z: 0 }, - BaseScale: 1, _motionList: [clip("mtn_idle", -1, 2, 0), clip("mtn_turn", -2, 1, 30), - ramp("mtn_nod", -4, 1, 20)], - _expressionList: ["exp_idle", "exp_closed"] }), - mb("CubismFadeController", { CubismFadeMotionList: { MotionInstanceIds: [-1, -2, -3, -4], CubismFadeMotionObjects: [ - fadeData("mtn_idle", 2, ["ParamAngleX", "ParamEyeLOpen"], [curve(2, 0, 0), curve(2, 1, 1)]), - fadeData("mtn_turn", 1, ["ParamAngleX"], [curve(1, 30, 30)]), - fadeData("misc_sway", 2, ["ParamLoopA", "ParamMissing"], [curve(2, 0, 1), curve(2, 0, 1)]), - fadeData("mtn_nod", 1, ["ParamAngleX"], [curve(1, 0, 20)]), - ] } }), + BaseScale: 1, _motionList: [idle, turn, nod].map(ref), _expressionList: ["exp_idle", "exp_closed"] }), + mb("CubismFadeController", { CubismFadeMotionList: ref(fades) }), mb("CubismExpressionController", { UseLegacyBlendCalculation: 0, CurrentExpressionIndex: -1, CurrentFadeInTime: -1, - ExpressionsList: { CubismExpressionObjects: [ - { name: "exp_idle.exp3", FadeInTime: 0.5, FadeOutTime: 0.5, Parameters: [] }, - { name: "exp_closed.exp3", FadeInTime: 0.5, FadeOutTime: 0.5, Parameters: [{ Id: "ParamEyeLOpen", Value: 0, Blend: 2 }] }] } }), + ExpressionsList: ref(expressions) }), mb("CubismHarmonicMotionController", { BlendMode: 1, ChannelTimescales: [1] }), mb("CubismAutoEyeBlinkInput", { Mean: 2.5, MaximumDeviation: 2, Timescale: 10 }), mb("CubismEyeBlinkController", { BlendMode: 2, EyeOpening: 1 }), @@ -98,14 +106,17 @@ const prefab = ({ mouthTags = ["ParamMouthOpenY"] } = {}) => { ...(id === "ParamBreath" ? [mb("CubismHarmonicMotionParameter", { Channel: 0, Direction: 2, NormalizedOrigin: 0.5, NormalizedRange: 0.5, Duration: 3 })] : [])])), node(`${root}/Anchors`), - head, + node(`${root}/Anchors/Head`, [], { localPosition: { x: 0.25, y: 1.5, z: 0 } }), node(`${root}/Drawables`), node(`${root}/Drawables/ArtMesh0`, [ mb("CubismDrawable", { _unmanagedIndex: 0 }), - { type: "MeshRenderer", m_Materials: [{ material: "Lit" }] }, - mb("CubismRenderer", { _localSortingOrder: 0, _color: { r: 1, g: 1, b: 1, a: 1 }, _mainTexture: {} }), + component("MeshRenderer", { m_Materials: [ref(materialId)] }), + mb("CubismRenderer", { _localSortingOrder: 0, _color: { r: 1, g: 1, b: 1, a: 1 }, _mainTexture: ref(textureId) }), ]), - ] }; + ]); + const options = { prefab: prefabId, moc: o.moc(new Uint8Array([0x4d, 0x4f, 0x43, 0x33, 5, 0, 0, 0, 0, 0, 0, 0])), + shaders: [lit], resources: {} }; + return { store: o.store(options), model: o.modelJson(options) }; }; // a loaded, shown character driven by a 30 fps loop (the story's hooks) @@ -114,8 +125,8 @@ const shown = async (opts = {}) => { globalThis.Live2DCubismCore = fakeCore(); try { const loop = new PlayerLoop(30); - const moc = new Uint8Array([0x4d, 0x4f, 0x43, 0x33, 5, 0, 0, 0, 0, 0, 0, 0]).buffer; - const ch = new Live2DCharacter(prefab(opts), moc, loop, { random: new UnityRandom(1) }); + const { store, model } = modelFiles(opts); + const ch = new Live2DCharacter(store, model, loop, { random: new UnityRandom(1) }); const requests = []; loop.on("update", () => { for (const f of requests.splice(0)) f(); ch.update(); }); loop.on("animation", () => ch.animatorUpdate()); diff --git a/tests/unit/live2d-model.test.mjs b/tests/unit/live2d-model.test.mjs index dd4053f..31e9d77 100644 --- a/tests/unit/live2d-model.test.mjs +++ b/tests/unit/live2d-model.test.mjs @@ -1,90 +1,97 @@ // model.json (src/live2d/model.js, docs/data-format.md "model.json"), shared by the model viewer and the story sessions; -// the models of a story's Character rows (src/story/models.js: a model id with the model under live2d//, or the -// {dir, moc3, prefab} form of story manifest /1); the shaders each character draws with (StoryRenderer and -// SimpleCaptureRenderer addCharacter) and the keywords the story renderer adds to them per quality. Synthetic files. +// the models of a story's Character rows (src/story/models.js: a model id, the model's model.json under live2d// +// and its objects in the store's object table); the drawing each character is given (StoryRenderer and +// SimpleCaptureRenderer addCharacter) and the keywords the story renderer adds to character draws per quality. +// Synthetic objects (live2d-objects.mjs). import assert from "node:assert/strict"; import { test } from "node:test"; import { AssetStore } from "../../src/data/assets.js"; import { PlayerLoop } from "../../src/engine/loop.js"; -import { MODEL_JSON_FORMATS, modelIndex } from "../../src/live2d/model.js"; +import { MODEL_JSON_FORMAT, modelIndex } from "../../src/live2d/model.js"; import { AdvQuality } from "../../src/story/field.js"; import { storyModels } from "../../src/story/models.js"; import { StoryRenderer } from "../../src/story/renderer.js"; import { SimpleCaptureRenderer } from "../../src/story/simple/render.js"; import { headlessGL } from "../../scripts/lib/headless.mjs"; +import { ModelObjects, node } from "./live2d-objects.mjs"; -const MASK = "Live2D Cubism/Mask"; -const resources = { cubismMask: { shader: { shader: MASK }, floats: { _Cull: 0 } }, cubismMaskCulling: { shader: { shader: MASK }, floats: { _Cull: 1 } } }; -const index = (extra = {}) => ({ format: 2, motionSync: true, name: "m1", moc3: "m1.moc3", prefab: "m1.prefab.json", - shaders: "shaders/shaders.json", resources, ...extra }); -const store = (files) => new AssetStore({ text: Object.fromEntries(Object.entries(files).map(([p, v]) => [p, JSON.stringify(v)])) }); +const LIT = "Live2D Cubism/Lit-URP-ADV-optimize", MASK = "Live2D Cubism/Mask"; -test("modelIndex: model.json at the store's root or in a directory, formats 1 and 2", () => { - assert.deepEqual(MODEL_JSON_FORMATS, [1, 2]); - const root = modelIndex(store({ "model.json": index({ prefab: "p/m1.prefab.json" }) })); - assert.equal(root.moc3, "m1.moc3"); - assert.equal(root.prefab, "p/m1.prefab.json"); - assert.equal(root.textureDir, "p"); // the drawables' textures are relative to the prefab - assert.equal(root.shaderDir, "shaders"); - assert.equal(root.motionSync, true); - assert.equal(root.resources, root.index.resources); - const sub = modelIndex(store({ "live2d/m1/model.json": index() }), "live2d/m1"); - assert.deepEqual([sub.moc3, sub.prefab, sub.shaders, sub.textureDir, sub.shaderDir], - ["live2d/m1/m1.moc3", "live2d/m1/m1.prefab.json", "live2d/m1/shaders/shaders.json", "live2d/m1", "live2d/m1/shaders"]); - const v1 = modelIndex(store({ "model.json": index({ format: 1, motionSync: undefined }) })); - assert.equal(v1.motionSync, null); // not stated in format 1 - const bad = (x, re, dir = "") => assert.throws(() => modelIndex(store({ [dir ? `${dir}/model.json` : "model.json"]: x }), dir), re); - bad(index({ format: 3 }), /^Error: model\.json: format 3 is not supported \(1 or 2 expected\)$/); - bad(index({ format: 3 }), /live2d\/x\/model\.json: format 3/, "live2d/x"); - bad(index({ motionSync: undefined }), /"motionSync" missing/); - bad(index({ moc3: "" }), /"moc3" missing/); - bad(index({ shaders: "shaders/index.json" }), /"shaders" must name a shaders\.json/); - assert.throws(() => modelIndex(store({}), "live2d/none"), /live2d\/none\/model\.json/); +// the objects of a model (its prefab, moc, two shaders, the mask materials, a clip) and the options of its model.json +const objects = () => { + const o = new ModelObjects(); + const lit = o.shader(LIT, { properties: [], subShaders: [] }, []); + const mask = o.shader(MASK, { properties: [], subShaders: [] }, []); + const clip = o.clip({ name: "idle", bindings: [{ cls: "CubismParameter", path: "Parameters/A", attribute: "Value" }], + constant: [0] }); + const model = { prefab: o.prefab([node("m1")]), moc: o.moc(new Uint8Array(4)), shaders: [lit, mask], motionSync: true, + resources: { cubismMask: o.material("Mask", { shader: mask, floats: { _Cull: 0 } }), + cubismMaskCulling: o.material("MaskCulling", { shader: mask, floats: { _Cull: 1 } }) } }; + return { o, lit, clip, model }; +}; + +test("modelIndex: model.json at the store's root or in a directory, the classes of its objects checked", () => { + assert.equal(MODEL_JSON_FORMAT, 3); + const { o, clip, model } = objects(); + const root = modelIndex(o.store(model)); + assert.deepEqual(root, o.modelJson(model)); + assert.deepEqual(Object.keys(root.bindings), [clip]); + assert.equal(modelIndex(o.store({ dir: "live2d/m1", ...model }), "live2d/m1").moc, model.moc); + const bad = (edit, re, dir = "") => { + const text = { [dir ? `${dir}/model.json` : "model.json"]: JSON.stringify(edit(o.modelJson(model))) }; + assert.throws(() => modelIndex(new AssetStore({ text, objects: o.objects }), dir), re); + }; + bad((m) => ({ ...m, format: 2 }), /^Error: model\.json: format 2 is not supported \(3 expected\)$/); + bad((m) => ({ ...m, format: 2 }), /live2d\/x\/model\.json: format 2/, "live2d/x"); + bad(({ motionSync, ...m }) => m, /model\.json: "motionSync" missing/); + bad(({ shaders, ...m }) => m, /model\.json: "shaders" missing/); + bad((m) => ({ ...m, prefab: m.moc }), /model\.json: CAB-test:\d+: a MonoBehaviour, not a GameObject/); + const list = o.mono("CubismFadeMotionList", "common.fadeMotionList"); + bad((m) => ({ ...m, moc: list }), new RegExp(`model\\.json: ${list}: not a CubismMoc`)); + bad((m) => ({ ...m, resources: { ...m.resources, cubismMask: "CAB-test:999" } }), /model\.json: object not found: CAB-test:999/); + bad((m) => ({ ...m, bindings: { [m.moc]: [] } }), /model\.json: CAB-test:\d+: a MonoBehaviour, not a AnimationClip/); + assert.throws(() => modelIndex(o.store(model), "live2d/none"), /live2d\/none\/model\.json/); }); -test("storyModels: a model id (live2d//, its own shaders, one ShaderLib per id) or {dir, moc3, prefab}", () => { +test("storyModels: a model id names the model.json under live2d//; one model and ShaderLib per id", () => { const gl = headlessGL(); - const files = { - "live2d/m1/model.json": index(), "live2d/m1/shaders/shaders.json": [], - "shaders/shaders.json": [], - }; - const s = store(files); + const { o, model } = objects(); + const s = o.store({ dir: "live2d/m1", ...model }); const story = { models: { "Character/Live2D/g/a/model/a": "m1", "Character/Live2D/g/b/model/b": "m1", - "Character/Live2D/g/c/model/c": { dir: "live2d/c", moc3: "c.moc3", prefab: "c.prefab.json" } } }; - const storyLib = { story: true }, sceneResources = { scene: true }; - const modelOf = storyModels(gl, s, story, "story 7", { lib: storyLib, resources: sceneResources }); - const a = modelOf("Character/Live2D/g/a/model/a"), b = modelOf("Character/Live2D/g/b/model/b"); + "Character/Live2D/g/c/model/c": "" } }; + const modelOf = storyModels(gl, s, story, "story 7"); + const a = modelOf("Character/Live2D/g/a/model/a"); + assert.equal(a, modelOf("Character/Live2D/g/b/model/b")); // one model (and ShaderLib) per id assert.equal(a.id, "m1"); - assert.equal(a, b); // one model (and ShaderLib) per id - assert.equal(a.lib.base, "live2d/m1/shaders"); - assert.equal(a.model.prefab, "live2d/m1/m1.prefab.json"); - assert.deepEqual(a.model.resources, resources); // model.json resources - const c = modelOf("Character/Live2D/g/c/model/c"); - assert.equal(c.id, null); - assert.equal(c.lib, storyLib); // manifest /1: the story's shaders - assert.deepEqual([c.model.prefab, c.model.moc3, c.model.textureDir], ["live2d/c/c.prefab.json", "live2d/c/c.moc3", "live2d/c"]); - assert.equal(c.model.resources, sceneResources); // and scene.json resources + assert.deepEqual(a.model, o.modelJson(model)); + assert.equal(a.lib.info(LIT).name, LIT); // the ShaderLib of model.json shaders assert.throws(() => modelOf("Character/Live2D/g/x/model/x"), /story 7: model Character\/Live2D\/g\/x\/model\/x is not in the story/); + assert.throws(() => modelOf("Character/Live2D/g/c/model/c"), /story 7: model Character\/Live2D\/g\/c\/model\/c is not in the story/); const headless = storyModels(null, s, story, "story 7")("Character/Live2D/g/a/model/a"); assert.equal(headless.lib, null); // no GL: no shaders - assert.equal(headless.model.moc3, "live2d/m1/m1.moc3"); + assert.equal(headless.model.moc, model.moc); }); -test("addCharacter: each character draws with the ShaderLib, textures and mask materials it is given", () => { +test("addCharacter: each character draws with the ShaderLib it is given and the mask materials of its model.json", () => { const gl = headlessGL(); + const { o, lit, model } = objects(); + const assets = o.store(model), m = o.modelJson(model); const character = { name: "c", renderers: [] }; const lib = { model: 1 }; const loop = new PlayerLoop(30); - const r = new StoryRenderer(gl, { story: 1 }, { camera: null, session: { stage: null } }, { unityLighting: false }, loop, { assets: null }); - const g = r.addCharacter(character, { lib, dir: "live2d/m1", resources }); + const r = new StoryRenderer(gl, { story: 1 }, { camera: null, session: { stage: null } }, { unityLighting: false }, loop, { assets }); + const g = r.addCharacter(character, { lib, model: m }); assert.equal(g.lib, lib); - assert.equal(g.dir, "live2d/m1"); - assert.equal(g.maskMats[1], resources.cubismMaskCulling); - const sr = new SimpleCaptureRenderer(gl, loop, { assets: null }); - const h = sr.addCharacter(character, { lib, dir: "live2d/m1", resources }); + assert.deepEqual(g.maskMats[1], assets.material(m.resources.cubismMaskCulling)); + assert.equal(g.maskMats[1].floats._Cull, 1); + const sr = new SimpleCaptureRenderer(gl, loop, { assets }); + const h = sr.addCharacter(character, { lib, model: m }); assert.equal(h.lib, lib); assert.equal(sr.drawings.get(character), h); - assert.throws(() => sr.addCharacter(character, { lib, dir: "", resources: {} }), /resources\.cubismMask/); + assert.throws(() => sr.addCharacter(character, { lib, model: { ...m, shaders: { ...m.shaders, [MASK]: lit } } }), + /mask material Mask: not drawn with Live2D Cubism\/Mask/); + const masked = { name: "d", renderers: [{ name: "ArtMesh0", material: assets.material(m.resources.cubismMask) }] }; + assert.throws(() => sr.addCharacter(masked, { lib, model: m }), /ArtMesh0: material Mask is not drawn with Live2D Cubism\/Lit-URP-ADV-optimize/); }); test("the keywords the story renderer adds to character draws: _ADDITIONAL_LIGHTS_VERTEX at quality 4 only", () => { diff --git a/tests/unit/live2d-objects.mjs b/tests/unit/live2d-objects.mjs new file mode 100644 index 0000000..c60c1e6 --- /dev/null +++ b/tests/unit/live2d-objects.mjs @@ -0,0 +1,115 @@ +// Synthetic Live2D models in the product form (docs/data-format.md "Objects" and "model.json"): objects in memory that +// name each other by reference, and model.json format 3, for the tests of the model viewer and the story's characters. +// +// const o = new ModelObjects(); +// const clip = o.clip({ name: "idle", bindings: [{ cls: "CubismParameter", path: "Parameters/ParamAngleX", +// attribute: "Value" }], constant: [0], ... }); +// const prefab = o.prefab([node("model", [mb("Live2DCharacter", { _motionList: [ref(clip)], ... })]), ...]); +// const store = o.store({ prefab, moc: o.moc(bytes), shaders: [lit, mask], resources: {...}, motionSync: false }); +import { AssetStore } from "../../src/data/assets.js"; + +export const FILE = "CAB-test"; + +// a reference to an object id +export const ref = (id) => { + const i = id.lastIndexOf(":"); + return { $ref: { file: id.slice(0, i), pathId: id.slice(i + 1) } }; +}; + +// prefab.json nodes and components +export const node = (path, components = [], extra = {}) => ({ + path, name: path.split("/").pop(), pathId: "0", transformPathId: "0", active: true, layer: 0, + localPosition: { x: 0, y: 0, z: 0 }, localRotation: { x: 0, y: 0, z: 0, w: 1 }, localScale: { x: 1, y: 1, z: 1 }, + ...extra, components, +}); +export const mb = (cls, fields = {}) => ({ type: "MonoBehaviour", pathId: "0", script: `Test||${cls}`, fields }); +export const component = (type, fields = {}) => ({ type, pathId: "0", fields }); + +export class ModelObjects { + constructor(file = FILE) { + this.file = file; + this.next = 1; + this.objects = {}; // id -> {class, name, files} + this.bindings = {}; // clip id -> [{path, attribute}] (model.json bindings) + } + + add(cls, name, files) { + const id = `${this.file}:${this.next++}`; + this.objects[id] = { class: cls, name, files }; + return id; + } + + // a MonoBehaviour outside the hierarchy (mono.json) + mono(cls, name, fields = {}) { + return this.add("MonoBehaviour", name, { json: JSON.stringify({ $script: `Test||${cls}`, m_Name: name, ...fields }) }); + } + + // an AnimationClip whose bindings are {cls, path, attribute}: clip.json (a MonoBehaviour binding names its script + // class) and the names model.json gives them + clip({ name, bindings = [], events = [], stopTime = 1, loopTime = false, startTime = 0, cycleOffset = 0, + streamed = { curveCount: 0, frames: [] }, dense = { curveCount: 0 }, constant = [] }) { + const doc = { + name, sampleRate: 30, wrapMode: 0, startTime, stopTime, loopTime, cycleOffset, events, + bindings: bindings.map((b) => ({ pathCrc: 0, attributeCrc: 0, typeID: 114, class: "MonoBehaviour", customType: 0, + pptr: false, int: false, serializeReference: false, scriptClass: `Test||${b.cls}` })), + streamed, dense, constant, pptrCurveMapping: [], + }; + const id = this.add("AnimationClip", name, { json: JSON.stringify(doc) }); + this.bindings[id] = bindings.map((b) => ({ path: b.path, attribute: b.attribute })); + return id; + } + + // a Texture2D: its typetree (size, mip count, settings) and its image + texture(name, { width = 4, height = 4, mipCount = 1, settings = { m_FilterMode: 1, m_WrapU: 1, m_WrapV: 1 }, + image = new Uint8Array(0) } = {}) { + const json = { m_Name: name, m_Width: width, m_Height: height, m_MipCount: mipCount, m_ColorSpace: 1, + m_TextureSettings: settings }; + return this.add("Texture2D", name, { json: JSON.stringify(json), image }); + } + + material(name, { shader, keywords = [], floats = {}, colors = {} }) { + const doc = { material: name, shader: ref(shader), keywords, invalidKeywords: [], renderQueue: -1, tags: {}, + disabledPasses: [], textures: {}, ints: {}, floats, colors }; + return this.add("Material", name, { json: JSON.stringify(doc) }); + } + + // a Shader: its parsed form (`parsed`), and per variant [keywords, GLSL] a GLES3 program of sub shader 0, pass 0 + shader(name, parsed, variants) { + const files = { json: JSON.stringify({ name, ...parsed }) }; + const index = { name, variants: variants.map(([keywords], i) => ({ + file: `x/gles3/s0p0_vertex_${i}.glsl`, platform: "gles3", subShader: 0, pass: 0, stage: "vertex", type: "GLES3", + keywords, role: `program:gles3/s0p0_vertex_${i}` })) }; + files.index = JSON.stringify(index); + variants.forEach(([, glsl], i) => { files[`program:gles3/s0p0_vertex_${i}`] = glsl; }); + return this.add("Shader", name, files); + } + + // the CubismMoc MonoBehaviour holding the moc3 + moc(bytes, name = "model") { + return this.add("MonoBehaviour", name, { json: JSON.stringify({ $script: "Test||CubismMoc", m_Name: name }), + "blob:_bytes": bytes }); + } + + // a root GameObject's prefab.json + prefab(nodes, name = nodes[0].name) { + return this.add("GameObject", name, { json: JSON.stringify({ name, nodes }) }); + } + + // model.json format 3 of these objects (shaders: Shader ids) + modelJson({ prefab, moc, shaders, resources, motionSync = false, name = "model", group = "test" }) { + return { + format: 3, name, key: `Character/Live2D/${group}/${name}/model/${name}`, prefab, moc, + canvas: { pixelsPerUnit: 600, originX: 300, originY: 450, width: 600, height: 900, mocVersion: 5 }, + textures: Object.entries(this.objects).filter(([, o]) => o.class === "Texture2D").map(([id]) => id), + shaders: Object.fromEntries(shaders.map((id) => [this.objects[id].name, id])), + resources, motionSync, bindings: this.bindings, + }; + } + + // an AssetStore with model.json at `dir` (modelJson's options), the objects, and the other files `text` / `bytes` + store({ dir = "", text = {}, bytes = {}, info = null, ...model }) { + const where = dir ? `${dir}/model.json` : "model.json"; + return new AssetStore({ text: { ...text, [where]: JSON.stringify(this.modelJson(model)) }, bytes, + objects: this.objects, info }); + } +} diff --git a/tests/unit/live2d-session.test.mjs b/tests/unit/live2d-session.test.mjs index d42819e..49d3e21 100644 --- a/tests/unit/live2d-session.test.mjs +++ b/tests/unit/live2d-session.test.mjs @@ -1,9 +1,10 @@ // ModelSession over a synthetic model: a stand-in for Live2D Cubism Core (a few parameters, three drawables whose -// vertices follow ParamAngleX, one of them masked), a prefab with the components the runtime reads, shaders in the -// packed Unity layout, all in memory; drawn into the no-op WebGL2 context of scripts/lib/headless.mjs. +// vertices follow ParamAngleX, one of them masked), the model's objects (a prefab with the components the runtime +// reads, clips, fade motions, expressions, materials, a texture, shaders with their GLES3 programs) and model.json, all +// in memory (live2d-objects.mjs); drawn into the no-op WebGL2 context of scripts/lib/headless.mjs. import assert from "node:assert/strict"; import { test } from "node:test"; -import { AssetStore } from "../../src/data/assets.js"; +import { ModelObjects, component, mb, node, ref } from "./live2d-objects.mjs"; import { MODEL_FRAME_RATE, ModelSession } from "../../src/live2d/session.js"; import { headlessGL, headlessImages } from "../../scripts/lib/headless.mjs"; @@ -63,41 +64,62 @@ const fakeCore = () => { const MOC = new Uint8Array([0x4d, 0x4f, 0x43, 0x33, 5, 0, 0, 0, 0, 0, 0, 0]); -// ---- prefab +// ---- the model's objects +const LIT = "Live2D Cubism/Lit-URP-ADV-optimize", MASK = "Live2D Cubism/Mask"; const fadeData = (name, len, value) => ({ MotionName: name, FadeInTime: 1, FadeOutTime: 1, MotionLength: len, ParameterFadeInTimes: [-1], ParameterFadeOutTimes: [-1], ParameterIds: ["ParamAngleX"], ParameterCurves: [{ m_Curve: [{ time: 0, value, inSlope: 0, outSlope: 0 }, { time: len, value, inSlope: 0, outSlope: 0 }] }], }); -const clip = (name, id, len, value, opacity) => ({ - clip: name, stopTime: len, loopTime: true, startTime: 0, cycleOffset: 0, - events: [{ functionName: "InstanceId", intParameter: id }], dense: { curveCount: 0 }, - bindings: [{ path: "Parameters/ParamAngleX", class: "CubismParameter", attribute: "Value" }, - ...(opacity === undefined ? [] : [{ path: "", class: "CubismRenderController", attribute: "Opacity" }])], - streamed: { curveCount: 0, frames: [] }, constant: [value, ...(opacity === undefined ? [] : [opacity])], +const clip = (o, name, id, len, value, opacity) => o.clip({ + name, stopTime: len, loopTime: true, events: [{ functionName: "InstanceId", intParameter: id }], + bindings: [{ cls: "CubismParameter", path: "Parameters/ParamAngleX", attribute: "Value" }, + ...(opacity === undefined ? [] : [{ cls: "CubismRenderController", path: "", attribute: "Opacity" }])], + constant: [value, ...(opacity === undefined ? [] : [opacity])], }); -const tex = { texture: "textures/texture_00.png", name: "texture_00", width: 4, height: 4, mipCount: 1, - settings: { m_FilterMode: 1, m_WrapU: 1, m_WrapV: 1 } }; -const material = (keywords) => ({ material: keywords.length ? "LitMasked" : "Lit", shader: { shader: "Live2D Cubism/Lit-URP-ADV-optimize" }, - keywords, floats: { _SrcColor: 1, _DstColor: 10, _SrcAlpha: 1, _DstAlpha: 10, _Cull: 0 }, colors: {} }); -const prefab = ({ physics = true, mipCount = 1, opacity, missing = false } = {}) => { +// ---- shaders: the parsed forms and GLES3 programs +const program = "#ifdef VERTEX\n#version 300 es\nvoid main() {}\n#endif\n#ifdef FRAGMENT\n#version 300 es\nvoid main() {}\n#endif\n"; +const V = (val, name = "") => ({ val, name }); +const pass = (blend) => ({ state: { + rtBlend0: { srcBlend: blend[0], destBlend: blend[1], srcBlendAlpha: blend[2], destBlendAlpha: blend[3], blendOp: V(0), + blendOpAlpha: V(0), colMask: V(15) }, + zTest: V(4), zWrite: V(0), culling: V(0, "_Cull"), offsetFactor: V(0), offsetUnits: V(0), stencilRef: V(0), + stencilReadMask: V(255), stencilWriteMask: V(255), rtSeparateBlend: false, alphaToMask: V(0), + stencilOp: { comp: V(8), pass: V(0), fail: V(0), zFail: V(0) }, + stencilOpFront: { comp: V(8), pass: V(0), fail: V(0), zFail: V(0) }, stencilOpBack: { comp: V(8), pass: V(0), fail: V(0), zFail: V(0) }, +} }); +const litPass = pass([V(1, "_SrcColor"), V(10, "_DstColor"), V(1, "_SrcAlpha"), V(10, "_DstAlpha")]); +const maskPass = pass([V(1), V(1), V(1), V(1)]); +const parsed = (p) => ({ properties: [], subShaders: [{ tags: { tags: [] }, passes: [p] }] }); + +const modelFiles = ({ motionSync = false, physics = true, mipCount = 1, opacity, missing = false, json = null } = {}) => { + const o = new ModelObjects(); + const lit = o.shader(LIT, parsed(litPass), [[[], program], [["CUBISM_MASK_ON"], program]]); + const mask = o.shader(MASK, parsed(maskPass), [[[], program]]); + const material = (keywords) => o.material(keywords.length ? "LitMasked" : "Lit", { + shader: lit, keywords, floats: { _SrcColor: 1, _DstColor: 10, _SrcAlpha: 1, _DstAlpha: 10, _Cull: 0 } }); + const lits = [material([]), material(["CUBISM_MASK_ON"])]; + const png = new Uint8Array(24); + png.set([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a, 0, 0, 0, 13, 0x49, 0x48, 0x44, 0x52, 0, 0, 0, 4, 0, 0, 0, 4]); + const tex = o.texture("texture_00", { mipCount, image: png }); + const fades = o.mono("CubismFadeMotionList", "common.fadeMotionList", { MotionInstanceIds: [-1, -2], + CubismFadeMotionObjects: [o.mono("CubismFadeMotionData", "mtn_idle.fade", fadeData("mtn_idle", 2, 0)), + o.mono("CubismFadeMotionData", "mtn_turn.fade", fadeData("mtn_turn", 1, 30))].map(ref) }); + const expressions = o.mono("CubismExpressionList", "common.expressionList", { CubismExpressionObjects: [ + o.mono("CubismExpressionData", "exp_idle.exp3", { FadeInTime: 0.5, FadeOutTime: 0.5, Parameters: [] }), + o.mono("CubismExpressionData", "exp_closed.exp3", { FadeInTime: 0.5, FadeOutTime: 0.5, Parameters: [ + ...(missing ? [{ Id: "ParamMissing", Value: 1, Blend: 1 }] : []), { Id: "ParamEyeLOpen", Value: 0, Blend: 2 }] }), + ].map(ref) }); const root = "model"; - const node = (path, components = []) => ({ path, name: path.split("/").pop(), active: true, layer: 0, - localPosition: { x: 0, y: 0, z: 0 }, localRotation: { x: 0, y: 0, z: 0, w: 1 }, localScale: { x: 1, y: 1, z: 1 }, components }); - const mb = (cls, fields) => ({ type: "MonoBehaviour", class: cls, ...fields }); - return { key: "model", canvas: {}, nodes: [ + const prefab = o.prefab([ node(root, [ mb("Live2DCharacter", { DefaultMotionName: "mtn_idle", DefaultExpressionName: "exp_idle", BasePosition: { x: 0, y: -0.41, z: 0 }, - BaseScale: 1.6, _motionList: [clip("mtn_idle", -1, 2, 0), clip("mtn_turn", -2, 1, 30, opacity)], + BaseScale: 1.6, _motionList: [clip(o, "mtn_idle", -1, 2, 0), clip(o, "mtn_turn", -2, 1, 30, opacity)].map(ref), _expressionList: ["exp_idle", "exp_closed"] }), - mb("CubismFadeController", { CubismFadeMotionList: { MotionInstanceIds: [-1, -2], - CubismFadeMotionObjects: [fadeData("mtn_idle", 2, 0), fadeData("mtn_turn", 1, 30)] } }), + mb("CubismFadeController", { CubismFadeMotionList: ref(fades) }), mb("CubismExpressionController", { UseLegacyBlendCalculation: 0, CurrentExpressionIndex: -1, CurrentFadeInTime: -1, - ExpressionsList: { CubismExpressionObjects: [ - { name: "exp_idle.exp3", FadeInTime: 0.5, FadeOutTime: 0.5, Parameters: [] }, - { name: "exp_closed.exp3", FadeInTime: 0.5, FadeOutTime: 0.5, Parameters: [ - ...(missing ? [{ Id: "ParamMissing", Value: 1, Blend: 1 }] : []), { Id: "ParamEyeLOpen", Value: 0, Blend: 2 }] }] } }), + ExpressionsList: ref(expressions) }), mb("CubismHarmonicMotionController", { BlendMode: 1, ChannelTimescales: [1] }), mb("CubismAutoEyeBlinkInput", { Mean: 2.5, MaximumDeviation: 2, Timescale: 10 }), mb("CubismEyeBlinkController", { BlendMode: 2, EyeOpening: 1 }), @@ -106,53 +128,23 @@ const prefab = ({ physics = true, mipCount = 1, opacity, missing = false } = {}) mb("CubismRenderController", { _sortingOrder: 0, Opacity: 1 }), ]), node(`${root}/Parameters`), - ...IDS.map((id) => node(`${root}/Parameters/${id}`, [mb("CubismParameter", {}), - ...(id === "ParamEyeLOpen" ? [mb("CubismEyeBlinkParameter", {})] : []), + ...IDS.map((id) => node(`${root}/Parameters/${id}`, [mb("CubismParameter"), + ...(id === "ParamEyeLOpen" ? [mb("CubismEyeBlinkParameter")] : []), ...(id === "ParamBreath" ? [mb("CubismHarmonicMotionParameter", { Channel: 0, Direction: 2, NormalizedOrigin: 0.5, NormalizedRange: 0.5, Duration: 3 })] : [])])), node(`${root}/Drawables`), ...DRAWABLES.map((id, i) => node(`${root}/Drawables/${id}`, [ mb("CubismDrawable", { _unmanagedIndex: i }), - { type: "MeshRenderer", m_Materials: [material(i === 2 ? ["CUBISM_MASK_ON"] : [])] }, - mb("CubismRenderer", { _localSortingOrder: 0, _color: { r: 1, g: 1, b: 1, a: 1 }, _mainTexture: { ...tex, mipCount } }), + component("MeshRenderer", { m_Materials: [ref(lits[i === 2 ? 1 : 0])] }), + mb("CubismRenderer", { _localSortingOrder: 0, _color: { r: 1, g: 1, b: 1, a: 1 }, _mainTexture: ref(tex) }), ])), - ] }; -}; - -// ---- shaders (packed Unity layout) -const program = "#ifdef VERTEX\n#version 300 es\nvoid main() {}\n#endif\n#ifdef FRAGMENT\n#version 300 es\nvoid main() {}\n#endif\n"; -const V = (val, name = "") => ({ val, name }); -const pass = (blend) => ({ state: { - rtBlend0: { srcBlend: blend[0], destBlend: blend[1], srcBlendAlpha: blend[2], destBlendAlpha: blend[3], blendOp: V(0), - blendOpAlpha: V(0), colMask: V(15) }, - zTest: V(4), zWrite: V(0), culling: V(0, "_Cull"), offsetFactor: V(0), offsetUnits: V(0), stencilRef: V(0), - stencilReadMask: V(255), stencilWriteMask: V(255), rtSeparateBlend: false, alphaToMask: V(0), - stencilOp: { comp: V(8), pass: V(0), fail: V(0), zFail: V(0) }, - stencilOpFront: { comp: V(8), pass: V(0), fail: V(0), zFail: V(0) }, stencilOpBack: { comp: V(8), pass: V(0), fail: V(0), zFail: V(0) }, -} }); -const litPass = pass([V(1, "_SrcColor"), V(10, "_DstColor"), V(1, "_SrcAlpha"), V(10, "_DstAlpha")]); -const maskPass = pass([V(1), V(1), V(1), V(1)]); - -const modelFiles = ({ format = 1, motionSync, physics = true, mipCount = 1, opacity, missing = false } = {}) => { - const lit = "Live2D Cubism/Lit-URP-ADV-optimize", mask = "Live2D Cubism/Mask"; - const text = { - "model.json": JSON.stringify({ format, ...(motionSync === undefined ? {} : { motionSync }), name: "model", - moc3: "model.moc3", prefab: "model.prefab.json", shaders: "shaders/shaders.json", resources: { - cubismMask: { material: "Mask", shader: { shader: mask }, keywords: [], floats: { _Cull: 0 }, colors: {} }, - cubismMaskCulling: { material: "MaskCulling", shader: { shader: mask }, keywords: [], floats: { _Cull: 1 }, colors: {} } } }), - "model.prefab.json": JSON.stringify(prefab({ physics, mipCount, opacity, missing })), - "shaders/shaders.json": JSON.stringify([ - { name: lit, parsed: "lit.json", variants: [{ file: "lit/0.glsl", subShader: 0, pass: 0, keywords: [] }, - { file: "lit/1.glsl", subShader: 0, pass: 0, keywords: ["CUBISM_MASK_ON"] }] }, - { name: mask, parsed: "mask.json", variants: [{ file: "mask/0.glsl", subShader: 0, pass: 0, keywords: [] }] }]), - "shaders/lit.json": JSON.stringify({ properties: [], subShaders: [{ tags: { tags: [] }, passes: [litPass] }] }), - "shaders/mask.json": JSON.stringify({ properties: [], subShaders: [{ tags: { tags: [] }, passes: [maskPass] }] }), - "shaders/lit/0.glsl": program, "shaders/lit/1.glsl": program, "shaders/mask/0.glsl": program, - }; - const png = new Uint8Array(24); - png.set([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a, 0, 0, 0, 13, 0x49, 0x48, 0x44, 0x52, 0, 0, 0, 4, 0, 0, 0, 4]); - return headlessImages(new AssetStore({ text, bytes: { "model.moc3": MOC, "textures/texture_00.png": png }, - info: { id: "model", model: { group: "test" } } })); + ]); + const resources = { cubismMask: o.material("Mask", { shader: mask, floats: { _Cull: 0 } }), + cubismMaskCulling: o.material("MaskCulling", { shader: mask, floats: { _Cull: 1 } }) }; + const store = o.store({ prefab, moc: o.moc(MOC), shaders: [lit, mask], resources, motionSync, + info: { schema: "nnnotes.product/1", product: "live2d.model", subject: "model" } }); + if (json) store._text.set("model.json", JSON.stringify(json(JSON.parse(store.text("model.json"))))); + return headlessImages(store); }; const withCore = async (fn) => { @@ -176,7 +168,7 @@ test("a model loads, warms up and is shown with its default motion and expressio assert.deepEqual(s.expressions, ["exp_idle", "exp_closed"]); assert.equal(s.motion, "mtn_idle"); assert.equal(s.expression, "exp_idle"); - assert.deepEqual(s.info, { id: "model", model: { group: "test" } }); + assert.deepEqual(s.info, { schema: "nnnotes.product/1", product: "live2d.model", subject: "model" }); assert.ok(s.loop.frameCount > 8 && s.loop.frameCount < 20, `load frames ${s.loop.frameCount}`); assert.equal(s.character.isShowing, true); assert.equal(s.physics, true); @@ -332,7 +324,7 @@ test("a mipmapped texture gets its other levels generated", () => withCore(async } })); -test("errors: no Core, unknown format, unknown initial motion; a disposed session; model.json formats 1 and 2", async () => { +test("errors: no Core, unknown format, unknown initial motion; a disposed session; model.json checks", async () => { const saved = globalThis.Live2DCubismCore; delete globalThis.Live2DCubismCore; try { @@ -340,9 +332,14 @@ test("errors: no Core, unknown format, unknown initial motion; a disposed sessio } finally { if (saved) globalThis.Live2DCubismCore = saved; } await withCore(async () => { const gl = headlessGL(); - await assert.rejects(ModelSession.create({ gl, assets: modelFiles({ format: 3 }) }), /model\.json: format 3 is not supported/); - await assert.rejects(ModelSession.create({ gl, assets: modelFiles({ format: 2 }) }), /model\.json: "motionSync" missing/); - await (await ModelSession.create({ gl, assets: modelFiles({ format: 2, motionSync: false }) })).dispose(); + const edit = (json) => ModelSession.create({ gl, assets: modelFiles({ json }) }); + await assert.rejects(edit((m) => ({ ...m, format: 2 })), /model\.json: format 2 is not supported \(3 expected\)/); + await assert.rejects(edit(({ motionSync, ...m }) => m), /model\.json: "motionSync" missing/); + await assert.rejects(edit((m) => ({ ...m, moc: "CAB-test:999" })), /model\.json: object not found: CAB-test:999/); + await assert.rejects(edit((m) => ({ ...m, prefab: m.moc })), /model\.json: CAB-test:\d+: a MonoBehaviour, not a GameObject/); + await assert.rejects(edit((m) => ({ ...m, bindings: {} })), /mtn_idle: no names for its bindings/); + await assert.rejects(edit((m) => ({ ...m, shaders: { [LIT]: m.shaders[LIT] } })), /mask material Mask: not drawn with Live2D Cubism\/Mask/); + await (await ModelSession.create({ gl, assets: modelFiles({ motionSync: true }) })).dispose(); await assert.rejects(ModelSession.create({ gl, assets: modelFiles(), motion: "mtn_none" }), /no motion "mtn_none"/); const s = await ModelSession.create({ gl, assets: modelFiles(), motion: "mtn_turn", expression: "exp_closed" }); assert.equal(s.motion, "mtn_turn"); diff --git a/tests/unit/live2d.test.mjs b/tests/unit/live2d.test.mjs index 7a27f7e..9b0fee1 100644 --- a/tests/unit/live2d.test.mjs +++ b/tests/unit/live2d.test.mjs @@ -59,20 +59,23 @@ test("parameter writes clamp to the moc3 range (override, add, multiply) and the assert.deepEqual([...parts], [1, 0.5]); }); -const clipData = (extra = {}) => ({ - clip: "mtn_test", stopTime: 2, loopTime: true, startTime: 0, cycleOffset: 0, +// [clip.json, model.json names] of a clip whose `bindings` are {path, class, attribute}: a MonoBehaviour binding +// names its script class, a Transform binding its type +const clipData = ({ bindings = [{ path: "Parameters/A", class: "CubismParameter", attribute: "Value" }, + { path: "Parameters/B", class: "CubismParameter", attribute: "Value" }], ...extra } = {}) => [{ + name: "mtn_test", stopTime: 2, loopTime: true, startTime: 0, cycleOffset: 0, events: [{ functionName: "InstanceId", intParameter: -7 }], dense: { curveCount: 0 }, - bindings: [{ path: "Parameters/A", class: "CubismParameter", attribute: "Value" }, - { path: "Parameters/B", class: "CubismParameter", attribute: "Value" }], + bindings: bindings.map((b) => (b.class === "Transform" ? { typeID: 4, class: "Transform", pptr: false } + : { typeID: 114, class: "MonoBehaviour", scriptClass: `Test||${b.class}`, pptr: false })), streamed: { curveCount: 1, frames: [[0, [[0, 0, 0, 1, 0]]], [1, [[0, 0, 0, 0, 1]]]] }, constant: [4], ...extra, -}); +}, bindings.map(({ path, attribute }) => ({ path, attribute }))]; test("a clip writes its streamed cubic keys and constants raw, wrapped past its length", () => { const P = new Live2DParameters(fakeCore(["A", "B"], [-10, -10], [10, 10], [0, 0])); - const c = new Live2DClip(clipData(), P); + const c = new Live2DClip(...clipData(), P); assert.equal(c.instanceId, -7); const v = new Float32Array(2); c.write(0.5, v); @@ -81,15 +84,15 @@ test("a clip writes its streamed cubic keys and constants raw, wrapped past its assert.equal(v[0], 1); // segment 1: constant 1 c.write(2.5, v); // past the length: 0.5 wrapped assert.equal(v[0], 0.5); - assert.equal(new Live2DClip(clipData({ loopTime: false }), P).localTime(2.5), 2); - assert.throws(() => new Live2DClip(clipData({ dense: { curveCount: 1 } }), P), /dense/); - assert.throws(() => new Live2DClip(clipData({ bindings: [{ path: "X", class: "Transform", attribute: "m_LocalPosition" }] }), P), + assert.equal(new Live2DClip(...clipData({ loopTime: false }), P).localTime(2.5), 2); + assert.throws(() => new Live2DClip(...clipData({ dense: { curveCount: 1 } }), P), /dense/); + assert.throws(() => new Live2DClip(...clipData({ bindings: [{ path: "X", class: "Transform", attribute: "m_LocalPosition" }] }), P), /binding/); }); test("a clip skips unbound curves and writes the model curves' controller fields", () => { const P = new Live2DParameters(fakeCore(["A", "B"], [-10, -10], [10, 10], [0, 0])); - const c = new Live2DClip(clipData({ + const c = new Live2DClip(...clipData({ bindings: [{ path: "", class: "CubismEyeBlinkController", attribute: "EyeOpening" }, { path: null, class: "CubismParameter", attribute: null }, { path: "Parameters/B", class: "CubismParameter", attribute: "Value" }, diff --git a/tests/unit/site-fixture.mjs b/tests/unit/site-fixture.mjs index 7044622..300b6e0 100644 --- a/tests/unit/site-fixture.mjs +++ b/tests/unit/site-fixture.mjs @@ -1,10 +1,11 @@ // Synthetic site pieces for the validator tests: content-addressed assets, stored as they are or gzip- / brotli-encoded -// (docs/data-format.md "File entries"), and a Live2D model whose files validate as a model (docs/data-format.md +// (docs/data-format.md "File entries"), and a Live2D model whose product validates as a model (docs/data-format.md // "Live2D models"). No game data. import crypto from "node:crypto"; import fs from "node:fs"; import path from "node:path"; import zlib from "node:zlib"; +import { ModelObjects, component, mb, node, ref } from "./live2d-objects.mjs"; export const LIT = "Live2D Cubism/Lit-URP-ADV-optimize", MASK = "Live2D Cubism/Mask"; export const STORY_LIGHTS = "_ADDITIONAL_LIGHTS_VERTEX"; @@ -12,7 +13,8 @@ export const STORY_LIGHTS = "_ADDITIONAL_LIGHTS_VERTEX"; export const COMPRESSIBLE = new Set(["json", "glsl", "moc3", "atlas", "skel", "bin", "wav", "glb"]); export const sha256 = (b) => crypto.createHash("sha256").update(b).digest("hex"); -export const bufferOf = (v) => (Buffer.isBuffer(v) ? v : Buffer.from(typeof v === "string" ? v : JSON.stringify(v))); +export const bufferOf = (v) => (Buffer.isBuffer(v) ? v : v instanceof Uint8Array ? Buffer.from(v.buffer, v.byteOffset, v.length) + : Buffer.from(typeof v === "string" ? v : JSON.stringify(v))); export const encode = (b, encoding) => (encoding === "gzip" ? zlib.gzipSync(b, { level: 9 }) : zlib.brotliCompressSync(b, { params: { [zlib.constants.BROTLI_PARAM_QUALITY]: 11 } })); @@ -44,62 +46,65 @@ export const png = (w, h) => { }; export const glsl = "#ifdef VERTEX\n#version 300 es\nvoid main() {}\n#endif\n#ifdef FRAGMENT\n#version 300 es\nvoid main() {}\n#endif\n"; -const node = (p, components = []) => ({ path: p, name: p.split("/").pop(), active: true, layer: 0, - localPosition: { x: 0, y: 0, z: 0 }, localRotation: { x: 0, y: 0, z: 0, w: 1 }, localScale: { x: 1, y: 1, z: 1 }, components }); -const mb = (cls, fields = {}) => ({ type: "MonoBehaviour", class: cls, ...fields }); -const material = (keywords) => ({ material: "Lit", shader: { shader: LIT }, keywords, - floats: { _SrcColor: 1, _DstColor: 10, _SrcAlpha: 1, _DstAlpha: 10, _Cull: 0 }, colors: {} }); -const maskMaterial = (cull) => ({ material: cull ? "MaskCulling" : "Mask", shader: { shader: MASK }, keywords: [], - floats: { _Cull: cull }, colors: {} }); +const PARSED = { properties: [], subShaders: [{ passes: [{ state: {} }] }] }; -// the model prefab: the root components the viewer reads, one clip with its fade motion, two drawables (the second -// masked when `masked`), and with `motionSync` the MotionSync controller with its CRI audio input -const prefab = (name, { masked, motionSync }) => ({ key: name, canvas: {}, nodes: [ - node(name, [ - mb("Live2DCharacter", { DefaultMotionName: "mtn_idle", DefaultExpressionName: "exp_idle", BasePosition: { x: 0, y: 0, z: 0 }, - BaseScale: 1, _motionList: [{ clip: "mtn_idle", stopTime: 1, loopTime: true, startTime: 0, cycleOffset: 0, - events: [{ functionName: "InstanceId", intParameter: -1 }] }], - _expressionList: ["exp_idle"] }), - mb("CubismFadeController", { CubismFadeMotionList: { MotionInstanceIds: [-1], CubismFadeMotionObjects: [{ MotionName: "mtn_idle" }] } }), - mb("CubismExpressionController", { UseLegacyBlendCalculation: 0, - ExpressionsList: { CubismExpressionObjects: [{ name: "exp_idle.exp3", Parameters: [] }] } }), - mb("CubismHarmonicMotionController", { BlendMode: 1 }), mb("CubismAutoEyeBlinkInput"), mb("CubismEyeBlinkController"), - mb("CubismMouthController"), mb("CubismRenderController"), - ...(motionSync ? [mb("CubismMotionSyncController", { _motionSyncData: { Settings: [{}] } }), - mb("Live2DMotionSyncCriAudioInput", { ListeningChannel: 0 })] : []), - ]), - node(`${name}/Drawables`), - ...[0, 1].map((i) => node(`${name}/Drawables/ArtMesh${i}`, [ - mb("CubismDrawable", { _unmanagedIndex: i }), - { type: "MeshRenderer", m_Materials: [material(masked && i === 1 ? ["CUBISM_MASK_ON"] : [])] }, - mb("CubismRenderer", { _mainTexture: { texture: "textures/texture_00.png", name: "texture_00", width: 4, height: 4, mipCount: 1 } }), - ])), -] }); - -// The logical files of a model (model.json at the root): -// format model.json format (2: with motionSync) -// motionSync the prefab's MotionSync controller (and model.json motionSync with format 2) -// storyLights the Lit variants with _ADDITIONAL_LIGHTS_VERTEX (the story renderer's at quality 4) are packed -// masked one drawable is masked (CUBISM_MASK_ON; the mask shader is packed) -export const modelFiles = (name = "m1", { format = 2, motionSync = false, storyLights = true, masked = true } = {}) => { +// The objects of a Live2D model (live2d-objects.mjs) and its model.json: +// motionSync the prefab's MotionSync controller with its CRI audio input (model.json motionSync) +// storyLights the Lit programs with _ADDITIONAL_LIGHTS_VERTEX (the story renderer's at quality 4) +// masked the second drawable is masked (CUBISM_MASK_ON; the mask shader's program draws the mask) +// The prefab has the root components the model viewer reads, one clip with its fade motion, one expression and two +// drawables. -> {objects: ModelObjects, model: model.json} +export const modelObjects = (name = "m1", { motionSync = false, storyLights = true, masked = true } = {}) => { + const o = new ModelObjects(`CAB-${name}`); const sets = [[], ...(masked ? [["CUBISM_MASK_ON"]] : [])]; - const litSets = sets.flatMap((k) => (storyLights ? [k, [...k, STORY_LIGHTS]] : [k])); - const shaders = [{ name: LIT, parsed: "lit.json", variants: litSets.map((k, i) => ({ file: `lit/${i}.glsl`, subShader: 0, pass: 0, keywords: k })) }, - ...(masked ? [{ name: MASK, parsed: "mask.json", variants: [{ file: "mask/0.glsl", subShader: 0, pass: 0, keywords: [] }] }] : [])]; - const files = { - "model.json": { format, ...(format >= 2 ? { motionSync } : {}), name, key: `Character/Live2D/g/${name}`, - moc3: `${name}.moc3`, prefab: `${name}.prefab.json`, textures: ["textures/texture_00.png"], - shaders: "shaders/shaders.json", resources: { cubismMask: maskMaterial(0), cubismMaskCulling: maskMaterial(1) } }, - [`${name}.moc3`]: Buffer.concat([Buffer.from("MOC3", "latin1"), Buffer.alloc(252, 5)]), - [`${name}.prefab.json`]: prefab(name, { masked, motionSync }), - "textures/texture_00.png": png(4, 4), - "shaders/shaders.json": shaders, - "shaders/lit.json": { properties: [], subShaders: [{ passes: [{ state: {} }] }] }, - }; - litSets.forEach((_, i) => { files[`shaders/lit/${i}.glsl`] = glsl; }); - if (masked) { files["shaders/mask.json"] = { properties: [], subShaders: [{ passes: [{ state: {} }] }] }; files["shaders/mask/0.glsl"] = glsl; } - return files; + const lit = o.shader(LIT, PARSED, sets.flatMap((k) => (storyLights ? [k, [...k, STORY_LIGHTS]] : [k])).map((k) => [k, glsl])); + const mask = o.shader(MASK, PARSED, masked ? [[[], glsl]] : []); + const material = (keywords) => o.material(keywords.length ? "LitMasked" : "Lit", { + shader: lit, keywords, floats: { _SrcColor: 1, _DstColor: 10, _SrcAlpha: 1, _DstAlpha: 10, _Cull: 0 } }); + const mats = sets.map(material); + const tex = o.texture("texture_00", { image: png(4, 4) }); + const clip = o.clip({ name: "mtn_idle", loopTime: true, events: [{ functionName: "InstanceId", intParameter: -1 }] }); + const fades = o.mono("CubismFadeMotionList", "common.fadeMotionList", { MotionInstanceIds: [-1], CubismFadeMotionObjects: [ + ref(o.mono("CubismFadeMotionData", "mtn_idle.fade", { MotionName: "mtn_idle", FadeInTime: 0.5, FadeOutTime: 0.5, MotionLength: 1, + ParameterIds: [], ParameterCurves: [], ParameterFadeInTimes: [], ParameterFadeOutTimes: [] }))] }); + const expressions = o.mono("CubismExpressionList", "common.expressionList", { CubismExpressionObjects: [ + ref(o.mono("CubismExpressionData", "exp_idle.exp3", { FadeInTime: 0.5, FadeOutTime: 0.5, Parameters: [] }))] }); + const prefab = o.prefab([ + node(name, [ + mb("Live2DCharacter", { DefaultMotionName: "mtn_idle", DefaultExpressionName: "exp_idle", BasePosition: { x: 0, y: 0, z: 0 }, + BaseScale: 1, _motionList: [ref(clip)], _expressionList: ["exp_idle"] }), + mb("CubismFadeController", { CubismFadeMotionList: ref(fades) }), + mb("CubismExpressionController", { UseLegacyBlendCalculation: 0, CurrentExpressionIndex: -1, ExpressionsList: ref(expressions) }), + mb("CubismHarmonicMotionController", { BlendMode: 1, ChannelTimescales: [1] }), mb("CubismAutoEyeBlinkInput"), + mb("CubismEyeBlinkController"), mb("CubismMouthController"), mb("CubismRenderController", { Opacity: 1 }), + ...(motionSync ? [mb("CubismMotionSyncController", { _motionSyncData: { Settings: [{}] } }), + mb("Live2DMotionSyncCriAudioInput", { ListeningChannel: 0 })] : []), + ]), + node(`${name}/Drawables`), + ...[0, 1].map((i) => node(`${name}/Drawables/ArtMesh${i}`, [ + mb("CubismDrawable", { _unmanagedIndex: i }), + component("MeshRenderer", { m_Materials: [ref(mats[masked && i === 1 ? 1 : 0])] }), + mb("CubismRenderer", { _localSortingOrder: 0, _color: { r: 1, g: 1, b: 1, a: 1 }, _mainTexture: ref(tex) }), + ])), + ]); + const resources = { cubismMask: o.material("Mask", { shader: mask, floats: { _Cull: 0 } }), + cubismMaskCulling: o.material("MaskCulling", { shader: mask, floats: { _Cull: 1 } }) }; + const moc = o.moc(Buffer.concat([Buffer.from("MOC3", "latin1"), Buffer.alloc(252, 5)]), name); + return { objects: o, model: o.modelJson({ prefab, moc, shaders: [lit, mask], resources, motionSync, name, group: "g" }) }; }; -// a model manifest (format 3) over stored files -export const modelManifest = (id, files) => ({ format: 3, id, key: `Character/Live2D/g/${id}`, model: { group: "g", textures: 1 }, files }); +// the extension of an object's file of `role` (as a product layout names its asset) +const roleExt = (role) => (role === "image" ? "png" : role.startsWith("program:") ? "glsl" : role === "blob:_bytes" ? "moc3" : "json"); + +// The product manifest (nnnotes.product/1 of live2d.model) of a model (modelObjects), its files and the files of its +// objects stored under dir/assets (`encoding` as putFiles) +export const modelProduct = (dir, { objects, model }, encoding = null) => ({ + schema: "nnnotes.product/1", product: "live2d.model", subject: model.key, + files: { "model.json": putAsset(dir, "model.json", model, encoding) }, + objects: Object.fromEntries(Object.entries(objects.objects).map(([id, o]) => [id, { class: o.class, name: o.name, + files: Object.fromEntries(Object.entries(o.files).map(([role, v]) => [role, putAsset(dir, `x.${roleExt(role)}`, v, encoding)])) }])), +}); + +// the sum of size over the files of a product manifest and of its objects +export const productSize = (man) => [man.files, ...Object.values(man.objects).map((o) => o.files)] + .reduce((n, files) => n + Object.values(files).reduce((k, f) => k + f.size, 0), 0); diff --git a/tests/unit/story-assets.test.mjs b/tests/unit/story-assets.test.mjs index 1ba3d1c..eecd329 100644 --- a/tests/unit/story-assets.test.mjs +++ b/tests/unit/story-assets.test.mjs @@ -1,7 +1,7 @@ -// Loading a story from a site (src/story/assets.js): manifest format /2 (the model manifests fetched and their files -// merged under live2d//, the site root from the manifest's `root`), format /1 (the models among the story's own -// files), and the Node openers of scripts/lib/headless.mjs (a manifest on disk with encoded assets, a story directory -// with its models directory). Synthetic sites only. +// Loading a story from a site (src/story/assets.js): the model manifests (product manifests of live2d.model) fetched, +// their model.json merged under live2d// and their objects into the store's object table, the site root from the +// manifest's `root`; and the Node openers of scripts/lib/headless.mjs (a manifest on disk with encoded assets, a +// story directory with its models directory). Synthetic sites only. import assert from "node:assert/strict"; import fs from "node:fs"; import os from "node:os"; @@ -9,10 +9,10 @@ import path from "node:path"; import { test } from "node:test"; import { gzipSync } from "node:zlib"; import { modelIndex } from "../../src/live2d/model.js"; -import { MODEL_MANIFEST_FORMATS, STORY_MANIFEST_FORMAT, STORY_MANIFEST_FORMATS, fetchStoryManifest, loadStoryStore } +import { MODEL_PRODUCT, STORY_MANIFEST_FORMAT, STORY_MANIFEST_FORMATS, fetchStoryManifest, loadStoryStore } from "../../src/story/assets.js"; import { nodeDecode, openStory, storyDirStore } from "../../scripts/lib/headless.mjs"; -import { putFiles } from "./site-fixture.mjs"; +import { modelObjects, modelProduct, putFiles, sha256 } from "./site-fixture.mjs"; const enc = new TextEncoder(); const ROOT = "https://example.test/site/"; @@ -32,14 +32,22 @@ const site = (files) => { return { fetch, requested }; }; -const modelJson = (name) => JSON.stringify({ format: 2, motionSync: false, name, moc3: `${name}.moc3`, prefab: `${name}.prefab.json`, - shaders: "shaders/shaders.json", resources: { cubismMask: {}, cubismMaskCulling: {} } }); +// a product manifest of model `id` whose model.json is `doc` (gzip-encoded with `gz`) and whose one object, a +// Texture2D `oid` (default: one of its own), has a json file: [manifest, {URL: contents}] +const product = (id, doc, { gz = false, oid = `CAB-${id}:1` } = {}) => { + const text = JSON.stringify(doc), z = new Uint8Array(gzipSync(enc.encode(text))), tj = JSON.stringify({ m_Name: oid }); + const asset = `assets/${sha256(Buffer.from(text))}.json`, tasset = `assets/${sha256(Buffer.from(tj))}.json`; + const man = { schema: "nnnotes.product/1", product: MODEL_PRODUCT, subject: `Character/Live2D/g/${id}/model/${id}`, + files: { "model.json": gz ? { asset: `${asset}.gz`, size: size(text), stored: z.byteLength } : { asset, size: size(text) } }, + objects: { [oid]: { class: "Texture2D", name: "t", files: { json: { asset: tasset, size: size(tj) } } } } }; + return [man, { [`${ROOT}${gz ? `${asset}.gz` : asset}`]: gz ? z : text, [`${ROOT}${tasset}`]: tj }]; +}; -// a site of one region's story (stories/tw/7.json, format /2) using models m1 (format 3, its model.json gzip-encoded) -// and m2 (format 2) +// a site of one region's story (stories/tw/7.json) using models m1 (its model.json gzip-encoded) and m2, which share +// the object CAB-s:1 const fixture = () => { const story = JSON.stringify({ advId: 7, models: { "Character/Live2D/g/m1/model/m1": "m1", "Character/Live2D/g/m2/model/m2": "m2" } }); - const m1 = modelJson("m1"), z = new Uint8Array(gzipSync(enc.encode(m1))), m2 = modelJson("m2"); + const [m1, f1] = product("m1", { name: "m1" }, { gz: true, oid: "CAB-s:1" }), [m2, f2] = product("m2", { name: "m2" }, { oid: "CAB-s:1" }); const man = { format: STORY_MANIFEST_FORMAT, advId: 7, root: "../../", language: "en", story: { advId: 7, playbackMode: 0 }, requires: { commands: [] }, models: { m1: "models/m1.json", m2: "models/m2.json" }, @@ -48,109 +56,73 @@ const fixture = () => { ja: { files: { "ui/languages.json": { asset: "assets/ja.json", size: 13 } } } }, }; const files = { - [`${ROOT}stories/tw/7.json`]: man, - [`${ROOT}models/m1.json`]: { format: 3, id: "m1", files: { - "model.json": { asset: "assets/m1.json.gz", size: size(m1), stored: z.byteLength }, - "m1.moc3": { asset: "assets/m1.moc3", size: 4 } } }, - [`${ROOT}models/m2.json`]: { format: 2, id: "m2", files: { "model.json": { asset: "assets/m2.json", size: size(m2) } } }, + [`${ROOT}stories/tw/7.json`]: man, [`${ROOT}models/m1.json`]: m1, [`${ROOT}models/m2.json`]: m2, ...f1, ...f2, [`${ROOT}assets/story.json`]: story, [`${ROOT}assets/en.json`]: '{"lang":"en"}', [`${ROOT}assets/ja.json`]: '{"lang":"ja"}', - [`${ROOT}assets/m1.json.gz`]: z, [`${ROOT}assets/m1.moc3`]: new Uint8Array([77, 79, 67, 51]), [`${ROOT}assets/m2.json`]: m2, }; - return { man, files, m1, z }; + return { man, files, m1, m2 }; }; -test("formats: story manifest /2 (current) and /1; model manifests 2 and 3", () => { +test("formats: story manifest /2; model manifests are products of live2d.model", () => { assert.equal(STORY_MANIFEST_FORMAT, "ournotes.story-manifest/2"); - assert.deepEqual(STORY_MANIFEST_FORMATS, ["ournotes.story-manifest/1", "ournotes.story-manifest/2"]); - assert.deepEqual(MODEL_MANIFEST_FORMATS, [2, 3]); + assert.deepEqual(STORY_MANIFEST_FORMATS, [STORY_MANIFEST_FORMAT]); + assert.equal(MODEL_PRODUCT, "live2d.model"); }); -test("format /2: the model manifests' files merged under live2d//, every path from the manifest's root", async () => { - const { files, m1, z } = fixture(); +test("the models' model.json merged under live2d// and their objects into one table, every path from the manifest's root", async () => { + const { files } = fixture(); const { fetch, requested } = site(files); const progress = []; const store = await loadStoryStore(`${ROOT}stories/tw/7.json`, { fetch, onProgress: (a, b) => progress.push([a, b]) }); - assert.deepEqual(store.list().sort(), ["live2d/m1/m1.moc3", "live2d/m1/model.json", "live2d/m2/model.json", "story.json", - "ui/languages.json"]); - assert.equal(store.text("live2d/m1/model.json"), m1); // decoded by the player - assert.deepEqual([...store.bytes("live2d/m1/m1.moc3")], [77, 79, 67, 51]); + assert.deepEqual(store.list().sort(), ["live2d/m1/model.json", "live2d/m2/model.json", "story.json", "ui/languages.json"]); + assert.deepEqual(store.json("live2d/m1/model.json"), { name: "m1" }); // decoded by the player + assert.deepEqual(store.objectJson("CAB-s:1"), { m_Name: "CAB-s:1" }); // one object of both models assert.equal(store.json("ui/languages.json").lang, "en"); assert.equal(store.info.loadedLanguage, "en"); assert.deepEqual(store.info.models, { m1: "models/m1.json", m2: "models/m2.json" }); assert.deepEqual(Object.keys(store.info.languages), ["en", "ja"]); assert.equal(store.info.files, undefined); + assert.equal(store.info.objects, undefined); // the region's manifest resolves against the site root (../../), not stories/ assert.ok(requested.includes(`${ROOT}models/m1.json`) && requested.includes(`${ROOT}models/m2.json`)); assert.ok(requested.filter((u) => u.includes("/assets/")).every((u) => u.startsWith(`${ROOT}assets/`)), requested.join("\n")); - const total = size(files[`${ROOT}assets/story.json`]) + 13 + z.byteLength + 4 + size(files[`${ROOT}assets/m2.json`]); - assert.deepEqual(progress.at(-1), [total, total]); - const m = modelIndex(store, "live2d/m1"); - assert.equal(m.moc3, "live2d/m1/m1.moc3"); - assert.equal(m.shaderDir, "live2d/m1/shaders"); + const shared = `${ROOT}assets/${sha256(Buffer.from(JSON.stringify({ m_Name: "CAB-s:1" })))}.json`; + assert.equal(requested.filter((u) => u === shared).length, 1); // a shared object's file fetched once + assert.equal(progress.at(-1)[0], progress.at(-1)[1]); // the other language; an explicit base assert.equal((await loadStoryStore(`${ROOT}stories/tw/7.json`, { fetch, lang: "ja" })).json("ui/languages.json").lang, "ja"); const moved = Object.fromEntries(Object.entries(files).map(([u, v]) => [u.startsWith(`${ROOT}stories/`) ? u.replace(ROOT, "https://cdn.example.test/m/") : u, v])); const b = await loadStoryStore("https://cdn.example.test/m/stories/tw/7.json", { ...site(moved), base: ROOT }); - assert.equal(b.text("live2d/m1/model.json"), m1); + assert.deepEqual(b.json("live2d/m1/model.json"), { name: "m1" }); }); -test("format /2: a model manifest missing or of another format, a model of another id, a story file under live2d/", async () => { +test("a model manifest missing or not a product of live2d.model, a model of another id, an object two models differ on", async () => { const url = `${ROOT}stories/tw/7.json`; - { - const { files } = fixture(); - delete files[`${ROOT}models/m2.json`]; - await assert.rejects(loadStoryStore(url, site(files)), /models\/m2\.json: HTTP 404/); - } - { - const { files } = fixture(); - files[`${ROOT}models/m2.json`].format = 4; - await assert.rejects(loadStoryStore(url, site(files)), /models\/m2\.json: not a model manifest \(format 2 or 3\)/); - } - { - const { files } = fixture(); - files[`${ROOT}models/m2.json`].id = "m3"; - await assert.rejects(loadStoryStore(url, site(files)), /model m3, the story names it m2/); - } - { - const { man, files } = fixture(); - man.files["live2d/m1/model.json"] = man.files["story.json"]; - await assert.rejects(loadStoryStore(url, site(files)), /live2d\/m1\/model\.json: live2d\/ holds the models' files/); - } - { - const { man, files } = fixture(); - delete man.root; - await assert.rejects(loadStoryStore(url, site(files)), /without root \/ models/); - } - { - const { man, files } = fixture(); - man.format = "ournotes.story-manifest/3"; - await assert.rejects(fetchStoryManifest(url, site(files)), /not a story manifest \(ournotes\.story-manifest\/1 or ournotes\.story-manifest\/2\)/); - } -}); - -test("format /1: the models' files among the story's, assets from the directory above the manifest's", async () => { - const story = JSON.stringify({ advId: 7, models: { "Character/Live2D/g/m1/model/m1": { dir: "live2d/m1", moc3: "m1.moc3", prefab: "m1.prefab.json" } } }); - const man = { format: "ournotes.story-manifest/1", advId: 7, language: "en", story: { advId: 7, playbackMode: 0 }, requires: { commands: [] }, - files: { "story.json": { asset: "assets/story.json", size: size(story) }, "live2d/m1/m1.moc3": { asset: "assets/m1.moc3", size: 4 } }, - languages: { en: { files: {} } } }; - const { fetch, requested } = site({ [`${ROOT}stories/7.json`]: man, [`${ROOT}assets/story.json`]: story, - [`${ROOT}assets/m1.moc3`]: new Uint8Array([77, 79, 67, 51]) }); - const store = await loadStoryStore(`${ROOT}stories/7.json`, { fetch }); - assert.deepEqual(store.list().sort(), ["live2d/m1/m1.moc3", "story.json"]); - assert.ok(!requested.some((u) => u.includes("/models/"))); - assert.equal(store.info.format, "ournotes.story-manifest/1"); + const rejects = async (edit, re) => { + const f = fixture(); + edit(f); + await assert.rejects(loadStoryStore(url, site(f.files)), re); + }; + await rejects((f) => { delete f.files[`${ROOT}models/m2.json`]; }, /models\/m2\.json: HTTP 404/); + await rejects((f) => { f.m2.product = "live2d.cubism"; }, /models\/m2\.json: not a model manifest \(nnnotes\.product\/1 of live2d\.model\)/); + await rejects((f) => { delete f.m2.objects; }, /models\/m2\.json: not a model manifest/); + await rejects((f) => { f.m2.subject = "Character/Live2D/g/m3/model/m3"; }, /model m3, the story names it m2/); + await rejects((f) => { f.m2.objects["CAB-s:1"].class = "Material"; }, /story 7: object CAB-s:1 differs between the manifests of its models/); + await rejects((f) => { f.man.files["live2d/m1/model.json"] = f.man.files["story.json"]; }, /live2d\/m1\/model\.json: live2d\/ holds the models' files/); + await rejects((f) => { delete f.man.root; }, /without root \/ models/); + const { man, files } = fixture(); + man.format = "ournotes.story-manifest/1"; + await assert.rejects(fetchStoryManifest(url, site(files)), /not a story manifest \(ournotes\.story-manifest\/2\)/); }); // ------------------------------------------------------------------------------------------------ Node openers -test("openStory: a manifest on disk with encoded assets (node:zlib); storyDirStore: a story directory with modelsDir", async () => { +test("openStory: a manifest on disk with encoded assets (node:zlib), its models' objects loaded", async () => { const dir = fs.mkdtempSync(path.join(os.tmpdir(), "ournotes-story-open-")); try { - const m1 = { format: 2, motionSync: false, name: "m1", moc3: "m1.moc3", prefab: "m1.prefab.json", shaders: "shaders/shaders.json", - resources: {}, pad: "p".repeat(200) }; - const mf = putFiles(dir, { "model.json": m1 }, "br"); - assert.ok(mf["model.json"].asset.endsWith(".json.br")); + const m = modelObjects("m1"); fs.mkdirSync(path.join(dir, "models")); - fs.writeFileSync(path.join(dir, "models", "m1.json"), JSON.stringify({ format: 3, id: "m1", files: mf })); + const mm = modelProduct(dir, m, "br"); + assert.ok(Object.values(mm.objects).some((o) => Object.values(o.files).some((f) => f.asset.endsWith(".json.br")))); + fs.writeFileSync(path.join(dir, "models", "m1.json"), JSON.stringify(mm)); const sf = putFiles(dir, { "story.json": { advId: 7, models: { "Character/Live2D/g/m1/model/m1": "m1" }, pad: "q".repeat(200) } }, "gzip"); const lf = putFiles(dir, { "ui/languages.json": { language: "en" } }); fs.mkdirSync(path.join(dir, "stories", "tw"), { recursive: true }); @@ -159,26 +131,28 @@ test("openStory: a manifest on disk with encoded assets (node:zlib); storyDirSto story: { playbackMode: 0 }, requires: { commands: [] }, models: { m1: "models/m1.json" }, files: sf, languages: { en: { files: lf } } })); const store = await openStory(manifest); - assert.deepEqual(store.json("live2d/m1/model.json"), m1); + assert.deepEqual(modelIndex(store, "live2d/m1"), m.model); + assert.equal(store.prefab(m.model.prefab).nodes[0].name, "m1"); assert.equal(store.json("story.json").advId, 7); assert.equal(typeof store.image, "function"); assert.equal(Buffer.from(await nodeDecode(gzipSync(Buffer.from("abc")), "gzip")).toString(), "abc"); await assert.rejects(nodeDecode(Buffer.from("x"), "zstd"), /unknown content encoding/); + } finally { fs.rmSync(dir, { recursive: true, force: true }); } +}); - // a story directory: DIR/story.json with modelsDir, the models in DIR/../live2d//; DIR's own live2d/ hidden +test("storyDirStore: a story directory with its models directory mounted at live2d/", () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), "ournotes-story-dir-")); + try { + // DIR/story.json with modelsDir, the models in DIR/../live2d//; DIR's own live2d/ hidden const sdir = path.join(dir, "out", "7"), mdir = path.join(dir, "out", "live2d", "m1"); fs.mkdirSync(path.join(sdir, "live2d", "stale"), { recursive: true }); - fs.mkdirSync(path.join(mdir, "shaders"), { recursive: true }); + fs.mkdirSync(mdir, { recursive: true }); fs.writeFileSync(path.join(sdir, "story.json"), JSON.stringify({ advId: 7, models: { "Character/Live2D/g/m1/model/m1": "m1" }, modelsDir: "../live2d" })); fs.writeFileSync(path.join(sdir, "live2d", "stale", "model.json"), "{}"); - fs.writeFileSync(path.join(mdir, "model.json"), JSON.stringify(m1)); - fs.writeFileSync(path.join(mdir, "shaders", "shaders.json"), "[]"); + fs.writeFileSync(path.join(mdir, "model.json"), JSON.stringify({ name: "m1" })); const ds = storyDirStore(sdir); assert.equal(ds.has("live2d/m1/model.json"), true); assert.equal(ds.has("live2d/stale/model.json"), false); - assert.deepEqual(ds.list(), ["live2d/m1/model.json", "live2d/m1/shaders/shaders.json", "story.json"]); - assert.equal(modelIndex(ds, "live2d/m1").shaderDir, "live2d/m1/shaders"); - const opened = await openStory(sdir); - assert.deepEqual(opened.json("live2d/m1/model.json"), m1); + assert.deepEqual(ds.list(), ["live2d/m1/model.json", "story.json"]); } finally { fs.rmSync(dir, { recursive: true, force: true }); } }); diff --git a/tests/unit/story-author.test.mjs b/tests/unit/story-author.test.mjs index ae1046f..fd93ab2 100644 --- a/tests/unit/story-author.test.mjs +++ b/tests/unit/story-author.test.mjs @@ -17,7 +17,8 @@ import { fileFetch, nodeDecode } from "../../scripts/lib/headless.mjs"; import { loadStoryStore } from "../../src/story/assets.js"; import { StoryPlayerCore } from "../../src/story/player-core.js"; import "../../src/story/features/index.js"; -import { modelFiles, modelManifest, png, putFiles } from "./site-fixture.mjs"; +import { ModelObjects, mb, node, ref } from "./live2d-objects.mjs"; +import { modelObjects, modelProduct, png, putFiles } from "./site-fixture.mjs"; const CLI = path.resolve("scripts/story-author.mjs"); const LT = String.fromCharCode(0xFF1C), WJ = String.fromCharCode(0x2060); @@ -85,11 +86,17 @@ test("the base story's stages with short ids and captions, its music and a stage }); test("a model's motions and expressions; the clip a motion plays at a position", () => { - const prefab = { nodes: [{ components: [{ class: "CubismModel" }, { class: "Live2DCharacter", DefaultMotionName: "mtn_idle01_C", - DefaultExpressionName: "exp_idle01", _motionList: [{ clip: "mtn_idle01_C" }, { clip: "mtn_nod01_L" }], _expressionList: ["exp_idle01"] }] }] }; - assert.deepEqual(modelActions(prefab), { motions: ["mtn_idle01_C", "mtn_nod01_L"], expressions: ["exp_idle01"], - defaultMotion: "mtn_idle01_C", defaultExpression: "exp_idle01" }); - assert.throws(() => modelActions({ nodes: [{ components: [] }] }), /Live2DCharacter/); + const o = new ModelObjects(); + const clips = ["mtn_idle01_C", "mtn_nod01_L"].map((name) => ref(o.clip({ name }))); + const actions = (components) => { + const prefab = o.prefab([node("m", components)]); + return modelActions(o.store({ prefab, moc: prefab, shaders: [], resources: {} }), { prefab }); + }; + assert.deepEqual(actions([mb("CubismModel"), mb("Live2DCharacter", { DefaultMotionName: "mtn_idle01_C", DefaultExpressionName: "exp_idle01", + _motionList: clips, _expressionList: ["exp_idle01"] })]), + { motions: ["mtn_idle01_C", "mtn_nod01_L"], expressions: ["exp_idle01"], defaultMotion: "mtn_idle01_C", + defaultExpression: "exp_idle01" }); + assert.throws(() => actions([]), /Live2DCharacter/); const m = ACTIONS.m_a.motions; assert.equal(resolveMotion("mtn_smile01", 3, m), "mtn_smile01_L"); assert.equal(resolveMotion("mtn_smile01", 9, m), "mtn_smile01_C"); // no _R: the centre one @@ -387,15 +394,15 @@ const makeSite = () => { const dir = fs.mkdtempSync(path.join(os.tmpdir(), "story-author-test-")); const models = []; for (const id of ["m_a", "m_b"]) { - const man = modelManifest(id, putFiles(dir, modelFiles(id))); + const man = modelProduct(dir, modelObjects(id)); fs.mkdirSync(path.join(dir, "models"), { recursive: true }); fs.writeFileSync(path.join(dir, "models", `${id}.json`), JSON.stringify(man)); - models.push({ id, group: "g", key: man.key, manifest: `models/${id}.json` }); + models.push({ id, group: "g", key: man.subject, manifest: `models/${id}.json` }); } fs.writeFileSync(path.join(dir, "models.json"), JSON.stringify({ format: 2, models })); const scene = { ...SCENE, stages: { "st/garden": { nodes: [{ components: [{ class: "AdvStage", _backgroundSprite: { texture: { texture: "textures/bg.png" } } }] }] }, "st/hall": {} } }; - const story = { ...STORY, models: { "Character/Live2D/g/m_a": "m_a" } }; + const story = { ...STORY, models: { "Character/Live2D/g/m_a/model/m_a": "m_a" } }; const files = putFiles(dir, { "story.json": story, "episode.json": EPISODE, "scene.json": scene, "ui/ui.json": { nodes: [] }, "frames.json": {}, "chat.json": {}, "textures/bg.png": png(8, 4), "textures/unused.png": png(2, 2), @@ -481,7 +488,7 @@ test("the commands without a browser: new, check, build; the built story loads", assert.equal(man.audio, true); const store = await loadStoryStore(url, { fetch: fileFetch, decode: nodeDecode }); assert.equal(store.json("episode.json").commands.filter((c) => c.cmd === "Talk").length, 2); - assert.deepEqual(store.json("story.json").models, { "Character/Live2D/g/m_a": "m_a", "Character/Live2D/g/m_b": "m_b" }); + assert.deepEqual(store.json("story.json").models, { "Character/Live2D/g/m_a/model/m_a": "m_a", "Character/Live2D/g/m_b/model/m_b": "m_b" }); assert.deepEqual(store.json("ui/fonts.json").fonts["F SDF"].browser.family.split(",")[0], "\"Noto Sans\""); assert.ok(store.has("live2d/m_b/model.json")); diff --git a/tests/unit/story-schema.test.mjs b/tests/unit/story-schema.test.mjs index ad7fc53..f2c4d4e 100644 --- a/tests/unit/story-schema.test.mjs +++ b/tests/unit/story-schema.test.mjs @@ -11,7 +11,7 @@ import path from "node:path"; import { test } from "node:test"; import { fileURLToPath } from "node:url"; import { compile } from "../../scripts/lib/json-schema.mjs"; -import { encode, modelFiles, modelManifest, putFiles, sha256 } from "./site-fixture.mjs"; +import { encode, modelObjects, modelProduct, productSize, putFiles, sha256 } from "./site-fixture.mjs"; const here = path.dirname(fileURLToPath(import.meta.url)); const script = path.join(here, "..", "..", "scripts", "validate-data.mjs"); @@ -98,7 +98,8 @@ const language = (lang, mode, field) => ({ format: "ournotes.story-language/1", // logical files -> a content-addressed site with stories/7.json and stories.json; `edit(parts)` changes the pieces // before they are stored: // common, groups, man the story's common files, language groups and manifest keys -// models model id -> its logical files (models/.json; the manifest's models list them all) +// models model id -> its objects and model.json (site-fixture modelObjects; its product manifest +// models/.json; the manifest's models list them all) // modelsIndex write models.json listing the models (true), or the ids of a list // encoding "gzip" | "br": every compressible asset encoded where that is smaller // where the manifest's path (stories/7.json; stories//7.json for a region's own) @@ -130,16 +131,15 @@ function buildSite(edit = () => {}) { const ids = Object.keys(parts.models).sort(), modelSize = {}; fs.mkdirSync(path.join(dir, "models"), { recursive: true }); for (const id of ids) { - const mm = modelManifest(id, entries(parts.models[id])); - modelSize[id] = Object.values(mm.files).reduce((n, f) => n + f.size, 0); + const mm = modelProduct(dir, parts.models[id], parts.encoding); + modelSize[id] = productSize(mm); fs.writeFileSync(path.join(dir, "models", `${id}.json`), JSON.stringify(mm)); } if (parts.modelsIndex) { const listed = Array.isArray(parts.modelsIndex) ? parts.modelsIndex : ids; fs.writeFileSync(path.join(dir, "models.json"), JSON.stringify({ format: 2, models: listed.map((id) => ({ id, manifest: `models/${id}.json` })) })); } - const sited = parts.man.format === "ournotes.story-manifest/2"; // format /1: no models of the site - const m = { ...(sited ? { models: Object.fromEntries(ids.map((id) => [id, `models/${id}.json`])) } : {}), ...parts.man, + const m = { models: Object.fromEntries(ids.map((id) => [id, `models/${id}.json`])), ...parts.man, files: entries(parts.common), languages: Object.fromEntries(Object.entries(parts.groups).map(([l, g]) => [l, { files: entries(g) }])) }; fs.mkdirSync(path.dirname(path.join(dir, parts.where)), { recursive: true }); @@ -147,7 +147,7 @@ function buildSite(edit = () => {}) { if (parts.index) { const size = (files) => Object.values(files).reduce((n, f) => n + f.size, 0); const entry = { id: "7", manifest: parts.where, audio: m.audio, audioFormat: m.audioFormat, fonts: m.fonts, ...m.story, - size: { common: size(m.files), ...(sited ? { models: ids.reduce((n, id) => n + modelSize[id], 0) } : {}), + size: { common: size(m.files), models: ids.reduce((n, id) => n + modelSize[id], 0), languages: Object.fromEntries(Object.entries(m.languages).map(([l, g]) => [l, size(g.files)])) } }; if (typeof parts.index === "function") parts.index(entry); fs.writeFileSync(path.join(dir, "stories.json"), JSON.stringify({ format: "ournotes.stories/1", language: "en", languages: ["ja", "en"], stories: [entry] })); @@ -508,7 +508,7 @@ test("fallback font assets: the fallback material of each text material (GetFall // a Live2D model m1 used by a Character row: its files in models/m1.json, story.json naming it, the story's shaders const ADDRESS = "Character/Live2D/g/m1/model/m1"; const withModel = (p, opts = {}) => { - p.models.m1 = modelFiles("m1", opts); + p.models.m1 = modelObjects("m1", opts); p.common["story.json"].models = { [ADDRESS]: "m1" }; const ep = p.common["episode.json"]; ep.commands.push({ i: 3, cmd: "Character", raw: 1, TargetName: "a", TargetAssetName: "g/m1/model/m1" }); @@ -543,7 +543,7 @@ test("models: the manifest's model manifests, valid as models; live2d/ holds the }, "manifest: models.m2: models/m2.json missing"); failsWith((p) => { withModel(p); - p.models.m0 = modelFiles("m0"); + p.models.m0 = modelObjects("m0"); p.man.models = { m1: "models/m1.json", m0: "models/m0.json" }; p.common["story.json"].models["Character/Live2D/g/m0/model/m0"] = "m0"; }, "manifest: models: keys not sorted"); @@ -553,12 +553,12 @@ test("models: the manifest's model manifests, valid as models; live2d/ holds the failsWith((p) => { withModel(p); p.common["live2d/m1/model.json"] = {}; }, "manifest: files: live2d/m1/model.json: live2d/ holds the files of the models"); failsWith((p) => { withModel(p); p.groups.en["live2d/m1/x.json"] = {}; }, "manifest: languages.en: live2d/m1/x.json: live2d/ holds"); // a model manifest that is not a valid model fails the model and the story - const bad = validate((p) => { withModel(p); p.models.m1["extra.json"] = {}; }); + const bad = validate((p) => { withModel(p); p.models.m1.objects.mono("CubismFadeMotionData", "extra"); }); assert.equal(bad.status, 1); assert.ok(bad.lines.includes("FAIL m1") && bad.lines.includes("FAIL 7"), bad.lines.join("\n")); - assert.ok(bad.lines.some((l) => l.startsWith("models/m1.json: not a valid model (1 problem): extra.json: listed but read by neither")), + assert.ok(bad.lines.some((l) => /^models\/m1\.json: not a valid model \(1 problem\): CAB-m1:\d+#json: listed but read by neither/.test(l)), bad.lines.join("\n")); - failsWith((p) => { withModel(p); p.models.m1["model.json"].motionSync = true; }, + failsWith((p) => { withModel(p); p.models.m1.model.motionSync = true; }, "model.json: motionSync true, the prefab's root has no MotionSync controller with a CRI audio input"); }); @@ -569,19 +569,15 @@ test("root: the site root seen from the manifest, ../../ for a region's own", () failsWith((p) => { p.man.root = "../../"; }, 'manifest: root "../../", the site root from stories/7.json is "../"'); }); -test("models: the variants the story renderer draws with at every quality", () => { - // model.json format 1: valid for the model viewer without the quality-4 variants, but not for a story - const r = validate((p) => withModel(p, { format: 1, storyLights: false })); +test("models: the programs the story renderer draws with at every quality", () => { + // a model without the Lit programs of quality 4 is not a valid model, and its stories fail + const r = validate((p) => withModel(p, { storyLights: false })); assert.equal(r.status, 1); - assert.ok(r.lines.includes("1/1 models valid") && r.lines.includes("0/1 stories valid"), r.lines.join("\n")); + assert.ok(r.lines.includes("FAIL m1") && r.lines.includes("0/1 models valid") && r.lines.includes("0/1 stories valid"), r.lines.join("\n")); const lit = "Live2D Cubism/Lit-URP-ADV-optimize"; - assert.ok(r.lines.includes(`models/m1.json: shaders/shaders.json: ${lit}: no variant for [_ADDITIONAL_LIGHTS_VERTEX] (the story renderer's keywords at quality 4)`), + assert.ok(r.lines.includes(`shader ${lit}: ${lit}: no variant for [_ADDITIONAL_LIGHTS_VERTEX] (the story renderer's keywords at quality 4)`), r.lines.join("\n")); assert.ok(r.lines.some((l) => l.includes("no variant for [CUBISM_MASK_ON _ADDITIONAL_LIGHTS_VERTEX]")), r.lines.join("\n")); - assert.equal(validate((p) => withModel(p, { format: 1 })).status, 0); - // model.json format 2 needs them for the model itself - const m = validate((p) => withModel(p, { storyLights: false })); - assert.ok(m.lines.includes("FAIL m1") && m.lines.includes("0/1 models valid"), m.lines.join("\n")); }); test("encoded assets: gzip and brotli, decoded and checked against their name, size and stored length", () => { @@ -650,57 +646,13 @@ test("the schemas: stored exactly with a .gz / .br asset, parts of four items, m assert.ok(S(withFiles(bad)).length > 0, JSON.stringify(bad)); assert.ok(S({ ...man, root: "./" }).some((e) => e.includes("/root"))); assert.ok(S({ ...man, models: { m1: "live2d/m1.json" } }).some((e) => e.includes("/models/m1"))); - const { root, models, ...legacy } = man; + const { root, models, ...rest } = man; assert.equal(root, "../"); - assert.ok(S({ ...legacy, models }).length > 0); // format /2 needs root and models - assert.ok(S({ ...legacy, root }).length > 0); - assert.deepEqual(S({ ...legacy, format: "ournotes.story-manifest/1" }), []); + assert.ok(S({ ...rest, models }).length > 0); // a manifest needs root and models + assert.ok(S({ ...rest, root }).length > 0); + assert.ok(S({ ...man, format: "ournotes.story-manifest/1" }).some((e) => e.includes("/format"))); assert.deepEqual(schema("story")(story), []); assert.deepEqual(schema("story")({ ...story, modelsDir: "../live2d" }), []); - assert.ok(schema("story")({ ...story, models: { [ADDRESS]: { dir: "live2d/m1" } } }).some((e) => e.includes("/models/"))); -}); - -// ------------------------------------------------------------------------------------------------ manifest format /1 -// a story of manifest format /1: the model's files among its common files (live2d/m1/), story.json models naming them -// by {dir, moc3, prefab}, the story's shaders holding the Lit and mask variants, scene.json resources -const withEmbeddedModel = (p) => { - withModel(p); - const files = p.models.m1; - delete p.models.m1; - p.man.format = "ournotes.story-manifest/1"; - delete p.man.root; - for (const k of ["m1.moc3", "m1.prefab.json", "textures/texture_00.png"]) p.common[`live2d/m1/${k}`] = files[k]; - for (const [k, v] of Object.entries(files)) if (k.startsWith("shaders/")) p.common[k] = v; - const sh = p.common["shaders/shaders.json"]; - p.common["shaders/shaders.json"] = [...sh, ...shaderIndex(["Adv/AlphaBlend"])]; - p.common["story.json"].models = { [ADDRESS]: { dir: "live2d/m1", moc3: "m1.moc3", prefab: "m1.prefab.json" } }; - p.common["scene.json"].resources = files["model.json"].resources; -}; - -test("manifest format /1: the models' files among the story's, drawn with the story's shaders", () => { - const { dir, man } = buildSite(withEmbeddedModel); - const index = JSON.parse(fs.readFileSync(path.join(dir, "stories.json"), "utf8")); - const story = JSON.parse(fs.readFileSync(path.join(dir, man.files["story.json"].asset), "utf8")); - fs.rmSync(dir, { recursive: true, force: true }); - assert.deepEqual(schema("story-manifest")(man), []); - assert.deepEqual(schema("stories")(index), []); - assert.deepEqual(Object.keys(index.stories[0].size), ["common", "languages"]); - assert.deepEqual(schema("story")(story), []); - const r = validate(withEmbeddedModel); - assert.equal(r.status, 0, r.lines.join("\n") + r.stderr); - assert.deepEqual(r.lines, ["1/1 stories valid"]); - assert.equal(validate((p) => { withEmbeddedModel(p); p.encoding = "gzip"; }).status, 0); - // the story's shaders hold a variant per drawable material - failsWith((p) => { - withEmbeddedModel(p); - const lit = p.common["shaders/shaders.json"][0]; - lit.variants = lit.variants.filter((v) => v.keywords.join(" ") !== "CUBISM_MASK_ON"); - }, "shaders/shaders.json: Live2D Cubism/Lit-URP-ADV-optimize: no variant for [CUBISM_MASK_ON]"); - failsWith((p) => { withEmbeddedModel(p); delete p.common["live2d/m1/m1.moc3"]; }, `story.json: models.${ADDRESS}: live2d/m1/m1.moc3 is not a common file`); - failsWith((p) => { withEmbeddedModel(p); delete p.common["live2d/m1/textures/texture_00.png"]; }, "live2d/m1/textures/texture_00.png: texture of a drawable not in the manifest"); - // each format names its models in its own form - failsWith((p) => { withEmbeddedModel(p); p.common["story.json"].models = { [ADDRESS]: "m1" }; }, `story.json: models.${ADDRESS}: not {dir, moc3, prefab}`); - failsWith((p) => { withModel(p); p.common["story.json"].models = { [ADDRESS]: { dir: "live2d/m1", moc3: "m1.moc3", prefab: "m1.prefab.json" } }; }, - `story.json: models.${ADDRESS}: not a model id`); - failsWith((p) => { withEmbeddedModel(p); p.index = (e) => { e.size.models = 0; }; }, "stories.json: size"); + assert.ok(schema("story")({ ...story, models: { [ADDRESS]: { dir: "live2d/m1", moc3: "m1.moc3", prefab: "m1.prefab.json" } } }) + .some((e) => e.includes("/models/"))); }); diff --git a/tests/unit/validate-manifest.test.mjs b/tests/unit/validate-manifest.test.mjs index 5964682..1600d55 100644 --- a/tests/unit/validate-manifest.test.mjs +++ b/tests/unit/validate-manifest.test.mjs @@ -1,7 +1,8 @@ // scripts/validate-data.mjs on the manifest keys of a site of several regions / languages and on model manifests: -// the schemas describe regions, language, titles, bandNames and the model facts; without charts.json the region -// manifests charts//.json are validated too; models.json facts must equal the manifest's. Synthetic sites -// (the manifests list no files, so the content checks fail; the lines checked here are the schema and index ones). +// the schemas describe regions, language, titles and bandNames, a model manifest is a product manifest of live2d.model; +// without charts.json the region manifests charts//.json are validated too; a models.json entry agrees +// with its model. Synthetic sites (the chart manifests list no files, so their content checks fail; the lines checked +// here are the schema and index ones). import assert from "node:assert/strict"; import { spawnSync } from "node:child_process"; import fs from "node:fs"; @@ -10,13 +11,16 @@ import path from "node:path"; import { test } from "node:test"; import { fileURLToPath } from "node:url"; import { compile } from "../../scripts/lib/json-schema.mjs"; +import { modelObjects, modelProduct, productSize } from "./site-fixture.mjs"; const here = path.dirname(fileURLToPath(import.meta.url)); const script = path.join(here, "..", "..", "scripts", "validate-data.mjs"); const schema = (name) => compile(JSON.parse(fs.readFileSync(path.join(here, "..", "..", "schema", `${name}.schema.json`), "utf8"))); +// validates a site of the JSON files `files` (path -> value; a function: called with the site directory first) const site = (files) => { const dir = fs.mkdtempSync(path.join(os.tmpdir(), "ournotes-site-")); + if (typeof files === "function") files = files(dir); for (const [p, v] of Object.entries(files)) { fs.mkdirSync(path.dirname(path.join(dir, p)), { recursive: true }); fs.writeFileSync(path.join(dir, p), JSON.stringify(v)); @@ -42,16 +46,6 @@ test("manifest schema: regions, language, titles and bandNames; a manifest witho assert.ok(S(manifest({}, { language: "" })).some((e) => e.includes("/chart/language"))); }); -test("model manifest schema: the model facts", () => { - const S = schema("model"); - const m = (model) => ({ format: 2, id: "m1", files: {}, ...(model === undefined ? {} : { model }) }); - assert.deepEqual(S(m()), []); - assert.deepEqual(S(m({ group: "003_adv", textures: 2, nodes: 525, extra: true, - canvas: { pixelsPerUnit: 6000, originX: 3000, originY: 4500, width: 6000, height: 9000, mocVersion: 5 } })), []); - assert.ok(S(m({ textures: -1 })).some((e) => e.includes("/model/textures"))); - assert.ok(S(m({ canvas: { mocVersion: 5.5 } })).some((e) => e.includes("/model/canvas/mocVersion"))); -}); - test("without charts.json the region manifests are validated too", () => { const r = site({ "charts/1_easy.json": manifest({ regions: ["tw"] }), "charts/kr/1_easy.json": manifest({ regions: ["K R"] }) }); assert.equal(r.status, 1); @@ -60,68 +54,69 @@ test("without charts.json the region manifests are validated too", () => { assert.equal(r.lines.filter((l) => l.startsWith("manifest:") && l.includes("/regions/0")).length, 1); }); -test("a models.json entry's key and facts equal its manifest's", () => { - const man = { format: 2, id: "m1", key: "Character/Live2D/g/m1", model: { group: "g", textures: 2 }, files: {} }; - const ok = site({ "models.json": { format: 2, models: [{ id: "m1", manifest: "models/m1.json", key: man.key, group: "g", textures: 2 }] }, - "models/m1.json": man }); - assert.ok(!ok.lines.some((l) => l.startsWith("models.json:")), ok.lines.join("\n")); - const bad = site({ "models.json": { format: 2, models: [{ id: "m1", manifest: "models/m1.json", key: "other", group: "h", textures: 2 }] }, - "models/m1.json": man }); +const KEY = "Character/Live2D/g/m1/model/m1"; + +test("a models.json entry agrees with its model: id, key and group of the subject; canvas and textures of model.json; bytes and files", () => { + const lines = (entry, edit = (m) => m) => site((dir) => { + const m = modelObjects("m1"), man = edit(modelProduct(dir, m)); + const files = Object.keys(man.files).length + Object.values(man.objects).reduce((n, o) => n + Object.keys(o.files).length, 0); + const full = { id: "m1", manifest: "models/m1.json", key: KEY, group: "g", canvas: m.model.canvas, textures: 1, + bytes: productSize(man), files }; + return { "models.json": { format: 2, models: [{ ...full, ...entry }] }, "models/m1.json": man }; + }); + const ok = lines({}); + assert.equal(ok.status, 0, ok.lines.join("\n")); + assert.deepEqual(ok.lines, ["1/1 models valid"]); + const bad = lines({ key: "other", group: "h", textures: 2, bytes: 1, files: 1, canvas: {} }); assert.equal(bad.status, 1); - assert.ok(bad.lines.includes("models.json: key differs from the manifest")); - assert.ok(bad.lines.includes("models.json: group differs from the manifest's model.group")); - assert.ok(!bad.lines.some((l) => l.includes("model.textures"))); + for (const l of [`models.json: key other, manifest subject ${KEY}`, "models.json: group h, the manifest subject's g", + "models.json: textures 2, model.json 1", "models.json: canvas differs from model.json canvas"]) + assert.ok(bad.lines.includes(l), `${l} in\n${bad.lines.join("\n")}`); + assert.ok(bad.lines.some((l) => l.startsWith("models.json: bytes 1, manifest files total")), bad.lines.join("\n")); + assert.ok(bad.lines.some((l) => l.startsWith("models.json: files 1, manifest")), bad.lines.join("\n")); }); -test("model display names: schemas, and equal in models.json and the manifest when either has them", () => { - const names = { ja: "エー", "zh-Hant": "A" }; - assert.deepEqual(schema("model")({ format: 2, id: "m1", files: {}, model: { character: 3, names, label: "A" } }), []); +test("models.json: display names of the models", () => { + const names = { ja: "ア", en: "A" }; assert.deepEqual(schema("models")({ format: 2, models: [{ id: "m1", manifest: "models/m1.json", character: 3, names, label: "A" }] }), []); - assert.ok(schema("model")({ format: 2, id: "m1", files: {}, model: { names: { ja: "" } } }).some((e) => e.includes("/model/names/ja"))); assert.ok(schema("models")({ format: 2, models: [{ id: "m1", manifest: "models/m1.json", character: 1.5 }] }) .some((e) => e.includes("/models/0/character"))); - const man = (model) => ({ format: 2, id: "m1", key: "k", model: { group: "g", ...model }, files: {} }); - const entry = (extra) => ({ format: 2, models: [{ id: "m1", manifest: "models/m1.json", ...extra }] }); - const lines = (index, m) => site({ "models.json": index, "models/m1.json": m }).lines.filter((l) => l.startsWith("models.json:")); - assert.deepEqual(lines(entry({ character: 3, names, label: "A" }), man({ character: 3, names, label: "A" })), []); - assert.deepEqual(lines(entry({}), man({})), []); - assert.deepEqual(lines(entry({ character: 3, label: "A" }), man({ names })), [ - "models.json: character differs from the manifest's model.character", - "models.json: names differs from the manifest's model.names", - "models.json: label differs from the manifest's model.label", - ]); + assert.ok(schema("models")({ format: 2, models: [{ id: "m1", manifest: "models/m1.json", names: { ja: "" } }] }) + .some((e) => e.includes("/models/0/names/ja"))); }); -test("chart and model manifests: formats 2 and 3, file entries stored as they are or encoded", () => { +test("chart manifests: formats 2 and 3, file entries stored as they are or encoded", () => { const a = "a".repeat(64); const files = { "live.json": { asset: `assets/${a}.json.gz`, size: 100, stored: 60 }, "x.glsl": { asset: `assets/${a}.glsl.br`, size: 100, stored: 40 }, "t.png": { asset: `assets/${a}.png`, size: 10 }, "n.json": { parts: [["k", `assets/${a}.json.gz`, 50, 20], ["l", `assets/${a}.json`, 5]], size: 63 } }; - for (const format of [2, 3]) { - assert.deepEqual(schema("manifest")(manifest({ format, files })), []); - assert.deepEqual(schema("model")({ format, id: "m1", files }), []); - } - for (const format of [1, 4]) { - assert.ok(schema("manifest")(manifest({ format })).some((e) => e.includes("/format"))); - assert.ok(schema("model")({ format, id: "m1", files: {} }).some((e) => e.includes("/format"))); - } + for (const format of [2, 3]) assert.deepEqual(schema("manifest")(manifest({ format, files })), []); + for (const format of [1, 4]) assert.ok(schema("manifest")(manifest({ format })).some((e) => e.includes("/format"))); assert.ok(schema("manifest")(manifest({ files: { "live.json": { asset: `assets/${a}.json.gz`, size: 100 } } })).length > 0); - assert.ok(schema("model")({ format: 3, id: "m1", files: { "n.json": { parts: [["k", `assets/${a}.json`, 5, 4]], size: 9 } } }).length > 0); - // model.json: format 2 states motionSync - const mj = { format: 2, motionSync: false, moc3: "m.moc3", prefab: "m.prefab.json", shaders: "shaders/shaders.json", - resources: { cubismMask: { shader: { shader: "Live2D Cubism/Mask" }, floats: { _Cull: 0 } }, - cubismMaskCulling: { shader: { shader: "Live2D Cubism/Mask" }, floats: { _Cull: 1 } } } }; - assert.deepEqual(schema("model-json")(mj), []); - const { motionSync, ...noSync } = mj; +}); + +test("product manifests and model.json: object tables of whole or encoded files; model.json format 3", () => { + const a = "a".repeat(64), S = schema("product"); + const man = { schema: "nnnotes.product/1", product: "live2d.model", subject: KEY, + files: { "model.json": { asset: `assets/${a}.json.gz`, size: 100, stored: 60 } }, + objects: { "CAB-x:-12": { class: "Texture2D", name: "t", files: { json: { asset: `assets/${a}.json`, size: 9 }, + image: { asset: `assets/${a}.png`, size: 10 } } } } }; + assert.deepEqual(S(man), []); + assert.ok(S({ ...man, schema: "nnnotes.product/2" }).some((e) => e.includes("/schema"))); + assert.ok(S({ ...man, objects: { "CAB-x": man.objects["CAB-x:-12"] } }).length > 0); // not an object id + assert.ok(S({ ...man, files: { "n.json": { parts: [["k", `assets/${a}.json`, 5]], size: 7 } } }).length > 0); // no parts + const { model } = modelObjects("m1"); + assert.deepEqual(schema("model-json")(model), []); + assert.ok(schema("model-json")({ ...model, format: 2 }).some((e) => e.includes("/format"))); + const { motionSync, ...noSync } = model; assert.equal(motionSync, false); - assert.deepEqual(schema("model-json")({ ...noSync, format: 1 }), []); assert.ok(schema("model-json")(noSync).length > 0); - assert.ok(schema("model-json")({ ...mj, format: 3 }).some((e) => e.includes("/format"))); + assert.ok(schema("model-json")({ ...model, moc: "m1.moc3" }).some((e) => e.includes("/moc"))); }); -test("a model manifest's id is its name without models.json", () => { - const r = site({ "models/m1.json": { format: 3, id: "m2", files: {} } }); +test("a model manifest's subject names the model of the manifest's name", () => { + const r = site((dir) => ({ "models/m2.json": modelProduct(dir, modelObjects("m1")) })); assert.equal(r.status, 1); - assert.ok(r.lines.includes("manifest: id m2, expected m1 (the manifest's name)"), r.lines.join("\n")); + assert.ok(r.lines.includes("manifest: subject: model m1, expected m2 (the manifest's name)"), r.lines.join("\n")); }); diff --git a/types/index.d.ts b/types/index.d.ts index 90ddd23..c5b22be 100644 --- a/types/index.d.ts +++ b/types/index.d.ts @@ -54,9 +54,35 @@ export interface FromManifestOptions { type PathMap = Map | Record; -/** The files of one chart, read synchronously once loaded. */ +/** A reference to an object in an object's JSON: the object `:`. */ +export interface ObjectRef { + $ref: { file: string; pathId: string; class?: string; name?: string }; +} + +/** An object of a store's object table: its class, name and files by role (`json`, `image`, `index`, + * `program:/`, `blob:`). */ +export interface ObjectEntry { + class: string; + name?: string | null; + files: Record; +} + +/** A Texture2D object: its size, mip count, colour space and sampler settings. */ +export interface TextureObject { + id: string; + name: string; + width: number; + height: number; + mipCount: number; + colorSpace: number; + settings: Record; +} + +/** The files of one chart or product and the objects it uses (a product manifest's object table), read synchronously + * once loaded. An object id is `:`. */ export class AssetStore { - constructor(files?: { text?: PathMap; bytes?: PathMap; info?: ManifestInfo | null }); + constructor(files?: { text?: PathMap; bytes?: PathMap; + objects?: Record; info?: ManifestInfo | null }); static fromManifest(url: string | URL, options?: FromManifestOptions): Promise; info: ManifestInfo | null; has(path: string): boolean; @@ -68,6 +94,27 @@ export class AssetStore { /** A PNG decoded with its texel values as stored, row 0 at the bottom. */ image(path: string): Promise; list(prefix?: string): string[]; + hasObject(id: string): boolean; + /** The object's class and name; throws for an unknown object or, with `cls`, an object of another class. */ + object(id: string, cls?: string | null): { class: string; name: string | null }; + /** The roles of the object's files. */ + roles(id: string): string[]; + objectText(id: string, role?: string): string; + /** The parsed file (cached: callers must not change it). */ + objectJson(id: string, role?: string): T; + /** A copy of the file's bytes. */ + objectBytes(id: string, role: string): Uint8Array; + /** The Texture2D's image, decoded as `image` decodes a PNG. */ + objectImage(id: string, role?: string): Promise; + /** The object id a reference names (null for null), checked to be an object of the store (of class `cls`). */ + ref(value: ObjectRef | null, cls?: string | null): string | null; + /** A GameObject's hierarchy (prefab.json). */ + prefab(id: string): { name: string; nodes: any[] }; + material(id: string): any; + clip(id: string): any; + /** A MonoBehaviour's fields and `$script`. */ + mono(id: string): any; + texture(id: string): TextureObject; } export interface ChartSessionOptions { diff --git a/types/live2d.d.ts b/types/live2d.d.ts index 4973ccc..396dbc5 100644 --- a/types/live2d.d.ts +++ b/types/live2d.d.ts @@ -8,26 +8,12 @@ export type { FromManifestOptions, ManifestInfo }; /** Game time per step: the ADV's 30 frames per second. */ export const MODEL_FRAME_RATE: 30; -/** A model manifest without its `files` map (`id`, `key`, `model`, ... as the site provides them). */ +/** A model manifest (a product manifest of live2d.model) without its `files` and `objects`. */ export interface ModelInfo { - format?: number; - id?: string; - key?: string; - /** Facts about the model: its group, the moc3 canvas, the atlas page and node counts, the character's names. */ - model?: { - group?: string; - canvas?: { pixelsPerUnit?: number; originX?: number; originY?: number; width?: number; height?: number; - mocVersion?: number; [key: string]: unknown }; - textures?: number; - nodes?: number; - /** The character's id in the game's master data. */ - character?: number; - /** The character's name per language (`ja`, `en`, `zh-Hant`, `zh-Hans`, `ko`). */ - names?: Record; - /** A display text: the character's name in the site's language. */ - label?: string; - [key: string]: unknown; - }; + schema?: "nnnotes.product/1"; + product?: "live2d.model"; + /** The model's key, Character/Live2D///model/. */ + subject?: string; [key: string]: unknown; } diff --git a/types/story.d.ts b/types/story.d.ts index 74556ac..e4685a6 100644 --- a/types/story.d.ts +++ b/types/story.d.ts @@ -183,18 +183,17 @@ export interface StoryManifestInfo { loadedLanguage?: StoryLanguage; audio?: boolean; requires?: { commands?: string[]; cubismCore?: boolean; motionSync?: boolean }; - /** Format /2: the site root, relative to the manifest. */ + /** The site root, relative to the manifest. */ root?: string; - /** Format /2: model id -> the path of its model manifest, relative to the site root. */ + /** Model id -> the path of its model manifest (a product manifest of live2d.model), relative to the site root. */ models?: Record; [key: string]: unknown; } -/** The current story manifest format. */ +/** The story manifest format. */ export const STORY_MANIFEST_FORMAT: "ournotes.story-manifest/2"; -/** The story manifest formats the player reads: /1 (the models' files among the story's) and /2 (the models in model - * manifests of the site). */ -export const STORY_MANIFEST_FORMATS: readonly ["ournotes.story-manifest/1", "ournotes.story-manifest/2"]; +/** The story manifest formats the player reads. */ +export const STORY_MANIFEST_FORMATS: readonly ["ournotes.story-manifest/2"]; /** Fetches a story manifest; refuses a story whose commands this player lacks. */ export function fetchStoryManifest(url: string | URL, options?: { fetch?: typeof globalThis.fetch; signal?: AbortSignal | null }): @@ -207,9 +206,9 @@ export interface LoadStoryOptions extends FromManifestOptions { manifest?: { url: string; manifest: any } | null; } -/** The AssetStore of one language of a story: the manifest's common files and that language's files; with format /2 - * also the files of the model manifests it lists, under live2d//. `base` defaults to the manifest's `root` - * (format /2) or the directory above the manifest's directory (format /1). */ +/** The AssetStore of one language of a story: the manifest's common files and that language's files, the model.json + * of every model manifest it lists under live2d// and their objects in one object table. `base` defaults to the + * manifest's `root`. */ export function loadStoryStore(url: string | URL, options?: LoadStoryOptions): Promise; /** A talk log entry (AdvTalkHelper.AddLogEntry): Talk, Location, subtitles and chat rows, also those the shortcut From 6f46864adeb9b6c74430b93a3886577305a93fec Mon Sep 17 00:00:00 2001 From: nichinichisou Date: Thu, 15 Oct 2026 13:13:59 +0800 Subject: [PATCH 02/18] feat(live2d): read model manifests of format 2 The model viewer reads a model manifest of format 2: one model whose clips, fade motions, expressions, materials, main textures and moc are inline in its prefab, with a shader directory and a model.json of format 1. legacy.js turns such a store into a product of live2d.model when the session loads it: every inline object becomes an object of an object table that the prefab's fields name by reference, equal materials one object, an atlas page a Texture2D with the descriptor's size, mip count, colour space and sampler settings, a shader of the directory a Shader with its GLES3 programs, and a shader the directory lacks (the mask shader of a model without masked drawables) a Shader without programs. The model.json of format 3 carries the clips' bindings and motionSync from the root's components, and `info` takes the product form with the model's key as its subject. The tests build one synthetic model in both forms (live2d-fixture.mjs) and check that the converted model plays as the product form does. Co-Authored-By: Claude Opus 5.5 --- README.en.md | 3 +- README.md | 2 +- docs/data-format.md | 24 ++- docs/live2d.md | 11 +- src/live2d/legacy.js | 130 ++++++++++++++++ src/live2d/model.js | 2 + src/live2d/session.js | 5 +- src/story/assets.js | 4 +- tests/unit/live2d-fixture.mjs | 236 +++++++++++++++++++++++++++++ tests/unit/live2d-legacy.test.mjs | 118 +++++++++++++++ tests/unit/live2d-session.test.mjs | 155 +------------------ types/live2d.d.ts | 7 +- 12 files changed, 532 insertions(+), 165 deletions(-) create mode 100644 src/live2d/legacy.js create mode 100644 tests/unit/live2d-fixture.mjs create mode 100644 tests/unit/live2d-legacy.test.mjs diff --git a/README.en.md b/README.en.md index 8e9974c..34d33be 100644 --- a/README.en.md +++ b/README.en.md @@ -197,7 +197,8 @@ nnnotes builds such a site from the user's own game files, for example: nnnotes web out/site --all --all-live2d --all-stories --player ``` -- The player also reads assets stored as they are (not encoded). +- The player also reads assets stored as they are (not encoded), and the model viewer model manifests of format 2 + ([docs/data-format.md](docs/data-format.md#model-manifests-of-format-2)). - Hosting (paths, CORS, encoded assets, caching): [docs/embedding.md](docs/embedding.md#hosting-the-data). A site encoded with brotli plays in Chromium-based browsers only when its `.br` assets are served with `Content-Encoding: br`. diff --git a/README.md b/README.md index 507fba9..e4437a0 100644 --- a/README.md +++ b/README.md @@ -152,7 +152,7 @@ nnnotes 从使用者自己的游戏文件生成这样的站点，例如： nnnotes web out/site --all --all-live2d --all-stories --player ``` -- 播放器也读取未编码存放的资源。 +- 播放器也读取未编码存放的资源；模型查看器也读取格式 2 的模型清单（[说明](docs/data-format.md#model-manifests-of-format-2)）。 - 托管方式（路径、CORS、编码资源、缓存）：[docs/embedding.md](docs/embedding.md#hosting-the-data)。以 brotli 编码的站点要在 Chromium 系浏览器中播放，需以 `Content-Encoding: br` 提供 `.br` 资源。 - 校验：`npm run validate-data -- <站点目录>` 按数据格式与 `schema/` 检查谱面、模型与剧情，包括剧情与其模型之间的对应。 diff --git a/docs/data-format.md b/docs/data-format.md index 98d9ae8..23c0a23 100644 --- a/docs/data-format.md +++ b/docs/data-format.md @@ -526,7 +526,7 @@ paths of a manifest against `models/..`, the site root, as for charts. | Key | Meaning | |---|---| -| `schema`, `product` | `"nnnotes.product/1"` and `"live2d.model"`; the viewer and the story player refuse others. | +| `schema`, `product` | `"nnnotes.product/1"` and `"live2d.model"`; the story player refuses others, and the viewer reads others only as a [model manifest of format 2](#model-manifests-of-format-2). | | `subject` | The model's key, `Character/Live2D///model/`; `` is the manifest's ``. | | `files` | The product's own files: `model.json` alone. A [file entry](#file-entries), whole or encoded. | | `objects` | The objects the model uses: object id → its `class`, `name` and `files` by role, each a file entry, whole or encoded. | @@ -629,6 +629,28 @@ only the GLES3 programs (platform `gles3`, type `GLES3`, subshader 0, pass 0) th qualities. No player enables `_ADDITIONAL_LIGHTS`, so the programs with it are not read. - "Live2D Cubism/Mask": its program without keywords, when some drawable is masked (a material with `CUBISM_MASK_ON`). +### Model manifests of format 2 + +The model viewer also reads a model manifest of format 2, `{ "format": 2, "id": "…", "key": "…", "model": { … }, +"files": { … } }` (no `schema`), whose `files` (file entries as in a [chart manifest](#file-entries)) hold one model +with its objects inline: + +| Logical file | Content | +|---|---| +| `model.json` | `{ "format": 1, name, key, moc3, prefab, textures, canvas, shaders, resources }`: the paths of the moc3, the prefab and the shader index, and in `resources` the mask materials `cubismMask` and `cubismMaskCulling` inline (`{ material, shader: { shader: }, keywords, floats, colors, … }`). | +| `.moc3` | The moc3. | +| `.prefab.json` | `{ key, nodes, canvas }`: the model's node list; a component is `{ type, class, …its fields }` (`class`: a MonoBehaviour's script class). Inline in it: the CubismMoc (`{ asset: "CubismMoc", name }`), the clips of `_motionList` (`{ clip: , bindings: [{ path, typeID, class, attribute }], … }` and the clip's data as in an AnimationClip's `json`), the fade motion list, the expression list and their entries (`{ asset: , name, …their fields }`), the drawables' materials and their main textures (a [texture descriptor](#texture-descriptors), its path relative to the prefab's directory). | +| `textures/…` | The atlas pages. | +| `shaders/shaders.json`, `shaders/…` | The shaders in the layout of a [shader directory](#shaders), the mask shader only when some drawable is masked. | + +The viewer reads it as a model manifest of `live2d.model`: each inline object becomes an object `:` (`` +the model's name) that the fields name by reference, equal materials one object, an atlas page a Texture2D whose +fields are the descriptor's (`m_Width`, `m_Height`, `m_MipCount`, `m_ColorSpace`, `m_TextureSettings`), a shader of the +directory a Shader whose programs are its variants' files, and a shader the directory lacks a Shader without programs. +The `model.json` of format 3 names them, with `bindings` from the clips' bindings and `motionSync` from the root's +components; `info` is `{ schema: "nnnotes.product/1", product: "live2d.model", subject: }`. The story player +reads model manifests of `live2d.model` only. + ## Conventions - **Unity names.** Serialized fields keep their Unity names: engine fields with `m_` (`m_Enabled`, `m_Materials`), diff --git a/docs/live2d.md b/docs/live2d.md index ca8b324..6bffb70 100644 --- a/docs/live2d.md +++ b/docs/live2d.md @@ -81,10 +81,11 @@ its shadow root; the div fills the host, and a host without a height gets a box its width. It resolves once the model is shown, after the `ready` event, and rejects when loading fails (after an `error` event). -Options: `src` (a model manifest URL) or `assets` (an `AssetStore`); `motion`, `expression`, `loop`, `physics`, -`breath`, `seed` as for `ModelSession`; `paused` (start paused); `pixelRatio` (device pixels per CSS pixel, default -`devicePixelRatio`); `signal` (an `AbortSignal` that cancels the loading); `on` (`{type: listener}` added before -loading starts). +Options: `src` (a model manifest URL: a product manifest of `live2d.model` or a +[model manifest of format 2](data-format.md#model-manifests-of-format-2)) or `assets` (an `AssetStore`); `motion`, +`expression`, `loop`, `physics`, `breath`, `seed` as for `ModelSession`; `paused` (start paused); `pixelRatio` (device +pixels per CSS pixel, default `devicePixelRatio`); `signal` (an `AbortSignal` that cancels the loading); `on` +(`{type: listener}` added before loading starts). | | | |---|---| @@ -101,7 +102,7 @@ loading starts). | `physics`, `breath` | Read / write. | | `hasPhysics` | The model has physics (a `CubismPhysicsController`); without it `physics` stays `false`. | | `play()`, `pause()`, `paused` | While paused no frame advances; the last frame stays shown. | -| `info` | The manifest without `files`. | +| `info` | The manifest without `files` and `objects`: `{ schema, product, subject }` (for a [model manifest of format 2](data-format.md#model-manifests-of-format-2), that form with its key as `subject`). | | `session`, `root`, `canvas` | The `ModelSession`, the appended div and the canvas. | | `dispose()` | Stops the player, releases the WebGL context and the model and removes the div. | diff --git a/src/live2d/legacy.js b/src/live2d/legacy.js new file mode 100644 index 0000000..434d8e9 --- /dev/null +++ b/src/live2d/legacy.js @@ -0,0 +1,130 @@ +import { AssetStore, PRODUCT_MANIFEST } from "../data/assets.js"; +import { join } from "../engine/core.js"; +import { componentClass } from "../engine/prefab.js"; +import { MODEL_JSON_FORMAT, MODEL_PRODUCT } from "./model.js"; + +// Live2D models of the site layout with inline objects (docs/data-format.md "Model manifests of format 2"): a model +// manifest of format 2 (no `schema`) whose files are the moc3, the prefab with the clips, the fade motions, the +// expressions and the materials inline, the atlas pages and a shader directory, indexed by a model.json of format 1. +// legacyModel() reads such a store as a product of live2d.model: the same data as an object table and a model.json of +// format 3, so the viewer reads both through one path. + +export const LEGACY_MODEL_MANIFEST_FORMAT = 2, LEGACY_MODEL_JSON_FORMAT = 1; + +// whether `info` (a store's manifest without its files) is a model manifest of format 2 +export const isLegacyModel = (info) => !!info && info.schema === undefined && info.format === LEGACY_MODEL_MANIFEST_FORMAT; + +const dirOf = (p) => p.slice(0, p.lastIndexOf("/") + 1); + +// The store of a model manifest of format 2 read as a product of live2d.model (a new AssetStore): +// model.json format 3 of the objects below; `motionSync` from the root's components +// objects ":": the prefab's GameObject, the CubismMoc (its moc3 as blob:_bytes), a Texture2D per +// atlas page the drawables use (the descriptor's fields as its typetree, the PNG as its image), a +// Material per distinct inline material (its shader a reference), the Shaders of the shader directory +// (parsed form, variant index, GLES3 programs) and of the materials, the clips (model.json bindings: the paths and attributes +// the inline clips name), the fade motion list and its fade motions, the expression list and its +// expressions +// info {schema: nnnotes.product/1, product: live2d.model, subject: the model's key} +export const legacyModel = (store) => { + const m = store.json("model.json"); + if (!m || m.format !== LEGACY_MODEL_JSON_FORMAT) + throw new Error(`model.json: format ${m && m.format} is not supported (${LEGACY_MODEL_JSON_FORMAT} expected)`); + const objects = {}; + let next = 1; + const add = (cls, name, files) => { + const id = `${m.name}:${next++}`; + objects[id] = { class: cls, name, files }; + return id; + }; + const ref = (id) => { + if (id === null) return null; + const i = id.lastIndexOf(":"); + return { $ref: { file: id.slice(0, i), pathId: id.slice(i + 1) } }; + }; + const json = (v) => JSON.stringify(v); + + // the shader directory: per shader its parsed form, its variant index (a role per GLES3 program) and the programs + const shaderDir = dirOf(m.shaders), shaders = {}; + for (const r of store.json(m.shaders)) { + const files = { json: store.text(shaderDir + r.parsed) }; + const variants = r.variants.map((v) => { + const role = `program:${v.platform}/${v.file.slice(v.file.lastIndexOf("/") + 1).replace(/\.glsl$/, "")}`; + if (files[role] !== undefined) throw new Error(`${m.shaders}: ${r.name}: two variants of ${role}`); + files[role] = store.text(shaderDir + v.file); + return { ...v, role }; + }); + files.index = json({ name: r.name, variants }); + shaders[r.name] = add("Shader", r.name, files); + } + // an inline material; equal ones are one object. A shader the directory lacks (the mask shader of a model without + // masked drawables) is a Shader object with its name and no programs. + const materials = new Map(); + const material = (mat) => { + const name = mat.shader && mat.shader.shader; + if (typeof name !== "string") throw new Error(`material ${mat.material}: no shader`); + if (!shaders[name]) shaders[name] = add("Shader", name, { json: json({ name }), index: json({ name, variants: [] }) }); + const doc = json({ ...mat, shader: ref(shaders[name]) }); + if (!materials.has(doc)) materials.set(doc, add("Material", mat.material, { json: doc })); + return materials.get(doc); + }; + // a texture descriptor (its path relative to the prefab's directory); one object per path + const textures = new Map(); + const texture = (d) => { + const path = join(dirOf(m.prefab), d.texture); + if (!textures.has(path)) { + textures.set(path, add("Texture2D", d.name, { + json: json({ m_Name: d.name, m_Width: d.width, m_Height: d.height, m_TextureFormat: d.format, + m_MipCount: d.mipCount, m_ColorSpace: d.colorSpace, m_TextureSettings: d.settings }), + image: store.bytes(path), + })); + } + return textures.get(path); + }; + // an inline MonoBehaviour asset ({asset: its class, name, ...its fields}) with the fields `refs` replaces + const mono = ({ asset, name, ...fields }, refs = {}) => + add("MonoBehaviour", fields.m_Name ?? name, { json: json({ $script: `||${asset}`, ...fields, ...refs }) }); + // an inline clip; a binding of a MonoBehaviour (type id 114) names its class as its script class + const bindings = {}; + const clip = ({ clip: name, bindings: bs, ...fields }) => { + const stored = bs.map(({ path: _path, attribute: _attribute, class: cls, ...b }) => (b.typeID === 114 + ? { ...b, class: "MonoBehaviour", scriptClass: cls ? `||${cls}` : null } : { ...b, class: cls })); + const id = add("AnimationClip", name, { json: json({ name, ...fields, bindings: stored }) }); + bindings[id] = bs.map((b) => ({ path: b.path, attribute: b.attribute })); + return id; + }; + const moc = add("MonoBehaviour", m.name, { json: json({ $script: "||CubismMoc", m_Name: m.name }), + "blob:_bytes": store.bytes(m.moc3) }); + + // the fields of a component with its inline objects replaced by references + const fields = (cls, f) => { + switch (cls) { + case "CubismModel": return { ...f, _moc: ref(moc) }; + case "Live2DCharacter": return { ...f, _motionList: f._motionList.map((c) => ref(clip(c))) }; + case "CubismFadeController": { + const l = f.CubismFadeMotionList; + return { ...f, CubismFadeMotionList: ref(mono(l, { CubismFadeMotionObjects: l.CubismFadeMotionObjects.map((o) => ref(mono(o))) })) }; + } + case "CubismExpressionController": { + const l = f.ExpressionsList; + return { ...f, ExpressionsList: l === null ? null + : ref(mono(l, { CubismExpressionObjects: l.CubismExpressionObjects.map((o) => ref(mono(o))) })) }; + } + case "CubismRenderer": return { ...f, _mainTexture: f._mainTexture === null ? null : ref(texture(f._mainTexture)) }; + case "MeshRenderer": return { ...f, m_Materials: f.m_Materials.map((x) => ref(material(x))) }; + default: return f; + } + }; + const { nodes } = store.json(m.prefab); + const converted = nodes.map((n) => ({ ...n, components: n.components.map(({ type, class: cls, ...f }) => + (cls === undefined ? { type, fields: fields(type, f) } : { type, class: cls, fields: fields(cls, f) })) })); + const prefab = add("GameObject", m.name, { json: json({ name: m.name, nodes: converted }) }); + const root = new Set(converted[0].components.map(componentClass)); + + const model = { + format: MODEL_JSON_FORMAT, name: m.name, key: m.key, prefab, moc, canvas: m.canvas, textures: [...textures.values()], + shaders, resources: { cubismMask: material(m.resources.cubismMask), cubismMaskCulling: material(m.resources.cubismMaskCulling) }, + motionSync: root.has("CubismMotionSyncController") && root.has("Live2DMotionSyncCriAudioInput"), bindings, + }; + return new AssetStore({ text: { "model.json": json(model) }, objects, + info: { schema: PRODUCT_MANIFEST, product: MODEL_PRODUCT, subject: m.key } }); +}; diff --git a/src/live2d/model.js b/src/live2d/model.js index acc4e5a..af59de9 100644 --- a/src/live2d/model.js +++ b/src/live2d/model.js @@ -3,6 +3,8 @@ // are in the store's object table. export const MODEL_JSON_FORMAT = 3; +// the product of a model manifest (a product manifest, nnnotes.product/1) +export const MODEL_PRODUCT = "live2d.model"; // The model.json at `dir`/model.json of `assets` (dir "" for the store's root), its objects checked: // prefab the model's root GameObject (its prefab.json) diff --git a/src/live2d/session.js b/src/live2d/session.js index 12a4126..1b14ee0 100644 --- a/src/live2d/session.js +++ b/src/live2d/session.js @@ -8,6 +8,7 @@ import { GLTex } from "../engine/texture.js"; import { Live2DCharacter } from "./character.js"; import { cubismCore } from "./cubism.js"; import { Live2DDrawing } from "./drawing.js"; +import { isLegacyModel, legacyModel } from "./legacy.js"; import { modelIndex } from "./model.js"; // ModelSession: one Live2D model of the game's ADV, running idle as the story shows a character (auto eye blink, @@ -60,7 +61,8 @@ export class ModelSession { // opts: // gl WebGL2RenderingContext (required); used by this session alone while it lives - // assets AssetStore of the model (required): model.json and the objects it names + // assets AssetStore of the model (required): model.json and the objects it names, or the files of a model + // manifest of format 2 (read through legacy.js; `info` is then the product form) // motion, expression shown first (default: the model's default motion / expression); loop: replay that motion // physics, breath CubismPhysicsController and the harmonic breath on (default true) // seed seed of the auto eye blink's random intervals (default: from the clock) @@ -82,6 +84,7 @@ export class ModelSession { width, height, onMotion = null } = {}) { if (!gl) throw new Error("ModelSession: a WebGL2 context is required"); if (!assets) throw new Error("ModelSession: an AssetStore is required"); + if (isLegacyModel(assets.info)) assets = legacyModel(assets); await cubismCore(); bindAssets(gl, assets); this.gl = gl; this.assets = assets; diff --git a/src/story/assets.js b/src/story/assets.js index 4059a0f..ab48c6a 100644 --- a/src/story/assets.js +++ b/src/story/assets.js @@ -1,4 +1,5 @@ import { AssetStore, PRODUCT_MANIFEST, decodeAsset } from "../data/assets.js"; +import { MODEL_PRODUCT } from "../live2d/model.js"; import { ADV_PLAYBACK_MODE, StoryCommandError } from "./interfaces.js"; import { StoryPlayerCore } from "./player-core.js"; import { simpleUnsupportedCommands } from "./simple/validator.js"; @@ -9,8 +10,7 @@ import { simpleUnsupportedCommands } from "./simple/validator.js"; // the story manifest format, the formats this player reads export const STORY_MANIFEST_FORMAT = "ournotes.story-manifest/2"; export const STORY_MANIFEST_FORMATS = Object.freeze([STORY_MANIFEST_FORMAT]); -// the product of a model manifest (a product manifest, nnnotes.product/1) -export const MODEL_PRODUCT = "live2d.model"; +export { MODEL_PRODUCT }; // Checks that `mm` (fetched from `where`) is the model manifest of model `id`: a product manifest of live2d.model whose // subject is a model key Character/Live2D///model/. diff --git a/tests/unit/live2d-fixture.mjs b/tests/unit/live2d-fixture.mjs new file mode 100644 index 0000000..07a1304 --- /dev/null +++ b/tests/unit/live2d-fixture.mjs @@ -0,0 +1,236 @@ +// A synthetic Live2D model for the model viewer's tests: a stand-in for Live2D Cubism Core (a few parameters, three +// drawables whose vertices follow ParamAngleX, one of them masked) and one model in two forms, in memory: +// modelFiles() the product form: the model's objects (a prefab with the components the runtime reads, clips, fade +// motions, expressions, materials, a texture, shaders with their GLES3 programs) and model.json +// format 3 (live2d-objects.mjs) +// legacyFiles() the files of a model manifest of format 2: the same model with its clips, fade motions, +// expressions and materials inline in the prefab, a shader directory and model.json format 1 +import { AssetStore } from "../../src/data/assets.js"; +import { ModelObjects, component, mb, node, ref } from "./live2d-objects.mjs"; +import { headlessImages } from "../../scripts/lib/headless.mjs"; + +export const IDS = ["ParamAngleX", "ParamEyeLOpen", "ParamMouthOpenY", "ParamBreath"]; +const MIN = [-30, 0, 0, 0], MAX = [30, 1, 1, 1], DEF = [0, 1, 0, 0]; +export const DRAWABLES = ["ArtMesh0", "ArtMesh1", "ArtMesh2"]; +const QUAD = [-0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1]; + +// the stand-in Core: Moc / Model with parameter, part and drawable arrays; update() moves every drawable by +// ParamAngleX / 300 in x (and reports its vertices as changed on every update, as a model in motion does) and sets +// drawable 1's opacity to ParamEyeLOpen +export const fakeCore = () => { + const made = []; + const core = { + Version: { csmGetVersion: () => 0x05010000, csmGetLatestMocVersion: () => 5 }, + Moc: { fromArrayBuffer: () => ({ _release() { core.releasedMocs++; } }) }, + Model: { + fromMoc() { + const n = DRAWABLES.length; + const d = { + count: n, ids: DRAWABLES.slice(), constantFlags: new Uint8Array([4, 4, 4]), dynamicFlags: new Uint8Array([1, 1, 1]), + opacities: new Float32Array([1, 1, 1]), renderOrders: new Int32Array([2, 0, 1]), maskCounts: new Int32Array([0, 0, 1]), + masks: [new Int32Array(0), new Int32Array(0), new Int32Array([0])], + vertexPositions: DRAWABLES.map((_, i) => Float32Array.from(QUAD, (v, k) => v + (k % 2 ? 0 : i * 0.2))), + vertexUvs: DRAWABLES.map(() => new Float32Array([0, 0, 1, 0, 1, 1, 0, 1])), + indices: DRAWABLES.map(() => new Uint16Array([0, 1, 2, 0, 2, 3])), + multiplyColors: new Float32Array(4 * n).fill(1), screenColors: new Float32Array([0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1]), + resetDynamicFlags() { for (let i = 0; i < n; i++) this.dynamicFlags[i] &= 1; }, + }; + const m = { + parameters: { count: IDS.length, ids: IDS.slice(), minimumValues: Float32Array.from(MIN), + maximumValues: Float32Array.from(MAX), defaultValues: Float32Array.from(DEF), values: Float32Array.from(DEF) }, + parts: { count: 1, ids: ["Part0"], opacities: new Float32Array([1]) }, + drawables: d, + canvasinfo: { CanvasWidth: 600, CanvasHeight: 900, CanvasOriginX: 300, CanvasOriginY: 450, PixelsPerUnit: 600 }, + updates: 0, + update() { + this.updates++; + const ax = this.parameters.values[0] / 300; + for (let i = 0; i < n; i++) { + d.vertexPositions[i].set(Float32Array.from(QUAD, (v, k) => v + (k % 2 ? 0 : i * 0.2 + ax))); + d.dynamicFlags[i] |= 32; + } + const o = Math.fround(this.parameters.values[1]); + if (o !== d.opacities[1]) { d.opacities[1] = o; d.dynamicFlags[1] |= 4; } + }, + release() { core.releasedModels++; }, + }; + made.push(m); + return m; + }, + }, + made, releasedModels: 0, releasedMocs: 0, + }; + return core; +}; + +export const MOC = new Uint8Array([0x4d, 0x4f, 0x43, 0x33, 5, 0, 0, 0, 0, 0, 0, 0]); + +// ---- the model's objects +export const LIT = "Live2D Cubism/Lit-URP-ADV-optimize", MASK = "Live2D Cubism/Mask"; +const fadeData = (name, len, value) => ({ + MotionName: name, FadeInTime: 1, FadeOutTime: 1, MotionLength: len, ParameterFadeInTimes: [-1], ParameterFadeOutTimes: [-1], + ParameterIds: ["ParamAngleX"], ParameterCurves: [{ m_Curve: [{ time: 0, value, inSlope: 0, outSlope: 0 }, + { time: len, value, inSlope: 0, outSlope: 0 }] }], +}); +const clip = (o, name, id, len, value, opacity) => o.clip({ + name, stopTime: len, loopTime: true, events: [{ functionName: "InstanceId", intParameter: id }], + bindings: [{ cls: "CubismParameter", path: "Parameters/ParamAngleX", attribute: "Value" }, + ...(opacity === undefined ? [] : [{ cls: "CubismRenderController", path: "", attribute: "Opacity" }])], + constant: [value, ...(opacity === undefined ? [] : [opacity])], +}); + +// ---- shaders: the parsed forms and GLES3 programs +const program = "#ifdef VERTEX\n#version 300 es\nvoid main() {}\n#endif\n#ifdef FRAGMENT\n#version 300 es\nvoid main() {}\n#endif\n"; +const V = (val, name = "") => ({ val, name }); +const pass = (blend) => ({ state: { + rtBlend0: { srcBlend: blend[0], destBlend: blend[1], srcBlendAlpha: blend[2], destBlendAlpha: blend[3], blendOp: V(0), + blendOpAlpha: V(0), colMask: V(15) }, + zTest: V(4), zWrite: V(0), culling: V(0, "_Cull"), offsetFactor: V(0), offsetUnits: V(0), stencilRef: V(0), + stencilReadMask: V(255), stencilWriteMask: V(255), rtSeparateBlend: false, alphaToMask: V(0), + stencilOp: { comp: V(8), pass: V(0), fail: V(0), zFail: V(0) }, + stencilOpFront: { comp: V(8), pass: V(0), fail: V(0), zFail: V(0) }, stencilOpBack: { comp: V(8), pass: V(0), fail: V(0), zFail: V(0) }, +} }); +const litPass = pass([V(1, "_SrcColor"), V(10, "_DstColor"), V(1, "_SrcAlpha"), V(10, "_DstAlpha")]); +const maskPass = pass([V(1), V(1), V(1), V(1)]); +const parsed = (p) => ({ properties: [], subShaders: [{ tags: { tags: [] }, passes: [p] }] }); + +// a 4 x 4 PNG as far as its header goes +const PNG = new Uint8Array(24); +PNG.set([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a, 0, 0, 0, 13, 0x49, 0x48, 0x44, 0x52, 0, 0, 0, 4, 0, 0, 0, 4]); + +export const modelFiles = ({ motionSync = false, physics = true, mipCount = 1, opacity, missing = false, json = null } = {}) => { + const o = new ModelObjects(); + const lit = o.shader(LIT, parsed(litPass), [[[], program], [["CUBISM_MASK_ON"], program]]); + const mask = o.shader(MASK, parsed(maskPass), [[[], program]]); + const material = (keywords) => o.material(keywords.length ? "LitMasked" : "Lit", { + shader: lit, keywords, floats: { _SrcColor: 1, _DstColor: 10, _SrcAlpha: 1, _DstAlpha: 10, _Cull: 0 } }); + const lits = [material([]), material(["CUBISM_MASK_ON"])]; + const tex = o.texture("texture_00", { mipCount, image: PNG }); + const fades = o.mono("CubismFadeMotionList", "common.fadeMotionList", { MotionInstanceIds: [-1, -2], + CubismFadeMotionObjects: [o.mono("CubismFadeMotionData", "mtn_idle.fade", fadeData("mtn_idle", 2, 0)), + o.mono("CubismFadeMotionData", "mtn_turn.fade", fadeData("mtn_turn", 1, 30))].map(ref) }); + const expressions = o.mono("CubismExpressionList", "common.expressionList", { CubismExpressionObjects: [ + o.mono("CubismExpressionData", "exp_idle.exp3", { FadeInTime: 0.5, FadeOutTime: 0.5, Parameters: [] }), + o.mono("CubismExpressionData", "exp_closed.exp3", { FadeInTime: 0.5, FadeOutTime: 0.5, Parameters: [ + ...(missing ? [{ Id: "ParamMissing", Value: 1, Blend: 1 }] : []), { Id: "ParamEyeLOpen", Value: 0, Blend: 2 }] }), + ].map(ref) }); + const root = "model"; + const prefab = o.prefab([ + node(root, [ + mb("Live2DCharacter", { DefaultMotionName: "mtn_idle", DefaultExpressionName: "exp_idle", BasePosition: { x: 0, y: -0.41, z: 0 }, + BaseScale: 1.6, _motionList: [clip(o, "mtn_idle", -1, 2, 0), clip(o, "mtn_turn", -2, 1, 30, opacity)].map(ref), + _expressionList: ["exp_idle", "exp_closed"] }), + mb("CubismFadeController", { CubismFadeMotionList: ref(fades) }), + mb("CubismExpressionController", { UseLegacyBlendCalculation: 0, CurrentExpressionIndex: -1, CurrentFadeInTime: -1, + ExpressionsList: ref(expressions) }), + mb("CubismHarmonicMotionController", { BlendMode: 1, ChannelTimescales: [1] }), + mb("CubismAutoEyeBlinkInput", { Mean: 2.5, MaximumDeviation: 2, Timescale: 10 }), + mb("CubismEyeBlinkController", { BlendMode: 2, EyeOpening: 1 }), + mb("CubismMouthController", { BlendMode: 0, MouthOpening: 0 }), + ...(physics ? [mb("CubismPhysicsController", { _rig: { Fps: 60, Gravity: { x: 0, y: -1 }, Wind: { x: 0, y: 0 }, SubRigs: [] } })] : []), + mb("CubismRenderController", { _sortingOrder: 0, Opacity: 1 }), + ]), + node(`${root}/Parameters`), + ...IDS.map((id) => node(`${root}/Parameters/${id}`, [mb("CubismParameter"), + ...(id === "ParamEyeLOpen" ? [mb("CubismEyeBlinkParameter")] : []), + ...(id === "ParamBreath" ? [mb("CubismHarmonicMotionParameter", { Channel: 0, Direction: 2, NormalizedOrigin: 0.5, + NormalizedRange: 0.5, Duration: 3 })] : [])])), + node(`${root}/Drawables`), + ...DRAWABLES.map((id, i) => node(`${root}/Drawables/${id}`, [ + mb("CubismDrawable", { _unmanagedIndex: i }), + component("MeshRenderer", { m_Materials: [ref(lits[i === 2 ? 1 : 0])] }), + mb("CubismRenderer", { _localSortingOrder: 0, _color: { r: 1, g: 1, b: 1, a: 1 }, _mainTexture: ref(tex) }), + ])), + ]); + const resources = { cubismMask: o.material("Mask", { shader: mask, floats: { _Cull: 0 } }), + cubismMaskCulling: o.material("MaskCulling", { shader: mask, floats: { _Cull: 1 } }) }; + const store = o.store({ prefab, moc: o.moc(MOC), shaders: [lit, mask], resources, motionSync, + info: { schema: "nnnotes.product/1", product: "live2d.model", subject: "model" } }); + if (json) store._text.set("model.json", JSON.stringify(json(JSON.parse(store.text("model.json"))))); + return headlessImages(store); +}; + +// ---- the same model as the files of a model manifest of format 2 (masked: drawable 2's material masks it, and the +// shader directory holds the mask shader) +export const LEGACY_KEY = "Character/Live2D/test/model/model/model"; +export const legacyFiles = ({ masked = true } = {}) => { + const lnode = (path, components = []) => ({ path, name: path.split("/").pop(), active: true, layer: 0, + localPosition: { x: 0, y: 0, z: 0 }, localRotation: { x: 0, y: 0, z: 0, w: 1 }, localScale: { x: 1, y: 1, z: 1 }, components }); + const lmb = (cls, fields = {}) => ({ type: "MonoBehaviour", class: cls, ...fields }); + const inline = (asset, name, fields) => ({ asset, name, m_Name: name, ...fields }); + const material = (name, shader, keywords, floats) => ({ material: name, shader: { shader }, keywords, invalidKeywords: [], + renderQueue: -1, tags: {}, disabledPasses: [], textures: {}, ints: {}, floats, colors: {} }); + const lit = (keywords) => material(keywords.length ? "LitMasked" : "Lit", LIT, keywords, + { _SrcColor: 1, _DstColor: 10, _SrcAlpha: 1, _DstAlpha: 10, _Cull: 0 }); + const clip = (name, id, len, value) => ({ + clip: name, sampleRate: 30, wrapMode: 0, startTime: 0, stopTime: len, loopTime: true, cycleOffset: 0, + events: [{ functionName: "InstanceId", intParameter: id }], + bindings: [{ path: "Parameters/ParamAngleX", typeID: 114, class: "CubismParameter", attribute: "Value" }], + streamed: { curveCount: 0, frames: [] }, dense: { curveCount: 0 }, constant: [value], + }); + const texture = { texture: "textures/texture_00-0.png", name: "texture_00", width: 4, height: 4, format: 4, mipCount: 1, + colorSpace: 1, settings: { m_FilterMode: 1, m_WrapU: 1, m_WrapV: 1 } }; + const root = "model"; + const nodes = [ + lnode(root, [ + lmb("CubismModel", { _moc: { asset: "CubismMoc", name: "model" } }), + lmb("Live2DCharacter", { DefaultMotionName: "mtn_idle", DefaultExpressionName: "exp_idle", BasePosition: { x: 0, y: -0.41, z: 0 }, + BaseScale: 1.6, _motionList: [clip("mtn_idle", -1, 2, 0), clip("mtn_turn", -2, 1, 30)], + _expressionList: ["exp_idle", "exp_closed"] }), + lmb("CubismFadeController", { CubismFadeMotionList: inline("CubismFadeMotionList", "common.fadeMotionList", { + MotionInstanceIds: [-1, -2], + CubismFadeMotionObjects: [inline("CubismFadeMotionData", "mtn_idle.fade", fadeData("mtn_idle", 2, 0)), + inline("CubismFadeMotionData", "mtn_turn.fade", fadeData("mtn_turn", 1, 30))] }) }), + lmb("CubismExpressionController", { UseLegacyBlendCalculation: 0, CurrentExpressionIndex: -1, CurrentFadeInTime: -1, + ExpressionsList: inline("CubismExpressionList", "common.expressionList", { CubismExpressionObjects: [ + inline("CubismExpressionData", "exp_idle.exp3", { FadeInTime: 0.5, FadeOutTime: 0.5, Parameters: [] }), + inline("CubismExpressionData", "exp_closed.exp3", { FadeInTime: 0.5, FadeOutTime: 0.5, + Parameters: [{ Id: "ParamEyeLOpen", Value: 0, Blend: 2 }] }), + ] }) }), + lmb("CubismHarmonicMotionController", { BlendMode: 1, ChannelTimescales: [1] }), + lmb("CubismAutoEyeBlinkInput", { Mean: 2.5, MaximumDeviation: 2, Timescale: 10 }), + lmb("CubismEyeBlinkController", { BlendMode: 2, EyeOpening: 1 }), + lmb("CubismMouthController", { BlendMode: 0, MouthOpening: 0 }), + lmb("CubismPhysicsController", { _rig: { Fps: 60, Gravity: { x: 0, y: -1 }, Wind: { x: 0, y: 0 }, SubRigs: [] } }), + lmb("CubismRenderController", { _sortingOrder: 0, Opacity: 1 }), + ]), + lnode(`${root}/Parameters`), + ...IDS.map((id) => lnode(`${root}/Parameters/${id}`, [lmb("CubismParameter"), + ...(id === "ParamEyeLOpen" ? [lmb("CubismEyeBlinkParameter")] : []), + ...(id === "ParamBreath" ? [lmb("CubismHarmonicMotionParameter", { Channel: 0, Direction: 2, NormalizedOrigin: 0.5, + NormalizedRange: 0.5, Duration: 3 })] : [])])), + lnode(`${root}/Drawables`), + ...DRAWABLES.map((id, i) => lnode(`${root}/Drawables/${id}`, [ + lmb("CubismDrawable", { _unmanagedIndex: i }), + { type: "MeshRenderer", m_Materials: [lit(masked && i === 2 ? ["CUBISM_MASK_ON"] : [])] }, + lmb("CubismRenderer", { _localSortingOrder: 0, _color: { r: 1, g: 1, b: 1, a: 1 }, _mainTexture: texture }), + ])), + ]; + const variant = (dir, i, keywords) => ({ file: `${dir}/gles3/s0p0_vertex_${i}.glsl`, platform: "gles3", subShader: 0, + pass: 0, stage: "vertex", type: "GLES3", keywords }); + const shaders = [{ name: LIT, source: "CAB-test", parsed: "lit.json", variants: [variant("lit", 0, []), variant("lit", 1, ["CUBISM_MASK_ON"])] }, + ...(masked ? [{ name: MASK, source: "CAB-test", parsed: "mask.json", variants: [variant("mask", 0, [])] }] : [])]; + const text = { + "model.json": JSON.stringify({ + format: 1, name: "model", key: LEGACY_KEY, moc3: "model.moc3", prefab: "model.prefab.json", textures: [texture.texture], + canvas: { pixelsPerUnit: 600, originX: 300, originY: 450, width: 600, height: 900, mocVersion: 5 }, + shaders: "shaders/shaders.json", + resources: { cubismMask: material("Mask", MASK, [], { _Cull: 0 }), cubismMaskCulling: material("MaskCulling", MASK, [], { _Cull: 1 }) }, + }), + "model.prefab.json": JSON.stringify({ key: LEGACY_KEY, nodes, canvas: {} }), + "shaders/shaders.json": JSON.stringify(shaders), + "shaders/lit.json": JSON.stringify({ name: LIT, ...parsed(litPass) }), + "shaders/lit/gles3/s0p0_vertex_0.glsl": program, + "shaders/lit/gles3/s0p0_vertex_1.glsl": program, + ...(masked ? { "shaders/mask.json": JSON.stringify({ name: MASK, ...parsed(maskPass) }), + "shaders/mask/gles3/s0p0_vertex_0.glsl": program } : {}), + }; + return new AssetStore({ text, bytes: { "model.moc3": MOC, [texture.texture]: PNG }, + info: { format: 2, id: "model", key: LEGACY_KEY, model: { group: "test", textures: 1 } } }); +}; + +export const withCore = async (fn) => { + const saved = globalThis.Live2DCubismCore, core = fakeCore(); + globalThis.Live2DCubismCore = core; + try { return await fn(core); } finally { if (saved) globalThis.Live2DCubismCore = saved; else delete globalThis.Live2DCubismCore; } +}; diff --git a/tests/unit/live2d-legacy.test.mjs b/tests/unit/live2d-legacy.test.mjs new file mode 100644 index 0000000..7c39a47 --- /dev/null +++ b/tests/unit/live2d-legacy.test.mjs @@ -0,0 +1,118 @@ +// Models of a model manifest of format 2 (src/live2d/legacy.js): read as a product of live2d.model, and played like +// the same model in the product form (live2d-fixture.mjs). +import assert from "node:assert/strict"; +import { test } from "node:test"; +import { LEGACY_KEY, LIT, MASK, MOC, legacyFiles, modelFiles, withCore } from "./live2d-fixture.mjs"; +import { AssetStore } from "../../src/data/assets.js"; +import { componentClass } from "../../src/engine/prefab.js"; +import { isLegacyModel, legacyModel } from "../../src/live2d/legacy.js"; +import { modelIndex } from "../../src/live2d/model.js"; +import { ModelSession } from "../../src/live2d/session.js"; +import { headlessGL, headlessImages } from "../../scripts/lib/headless.mjs"; + +const PRODUCT = { schema: "nnnotes.product/1", product: "live2d.model", subject: LEGACY_KEY }; + +test("a model manifest of format 2 is recognised by its info", () => { + assert.equal(isLegacyModel(legacyFiles().info), true); + assert.equal(isLegacyModel({ format: 3, id: "model" }), false); + assert.equal(isLegacyModel({ schema: "nnnotes.product/1", product: "live2d.model", format: 2 }), false); + assert.equal(isLegacyModel(null), false); +}); + +test("its files become an object table and model.json format 3", () => { + const store = legacyModel(legacyFiles()); + assert.deepEqual(store.info, PRODUCT); + const m = modelIndex(store); + assert.equal(m.key, LEGACY_KEY); + assert.equal(m.motionSync, false); + assert.deepEqual(Object.keys(m.shaders).sort(), [LIT, MASK]); + assert.deepEqual([...store.objectBytes(m.moc, "blob:_bytes")], [...MOC]); + // the prefab's inline objects are references: the moc, the clips, the fade list, the expressions + const root = store.prefab(m.prefab).nodes[0]; + const fields = (cls) => root.components.find((c) => componentClass(c) === cls).fields; + assert.equal(store.ref(fields("CubismModel")._moc, "MonoBehaviour"), m.moc); + const clips = fields("Live2DCharacter")._motionList.map((r) => store.ref(r, "AnimationClip")); + assert.deepEqual(clips.map((id) => store.clip(id).name), ["mtn_idle", "mtn_turn"]); + assert.deepEqual(Object.keys(m.bindings), clips); + assert.deepEqual(m.bindings[clips[0]], [{ path: "Parameters/ParamAngleX", attribute: "Value" }]); + assert.deepEqual(store.clip(clips[0]).bindings, [{ typeID: 114, class: "MonoBehaviour", scriptClass: "||CubismParameter" }]); + const fades = store.mono(store.ref(fields("CubismFadeController").CubismFadeMotionList, "MonoBehaviour")); + assert.deepEqual(fades.CubismFadeMotionObjects.map((r) => store.mono(store.ref(r)).m_Name), ["mtn_idle.fade", "mtn_turn.fade"]); + const exprs = store.mono(store.ref(fields("CubismExpressionController").ExpressionsList, "MonoBehaviour")); + assert.deepEqual(exprs.CubismExpressionObjects.map((r) => store.mono(store.ref(r)).m_Name), ["exp_idle.exp3", "exp_closed.exp3"]); + // one Texture2D per atlas page, one Material per distinct material (two drawables share Lit) + const ofClass = (cls) => [...store._objects.keys()].filter((id) => store.object(id).class === cls); + assert.deepEqual(m.textures, ofClass("Texture2D")); + assert.deepEqual(store.texture(m.textures[0]), + { id: m.textures[0], name: "texture_00", width: 4, height: 4, mipCount: 1, colorSpace: 1, + settings: { m_FilterMode: 1, m_WrapU: 1, m_WrapV: 1 } }); + assert.deepEqual(ofClass("Material").map((id) => store.material(id).material).sort(), ["Lit", "LitMasked", "Mask", "MaskCulling"]); + assert.equal(store.ref(store.material(m.resources.cubismMask).shader, "Shader"), m.shaders[MASK]); + // the shader directory's variants are programs of the Shader objects + const index = store.objectJson(m.shaders[LIT], "index"); + assert.deepEqual(index.variants.map((v) => [v.role, v.keywords]), + [["program:gles3/s0p0_vertex_0", []], ["program:gles3/s0p0_vertex_1", ["CUBISM_MASK_ON"]]]); + assert.match(store.objectText(m.shaders[LIT], "program:gles3/s0p0_vertex_1"), /#ifdef VERTEX/); + assert.equal(store.objectJson(m.shaders[LIT]).name, LIT); +}); + +test("a model without masked drawables has a mask shader without programs", () => { + const store = legacyModel(legacyFiles({ masked: false })); + const m = modelIndex(store); + assert.deepEqual(store.roles(m.shaders[MASK]).sort(), ["index", "json"]); + assert.deepEqual(store.objectJson(m.shaders[MASK], "index"), { name: MASK, variants: [] }); + assert.equal(store.ref(store.material(m.resources.cubismMaskCulling).shader, "Shader"), m.shaders[MASK]); +}); + +test("model.json other than format 1 is refused", () => { + const files = legacyFiles(); + files._text.set("model.json", JSON.stringify({ ...files.json("model.json"), format: 2 })); + assert.throws(() => legacyModel(files), /model\.json: format 2 is not supported \(1 expected\)/); +}); + +test("the model plays as the same model in the product form does", () => withCore(async () => { + const open = (assets) => ModelSession.create({ gl: headlessGL(), assets, seed: 3 }); + const a = await open(headlessImages(legacyModel(legacyFiles()))), b = await open(modelFiles()); + for (const k of ["name", "motions", "expressions", "defaultMotion", "defaultExpression", "hasPhysics"]) assert.deepEqual(a[k], b[k], k); + // what the character made of the objects: clips, fades, expressions, the drawables' materials and textures + const built = (s) => { + const ch = s.character, plain = (v) => JSON.parse(JSON.stringify(v)); + return plain({ + clips: [...ch.clips.values()].map((c) => ({ name: c.name, instanceId: c.instanceId, streamed: c.streamed, constants: c.constants, + fieldCurves: c.fieldCurves, fieldConstants: c.fieldConstants })), + fades: [...ch.fadeById.entries()].map(([id, f]) => [id, f.name, f.objectName, f.length, [...f.curveOf.entries()]]), + expressions: ch.expressions, + renderers: ch.renderers.map((r) => { + const { shader: _s, ...mat } = r.material, { id: _id, ...tex } = s.assets.texture(r.texture); + return { mat, tex }; + }), + }); + }; + assert.deepEqual(built(a), built(b)); + const state = (s) => ({ params: [...s.character.params.value], shown: s.character.renderers.map((r) => [...s.character.displayed(r).pos]), + junctions: s.character.junctions.map((j) => j.tile) }); + for (const [motion, expression] of [["mtn_turn", "exp_closed"], ["mtn_idle", "exp_idle"]]) { + for (const s of [a, b]) { s.playMotion(motion, { fade: 0 }); s.setExpression(expression); } + for (let k = 0; k < 40; k++) { + await a.step(); await b.step(); + assert.deepEqual(state(a), state(b), `${motion} step ${k}`); + } + } + await a.dispose(); await b.dispose(); +})); + +test("ModelSession reads the files of a model manifest of format 2 itself", () => withCore(async () => { + const saved = globalThis.createImageBitmap; + globalThis.createImageBitmap = async () => ({ width: 4, height: 4, close() {} }); + try { + const files = legacyFiles(); + assert.ok(files instanceof AssetStore); + const s = await ModelSession.create({ gl: headlessGL(), assets: files, seed: 1 }); + assert.deepEqual(s.info, PRODUCT); + assert.deepEqual(s.motions, ["mtn_idle", "mtn_turn"]); + assert.equal(s.motion, "mtn_idle"); + await s.dispose(); + } finally { + if (saved) globalThis.createImageBitmap = saved; else delete globalThis.createImageBitmap; + } +})); diff --git a/tests/unit/live2d-session.test.mjs b/tests/unit/live2d-session.test.mjs index 49d3e21..98983c9 100644 --- a/tests/unit/live2d-session.test.mjs +++ b/tests/unit/live2d-session.test.mjs @@ -1,157 +1,10 @@ -// ModelSession over a synthetic model: a stand-in for Live2D Cubism Core (a few parameters, three drawables whose -// vertices follow ParamAngleX, one of them masked), the model's objects (a prefab with the components the runtime -// reads, clips, fade motions, expressions, materials, a texture, shaders with their GLES3 programs) and model.json, all -// in memory (live2d-objects.mjs); drawn into the no-op WebGL2 context of scripts/lib/headless.mjs. +// ModelSession over the synthetic model of live2d-fixture.mjs (its product form) and the stand-in for Live2D Cubism +// Core, drawn into the no-op WebGL2 context of scripts/lib/headless.mjs. import assert from "node:assert/strict"; import { test } from "node:test"; -import { ModelObjects, component, mb, node, ref } from "./live2d-objects.mjs"; +import { LIT, modelFiles, withCore } from "./live2d-fixture.mjs"; import { MODEL_FRAME_RATE, ModelSession } from "../../src/live2d/session.js"; -import { headlessGL, headlessImages } from "../../scripts/lib/headless.mjs"; - -const IDS = ["ParamAngleX", "ParamEyeLOpen", "ParamMouthOpenY", "ParamBreath"]; -const MIN = [-30, 0, 0, 0], MAX = [30, 1, 1, 1], DEF = [0, 1, 0, 0]; -const DRAWABLES = ["ArtMesh0", "ArtMesh1", "ArtMesh2"]; -const QUAD = [-0.1, -0.1, 0.1, -0.1, 0.1, 0.1, -0.1, 0.1]; - -// the stand-in Core: Moc / Model with parameter, part and drawable arrays; update() moves every drawable by -// ParamAngleX / 300 in x (and reports its vertices as changed on every update, as a model in motion does) and sets -// drawable 1's opacity to ParamEyeLOpen -const fakeCore = () => { - const made = []; - const core = { - Version: { csmGetVersion: () => 0x05010000, csmGetLatestMocVersion: () => 5 }, - Moc: { fromArrayBuffer: () => ({ _release() { core.releasedMocs++; } }) }, - Model: { - fromMoc() { - const n = DRAWABLES.length; - const d = { - count: n, ids: DRAWABLES.slice(), constantFlags: new Uint8Array([4, 4, 4]), dynamicFlags: new Uint8Array([1, 1, 1]), - opacities: new Float32Array([1, 1, 1]), renderOrders: new Int32Array([2, 0, 1]), maskCounts: new Int32Array([0, 0, 1]), - masks: [new Int32Array(0), new Int32Array(0), new Int32Array([0])], - vertexPositions: DRAWABLES.map((_, i) => Float32Array.from(QUAD, (v, k) => v + (k % 2 ? 0 : i * 0.2))), - vertexUvs: DRAWABLES.map(() => new Float32Array([0, 0, 1, 0, 1, 1, 0, 1])), - indices: DRAWABLES.map(() => new Uint16Array([0, 1, 2, 0, 2, 3])), - multiplyColors: new Float32Array(4 * n).fill(1), screenColors: new Float32Array([0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1]), - resetDynamicFlags() { for (let i = 0; i < n; i++) this.dynamicFlags[i] &= 1; }, - }; - const m = { - parameters: { count: IDS.length, ids: IDS.slice(), minimumValues: Float32Array.from(MIN), - maximumValues: Float32Array.from(MAX), defaultValues: Float32Array.from(DEF), values: Float32Array.from(DEF) }, - parts: { count: 1, ids: ["Part0"], opacities: new Float32Array([1]) }, - drawables: d, - canvasinfo: { CanvasWidth: 600, CanvasHeight: 900, CanvasOriginX: 300, CanvasOriginY: 450, PixelsPerUnit: 600 }, - updates: 0, - update() { - this.updates++; - const ax = this.parameters.values[0] / 300; - for (let i = 0; i < n; i++) { - d.vertexPositions[i].set(Float32Array.from(QUAD, (v, k) => v + (k % 2 ? 0 : i * 0.2 + ax))); - d.dynamicFlags[i] |= 32; - } - const o = Math.fround(this.parameters.values[1]); - if (o !== d.opacities[1]) { d.opacities[1] = o; d.dynamicFlags[1] |= 4; } - }, - release() { core.releasedModels++; }, - }; - made.push(m); - return m; - }, - }, - made, releasedModels: 0, releasedMocs: 0, - }; - return core; -}; - -const MOC = new Uint8Array([0x4d, 0x4f, 0x43, 0x33, 5, 0, 0, 0, 0, 0, 0, 0]); - -// ---- the model's objects -const LIT = "Live2D Cubism/Lit-URP-ADV-optimize", MASK = "Live2D Cubism/Mask"; -const fadeData = (name, len, value) => ({ - MotionName: name, FadeInTime: 1, FadeOutTime: 1, MotionLength: len, ParameterFadeInTimes: [-1], ParameterFadeOutTimes: [-1], - ParameterIds: ["ParamAngleX"], ParameterCurves: [{ m_Curve: [{ time: 0, value, inSlope: 0, outSlope: 0 }, - { time: len, value, inSlope: 0, outSlope: 0 }] }], -}); -const clip = (o, name, id, len, value, opacity) => o.clip({ - name, stopTime: len, loopTime: true, events: [{ functionName: "InstanceId", intParameter: id }], - bindings: [{ cls: "CubismParameter", path: "Parameters/ParamAngleX", attribute: "Value" }, - ...(opacity === undefined ? [] : [{ cls: "CubismRenderController", path: "", attribute: "Opacity" }])], - constant: [value, ...(opacity === undefined ? [] : [opacity])], -}); - -// ---- shaders: the parsed forms and GLES3 programs -const program = "#ifdef VERTEX\n#version 300 es\nvoid main() {}\n#endif\n#ifdef FRAGMENT\n#version 300 es\nvoid main() {}\n#endif\n"; -const V = (val, name = "") => ({ val, name }); -const pass = (blend) => ({ state: { - rtBlend0: { srcBlend: blend[0], destBlend: blend[1], srcBlendAlpha: blend[2], destBlendAlpha: blend[3], blendOp: V(0), - blendOpAlpha: V(0), colMask: V(15) }, - zTest: V(4), zWrite: V(0), culling: V(0, "_Cull"), offsetFactor: V(0), offsetUnits: V(0), stencilRef: V(0), - stencilReadMask: V(255), stencilWriteMask: V(255), rtSeparateBlend: false, alphaToMask: V(0), - stencilOp: { comp: V(8), pass: V(0), fail: V(0), zFail: V(0) }, - stencilOpFront: { comp: V(8), pass: V(0), fail: V(0), zFail: V(0) }, stencilOpBack: { comp: V(8), pass: V(0), fail: V(0), zFail: V(0) }, -} }); -const litPass = pass([V(1, "_SrcColor"), V(10, "_DstColor"), V(1, "_SrcAlpha"), V(10, "_DstAlpha")]); -const maskPass = pass([V(1), V(1), V(1), V(1)]); -const parsed = (p) => ({ properties: [], subShaders: [{ tags: { tags: [] }, passes: [p] }] }); - -const modelFiles = ({ motionSync = false, physics = true, mipCount = 1, opacity, missing = false, json = null } = {}) => { - const o = new ModelObjects(); - const lit = o.shader(LIT, parsed(litPass), [[[], program], [["CUBISM_MASK_ON"], program]]); - const mask = o.shader(MASK, parsed(maskPass), [[[], program]]); - const material = (keywords) => o.material(keywords.length ? "LitMasked" : "Lit", { - shader: lit, keywords, floats: { _SrcColor: 1, _DstColor: 10, _SrcAlpha: 1, _DstAlpha: 10, _Cull: 0 } }); - const lits = [material([]), material(["CUBISM_MASK_ON"])]; - const png = new Uint8Array(24); - png.set([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a, 0, 0, 0, 13, 0x49, 0x48, 0x44, 0x52, 0, 0, 0, 4, 0, 0, 0, 4]); - const tex = o.texture("texture_00", { mipCount, image: png }); - const fades = o.mono("CubismFadeMotionList", "common.fadeMotionList", { MotionInstanceIds: [-1, -2], - CubismFadeMotionObjects: [o.mono("CubismFadeMotionData", "mtn_idle.fade", fadeData("mtn_idle", 2, 0)), - o.mono("CubismFadeMotionData", "mtn_turn.fade", fadeData("mtn_turn", 1, 30))].map(ref) }); - const expressions = o.mono("CubismExpressionList", "common.expressionList", { CubismExpressionObjects: [ - o.mono("CubismExpressionData", "exp_idle.exp3", { FadeInTime: 0.5, FadeOutTime: 0.5, Parameters: [] }), - o.mono("CubismExpressionData", "exp_closed.exp3", { FadeInTime: 0.5, FadeOutTime: 0.5, Parameters: [ - ...(missing ? [{ Id: "ParamMissing", Value: 1, Blend: 1 }] : []), { Id: "ParamEyeLOpen", Value: 0, Blend: 2 }] }), - ].map(ref) }); - const root = "model"; - const prefab = o.prefab([ - node(root, [ - mb("Live2DCharacter", { DefaultMotionName: "mtn_idle", DefaultExpressionName: "exp_idle", BasePosition: { x: 0, y: -0.41, z: 0 }, - BaseScale: 1.6, _motionList: [clip(o, "mtn_idle", -1, 2, 0), clip(o, "mtn_turn", -2, 1, 30, opacity)].map(ref), - _expressionList: ["exp_idle", "exp_closed"] }), - mb("CubismFadeController", { CubismFadeMotionList: ref(fades) }), - mb("CubismExpressionController", { UseLegacyBlendCalculation: 0, CurrentExpressionIndex: -1, CurrentFadeInTime: -1, - ExpressionsList: ref(expressions) }), - mb("CubismHarmonicMotionController", { BlendMode: 1, ChannelTimescales: [1] }), - mb("CubismAutoEyeBlinkInput", { Mean: 2.5, MaximumDeviation: 2, Timescale: 10 }), - mb("CubismEyeBlinkController", { BlendMode: 2, EyeOpening: 1 }), - mb("CubismMouthController", { BlendMode: 0, MouthOpening: 0 }), - ...(physics ? [mb("CubismPhysicsController", { _rig: { Fps: 60, Gravity: { x: 0, y: -1 }, Wind: { x: 0, y: 0 }, SubRigs: [] } })] : []), - mb("CubismRenderController", { _sortingOrder: 0, Opacity: 1 }), - ]), - node(`${root}/Parameters`), - ...IDS.map((id) => node(`${root}/Parameters/${id}`, [mb("CubismParameter"), - ...(id === "ParamEyeLOpen" ? [mb("CubismEyeBlinkParameter")] : []), - ...(id === "ParamBreath" ? [mb("CubismHarmonicMotionParameter", { Channel: 0, Direction: 2, NormalizedOrigin: 0.5, - NormalizedRange: 0.5, Duration: 3 })] : [])])), - node(`${root}/Drawables`), - ...DRAWABLES.map((id, i) => node(`${root}/Drawables/${id}`, [ - mb("CubismDrawable", { _unmanagedIndex: i }), - component("MeshRenderer", { m_Materials: [ref(lits[i === 2 ? 1 : 0])] }), - mb("CubismRenderer", { _localSortingOrder: 0, _color: { r: 1, g: 1, b: 1, a: 1 }, _mainTexture: ref(tex) }), - ])), - ]); - const resources = { cubismMask: o.material("Mask", { shader: mask, floats: { _Cull: 0 } }), - cubismMaskCulling: o.material("MaskCulling", { shader: mask, floats: { _Cull: 1 } }) }; - const store = o.store({ prefab, moc: o.moc(MOC), shaders: [lit, mask], resources, motionSync, - info: { schema: "nnnotes.product/1", product: "live2d.model", subject: "model" } }); - if (json) store._text.set("model.json", JSON.stringify(json(JSON.parse(store.text("model.json"))))); - return headlessImages(store); -}; - -const withCore = async (fn) => { - const saved = globalThis.Live2DCubismCore, core = fakeCore(); - globalThis.Live2DCubismCore = core; - try { return await fn(core); } finally { if (saved) globalThis.Live2DCubismCore = saved; else delete globalThis.Live2DCubismCore; } -}; +import { headlessGL } from "../../scripts/lib/headless.mjs"; const open = (opts = {}) => { const calls = []; diff --git a/types/live2d.d.ts b/types/live2d.d.ts index 396dbc5..bcca139 100644 --- a/types/live2d.d.ts +++ b/types/live2d.d.ts @@ -8,7 +8,8 @@ export type { FromManifestOptions, ManifestInfo }; /** Game time per step: the ADV's 30 frames per second. */ export const MODEL_FRAME_RATE: 30; -/** A model manifest (a product manifest of live2d.model) without its `files` and `objects`. */ +/** A model manifest (a product manifest of live2d.model) without its `files` and `objects`; for a model manifest of + * format 2, that form with its key as `subject`. */ export interface ModelInfo { schema?: "nnnotes.product/1"; product?: "live2d.model"; @@ -47,7 +48,7 @@ export type ModelMotionCallback = (type: "motionstart" | "motionend", detail: Mo export interface ModelSessionOptions { /** The context to draw into; used by this session alone while it lives. */ gl: WebGL2RenderingContext; - /** The model's files: model.json and the files it names. */ + /** The model's files: model.json and the objects it names, or the files of a model manifest of format 2. */ assets: AssetStore; /** Motion shown first (default: the model's default motion). */ motion?: string; @@ -118,7 +119,7 @@ export class ModelSession { } export interface ModelPlayerOptions { - /** URL of a model manifest (models/.json). */ + /** URL of a model manifest (models/.json): a product manifest of live2d.model or a model manifest of format 2. */ src?: string | URL; /** A loaded AssetStore instead of `src`. */ assets?: AssetStore; From 6ff2f3da1c00db73392b8b4cd5f31cb0f3cc979c Mon Sep 17 00:00:00 2001 From: nichinichisou Date: Thu, 15 Oct 2026 19:06:51 +0800 Subject: [PATCH 03/18] feat(live): load a live from its product manifest and its parts A live loads from nnnotes' product layout: `loadLiveStore(url, {variants})` fetches the live manifest (`lives/_.json`), the manifests its `live.json` `parts` names and the variant products a page offers, and puts each product's files under its manifest path without `.json`, with one object table. - `LiveData` reads a live's documents from its store: `live.json`, the score and its mirrored form, and the sounds of the `live.sounds` product that offers the most note sound sets together with the music of `live.music`. - `ChartSession` reads its score and sounds through `LiveData`; a chart manifest of the site layout is read as a live (`src/live/legacy.js`), so the existing chart sites keep playing. - `ChartPlayer` takes a live manifest or a chart manifest as `src`, and the new `variants` option; `loadLiveStore` is exported. - The headless helper opens live manifests too. Co-Authored-By: Claude Opus 5.5 --- docs/api.md | 22 +++-- docs/data-format.md | 40 ++++++++ scripts/lib/headless.mjs | 13 +-- src/index.js | 2 + src/live/data.js | 64 +++++++++++++ src/live/legacy.js | 72 +++++++++++++++ src/live/session.js | 14 +-- src/live/store.js | 99 ++++++++++++++++++++ src/player/player.js | 12 ++- tests/unit/gltrack.test.mjs | 2 +- tests/unit/live-store.test.mjs | 149 ++++++++++++++++++++++++++++++ tests/unit/read-set.test.mjs | 15 ++- tests/unit/session-audio.test.mjs | 4 +- types/index.d.ts | 17 +++- 14 files changed, 493 insertions(+), 32 deletions(-) create mode 100644 src/live/data.js create mode 100644 src/live/legacy.js create mode 100644 src/live/store.js create mode 100644 tests/unit/live-store.test.mjs diff --git a/docs/api.md b/docs/api.md index 1cd7265..540953c 100644 --- a/docs/api.md +++ b/docs/api.md @@ -8,7 +8,7 @@ live UI, with the chart's music and sound effects, auto-played at Perfect. The d | Import | Contents | |---|---| -| `ournotes-player` | `ChartPlayer`, `ChartSession`, `AssetStore`, `defineOurnotesPlayer`, `OurnotesPlayerElement`, `LIVE_SPEEDS`, `LIVE_OPTIONS`, `LIVE_OPTION_GROUPS`, `LiveSettingsError`, `PLAYER_LANGUAGES`, `formatTime` | +| `ournotes-player` | `ChartPlayer`, `ChartSession`, `AssetStore`, `loadLiveStore`, `defineOurnotesPlayer`, `OurnotesPlayerElement`, `LIVE_SPEEDS`, `LIVE_OPTIONS`, `LIVE_OPTION_GROUPS`, `LiveSettingsError`, `PLAYER_LANGUAGES`, `formatTime` | | `ournotes-player/element` | the same exports; importing it defines `` | | `ournotes-player/ui` (experimental) | `UILibrary`, `UISession`, `UIPlayer`, `OurnotesUIElement`, `defineOurnotesUI`, `cameraProjection`; [prefab preview API and limits](ui-preview.md) | | `ournotes-player/ui/element` (experimental) | the same UI exports; importing it defines `` | @@ -25,7 +25,7 @@ Chart API times are in milliseconds of chart time. The independent UI preview AP | Attribute | Property | Meaning | |---|---|---| -| `src` | `src` | URL of a chart manifest. Setting it loads that chart (the previous one is disposed). | +| `src` | `src` | URL of a live manifest or of a chart manifest. Setting it loads that chart (the previous one is disposed). | | `controls` | `controls` | Shows the control bar (boolean attribute). | | `autoplay` | `autoplay` | Starts playing once loaded (boolean attribute). See [Audio](#audio). | | `speed` | `speed` | Playback speed, default `1`. | @@ -68,8 +68,9 @@ Options: | Option | Default | | |---|---|---| -| `src` | | URL of a chart manifest. | -| `assets` | | An `AssetStore` to use instead of `src`. | +| `src` | | URL of a live manifest ([data-format.md](data-format.md#lives)) or of a chart manifest. | +| `variants` | `[]` | With `src`: manifest paths of further note skins, note effect sets and note sound sets (see `loadLiveStore`). | +| `assets` | | An `AssetStore` to use instead of `src` (`loadLiveStore`'s, or a chart manifest's). | | `controls` | `true` | Shows the control bar. | | `autoplay` | `false` | Calls `play()` once loaded. | | `speed` | `1` | Playback speed (> 0). The control bar offers `LIVE_SPEEDS`: 0.5, 0.75, 1, 1.25, 1.5. | @@ -148,9 +149,9 @@ The chart itself, without DOM access: it draws into a WebGL2 context you give it needs `AudioContext` (window only) even for a chart without audio files. ```js -import { AssetStore, ChartSession } from "ournotes-player"; +import { ChartSession, loadLiveStore } from "ournotes-player"; -const assets = await AssetStore.fromManifest("https://example.org/site/charts/100001_expert.json"); +const assets = await loadLiveStore("https://example.org/out/lives/100001_expert.json"); const gl = canvas.getContext("webgl2", { alpha: false, antialias: false, depth: false, stencil: false, premultipliedAlpha: false }); const session = await ChartSession.create({ gl, assets, width: canvas.width, height: canvas.height }); @@ -183,6 +184,15 @@ the chart. The context serves this session alone while it lives (one session per The files of one chart, loaded before the session starts and read synchronously by it. +- `loadLiveStore(url, {variants, fetch, onProgress, signal, base, concurrency, decode})`: the store of a live + ([data-format.md](data-format.md#lives)): the live manifest at `url`, the manifests of the products its + `live.json` `parts` names and those of `variants` (manifest paths, relative to `base`, of `live.noteskin`, + `live.noteeffect` and `live.sounds` products the player offers besides), then every asset they list. Each + product's files are under its manifest path without `.json`; the objects of all of them are one table (an object + two products list must be the same in both). `base`: the root the manifest paths and asset paths are relative to + (default: the directory above the live manifest's directory). The other options are those of `fromManifest`. A + chart manifest gives the store of its files, as `fromManifest` does. + - `AssetStore.fromManifest(url, {fetch, onProgress, signal, base, concurrency, decode})`: fetches the manifest, then every asset it lists (6 requests at a time), and checks each file's size against the manifest. Asset paths are relative to `base`, by default the directory above the manifest's directory (the site layout `charts/.json` + diff --git a/docs/data-format.md b/docs/data-format.md index 23c0a23..9e72140 100644 --- a/docs/data-format.md +++ b/docs/data-format.md @@ -15,6 +15,7 @@ describes its shape and the conventions it follows rather than every serialized Contents: [Site layout](#site-layout) · +[Lives](#lives) · [charts.json](#chartsjson) · [Chart manifest](#chart-manifest) · [Logical files](#logical-files) · @@ -57,6 +58,45 @@ Contents: A page can also hand the player the logical files directly, without a site; the files and their contents are the same (see [docs/api.md](api.md)). +## Lives + +A live is the product `live` of nnnotes' product layout and the products it plays with. Each has a product manifest +(`nnnotes.product/1`: `files` and the `objects` of its object table) at `/.json`, its files under +`assets/.`: + +``` +/ + lives/_.json the live (live.json, the score) + scenes/default.json the live scene (live.scene) + notes/default.json the note highway (live.notes) + noteskins/.json a note skin (live.noteskin) + noteeffects/.json a note effect set (live.noteeffect) + sounds/.json the note and live sounds (live.sounds) + music/.json the music (live.music) + assets/. +``` + +The player loads the live manifest, the manifests its `live.json` `parts` names (the defaults) and the manifests a page +gives as variants (the other note skins, note effect sets and note sound sets it offers), and puts each product's files +under its manifest path without `.json` (`lives/100001_expert/live.json`, `sounds/default/audio/LiveSe/.flac`). + +`live.json` (format 1), as far as the player reads it: + +| Key | Meaning | +|---|---| +| `format` | `1`. | +| `notes`, `notesMirror` | The score and the score converted with `MirrorChart`, files of the live ([Score](#score)). | +| `parts` | Part → manifest path of the product: `scene`, `notes`, `noteskin`, `noteeffect`, `sounds`, `music`. | + +`sounds.json` of `live.sounds` (format 1) holds the keys of [Sounds](#sounds-audiolive-audiojson) except `music`, with +the sounds of the note sound sets and the live sounds; `music.json` of `live.music` (format 1) holds `music` +(`{ soundId }`) and `sounds` with the music's cue. A layer's `file` is relative to its product's directory. The player +uses the `live.sounds` product that offers the most note sound sets (one with `NoteSePatternId=` holds the +default set and the others) together with the music. + +A chart manifest of the site layout ([Chart manifest](#chart-manifest)) is read as a live as well: its `live.json` +gives the score and its `audio/live-audio.json` the sounds and the music. + ## charts.json The index of the charts a site offers ([schema](../schema/charts.schema.json)). Chart listings read it; the player diff --git a/scripts/lib/headless.mjs b/scripts/lib/headless.mjs index 75a857a..ecd4813 100644 --- a/scripts/lib/headless.mjs +++ b/scripts/lib/headless.mjs @@ -263,11 +263,12 @@ export const openStory = async (where, { lang = null } = {}) => { return headlessImages(await loadStoryStore(pathToFileURL(p).href, { fetch: fileFetch, lang, decode: nodeDecode })); }; -// a chart manifest file (site/charts/.json, assets under site/assets) or a directory holding live.json (its files -// read on demand) -export const openChart = async (where) => { +// a live manifest file (lives/_.json of a product layout; `variants` as loadLiveStore), a chart +// manifest file (site/charts/.json, assets under site/assets) or a directory holding live.json (its files read on +// demand) +export const openChart = async (where, { variants = [] } = {}) => { const p = path.resolve(where); - const store = fs.statSync(p).isDirectory() ? new DirStore(p) - : await AssetStore.fromManifest(pathToFileURL(p).href, { fetch: fileFetch, decode: nodeDecode }); - return headlessImages(store); + if (fs.statSync(p).isDirectory()) return headlessImages(new DirStore(p)); + const { loadLiveStore } = await import("../../src/live/store.js"); + return headlessImages(await loadLiveStore(pathToFileURL(p).href, { variants, fetch: fileFetch, decode: nodeDecode })); }; diff --git a/src/index.js b/src/index.js index 3a40770..29b5bc0 100644 --- a/src/index.js +++ b/src/index.js @@ -4,10 +4,12 @@ // defines it on import) // ChartSession the DOM-free chart session over a WebGL2 context (drive it yourself) // AssetStore the files of one chart (from a chart manifest or in memory) +// loadLiveStore the store of a live: its product manifest and the products it plays with // LIVE_OPTIONS, LIVE_OPTION_GROUPS, LiveSettingsError the game's Live options the player reproduces (settings) // PLAYER_LANGUAGES the languages of the controls export { AssetStore } from "./data/assets.js"; export { ChartSession, LIVE_SPEEDS } from "./live/session.js"; +export { loadLiveStore } from "./live/store.js"; export { LIVE_OPTIONS, LIVE_OPTION_GROUPS, LiveSettingsError } from "./live/settings.js"; export { PLAYER_LANGUAGES } from "./player/strings.js"; export { ChartPlayer } from "./player/player.js"; diff --git a/src/live/data.js b/src/live/data.js new file mode 100644 index 0000000..c6fe159 --- /dev/null +++ b/src/live/data.js @@ -0,0 +1,64 @@ +import { LIVE_PRODUCT, productDir } from "./store.js"; + +// LiveData: the documents of one live in its store (loadLiveStore, or a chart store read through legacy.js): +// live live.json of the live product: the chart, the score files, the parts +// score(m) the converted score (notes.json; notes.mirror.json with the MirrorChart option) +// audio() the sound data of the live: the sounds of a live.sounds product and the music of the live.music part +// sceneDoc, noteAssets the live scene and the note assets (livescene/scene.json, livenotes/notes.json) +// A product's files are under its directory in the store (store.js productDir). + +export const LIVE_JSON_FORMAT = 1, SOUNDS_FORMAT = 1, MUSIC_FORMAT = 1; +const SOUNDS = "live.sounds"; + +const checkFormat = (doc, format, where) => { + if (!doc || doc.format !== format) throw new Error(`${where}: format ${doc && doc.format} (${format} expected)`); + return doc; +}; + +export class LiveData { + constructor(assets) { + const info = assets.info; + if (!info || info.product !== LIVE_PRODUCT || typeof info.dir !== "string" || !info.products) + throw new Error("not the store of a live (loadLiveStore)"); + this.assets = assets; + this.dir = info.dir; + this.products = info.products; + this.live = checkFormat(assets.json(`${this.dir}/live.json`), LIVE_JSON_FORMAT, `${this.dir}/live.json`); + this.parts = Object.fromEntries(Object.entries(this.live.parts).map(([part, p]) => [part, productDir(p)])); + } + + // the JSON file `file` of the product in directory `dir` + json(dir, file) { return this.assets.json(`${dir}/${file}`); } + + // the converted score; `mirror`: the one converted with MirrorChart (live.json notesMirror) + score(mirror = false) { + const f = mirror ? this.live.notesMirror : this.live.notes; + if (!f) throw new Error(`${this.dir}: no ${mirror ? "mirrored " : ""}score`); + return this.json(this.dir, f); + } + + // The live's sounds: the live.sounds product with the most note sound sets (it holds the default set and the sets a + // page offers), with the music of the live.music part. Every layer's `file` is its path in the store. + audio() { + const loaded = Object.values(this.products[SOUNDS] || {}).map((dir) => { + const doc = checkFormat(this.json(dir, "sounds.json"), SOUNDS_FORMAT, `${dir}/sounds.json`); + return { dir, doc, sets: doc.noteSe && doc.noteSe.groups ? Object.keys(doc.noteSe.groups).length : 1 }; + }); + if (!loaded.length) throw new Error(`${this.dir}: no ${SOUNDS} product`); + const most = Math.max(...loaded.map((s) => s.sets)), pick = loaded.filter((s) => s.sets === most); + if (pick.length > 1) throw new Error(`${pick.map((s) => s.dir).join(", ")}: two ${SOUNDS} products offer as many note sound sets`); + const { dir, doc } = pick[0]; + const musicDir = this.parts.music, music = checkFormat(this.json(musicDir, "music.json"), MUSIC_FORMAT, `${musicDir}/music.json`); + const inDir = (d, sounds) => Object.fromEntries(Object.entries(sounds).map(([id, cue]) => + [id, { ...cue, layers: cue.layers.map((L) => ({ ...L, file: `${d}/${L.file}` })) }])); + const sounds = { ...inDir(dir, doc.sounds) }; + for (const [id, cue] of Object.entries(inDir(musicDir, music.sounds))) { + if (id in sounds) throw new Error(`sound ${id}: in ${dir}/sounds.json and ${musicDir}/music.json`); + sounds[id] = cue; + } + return { ...doc, sounds, music: music.music }; + } + + get sceneDoc() { return this.assets.json("livescene/scene.json"); } + get noteAssets() { return this.assets.json("livenotes/notes.json"); } +} diff --git a/src/live/legacy.js b/src/live/legacy.js new file mode 100644 index 0000000..e60d882 --- /dev/null +++ b/src/live/legacy.js @@ -0,0 +1,72 @@ +import { AssetStore } from "../data/assets.js"; +import { LIVE_PRODUCT } from "./store.js"; +import { LIVE_JSON_FORMAT, MUSIC_FORMAT, SOUNDS_FORMAT } from "./data.js"; + +// Charts of the site layout with a live directory (docs/data-format.md "Chart manifest"): a chart manifest without +// `schema` whose files are a live.json index, the score, audio/live-audio.json with its waveforms, livescene/ and +// livenotes/. legacyLive() reads such a store as the store of a live: a live.json of the live product, sounds.json and +// music.json of the live.sounds and live.music products, so ChartSession reads both through LiveData. + +// whether `store` holds a chart of the site layout with a live directory (no product `schema`, a live.json) +export const isLegacyLive = (store) => !!store && (!store.info || store.info.schema === undefined) && store.has("live.json"); + +// A store over another one: files added in memory (`text`) and files that are another path of the inner store +// (`alias`); every other path reads the inner store as it is. +class OverlayStore extends AssetStore { + constructor(inner, { text = {}, alias = {}, info = null } = {}) { + super({ text, info }); + this.inner = inner; + this.alias = new Map(Object.entries(alias)); + } + + _path(p) { return this.alias.get(p) ?? p; } + has(p) { return this._text.has(p) || this.inner.has(this._path(p)); } + text(p) { return this._text.has(p) ? this._text.get(p) : this.inner.text(this._path(p)); } + bytes(p) { return this._text.has(p) ? super.bytes(p) : this.inner.bytes(this._path(p)); } + json(p) { return this._text.has(p) ? super.json(p) : this.inner.json(this._path(p)); } + image(p) { return this.inner.image(this._path(p)); } + list(prefix = "") { + return [...new Set([...this._text.keys(), ...this.alias.keys(), ...this.inner.list()])].filter((k) => k.startsWith(prefix)).sort(); + } +} + +// The store of a chart of the site layout read as the store of a live (an OverlayStore over `store`): +// lives/_/live.json, notes.json, notes.mirror.json the live: the score files (and the mirrored +// one when the chart has it), `parts` naming the products below +// sounds/default/sounds.json live-audio.json without the music; its waveforms under sounds/default/ +// music//music.json the music's sound; its waveforms under music// +// The scene and the note assets stay where they are (livescene/, livenotes/). `info` is the chart manifest's (chart, +// quality, options, audio) with the live's `product`, `subject`, `dir` and `products`. +export const legacyLive = (store) => { + const old = store.json("live.json"); + for (const k of ["musicId", "difficulty", "notes", "liveAudio"]) + if (old[k] === undefined) throw new Error(`live.json: no ${k}`); + const id = `${old.musicId}_${old.difficulty}`, dir = `lives/${id}`; + const sounds = "sounds/default", musicDir = `music/${old.musicId}`; + const text = {}, alias = {}; + alias[`${dir}/notes.json`] = old.notes; + if (old.notesMirror) alias[`${dir}/notes.mirror.json`] = old.notesMirror; + const parts = { scene: "scenes/default.json", notes: "notes/default.json", noteskin: "noteskins/default.json", + noteeffect: "noteeffects/default.json", sounds: `${sounds}.json`, music: `${musicDir}.json` }; + text[`${dir}/live.json`] = JSON.stringify({ + format: LIVE_JSON_FORMAT, musicId: old.musicId, difficulty: old.difficulty, notes: "notes.json", + ...(old.notesMirror ? { notesMirror: "notes.mirror.json" } : {}), parts, + }); + const audio = store.json(old.liveAudio); + const musicId = String(audio.music && audio.music.soundId); + if (!audio.sounds || !audio.sounds[musicId]) throw new Error(`${old.liveAudio}: no sound of the music (${musicId})`); + // a sound's waveforms under the product's directory + const moved = (into, cue) => ({ ...cue, layers: cue.layers.map((L) => { alias[`${into}/${L.file}`] = L.file; return L; }) }); + const { music, sounds: all, ...rest } = audio; + const se = Object.fromEntries(Object.entries(all).filter(([k]) => k !== musicId).map(([k, c]) => [k, moved(sounds, c)])); + text[`${sounds}/sounds.json`] = JSON.stringify({ format: SOUNDS_FORMAT, ...rest, sounds: se }); + text[`${musicDir}/music.json`] = JSON.stringify({ + format: MUSIC_FORMAT, musicId: old.musicId, music, sounds: { [musicId]: moved(musicDir, all[musicId]) }, + }); + const info = { + ...(store.info || {}), product: LIVE_PRODUCT, subject: `${old.musicId}/${old.difficulty}`, dir, + products: { [LIVE_PRODUCT]: { [`${old.musicId}/${old.difficulty}`]: dir }, "live.sounds": { default: sounds }, + "live.music": { [String(old.musicId)]: musicDir } }, + }; + return new OverlayStore(store, { text, alias, info }); +}; diff --git a/src/live/session.js b/src/live/session.js index 1f04b53..98223ea 100644 --- a/src/live/session.js +++ b/src/live/session.js @@ -5,7 +5,9 @@ import { trackGL } from "../engine/gltrack.js"; import { ShaderLib } from "../engine/glsl.js"; import { PlayerLoop } from "../engine/loop.js"; import { LiveLightWeightBackground } from "./background.js"; +import { LiveData } from "./data.js"; import { LiveFx } from "./fx.js"; +import { isLegacyLive, legacyLive } from "./legacy.js"; import { LiveNotes } from "./noteview.js"; import { LiveRenderer } from "./renderer.js"; import { LIVE_OPTION_BY_NAME, LiveOptionContext, LiveSettingsError, liveCategoryVolumes, liveDerived, liveNoteEffectName, @@ -80,7 +82,8 @@ export const LIVE_SPEEDS = [0.5, 0.75, 1, 1.25, 1.5]; // the playback speeds t export class ChartSession { // opts: // gl WebGL2RenderingContext (required); used by this session alone while it lives - // assets AssetStore of the chart (required) + // assets AssetStore of the live (required): loadLiveStore's, or the store of a chart manifest of the site + // layout with a live directory (read through legacy.js) // audioContext an AudioContext to play into, at any sample rate (the waveforms are decoded into its rate; default: // a 48 kHz one created and closed by the session) // settings Live options by name (settings.js LIVE_OPTIONS; default: the chart data's preset-1 values); values @@ -106,13 +109,12 @@ export class ChartSession { async _load({ gl, assets, audioContext = null, quality, seed, width, height, settings = null } = {}) { if (!gl) throw new Error("ChartSession: a WebGL2 context is required"); if (!assets) throw new Error("ChartSession: an AssetStore is required"); + if (isLegacyLive(assets)) assets = legacyLive(assets); bindAssets(gl, assets); this.gl = gl; this.assets = assets; this._gl = trackGL(gl); - const live = assets.json("live.json"); - const scene = assets.json(live.scene); - const noteAssets = assets.json(live.noteAssets); - const audioData = assets.json(live.liveAudio); + const data = this.data = new LiveData(assets); + const live = data.live, scene = data.sceneDoc, noteAssets = data.noteAssets, audioData = data.audio(); // Live options: defaults, ranges and the values the chart's files offer, then the settings of this session const ctx = new LiveOptionContext({ live, scene, notes: noteAssets, audio: audioData, info: assets.info || null }); if (quality !== undefined && quality !== null) ctx.defaults.LiveQuality = Number(quality); @@ -121,7 +123,7 @@ export class ChartSession { this._settings = s; this.derived = liveDerived(ctx, s); // MirrorChart (4): the score converted with isMirror (SsMusicScoreConverter), a file of its own - const score = assets.json(s.MirrorChart ? live.notesMirror : live.notes); + const score = data.score(s.MirrorChart); this.live = live; this.score = score; this.noteAssets = noteAssets; const loop = new PlayerLoop(60); diff --git a/src/live/store.js b/src/live/store.js new file mode 100644 index 0000000..27fb40b --- /dev/null +++ b/src/live/store.js @@ -0,0 +1,99 @@ +import { AssetStore, PRODUCT_MANIFEST, decodeAsset } from "../data/assets.js"; + +// Loading a live: the product manifest of the live (lives/_.json, the `live` product of nnnotes' +// product layout), the manifests of the products its live.json `parts` names, and any further products of note skins, +// note effect sets and note sound sets a page offers (`variants`). The store holds every product's files under the +// product's directory, its manifest path without ".json" (lives/100001_expert/live.json, scenes/default/scene.json, +// sounds/default/audio/LiveSe/.flac, ...), and the objects of all of them in one table. + +export const LIVE_PRODUCT = "live"; +// the part of a live (live.json `parts`) -> the product it is +export const LIVE_PARTS = Object.freeze({ + scene: "live.scene", notes: "live.notes", noteskin: "live.noteskin", noteeffect: "live.noteeffect", + sounds: "live.sounds", music: "live.music", +}); +// the products a page may offer besides the parts: other note skins, note effect sets and note sound sets +export const LIVE_VARIANTS = Object.freeze(["live.noteskin", "live.noteeffect", "live.sounds"]); + +const here = () => (globalThis.document ? globalThis.document.baseURI : globalThis.location ? globalThis.location.href : undefined); + +// the directory of a product's files in a live store: its manifest path without ".json" +export const productDir = (manifestPath) => { + const p = String(manifestPath); + if (!/^[^/]+\/[^/]+\.json$/.test(p)) throw new Error(`${p}: not a product manifest path /.json`); + return p.slice(0, -".json".length); +}; + +const isProduct = (m, product) => !!m && m.schema === PRODUCT_MANIFEST && m.product === product + && !!m.files && typeof m.files === "object" && !!m.objects && typeof m.objects === "object"; + +// A chart manifest of the site layout (docs/data-format.md "Chart manifest"): no `schema`, a `live.json` file. Its +// store is read through legacy.js. +export const isChartManifest = (m) => !!m && m.schema === undefined && !!m.files && !!m.files["live.json"]; + +// The store of the live at `url`: a live product manifest, or a chart manifest (its files as they are; ChartSession +// reads them through legacy.js). `variants`: manifest paths (relative to `base`) of further products of LIVE_VARIANTS. +// `base`: the root the manifest and asset paths are relative to (default: the directory above the manifest's). +// `fetch`, `signal`, `onProgress`, `decode`, `concurrency`: as AssetStore.fromManifest. The store's `info` is the live +// manifest without its file lists, plus `dir` (the live's directory) and `products` ({product: {subject: directory}} +// of every product loaded, the live included). +export const loadLiveStore = async (url, { variants = [], fetch = globalThis.fetch, signal = null, onProgress = null, + base = null, decode = decodeAsset, concurrency } = {}) => { + if (!url) throw new Error("no live manifest URL"); + if (typeof fetch !== "function") throw new Error("no fetch function"); + const href = new URL(String(url), here()).href; + const getJson = async (u) => { + if (signal) signal.throwIfAborted(); + const r = await fetch(u, signal ? { signal } : undefined); + if (!r.ok) throw new Error(`${u}: HTTP ${r.status}`); + return r.json(); + }; + const man = await getJson(href); + const serve = (doc) => (u, init) => (String(u) === href + ? Promise.resolve(new Response(JSON.stringify(doc), { headers: { "content-type": "application/json" } })) + : fetch(u, init)); + const opts = { signal, decode, concurrency }; + if (isChartManifest(man)) { + if (variants.length) throw new Error(`${href}: a chart manifest has no variant products`); + return AssetStore.fromManifest(href, { ...opts, fetch: serve(man), base, onProgress }); + } + if (!isProduct(man, LIVE_PRODUCT)) throw new Error(`${href}: not a live manifest (${PRODUCT_MANIFEST} of ${LIVE_PRODUCT})`); + const root = base ? new URL(String(base), here()) : new URL("../", href); + if (!href.startsWith(root.href)) throw new Error(`${href}: not under the root ${root.href}`); + const liveDir = productDir(href.slice(root.href.length)); + // live.json names the parts: read it first (through fromManifest, which decodes and checks it) + const head = await AssetStore.fromManifest(href, { ...opts, base: root.href, + fetch: serve({ files: { "live.json": man.files["live.json"] } }) }); + const live = head.json("live.json"); + if (!live || !live.parts || typeof live.parts !== "object") throw new Error(`${href}: live.json without parts`); + const wanted = []; // [manifest path, the products it may be] + for (const [part, product] of Object.entries(LIVE_PARTS)) { + if (typeof live.parts[part] !== "string") throw new Error(`${href}: live.json names no ${part} part`); + wanted.push([live.parts[part], [product]]); + } + for (const p of variants) wanted.push([String(p), LIVE_VARIANTS]); + const loaded = await Promise.all(wanted.map(async ([p, products]) => { + const u = new URL(p, root).href, m = await getJson(u); + if (!products.some((x) => isProduct(m, x))) + throw new Error(`${u}: not a manifest of ${products.join(" or ")}`); + return [productDir(p), m]; + })); + const files = {}, objects = {}, products = {}; + for (const [dir, m] of [[liveDir, man], ...loaded]) { + const subjects = (products[m.product] ||= {}); + if (m.subject in subjects) { + if (subjects[m.subject] !== dir) throw new Error(`${m.product} ${m.subject}: two manifests (${subjects[m.subject]}, ${dir})`); + continue; // a variant that is also a part + } + subjects[m.subject] = dir; + for (const [p, f] of Object.entries(m.files)) files[`${dir}/${p}`] = f; + for (const [id, o] of Object.entries(m.objects)) { + if (id in objects && JSON.stringify(objects[id]) !== JSON.stringify(o)) + throw new Error(`object ${id} differs between the manifests of the live's products`); + objects[id] = o; + } + } + const { files: _f, objects: _o, ...info } = man; + const merged = { ...info, dir: liveDir, products, files, objects }; + return AssetStore.fromManifest(href, { ...opts, fetch: serve(merged), base: root.href, onProgress }); +}; diff --git a/src/player/player.js b/src/player/player.js index 293ecec..18dee9c 100644 --- a/src/player/player.js +++ b/src/player/player.js @@ -1,6 +1,6 @@ -import { AssetStore } from "../data/assets.js"; import { ChartSession } from "../live/session.js"; import { liveSettingsChanges } from "../live/settings.js"; +import { loadLiveStore } from "../live/store.js"; import { ChartControls, PLAYER_CSS } from "./controls.js"; import { formatString, playerStrings } from "./strings.js"; @@ -26,8 +26,11 @@ const MAX_STEPS = 4; // steps per animation frame at most (a late f export class ChartPlayer extends EventTarget { // host: an Element or a ShadowRoot. opts: - // src URL of a chart manifest (charts/.json), or - // assets an AssetStore + // src URL of a live manifest (lives/_.json) or of a chart manifest (charts/.json), + // or + // assets an AssetStore (loadLiveStore's, or a chart manifest's) + // variants with `src`: manifest paths of further note skins, note effect sets and note sound sets the player + // offers (loadLiveStore) // controls show the control bar (default true) // autoplay start playing once loaded (default false); a browser that blocks audio without a user gesture // keeps the player paused until play() is called from a gesture @@ -84,7 +87,8 @@ export class ChartPlayer extends EventTarget { const o = this.opts, signal = o.signal ? anySignal([o.signal, this._abort.signal]) : this._abort.signal; this._status(this.strings.loading); const mb = (n) => (n / 1048576).toFixed(1); - const assets = o.assets || await AssetStore.fromManifest(o.src, { + const assets = o.assets || await loadLiveStore(o.src, { + variants: o.variants || [], signal, onProgress: (loaded, total) => { this._status(formatString(this.strings.loadingMB, { loaded: mb(loaded), total: mb(total) })); diff --git a/tests/unit/gltrack.test.mjs b/tests/unit/gltrack.test.mjs index 7f0cb29..ebde677 100644 --- a/tests/unit/gltrack.test.mjs +++ b/tests/unit/gltrack.test.mjs @@ -50,7 +50,7 @@ test("nothing is deleted on a lost context; other contexts are not tracked", () test("ChartSession tracks its context and releases it when it fails to load", () => withWebGL2(async () => { const gl = new FakeGL2(), set = []; const spy = new Proxy(gl, { set(o, k, v) { set.push(k); o[k] = v; return true; } }); - await assert.rejects(ChartSession.create({ gl: spy, assets: new AssetStore() }), /live\.json/); + await assert.rejects(ChartSession.create({ gl: spy, assets: new AssetStore() }), /not the store of a live/); assert.deepEqual(set.slice(0, 2), ["createBuffer", "deleteBuffer"]); assert.deepEqual(Object.keys(gl).filter((k) => typeof gl[k] === "function"), []); })); diff --git a/tests/unit/live-store.test.mjs b/tests/unit/live-store.test.mjs new file mode 100644 index 0000000..20e17f7 --- /dev/null +++ b/tests/unit/live-store.test.mjs @@ -0,0 +1,149 @@ +// Loading a live (src/live/store.js), its documents (src/live/data.js) and charts of the site layout read as lives +// (src/live/legacy.js), over a product layout and a chart site in memory. Synthetic data only. +import assert from "node:assert/strict"; +import { test } from "node:test"; +import { AssetStore, PRODUCT_MANIFEST } from "../../src/data/assets.js"; +import { LiveData } from "../../src/live/data.js"; +import { isLegacyLive, legacyLive } from "../../src/live/legacy.js"; +import { LIVE_PARTS, loadLiveStore, productDir } from "../../src/live/store.js"; + +const ROOT = "https://example.org/out/"; + +// a product layout in memory: path -> JSON document or bytes; a product's files are assets named by their path +const layout = () => { + const files = new Map(); + const product = (manifest, product, subject, own, objects = {}) => { + const entries = {}; + for (const [p, body] of Object.entries(own)) { + const bytes = body instanceof Uint8Array ? body : new TextEncoder().encode(JSON.stringify(body)); + const asset = `assets/${manifest.replace(/[/.]/g, "_")}_${p.replace(/[/.]/g, "_")}${p.endsWith(".json") ? ".json" : ".bin"}`; + files.set(asset, bytes); + entries[p] = { asset, size: bytes.byteLength }; + } + files.set(manifest, { schema: PRODUCT_MANIFEST, product, subject, files: entries, objects }); + }; + const fetch = async (u) => { + const p = String(u).slice(ROOT.length), v = files.get(p); + if (v === undefined) return { ok: false, status: 404 }; + const bytes = v instanceof Uint8Array ? v : new TextEncoder().encode(JSON.stringify(v)); + return { ok: true, status: 200, json: async () => JSON.parse(new TextDecoder().decode(bytes)), + arrayBuffer: async () => bytes.buffer.slice(bytes.byteOffset, bytes.byteOffset + bytes.byteLength) }; + }; + return { files, product, fetch }; +}; + +const TEX = { class: "Texture2D", name: "t", files: { json: { asset: "assets/tex.json", size: 2 } } }; +const PARTS = { scene: "scenes/default.json", notes: "notes/default.json", noteskin: "noteskins/skin001.json", + noteeffect: "noteeffects/effect001.json", sounds: "sounds/default.json", music: "music/7.json" }; +const layer = (file) => ({ file, sampleRate: 48000, channels: 2, samples: 4800, loopFlag: 0, loopStart: null, + loopEnd: null, volume: 1 }); +const cue = (sheet, file, cats) => ({ sheet, cue: sheet, category: 1, categories: cats, volume: 1, + row: { _isRandomPitch: false }, layers: [layer(file)] }); + +const live = () => { + const L = layout(); + L.files.set("assets/tex.json", new TextEncoder().encode("{}")); + L.product("lives/7_expert.json", "live", "7/expert", + { "live.json": { format: 1, musicId: 7, difficulty: "expert", notes: "notes.json", + notesMirror: "notes.mirror.json", parts: PARTS }, + "notes.json": { notes: [1] }, "notes.mirror.json": { notes: [2] } }, { "a:1": TEX }); + L.product("scenes/default.json", "live.scene", "default", { "scene.json": { format: 1 } }, { "a:1": TEX }); + L.product("notes/default.json", "live.notes", "default", { "notes.json": { format: 1 } }); + for (const s of ["skin001", "skin002"]) L.product(`noteskins/${s}.json`, "live.noteskin", s, { "noteskin.json": { name: s } }); + L.product("noteeffects/effect001.json", "live.noteeffect", "effect001", { "noteeffect.json": {} }); + L.product("sounds/default.json", "live.sounds", "default", { + "sounds.json": { format: 1, sounds: { 10: cue("se1", "audio/se1/a.flac", ["LiveSe"]) }, categories: { LiveSe: 1 }, + noteSe: { types: { 1: 10 }, patternId: 1 } }, + "audio/se1/a.flac": new Uint8Array([1]) }); + L.product("sounds/NoteSePatternId=1,3.json", "live.sounds", "NoteSePatternId=1,3", { + "sounds.json": { format: 1, sounds: { 10: cue("se1", "audio/se1/a.flac", ["LiveSe"]), 30: cue("se3", "audio/se3/a.flac", ["LiveSe"]) }, + categories: { LiveSe: 1 }, noteSe: { types: { 1: 10 }, patternId: 1, groups: { 1: { 1: 10 }, 3: { 1: 30 } } } }, + "audio/se1/a.flac": new Uint8Array([1]), "audio/se3/a.flac": new Uint8Array([3]) }); + L.product("music/7.json", "live.music", "7", { + "music.json": { format: 1, musicId: 7, music: { soundId: 1 }, sounds: { 1: cue("bgm_7", "audio/bgm_7/bgm_7.m4a", ["LiveBgm"]) } }, + "audio/bgm_7/bgm_7.m4a": new Uint8Array([7]) }); + return L; +}; + +test("productDir: a manifest path without .json, one directory and a file", () => { + assert.equal(productDir("scenes/default.json"), "scenes/default"); + assert.equal(productDir("sounds/NoteSePatternId=1,3.json"), "sounds/NoteSePatternId=1,3"); + for (const bad of ["scene.json", "a/b/c.json", "scenes/default"]) assert.throws(() => productDir(bad), /not a product manifest path/); +}); + +test("loadLiveStore: the live, its parts and the variants, each under its directory, one object table", async () => { + const L = live(); + const store = await loadLiveStore(`${ROOT}lives/7_expert.json`, { fetch: L.fetch, variants: ["noteskins/skin002.json"] }); + assert.equal(store.info.dir, "lives/7_expert"); + assert.deepEqual(store.info.products, { + live: { "7/expert": "lives/7_expert" }, "live.scene": { default: "scenes/default" }, + "live.notes": { default: "notes/default" }, "live.noteskin": { skin001: "noteskins/skin001", skin002: "noteskins/skin002" }, + "live.noteeffect": { effect001: "noteeffects/effect001" }, "live.sounds": { default: "sounds/default" }, + "live.music": { 7: "music/7" } }); + assert.deepEqual(store.json("noteskins/skin002/noteskin.json"), { name: "skin002" }); + assert.deepEqual([...store.bytes("music/7/audio/bgm_7/bgm_7.m4a")], [7]); + assert.deepEqual(store.object("a:1", "Texture2D"), { id: "a:1", class: "Texture2D", name: "t" }); // in two products + assert.equal(Object.keys(LIVE_PARTS).length, 6); +}); + +test("loadLiveStore refuses a manifest of another product, a part of the wrong product and differing objects", async () => { + const L = live(); + await assert.rejects(loadLiveStore(`${ROOT}scenes/default.json`, { fetch: L.fetch }), /not a live manifest/); + await assert.rejects(loadLiveStore(`${ROOT}lives/7_expert.json`, { fetch: L.fetch, variants: ["scenes/default.json"] }), + /not a manifest of live\.noteskin or live\.noteeffect or live\.sounds/); + L.files.get("scenes/default.json").objects["a:1"] = { ...TEX, name: "other" }; + await assert.rejects(loadLiveStore(`${ROOT}lives/7_expert.json`, { fetch: L.fetch }), /object a:1 differs/); + L.files.delete("music/7.json"); + await assert.rejects(loadLiveStore(`${ROOT}lives/7_expert.json`, { fetch: L.fetch }), /music\/7\.json: HTTP 404/); +}); + +test("LiveData: the score, and the sounds of the live.sounds product with the most sets with the music", async () => { + const L = live(); + const one = new LiveData(await loadLiveStore(`${ROOT}lives/7_expert.json`, { fetch: L.fetch })); + assert.deepEqual([one.score().notes, one.score(true).notes], [[1], [2]]); + assert.deepEqual(one.parts, Object.fromEntries(Object.entries(PARTS).map(([k, p]) => [k, p.slice(0, -5)]))); + const a = one.audio(); + assert.deepEqual(Object.keys(a.sounds), ["1", "10"]); + assert.equal(a.sounds[10].layers[0].file, "sounds/default/audio/se1/a.flac"); + assert.equal(a.sounds[1].layers[0].file, "music/7/audio/bgm_7/bgm_7.m4a"); + assert.deepEqual([a.music, a.categories], [{ soundId: 1 }, { LiveSe: 1 }]); + const two = new LiveData(await loadLiveStore(`${ROOT}lives/7_expert.json`, + { fetch: L.fetch, variants: ["sounds/NoteSePatternId=1,3.json"] })); + const b = two.audio(); + assert.deepEqual(Object.keys(b.noteSe.groups), ["1", "3"]); + assert.equal(b.sounds[30].layers[0].file, "sounds/NoteSePatternId=1,3/audio/se3/a.flac"); + assert.throws(() => new LiveData(new AssetStore()), /not the store of a live/); +}); + +// a chart of the site layout: live.json, the score, live-audio.json and its waveforms +const chart = (extra = {}) => new AssetStore({ + text: { + "live.json": JSON.stringify({ musicId: 7, difficulty: "expert", scene: "livescene/scene.json", + notes: "score/m7_03.notes.json", notesMirror: "score/m7_03.mirror.notes.json", + noteAssets: "livenotes/notes.json", liveAudio: "audio/live-audio.json", ...extra }), + "score/m7_03.notes.json": JSON.stringify({ notes: [1] }), "score/m7_03.mirror.notes.json": JSON.stringify({ notes: [2] }), + "audio/live-audio.json": JSON.stringify({ + sounds: { 1: cue("bgm_7", "audio/bgm_7/bgm_7.m4a", ["LiveBgm"]), 10: cue("se1", "audio/se1/a.flac", ["LiveSe"]) }, + categories: { LiveBgm: 1, LiveSe: 1 }, react: [], music: { soundId: 1 }, noteSe: { types: { 1: 10 } }, liveSe: {}, + timeline: { startCheerDelaySec: 1 } }), + "livescene/scene.json": "{}", "livenotes/notes.json": "{}", + }, + bytes: { "audio/bgm_7/bgm_7.m4a": new Uint8Array([7]), "audio/se1/a.flac": new Uint8Array([1]) }, + info: { format: 3, chart: { title: "x" }, quality: 1 }, +}); + +test("legacyLive: a chart of the site layout read as the store of a live", () => { + const old = chart(); + assert.ok(isLegacyLive(old) && !isLegacyLive(new AssetStore({ text: { "live.json": "{}" }, info: { schema: PRODUCT_MANIFEST } }))); + const store = legacyLive(old), data = new LiveData(store); + assert.deepEqual([data.score().notes, data.score(true).notes], [[1], [2]]); + assert.equal(store.info.dir, "lives/7_expert"); + assert.deepEqual([store.info.chart, store.info.quality, store.info.product], [{ title: "x" }, 1, "live"]); + const a = data.audio(); + assert.deepEqual([a.music, Object.keys(a.sounds)], [{ soundId: 1 }, ["1", "10"]]); + assert.deepEqual([...store.bytes(a.sounds[1].layers[0].file)], [7]); + assert.deepEqual([...store.bytes(a.sounds[10].layers[0].file)], [1]); + assert.deepEqual([data.sceneDoc, data.noteAssets], [{}, {}]); + assert.equal(new LiveData(legacyLive(chart({ notesMirror: undefined }))).live.notesMirror, undefined); + assert.throws(() => legacyLive(chart({ musicId: undefined })), /live\.json: no musicId/); +}); diff --git a/tests/unit/read-set.test.mjs b/tests/unit/read-set.test.mjs index 6c9379b..965e481 100644 --- a/tests/unit/read-set.test.mjs +++ b/tests/unit/read-set.test.mjs @@ -125,13 +125,18 @@ test("read-set --serve answers each request line in order, one JSON line each, u } finally { fs.rmSync(dir, { recursive: true, force: true }); } }); -// a live directory whose session gets as far as the Live options: live.json and empty scene / note asset / audio -// files, no score (a load with accepted settings fails at the score) +// a live directory whose session gets as far as the Live options: live.json, empty scene and note asset files, an +// audio file with the music's sound, no score (a load with accepted settings fails at the score) const settingsDir = () => { const dir = fs.mkdtempSync(path.join(os.tmpdir(), "ournotes-settings-")); - fs.writeFileSync(path.join(dir, "live.json"), JSON.stringify({ scene: "scene.json", noteAssets: "notes.json", - liveAudio: "audio.json", notes: "score.json" })); - for (const f of ["scene.json", "notes.json", "audio.json"]) fs.writeFileSync(path.join(dir, f), "{}"); + fs.writeFileSync(path.join(dir, "live.json"), JSON.stringify({ + musicId: 1, difficulty: "expert", scene: "livescene/scene.json", noteAssets: "livenotes/notes.json", + liveAudio: "audio.json", notes: "score.json" })); + for (const d of ["livescene", "livenotes"]) { + fs.mkdirSync(path.join(dir, d)); + fs.writeFileSync(path.join(dir, d, d === "livescene" ? "scene.json" : "notes.json"), "{}"); + } + fs.writeFileSync(path.join(dir, "audio.json"), JSON.stringify({ music: { soundId: 1 }, sounds: { 1: { layers: [] } } })); return dir; }; diff --git a/tests/unit/session-audio.test.mjs b/tests/unit/session-audio.test.mjs index d846970..11b1269 100644 --- a/tests/unit/session-audio.test.mjs +++ b/tests/unit/session-audio.test.mjs @@ -139,9 +139,9 @@ test("the headless AudioContext decodes into its own sample rate", async () => { test("ChartSession accepts an AudioContext at any sample rate", async () => { for (const rate of [44100, 48000, 96000]) { const ctx = new FakeAudioContext(rate, PRIMED); - // an empty store: the session fails on its first file, not on the context + // an empty store: the session fails on its data, not on the context await assert.rejects(ChartSession.create({ gl: headlessGL(), assets: new AssetStore(), audioContext: ctx }), - /asset not found: live\.json/); + /not the store of a live/); } }); diff --git a/types/index.d.ts b/types/index.d.ts index c5b22be..8a8aebe 100644 --- a/types/index.d.ts +++ b/types/index.d.ts @@ -220,10 +220,23 @@ export class ChartSession { /** The playback speeds the controls offer. */ export const LIVE_SPEEDS: readonly number[]; +export interface LoadLiveStoreOptions extends FromManifestOptions { + /** Manifest paths (relative to `base`) of further products of note skins, note effect sets and note sound sets + * (`live.noteskin`, `live.noteeffect`, `live.sounds`) the player offers. */ + variants?: string[]; +} + +/** The store of a live: its product manifest (`lives/_.json`), the products its live.json + * `parts` names and the `variants`, each product's files under its manifest path without `.json`. A chart manifest + * (`charts/.json`) gives the store of its files. */ +export function loadLiveStore(url: string | URL, options?: LoadLiveStoreOptions): Promise; + export interface ChartPlayerOptions { - /** URL of a chart manifest (charts/.json). */ + /** URL of a live manifest (lives/_.json) or of a chart manifest (charts/.json). */ src?: string | URL; - /** A loaded AssetStore instead of `src`. */ + /** With `src`: manifest paths of further note skins, note effect sets and note sound sets (`loadLiveStore`). */ + variants?: string[]; + /** A loaded AssetStore instead of `src` (`loadLiveStore`'s, or a chart manifest's). */ assets?: AssetStore; /** Show the control bar (default true). */ controls?: boolean; From d197e3e44fdf07af40160830e74bcecc8b67c197 Mon Sep 17 00:00:00 2001 From: nichinichisou Date: Thu, 15 Oct 2026 21:11:38 +0800 Subject: [PATCH 04/18] feat(live)!: read the scene and the note highway from their products A live's scene, note views, note skin and note effect set are read from the live.scene, live.notes, live.noteskin and live.noteeffect products: - LiveObjects resolves prefab.json roots, materials, sprites, textures, meshes, clips, controllers and MonoBehaviour assets by object id. - One shader library serves the whole live: Shader objects by id, with the products' name tables answering lookups by name. - Clips are named by the products' bindings tables (AnimBindings, AnimClip.fromObject) for the Animator that plays them. - The engine stays independent of the data layout: FxMaterials reads materials from a directory (stories) or from objects (lives), and Prefab.fromObjects joins the roots of a scene into one hierarchy. Charts of the site layout (a chart manifest with a live directory) are converted when loaded into the same documents and objects (src/live/legacy.js): inline objects become objects of two serialized files, packed shader directories become Shader objects, references between scene roots become component references, and the clips' bindings become bindings tables. Such a chart draws exactly as before. Live options offer a note design or effect set when its product is loaded, and bar lines when the scene has the bar line view prefab. BREAKING CHANGE: the live renderers, note views and effects take LiveObjects and the product documents; the scene.json/notes.json inline formats and the livescene/livenotes directory lookups are no longer read directly. Co-Authored-By: Claude Opus 5.5 --- src/data/assets.js | 78 +++++ src/engine/anim.js | 68 ++++- src/engine/glsl.js | 82 ++++-- src/engine/particles.js | 105 +++++-- src/engine/postfx.js | 11 +- src/engine/prefab.js | 23 +- src/live/background.js | 23 +- src/live/canvas.js | 18 +- src/live/data.js | 47 ++- src/live/fx-effects.js | 137 ++++----- src/live/fx-ui.js | 92 +++--- src/live/fx.js | 25 +- src/live/intro.js | 100 ++++--- src/live/lane.js | 76 +++-- src/live/legacy.js | 456 +++++++++++++++++++++++++++-- src/live/noteview.js | 172 ++++++----- src/live/objects.js | 134 +++++++++ src/live/particles.js | 11 +- src/live/renderer.js | 51 +++- src/live/session.js | 46 +-- src/live/settings.js | 49 ++-- src/live/stage.js | 3 +- src/story/features/clips.js | 2 +- src/story/renderer.js | 2 +- src/story/simple/home/host.js | 2 +- tests/unit/live-legacy-fixture.mjs | 129 ++++++++ tests/unit/live-settings.test.mjs | 119 ++++---- tests/unit/live-store.test.mjs | 82 ++++-- tests/unit/read-set.test.mjs | 81 ++--- 29 files changed, 1639 insertions(+), 585 deletions(-) create mode 100644 src/live/objects.js create mode 100644 tests/unit/live-legacy-fixture.mjs diff --git a/src/data/assets.js b/src/data/assets.js index dd821bf..5a36a9a 100644 --- a/src/data/assets.js +++ b/src/data/assets.js @@ -38,6 +38,15 @@ export const refId = (v) => { return `${r.file}:${r.pathId}`; }; +// A byte array as an object stores it ({"$hex": ""}) -> Uint8Array. +export const hexBytes = (v) => { + const h = v && v.$hex; + if (typeof h !== "string" || h.length % 2 || /[^0-9a-f]/i.test(h)) throw new Error(`not a byte array: ${JSON.stringify(v)}`); + const out = new Uint8Array(h.length / 2); + for (let i = 0; i < out.length; i++) out[i] = parseInt(h.slice(i * 2, i * 2 + 2), 16); + return out; +}; + // the content encoding of an asset by its name: "gzip" (.gz), "br" (.br), or null (stored as it is) export const assetEncoding = (asset) => { const m = /\.(gz|br)$/.exec(String(asset)); @@ -79,6 +88,9 @@ export class AssetStore { this._objects.set(id, { class: o.class, name: o.name ?? null, files }); } this._parsed = new Map(); // "#" -> the parsed JSON of an object's file + this._sprites = new Map(); // id -> sprite() + this._meshes = new Map(); // id -> mesh() + this._contained = new Map(); // serialized file -> pathId -> contained() this.info = info; } @@ -309,6 +321,72 @@ export class AssetStore { return { id, name: t.m_Name, width: t.m_Width, height: t.m_Height, mipCount: t.m_MipCount, colorSpace: t.m_ColorSpace, settings: t.m_TextureSettings }; } + + // A Sprite as it is drawn from its render data (sprite.json: the typetree `sprite`, its render mesh `mesh`, the + // render data `renderData`): {id, name, texture (texture() of the page it is drawn from), rect, pivot, border, + // pixelsToUnits, offset, textureRect, textureRectOffset, atlasRectOffset, downscaleMultiplier, settingsRaw, + // uvTransform, vertices, uv, indices}. The texture coordinates of a vertex p are ((p.x * uvTransform.x + + // uvTransform.y) / width, (p.y * uvTransform.z + uvTransform.w) / height) of the page, in float32; vertices, uv and + // indices are null for a sprite without a render mesh. Made once and shared: callers do not modify it. + sprite(id) { + this.object(id, "Sprite"); + let s = this._sprites.get(id); + if (s) return s; + const j = this.objectJson(id), t = j.sprite, rd = j.renderData, ut = rd.uvTransform; + const tex = this.texture(this.ref(rd.texture, "Texture2D")), f = Math.fround; + const m = j.mesh; + if (m && m.submeshes.length !== 1) throw new Error(`sprite ${id} (${t.m_Name}): ${m.submeshes.length} submeshes`); + s = { id, name: t.m_Name, texture: tex, rect: t.m_Rect, pivot: t.m_Pivot, border: t.m_Border, + pixelsToUnits: t.m_PixelsToUnits, offset: t.m_Offset, textureRect: rd.textureRect, + textureRectOffset: rd.textureRectOffset, atlasRectOffset: rd.atlasRectOffset, + downscaleMultiplier: rd.downscaleMultiplier, settingsRaw: rd.settingsRaw, uvTransform: ut, + vertices: m ? m.positions : null, + uv: m ? m.positions.map((p) => [f(f(f(p[0] * ut.x) + ut.y) / tex.width), f(f(f(p[1] * ut.z) + ut.w) / tex.height)]) : null, + indices: m ? m.submeshes[0].flat() : null }; + this._sprites.set(id, s); + return s; + } + + // A Mesh in Unity's space, from its `arrays` file: {id, name, vertices, normals, tangents, colors, uv0 .. uv7 (each + // [[...]] per vertex, absent channels null), submeshes (the triangle indices of each submesh, flat), skin ({joints, + // weights}: four per vertex, or null), bindPoses (the typetree's m_BindPose)}. Made once and shared. + mesh(id) { + const o = this.object(id, "Mesh"); + let m = this._meshes.get(id); + if (m) return m; + const a = this.objectJson(id, "arrays"), t = this.objectJson(id); + m = { id, name: o.name, vertices: a.positions, submeshes: a.submeshes.map((s) => s.flat()), + skin: a.joints ? { joints: a.joints, weights: a.weights } : null, bindPoses: t.m_BindPose || [] }; + for (const k of ["normals", "tangents", "colors", "uv0", "uv1", "uv2", "uv3", "uv4", "uv5", "uv6", "uv7"]) m[k] = a[k] ?? null; + this._meshes.set(id, m); + return m; + } + + // controller.json: the AnimatorController's layers, state machines and parameters; `clips` are references + controller(id) { this.object(id, "AnimatorController"); return this.objectJson(id); } + + // The node of a prefab that a reference names when it is not an object of the store: a GameObject or a component + // inside a root GameObject's prefab.json of the same serialized file: {prefab (the root GameObject id), path (the + // node's path), component (the prefab.json component, null for a GameObject)}. + contained(v) { + const r = v && v.$ref; + if (!r || typeof r.file !== "string" || typeof r.pathId !== "string") throw new Error(`not a reference: ${JSON.stringify(v)}`); + let byId = this._contained.get(r.file); + if (!byId) { + byId = new Map(); + for (const [id, o] of this._objects) { + if (o.class !== "GameObject" || !id.startsWith(`${r.file}:`)) continue; + for (const n of this.prefab(id).nodes) { + byId.set(n.pathId, { prefab: id, path: n.path, component: null }); + for (const c of n.components) byId.set(c.pathId, { prefab: id, path: n.path, component: c }); + } + } + this._contained.set(r.file, byId); + } + const hit = byId.get(r.pathId); + if (!hit) throw new Error(`${r.file}:${r.pathId}: in no prefab of the store`); + return hit; + } } // The store of the session that draws into a WebGL context. Shader libraries and texture loads reach the files through diff --git a/src/engine/anim.js b/src/engine/anim.js index 613005b..c851df8 100644 --- a/src/engine/anim.js +++ b/src/engine/anim.js @@ -57,27 +57,45 @@ export class AnimClip { this.events = (desc.events || []).map((e) => ({ ...e, time: F(e.time) })); } - // raw Mecanim clip. The discrete curves supported are PPtr curves on SpriteRenderer.m_Sprite (the SpriteRenderer - // binding with attribute 0) in the streamed clip. Dense curves, other discrete curves and discrete curves outside - // the streamed clip are not implemented (raising keeps other data from being sampled wrongly). + // A raw Mecanim clip whose bindings carry their path and attribute ({path, typeID, class, attribute, curves, pptr}). + // The discrete curves supported are PPtr curves on SpriteRenderer.m_Sprite (the SpriteRenderer binding with + // attribute 0) in the streamed clip. Dense curves, other discrete curves and discrete curves outside the streamed + // clip are not implemented (raising keeps other data from being sampled wrongly). static fromMecanim(clip, key = clip.name) { - const name = key; + // a PPtr curve on m_Sprite is stored without attribute (the exporter keeps none for object curves) + return AnimClip._mecanim(clip, key, clip.bindings.map((b) => ({ + path: b.path, cls: b.class, typeID: b.typeID, curves: b.curves || 1, pptr: !!b.pptr, + attribute: b.pptr && b.attribute == null && !b.attributeCrc ? "m_Sprite" : b.attribute }))); + } + + // An AnimationClip object (clip.json: the raw Mecanim data and the bindings as stored) played by an Animator whose + // hierarchy names its bindings: `names[i]` = {path, attribute} of binding i in that hierarchy (a product's bindings + // table, AnimBindings); a path null binds nothing there. A binding animates `curves` consecutive curves (default 1). + static fromObject(clip, names, key = clip.name) { + if (!Array.isArray(names) || names.length !== clip.bindings.length) + throw new AnimError(`clip ${key}: ${names ? names.length : "no"} names for ${clip.bindings.length} bindings`); + return AnimClip._mecanim(clip, key, clip.bindings.map((b, i) => ({ + path: names[i].path, cls: b.class, typeID: b.typeID, attribute: names[i].attribute, curves: b.curves || 1, + pptr: !!b.pptr }))); + } + + static _mecanim(clip, name, named) { if (clip.startTime !== 0 || clip.cycleOffset) throw new AnimError(`clip ${name}: start time / cycle offset`); if (clip.dense.curveCount) throw new AnimError(`clip ${name}: dense curves not implemented`); if (clip.discreteCurveCount) throw new AnimError(`clip ${name}: discrete curves outside the streamed clip`); const nD = clip.streamed.discreteCurveCount || 0, objects = clip.pptrCurveMapping || []; const bindings = []; - for (const b of clip.bindings) { + for (const b of named) { if (b.pptr) { - if (b.typeID !== 212 || b.attribute != null || b.attributeCrc) - throw new AnimError(`clip ${name}: PPtr curve ${b.class}.${b.attribute ?? b.attributeCrc} not implemented`); - bindings.push({ path: b.path, cls: b.class, typeID: b.typeID, attr: "m_Sprite", pptr: true }); + if (b.typeID !== 212 || b.attribute !== "m_Sprite") + throw new AnimError(`clip ${name}: PPtr curve ${b.cls}.${b.attribute} not implemented`); + bindings.push({ path: b.path, cls: b.cls, typeID: b.typeID, attr: "m_Sprite", pptr: true }); continue; } - const comps = ANIM_COMPONENTS[b.curves || 1]; + const comps = ANIM_COMPONENTS[b.curves]; if (!comps) throw new AnimError(`clip ${name}: binding with ${b.curves} curves`); for (const c of comps) - bindings.push({ path: b.path, cls: b.class, typeID: b.typeID, attr: `${b.attribute}${c}` }); + bindings.push({ path: b.path, cls: b.cls, typeID: b.typeID, attr: `${b.attribute}${c}` }); } const nS = clip.streamed.curveCount + nD; if (bindings.length !== nS + clip.constant.length) throw new AnimError(`clip ${name}: binding count`); @@ -181,11 +199,11 @@ export const crc32 = (s) => { // (index), anyState: [transition], parameters: Map name -> {type, value}}. transition = {destination, duration, // offset, exitTime, hasExitTime, fixedDuration, canTransitionToSelf, conditions: [{mode, event, threshold}]}. export const AnimController = { - // raw AnimatorController: clips[i].clip keys into `clipOf(key)` -> AnimClip + // raw AnimatorController: `clipOf(entry, index)` -> AnimClip for each entry of its `clips` (a clip reference) fromMecanim(ctrl, clipOf) { if (ctrl.layers.length !== 1) throw new AnimError(`${ctrl.name}: ${ctrl.layers.length} layers`); const sm = ctrl.stateMachines[ctrl.layers[0].stateMachine]; - const clips = ctrl.clips.map((c) => clipOf(c.clip)); + const clips = ctrl.clips.map((c, i) => clipOf(c, i)); const states = sm.states.map((s) => { if (s.cycleOffset || s.mirror || s.speedParam || s.timeParam) throw new AnimError(`${ctrl.name}.${s.name}: state settings`); if (s.blendTrees.length > 1 || (s.blendTrees[0] && s.blendTrees[0].length !== 1)) @@ -200,6 +218,32 @@ export const AnimController = { }, }; +// The clip bindings of a product's documents: {prefab GameObject id: {Animator node path: {clip id: [{path, +// attribute}]}}}, the path and attribute each binding of a clip names when the Animator at that node plays it (paths +// relative to the Animator's GameObject; null: nothing of the hierarchy there). Several tables merge; an entry two +// tables both give must be the same. +export class AnimBindings { + constructor(tables = []) { + this.table = new Map(); // "\u0000\u0000" -> names + for (const t of tables) + for (const [prefab, animators] of Object.entries(t || {})) + for (const [path, clips] of Object.entries(animators)) + for (const [clip, names] of Object.entries(clips)) { + const k = `${prefab}\u0000${path}\u0000${clip}`, seen = this.table.get(k); + if (seen && JSON.stringify(seen) !== JSON.stringify(names)) + throw new AnimError(`bindings of clip ${clip} at ${path} of ${prefab} differ between documents`); + this.table.set(k, names); + } + } + + // the names of clip `clip` played by the Animator at `path` of the prefab (or scene root) `prefab` + names(prefab, path, clip) { + const n = this.table.get(`${prefab}\u0000${path}\u0000${clip}`); + if (!n) throw new AnimError(`no bindings of clip ${clip} for the Animator at ${path} of ${prefab}`); + return n; + } +} + // ------------------------------------------------------------------- Animator // The state time advances in the animation phase before sampling, so a state entered this frame is first sampled // after one advance. Update mode 0 (Normal): scaled delta time * Animator.speed. diff --git a/src/engine/glsl.js b/src/engine/glsl.js index d409ba2..7baaf07 100644 --- a/src/engine/glsl.js +++ b/src/engine/glsl.js @@ -315,12 +315,41 @@ export const applyState = (gl, s) => { export const SHADER_PLATFORM = "gles3", SHADER_PROGRAM_TYPE = "GLES3"; const RECORDS = Symbol("shader records"); -// Shader index + variant selection + program cache. The shaders are named by their Unity names: +// The Shader objects of a store by object id (each object's `json`: the parsed form, `index`: its variants, +// `program:/`: their GLSL; docs/data-format.md "Shaders"), and a name table {Unity name: object id} +// that answers lookups by name (Shader.Find). +class ShaderObjects { + constructor(assets, names) { + this.assets = assets; + this.names = names; + this.records = new Map(); + } + + get(key) { + const id = this.names.get(key) ?? key; + let r = this.records.get(id); + if (!r) { + if (!this.assets.hasObject(id)) return undefined; + this.assets.object(id, "Shader"); + const index = this.assets.objectJson(id, "index"); + r = { + name: index.name, + variants: index.variants.filter((v) => v.platform === SHADER_PLATFORM && v.type === SHADER_PROGRAM_TYPE) + .map((v) => ({ ...v, key: `${id}#${v.role}` })), + info: () => this.assets.objectJson(id), + text: (v) => this.assets.objectText(id, v.role), + }; + this.records.set(id, r); + } + return r; + } +} + +// Shader index + variant selection + program cache. A shader is named by its key: // new ShaderLib(gl, base, assets) a packed shader directory `base` (shaders.json + -// /gles3/*.glsl + .json) -// ShaderLib.fromObjects(gl, shaders, assets) the Shader objects of a shader table {name: object id}: each -// object's `json` (the parsed form), `index` (its variants) and -// `program:/` files (docs/data-format.md "Shaders") +// /gles3/*.glsl + .json): the key is the Unity name +// ShaderLib.fromObjects(gl, names, assets) the Shader objects of the store: the key is the object id, or a +// Unity name of the name table `names` {name: object id} export class ShaderLib { constructor(gl, base = "shaders", assets = assetsOf(gl)) { this.gl = gl; @@ -338,28 +367,30 @@ export class ShaderLib { this.states = new Map(); // "name|subShader|pass" -> {state, names, byValues}: passState per value set } - // The shaders of a shader table {Unity name: Shader object id}; a name must be the one its object's index states. - static fromObjects(gl, shaders, assets = assetsOf(gl)) { - const records = new Map(Object.entries(shaders).map(([name, id]) => { - assets.object(id, "Shader"); - const index = assets.objectJson(id, "index"); - if (index.name !== name) throw new Error(`shader ${id} is ${index.name}, the table names it ${name}`); - return [name, { - variants: index.variants.filter((v) => v.platform === SHADER_PLATFORM && v.type === SHADER_PROGRAM_TYPE) - .map((v) => ({ ...v, key: `${id}#${v.role}` })), - info: () => assets.objectJson(id), - text: (v) => assets.objectText(id, v.role), - }]; - })); + // The Shader objects of the store; `names` {Unity name: Shader object id} names some of them (a name must be the one + // its object's index states). + static fromObjects(gl, names = {}, assets = assetsOf(gl)) { + const table = new Map(Object.entries(names)); + const records = new ShaderObjects(assets, table); + for (const [name, id] of table) { + const r = records.get(id); + if (!r) throw new Error(`shader ${name}: object ${id} not found`); + if (r.name !== name) throw new Error(`shader ${id} is ${r.name}, the table names it ${name}`); + } return new ShaderLib(gl, { [RECORDS]: records }, assets); } + // the Unity name of the shader a key names + nameOf(key) { return this._record(key).name ?? key; } + + _record(key) { + const rec = this.index.get(key); + if (!rec) throw new Error(`shader not packed: ${key}`); + return rec; + } + info(name) { - if (!this.parsed.has(name)) { - const rec = this.index.get(name); - if (!rec) throw new Error(`shader not packed: ${name}`); - this.parsed.set(name, rec.info()); - } + if (!this.parsed.has(name)) this.parsed.set(name, this._record(name).info()); return this.parsed.get(name); } @@ -372,9 +403,8 @@ export class ShaderLib { return prog; } - _program(name, pass, keywords, subShader) { - const rec = this.index.get(name); - if (!rec) throw new Error(`shader not packed: ${name}`); + _program(key, pass, keywords, subShader) { + const rec = this._record(key), name = rec.name ?? key; const vs = rec.variants.filter((v) => v.subShader === subShader && v.pass === pass); if (!vs.length) throw new Error(`${name}: no GLES3 variants for subshader ${subShader} pass ${pass}`); const known = new Set(vs.flatMap((v) => v.keywords)); diff --git a/src/engine/particles.js b/src/engine/particles.js index 6fa3f85..e54a3ba 100644 --- a/src/engine/particles.js +++ b/src/engine/particles.js @@ -1,6 +1,6 @@ import { Gradient } from "./anim.js"; import { F, join } from "./core.js"; -import { assetsOf } from "../data/assets.js"; +import { assetsOf, hexBytes } from "../data/assets.js"; import { ShaderLib, applyState } from "./glsl.js"; import { mat4, quat } from "./math.js"; import { UnityRandom } from "./random.js"; @@ -471,7 +471,7 @@ export class FxParticleSystem { const r = rendererComp && rendererComp.m_Enabled ? rendererComp : null; this._rendererComp = rendererComp || null; this._rendererConfig = undefined; - this.renderer = r ? this._alignSettings(FxParticleSystem._rendererSettings(r)) : null; + this.renderer = r ? this._alignSettings(this._rendererSettings(r)) : null; this.sortingOrder = r ? r.m_SortingOrder | 0 : 0; // LiveParticleOrderInLayerSetter may overwrite this.renderUnsupported = []; if (this.renderer) { @@ -498,15 +498,25 @@ export class FxParticleSystem { this._pausedFrom = null; } - // ParticleSystemRenderer serialized fields -> renderer settings - static _rendererSettings(r) { + // a mesh value of the data (ParticleSystemRenderer.m_Mesh, ShapeModule.m_Mesh): {vertices, normals, uv0, submeshes} + // through the materials' source (FxMaterials.mesh), the value itself without materials + _mesh(v) { return this.materials && v ? this.materials.mesh(v) : v || null; } + + _rendererSettings(r) { + const bytes = (v) => (this.materials ? this.materials.bytes(v) : v); + return { ...FxParticleSystem._rendererSettings(r, bytes), mesh: this._mesh(r.m_Mesh) }; + } + + // ParticleSystemRenderer serialized fields -> renderer settings (`mesh` as stored); bytes(v): a byte array of the + // data as numbers + static _rendererSettings(r, bytes = (v) => v) { const v3 = (o) => [F(o.x), F(o.y), F(o.z)]; return { renderMode: r.m_RenderMode, alignment: r.m_RenderAlignment, pivot: v3(r.m_Pivot), flip: v3(r.m_Flip), minSize: F(r.m_MinParticleSize), maxSize: F(r.m_MaxParticleSize), sortMode: r.m_SortMode, sortingLayer: r.m_SortingLayer | 0, sortingLayerID: r.m_SortingLayerID | 0, normalDirection: F(r.m_NormalDirection), maskInteraction: r.m_MaskInteraction | 0, sortingFudge: F(r.m_SortingFudge || 0), - streams: r.m_UseCustomVertexStreams ? r.m_VertexStreams.slice() : [0, 1, 3, 4], + streams: r.m_UseCustomVertexStreams ? Array.from(bytes(r.m_VertexStreams)) : [0, 1, 3, 4], mesh: r.m_Mesh || null, materials: (r.m_Materials || []).filter(Boolean), lengthScale: F(r.m_LengthScale ?? 2), velocityScale: F(r.m_VelocityScale || 0), cameraVelocityScale: F(r.m_CameraVelocityScale || 0), freeformStretching: !!r.m_FreeformStretching, @@ -555,7 +565,7 @@ export class FxParticleSystem { // renderer settings also for a disabled renderer (null without a ParticleSystemRenderer component) get rendererConfig() { if (this._rendererConfig === undefined) - this._rendererConfig = this.renderer || (this._rendererComp ? this._alignSettings(FxParticleSystem._rendererSettings(this._rendererComp)) : null); + this._rendererConfig = this.renderer || (this._rendererComp ? this._alignSettings(this._rendererSettings(this._rendererComp)) : null); return this._rendererConfig; } @@ -937,7 +947,7 @@ export class FxParticleSystem { // m_MeshMaterialIndex with m_UseMeshMaterialIndex), with cumulative areas and edge lengths for the placement modes // (placementMode: 0 Vertex, 1 Edge, 2 Triangle). Spawn modes other than Random and mesh colours raise FxError. _meshShape(S) { - const m = S.m_Mesh; + const m = this._mesh(S.m_Mesh); if (!m || !m.vertices || !m.vertices.length) { this.unsupported.push("shape mesh without mesh data"); return; } if (S.m_MeshSpawn && (S.m_MeshSpawn.mode | 0) !== 0) this.unsupported.push(`shape mesh spawn mode ${S.m_MeshSpawn.mode}`); if (S.m_UseMeshColors && m.colors) this.unsupported.push("shape mesh colours"); @@ -1578,11 +1588,19 @@ export class FxParticleSystem { }; // ------------------------------------------------------------------------------------------------ materials -// Entries of a renderer's m_Materials (exported: material, shader.shader, keywords, renderQueue, textures -// {name: {texture: desc | null, scale, offset}}, floats, ints, colors) drawn with the game's HLSLcc GLES3 programs -// through ShaderLib(gl, "/shaders"), resolved as for the other live materials: shader -// property defaults under the material values, the keywords select the variant (pass 0 of subshader 0), render queue -// -1 -> the shader's Queue tag. Used by the particle renderers and by the effect SpriteRenderers (fx-effects.js). +// The materials of renderers (m_Materials entries) drawn with the game's HLSLcc GLES3 programs, resolved as for the +// other live materials: shader property defaults under the material values, the keywords select the variant (pass 0 +// of subshader 0), render queue -1 -> the shader's Queue tag. Used by the particle renderers and by the effect +// SpriteRenderers (fx-effects.js). The values come from one of two sources: +// new FxMaterials(gl, base) inline records of a data file: a material {material, shader: {shader: name}, +// keywords, renderQueue, textures {name: {texture: descriptor | null, scale, +// offset}}, floats, ints, colors}, textures as exported descriptors under `base`, +// sprites, meshes and byte arrays inline, shaders packed under /shaders +// FxMaterials.fromObjects(gl, objs) references to the objects of a store, read through `objs`: material(v) -> +// {id, material, shader (Shader object id), keywords, renderQueue, textures {name: +// {texture: texture record | null, scale, offset}}, floats, ints, colors}, +// texture(id) -> Promise, sprite(v), mesh(v), shaderInfo(id), and its +// ShaderLib `lib` (by Shader object id) // Uniform sheets per draw, highest priority first: the draw's sheet (MaterialPropertyBlock-like), ctx.perObject(M), // the material, the shader defaults, ctx.globals (per camera), then FxMaterials.ENGINE (values no one sets). // ENGINE: _TextureSampleAdd / _ClipRect / _UIMaskSoftnessX|Y are set by uGUI's CanvasRenderer only; 0 outside a canvas. @@ -1593,9 +1611,10 @@ export class FxParticleSystem { // (m_EnableGPUInstancing) is not used: the shaders have no procedural-instancing variant, so Unity builds the vertices // on the CPU as here. export class FxMaterials { - constructor(gl, base = "livenotes") { + constructor(gl, base = "livenotes", objects = null) { this.gl = gl; this.base = base; + this.objects = objects; this.mats = new Map(); this.lib = null; this.loaded = false; @@ -1605,17 +1624,41 @@ export class FxMaterials { this._tex = new Map(); } - // parsed shader JSON through assets (works with gl = null when assets can read the packed files) - shaderInfo(name) { - if (!this._info.has(name)) { + static fromObjects(gl, objects) { return new FxMaterials(gl, null, objects); } + + // the material a value of the data names: {key, material, shader, keywords, renderQueue, textures, floats, ints, + // colors} (key: one per material of the source) + record(v) { + if (!v) throw new FxError("material record missing"); + if (this.objects) { const m = this.objects.material(v); return { ...m, key: m.id }; } + if (!v.shader) throw new FxError("material record missing"); + return { ...v, key: `${v.material}\u0000${v.shader.shader}`, shader: v.shader.shader }; + } + + // a sprite value of the data: {name, texture, rect, border, pivot, pixelsToUnits, vertices, uv, indices, settingsRaw} + sprite(v) { + if (!v) return null; + return this.objects ? this.objects.sprite(v) : { ...v, name: v.sprite }; + } + + // a mesh value of the data: {vertices, normals, uv0, submeshes (flat triangle indices)} + mesh(v) { return this.objects ? this.objects.mesh(v) : v; } + + // a byte array of the data ({"$hex"} in an object, numbers in a packed directory) + bytes(v) { return this.objects ? hexBytes(v) : v; } + + // parsed shader JSON (works with gl = null when the store can read the shader files) + shaderInfo(shader) { + if (this.objects) return this.objects.shaderInfo(shader); + if (!this._info.has(shader)) { const assets = assetsOf(this.gl); if (!this._index) this._index = new Map(assets.json(join(this.base, "shaders", "shaders.json")).map((r) => [r.name, r])); - const rec = this._index.get(name); - if (!rec) throw new FxError(`shader not packed: ${name}`); - this._info.set(name, assets.json(join(this.base, "shaders", rec.parsed))); + const rec = this._index.get(shader); + if (!rec) throw new FxError(`shader not packed: ${shader}`); + this._info.set(shader, assets.json(join(this.base, "shaders", rec.parsed))); } - return this._info.get(name); + return this._info.get(shader); } // ShaderLab "Queue" tag -> render queue (Background 1000, Geometry 2000, AlphaTest 2450, GeometryLast 2500, @@ -1636,13 +1679,12 @@ export class FxMaterials { return FxMaterials.queueFromTag(q ? q[1] : undefined); } - get(record) { - if (!record || !record.shader) throw new FxError("material record missing"); - const key = `${record.material}\u0000${record.shader.shader}`; - let m = this.mats.get(key); + get(v) { + const record = this.record(v); + let m = this.mats.get(record.key); if (!m) { m = new FxMaterial(this, record); - this.mats.set(key, m); + this.mats.set(record.key, m); if (this.loaded && this.gl) m._pending = m._load(); } return m; @@ -1655,7 +1697,7 @@ export class FxMaterials { const gl = this.gl; if (!gl) return; if (!this.lib) { - this.lib = new ShaderLib(gl, join(this.base, "shaders")); + this.lib = this.objects ? this.objects.lib : new ShaderLib(gl, join(this.base, "shaders")); this.solid = { white: GLTex.solid(gl, [255, 255, 255, 255], "white"), black: GLTex.solid(gl, [0, 0, 0, 255], "black"), gray: GLTex.solid(gl, [128, 128, 128, 255], "gray"), bump: GLTex.solid(gl, [128, 128, 255, 255], "bump"), @@ -1667,9 +1709,11 @@ export class FxMaterials { for (const m of [...this.mats.values()]) await (m._pending || (m._pending = m._load())); } - texture(desc) { - if (!this._tex.has(desc.texture)) this._tex.set(desc.texture, GLTex.load(this.gl, this.base, desc)); - return this._tex.get(desc.texture); + // the GL texture of a material's or sprite's texture (an exported descriptor, or a texture record of the store) + texture(t) { + if (this.objects) return this.objects.texture(t.id); + if (!this._tex.has(t.texture)) this._tex.set(t.texture, GLTex.load(this.gl, this.base, t)); + return this._tex.get(t.texture); } }; @@ -1677,11 +1721,12 @@ FxMaterials.ENGINE = { _TextureSampleAdd: [0, 0, 0, 0], _ClipRect: [0, 0, 0, 0], FxMaterials.ENGINE_STATE = { unity_GUIZTestMode: 4 }; export class FxMaterial { + // record: FxMaterials.record constructor(owner, record) { this.owner = owner; this.record = record; this.name = record.material; - this.shader = record.shader.shader; + this.shader = record.shader; this.keywords = record.keywords || []; this.ready = false; this._queue = undefined; diff --git a/src/engine/postfx.js b/src/engine/postfx.js index 78862d6..bba075c 100644 --- a/src/engine/postfx.js +++ b/src/engine/postfx.js @@ -183,12 +183,13 @@ export class URPPost { } // ------------------------------------------------------------------ resources - // film grain textures by FilmGrain.type (the renderer's post data); the Uber shader samples them with its inline - // sampler_LinearRepeat - static async loadGrain(gl, base, descs) { + // film grain textures by FilmGrain.type (the renderer's post data), each loaded by `load(item)`; the Uber shader + // samples them with its inline sampler_LinearRepeat + static async loadGrain(gl, items, load) { const out = []; - for (const d of descs) { - const t = await GLTex.load(gl, base, d); + for (const d of items) { + const t = await load(d); + gl.bindTexture(gl.TEXTURE_2D, t.glTexture); GLTex.sampler(gl, { m_FilterMode: 1, m_WrapU: 0, m_WrapV: 0 }, 1); out.push(t); } diff --git a/src/engine/prefab.js b/src/engine/prefab.js index 9ad1755..f9776ce 100644 --- a/src/engine/prefab.js +++ b/src/engine/prefab.js @@ -3,12 +3,16 @@ import { Transform } from "./math.js"; // Prefab: Transform hierarchy and component lookup over an exported prefab node list // (hierarchy order, local TRS, components in GameObject order). -// A component's class: a MonoBehaviour's script class (the last part of a prefab.json component's `script`, -// "assembly|namespace|class"; the `class` of a component in a chart or story file), else the component's type. -export const componentClass = (c) => - (typeof c.script === "string" ? c.script.slice(c.script.lastIndexOf("|") + 1) : (c.class || c.type)); +// the class of a script "assembly|namespace|class" (a prefab.json component's `script`, a MonoBehaviour object's +// `$script`) +export const scriptClass = (script) => script.slice(script.lastIndexOf("|") + 1); + +// A component's class: a MonoBehaviour's script class (prefab.json `script`; the `class` of a component in a story +// file), else the component's type. +export const componentClass = (c) => (typeof c.script === "string" ? scriptClass(c.script) : (c.class || c.type)); export class Prefab { + // exported: {key, nodes} constructor(exported) { this.key = exported.key; this.nodes = new Map(); // path -> {node, transform} @@ -28,6 +32,17 @@ export class Prefab { } } + // The prefab.json roots `ids` of the store (AssetStore.prefab) as one hierarchy: a prefab, or the roots of a scene + // (their node paths start with the root's name). `ids` keeps the root ids. + static fromObjects(assets, ids, key = ids.join(" ")) { + const p = new Prefab({ key, nodes: ids.flatMap((id) => assets.prefab(id).nodes) }); + p.ids = ids.slice(); + return p; + } + + // the `fields` of the unique component of class `cls` on `path` (prefab.json) + fields(path, cls) { return this.component(path, cls).fields; } + get root() { if (this.roots.length !== 1) throw new Error(`${this.key}: ${this.roots.length} roots`); return this.roots[0]; diff --git a/src/live/background.js b/src/live/background.js index b0ecf33..1c16034 100644 --- a/src/live/background.js +++ b/src/live/background.js @@ -23,8 +23,9 @@ export const LiveLightWeightBackground = { ROOT: "Live/RenderCanvas/LightWeightCompositeRoot", LAYERS: ["Layer_JacketLarge", "Layer_BaseWhite", "Layer_JacketSmall", "Layer_StageBg"], // CreateCompositedRenderTexture array order - // normalised layer rects in the root (pure) - layers(scene, prefab) { + // normalised layer rects in the root (pure); objects: LiveObjects, sprites: {stage, jacket} (the band's LightWeight + // background sprite, the music's jacket sprite: object ids) + layers(objects, prefab, sprites) { const root = prefab.nodes.get(LiveLightWeightBackground.ROOT).node; const rw = root.rect.m_SizeDelta.x, rh = root.rect.m_SizeDelta.y; // root anchors 0.5/0.5: rect = sizeDelta // the root's rect is its sizeDelta because both its anchors are 0.5 (the RenderCanvas rect does not enter); @@ -34,7 +35,7 @@ export const LiveLightWeightBackground = { width: W, height: H, layers: LiveLightWeightBackground.LAYERS.map((name) => { const path = `${LiveLightWeightBackground.ROOT}/${name}`; - const n = prefab.nodes.get(path).node, img = prefab.component(path, "Image"); + const n = prefab.nodes.get(path).node, img = prefab.fields(path, "Image"); const r = n.rect, s = n.localScale; if (r.m_AnchorMin.x !== 0.5 || r.m_AnchorMax.x !== 0.5 || r.m_AnchorMin.y !== 0.5 || r.m_AnchorMax.y !== 0.5) throw new Error(`${path}: anchors`); @@ -43,18 +44,18 @@ export const LiveLightWeightBackground = { const x0 = F(F(F(cx - F(w / 2)) + F(rw / 2)) / rw), y0 = F(F(F(cy - F(h / 2)) + F(rh / 2)) / rh); const x1 = F(F(F(cx + F(w / 2)) + F(rw / 2)) / rw), y1 = F(F(F(cy + F(h / 2)) + F(rh / 2)) / rh); // texture: jacket (MusicJacket), the Image sprite's texture (whole atlas page for BaseWhite), stage sprite - const tex = name === "Layer_StageBg" ? scene.sprites.lightweightBackground.texture - : name.startsWith("Layer_Jacket") ? scene.sprites.jacket.texture - : img.m_Sprite ? img.m_Sprite.texture : null; + const sprite = name === "Layer_StageBg" ? sprites.stage : name.startsWith("Layer_Jacket") ? sprites.jacket : img.m_Sprite; + const tex = sprite ? objects.sprite(sprite).texture : null; const c = img.m_Color; return { name, tint: [c.r, c.g, c.b, c.a], rectMin: [x0, y0, 0, 0], rectMax: [x1, y1, 0, 0], texture: tex }; }), }; }, - // Builds blur_result; returns an GLTarget (RGB565, bilinear, clamp). `vao` = an empty VAO. - async build(gl, lib, scene, prefab, base, vao) { - const L = LiveLightWeightBackground.layers(scene, prefab), W = L.width, H = L.height; + // Builds blur_result; returns an GLTarget (RGB565, bilinear, clamp). `vao` = an empty VAO. The layer textures are + // uploaded for the build and deleted after it. + async build(gl, objects, prefab, sprites, vao) { + const lib = objects.lib, L = LiveLightWeightBackground.layers(objects, prefab, sprites), W = L.width, H = L.height; const composite = new GLTarget(gl, W, H, { label: "composite_bg" }); // ARGB32 const R565 = { internal: gl.RGB565, format: gl.RGB, type: gl.UNSIGNED_SHORT_5_6_5 }; const result = new GLTarget(gl, W, H, { ...R565, label: "blur_result" }); @@ -91,8 +92,8 @@ export const LiveLightWeightBackground = { if (!(l.tint[3] > 0)) continue; let t = white; if (l.texture) { - if (!textures.has(l.texture.texture)) textures.set(l.texture.texture, await GLTex.load(gl, base, l.texture)); - t = textures.get(l.texture.texture); + if (!textures.has(l.texture.id)) textures.set(l.texture.id, await GLTex.loadObject(gl, l.texture.id, objects.assets)); + t = textures.get(l.texture.id); } blit("Hidden/CompositeRectBlit", 0, { _MainTex: t, _Tint: l.tint, _RectMin: l.rectMin, _RectMax: l.rectMax }, composite); } diff --git a/src/live/canvas.js b/src/live/canvas.js index ba3a67b..e01d3ba 100644 --- a/src/live/canvas.js +++ b/src/live/canvas.js @@ -1,5 +1,4 @@ import { F } from "../engine/core.js"; -import { GLTex } from "../engine/texture.js"; import { UIDraw, UIImage, UILayout, UIMesh, UINode, UISprite, uiColor32 } from "../engine/ugui.js"; import { LiveCopyProgram, LiveLightWeightBackground } from "./background.js"; import { LiveLane, LiveLaneLayout } from "./lane.js"; @@ -29,7 +28,7 @@ export class LiveRenderCanvas { constructor(renderer) { this.r = renderer; this.gl = renderer.gl; this.lib = renderer.lib; this.prefab = renderer.prefab; this.scene = renderer.scene; this.ROOT = "Live/RenderCanvas"; - this.canvasComp = this.prefab.component(this.ROOT, "Canvas"); + this.canvasComp = this.prefab.fields(this.ROOT, "Canvas"); this.sortingOrder = LiveLane.sortingOrder(this.prefab, this.ROOT, this.canvasComp.m_SortingOrder); // UINode tree of the canvas (CanvasGroup components folded into the records UINode reads) const nodes = new Map(); @@ -37,7 +36,7 @@ export class LiveRenderCanvas { if (path !== this.ROOT && !path.startsWith(`${this.ROOT}/`)) continue; const cg = e.node.components.find((c) => c.type === "CanvasGroup"); const parent = path === this.ROOT ? null : nodes.get(path.slice(0, path.lastIndexOf("/"))); - nodes.set(path, new UINode({ ...e.node, canvasGroup: cg }, parent)); + nodes.set(path, new UINode({ ...e.node, canvasGroup: cg ? cg.fields : null }, parent)); } this.nodes = nodes; this.root = nodes.get(this.ROOT); @@ -50,7 +49,7 @@ export class LiveRenderCanvas { for (const k of ["LightWeightBackgroundStageImage", "LightWeightBackgroundShadowImage"]) n(k).activeSelf = true; for (const k of ["LightWeightCompositeRoot", "LightWeightBackgroundSuppertCardImage", "MVImage"]) n(k).activeSelf = false; this.configure(renderer.settings); - const comp = (node, cls) => this.prefab.component(node.path, cls); + const comp = (node, cls) => this.prefab.fields(node.path, cls); this.graphics = new Map([ [this.stage, { kind: "raw", g: comp(this.stage, "RawImage") }], [this.shadow, { kind: "image", g: comp(this.shadow, "Image") }], @@ -61,7 +60,8 @@ export class LiveRenderCanvas { } _material(m) { // m_Material null -> Default UI Material (UI/Default, property defaults) - const mat = m || { material: "Default UI Material", shader: { shader: "UI/Default" }, keywords: [], floats: {}, colors: {} }; + const mat = m ? this.r.objects.material(m) + : { material: "Default UI Material", shader: "UI/Default", keywords: [], floats: {}, colors: {} }; return liveMaterialSheets(this.lib, mat, this.r.tex); } @@ -77,10 +77,10 @@ export class LiveRenderCanvas { async load() { const gl = this.gl; - this.bg = await LiveLightWeightBackground.build(gl, this.lib, this.scene, this.prefab, this.r.base, this.r.vao); - const sp = this.graphics.get(this.shadow).g.m_Sprite; - const atlas = await GLTex.load(gl, this.r.base, sp.texture); - this.shadowSprite = new UISprite(sp.sprite, { ...sp, pixelsPerUnit: sp.pixelsToUnits }, atlas); + this.bg = await LiveLightWeightBackground.build(gl, this.r.objects, this.prefab, this.r.sprites, this.r.vao); + const sp = this.r.objects.sprite(this.graphics.get(this.shadow).g.m_Sprite); + const atlas = await this.r.objects.texture(sp.texture.id); + this.shadowSprite = new UISprite(sp.name, { ...sp, pixelsPerUnit: sp.pixelsToUnits }, atlas); this.buf = { vao: gl.createVertexArray(), vbo: gl.createBuffer(), ibo: gl.createBuffer() }; for (const e of this.graphics.values()) e.mat = this._material(e.g.m_Material); // AspectRatioFitter.aspectRatio = (float)tex.width / (float)tex.height of the composite (FullInitialize) diff --git a/src/live/data.js b/src/live/data.js index c6fe159..36d94c3 100644 --- a/src/live/data.js +++ b/src/live/data.js @@ -1,14 +1,17 @@ import { LIVE_PRODUCT, productDir } from "./store.js"; // LiveData: the documents of one live in its store (loadLiveStore, or a chart store read through legacy.js): -// live live.json of the live product: the chart, the score files, the parts +// live live.json of the live product: the chart, the score files, the band, the parts // score(m) the converted score (notes.json; notes.mirror.json with the MirrorChart option) // audio() the sound data of the live: the sounds of a live.sounds product and the music of the live.music part -// sceneDoc, noteAssets the live scene and the note assets (livescene/scene.json, livenotes/notes.json) +// scene, notes scene.json of the live.scene part, notes.json of the live.notes part +// noteskin(n), noteeffect(n) noteskin.json / noteeffect.json of the note skin / effect set named n (default: the +// part's); noteskins, noteeffects: the names of those loaded (the parts and the variants) // A product's files are under its directory in the store (store.js productDir). export const LIVE_JSON_FORMAT = 1, SOUNDS_FORMAT = 1, MUSIC_FORMAT = 1; -const SOUNDS = "live.sounds"; +export const SCENE_FORMAT = 1, NOTES_FORMAT = 1, NOTESKIN_FORMAT = 1, NOTEEFFECT_FORMAT = 1; +const SOUNDS = "live.sounds", NOTESKIN = "live.noteskin", NOTEEFFECT = "live.noteeffect"; const checkFormat = (doc, format, where) => { if (!doc || doc.format !== format) throw new Error(`${where}: format ${doc && doc.format} (${format} expected)`); @@ -25,11 +28,46 @@ export class LiveData { this.products = info.products; this.live = checkFormat(assets.json(`${this.dir}/live.json`), LIVE_JSON_FORMAT, `${this.dir}/live.json`); this.parts = Object.fromEntries(Object.entries(this.live.parts).map(([part, p]) => [part, productDir(p)])); + this._docs = new Map(); } // the JSON file `file` of the product in directory `dir` json(dir, file) { return this.assets.json(`${dir}/${file}`); } + // a document of a product, read once (shared: callers do not modify it) + _doc(dir, file, format) { + const k = `${dir}/${file}`; + if (!this._docs.has(k)) this._docs.set(k, checkFormat(this.json(dir, file), format, k)); + return this._docs.get(k); + } + + get scene() { return this._doc(this.parts.scene, "scene.json", SCENE_FORMAT); } + get notes() { return this._doc(this.parts.notes, "notes.json", NOTES_FORMAT); } + + // the subject of the loaded product of `product` in directory `dir` + _subject(product, dir) { + const hit = Object.entries(this.products[product] || {}).find(([, d]) => d === dir); + if (!hit) throw new Error(`${dir}: not a loaded ${product} product`); + return hit[0]; + } + + get noteskins() { return Object.keys(this.products[NOTESKIN] || {}); } + get noteeffects() { return Object.keys(this.products[NOTEEFFECT] || {}); } + + // the note skin `name` (MasterLiveNoteSkin._assetName, notes.json settings.skins); default: the live's part + noteskin(name = this._subject(NOTESKIN, this.parts.noteskin)) { + const dir = (this.products[NOTESKIN] || {})[name]; + if (!dir) throw new Error(`note skin ${name}: not loaded`); + return this._doc(dir, "noteskin.json", NOTESKIN_FORMAT); + } + + // the note effect set `name` (MasterLiveNoteEffectSkin._assetName, notes.json settings.effects); default: the part + noteeffect(name = this._subject(NOTEEFFECT, this.parts.noteeffect)) { + const dir = (this.products[NOTEEFFECT] || {})[name]; + if (!dir) throw new Error(`note effect set ${name}: not loaded`); + return this._doc(dir, "noteeffect.json", NOTEEFFECT_FORMAT); + } + // the converted score; `mirror`: the one converted with MirrorChart (live.json notesMirror) score(mirror = false) { const f = mirror ? this.live.notesMirror : this.live.notes; @@ -58,7 +96,4 @@ export class LiveData { } return { ...doc, sounds, music: music.music }; } - - get sceneDoc() { return this.assets.json("livescene/scene.json"); } - get noteAssets() { return this.assets.json("livenotes/notes.json"); } } diff --git a/src/live/fx-effects.js b/src/live/fx-effects.js index 7a1b178..86c556f 100644 --- a/src/live/fx-effects.js +++ b/src/live/fx-effects.js @@ -1,7 +1,7 @@ -import { AnimClip, AnimController, AnimTargets, Animator } from "../engine/anim.js"; +import { AnimTargets, Animator } from "../engine/anim.js"; import { F } from "../engine/core.js"; import { Transform, mat4 } from "../engine/math.js"; -import { Prefab } from "../engine/prefab.js"; +import { Prefab, componentClass } from "../engine/prefab.js"; import { FxParticleSystem, PS_STOP } from "../engine/particles.js"; // Note effects, slide hold loop and lane effects of the live chart player: FxPrefab (one effect prefab instance), @@ -22,9 +22,11 @@ import { FxParticleSystem, PS_STOP } from "../engine/particles.js"; // UpdateFrame; LiveGameLaneEffectBase.Initialize / SetWidth / Play / // Stop / IsPlaying; Live{Renderer,Particle}OrderInLayerSetter.Awake, // LiveOrderInLayerUtility.GetOrderInLayer. -// - Data: livenotes/notes.json (assets[...] prefab node lists with raw components, clips, controllers, -// LiveNoteEffectAssetSettings, LiveLaneEffectAssetSettings); score/.notes.json; -// constants of the scene [EmbScene/Live] quoted below (as in livescene/scene.json). +// - Data: the live's objects (LiveObjects): the effect set's LiveNoteEffectAssetSettings (noteeffect.json +// settings) and the lane effects' LiveLaneEffectAssetSettings (notes.json laneEffect), each field a reference to +// the LiveNoteEffectAsset / LiveLaneEffectAsset of an effect prefab; the prefabs' prefab.json, controllers and +// clips (bindings named by the documents' bindings tables); the score; constants of the scene [EmbScene/Live] +// quoted below. // - Engine behaviour that is native (Animator, AnimationEvent dispatch, SpriteRenderer mesh generation, GameObject // activation) follows Unity's documented behaviour and is marked `// ENGINE:`. // @@ -35,8 +37,9 @@ import { FxParticleSystem, PS_STOP } from "../engine/particles.js"; export class FxEffectsError extends Error {}; // ------------------------------------------------------------------------------------------ scene values -// [EmbScene/Live] values the effect views read, taken from livescene/scene.json (parsed by the caller): the lane -// width, the judgement root, and the effect containers in their serialized order. Which LiveNoteEffectAssetSettings / +// [EmbScene/Live] values the effect views read, taken from the scene's hierarchy (a Prefab of the scene roots as +// loaded, before any animation): the lane width, the judgement root, and the effect containers in their serialized +// order. Which LiveNoteEffectAssetSettings / // LiveLaneEffectAssetSettings field a container type uses is code: LiveAllNoteEffectViewContainer.GetEffectAsset // (FX_NOTE_FIELD) and the lane settings fields per LiveLaneEffectType (FX_LANE_FIELD). export const FX_NOTE_FIELD = { 10: "NormalEffect", 20: "SlideEffect", 21: "SlideLoopConnectEffect", 22: "SlideLoopEffect", @@ -44,12 +47,11 @@ export const FX_NOTE_FIELD = { 10: "NormalEffect", 20: "SlideEffect", 21: "Slide export const FX_LANE_FIELD = { 2: "InVainEffect", 10: "NormalEffect", 20: "SlideEffect", 30: "FlickEffect", 31: "FlickLeftEffect", 32: "FlickRightEffect" }; export const FxScene = { - read(scene) { - const byPath = new Map(scene.scene.nodes.map((n) => [n.path, n])); + read(pf) { const comp = (path, cls) => { - const n = byPath.get(path), c = n && n.components.find((x) => (x.class || x.type) === cls); + const c = pf.nodes.has(path) ? pf.components(path, cls)[0] : null; if (!c) throw new FxEffectsError(`scene: no ${cls} on ${path}`); - return c; + return c.fields; }; const field = (table, type, path) => { if (!(type in table)) throw new FxEffectsError(`scene: ${path}: effect type ${type} has no settings field`); @@ -59,7 +61,6 @@ export const FxScene = { // LiveNoteJudgementRoot3D.Initialize, LiveAllNoteEffectViewContainer.Initialize (baseLaneWidth) and the lane effects. const laneWidth = comp("LiveGameView/root/LiveGameLane", "LiveLaneView")._laneWidth; // LiveGameView/root/LiveGameLane/judgement_root (local (0, 0, 9.62) under identity parents) - const pf = new Prefab({ key: "scene", nodes: scene.scene.nodes }); const judgementRoot = pf.transform("LiveGameView/root/LiveGameLane/judgement_root").localToWorld(); // LiveAllNoteEffectViewContainer._containers -> LiveGameNoteEffectContainer (_effectType, capacities) const noteContainers = comp("LiveGameView/root/LiveGameNoteEffectView/container_root", "LiveAllNoteEffectViewContainer") @@ -119,7 +120,7 @@ export const FxLiveMath = { // m_Color, material = m_Materials[0] with the sprite's texture as _MainTex. A program with the per-draw sprite // constants (URP 2D Sprite-Unlit-Default: unity_SpriteColor, unity_SpriteProps) takes the colour and the flip there, // over white vertices at their unflipped positions (LiveLane.spriteItem, the stage background sprite). -// drawMode 0 (Simple): the sprite's own mesh (exported m_Sprite.vertices / uv / indices, units relative to the pivot). +// drawMode 0 (Simple): the sprite's own mesh (its render mesh vertices / uv / indices, units relative to the pivot). // drawMode 1 (Sliced): a 9-slice of m_Size: outer rectangle (-pivot * size .. (1 - pivot) * size), borders // (m_Sprite.border px / pixelsToUnits: x left, y bottom, z right, w top); UVs of the sprite rect in its texture. // drawMode 2 (Tiled) is not used by any effect prefab and raises. @@ -136,8 +137,9 @@ export const FxLiveMath = { // ENGINE: a program with the per-draw sprite constants gets white, unflipped vertices; colour and flip come per draw. // (unity_SpriteColor = SpriteRenderer.color, unity_SpriteProps = (flipX ? -1 : 1, flipY ? -1 : 1, 0, 0)) export class FxSpriteRenderer { + // comp: the SpriteRenderer's fields; its sprite and material through `materials` (FxMaterials) constructor(comp, transform, { materials = null, name = "" } = {}) { - const sp = comp.m_Sprite; + const sp = materials ? materials.sprite(comp.m_Sprite) : comp.m_Sprite && { ...comp.m_Sprite, name: comp.m_Sprite.sprite }; if (!sp) throw new FxEffectsError(`${name}: SpriteRenderer without sprite`); if (comp.m_DrawMode !== 0 && comp.m_DrawMode !== 1) throw new FxEffectsError(`${name}: SpriteRenderer drawMode ${comp.m_DrawMode} not implemented`); @@ -149,7 +151,7 @@ export class FxSpriteRenderer { this.enabled = !!comp.m_Enabled; this.drawMode = comp.m_DrawMode; this.sprite = { - name: sp.sprite, texture: sp.texture, rect: { ...sp.rect }, border: { ...sp.border }, pivot: { ...sp.pivot }, + name: sp.name, texture: sp.texture, rect: { ...sp.rect }, border: { ...sp.border }, pivot: { ...sp.pivot }, ppu: F(sp.pixelsToUnits), vertices: sp.vertices, uv: sp.uv, indices: sp.indices, packed: (sp.settingsRaw & 1) === 1, }; if (this.drawMode === 1 && this.sprite.packed) throw new FxEffectsError(`${name}: sliced packed sprite not implemented`); @@ -281,11 +283,11 @@ export class FxSpriteRenderer { }; // ------------------------------------------------------------------------------------------ prefab instance -// One instance of an exported effect prefab (notes.json assets[key].nodes) inside its pool element: the element +// One instance of an effect prefab (a root GameObject's prefab.json) inside its pool element: the element // GameObject (clone of the scene placeholder container_root/prefabs/, identity, the LiveGameNoteEffectBase / // LiveGameLaneEffectBase host) -> placeholder child `root` (identity, _prefabAssetRoot) -> the prefab root. -// Components: Animator (controller from notes.controllers, clips from notes.clips, shared between instances like -// Unity's assets), SpriteRenderers (FxSpriteRenderer), ParticleSystems (FxParticleSystem, children filled in +// Components: Animator (its controller and clips through LiveObjects, shared between instances like Unity's +// assets), SpriteRenderers (FxSpriteRenderer), ParticleSystems (FxParticleSystem, children filled in // hierarchy order), Live{Renderer,Particle}OrderInLayerSetter, LiveNoteEffectAsset / LiveLaneEffectAsset (width // fitting, Play / Stop), AnimationFinishEventTrigger (event at clip.length -> onStop). // GameObject activity: activeSelf per node, activeInHierarchy propagated; a change of activeInHierarchy calls @@ -294,14 +296,14 @@ export class FxSpriteRenderer { // ENGINE: activation is native; OnEnable / OnDisable run immediately inside SetActive and m_IsActive writes, parent first. // The order among siblings has no visible effect here. export class FxPrefab { - // exported: notes.assets[key]; opts {notes (livenotes/notes.json), materials (FxMaterials | null), - // rng (UnityRandom), parent (Transform of the pool container), cache ({clips: Map, ctrls: Map}, shared), name} - constructor(exported, { notes, materials = null, rng = null, parent = null, cache = null, name = "" } = {}) { - this.key = exported.key; - this.name = name || exported.key; - this.notes = notes; - this.cache = cache || { clips: new Map(), ctrls: new Map() }; - const pf = new Prefab(exported); + // id: the prefab's root GameObject; opts {objects (LiveObjects), materials (FxMaterials | null), rng (UnityRandom), + // parent (Transform of the pool container), name} + constructor(id, { objects, materials = null, rng = null, parent = null, name = "" } = {}) { + const pf = Prefab.fromObjects(objects.assets, [id], id); // a hierarchy of its own per instance + this.id = id; + this.key = pf.root.name; + this.name = name || this.key; + this.objects = objects; this.prefab = pf; // element (placeholder clone) and its `root` child: identity transforms (as in the scene) this.element = new Transform(`${pf.root.name}(element)`, parent); @@ -332,26 +334,25 @@ export class FxPrefab { this.asset = null; this.assetClass = null; this.animatorEntry = null; this.animatorComp = null; const setters = []; - const cls = (c) => c.class || c.type; for (const e of this.order) { const comps = e.node.components; for (const c of comps) { - const k = cls(c); + const k = componentClass(c); if (k === "ParticleSystem") { - const r = comps.find((x) => cls(x) === "ParticleSystemRenderer") || null; - e.system = new FxParticleSystem(c, r, e.transform, { rng, materials, name: `${this.key}:${e.path}` }); + const r = comps.find((x) => componentClass(x) === "ParticleSystemRenderer") || null; + e.system = new FxParticleSystem(c.fields, r && r.fields, e.transform, { rng, materials, name: `${this.key}:${e.path}` }); this.systems.push(e.system); } else if (k === "SpriteRenderer") { - e.sprite = new FxSpriteRenderer(c, e.transform, { materials, name: `${this.key}:${e.path}` }); + e.sprite = new FxSpriteRenderer(c.fields, e.transform, { materials, name: `${this.key}:${e.path}` }); this.sprites.push(e.sprite); } else if (k === "Animator") { if (this.animatorEntry) throw new FxEffectsError(`${this.key}: two Animators`); - this.animatorEntry = e; this.animatorComp = c; + this.animatorEntry = e; this.animatorComp = c.fields; } else if (k === "LiveNoteEffectAsset" || k === "LiveLaneEffectAsset") { if (this.asset) throw new FxEffectsError(`${this.key}: two effect assets`); - this.asset = c; this.assetClass = k; + this.asset = c.fields; this.assetClass = k; } else if (k === "LiveRendererOrderInLayerSetter" || k === "LiveParticleOrderInLayerSetter") { - setters.push(c); + setters.push(c.fields); } else if (k === "LiveGameParticleCallback") { e.particleCallback = true; } else if (k !== "ParticleSystemRenderer") { @@ -433,29 +434,13 @@ export class FxPrefab { } // ----------------------------------------------------------------------------------------- Animator - _clip(key) { - const cs = this.cache.clips; - if (!cs.has(key)) { - const raw = this.notes.clips[key]; - if (!raw) throw new FxEffectsError(`clip ${key} not exported`); - const clip = AnimClip.fromMecanim(raw, key); - clip.rawName = raw.name; // AnimationClip.name (AnimationFinishEventTrigger matches it) - cs.set(key, clip); - } - return cs.get(key); - } - + // The controller's clips are named for this Animator's node (LiveObjects.controller); `clipList` is + // runtimeAnimatorController.animationClips, `rawName` the AnimationClip.name AnimationFinishEventTrigger matches. _buildAnimator(comp) { if (comp.m_KeepAnimatorStateOnDisable || comp.m_WriteDefaultValuesOnDisable || comp.m_UpdateMode !== 0 || comp.m_ApplyRootMotion) throw new FxEffectsError(`${this.key}: Animator settings not implemented`); - const key = comp.m_Controller && comp.m_Controller.controller; - if (!key || !this.notes.controllers[key]) throw new FxEffectsError(`${this.key}: controller ${key} not exported`); - let ctrl = this.cache.ctrls.get(key); - if (!ctrl) { - ctrl = AnimController.fromMecanim(this.notes.controllers[key], (k) => this._clip(k)); - ctrl.clipList = this.notes.controllers[key].clips.map((c) => this._clip(c.clip)); // runtimeAnimatorController.animationClips - this.cache.ctrls.set(key, ctrl); - } + const ctrl = this.objects.controller(comp.m_Controller, this.id, this.animatorEntry.path); + if (!ctrl) throw new FxEffectsError(`${this.key}: Animator without controller`); this.controller = ctrl; this.unresolved = []; this.animator = new Animator(ctrl, (b) => { @@ -784,22 +769,17 @@ export const FxContainerChain = (names) => { // LiveAllNoteEffectView (LiveGameView/root/LiveGameNoteEffectView): one-shot note effects per judged note and the // slide hold loop per held long line. Effects are drawn by LiveEffectCamera (layer 29): camera "effect". export class LiveNoteEffects { - // opts {notes (livenotes/notes.json), score (score/.notes.json), materials (FxMaterials), rng (UnityRandom), - // sceneInfo (FxScene.read(scene)), judgementRoot3D (localToWorld of judgement_root; default = sceneInfo), - // effect (the effect set's asset name, NoteEffectId; default notes.settings.effect)} - constructor(gl, { notes, score, sceneInfo, materials = null, rng = null, judgementRoot3D = null, effect = null } = {}) { + // opts {objects (LiveObjects), effect (noteeffect.json of the effect set: MasterLiveNoteEffectSkin of NoteEffectId, + // its Light set at LiveQuality Low; SoloLiveResourceLoadStateNode.CreateLoadParameter), score + // (score/.notes.json), materials (FxMaterials), rng (UnityRandom), sceneInfo (FxScene.read), + // judgementRoot3D (localToWorld of judgement_root; default = sceneInfo)} + constructor(gl, { objects, effect, score, sceneInfo, materials = null, rng = null, judgementRoot3D = null } = {}) { if (!sceneInfo) throw new FxEffectsError("LiveNoteEffects: sceneInfo (FxScene.read) missing"); this.gl = gl; - this.notes = notes; this.score = score; this.materials = materials; this.rng = rng; - this.cache = { clips: new Map(), ctrls: new Map() }; - // effect set: MasterLiveNoteEffectSkin of NoteEffectId (default 1 -> effect001), LiveQuality Middle -> not the - // Light set (SoloLiveResourceLoadStateNode.CreateLoadParameter) - const dir = `Effect/Live/NoteEffect/${effect || notes.settings.effect}`; - const settings = notes.assets[`${dir}/LiveNoteEffectAssetSettings`]; - if (!settings) throw new FxEffectsError(`${dir}/LiveNoteEffectAssetSettings not exported`); + const settings = objects.mono(effect.settings), dir = effect.name; this.laneCount = score.laneCount; // _maxLaneCount (24) this.baseLaneWidth = F(sceneInfo.laneWidth); // (float)LiveLaneView._laneWidth // FullInitialize: judgement positions of the 3D root (LiveLaneView.noteJudgementRoot3D) cached once @@ -815,17 +795,16 @@ export class LiveNoteEffects { this.pools = new Map(); for (const c of sceneInfo.noteContainers) { const ref = settings[c.field]; - if (!ref || !ref.gameObject) throw new FxEffectsError(`${dir}: ${c.field} missing`); - const key = `${dir}/${ref.gameObject}`, exported = notes.assets[key]; - if (!exported) throw new FxEffectsError(`${key} not exported`); + if (!ref) throw new FxEffectsError(`${dir}: ${c.field} missing`); + const id = objects.assets.contained(ref).prefab; // the effect prefab of the LiveNoteEffectAsset const parent = FxContainerChain(["LiveGameView", "root", "LiveGameNoteEffectView", "container_root", c.name]); const create = (i) => { - const e = new FxPrefab(exported, { notes, materials, rng, parent, cache: this.cache, name: `${c.name}#${i}` }); + const e = new FxPrefab(id, { objects, materials, rng, parent, name: `${c.name}#${i}` }); e.initialize(c.type, this.baseLaneWidth); // OnInitializeElementView return e; }; const pool = new FxPool(create, c); - pool.name = c.name; pool.key = key; + pool.name = c.name; pool.key = `${dir}/${c.field}`; this.pools.set(c.type, pool); } this.activeOneShot = new Set(); // _activeOneShotEffects @@ -981,22 +960,18 @@ export class LiveNoteEffects { }; // ------------------------------------------------------------------------------------------ lane effects -export const LIVE_LANE_EFFECT_SET = "effect001"; // LaneEffectLoadStep.GetAssetPath // LiveLaneEffectView (LiveGameView/root/LiveGameLaneEffectView): per container 24 permanent instances, one per // lane; per frame the requested type per lane is played. Drawn by LiveGameCamera (layer 25): camera "game". export class LiveLaneEffects { // The lane effect set does not follow NoteEffectId or LiveQuality: LaneEffectLoadStep.GetAssetPath passes the - // constant "effect001" to LiveAddressablePath.GetLiveLaneEffectSkinAssetPath. - constructor(gl, { notes, score, sceneInfo, materials = null, rng = null } = {}) { + // constant "effect001" to LiveAddressablePath.GetLiveLaneEffectSkinAssetPath (notes.json laneEffect: its + // LiveLaneEffectAssetSettings). + constructor(gl, { objects, notes, score, sceneInfo, materials = null, rng = null } = {}) { if (!sceneInfo) throw new FxEffectsError("LiveLaneEffects: sceneInfo (FxScene.read) missing"); this.gl = gl; - this.notes = notes; - const dir = `Effect/Live/LaneEffect/${LIVE_LANE_EFFECT_SET}`; - const settings = notes.assets[`${dir}/LiveLaneEffectAssetSettings`]; - if (!settings) throw new FxEffectsError(`${dir}/LiveLaneEffectAssetSettings not exported`); + const settings = objects.mono(notes.laneEffect), dir = notes.settings.laneEffect; this.laneCount = score.laneCount; // LiveLaneView lane count (24) this.noteById = new Map(score.notes.map((x) => [x.id, x])); - this.cache = { clips: new Map(), ctrls: new Map() }; // FullInitialize: width = (float)_laneWidth / laneCount, instance i at // (width * 0.5 + (width * i - (float)_laneWidth * 0.5), 0, 0) world, SetWidth(width), SetActive(true) const lw = F(sceneInfo.laneWidth), n = this.laneCount; @@ -1006,11 +981,11 @@ export class LiveLaneEffects { this.pools = new Map(); for (const c of sceneInfo.laneContainers) { const ref = settings[c.field]; - const key = `${dir}/${ref.gameObject}`, exported = notes.assets[key]; - if (!exported) throw new FxEffectsError(`${key} not exported`); + if (!ref) throw new FxEffectsError(`${dir}: ${c.field} missing`); + const id = objects.assets.contained(ref).prefab; // the effect prefab of the LiveLaneEffectAsset const parent = FxContainerChain(["LiveGameView", "root", "LiveGameLaneEffectView", "container_root", c.name]); const create = (i) => { - const e = new FxPrefab(exported, { notes, materials, rng, parent, cache: this.cache, name: `lane-${c.name}#${i}` }); + const e = new FxPrefab(id, { objects, materials, rng, parent, name: `lane-${c.name}#${i}` }); e.initialize(c.type); return e; }; diff --git a/src/live/fx-ui.js b/src/live/fx-ui.js index 67a27db..ffd5519 100644 --- a/src/live/fx-ui.js +++ b/src/live/fx-ui.js @@ -1,6 +1,6 @@ -import { AnimClip, AnimController, Animator } from "../engine/anim.js"; +import { Animator } from "../engine/anim.js"; import { F } from "../engine/core.js"; -import { ShaderLib } from "../engine/glsl.js"; +import { componentClass } from "../engine/prefab.js"; import { GLTex } from "../engine/texture.js"; import { DOTRunner, DOTSequence, UIAffine, UIDraw, UIError, UIImage, UILayout, UINode, UISprite, UI_CLIP_TARGET, dotFloat, dotPunch, uiCanvasSize } from "../engine/ugui.js"; @@ -19,10 +19,10 @@ import { DOTRunner, DOTSequence, UIAffine, UIDraw, UIError, UIImage, UILayout, U // LiveCanvasOrderInLayerSetter.Awake + LiveOrderInLayerUtility.GetOrderInLayer (100 -> 10000); // DOTween: the DOTween model in engine/ugui.js (Punch, SetPunch, Vector3ArrayPlugin and the Sequence cycle follow // the game's DOTween build; the rest follows DOTween's documented behaviour). -// Data: livescene/scene.json (scene.nodes under Live/LiveUICanvas), livenotes/notes.json (prefabs.judge_effect_view, -// controllers / clips of EmbLive/Animation/Combo/*, assets Live/UiSpriteAssets/{LiveJudgementSpriteAsset, -// LiveComboSpriteAsset} with their inline sprite records), textures under livenotes/textures, UI/Default under -// livenotes/shaders. +// Data: the scene product through LiveObjects: scene.json `scene` (the nodes under Live/LiveUICanvas), +// prefabs.judgeEffectView (UILiveNoteJudgeEffectView), spriteAssets.judgement / combo (LiveJudgementSpriteAsset, +// LiveComboSpriteAsset), the UILiveCombo Animator's controller and clips (bindings named for that Animator), the sprites +// and textures they reference, and UI/Default from the shader name table. // // Frame: update(fr) in the update phase (LiveUIView.UpdateLiveSimulateFrame: ShowJudgement, then // UIComboCounterView.UpdateView), tweens(dt) in the DOTween phase, animate(dt) with the Animators, submit(renderer) in @@ -33,9 +33,6 @@ export const LIVEUI = { COMBO: "Live/LiveUICanvas/UISafeArea/ContentArea/UILiveCombo", JUDGEMENT: "Live/LiveUICanvas/UILiveJudgement", PREFAB: "UILiveNoteJudgeEffectView", - JUDGEMENT_SPRITES: "Live/UiSpriteAssets/LiveJudgementSpriteAsset", - COMBO_SPRITES: "Live/UiSpriteAssets/LiveComboSpriteAsset", - COMBO_CONTROLLER: "EmbLive/Animation/Combo/UILiveCombo", SHADER: "UI/Default", // UILiveNoteJudgeEffectView.Initialize: _judgementImages[i] <-> judgement KEYS[i] (static array) JUDGE_KEYS: [5, 4, 3, 2, 1, 6], @@ -43,17 +40,23 @@ export const LIVEUI = { PRIORITY_3_6: [3, 5, 10, 20], }; -export const liveComp = (rec, cls) => (rec.components || []).find((c) => (c.class || c.type) === cls) || null; +// the fields of a node record's component of class `cls` (prefab.json components), or null +export const liveComp = (rec, cls) => { + const c = (rec.components || []).find((x) => componentClass(x) === cls); + return c ? c.fields : null; +}; // ------------------------------------------------------------------- scene / prefab records -> UINode // The fields UINode reads (path, name, active, rect, localRotation, localScale, canvasGroup) plus the components // this module uses; graphics other than Image / TextMeshProUGUI do not occur in the subset (checked). export class LiveUITree { - constructor(refPPU) { + // objects: LiveObjects (the sprites and their textures) + constructor(refPPU, objects) { this.refPPU = refPPU; + this.objects = objects; this.nodes = new Map(); - this.sprites = new Map(); // "@" -> UISprite - this.textures = new Map(); // texture path (relative to livenotes/) -> exported descriptor, loaded by load() + this.sprites = new Map(); // Sprite object id -> UISprite + this.textures = new Map(); // Texture2D object id -> its record, loaded by load() } add(rec, path, parent) { @@ -89,23 +92,20 @@ export class LiveUITree { return n; } - // exported sprite record (inline in components and sprite assets) -> UISprite. load = true registers its texture - // for LiveUIFx.load(): only the sprite assets' sprites (livenotes/textures); sprites of scene / prefab image records - // are either replaced before they can draw or sit under disabled images / zero-alpha groups (a draw of an unloaded - // texture raises). - sprite(s, load = false) { - if (!s) return null; - const tex = s.texture; - const key = `${s.sprite}@${tex.texture}`; - if (!this.sprites.has(key)) { - if (s.downscaleMultiplier !== undefined && s.downscaleMultiplier !== 1) - throw new UIError(`sprite ${s.sprite}: atlas variant scale ${s.downscaleMultiplier}`); - this.sprites.set(key, new UISprite(s.sprite, { rect: s.rect, border: s.border, pixelsPerUnit: s.pixelsToUnits, + // a sprite reference (in components and sprite assets) -> UISprite. load = true registers its texture for + // LiveUIFx.load(): only the sprite assets' sprites; sprites of scene / prefab image records are either replaced + // before they can draw or sit under disabled images / zero-alpha groups (a draw of an unloaded texture raises). + sprite(v, load = false) { + if (!v) return null; + const s = this.objects.sprite(v), tex = s.texture; + if (!this.sprites.has(s.id)) { + if (s.downscaleMultiplier !== 1) throw new UIError(`sprite ${s.name}: atlas variant scale ${s.downscaleMultiplier}`); + this.sprites.set(s.id, new UISprite(s.name, { rect: s.rect, border: s.border, pixelsPerUnit: s.pixelsToUnits, pivot: s.pivot, textureRect: s.textureRect, textureRectOffset: s.textureRectOffset }, - { width: tex.width, height: tex.height, name: tex.texture })); + { width: tex.width, height: tex.height, name: tex.id })); } - if (load) this.textures.set(tex.texture, tex); - return this.sprites.get(key); + if (load) this.textures.set(tex.id, tex); + return this.sprites.get(s.id); } // GameObjectExtension.SetActiveFast: no-op when activeSelf already equals v @@ -329,10 +329,13 @@ export class LiveUIFx { // (Only matters if Init / Add arrive inside a running Init fade, which auto play at Perfect never does.) // settings: the live settings (settings.js) read by LiveUIView.Initialize (ComboCountDisplay 206, // ContinuationEffectDisplay 208, JudgeResultPositionType 103); absent: the defaults. - constructor(gl, { scene, notes, settings = null }, opts = {}) { + // objects: LiveObjects; scene: scene.json + constructor(gl, { objects, scene, settings = null }, opts = {}) { this.gl = gl; + this.objects = objects; const opt = (k, d) => (settings && settings[k] !== undefined ? settings[k] : d); - const recs = new Map(scene.scene.nodes.map((r) => [r.path, r])); + const nodes = scene.scene.flatMap((id) => objects.assets.prefab(id).nodes); + const recs = new Map(nodes.map((r) => [r.path, r])); const rec = (p) => { const r = recs.get(p); if (!r) throw new UIError(`scene node missing: ${p}`); return r; }; const canvasRec = rec(LIVEUI.CANVAS); const canvas = liveComp(canvasRec, "Canvas"), scaler = liveComp(canvasRec, "CanvasScaler"); @@ -347,13 +350,13 @@ export class LiveUIFx { if (!ol || ol._orderInLayer !== 100) throw new UIError("LiveUICanvas: order-in-layer setter differs"); this.sortingOrder = 10000 + ol._offset; this.planeDistance = canvas.m_PlaneDistance; - const tree = this.tree = new LiveUITree(scaler.m_ReferencePixelsPerUnit); + const tree = this.tree = new LiveUITree(scaler.m_ReferencePixelsPerUnit, objects); // subset: the canvas, the combo counter's chain and subtree, the judgement view and its effect_root // UISafeArea (flexible margin (0,0,0,40)) keeps ContentArea's serialized rect: the player has no device safe // area, and the serialized offsets (bottom 40, top 0) are that margin; UISafeAreaGlobalSettings overrides not read. const want = (p) => p === LIVEUI.CANVAS || p === `${LIVEUI.CANVAS}/UISafeArea` || p === `${LIVEUI.CANVAS}/UISafeArea/ContentArea` || p === LIVEUI.COMBO || p.startsWith(`${LIVEUI.COMBO}/`) || p === LIVEUI.JUDGEMENT || p.startsWith(`${LIVEUI.JUDGEMENT}/`); - for (const r of scene.scene.nodes) { // hierarchy order (parents first, siblings in order) + for (const r of nodes) { // hierarchy order (parents first, siblings in order) if (!want(r.path)) continue; const pp = r.path.slice(0, r.path.lastIndexOf("/")); tree.add(r, r.path, r.path === LIVEUI.CANVAS ? null : tree.node(pp)); @@ -364,15 +367,14 @@ export class LiveUIFx { // Instantiate(prefab, parent) keeps the prefab's local RectTransform) const jv = liveComp(rec(LIVEUI.JUDGEMENT), "LiveJudgementView"); const parentPath = jv._judgementEffectParent.transform; - const prefab = notes.prefabs.judge_effect_view; + const prefab = objects.assets.prefab(scene.prefabs.judgeEffectView); if (prefab.nodes[0].path !== LIVEUI.PREFAB) throw new UIError("judge effect prefab root differs"); const inst = (p) => `${parentPath}/${p}`; for (const r of prefab.nodes) { const pp = r.path.includes("/") ? inst(r.path.slice(0, r.path.lastIndexOf("/"))) : parentPath; tree.add(r, inst(r.path), tree.node(pp)); } - const jsa = notes.assets[LIVEUI.JUDGEMENT_SPRITES]; - if (!jsa) throw new UIError(`asset missing: ${LIVEUI.JUDGEMENT_SPRITES}`); + const jsa = objects.mono(scene.spriteAssets.judgement); // LiveJudgementSpriteAsset const judgeSprites = new Map([[5, jsa.PerfectSprite], [4, jsa.GreatSprite], [3, jsa.GoodSprite], [2, jsa.BadSprite], [1, jsa.MissSprite], [6, jsa.JustSprite]]); for (const s of judgeSprites.values()) tree.sprite(s, true); @@ -385,18 +387,14 @@ export class LiveUIFx { // combo: Animator UILiveCombo on UIComboCounterView's GameObject const comboNode = tree.node(LIVEUI.COMBO), comboRec = rec(LIVEUI.COMBO); const an = liveComp(comboRec, "Animator"); - const ctrlKey = notes.assets[LIVEUI.COMBO_CONTROLLER] && notes.assets[LIVEUI.COMBO_CONTROLLER].controller; - const raw = ctrlKey && notes.controllers[ctrlKey]; - if (!an || !raw || raw.name !== an.m_Controller.name) throw new UIError("UILiveCombo: controller not exported"); + if (!an || !an.m_Controller) throw new UIError("UILiveCombo: Animator without controller"); if (!an.m_Enabled || an.m_UpdateMode !== 0 || an.m_CullingMode !== 0 || an.m_KeepAnimatorStateOnDisable || an.m_ApplyRootMotion) throw new UIError("UILiveCombo: Animator settings outside the implemented subset"); - const ctrl = AnimController.fromMecanim(raw, (key) => { - if (!notes.clips[key]) throw new UIError(`clip ${key} not exported`); - return AnimClip.fromMecanim(notes.clips[key], key); - }); + const root = LIVEUI.COMBO.slice(0, LIVEUI.COMBO.indexOf("/")); + const rootId = scene.scene.find((id) => objects.assets.prefab(id).nodes[0].path === root); + const ctrl = objects.controller(an.m_Controller, rootId, LIVEUI.COMBO); this.animator = new Animator(ctrl, (b) => LiveUIFx.bind(comboNode, b), "UILiveCombo"); - const csa = notes.assets[LIVEUI.COMBO_SPRITES]; - if (!csa) throw new UIError(`asset missing: ${LIVEUI.COMBO_SPRITES}`); + const csa = objects.mono(scene.spriteAssets.combo); // LiveComboSpriteAsset for (const ts of [csa.NormalSprites, csa.FullComboSprites, csa.AllPerfectSprites]) for (const s of [ts.TitleSprite, ts.GlowSprite, ...ts.BaseDigitSprites, ...ts.FillDigitSprites]) if (s) tree.sprite(s, true); this.combo = new LiveComboCounter(tree, liveComp(comboRec, "UIComboCounterView"), csa, this.animator); @@ -488,7 +486,7 @@ export class LiveUIFx { } // ------------------------------------------------------------ GL - // the GL resources of an already loaded LiveUIFx (same scene and notes): what load() creates + // the GL resources of an already loaded LiveUIFx (same objects and scene): what load() creates adoptGL(prev) { for (const k of ["lib", "solid", "material", "tex", "vao", "vbo", "ibo"]) this[k] = prev[k]; } @@ -496,7 +494,7 @@ export class LiveUIFx { async load() { const gl = this.gl; if (!gl) return; - this.lib = new ShaderLib(gl, "livenotes/shaders"); + this.lib = this.objects.lib; this.solid = { white: GLTex.solid(gl, [255, 255, 255, 255], "white"), black: GLTex.solid(gl, [0, 0, 0, 255], "black"), gray: GLTex.solid(gl, [128, 128, 128, 255], "gray"), bump: GLTex.solid(gl, [128, 128, 255, 255], "bump"), @@ -508,7 +506,7 @@ export class LiveUIFx { this.material = { shader: LIVEUI.SHADER, keywords: [], floats: num, colors: {}, defaults }; this.lib.program(LIVEUI.SHADER, 0, []); this.tex = new Map(); - for (const [p, desc] of this.tree.textures) this.tex.set(p, await GLTex.load(gl, "livenotes", desc)); + for (const id of this.tree.textures.keys()) this.tex.set(id, await this.objects.texture(id)); this.vao = gl.createVertexArray(); this.vbo = gl.createBuffer(); this.ibo = gl.createBuffer(); } diff --git a/src/live/fx.js b/src/live/fx.js index cce57f3..8ab0934 100644 --- a/src/live/fx.js +++ b/src/live/fx.js @@ -1,3 +1,4 @@ +import { Prefab } from "../engine/prefab.js"; import { UnityRandom } from "../engine/random.js"; import { FxScene, LiveLaneEffects, LiveNoteEffects } from "./fx-effects.js"; import { LiveUIFx } from "./fx-ui.js"; @@ -23,24 +24,25 @@ import { LiveIntroStars } from "./particles.js"; // (The sync is PostLateUpdate ParticleSystemEndUpdateAll, so stepping in the animation phase is equivalent for drawing.) export class LiveFx { - // gl may be null (Node tests). notes = assets.json("livenotes/notes.json"), score = score/.notes.json, - // scene = assets.json("livescene/scene.json"). introStars = () => stage.intro.stars() (the 7 lane-in - // star emitters of the start timeline; the session constructs LiveFx after the stage so that this object's animation - // hook runs after the timeline evaluation of the frame, as Unity's DirectorUpdateAnimation precedes + // gl may be null (Node tests). objects = the live's LiveObjects; scene, notes, effect = scene.json, notes.json and + // the selected effect set's noteeffect.json; score = score/.notes.json. introStars = () => stage.intro.stars() + // (the 7 lane-in star emitters of the start timeline; the session constructs LiveFx after the stage so that this + // object's animation hook runs after the timeline evaluation of the frame, as Unity's DirectorUpdateAnimation precedes // ParticleSystemBeginUpdateAll). seed: UnityRandom seed of the particle stream. settings: the live settings - // (settings.js; the live UI's options and the note effect set), absent: the data's defaults. + // (settings.js; the live UI's options), absent: the data's defaults. // ENGINE: effect ParticleSystems use autoRandomSeed, so the game's values are not reproducible; deterministic per seed here. - constructor({ gl, loop, notes, score, scene, introStars = null, seed = 1, settings = null, effect = null }) { + constructor({ gl, loop, objects, scene, notes, effect, score, introStars = null, seed = 1, settings = null }) { this.gl = gl; this.loop = loop; - this.materials = new FxMaterials(gl, "livenotes"); - this.sceneMaterials = new FxMaterials(gl, "livescene"); // white_star (shader + texture under livescene/) + this.materials = FxMaterials.fromObjects(gl, objects); this.rng = new UnityRandom(seed); this.introStars = introStars; - this.stars = new LiveIntroStars(gl, { materials: this.sceneMaterials, rng: this.rng }); - const opts = { notes, score, sceneInfo: FxScene.read(scene), materials: this.materials, rng: this.rng, effect }; + this.stars = new LiveIntroStars(gl, { materials: this.materials, rng: this.rng }); + // the scene's hierarchy as loaded (before the timeline animates it) + const sceneInfo = FxScene.read(Prefab.fromObjects(objects.assets, scene.scene, "scene")); + const opts = { objects, notes, effect, score, sceneInfo, materials: this.materials, rng: this.rng }; this.lane = new LiveLaneEffects(gl, opts); this.note = new LiveNoteEffects(gl, opts); - this.uiArgs = { scene, notes, score, settings }; + this.uiArgs = { objects, scene, settings }; this.ui = new LiveUIFx(gl, this.uiArgs); this.frozen = false; // a step that does not count (the player's seek after a long clock jump) loop.on("tweens", (l) => { if (!this.frozen) this.ui.tweens(l.deltaTime); }); @@ -50,7 +52,6 @@ export class LiveFx { async load() { if (this.introStars) this.stars.prepare(this.introStars()); // registers the star material before loading await this.materials.load(); - await this.sceneMaterials.load(); await this.note.load(); await this.lane.load(); await this.ui.load(); diff --git a/src/live/intro.js b/src/live/intro.js index c3a9f28..30bc235 100644 --- a/src/live/intro.js +++ b/src/live/intro.js @@ -1,9 +1,6 @@ -import { AnimClip } from "../engine/anim.js"; import { F } from "../engine/core.js"; import { applyState } from "../engine/glsl.js"; import { mat4 } from "../engine/math.js"; -import { Prefab } from "../engine/prefab.js"; -import { GLTex } from "../engine/texture.js"; import { LIVE_UI_DEAD_INPUTS, LiveGeom, LiveLane } from "./lane.js"; import { liveFitSheets, liveMaterialSheets } from "./renderer.js"; @@ -32,10 +29,12 @@ import { liveFitSheets, liveMaterialSheets } from "./renderer.js"; // gradient veil + the simple jacket panel + TMP texts, T 1.5 .. 4.4167): the player has no start canvas; the 7 // ef_particle_star ParticleSystems are drawn by the effects (LiveFx; `stars()` gives their transforms and active // state). -// Clips are sampled with AnimClip (engine/anim.js). Bindings are resolved against each track's bound -// Animator (TimelineAsset track `boundAnimator` {key, gameObject}: the scene or the timeline prefab) and applied by -// attribute name: GameObject m_IsActive, Transform m_LocalPosition / m_LocalScale, SpriteRenderer m_Color.*, -// renderer material properties `material..` (Animator material property block, typeID 137). +// Clips are sampled with AnimClip (engine/anim.js). The timeline is the band's (scene.json bands[b].timeline); each +// animation track plays on the Animator its binding names (bands[b].tracks[track id]: {prefab: a scene root or the +// timeline prefab, animator: the Animator's node path}), its clips' bindings named for that Animator (scene.json +// bindings) and applied by attribute name: GameObject m_IsActive, Transform m_LocalPosition / m_LocalScale, +// SpriteRenderer m_Color.*, renderer material properties `material..` (Animator material property block, +// typeID 137). // TimelineClip -> clip-local time at timeline time T, or null when the clip does not drive the track at T // (m_Start, m_Duration, m_ClipIn, m_TimeScale, Hold extrapolation (1) within the pre/post extrapolation time). @@ -55,18 +54,24 @@ export const LIVE_SIGNAL_METHODS = { NotifyPlayVoice: "PlayVoice" }; export class LiveIntro { constructor(stage) { this.stage = stage; this.r = stage.r; this.scene = stage.scene; this.prefab = stage.r.prefab; - this.tl = new Prefab(this.scene.assets.startTimeline); + const O = this.r.objects, band = this.r.band; + this.tlId = band.timeline; + this.tl = O.prefab(band.timeline); const rootPath = this.tl.roots[0].name; - this.director = this.tl.component(rootPath, "PlayableDirector"); - const ta = this.director.m_PlayableAsset; - this.tracks = new Map(ta.m_Tracks.map((t) => [t.m_Name, { - bound: t.boundAnimator || null, - clips: t.m_Clips.map((c) => ({ tc: c, clip: AnimClip.fromMecanim(c.m_Asset.m_Clip, c.m_Asset.m_Clip.clip) })) }])); - this.duration = Math.max(...ta.m_Tracks.flatMap((t) => t.m_Clips.map((c) => c.m_Start + c.m_Duration))); // BasedOnClips + this.director = this.tl.fields(rootPath, "PlayableDirector"); + const tracks = O.mono(this.director.m_PlayableAsset).m_Tracks.map((v) => O.mono(v)); + // an animation track's clips (AnimationPlayableAsset.m_Clip) as its bound Animator plays them + this.tracks = new Map(tracks.filter((t) => t.class === "AnimationTrack").map((t) => { + const bound = band.tracks[t.id]; + if (!bound) throw new Error(`timeline track ${t.m_Name}: no bound Animator`); + return [t.m_Name, { bound, clips: t.m_Clips.map((c) => ({ + tc: c, clip: O.clip(O.id(O.mono(c.m_Asset).m_Clip, "AnimationClip"), bound.prefab, bound.animator) })) }]; + })); + this.duration = Math.max(...tracks.flatMap((t) => (t.m_Clips || []).map((c) => c.m_Start + c.m_Duration))); // BasedOnClips this.effectRoot = `${rootPath}/root`; this.time = 0; this.playing = false; this.done = false; this.onEnd = null; this.n = 0; this.ignored = 0; this.onSignal = null; // onSignal(name): Signal track markers (see _signalMarkers) - this.signals = this._signalMarkers(ta, rootPath); + this.signals = this._signalMarkers(tracks, rootPath); this.dt = F(1 / 60); this.effect = new LiveLaneInEffect(this); } @@ -91,19 +96,21 @@ export class LiveIntro { // Signal track markers: SignalEmitter {m_Time, m_Retroactive, m_EmitOnce, m_Asset (SignalAsset)} resolved through the // timeline prefab's SignalReceiver (LivePlayableTimeline._signalReceiver): the asset's reaction is a UnityEvent // whose persistent call names the LivePlayableTimeline method, mapped to the callback name here. - _signalMarkers(ta, rootPath) { - const recv = this.tl.component(rootPath, "SignalReceiver").m_Events; + _signalMarkers(tracks, rootPath) { + const O = this.r.objects, recv = this.tl.fields(rootPath, "SignalReceiver").m_Events; const out = []; - for (const tr of ta.m_Tracks) { - if (tr.asset !== "SignalTrack" || tr.m_Muted) continue; - for (const m of (tr.m_Markers && tr.m_Markers.m_Objects) || []) { - if (m.asset !== "SignalEmitter" || !m.m_Enabled) continue; - const k = recv.m_Signals.findIndex((a) => a && a.name === m.m_Asset.name); + for (const tr of tracks) { + if (tr.class !== "SignalTrack" || tr.m_Muted) continue; + for (const v of (tr.m_Markers && tr.m_Markers.m_Objects) || []) { + const m = O.mono(v); + if (m.class !== "SignalEmitter" || !m.m_Enabled) continue; + const asset = O.mono(m.m_Asset); + const k = recv.m_Signals.findIndex((a) => a && O.id(a) === asset.id); if (k < 0) continue; // no reaction for this signal on the receiver for (const c of recv.m_Events[k].m_PersistentCalls.m_Calls) { const name = LIVE_SIGNAL_METHODS[c.m_MethodName]; - if (!name) throw new Error(`signal ${m.m_Asset.name}: receiver method ${c.m_MethodName}`); - out.push({ time: m.m_Time, retroactive: !!m.m_Retroactive, emitOnce: !!m.m_EmitOnce, asset: m.m_Asset.name, name, fired: false }); + if (!name) throw new Error(`signal ${asset.m_Name}: receiver method ${c.m_MethodName}`); + out.push({ time: m.m_Time, retroactive: !!m.m_Retroactive, emitOnce: !!m.m_EmitOnce, asset: asset.m_Name, name, fired: false }); } } } @@ -145,9 +152,9 @@ export class LiveIntro { // node of a binding: the bound Animator's GameObject + binding path, in the scene or the timeline prefab _target(bound, b) { - const pf = bound.key === this.scene.scene.key ? this.prefab : bound.key === this.tl.key ? this.tl : null; - if (!pf) throw new Error(`timeline binding in ${bound.key}`); - const path = b.path ? `${bound.gameObject}/${b.path}` : bound.gameObject; + const pf = bound.prefab === this.tlId ? this.tl : this.prefab.ids.includes(bound.prefab) ? this.prefab : null; + if (!pf) throw new Error(`timeline binding in ${bound.prefab}`); + const path = b.path ? `${bound.animator}/${b.path}` : bound.animator; if (!pf.nodes.has(path)) throw new Error(`timeline binding: no node ${path}`); return { pf, path }; } @@ -200,7 +207,7 @@ export class LiveIntro { // area's animation alpha (created in the animation phase; first DOTween update next frame) _playTapAreaFadeIn() { this.fadeReserved = false; - const lane = this.stage.lane, view = this.prefab.component("LiveGameView/root/LiveGameLane", "LiveLaneView"); + const lane = this.stage.lane, view = this.prefab.fields("LiveGameView/root/LiveGameLane", "LiveLaneView"); const dur = view._tapAreaFadeInDuration, ease = view._tapAreaFadeInEase; if (typeof dur !== "number" || typeof ease !== "number") throw new Error("LiveLaneView tap-area fade fields missing"); this.fadeTween = this.loop.tweens.to(0, 1, dur, ease, (v) => { lane.tapArea.animationAlpha = F(v); }); @@ -230,22 +237,22 @@ export class LiveLaneInEffect { this.renderers = []; for (const [path, e] of this.tl.nodes) { if (!path.startsWith(`${this.root}/LiveGameLaneInEffect`)) continue; - const sr = e.node.components.find((c) => c.type === "SpriteRenderer"); + const c = e.node.components.find((x) => x.type === "SpriteRenderer"), sr = c && c.fields; if (sr) this.renderers.push({ path, sr, layer: e.node.layer, color: { ...sr.m_Color } }); } this.byPath = new Map(this.renderers.map((x) => [x.path, x])); const meshPath = `${this.root}/LiveGameLaneInEffect/lines/lane_line03_plus/lane_mesh/lane_mesh`; - this.mesh = new LiveSkinnedMesh(this.tl, meshPath, this.tl.component(meshPath, "SkinnedMeshRenderer"), this.tl.nodes.get(meshPath).node.layer); + this.mesh = new LiveSkinnedMesh(this.tl, meshPath, this.tl.fields(meshPath, "SkinnedMeshRenderer"), + this.tl.nodes.get(meshPath).node.layer, this.r.objects); } async load() { - const gl = this.r.gl, textures = new Map(); + const gl = this.r.gl; this.vbo = { pos: gl.createBuffer(), uv: gl.createBuffer(), col: gl.createBuffer(), idx: gl.createBuffer() }; await this.mesh.load(this.r); for (const x of this.renderers) { - const sp = x.sr.m_Sprite, d = sp.texture; - if (!textures.has(d.texture)) textures.set(d.texture, await GLTex.load(gl, this.r.base, d)); - x.tex = textures.get(d.texture); + const sp = this.r.objects.sprite(x.sr.m_Sprite), d = sp.texture; + x.tex = await this.r.objects.texture(d.id); if (x.sr.m_DrawMode === 1) { // Sliced (LiveGeom.sliced: uv from the sprite rect) const r = sp.textureRect || sp.rect, tw = d.width, th = d.height; x.mesh = LiveGeom.sliced(x.sr.m_Size, sp.rect, sp.pivot, [sp.border.x, sp.border.y, sp.border.z, sp.border.w], @@ -275,8 +282,8 @@ export class LiveLaneInEffect { // the SortingGroup a renderer belongs to (nearest ancestor with an enabled SortingGroup), or null sortingGroup(path) { for (let p = path; p.includes("/"); p = p.slice(0, p.lastIndexOf("/"))) { - const g = this.tl.nodes.get(p).node.components.find((c) => c.type === "SortingGroup" && c.m_Enabled); - if (g) return { path: p, group: g }; + const g = this.tl.nodes.get(p).node.components.find((c) => c.type === "SortingGroup" && c.fields.m_Enabled); + if (g) return { path: p, group: g.fields }; } return null; } @@ -314,7 +321,8 @@ export class LiveLaneInEffect { meshState() { return this.mesh ? this.mesh.state() : null; } - // for the effects (LiveFx): the 7 star emitters (ef_particle_star under lane_lineNN/star_icon) + // for the effects (LiveFx): the 7 star emitters (ef_particle_star under lane_lineNN/star_icon), with their node's + // prefab.json components stars() { const out = []; for (const [path] of this.tl.nodes) @@ -336,14 +344,16 @@ export class LiveLaneInEffect { // root bone; the same world positions). // ENGINE: the mesh has no colour channel; the missing COLOR input is taken as (1, 1, 1, 1). export class LiveSkinnedMesh { - constructor(prefab, path, smr, layer) { + // smr: the SkinnedMeshRenderer's fields; objects: LiveObjects (its mesh and material) + constructor(prefab, path, smr, layer, objects) { this.pf = prefab; this.path = path; this.smr = smr; this.layer = layer; - const m = smr.m_Mesh; - if (!m || !m.m_Skin || !m.m_BindPose) throw new Error(`${path}: skinned mesh without skin data`); + const m = objects.mesh(smr.m_Mesh); + if (!m || !m.skin || !m.bindPoses.length) throw new Error(`${path}: skinned mesh without skin data`); if (m.submeshes.length !== 1) throw new Error(`${path}: ${m.submeshes.length} submeshes`); this.vertices = m.vertices; this.uv = new Float32Array(m.uv0.flat()); this.idx = new Uint16Array(m.submeshes[0]); - this.skin = m.m_Skin; - this.bindPoses = m.m_BindPose.map((e) => { + // per vertex: the bone indices and weights of its influences, in the mesh's order + this.skin = m.skin.joints.map((boneIndex, k) => ({ boneIndex, weight: m.skin.weights[k] })); + this.bindPoses = m.bindPoses.map((e) => { const o = new Float32Array(16); for (let r = 0; r < 4; r++) for (let c = 0; c < 4; c++) o[c * 4 + r] = e[`e${r}${c}`]; return o; @@ -351,14 +361,14 @@ export class LiveSkinnedMesh { this.bones = smr.m_Bones.map((b) => b.transform); if (this.bones.length !== this.bindPoses.length) throw new Error(`${path}: bones / bind poses`); this.rootBone = smr.m_RootBone ? smr.m_RootBone.transform : path; - this.material = smr.m_Materials[0]; + this.material = objects.material(smr.m_Materials[0]); this.block = {}; // Animator material property block (material.. curves) this.skinWeights = 2; // QualitySettings.skinWeights of every quality level (player data) } async load(r) { this.r = r; - this.tex = await GLTex.load(r.gl, r.base, this.material.textures._MainTex.texture); + this.tex = await r.objects.texture(this.material.textures._MainTex.texture.id); this.vbo = { pos: r.gl.createBuffer(), uv: r.gl.createBuffer(), col: r.gl.createBuffer(), idx: r.gl.createBuffer() }; } @@ -428,7 +438,7 @@ export class LiveSkinnedMesh { this.lastCulled = culled; if (culled) return null; const pos = this.skinned(), mesh = { pos, uv: this.uv, idx: this.idx }; - const mat = this.material, shader = mat.shader.shader, tint = this.tint(), I = mat4.identity(); + const mat = this.material, shader = mat.shader, tint = this.tint(), I = mat4.identity(); return { sortingLayer: this.smr.m_SortingLayer || 0, sortingOrder: this.smr.m_SortingOrder, queue: LiveLane.queue(this.r.lib, mat), center: LiveLane._center(pos), diff --git a/src/live/lane.js b/src/live/lane.js index d7e5afa..2bc23dd 100644 --- a/src/live/lane.js +++ b/src/live/lane.js @@ -1,8 +1,7 @@ import { F } from "../engine/core.js"; import { applyState } from "../engine/glsl.js"; import { mat4 } from "../engine/math.js"; -import { Prefab } from "../engine/prefab.js"; -import { GLTex } from "../engine/texture.js"; +import { componentClass } from "../engine/prefab.js"; import { liveFitSheets, liveMaterialSheets } from "./renderer.js"; // LiveLane: the 3D lane of the live screen (LiveLaneLayout, LiveGeom, render order helpers): LiveLaneView / LiveLaneLineView / LiveLaneLine with the @@ -98,24 +97,34 @@ export const LiveLaneLayout = { SPLIT_MAIN: [0, 4, 6, 8, 12], // constant table of the game code, by LiveLaneSplitCountType SPLIT_SUB: [12, 12, 12, 0, 0], // constant table of the game code + // the preset-1 value of an option (MasterOptionDefault, scene.json settings.optionDefaults: a string) option(scene, name) { - const o = scene.master.optionDefaultsPreset1[name]; - if (!o) throw new Error(`option ${name} missing`); - return o.value; + const o = scene.settings.optionDefaults[name]; + if (o === undefined) throw new Error(`option ${name} missing`); + return o; }, - build(scene, prefab, settings = null) { + // the lane skin's sprites (scene.json laneSkins[settings.laneSkin]: lane_base, lane_tap_area, out_side_line) as + // LiveObjects sprites + skin(objects, scene) { + const ids = scene.laneSkins[scene.settings.laneSkin]; + if (!ids) throw new Error(`lane skin ${scene.settings.laneSkin} missing`); + return Object.fromEntries(["lane_base", "lane_tap_area", "out_side_line"].map((k) => [k, objects.sprite(ids[k])])); + }, + + // skin: LiveLaneLayout.skin + build(scene, prefab, skin, settings = null) { const L = LiveLaneLayout, n = L.LANE_COUNT; const opt = (k) => (settings && settings[k] !== undefined ? settings[k] : Number(L.option(scene, k))); const unit01 = (v) => { const x = F(v / 100); return x < 0 ? 0 : x > 1 ? 1 : x; }; - const view = prefab.component("LiveGameView/root/LiveGameLane/lines", "LiveLaneLineView"); + const view = prefab.fields("LiveGameView/root/LiveGameLane/lines", "LiveLaneLineView"); const width = F(view._laneWidth), length = F(view._lineLength), spaceLen = F(view._spaceLineLength); const judgementZ = F(prefab.transform("LiveGameView/root/LiveGameLane/judgement_root").localPosition.z); const split = opt("GuidelineCount"); const lineAlpha = unit01(opt("GuidelineOpacity")); const main = L.SPLIT_MAIN[split], sub = L.SPLIT_SUB[split]; // TrySetupSettingDictionary: OutSide widths = out_side_line texture height / 100 - const outW = F(scene.laneSkin.out_side_line.texture.height / 100); + const outW = F(skin.out_side_line.texture.height / 100); const lineSettings = new Map(view._settings.map((s) => [s.LineType, s.LineType === 1 ? { ...s, LineWidthFrom: outW, LineWidthTo: outW } : s])); const lines = []; @@ -181,23 +190,25 @@ export class LiveLane { // renderer.settings: the live settings (settings.js) or undefined (the data's defaults) constructor(renderer, scene) { this.r = renderer; this.scene = scene; this.prefab = renderer.prefab; - const P = "LiveGameView/root/LiveGameLane"; + const O = renderer.objects, P = "LiveGameView/root/LiveGameLane"; this.paths = { lane: P, base: `${P}/base`, laneBase: `${P}/base/lane_base`, tapArea: `${P}/judgement_root/tap_area` }; - this.laneBaseSR = this.prefab.component(this.paths.laneBase, "SpriteRenderer"); - this.tapSR = this.prefab.component(this.paths.tapArea, "SpriteRenderer"); - this.baseCanvas = this.prefab.component(this.paths.base, "Canvas"); - this.baseScaler = this.prefab.component(this.paths.base, "CanvasScaler"); - const lv = this.prefab.component(`${P}/lines`, "LiveLaneLineView"); - this.outsideMat = lv._outSideLineMaterial; - // inner lines: instances of LiveLaneLineView._linePrefab EmbLive/Prefabs/LiveGame/lane_line (scene.assets.laneLinePrefab) - const lp = this.linePrefab = new Prefab(scene.assets.laneLinePrefab), lpRoot = lp.root.name; - const lineLR = this.lineLR = lp.component(lpRoot, "LineRenderer"); - this.lineMat = lineLR.m_Materials[0]; + this.laneBaseSR = this.prefab.fields(this.paths.laneBase, "SpriteRenderer"); + this.tapSR = this.prefab.fields(this.paths.tapArea, "SpriteRenderer"); + this.baseCanvas = this.prefab.fields(this.paths.base, "Canvas"); + this.baseScaler = this.prefab.fields(this.paths.base, "CanvasScaler"); + this.skin = LiveLaneLayout.skin(O, scene); + const lv = this.prefab.fields(`${P}/lines`, "LiveLaneLineView"); + this.outsideMat = O.material(lv._outSideLineMaterial); + // inner lines: instances of LiveLaneLineView._linePrefab EmbLive/Prefabs/LiveGame/lane_line (scene.json + // prefabs.laneLine) + const lp = this.linePrefab = O.prefab(scene.prefabs.laneLine), lpRoot = lp.root.name; + const lineLR = this.lineLR = lp.fields(lpRoot, "LineRenderer"); + this.lineMat = O.material(lineLR.m_Materials[0]); // LineRenderer parameters of the three line kinds must be what LiveGeom.lineStrip builds (world space, // alignment View, texture mode Stretch, no corner / cap vertices); width and colour come from the setters. const renderers = [[lp, lpRoot, lineLR], [this.prefab, `${P}/lines/lane_line_left`, null], [this.prefab, `${P}/lines/lane_line_right`, null]]; for (const [pf, path, lr0] of renderers) { - const lr = lr0 || pf.component(path, "LineRenderer"), q = lr.m_Parameters; + const lr = lr0 || pf.fields(path, "LineRenderer"), q = lr.m_Parameters; if (!lr.m_UseWorldSpace || q.alignment !== 0 || q.textureMode !== 0 || q.numCornerVertices || q.numCapVertices || q.textureScale.x !== 1 || lr.m_Loop) throw new Error(`${path}: LineRenderer parameters outside LiveGeom.lineStrip`); @@ -214,7 +225,7 @@ export class LiveLane { // of the lane's draw, so a new value set during a live gives the lane a live started with it would have. configure(settings) { const P = "LiveGameView/root/LiveGameLane"; - this.layout = LiveLaneLayout.build(this.scene, this.prefab, settings); + this.layout = LiveLaneLayout.build(this.scene, this.prefab, this.skin, settings); // sorting orders: LiveRendererOrderInLayerSetter of each line (LiveLane 41 + 0 -> 4100) const innerOrder = LiveLane.sortingOrder(this.linePrefab, this.linePrefab.root.name, this.lineLR.m_SortingOrder); for (const l of this.layout.lines) @@ -226,8 +237,8 @@ export class LiveLane { } async load() { - const gl = this.r.gl, base = this.r.base, sk = this.scene.laneSkin; - const T = (d) => GLTex.load(gl, base, d); + const gl = this.r.gl, sk = this.skin; + const T = (t) => this.r.objects.texture(t.id); this.tex = { laneBase: await T(sk.lane_base.texture), tapArea: await T(sk.lane_tap_area.texture), outside: await T(sk.out_side_line.texture), // Material.SetTexture(_MainTex, out_side_line) line: await T(this.lineMat.textures._MainTex.texture) }; @@ -273,9 +284,9 @@ export class LiveLane { // ENGINE: Unity 6 passes the sprite colour via unity_SpriteColor (SRP-batched) or vertex colour; multiplied once. // unity_GUIZTestMode (a ZTest property of UI-style shaders drawn outside a canvas): 0 = no depth test. // ENGINE: unity_GUIZTestMode outside a canvas is native global state; 0 here, LEqual would draw the same pixels. - // (Nothing writes depth in the live cameras' lists.) + // (Nothing writes depth in the live cameras' lists.) `sr`: the SpriteRenderer's fields (its material a reference). static spriteItem(r, vbo, sr, mesh, M, tex, color, order) { - const mat = sr.m_Materials[0], shader = mat.shader.shader; + const mat = r.objects.material(sr.m_Materials[0]), shader = mat.shader; const cen = mat4.transformPoint(M, LiveLane._center(mesh.pos)); return { sortingLayer: sr.m_SortingLayer || 0, sortingOrder: order, queue: LiveLane.queue(r.lib, mat), center: cen, @@ -306,7 +317,7 @@ export class LiveLane { // one LineRenderer (world space, identity object matrix) with Sirius/Live/Sprite/Default _lineItem(l, camPos) { - const mat = l.material === "outside" ? this.outsideMat : this.lineMat, shader = mat.shader.shader; + const mat = l.material === "outside" ? this.outsideMat : this.lineMat, shader = mat.shader; const tex = l.material === "outside" ? this.tex.outside : this.tex.line; const an = l.controlled ? this.lines : this.sideLines; const c0 = [l.c0[0], l.c0[1], l.c0[2], F(l.c0[3] * an.animationStartAlpha)]; @@ -360,14 +371,14 @@ export const LIVE_UI_DEAD_INPUTS = { _ClipRect: [-32767, -32767, 32767, 32767], LiveLane.QUEUE = { Background: 1000, Geometry: 2000, AlphaTest: 2450, Transparent: 3000, Overlay: 4000 }; // Material.renderQueue: the material's custom queue (m_CustomRenderQueue >= 0), else the shader's "Queue" tag -// ("Transparent", "Geometry+1", ...) of subshader 0. +// ("Transparent", "Geometry+1", ...) of subshader 0. `mat`: a LiveObjects material. LiveLane.queue = (lib, mat) => { if (mat.renderQueue >= 0) return mat.renderQueue; - const tags = lib.info(mat.shader.shader).subShaders[0].tags; + const tags = lib.info(mat.shader).subShaders[0].tags; const q = ((tags && tags.tags) || []).find(([k]) => k.toUpperCase() === "QUEUE"); if (!q) return 2000; const m = /^(\w+)([+-]\d+)?$/.exec(q[1]); - if (!m || !(m[1] in LiveLane.QUEUE)) throw new Error(`${mat.shader.shader}: queue tag ${q[1]}`); + if (!m || !(m[1] in LiveLane.QUEUE)) throw new Error(`${lib.nameOf(mat.shader)}: queue tag ${q[1]}`); return LiveLane.QUEUE[m[1]] + Number(m[2] || 0); }; @@ -377,9 +388,10 @@ export const LIVE_ORDER_IN_LAYER = { 10: 1000, 25: 2500, 26: 5700, 30: 3000, 40: 50: 5000, 55: 5500, 60: 6000, 100: 10000, 101: 4002, 200: 10002, 750: 9464, 751: 10000, 752: 2600, 753: 4050, 9100: 4600, 9101: 4699 }; LiveLane.sortingOrder = (prefab, path, fallback) => { - const s = prefab.components(path).find((c) => c.class === "LiveRendererOrderInLayerSetter" || - c.class === "LiveCanvasOrderInLayerSetter"); - if (!s) return fallback; + const c = prefab.components(path).find((x) => componentClass(x) === "LiveRendererOrderInLayerSetter" || + componentClass(x) === "LiveCanvasOrderInLayerSetter"); + if (!c) return fallback; + const s = c.fields; const v = LIVE_ORDER_IN_LAYER[s._orderInLayer]; if (v === undefined) throw new Error(`${path}: LiveOrderInLayer ${s._orderInLayer}`); return v + (s._offset || 0); diff --git a/src/live/legacy.js b/src/live/legacy.js index e60d882..6a26bd0 100644 --- a/src/live/legacy.js +++ b/src/live/legacy.js @@ -1,22 +1,43 @@ import { AssetStore } from "../data/assets.js"; import { LIVE_PRODUCT } from "./store.js"; -import { LIVE_JSON_FORMAT, MUSIC_FORMAT, SOUNDS_FORMAT } from "./data.js"; +import { LIVE_JSON_FORMAT, MUSIC_FORMAT, NOTEEFFECT_FORMAT, NOTES_FORMAT, NOTESKIN_FORMAT, SCENE_FORMAT, SOUNDS_FORMAT } from "./data.js"; // Charts of the site layout with a live directory (docs/data-format.md "Chart manifest"): a chart manifest without // `schema` whose files are a live.json index, the score, audio/live-audio.json with its waveforms, livescene/ and -// livenotes/. legacyLive() reads such a store as the store of a live: a live.json of the live product, sounds.json and -// music.json of the live.sounds and live.music products, so ChartSession reads both through LiveData. +// livenotes/. legacyLive() reads such a store as the store of a live: the documents of the live, scene, notes, note +// skin, note effect, sounds and music products, and the objects they name, so ChartSession reads both through +// LiveData and LiveObjects. +// +// The chart's scene.json and notes.json hold their objects inline: prefabs as node lists, materials, sprites, meshes, +// MonoBehaviour assets and clips where they are used, textures as file descriptors, clips and controllers of the note +// assets by key, shaders as packed directories. Each becomes an object of the store: livescene/ and livenotes/ are one +// serialized file each, and an inline object becomes one object of its file per distinct content (a material, sprite, +// mesh, MonoBehaviour asset or clip), per texture file, per clip / controller key and per packed shader; a prefab's +// root GameObject gets the prefab's nodes as its prefab.json, its nodes and components numbered in the file. A +// reference between the scene's roots becomes a reference to the node or component; the effect settings' references +// to their effect prefabs become references to the effect asset component on the prefab's root. The clips' bindings +// already carry their paths and attributes: the bindings tables list them for every Animator that plays them (the +// controller's Animator, a LoopAnimationTimeSync's Animator and the flick views' for the skin's arrow clips, a +// timeline track's bound Animator). // whether `store` holds a chart of the site layout with a live directory (no product `schema`, a live.json) -export const isLegacyLive = (store) => !!store && (!store.info || store.info.schema === undefined) && store.has("live.json"); +export const isLegacyLive = (store) => !!store && (!store.info || (store.info.schema === undefined && store.info.product === undefined)) + && store.has("live.json"); // A store over another one: files added in memory (`text`) and files that are another path of the inner store -// (`alias`); every other path reads the inner store as it is. +// (`alias`); every other path reads the inner store as it is. `objects`: the store's objects (AssetStore); `images`: +// object id -> the inner store's file its image is (decoded as the inner store decodes it). class OverlayStore extends AssetStore { - constructor(inner, { text = {}, alias = {}, info = null } = {}) { - super({ text, info }); + constructor(inner, { text = {}, alias = {}, objects = {}, images = new Map(), info = null } = {}) { + super({ text, objects, info }); this.inner = inner; this.alias = new Map(Object.entries(alias)); + this.images = images; + } + + objectImage(id, role = "image") { + const p = role === "image" ? this.images.get(id) : undefined; + return p !== undefined ? this.inner.image(p) : super.objectImage(id, role); } _path(p) { return this.alias.get(p) ?? p; } @@ -30,33 +51,430 @@ class OverlayStore extends AssetStore { } } +const ref = (id) => { + const i = id.lastIndexOf(":"); + return { $ref: { file: id.slice(0, i), pathId: id.slice(i + 1) } }; +}; +// JSON text of a value; infinities are written as 1e999 / -1e999 (JSON.parse reads them back as Infinity) +const INF = "\u0000inf", NEG_INF = "\u0000-inf"; +const json = (v) => JSON.stringify(v, (k, x) => (x === Infinity ? INF : x === -Infinity ? NEG_INF : x)) + .replaceAll('"\\u0000inf"', "1e999").replaceAll('"\\u0000-inf"', "-1e999"); +const triangles = (flat) => { + if (flat.length % 3) throw new Error(`${flat.length} indices: not triangles`); + const out = []; + for (let i = 0; i < flat.length; i += 3) out.push(flat.slice(i, i + 3)); + return out; +}; +const hex = (bytes) => ({ $hex: bytes.map((b) => b.toString(16).padStart(2, "0")).join("") }); +const without = (o, keys) => Object.fromEntries(Object.entries(o).filter(([k]) => !keys.includes(k))); + +// the inline object kinds of the chart's files +const isMaterial = (v) => typeof v.material === "string" && !!v.shader && typeof v.shader.shader === "string"; +const isSprite = (v) => typeof v.sprite === "string" && !!v.texture && typeof v.texture === "object" && !!v.rect; +const isTexture = (v) => typeof v.texture === "string" && typeof v.width === "number"; +const isMesh = (v) => typeof v.mesh === "string" && Array.isArray(v.vertices); +const isClip = (v) => Array.isArray(v.bindings) && !!v.streamed; +const isMono = (v) => typeof v.asset === "string"; +const isNodeRef = (v) => typeof v.gameObject === "string" || typeof v.transform === "string"; +const only = (v, k) => { const ks = Object.keys(v); return ks.length === 1 && ks[0] === k && typeof v[k] === "string"; }; + +// The objects of one directory of the chart (livescene/ or livenotes/), one serialized file `file`, added to `out`. +// `notes`: notes.json (the clips and controllers by key) of the file that has them, `peer`: that file for the other +// one (its Animators' controllers); `names`: clip id -> [{path, attribute}] of its bindings, `images`: Texture2D id -> +// its image file, both shared by the files. +class LegacyFile { + constructor(store, base, file, out, { notes = null, peer = null, names = new Map(), images = new Map() } = {}) { + this.store = store; this.base = base; this.file = file; this.out = out; this.images = images; + this.notes = notes; this.peer = peer; + this.n = 0; + this.memo = new Map(); + this.names = names; + this.tracks = []; // timeline tracks with their bound Animator: {id, name, bound, clips} + this._shaders = null; + } + + pathId() { this.n += 1; return String(this.n); } + + // an object of the file (once per memo key): files() -> {role: text | bytes} + add(cls, name, files, key = null) { + if (key !== null && this.memo.has(key)) return this.memo.get(key); + const id = `${this.file}:${this.pathId()}`; + if (key !== null) this.memo.set(key, id); + this.out[id] = { class: cls, name, files: files() }; + return id; + } + + // the packed shaders of the directory as Shader objects: name -> object id + shaders() { + if (this._shaders) return this._shaders; + const dir = `${this.base}/shaders`, S = this.store; + this._shaders = {}; + for (const r of S.json(`${dir}/shaders.json`)) { + this._shaders[r.name] = this.add("Shader", r.name, () => { + const files = { json: S.text(`${dir}/${r.parsed}`) }; + const variants = r.variants.map(({ file, ...v }) => { + const role = `program:${file.slice(file.indexOf("/") + 1)}`; + files[role] = S.text(`${dir}/${file}`); + return { ...v, role }; + }); + files.index = json({ name: r.name, variants }); + return files; + }); + } + return this._shaders; + } + + // a Shader by name; a shader the directory does not pack is a reference to no object of the store + shader(name) { + const id = this.shaders()[name], k = `shader:${name}`; + if (id) return id; + if (!this.memo.has(k)) this.memo.set(k, `${this.file}:${this.pathId()}`); + return this.memo.get(k); + } + + // a texture descriptor {texture (path under the directory), name, width, height, mipCount, colorSpace, settings}; a + // texture whose image the chart does not carry has no image file + texture(d) { + const path = `${this.base}/${d.texture}`, has = this.store.has(path); + const id = this.add("Texture2D", d.name, () => ({ + json: json({ m_Name: d.name, m_Width: d.width, m_Height: d.height, m_MipCount: d.mipCount, m_ColorSpace: d.colorSpace, + m_TextureSettings: d.settings }), + ...(has ? { image: this.store.bytes(path) } : {}), + }), `texture:${d.texture}`); + if (has) this.images.set(id, path); + return id; + } + + // a sprite record {sprite, texture, rect, pivot, border, pixelsToUnits, offset, textureRect, textureRectOffset, + // atlasRectOffset, downscaleMultiplier, settingsRaw, uvTransform, vertices, indices} + sprite(s) { + const { key: _key, ...rec } = s; + return this.add("Sprite", s.sprite, () => ({ json: json({ + sprite: { m_Name: s.sprite, m_Rect: s.rect, m_Pivot: s.pivot, m_Border: s.border, m_PixelsToUnits: s.pixelsToUnits, + m_Offset: s.offset }, + renderData: { texture: ref(this.texture(s.texture)), textureRect: s.textureRect, textureRectOffset: s.textureRectOffset, + ...(s.atlasRectOffset ? { atlasRectOffset: s.atlasRectOffset } : {}), + downscaleMultiplier: s.downscaleMultiplier ?? 1, settingsRaw: s.settingsRaw, uvTransform: s.uvTransform }, + mesh: s.vertices ? { positions: s.vertices, submeshes: [triangles(s.indices)] } : null, + }) }), `sprite:${json(rec)}`); + } + + // a material record {material, shader: {shader}, keywords, ..., textures {slot: {texture: descriptor | null, scale, + // offset}}, ints, floats, colors} + material(m) { + return this.add("Material", m.material, () => ({ json: json({ + ...m, shader: ref(this.shader(m.shader.shader)), + textures: Object.fromEntries(Object.entries(m.textures || {}).map(([k, t]) => + [k, { ...t, texture: t.texture ? ref(this.texture(t.texture)) : null }])), + }) }), `material:${json(m)}`); + } + + // a mesh record {mesh, vertices, normals, uv0, submeshes (flat indices), m_Skin, m_BindPose, ...} + mesh(m) { + return this.add("Mesh", m.mesh, () => { + const arrays = { positions: m.vertices, normals: m.normals ?? null, uv0: m.uv0 ?? null, submeshes: m.submeshes.map(triangles) }; + if (m.m_Skin) { + arrays.joints = m.m_Skin.map((w) => w.boneIndex); + arrays.weights = m.m_Skin.map((w) => w.weight); + } + return { json: json(without(m, ["mesh", "vertices", "normals", "uv0", "submeshes", "m_Skin"])), arrays: json(arrays) }; + }, `mesh:${json(m)}`); + } + + // an exported clip (bindings with path and attribute); its bindings' names are kept for the bindings tables + clip(c) { + const { clip: _key, ...raw } = c; + const k = `clip:${json(raw)}`; + if (this.memo.has(k)) return this.memo.get(k); + const id = this.add("AnimationClip", raw.name, () => ({ json: json({ + ...raw, + bindings: raw.bindings.map((b) => ({ + pathCrc: b.pathCrc ?? 0, attributeCrc: b.attributeCrc ?? 0, typeID: b.typeID, class: b.class, + customType: b.customType ?? 0, pptr: !!b.pptr, int: !!b.isIntCurve, serializeReference: false, + ...(b.curves ? { curves: b.curves } : {}), + })), + }) }), k); + // a PPtr curve on m_Sprite is stored without attribute + this.names.set(id, raw.bindings.map((b) => ({ + path: b.path, attribute: b.pptr && b.attribute == null && !b.attributeCrc ? "m_Sprite" : b.attribute }))); + return id; + } + + // a clip / controller of notes.json by key + clipKey(key) { + const c = this.notes && this.notes.clips[key]; + if (!c) throw new Error(`livenotes: clip ${key} not exported`); + return this.clip(c); + } + + controllerKey(key) { + const c = this.notes && this.notes.controllers[key]; + if (!c) throw new Error(`livenotes: controller ${key} not exported`); + return this.add("AnimatorController", c.name, () => ({ + json: json({ ...c, clips: c.clips.map((e) => ref(this.clipKey(e.clip))) }), + }), `controller:${key}`); + } + + // Animator.m_Controller: {controller: key} of notes.json, or {name} of a controller notes.json has under that name + // (one only); else as it is (a controller the chart does not carry) + controller(v) { + if (!v) return v; + if (!this.notes) return this.peer ? this.peer.controller(v) : v; + if (only(v, "controller")) return ref(this.controllerKey(v.controller)); + const ctrls = Object.entries(this.notes.controllers).filter(([, c]) => c.name === v.name); + return ctrls.length === 1 ? ref(this.controllerKey(ctrls[0][0])) : v; + } + + // a MonoBehaviour asset {asset: class, name, m_Name, ...fields}; a timeline track's bound Animator is kept aside + mono(v, cx) { + const fields = this.value(without(v, ["asset", "name", "key", "boundAnimator"]), cx); + const id = this.add("MonoBehaviour", v.m_Name ?? v.name ?? v.asset, () => ({ json: json({ $script: v.asset, ...fields }) }), + `mono:${json(v)}`); + if (v.boundAnimator && !this.tracks.some((t) => t.id === id)) + this.tracks.push({ id, name: v.m_Name, bound: v.boundAnimator, + clips: (v.m_Clips || []).map((c) => this.clip(c.m_Asset.m_Clip)) }); + return id; + } + + // a field value with its inline objects as references; cx: {root, layout} of the prefab being converted + value(v, cx) { + if (Array.isArray(v)) return v.map((x) => this.value(x, cx)); + if (!v || typeof v !== "object" || v.$ref !== undefined) return v; + if (isMaterial(v)) return ref(this.material(v)); + if (isSprite(v)) return ref(this.sprite(v)); + if (isTexture(v)) return ref(this.texture(v)); + if (isMesh(v)) return ref(this.mesh(v)); + if (isClip(v)) return ref(this.clip(v)); + if (only(v, "clip") && this.notes) return ref(this.clipKey(v.clip)); + if (isMono(v)) return ref(this.mono(v, cx)); + if (isNodeRef(v) && cx && cx.layout) return this.nodeRef(v, cx); + return Object.fromEntries(Object.entries(v).map(([k, x]) => [k, this.value(x, cx)])); + } + + // a node / component reference {gameObject | transform, component?, class?}: as it is inside its root, a reference to + // the node or component in another root of the layout + nodeRef(v, cx) { + const p = v.gameObject ?? v.transform, n = cx.layout.get(p); + if (p.split("/")[0] === cx.root || !n) return v; + if (v.transform !== undefined) return ref(`${this.file}:${n.transformPathId}`); + if (v.component === undefined) return ref(`${this.file}:${n.pathId}`); + const c = n.components.find((x) => x.type === v.component && (v.class === undefined || x.class === v.class)); + if (!c) throw new Error(`${p}: no ${v.class || v.component}`); + return ref(`${this.file}:${c.pathId}`); + } + + // node path -> {pathId, transformPathId, components: [{type, class, pathId}]} of prefabs' nodes, numbered in the file + layout(prefabs, into = new Map()) { + for (const nodes of prefabs) for (const n of nodes) { + into.set(n.path, { pathId: this.pathId(), transformPathId: this.pathId(), + components: n.components.map((c) => ({ type: c.type, class: c.class, pathId: this.pathId() })) }); + } + return into; + } + + // a prefab's node list as its root GameObject (prefab.json); `layout` numbers its nodes (layout()) + prefab(nodes, layout) { + const root = nodes[0].path, cx = { root, layout }; + const out = nodes.map((n) => { + const L = layout.get(n.path); + return { + ...without(n, ["components"]), pathId: L.pathId, transformPathId: L.transformPathId, + components: n.components.map((c, i) => { + let fields = without(c, ["type", "class"]); + if (c.type === "Animator") fields = { ...fields, m_Controller: this.controller(fields.m_Controller) }; + if (c.type === "ParticleSystemRenderer") + for (const k of ["m_VertexStreams", "m_TrailVertexStreams"]) if (Array.isArray(fields[k])) fields = { ...fields, [k]: hex(fields[k]) }; + return { type: c.type, pathId: L.components[i].pathId, ...(c.class ? { script: c.class } : {}), fields: this.value(fields, cx) }; + }), + }; + }); + const id = `${this.file}:${layout.get(root).pathId}`; + this.out[id] = { class: "GameObject", name: nodes[0].name, files: { json: json({ name: nodes[0].name, nodes: out }) } }; + return id; + } + + // a prefab {key, nodes} with a layout of its own (once per key) + prefabOf(p) { + const k = `prefab:${p.key}`; + if (!this.memo.has(k)) this.memo.set(k, this.prefab(p.nodes, this.layout([p.nodes]))); + return this.memo.get(k); + } +} + +// the root of the first path segment's nodes: the node lists of the roots of a flat hierarchy, in order +const roots = (nodes) => { + const by = new Map(); + for (const n of nodes) { + const r = n.path.split("/")[0]; + if (!by.has(r)) by.set(r, []); + by.get(r).push(n); + } + return [...by.values()]; +}; + +// prefab id -> animator path -> clip id -> names +const addBindings = (table, prefab, animator, clip, names) => { + ((table[prefab] ??= {})[animator] ??= {})[clip] = names; +}; + +// The bindings of the Animators of a prefab that play clips: an Animator's controller clips, a LoopAnimationTimeSync's +// clip on its Animator, and `extra(sync)` clips on a LoopAnimationTimeSync's Animator (the skin's arrow clips). +const prefabBindings = (F, id, table, extra = null) => { + for (const n of JSON.parse(F.out[id].files.json).nodes) for (const c of n.components) { + if (c.type === "Animator" && c.fields.m_Controller && c.fields.m_Controller.$ref) { + const ctrl = JSON.parse(F.out[`${ref2id(c.fields.m_Controller)}`].files.json); + for (const r of ctrl.clips) addBindings(table, id, n.path, ref2id(r), F.names.get(ref2id(r))); + } + if (c.script === "LoopAnimationTimeSync") { + const at = c.fields._animator.gameObject; + if (c.fields._loopAnim) addBindings(table, id, at, ref2id(c.fields._loopAnim), F.names.get(ref2id(c.fields._loopAnim))); + for (const clip of extra ? extra() : []) addBindings(table, id, at, clip, F.names.get(clip)); + } + } +}; +const ref2id = (v) => `${v.$ref.file}:${v.$ref.pathId}`; + +const SCENE_FILE = "livescene", NOTES_FILE = "livenotes"; + +// scene.json and notes.json of the chart as the documents and objects of the live.scene, live.notes, live.noteskin and +// live.noteeffect products: {docs: {scene, notes, noteskin, noteeffect}, objects, band, jacket, skin, effect} +const legacyScene = (store, sceneFile, notesFile) => { + const objects = {}, names = new Map(), images = new Map(); + const sc = store.json(sceneFile), nt = store.json(notesFile); + const N = new LegacyFile(store, "livenotes", NOTES_FILE, objects, { notes: nt, names, images }); + const S = new LegacyFile(store, "livescene", SCENE_FILE, objects, { peer: N, names, images }); + + // the scene's roots, numbered together (references between them), then the timeline and the lane line prefab + const rootNodes = roots(sc.scene.nodes), layout = S.layout(rootNodes); + const scene = rootNodes.map((nodes) => S.prefab(nodes, layout)); + const rootOf = (path) => scene[rootNodes.findIndex((nodes) => nodes[0].path === path.split("/")[0])]; + const timeline = S.prefabOf(sc.assets.startTimeline), laneLine = S.prefabOf(sc.assets.laneLinePrefab); + + // the graphics settings: the default pipeline, its renderers and the film grain textures of their post data + const P = sc.player, pipe = P.pipelines[P.defaultPipeline], renderers = {}; + for (const [name, r] of Object.entries(P.renderers)) { + const post = r.postProcessData ? P.postProcessData[r.postProcessData] : null, grain = sc.postTextures[name]; + const postId = post ? S.add("MonoBehaviour", post.m_Name, () => ({ json: json({ $script: "PostProcessData", ...post, + ...(grain ? { textures: { ...post.textures, filmGrainTex: grain.filmGrainTex.map((d) => ref(S.texture(d))) } } : {}) }) })) : null; + renderers[name] = S.add("MonoBehaviour", name, () => ({ json: json({ $script: "UniversalRendererData", ...r, + postProcessData: postId ? ref(postId) : null }) })); + } + const pipeId = S.add("MonoBehaviour", pipe.m_Name, () => ({ json: json({ $script: "UniversalRenderPipelineAsset", ...pipe, + m_RendererDataList: pipe.m_RendererDataList.map((n) => (n ? ref(renderers[n]) : null)) }) })); + const graphics = { + graphics: S.add("GraphicsSettings", "", () => ({ json: json({ m_CustomRenderPipeline: ref(pipeId) }) })), + quality: S.add("QualitySettings", "", () => ({ json: json({ m_QualitySettings: P.qualityLevels }) })), + player: S.add("PlayerSettings", "", () => ({ json: json({ m_ActiveColorSpace: P.colorSpace === "Linear" ? 1 : 0 }) })), + }; + + // the band's start timeline: its tracks' bound Animators (the scene or the timeline prefab) + const band = sc.slice.band, tracks = {}, sceneBindings = {}; + for (const t of S.tracks) { + const prefab = t.bound.key === sc.assets.startTimeline.key ? timeline : t.bound.key === sc.scene.key ? rootOf(t.bound.gameObject) : null; + if (!prefab) throw new Error(`timeline track ${t.name}: bound in ${t.bound.key}`); + tracks[t.id] = { name: t.name, prefab, animator: t.bound.gameObject }; + for (const clip of t.clips) addBindings(sceneBindings, prefab, t.bound.gameObject, clip, S.names.get(clip)); + } + + // note assets: the note view prefabs, the effect prefabs, the UI prefab and sprite assets, the skin + const ns = nt.settings, judge = N.prefabOf(nt.prefabs.judge_effect_view); + for (const id of scene) prefabBindings(S, id, sceneBindings); + const prefabs = {}, notesBindings = {}, skinBindings = {}, effectBindings = {}; + const skin = N.mono(nt.noteSkin, null), skinDoc = JSON.parse(objects[skin].files.json); + const arrows = () => ["FlickNoteAsset", "LeftFlickNoteAsset", "RightFlickNote"].filter((k) => skinDoc[k]) + .map((k) => JSON.parse(objects[ref2id(skinDoc[k])].files.json)) + .filter((u) => u._arrowLoopAnimation).map((u) => ref2id(u._arrowLoopAnimation)); + for (const [name, p] of Object.entries(nt.prefabs)) { + if (name === "judge_effect_view" || name === "bar_line_view") continue; + prefabs[name] = N.prefabOf(p); + prefabBindings(N, prefabs[name], notesBindings); + prefabBindings(N, prefabs[name], skinBindings, arrows); + } + // an effect set's settings: each field a reference to the effect asset component on the root of its prefab + // (/) + const effectSet = (kind, set, cls, table) => { + const dir = `Effect/Live/${kind}/${set}`, s = nt.assets[`${dir}/${kind === "NoteEffect" ? "LiveNoteEffectAssetSettings" : "LiveLaneEffectAssetSettings"}`]; + if (!s) throw new Error(`livenotes: ${dir} settings not exported`); + const v = { ...s }; + for (const [k, x] of Object.entries(s)) { + const p = x && typeof x === "object" && x.class === cls ? nt.assets[`${dir}/${x.gameObject}`] : null; + if (!p) continue; + const id = N.prefabOf(p), root = JSON.parse(objects[id].files.json).nodes[0]; + prefabBindings(N, id, table); + v[k] = { $ref: { file: NOTES_FILE, pathId: root.components.find((c) => c.script === cls).pathId } }; + } + return N.mono(v, null); + }; + const laneSet = "effect001"; // LaneEffectLoadStep.GetAssetPath + const laneEffect = effectSet("LaneEffect", laneSet, "LiveLaneEffectAsset", notesBindings); + const effect = effectSet("NoteEffect", ns.effect, "LiveNoteEffectAsset", effectBindings); + const spriteAsset = (k) => { + const a = nt.assets[`Live/UiSpriteAssets/${k}`]; + if (!a) throw new Error(`livenotes: Live/UiSpriteAssets/${k} not exported`); + return N.mono(a, null); + }; + + const m = sc.master; + const docs = { + scene: { + format: SCENE_FORMAT, scene, renderSettings: null, graphics, + laneSkins: { [sc.slice.laneSkin]: Object.fromEntries(Object.entries(sc.laneSkin).map(([k, s]) => [k, S.sprite(s)])) }, + prefabs: { laneLine, judgeEffectView: judge, barLineView: nt.prefabs.bar_line_view ? N.prefabOf(nt.prefabs.bar_line_view) : null }, + spriteAssets: { combo: spriteAsset("LiveComboSpriteAsset"), judgement: spriteAsset("LiveJudgementSpriteAsset") }, + bands: { [String(band)]: { background: S.sprite(sc.sprites.lightweightBackground), timeline, tracks } }, + settings: { + judgementSprites: ns.judgementSprites, laneSkin: sc.slice.laneSkin, + laneSkins: Object.fromEntries(m.liveLaneSkin.map((r) => [String(r._id), r._assetName])), + optionDefaults: Object.fromEntries(Object.entries(m.optionDefaultsPreset1).map(([k, o]) => [k, o.value])), + optionRanges: m.optionRanges, qualitySettings: m.liveQualitySettings, + }, + bindings: sceneBindings, shaders: S.shaders(), + }, + notes: { + format: NOTES_FORMAT, prefabs, laneEffect, settings: { ...without(ns, ["judgementSprites"]), laneEffect: laneSet }, + bindings: notesBindings, shaders: N.shaders(), + }, + noteskin: { format: NOTESKIN_FORMAT, name: ns.skin, skin, bindings: skinBindings, shaders: {} }, + noteeffect: { format: NOTEEFFECT_FORMAT, name: ns.effect, settings: effect, bindings: effectBindings, shaders: {} }, + }; + return { docs, objects, images, band, jacket: S.sprite(sc.sprites.jacket), skin: ns.skin, effect: ns.effect }; +}; + // The store of a chart of the site layout read as the store of a live (an OverlayStore over `store`): // lives/_/live.json, notes.json, notes.mirror.json the live: the score files (and the mirrored -// one when the chart has it), `parts` naming the products below +// one when the chart has it), the band, the jacket, `parts` naming the products below +// scenes/default/scene.json, notes/default/notes.json, noteskins//noteskin.json, +// noteeffects//noteeffect.json the scene and the note assets (legacyScene) // sounds/default/sounds.json live-audio.json without the music; its waveforms under sounds/default/ // music//music.json the music's sound; its waveforms under music// -// The scene and the note assets stay where they are (livescene/, livenotes/). `info` is the chart manifest's (chart, -// quality, options, audio) with the live's `product`, `subject`, `dir` and `products`. +// `info` is the chart manifest's (chart, quality, options, audio) with the live's `product`, `subject`, `dir` and +// `products`. export const legacyLive = (store) => { const old = store.json("live.json"); - for (const k of ["musicId", "difficulty", "notes", "liveAudio"]) + for (const k of ["musicId", "difficulty", "notes", "liveAudio", "scene", "noteAssets"]) if (old[k] === undefined) throw new Error(`live.json: no ${k}`); const id = `${old.musicId}_${old.difficulty}`, dir = `lives/${id}`; const sounds = "sounds/default", musicDir = `music/${old.musicId}`; + const L = legacyScene(store, old.scene, old.noteAssets); + const skinDir = `noteskins/${L.skin}`, effectDir = `noteeffects/${L.effect}`; const text = {}, alias = {}; alias[`${dir}/notes.json`] = old.notes; if (old.notesMirror) alias[`${dir}/notes.mirror.json`] = old.notesMirror; - const parts = { scene: "scenes/default.json", notes: "notes/default.json", noteskin: "noteskins/default.json", - noteeffect: "noteeffects/default.json", sounds: `${sounds}.json`, music: `${musicDir}.json` }; + const parts = { scene: "scenes/default.json", notes: "notes/default.json", noteskin: `${skinDir}.json`, + noteeffect: `${effectDir}.json`, sounds: `${sounds}.json`, music: `${musicDir}.json` }; text[`${dir}/live.json`] = JSON.stringify({ - format: LIVE_JSON_FORMAT, musicId: old.musicId, difficulty: old.difficulty, notes: "notes.json", - ...(old.notesMirror ? { notesMirror: "notes.mirror.json" } : {}), parts, + format: LIVE_JSON_FORMAT, musicId: old.musicId, difficulty: old.difficulty, band: { band: L.band }, jacket: L.jacket, + notes: "notes.json", ...(old.notesMirror ? { notesMirror: "notes.mirror.json" } : {}), parts, }); + text["scenes/default/scene.json"] = json(L.docs.scene); + text["notes/default/notes.json"] = json(L.docs.notes); + text[`${skinDir}/noteskin.json`] = json(L.docs.noteskin); + text[`${effectDir}/noteeffect.json`] = json(L.docs.noteeffect); const audio = store.json(old.liveAudio); const musicId = String(audio.music && audio.music.soundId); if (!audio.sounds || !audio.sounds[musicId]) throw new Error(`${old.liveAudio}: no sound of the music (${musicId})`); // a sound's waveforms under the product's directory - const moved = (into, cue) => ({ ...cue, layers: cue.layers.map((L) => { alias[`${into}/${L.file}`] = L.file; return L; }) }); + const moved = (into, cue) => ({ ...cue, layers: cue.layers.map((Lr) => { alias[`${into}/${Lr.file}`] = Lr.file; return Lr; }) }); const { music, sounds: all, ...rest } = audio; const se = Object.fromEntries(Object.entries(all).filter(([k]) => k !== musicId).map(([k, c]) => [k, moved(sounds, c)])); text[`${sounds}/sounds.json`] = JSON.stringify({ format: SOUNDS_FORMAT, ...rest, sounds: se }); @@ -65,8 +483,10 @@ export const legacyLive = (store) => { }); const info = { ...(store.info || {}), product: LIVE_PRODUCT, subject: `${old.musicId}/${old.difficulty}`, dir, - products: { [LIVE_PRODUCT]: { [`${old.musicId}/${old.difficulty}`]: dir }, "live.sounds": { default: sounds }, + products: { [LIVE_PRODUCT]: { [`${old.musicId}/${old.difficulty}`]: dir }, "live.scene": { default: "scenes/default" }, + "live.notes": { default: "notes/default" }, "live.noteskin": { [L.skin]: skinDir }, + "live.noteeffect": { [L.effect]: effectDir }, "live.sounds": { default: sounds }, "live.music": { [String(old.musicId)]: musicDir } }, }; - return new OverlayStore(store, { text, alias, info }); + return new OverlayStore(store, { text, alias, objects: L.objects, images: L.images, info }); }; diff --git a/src/live/noteview.js b/src/live/noteview.js index 259be2f..6c4c3d0 100644 --- a/src/live/noteview.js +++ b/src/live/noteview.js @@ -1,7 +1,8 @@ -import { AnimClip, AnimTargets, Gradient } from "../engine/anim.js"; +import { AnimTargets, Gradient } from "../engine/anim.js"; import { F } from "../engine/core.js"; -import { ShaderLib, applyState } from "../engine/glsl.js"; +import { applyState } from "../engine/glsl.js"; import { Transform, mat4 } from "../engine/math.js"; +import { componentClass } from "../engine/prefab.js"; import { GLTex } from "../engine/texture.js"; import { LiveLane } from "./lane.js"; import { NoteGeo } from "./notegeo.js"; @@ -20,7 +21,7 @@ import { LiveSettingsMath } from "./settings.js"; // ------------------------------------------------------------------ SpriteRenderer meshes export const SpriteMesh = { - // DrawMode Simple: the sprite's own mesh (exported vertices / uv / indices, atlas packing already resolved) + // DrawMode Simple: the sprite's own mesh (its render mesh vertices / uv / indices, atlas packing already resolved) simple(sp) { return { pos: Float32Array.from(sp.vertices.flatMap((v) => [v[0], v[1], v[2] ?? 0])), uv: Float32Array.from(sp.uv.flat()), idx: Uint16Array.from(sp.indices) }; @@ -29,9 +30,9 @@ export const SpriteMesh = { // DrawMode Sliced: 4 x 4 vertex grid over the rect at `size`, borders shrunk proportionally when they do not fit sliced(sp, sx, sy) { const ppu = sp.pixelsToUnits, r = sp.rect, tr = sp.textureRect, b = sp.border, tex = sp.texture; - if (((sp.settingsRaw >> 2) & 15) !== 0) throw new Error(`${sp.sprite}: rotated atlas packing with sliced draw mode`); + if (((sp.settingsRaw >> 2) & 15) !== 0) throw new Error(`${sp.name}: rotated atlas packing with sliced draw mode`); if (Math.abs(tr.width - r.width) > 1e-3 || Math.abs(tr.height - r.height) > 1e-3) - throw new Error(`${sp.sprite}: trimmed texture rect with sliced draw mode`); + throw new Error(`${sp.name}: trimmed texture rect with sliced draw mode`); const axis = (size, lo, hi, pivot, t0, tw, texSize) => { let a = lo / ppu, c = hi / ppu; if (a + c > size) { const k = size / (a + c); a *= k; c *= k; } @@ -56,12 +57,13 @@ export const SpriteMesh = { // ------------------------------------------------------------------ GL drawing of note renderers export class NoteGL { - constructor(gl, base) { + // objects: LiveObjects (its ShaderLib and GL textures) + constructor(gl, objects) { this.gl = gl; - this.base = base; // packed livenotes directory ("livenotes") - this.lib = new ShaderLib(gl, `${base}/shaders`); + this.objects = objects; + this.lib = objects.lib; this.vao = gl.createVertexArray(); - this.tex = new Map(); // texture path -> GLTex + this.tex = new Map(); // Texture2D object id -> GLTex (loaded) this.meshes = new Map(); // key -> {buffers, count, bounds} this.defTex = { white: GLTex.solid(gl, [255, 255, 255, 255], "white"), black: GLTex.solid(gl, [0, 0, 0, 255], "black"), @@ -71,14 +73,15 @@ export class NoteGL { idx: gl.createBuffer() }; } - async texture(desc) { - if (!this.tex.has(desc.texture)) this.tex.set(desc.texture, await GLTex.load(this.gl, this.base, desc)); - return this.tex.get(desc.texture); + // the GL texture of a texture record (AssetStore.texture) + async texture(t) { + if (!this.tex.has(t.id)) this.tex.set(t.id, await this.objects.texture(t.id)); + return this.tex.get(t.id); } // static sprite mesh, cached by sprite / draw mode / size spriteMesh(sp, drawMode, sx, sy) { - const key = drawMode === 1 ? `${sp.sprite}|${sx}|${sy}` : sp.sprite; + const key = drawMode === 1 ? `${sp.id}|${sx}|${sy}` : sp.id; let m = this.meshes.get(key); if (!m) { const g = this.gl, d = drawMode === 1 ? SpriteMesh.sliced(sp, sx, sy) : SpriteMesh.simple(sp); @@ -106,10 +109,10 @@ export class NoteGL { return [mat.floats, mat.colors, texST, this.lib.defaults(shader, this.defTex)]; } - // one SpriteRenderer: sr = serialized component (material, flip, colour), sprite + mesh, world matrix M; mpb: the - // renderer's MaterialPropertyBlock values (over the material's), or null + // one SpriteRenderer: sr = {m_Materials (LiveObjects materials), m_FlipX, m_FlipY, m_Color}, sprite + mesh, world + // matrix M; mpb: the renderer's MaterialPropertyBlock values (over the material's), or null drawSprite(ctx, sr, tex, mesh, M, color, mpb = null) { - const gl = this.gl, mat = sr.m_Materials[0], shader = mat.shader.shader; + const gl = this.gl, mat = sr.m_Materials[0], shader = mat.shader; const prog = this.lib.program(shader, 0, mat.keywords); const c = color || sr.m_Color, fx = sr.m_FlipX ? -1 : 1, fy = sr.m_FlipY ? -1 : 1; prog.apply([{ _MainTex: tex, _RendererColor: [c.r, c.g, c.b, c.a], _Flip: [fx, fy], @@ -133,7 +136,7 @@ export class NoteGL { // defaults), without drawing: the read-set plan (scripts/read-set.mjs --plan) makes them for the materials of // LiveNotes.plannedMaterials(), so the files they read are read without stepping the chart prepare(mat) { - const shader = mat.shader.shader; + const shader = mat.shader; this.lib.program(shader, 0, mat.keywords); this.lib.state(shader, 0, mat.floats); this.lib.defaults(shader, this.defTex); @@ -141,7 +144,7 @@ export class NoteGL { // one body mesh (Live/Unlit/SlideLine): m = NoteGeo.bodyMesh result, mpb = {_GradientTex, _Color, _GradientState} drawBody(ctx, mat, m, M, mpb) { - const gl = this.gl, shader = mat.shader.shader, prog = this.lib.program(shader, 0, mat.keywords); + const gl = this.gl, shader = mat.shader, prog = this.lib.program(shader, 0, mat.keywords); prog.apply([mpb, ctx.perObject(M), ...this._material(mat, shader), ctx.globals]); applyState(gl, this.lib.state(shader, 0, mat.floats)); gl.frontFace(gl.CW); // Unity's clockwise front faces (Cull Back in this pass) @@ -207,18 +210,21 @@ export const NoteViewType = { }; // ------------------------------------------------------------------ one prefab instance (transforms + renderers) +// prefab: a prefab.json; a node's `comp` is the fields of its SpriteRenderer / MeshRenderer and `materials` that +// renderer's materials (LiveObjects materials) export class NotePrefab { - constructor(prefab, skipPrefix) { + constructor(prefab, skipPrefix, objects) { this.root = null; - this.nodes = new Map(); // path -> {t: Transform, sr?, active} + this.nodes = new Map(); // path -> {t: Transform, comp, materials, active, ...} for (const n of prefab.nodes) { if (skipPrefix && skipPrefix.some((s) => n.path.includes(s))) continue; const cut = n.path.lastIndexOf("/"), parent = cut < 0 ? null : this.nodes.get(n.path.slice(0, cut)); if (cut >= 0 && !parent) continue; // below a skipped node const t = new Transform(n.name, parent ? parent.t : null); t.localPosition = { ...n.localPosition }; t.localRotation = { ...n.localRotation }; t.localScale = { ...n.localScale }; - const srC = n.components.find((c) => c.type === "SpriteRenderer" || c.type === "MeshRenderer"); - const node = { t, active: n.active && (!parent || parent.active), path: n.path, comp: srC || null, + const c = n.components.find((x) => x.type === "SpriteRenderer" || x.type === "MeshRenderer"), srC = c ? c.fields : null; + const node = { t, active: n.active && (!parent || parent.active), path: n.path, comp: srC, + materials: srC ? srC.m_Materials.map((m) => objects.material(m)) : null, sprite: null, size: srC && srC.m_Size ? { ...srC.m_Size } : null, flipX: srC ? !!srC.m_FlipX : false, enabled: srC ? !!srC.m_Enabled : false, order: srC ? srC.m_SortingOrder : 0, color: srC && srC.m_Color ? { ...srC.m_Color } : null }; @@ -269,17 +275,19 @@ export class ArrowGradient { export class NoteHeadView { constructor(owner, type) { this.owner = owner; this.type = type; - const prefab = owner.ln.prefabs[NoteViewType.prefab[type]]; - if (!prefab) throw new Error(`note view type ${type}: prefab missing`); - this.p = new NotePrefab(prefab, ["gekisou"]); + const id = owner.ln.prefabs[NoteViewType.prefab[type]]; + if (!id) throw new Error(`note view type ${type}: prefab missing`); + const prefab = owner.objects.assets.prefab(id); + this.prefabId = id; + this.p = new NotePrefab(prefab, ["gekisou"], owner.objects); const root = this.p.root.comp ? null : prefab.nodes[0].components.find((c) => c.type === "MonoBehaviour"); - this.mb = root || {}; + this.mb = root ? root.fields : {}; const ref = (k) => (this.mb[k] && this.mb[k].gameObject ? this.p.nodes.get(this.mb[k].gameObject) || null : null); this.main = ref("_mainSpriteRenderer"); this.mark = ref("_markSpriteRenderer"); this.left = ref("_leftSpriteRenderer"); this.right = ref("_rightSpriteRenderer"); this.arrow = ref("_flickArrowSpriteRenderer"); this.subArrow = ref("_subArrowSpriteRenderer"); this.parts = ref("_partsRenderer"); - this.unit = NoteViewType.skin[type] ? (owner.skin || owner.ln.noteSkin)[NoteViewType.skin[type]] : null; + this.unit = NoteViewType.skin[type] ? owner.skin[NoteViewType.skin[type]] : null; this.noteId = -1; } @@ -290,7 +298,7 @@ export class NoteHeadView { if (this.main) this.main.sprite = u._mainSprite; if (this.mark) this.mark.sprite = u._centerMarkSprite; if (this.arrow) { // LiveFlickNoteView / LiveDirectionFlickNoteView.OnSetupNoteSkin - this.clip = u._arrowLoopAnimation ? this.owner.clip(u._arrowLoopAnimation.clip) : null; // OverrideLoopAnimation + this.clip = u._arrowLoopAnimation ? this.owner.arrowClip(this, u._arrowLoopAnimation) : null; // OverrideLoopAnimation // the tilt (the tilt material and its _Slope / _Center / _HalfScale block) only with _arrowTiltEnabled and a tilt // material; without the material the game skips it (and so does LateUpdate) if (u._arrowTiltEnabled && u._arrowTiltMaterial) throw new Error("flick arrow tilt path not implemented (skin _arrowTiltEnabled)"); @@ -364,9 +372,9 @@ export class NoteHeadView { export class PairLineView { constructor(owner) { this.owner = owner; - this.p = new NotePrefab(owner.ln.prefabs.pair_note_line); + this.p = new NotePrefab(owner.objects.assets.prefab(owner.ln.prefabs.pair_note_line), null, owner.objects); this.r = this.p.nodes.get(this.p.root.comp ? this.p.root.path : `${this.p.root.path}/renderer`); - this.r.sprite = this.r.comp.m_Sprite; + this.r.sprite = owner.objects.sprite(this.r.comp.m_Sprite); } // LiveAllNoteView.TrySpawnPairNoteLine -> Setup(spawn, mid, w) setup(a, b) { @@ -387,9 +395,10 @@ export class PairLineView { export class NoteLineView { constructor(owner, kind) { this.owner = owner; this.kind = kind; - const pf = owner.ln.prefabs[kind === "slide" ? "slide_line_view" : "guide_line_view"]; - this.mr = pf.nodes[0].components.find((c) => c.type === "MeshRenderer"); - this.mb = pf.nodes[0].components.find((c) => c.type === "MonoBehaviour" && /NoteLineView$/.test(c.class)); + const pf = owner.objects.assets.prefab(owner.ln.prefabs[kind === "slide" ? "slide_line_view" : "guide_line_view"]); + const comp = (f) => { const c = pf.nodes[0].components.find(f); return c ? c.fields : null; }; + this.mr = comp((c) => c.type === "MeshRenderer"); + this.mb = comp((c) => /NoteLineView$/.test(componentClass(c))); this.lineId = -1; } // LiveAllNoteView.TrySpawnNoteLine -> SetActiveLineView(true), SetSlideAlpha, ApplyNoteLine @@ -446,7 +455,8 @@ export const musicScorePosition = (score, timeMs) => { return { bar: ref.bar + n, rhythm: Math.trunc(F(f * 8)) % 8, rhythmicUnit: 8, barProgress: F(f - n), timeMs }; }; -// LiveAllBarLineView with its LiveBarLineViewContainer and LiveBarLineView (the view prefab: prefabs.bar_line_view). +// LiveAllBarLineView with its LiveBarLineViewContainer and LiveBarLineView (the view prefab: scene.json +// prefabs.barLineView). // Initialize: the score's BarLineTimeMsList and _beforePlayingTimeMs = (float) the display offset; each view rented // from the container gets SpriteRenderer.size.x = LaneCount x the lane unit and is parented to the container's // _elementRoot. UpdateFrame (at the top of UpdateNoteView): from the bar after the frame position's Bar, every bar @@ -455,22 +465,24 @@ export const musicScorePosition = (score, timeMs) => { // reused are returned. LiveBarLineView.UpdateView: localPosition (0, y of the lerp from the spawn root to // GetNoteJudgementPosition(_laneCenter 0), 0), localScale the view progress. The sprite keeps its prefab sorting order. export class LiveBarLines { - constructor(owner, prefab, root) { + // id: the view prefab's root GameObject + constructor(owner, id, root) { + const O = owner.objects, prefab = O.assets.prefab(id); this.owner = owner; this.prefab = prefab; this.root = root; this.times = owner.score.barLineTimeMs || []; - const mb = prefab.nodes[0].components.find((c) => c.type === "MonoBehaviour" && c._renderer); - if (!mb || !mb._renderer.gameObject) throw new Error("bar line prefab: LiveBarLineView._renderer missing"); - this.rendererPath = mb._renderer.gameObject; - const sr = prefab.nodes.find((n) => n.path === this.rendererPath).components.find((c) => c.type === "SpriteRenderer"); - this.sprite = sr.m_Sprite; this.material = sr.m_Materials[0]; + const mb = prefab.nodes[0].components.find((c) => c.type === "MonoBehaviour" && c.fields._renderer); + if (!mb || !mb.fields._renderer.gameObject) throw new Error("bar line prefab: LiveBarLineView._renderer missing"); + this.rendererPath = mb.fields._renderer.gameObject; + const sr = prefab.nodes.find((n) => n.path === this.rendererPath).components.find((c) => c.type === "SpriteRenderer").fields; + this.sprite = O.sprite(sr.m_Sprite); this.material = O.material(sr.m_Materials[0]); this.active = []; this.pool = []; } // the container's _elementRoot relative to screen_root, from the live scene static elementRoot(scene) { - const nv = scene.component("LiveGameView/LiveGameCamera/screen_root/LiveGameAllNoteView", "LiveAllNoteView"); - const bl = scene.component(nv._barLineView.gameObject, "LiveAllBarLineView"); - const ct = scene.component(bl._container.gameObject, "LiveBarLineViewContainer"); + const nv = scene.fields("LiveGameView/LiveGameCamera/screen_root/LiveGameAllNoteView", "LiveAllNoteView"); + const bl = scene.fields(nv._barLineView.gameObject, "LiveAllBarLineView"); + const ct = scene.fields(bl._container.gameObject, "LiveBarLineViewContainer"); const top = scene.transform("LiveGameView/LiveGameCamera/screen_root"); let m = mat4.identity(); for (let t = scene.transform(ct._elementRoot.transform); t !== top; t = t.parent) { @@ -483,7 +495,7 @@ export class LiveBarLines { _rent() { const v = this.pool.pop(); if (v) return v; - const p = new NotePrefab(this.prefab), node = p.nodes.get(this.rendererPath), geo = this.owner.geo; + const p = new NotePrefab(this.prefab, null, this.owner.objects), node = p.nodes.get(this.rendererPath), geo = this.owner.geo; node.sprite = this.sprite; node.size = { x: F(F(geo.laneCount) * geo.unit), y: node.size.y }; // LiveBarLineView.Initialize return { p, node, progress: 0, dirty: false }; @@ -523,18 +535,20 @@ export class LiveBarLines { // ------------------------------------------------------------------ LiveAllNoteView (container of all views) export class LiveNotes { - // ln = livenotes/notes.json, score = score/.notes.json, opts.base = packed livenotes directory, - // opts.settings = the live settings (settings.js; absent: the data's option defaults), opts.displayOffsetMs = the - // executor's display offset, opts.skin = the note skin record (NoteDesignId; default ln.noteSkin) - constructor(gl, ln, score, opts = {}) { - this.gl = gl; this.ln = ln; this.score = score; this.base = opts.base || "livenotes"; - this.skin = opts.skin || ln.noteSkin; + // objects = the live's LiveObjects, ln = notes.json, score = score/.notes.json, opts.skin = the note skin's + // noteskin.json (NoteDesignId), opts.settings = the live settings (settings.js; absent: the data's option defaults), + // opts.displayOffsetMs = the executor's display offset, opts.scene = the live scene's Prefab and opts.barLineView = + // the bar line view prefab (scene.json prefabs.barLineView), for the bar lines + constructor(gl, objects, ln, score, opts = {}) { + this.gl = gl; this.objects = objects; this.ln = ln; this.score = score; + // LiveNoteSkinAsset with its units, sprites and materials resolved + this.skin = objects.resolve(objects.mono(opts.skin.skin)); this.notes = new Map(score.notes.map((n) => [n.id, n])); this.lines = new Map(score.lines.map((l) => [l.lineId, l])); - this._clips = new Map(); - this.gl_ = gl ? new NoteGL(gl, this.base) : null; - // the bar line views' _elementRoot relative to screen_root (opts.scene: the live scene's Prefab) - this.barLineRoot = opts.scene && ln.prefabs.bar_line_view ? LiveBarLines.elementRoot(opts.scene) : null; + this.gl_ = gl ? new NoteGL(gl, objects) : null; + // the bar line views' _elementRoot relative to screen_root + this.barLineView = opts.barLineView || null; + this.barLineRoot = opts.scene && this.barLineView ? LiveBarLines.elementRoot(opts.scene) : null; this.configure(opts.settings, opts.displayOffsetMs); } @@ -556,9 +570,9 @@ export class LiveNotes { this.displayOffsetMs = displayOffsetMs ?? s.noteDisplayTimeMs; // LiveAllBarLineView.Initialize runs always; with the option off (SetEnabled false) UpdateFrame does nothing if (this.showBarLines) { - if (!ln.prefabs.bar_line_view) throw new Error("MeasureLineDisplay: the bar line prefab (prefabs.bar_line_view) is not in the chart data"); + if (!this.barLineView) throw new Error("MeasureLineDisplay: the bar line prefab (scene.json prefabs.barLineView) is needed"); if (!this.barLineRoot) throw new Error("MeasureLineDisplay: the bar line container of the live scene is needed"); - this.barLines = new LiveBarLines(this, ln.prefabs.bar_line_view, this.barLineRoot); + this.barLines = new LiveBarLines(this, this.barLineView, this.barLineRoot); } else this.barLines = null; // line kinds: skin + container values const sk = this.skin; @@ -613,26 +627,32 @@ export class LiveNotes { // the bar line sprite's texture (MeasureLineDisplay turned on after load) async loadBarLineTexture() { - const pf = this.ln.prefabs.bar_line_view; - if (!this.gl_ || !pf) return; - for (const n of pf.nodes) for (const c of n.components) { - if (c.type === "SpriteRenderer" && c.m_Sprite && c.m_Sprite.texture) await this.gl_.texture(c.m_Sprite.texture); + if (!this.gl_ || !this.barLineView) return; + for (const n of this.objects.assets.prefab(this.barLineView).nodes) for (const c of n.components) { + if (c.type === "SpriteRenderer" && c.fields.m_Sprite) await this.gl_.texture(this.objects.sprite(c.fields.m_Sprite).texture); } } - clip(id) { - if (!this._clips.has(id)) this._clips.set(id, AnimClip.fromMecanim(this.ln.clips[id], id)); - return this._clips.get(id); + // a flick view's arrow loop clip (the skin's OverrideLoopAnimation) as the Animator its LoopAnimationTimeSync + // (_loopAnimationTimeSync._animator) plays it + arrowClip(view, v) { + const pf = this.objects.assets.prefab(view.prefabId), ref = view.mb._loopAnimationTimeSync; + const node = ref && pf.nodes.find((n) => n.path === ref.gameObject); + const sync = node && node.components.find((c) => componentClass(c) === "LoopAnimationTimeSync"); + if (!sync) throw new Error(`${pf.nodes[0].path}: no LoopAnimationTimeSync`); + return this.objects.clip(this.objects.id(v, "AnimationClip"), view.prefabId, sync.fields._animator.gameObject); } async load() { if (!this.gl_) return; - const g = this.gl_, want = new Map(); - const add = (sp) => { if (sp && sp.texture) want.set(sp.texture.texture, sp.texture); }; - const walk = (x) => { if (!x || typeof x !== "object") return; if (x.sprite && x.rect && x.texture) { add(x); return; } + const g = this.gl_, A = this.objects.assets, want = new Map(); + const add = (sp) => { want.set(sp.texture.id, sp.texture); }; + const isSprite = (x) => typeof x.id === "string" && A.hasObject(x.id) && A.object(x.id).class === "Sprite"; + const walk = (x) => { if (!x || typeof x !== "object") return; if (isSprite(x)) { add(x); return; } for (const v of Array.isArray(x) ? x : Object.values(x)) walk(v); }; walk(this.skin); - add(this.ln.prefabs.pair_note_line.nodes.flatMap((n) => n.components).find((c) => c.type === "SpriteRenderer").m_Sprite); + const pair = A.prefab(this.ln.prefabs.pair_note_line).nodes.flatMap((n) => n.components).find((c) => c.type === "SpriteRenderer"); + add(this.objects.sprite(pair.fields.m_Sprite)); if (this.barLines) add(this.barLines.sprite); for (const d of want.values()) await g.texture(d); this._lineTex = {}; @@ -696,13 +716,13 @@ export class LiveNotes { view.setup(note.id, cw.width, cw.center, note.timeMs, note.critical); view.setViewProgress(1); view.updateView(); - for (const n of view.renderers()) out.add(n.material || n.comp.m_Materials[0]); + for (const n of view.renderers()) out.add(n.material || n.materials[0]); if (this.showPairLines && note.pairNoteId && this.notes.has(note.pairNoteId)) pair = true; if (T.isLineBegin(note.op) && note.lineIds.length) body = true; } if (pair) { const pl = new PairLineView(this); - if (pl.r.enabled && pl.r.active) out.add(pl.r.comp.m_Materials[0]); + if (pl.r.enabled && pl.r.active) out.add(pl.r.materials[0]); } if (body) out.add(this.material); if (this.barLines && this.barLines.times.length) out.add(this.barLines.material); @@ -728,9 +748,9 @@ export class LiveNotes { const at = (M, c) => { const p = mat4.transformPoint(M, c); return { p, d: Math.hypot(p.x - cam.x, p.y - cam.y, p.z - cam.z) }; }; const sprite = (node, mpb = null) => { const comp = node.comp, M = mat4.mul(SR, node.t.localToWorld()); - const mesh = g.spriteMesh(node.sprite, comp.m_DrawMode, node.size.x, node.size.y), tex = g.tex.get(node.sprite.texture.texture); - if (!tex) throw new Error(`texture not loaded: ${node.sprite.texture.texture}`); - const mats = node.material ? [node.material] : comp.m_Materials; // sharedMaterial set by the view + const mesh = g.spriteMesh(node.sprite, comp.m_DrawMode, node.size.x, node.size.y), tex = g.tex.get(node.sprite.texture.id); + if (!tex) throw new Error(`texture not loaded: ${node.sprite.texture.name}`); + const mats = node.material ? [node.material] : node.materials; // sharedMaterial set by the view const { p, d } = at(M, mesh.center), sr = { m_Materials: mats, m_FlipX: node.flipX, m_FlipY: comp.m_FlipY }; const queue = node.material ? LiveLane.queue(g.lib, node.material) : 3000; renderer.submit("game", { sortingLayer: comp.m_SortingLayer, sortingOrder: node.order, queue, distance: d, @@ -745,10 +765,10 @@ export class LiveNotes { const BR = mat4.mul(SR, this.barLines.root); for (const n of this.barLines.renderers()) { const comp = n.comp, M = mat4.mul(BR, n.t.localToWorld()); - const mesh = g.spriteMesh(n.sprite, comp.m_DrawMode, n.size.x, n.size.y), tex = g.tex.get(n.sprite.texture.texture); - if (!tex) throw new Error(`texture not loaded: ${n.sprite.texture.texture}`); - const { p, d } = at(M, mesh.center), mat = comp.m_Materials[0]; - const sr = { m_Materials: comp.m_Materials, m_FlipX: n.flipX, m_FlipY: comp.m_FlipY }; + const mesh = g.spriteMesh(n.sprite, comp.m_DrawMode, n.size.x, n.size.y), tex = g.tex.get(n.sprite.texture.id); + if (!tex) throw new Error(`texture not loaded: ${n.sprite.texture.name}`); + const { p, d } = at(M, mesh.center), mat = n.materials[0]; + const sr = { m_Materials: n.materials, m_FlipX: n.flipX, m_FlipY: comp.m_FlipY }; renderer.submit("game", { sortingLayer: comp.m_SortingLayer, sortingOrder: n.order, queue: LiveLane.queue(g.lib, mat), distance: d, center: p, draw: (ctx) => g.drawSprite(ctx, sr, tex, mesh, M, n.color) }); } diff --git a/src/live/objects.js b/src/live/objects.js new file mode 100644 index 0000000..bb1b346 --- /dev/null +++ b/src/live/objects.js @@ -0,0 +1,134 @@ +import { refId } from "../data/assets.js"; +import { AnimBindings, AnimClip, AnimController } from "../engine/anim.js"; +import { ShaderLib } from "../engine/glsl.js"; +import { Prefab, scriptClass } from "../engine/prefab.js"; +import { GLTex } from "../engine/texture.js"; + +// LiveObjects: the objects of a live's store (docs/data-format.md "Objects") as the live modules read them, each made +// once per session. A value `v` is a reference as an object holds it ({"$ref": ...}) or an object id as a document +// holds it; null gives null. +// material(v) a Material: material.json with `id`, `shader` = the id of its Shader object (read when a draw +// needs it) and each texture slot's `texture` = the AssetStore.texture record of its Texture2D (or +// null) +// sprite(v), mesh(v), textureRecord(v) AssetStore.sprite / mesh / texture +// mono(v) a MonoBehaviour asset: mono.json with `id` and `class` (the script class) +// prefab(id) a root GameObject as a Prefab (Prefab.fromObjects; `ids` = [id]) +// clip(id, prefab, path) the AnimClip of clip `id` as the Animator at node `path` of the prefab or scene +// root `prefab` (a GameObject id) plays it: its bindings named by the bindings tables +// of the live's documents (AnimBindings); `rawName` = AnimationClip.name. The clips of +// one AnimationClip share its events (AnimationClip.AddEvent changes the asset). +// controller(v, prefab, path) the AnimController of an AnimatorController with its clips (clip()); `clipList`: +// the clips in the controller's order (runtimeAnimatorController.animationClips) +// resolve(v) a copy of a value with each reference to a Sprite, a Material or a MonoBehaviour replaced by +// sprite() / material() / the resolved mono() (other references kept) +// shaderInfo(id) the parsed Shader object +// With a GL context: `lib`, the ShaderLib over every Shader object of the store (by object id; `shaders` {Unity name: +// id} answers Shader.Find), and texture(id) -> Promise, the GL texture of a Texture2D object (uploaded once). +export class LiveObjects { + constructor(assets, { bindings = [], shaders = {}, gl = null } = {}) { + this.assets = assets; + this.bindings = new AnimBindings(bindings); + this.gl = gl; + this.lib = gl ? ShaderLib.fromObjects(gl, shaders, assets) : null; + this.shaderNames = shaders; + this._materials = new Map(); + this._monos = new Map(); + this._prefabs = new Map(); + this._clips = new Map(); + this._events = new Map(); // clip id -> the events every AnimClip of it shares + this._controllers = new Map(); + this._textures = new Map(); + } + + // the object id of a reference ({"$ref": ...}) or of an object id as a document gives it, checked to be in the store + // (and of class `cls`); null for null + id(v, cls = null) { + if (typeof v !== "string") return this.assets.ref(v, cls); + this.assets.object(v, cls); + return v; + } + + material(v) { + const id = this.id(v, "Material"); + if (id === null) return null; + let m = this._materials.get(id); + if (!m) { + const j = this.assets.material(id); + const textures = Object.fromEntries(Object.entries(j.textures || {}).map(([k, t]) => + [k, { ...t, texture: t.texture ? this.textureRecord(t.texture) : null }])); + m = { ...j, id, shader: refId(j.shader), textures }; + this._materials.set(id, m); + } + return m; + } + + sprite(v) { const id = this.id(v, "Sprite"); return id === null ? null : this.assets.sprite(id); } + mesh(v) { const id = this.id(v, "Mesh"); return id === null ? null : this.assets.mesh(id); } + textureRecord(v) { const id = this.id(v, "Texture2D"); return id === null ? null : this.assets.texture(id); } + + mono(v) { + const id = this.id(v, "MonoBehaviour"); + if (id === null) return null; + let m = this._monos.get(id); + if (!m) { + const j = this.assets.mono(id); + m = { ...j, id, class: scriptClass(j.$script) }; + this._monos.set(id, m); + } + return m; + } + + resolve(v) { + if (Array.isArray(v)) return v.map((x) => this.resolve(x)); + if (!v || typeof v !== "object") return v; + if (v.$ref !== undefined) { + const id = refId(v), cls = this.assets.hasObject(id) ? this.assets.object(id).class : null; + return cls === "Sprite" ? this.sprite(id) : cls === "Material" ? this.material(id) + : cls === "MonoBehaviour" ? this.resolve(this.mono(id)) : v; + } + return Object.fromEntries(Object.entries(v).map(([k, x]) => [k, this.resolve(x)])); + } + + shaderInfo(id) { this.assets.object(id, "Shader"); return this.assets.objectJson(id); } + + prefab(id) { + let p = this._prefabs.get(id); + if (!p) { p = Prefab.fromObjects(this.assets, [id], id); this._prefabs.set(id, p); } + return p; + } + + clip(id, prefab, path) { + const k = `${id}\u0000${prefab}\u0000${path}`; + let c = this._clips.get(k); + if (!c) { + const raw = this.assets.clip(id); + c = AnimClip.fromObject(raw, this.bindings.names(prefab, path, id), raw.name); + c.rawName = raw.name; + if (this._events.has(id)) c.events = this._events.get(id); else this._events.set(id, c.events); + this._clips.set(k, c); + } + return c; + } + + controller(v, prefab, path) { + const id = this.id(v, "AnimatorController"); + if (id === null) return null; + const k = `${id}\u0000${prefab}\u0000${path}`; + let ctrl = this._controllers.get(k); + if (!ctrl) { + const raw = this.assets.controller(id); + const clip = (e) => this.clip(this.id(e, "AnimationClip"), prefab, path); + ctrl = AnimController.fromMecanim(raw, clip); + ctrl.clipList = raw.clips.map(clip); + this._controllers.set(k, ctrl); + } + return ctrl; + } + + texture(id) { + if (!this.gl) throw new Error("LiveObjects: no GL context for textures"); + let t = this._textures.get(id); + if (!t) { t = GLTex.loadObject(this.gl, id, this.assets); this._textures.set(id, t); } + return t; + } +} diff --git a/src/live/particles.js b/src/live/particles.js index 1a753d8..d8ad653 100644 --- a/src/live/particles.js +++ b/src/live/particles.js @@ -1,12 +1,13 @@ import { FxError, FxMatrixTransform, FxParticleSystem } from "../engine/particles.js"; +import { componentClass } from "../engine/prefab.js"; import { UnityRandom } from "../engine/random.js"; // ------------------------------------------------------------------------------------------------ intro lane-in stars // The 7 `ef_particle_star` systems of the start timeline's LiveGameLaneInEffect // (live_start_playable_timeline/root/LiveGameLaneInEffect/lines/lane_lineNN/star_icon/ef_particle_star, -// livescene/scene.json). The stage hands them over through -// stage.intro.stars() -> [{path, active (activeInHierarchy), localToWorld, components}] every frame. -// new LiveIntroStars(gl, { materials /* FxMaterials(gl, "livescene") */, rng /* UnityRandom */ }) +// the band's timeline prefab). The stage hands them over through +// stage.intro.stars() -> [{path, active (activeInHierarchy), localToWorld, components (prefab.json)}] every frame. +// new LiveIntroStars(gl, { materials /* FxMaterials of the live's objects */, rng /* UnityRandom */ }) // prepare(stars) optional, before materials.load(): builds the systems so their material is loaded up front // update(stars, dt) animation phase, after the timeline evaluation of the frame (Unity: PlayableDirector in // DirectorUpdateAnimation, then ParticleSystemBeginUpdateAll): matrix, activation @@ -26,13 +27,13 @@ export class LiveIntroStars { _entry(s) { let e = this.entries.get(s.path); if (e) return e; - const comp = (type) => s.components.find((c) => c.type === type) || null; + const comp = (cls) => { const c = s.components.find((x) => componentClass(x) === cls); return c ? c.fields : null; }; const psc = comp("ParticleSystem"); if (!psc) throw new FxError(`${s.path}: no ParticleSystem`); const t = new FxMatrixTransform(s.localToWorld); const ps = new FxParticleSystem(psc, comp("ParticleSystemRenderer"), t, { rng: this.rng, materials: this.materials, name: s.path }); - const order = s.components.find((c) => c.type === "MonoBehaviour" && c.class === "LiveParticleOrderInLayerSetter"); + const order = comp("LiveParticleOrderInLayerSetter"); if (order) ps.sortingOrder = order._orderInLayer * 100 + order._offset; if (ps.renderer && this.materials) for (const m of ps.renderer.materials) this.materials.get(m); e = { ps, t }; diff --git a/src/live/renderer.js b/src/live/renderer.js index def23f0..989c99b 100644 --- a/src/live/renderer.js +++ b/src/live/renderer.js @@ -197,25 +197,30 @@ LiveCameraMath.sortItems = (items, opaqueDrawn = true) => { }; // ------------------------------------------------------------------------------------------------ renderer -// `scene` = parsed livescene/scene.json, `lib` = ShaderLib over "livescene/shaders", `loop` = PlayerLoop. +// `objects` = LiveObjects of the live (its `lib` draws), `scene` = scene.json of the live.scene part (the scene roots, +// settings, graphics objects), `loop` = PlayerLoop. The scene's root prefabs make one hierarchy (`prefab`). // Screen size: resize(W, H) (Screen.width / height; LiveGameView.FullInitialize runs again on a size change). // `settings`: the live settings (settings.js) read by the lane and the render canvas; absent: the data's defaults. +// `band`: the band's entry of scene.json bands (its start timeline and LightWeight background); `jacket`: the music's +// jacket sprite (live.json jacket). `sprites` {stage, jacket}: the sprites of the LightWeight background composite. export class LiveRenderer { - constructor(gl, lib, scene, loop, { quality = LIVE_QUALITY.Middle, base = "livescene", settings } = {}) { + constructor(gl, objects, scene, loop, { quality = LIVE_QUALITY.Middle, settings, band, jacket } = {}) { if (!gl.getExtension("EXT_color_buffer_float")) throw new Error("EXT_color_buffer_float is required"); - this.gl = gl; this.lib = lib; this.scene = scene; this.loop = loop; this.base = base; this.settings = settings; + if (!band || !jacket) throw new Error("LiveRenderer: the band and the jacket are required"); + this.gl = gl; this.objects = objects; this.lib = objects.lib; this.scene = scene; this.loop = loop; + this.settings = settings; this.band = band; this.sprites = { stage: band.background, jacket }; gl.frontFace(gl.CW); // Unity's front-face convention with its world-to-camera matrix this.vao = gl.createVertexArray(); - this.post = new URPPost(gl, lib, { vao: this.vao }); - this.prefab = new Prefab(scene.scene); + this.post = new URPPost(gl, this.lib, { vao: this.vao }); + this.prefab = Prefab.fromObjects(objects.assets, scene.scene, "scene"); // MasterLiveQualitySettings row of the LiveQuality - this.qualityRow = scene.master.liveQualitySettings.find((r) => r._quality === quality); + this.qualityRow = scene.settings.qualitySettings.find((r) => r._quality === quality); if (!this.qualityRow) throw new Error(`no MasterLiveQualitySettings for quality ${quality}`); this.quality = quality; const camera = (path, name, extra) => { const t = this.prefab.transform(path); - return new LiveCamera(name, t, this.prefab.component(path, "Camera"), - { uacd: this.prefab.component(path, "UniversalAdditionalCameraData"), ...extra }); + return new LiveCamera(name, t, this.prefab.fields(path, "Camera"), + { uacd: this.prefab.fields(path, "UniversalAdditionalCameraData"), ...extra }); }; this.cameras = { effect: camera("LiveGameView/LiveGameCamera/LiveEffectCamera", "effect", { opaqueDrawn: false }), @@ -232,15 +237,32 @@ export class LiveRenderer { // LiveEffectCamera's Volume (LiveGameVolume, global, priority 0) after // LiveViewPresenter.ApplyQualityEffectBloomSettings -> QualityConstants.ApplyQualityBloomMaxIterations: // quality != High -> maxIterations.Override(max(4, value - 1)) on the shared profile (applied once here). + // The profile's components are VolumeComponent assets (URPPost.evaluateStack reads them by class, `asset`). _effectStack() { - const vol = this.prefab.component("LiveGameView/LiveGameCamera/LiveEffectCamera", "Volume"); - const profile = JSON.parse(JSON.stringify(vol.sharedProfile)); + const vol = this.prefab.fields("LiveGameView/LiveGameCamera/LiveEffectCamera", "Volume"); + const shared = this.objects.mono(vol.sharedProfile); + const profile = { components: shared.components.map((v) => { + const c = JSON.parse(JSON.stringify(this.objects.mono(v))); + return { ...c, asset: c.class }; + }) }; if (this.quality !== LIVE_QUALITY.High) for (const c of profile.components) if (c.asset === "Bloom") c.maxIterations = { m_OverrideState: 1, m_Value: Math.max(4, c.maxIterations.m_Value - 1) }; return URPPost.evaluateStack([{ profile, weight: vol.weight }]); } + // The film grain textures of the effect camera's renderer: GraphicsSettings.m_CustomRenderPipeline (the URP asset) + // -> m_RendererDataList[UniversalAdditionalCameraData.m_RendererIndex, -1: m_DefaultRendererIndex] -> its + // postProcessData (SiriusPostProcessData) -> textures.filmGrainTex (Texture2D ids). + grainTextures() { + const O = this.objects, graphics = O.assets.objectJson(this.scene.graphics.graphics); + const pipeline = O.mono(graphics.m_CustomRenderPipeline); + const index = this.cameras.effect.uacd.m_RendererIndex; + const renderer = O.mono(pipeline.m_RendererDataList[index >= 0 ? index : pipeline.m_DefaultRendererIndex]); + const post = O.mono(renderer.postProcessData); + return post.textures.filmGrainTex.map((v) => O.id(v, "Texture2D")); + } + async load() { const gl = this.gl; this.tex = { @@ -249,8 +271,8 @@ export class LiveRenderer { gray: GLTex.solid(gl, [128, 128, 128, 255], "gray"), clear: GLTex.solid(gl, [0, 0, 0, 0], "clear"), }; - // SiriusPostProcessData of ForwardRendererLiveGameEffect (film grain textures; FilmGrain is off in LiveGameVolume) - this.grain = await URPPost.loadGrain(gl, this.base, this.scene.postTextures.ForwardRendererLiveGameEffect.filmGrainTex); + // SiriusPostProcessData of the effect camera's renderer (film grain textures; FilmGrain is off in LiveGameVolume) + this.grain = await URPPost.loadGrain(gl, this.grainTextures(), (id) => GLTex.loadObject(gl, id, this.objects.assets)); this.post.initLut(); this.canvas = new LiveRenderCanvas(this); await this.canvas.load(); @@ -378,9 +400,10 @@ export const liveFitSheets = (prog, sheets) => { }; // material property sheet [floats with every scalar shader default filled in, colours as arrays] for render state -// lookups (passState reads state properties such as _ColorMask / _Stencil* from the material floats) +// lookups (passState reads state properties such as _ColorMask / _Stencil* from the material floats); `mat`: a +// LiveObjects material (or a material of the same form whose `shader` is a key of `lib`) export const liveMaterialSheets = (lib, mat, tex) => { - const shader = mat.shader.shader, defaults = lib.defaults(shader, tex), floats = {}, colors = {}; + const shader = mat.shader, defaults = lib.defaults(shader, tex), floats = {}, colors = {}; for (const [k, v] of Object.entries(defaults)) if (typeof v === "number") floats[k] = v; Object.assign(floats, mat.floats || {}); for (const [k, v] of Object.entries(mat.colors || {})) colors[k] = [v.r, v.g, v.b, v.a]; diff --git a/src/live/session.js b/src/live/session.js index 98223ea..0d60580 100644 --- a/src/live/session.js +++ b/src/live/session.js @@ -2,13 +2,13 @@ import { bindAssets, unbindAssets } from "../data/assets.js"; import { F } from "../engine/core.js"; import { SOUND_CATEGORY } from "../engine/audio.js"; import { trackGL } from "../engine/gltrack.js"; -import { ShaderLib } from "../engine/glsl.js"; import { PlayerLoop } from "../engine/loop.js"; import { LiveLightWeightBackground } from "./background.js"; import { LiveData } from "./data.js"; import { LiveFx } from "./fx.js"; import { isLegacyLive, legacyLive } from "./legacy.js"; import { LiveNotes } from "./noteview.js"; +import { LiveObjects } from "./objects.js"; import { LiveRenderer } from "./renderer.js"; import { LIVE_OPTION_BY_NAME, LiveOptionContext, LiveSettingsError, liveCategoryVolumes, liveDerived, liveNoteEffectName, liveNoteSeMaps, liveOptionItems, liveSettingsChanges } from "./settings.js"; @@ -114,9 +114,10 @@ export class ChartSession { this.gl = gl; this.assets = assets; this._gl = trackGL(gl); const data = this.data = new LiveData(assets); - const live = data.live, scene = data.sceneDoc, noteAssets = data.noteAssets, audioData = data.audio(); - // Live options: defaults, ranges and the values the chart's files offer, then the settings of this session - const ctx = new LiveOptionContext({ live, scene, notes: noteAssets, audio: audioData, info: assets.info || null }); + const live = data.live, scene = data.scene, notes = data.notes, audioData = data.audio(); + // Live options: defaults, ranges and the values the live's files offer, then the settings of this session + const ctx = new LiveOptionContext({ live, scene, notes, audio: audioData, info: assets.info || null, + skins: data.noteskins, effects: data.noteeffects }); if (quality !== undefined && quality !== null) ctx.defaults.LiveQuality = Number(quality); const s = ctx.resolve(settings || {}); this.optionContext = ctx; @@ -125,15 +126,25 @@ export class ChartSession { // MirrorChart (4): the score converted with isMirror (SsMusicScoreConverter), a file of its own const score = data.score(s.MirrorChart); this.live = live; - this.score = score; this.noteAssets = noteAssets; + this.score = score; this.noteSettings = notes.settings; + // NoteDesignId (306): the note skin asset of MasterLiveNoteSkin; NoteEffectId (307) with LiveQuality: the note + // effect set (SoloLiveResourceLoadStateNode.CreateLoadParameter); the defaults are the live's parts + const skin = data.noteskin(s.NoteDesignId !== ctx.defaults.NoteDesignId ? notes.settings.skins[String(s.NoteDesignId)] : undefined); + const effect = data.noteeffect(liveNoteEffectName(ctx, s) ?? undefined); + // the live's objects: clip bindings of the scene, the notes and the selected skin and effect set; Shader.Find + // names of the scene + const objects = this.objects = new LiveObjects(assets, { gl, shaders: scene.shaders, + bindings: [scene.bindings, notes.bindings, skin.bindings, effect.bindings] }); const loop = new PlayerLoop(60); this.loop = loop; // FullInitialize: renderer (cameras, RTs, LightWeight background, post), stage (lane + start timeline), // simulation, note views, effects. The effects are built after the stage so that their animation hook runs // after the timeline evaluation (DirectorUpdateAnimation precedes ParticleSystemBeginUpdateAll). - const renderer = new LiveRenderer(gl, new ShaderLib(gl, "livescene/shaders", assets), scene, loop, - { quality: s.LiveQuality, settings: s }); + const band = scene.bands[String(live.band.band)]; + if (!band) throw new Error(`band ${live.band.band}: not in scene.json bands`); + const renderer = new LiveRenderer(gl, objects, scene, loop, + { quality: s.LiveQuality, settings: s, band, jacket: live.jacket }); await renderer.load(); this.renderer = renderer; const c = gl.canvas; @@ -143,15 +154,12 @@ export class ChartSession { await stage.load(); stage.attach(loop); this.stage = stage; - this.exec = new LiveExecutor(score, noteAssets.settings, this.derived); - // NoteDesignId (306): the note skin asset of MasterLiveNoteSkin (SoloLiveResourceLoadStateNode.CreateLoadParameter) - const skin = s.NoteDesignId !== ctx.defaults.NoteDesignId - ? noteAssets.noteSkins[noteAssets.settings.skins[String(s.NoteDesignId)]] : undefined; - this.notes = new LiveNotes(gl, noteAssets, score, { base: "livenotes", settings: s, displayOffsetMs: this.exec.D, skin, - scene: renderer.prefab }); + this.exec = new LiveExecutor(score, notes.settings, this.derived); + this.notes = new LiveNotes(gl, objects, notes, score, { settings: s, displayOffsetMs: this.exec.D, skin, + scene: renderer.prefab, barLineView: scene.prefabs.barLineView }); await this.notes.load(); - this.fx = new LiveFx({ gl, loop, notes: noteAssets, score, scene, introStars: () => stage.intro.stars(), - seed: seed ?? (Date.now() >>> 0), settings: s, effect: liveNoteEffectName(ctx, s) }); + this.fx = new LiveFx({ gl, loop, objects, scene, notes, effect, score, introStars: () => stage.intro.stars(), + seed: seed ?? (Date.now() >>> 0), settings: s }); await this.fx.load(); // live sound: SoundManager with the live's CRI routing, LiveSoundPlayer (cheers, note SE, BGM); every cue of @@ -477,7 +485,7 @@ export class ChartSession { // the chart state of a live booted with the current settings, at the current chart time (setSettings) async _restart() { if (!this.started) { // at the chart start: nothing simulated yet - this.exec = new LiveExecutor(this.score, this.noteAssets.settings, this.derived); + this.exec = new LiveExecutor(this.score, this.noteSettings, this.derived); this.notes.configure(this._settings, this.exec.D); this.fx.resetUI(this._settings); for (const d of this.preDts) this.fx.uiFrame(null, d); @@ -546,7 +554,7 @@ export class ChartSession { // DOTween / Animator steps of every frame up to the music start, clock gate reset. reconfigure: the note views and // the live UI take the current settings (setSettings) _resetChart(len, reconfigure = false) { - this.exec = new LiveExecutor(this.score, this.noteAssets.settings, this.derived); + this.exec = new LiveExecutor(this.score, this.noteSettings, this.derived); if (reconfigure) this.notes.configure(this._settings, this.exec.D); else this.notes.reset(); this.fx.resetUI(reconfigure ? this._settings : undefined); for (const d of this.preDts) this.fx.uiFrame(null, d); @@ -595,8 +603,8 @@ export class ChartSession { if (this._gl) this._gl.release(); unbindAssets(this.gl, this.assets); } - for (const k of ["renderer", "stage", "notes", "fx", "exec", "audio", "frame", "clock", "live", "score", "noteAssets", - "loop", "preDts", "_gl"]) this[k] = null; + for (const k of ["renderer", "stage", "notes", "fx", "exec", "audio", "frame", "clock", "live", "score", "noteSettings", + "objects", "loop", "preDts", "_gl"]) this[k] = null; } } diff --git a/src/live/settings.js b/src/live/settings.js index c778206..5ae2707 100644 --- a/src/live/settings.js +++ b/src/live/settings.js @@ -17,9 +17,9 @@ import { F } from "../engine/core.js"; // reload selects other files (score, skins, effect sets, quality): the session is created again // none read by the game, with no visible effect in an auto-played live (accepted, not applied) // -// Defaults: the chart data's preset-1 values (livenotes settings.optionDefaults, livescene master -// .optionDefaultsPreset1). Items the data does not carry fall back to the MasterOptionDefault / MasterOptionRange rows, -// written below as `def` / `range` (the master's stored units). +// Defaults: the live data's preset-1 values (notes.json and scene.json settings.optionDefaults). Items the data does +// not carry fall back to the MasterOptionDefault / MasterOptionRange rows, written below as `def` / `range` (the +// master's stored units). // // Live volumes (410-417): the game plays the live through the CRI categories Live*Config, whose volumes it keeps in // the device's local sound config, not in the options. A fresh install writes AppConfigDefaultData there @@ -184,17 +184,19 @@ const parseAs = (o, s) => { return Number.parseInt(s, 10); }; -// Resolved chart context: defaults, ranges and the values the chart's files offer. -// live live.json; scene livescene/scene.json; notes livenotes/notes.json; audio audio/live-audio.json -// info the chart manifest (quality, options) +// Resolved chart context: defaults, ranges and the values the live's files offer. +// live live.json; scene scene.json; notes notes.json; audio the live's sounds (LiveData.audio()) +// skins, effects the names of the loaded note skins / note effect sets (LiveData.noteskins / noteeffects) +// info the store's info (quality, options) export class LiveOptionContext { - constructor({ live = null, scene = null, notes = null, audio = null, info = null } = {}) { + constructor({ live = null, scene = null, notes = null, audio = null, info = null, skins = [], effects = [] } = {}) { this.live = live; this.scene = scene; this.notes = notes; this.audio = audio; this.info = info; + this.skins = skins; this.effects = effects; const ns = (notes && notes.settings) || {}; - const sm = (scene && scene.master) || {}; + const ss = (scene && scene.settings) || {}; this._def = ns.optionDefaults || {}; - this._def1 = sm.optionDefaultsPreset1 || {}; - this._range = { ...(sm.optionRanges || {}), ...(ns.optionRanges || {}) }; + this._def1 = ss.optionDefaults || {}; + this._range = { ...(ss.optionRanges || {}), ...(ns.optionRanges || {}) }; this.defaults = {}; for (const o of LIVE_OPTIONS) this.defaults[o.name] = this._default(o); } @@ -206,7 +208,7 @@ export class LiveOptionContext { } if (o.name === "NoteSePatternId" && this.audio && this.audio.noteSe && Number.isInteger(this.audio.noteSe.patternId)) return this.audio.noteSe.patternId; - const s = this._def[o.name] ?? (this._def1[o.name] && this._def1[o.name].value) ?? o.def; + const s = this._def[o.name] ?? this._def1[o.name] ?? o.def; return parseAs(o, s); } @@ -227,20 +229,20 @@ export class LiveOptionContext { const groupIds = groups ? Object.keys(groups).map(Number).sort((a, b) => a - b) : [this.defaults.NoteSePatternId]; switch (o.name) { case "MirrorChart": return this.live && this.live.notesMirror ? [false, true] : [false]; - // bar lines: the view prefab of the scene's LiveBarLineViewContainer (notes.json prefabs.bar_line_view) - case "MeasureLineDisplay": return this.notes && this.notes.prefabs && this.notes.prefabs.bar_line_view ? [false, true] : [false]; + // bar lines: the view prefab of the scene's LiveBarLineViewContainer (scene.json prefabs.barLineView) + case "MeasureLineDisplay": return this.scene && this.scene.prefabs && this.scene.prefabs.barLineView ? [false, true] : [false]; case "LiveQuality": { const q = this.info && this.info.options && this.info.options.LiveQuality; return Array.isArray(q) && q.length ? q.slice() : [this.defaults.LiveQuality]; } case "NoteDesignId": { - // settings.skins: design id -> skin asset name; the records of the other skins: notes.json noteSkins - const recs = (this.notes && this.notes.noteSkins) || null; - const skins = ns.skins && recs ? Object.keys(ns.skins).map(Number).filter((id) => recs[ns.skins[id]]) : []; + // settings.skins: design id -> skin asset name; the ids whose skin is loaded + const skins = ns.skins ? Object.keys(ns.skins).map(Number).filter((id) => this.skins.includes(ns.skins[id])) : []; return [...new Set([this.defaults.NoteDesignId, ...skins])].sort((a, b) => a - b); } case "NoteEffectId": { - const fx = ns.effects ? Object.keys(ns.effects).map(Number) : []; + // settings.effects: effect id -> effect set name; the ids whose set is loaded + const fx = ns.effects ? Object.keys(ns.effects).map(Number).filter((id) => this.effects.includes(ns.effects[id])) : []; return [...new Set([this.defaults.NoteEffectId, ...fx])].sort((a, b) => a - b); } case "NoteSePatternId": return groupIds; @@ -385,11 +387,11 @@ export const liveOptionItems = (ctx, s) => LIVE_OPTIONS.map((o) => ({ offered: ctx.offered(o), hidden: !!o.hidden || MUTE_ITEMS.has(o.name), mute: o.mute || null, })); -// The note effect set's asset name for NoteEffectId and LiveQuality, or null for the data's set (livenotes -// settings.effect). MasterLiveNoteEffectSkin._assetName; at LiveQuality Low NoteSkinLoadStep.CollectDownloadAddresses -// takes the set's Light variant (GetLightQualityAddress) when ResourceManager.ExistsAsset has it, else the set itself -// (here: when notes.json assets has the variant's LiveNoteEffectAssetSettings). The lane effects do not follow the -// set (LaneEffectLoadStep.GetAssetPath: always effect001). +// The note effect set's asset name for NoteEffectId and LiveQuality, or null for the live's set (its noteeffect part). +// MasterLiveNoteEffectSkin._assetName; at LiveQuality Low NoteSkinLoadStep.CollectDownloadAddresses takes the set's +// Light variant (GetLightQualityAddress) when ResourceManager.ExistsAsset has it, else the set itself (here: when the +// variant is among the loaded effect sets). The lane effects do not follow the set (LaneEffectLoadStep.GetAssetPath: +// always effect001). export const liveNoteEffectName = (ctx, s) => { if (s.NoteEffectId === ctx.defaults.NoteEffectId && s.LiveQuality === ctx.defaults.LiveQuality) return null; const ns = (ctx.notes && ctx.notes.settings) || {}; @@ -397,8 +399,7 @@ export const liveNoteEffectName = (ctx, s) => { (s.NoteEffectId === ctx.defaults.NoteEffectId ? ns.effect : undefined); if (!base) throw new LiveSettingsError(`NoteEffectId ${s.NoteEffectId}: this chart's data has no effect set for it`); if (s.LiveQuality !== 2) return base; - const assets = (ctx.notes && ctx.notes.assets) || {}; - return assets[`Effect/Live/NoteEffect/${base}Light/LiveNoteEffectAssetSettings`] ? `${base}Light` : base; + return ctx.effects.includes(`${base}Light`) ? `${base}Light` : base; }; // the changed names grouped by how they apply: {none, live, noteSe, boot, reload} (boot: re-simulation needed or a diff --git a/src/live/stage.js b/src/live/stage.js index 6c3a72f..8e85076 100644 --- a/src/live/stage.js +++ b/src/live/stage.js @@ -2,8 +2,7 @@ import { LiveIntro } from "./intro.js"; import { LiveLane } from "./lane.js"; // LiveStage: the 3D lane (LiveLane) and the start timeline (LiveIntro), updated in the loop phases and submitted -// to LiveRenderer each frame. The scene is `assets.json("livescene/scene.json")` (infinities are written as 1e999, -// which JSON.parse reads as Infinity). +// to LiveRenderer each frame. The scene is the live's scene.json (LiveData.scene). // // Use (the chart session): // const stage = new LiveStage(renderer, scene, loop); await stage.load(); stage.attach(loop); diff --git a/src/story/features/clips.js b/src/story/features/clips.js index ac44288..fbb0b28 100644 --- a/src/story/features/clips.js +++ b/src/story/features/clips.js @@ -47,7 +47,7 @@ export class AnimRecords { controllerOf(raw, where) { const own = ownClips(raw, (name) => new this.Err(`${where}: two clips named ${name}`)); const made = new Map(); - return AnimController.fromMecanim(raw, (key) => { + return AnimController.fromMecanim(raw, ({ clip: key }) => { if (!made.has(key)) made.set(key, AnimClip.fromMecanim(own.get(key) || this._get(this.clips, "clip", key, where), key)); return made.get(key); }); diff --git a/src/story/renderer.js b/src/story/renderer.js index a34b126..ada55e2 100644 --- a/src/story/renderer.js +++ b/src/story/renderer.js @@ -67,7 +67,7 @@ export class StoryRenderer { idx: this._buffer(new Uint16Array(m.submeshes[0]), gl.ELEMENT_ARRAY_BUFFER), count: m.submeshes[0].length, bounds: StoryRenderer.bounds(m.vertices) }; // post textures: film grain by FilmGrain.type, neutral colour curves, internal LUT - this.grain = await URPPost.loadGrain(gl, base, this.scene.postTextures.filmGrainTex); + this.grain = await URPPost.loadGrain(gl, this.scene.postTextures.filmGrainTex, (d) => GLTex.load(gl, base, d)); this.post.initLut(); for (const c of this.characters) { c.gl.white = this.tex.white; await c.gl.loadTextures(); c.gl.prepare(); } } diff --git a/src/story/simple/home/host.js b/src/story/simple/home/host.js index 941a712..da21a1a 100644 --- a/src/story/simple/home/host.js +++ b/src/story/simple/home/host.js @@ -383,7 +383,7 @@ export class SimpleHomeHost { this.kawase = new DualKawaseBlur(gl, this.lib); // the main camera's post chain: the renderer's film grain textures (only when the grain is drawn), its LUT this.urp = new URPPost(gl, this.postOpts.lib, { vao: this.vao }); - this.grain = post.grainDescs ? await URPPost.loadGrain(gl, "", post.grainDescs) : []; + this.grain = post.grainDescs ? await URPPost.loadGrain(gl, post.grainDescs, (d) => GLTex.load(gl, "", d)) : []; this.urp.initLut(); this.cameraColor = null; this.room.glb = null; // the file's bytes are no longer needed diff --git a/tests/unit/live-legacy-fixture.mjs b/tests/unit/live-legacy-fixture.mjs new file mode 100644 index 0000000..21cd597 --- /dev/null +++ b/tests/unit/live-legacy-fixture.mjs @@ -0,0 +1,129 @@ +// A synthetic chart of the site layout (docs/data-format.md "Chart manifest"): live.json, the score, live-audio.json +// with its waveforms, livescene/ and livenotes/ with their objects inline and their packed shaders. Every kind of +// inline object the live's reader converts appears once: prefabs with references between the scene's roots, materials +// (one of a shader the chart does not pack), sprites, textures with and without their image, a mesh, MonoBehaviour +// assets, a timeline whose track binds an Animator, clips and controllers of the note assets by key, the note skin +// with a flick arrow clip, the lane and note effect sets, the live UI sprite assets. +// chartFiles(extra) -> {text: {path: text}, bytes: {path: Uint8Array}}; `extra` is merged into live.json. + +const V0 = { x: 0, y: 0, z: 0 }, Q = { x: 0, y: 0, z: 0, w: 1 }, V1 = { x: 1, y: 1, z: 1 }; +const node = (path, components = []) => ({ path, name: path.split("/").pop(), active: true, layer: 0, tag: "Untagged", + localPosition: V0, localRotation: Q, localScale: V1, components }); +export const texture = (name) => ({ texture: `textures/${name}.png`, name, width: 4, height: 4, mipCount: 1, colorSpace: 1, + settings: { m_FilterMode: 1, m_WrapU: 1, m_WrapV: 1 } }); +export const sprite = (name) => ({ + sprite: name, texture: texture(name), rect: { x: 0, y: 0, width: 4, height: 4 }, pivot: { x: 0.5, y: 0.5 }, + border: { x: 0, y: 0, z: 0, w: 0 }, pixelsToUnits: 100, offset: { x: 0, y: 0 }, textureRect: { x: 0, y: 0, width: 4, height: 4 }, + textureRectOffset: { x: 0, y: 0 }, settingsRaw: 0, uvTransform: { x: 100, y: 2, z: 100, w: 2 }, +}); +const material = (name, shader = "Sprites/Default") => ({ + material: name, shader: { shader }, keywords: [], ints: {}, floats: { _Z: 1 }, colors: {}, + textures: { _MainTex: { texture: texture("white"), scale: { x: 1, y: 1 }, offset: { x: 0, y: 0 } } }, +}); +const clip = (name, path, attribute, extra = {}) => ({ + name, streamed: { curves: [] }, length: 1, loopTime: true, + bindings: [{ path, attribute, pathCrc: 11, attributeCrc: 22, typeID: 4, class: "Transform", ...extra }], +}); +const sr = (m, s) => ({ type: "SpriteRenderer", m_Enabled: 1, m_Materials: [m], m_Sprite: s, m_Size: V1, + m_Color: { r: 1, g: 1, b: 1, a: 1 }, m_SortingOrder: 0, m_SortingLayer: 0, m_DrawMode: 0 }); +const effectPrefab = (key, cls) => ({ key, nodes: [node(key, [{ type: "MonoBehaviour", class: cls, _duration: 1 }]), + node(`${key}/ps`, [{ type: "ParticleSystemRenderer", m_UseCustomVertexStreams: 1, m_VertexStreams: [0, 1, 3, 4] }])] }); + +export const SCENE = { + scene: { key: "scene", nodes: [ + node("Live", [{ type: "Animator", m_Controller: { controller: "UILiveCombo#5" } }]), + node("Live/judge", [sr(material("JudgeMat"), sprite("judge"))]), + node("Stage", [{ type: "MonoBehaviour", class: "StageView", + _judge: { gameObject: "Live/judge", component: "SpriteRenderer" }, _self: { gameObject: "Stage/mesh" } }]), + node("Stage/mesh", [{ type: "MeshRenderer", m_Materials: [material("Unpacked", "Hidden/NotPacked")] }, + { type: "MeshFilter", m_Mesh: { mesh: "quad", vertices: [0, 0, 0, 1, 0, 0, 0, 1, 0], uv0: [0, 0, 1, 0, 0, 1], + submeshes: [[0, 1, 2]], m_LocalAABB: {} } }]), + ] }, + assets: { + startTimeline: { key: "start_timeline", nodes: [node("StartTimeline", [ + { type: "PlayableDirector", m_PlayableAsset: { asset: "TimelineAsset", m_Name: "Start", m_Tracks: [{ + asset: "AnimationTrack", m_Name: "Band", boundAnimator: { key: "start_timeline", gameObject: "StartTimeline/band" }, + m_Clips: [{ m_Start: 0, m_Duration: 2, m_PostExtrapolationTime: Infinity, + m_Asset: { m_Clip: clip("band_in", "body", "m_LocalPosition.x") } }] }] } }]), + node("StartTimeline/band", [{ type: "Animator", m_Controller: null }])] }, + laneLinePrefab: { key: "lane_line", nodes: [node("lane_line", [sr(material("LaneMat"), sprite("lane_line"))])] }, + }, + player: { + pipelines: { URP: { m_Name: "URP", m_RendererDataList: ["Forward", null] } }, defaultPipeline: "URP", + renderers: { Forward: { m_Name: "Forward", postProcessData: "Post" } }, + postProcessData: { Post: { m_Name: "Post", textures: { filmGrainTex: [] } } }, + qualityLevels: [{ name: "Low" }], colorSpace: "Linear", + }, + postTextures: { Forward: { filmGrainTex: [texture("grain")] } }, + slice: { band: 3, laneSkin: "lane01" }, + laneSkin: { base: sprite("lane_base") }, + sprites: { lightweightBackground: sprite("bg"), jacket: sprite("jacket") }, + master: { liveLaneSkin: [{ _id: 1, _assetName: "lane01" }], optionDefaultsPreset1: { LaneOpacity: { value: "80" } }, + optionRanges: { LaneOpacity: [0, 100] }, liveQualitySettings: [] }, +}; + +export const NOTES = { + settings: { skin: "skin001", effect: "effect001", skins: { 1: "skin001" }, effects: { 1: "effect001" }, + judgementSprites: { perfect: "judge_perfect" }, optionDefaults: { NoteSpeed: "5.00" }, + optionRanges: { NoteSpeed: [100, 1200] }, liveSettings: {}, noteDisplayTimeMs: 1000 }, + prefabs: { + tap_note_view: { key: "tap_note_view", nodes: [node("tap_note_view", [{ type: "MonoBehaviour", class: "LiveNoteView" }])] }, + flick_note_view: { key: "flick_note_view", nodes: [ + node("flick_note_view", [{ type: "MonoBehaviour", class: "LiveFlickNoteView", + _loopAnimationTimeSync: { gameObject: "flick_note_view/arrow", component: "MonoBehaviour" } }]), + node("flick_note_view/arrow", [{ type: "MonoBehaviour", class: "LoopAnimationTimeSync", + _animator: { gameObject: "flick_note_view/arrow/anim_root", component: "Animator" }, + _loopAnim: { clip: "arrow_loop#1" } }]), + node("flick_note_view/arrow/anim_root", [{ type: "Animator", m_Controller: null }])] }, + judge_effect_view: { key: "judge_effect_view", nodes: [node("judge_effect_view", [sr(material("JudgeMat"), sprite("judge"))])] }, + }, + noteSkin: { asset: "LiveNoteSkinAsset", m_Name: "skin001", + TapNoteAsset: { asset: "NoteSkinAssetUnit", m_Name: "tap", _mainSprite: sprite("tap"), _centerMarkSprite: null }, + FlickNoteAsset: { asset: "NoteSkinAssetUnit", m_Name: "flick", _mainSprite: sprite("flick"), + _arrowLoopAnimation: { clip: "arrow_skin#2" } } }, + assets: { + "Effect/Live/LaneEffect/effect001/LiveLaneEffectAssetSettings": { asset: "LiveLaneEffectAssetSettings", m_Name: "lane", + _tap: { class: "LiveLaneEffectAsset", gameObject: "lane_tap" } }, + "Effect/Live/LaneEffect/effect001/lane_tap": effectPrefab("lane_tap", "LiveLaneEffectAsset"), + "Effect/Live/NoteEffect/effect001/LiveNoteEffectAssetSettings": { asset: "LiveNoteEffectAssetSettings", m_Name: "note", + _tap: { class: "LiveNoteEffectAsset", gameObject: "note_tap" } }, + "Effect/Live/NoteEffect/effect001/note_tap": effectPrefab("note_tap", "LiveNoteEffectAsset"), + "Live/UiSpriteAssets/LiveComboSpriteAsset": { asset: "LiveComboSpriteAsset", m_Name: "combo", _digits: [sprite("digit0")] }, + "Live/UiSpriteAssets/LiveJudgementSpriteAsset": { asset: "LiveJudgementSpriteAsset", m_Name: "judgement", + _perfect: sprite("judge_perfect") }, + }, + clips: { + "arrow_loop#1": clip("arrow_loop", "arrow", "m_LocalPosition.y"), + "arrow_skin#2": clip("arrow_skin", "arrow", null, { class: "SpriteRenderer", typeID: 212, pptr: true, attributeCrc: 0 }), + "combo#3": clip("combo", "Live/judge", "m_IsActive"), + }, + controllers: { "UILiveCombo#5": { name: "UILiveCombo", clips: [{ clip: "combo#3" }], layers: [] } }, +}; + +const SHADER = "Sprites/Default"; +const json = (v) => JSON.stringify(v, (k, x) => (x === Infinity ? "\u0000inf" : x)).replaceAll('"\\u0000inf"', "1e999"); +const layer = (file) => ({ file, sampleRate: 48000, channels: 2, samples: 4800, loopFlag: 0, loopStart: null, + loopEnd: null, volume: 1 }); +const cue = (sheet, file, categories) => ({ sheet, cue: sheet, category: 1, categories, volume: 1, + row: { _isRandomPitch: false }, layers: [layer(file)] }); + +export const chartFiles = (extra = {}) => ({ + text: { + "live.json": JSON.stringify({ musicId: 7, difficulty: "expert", scene: "livescene/scene.json", + notes: "score/m7_03.notes.json", notesMirror: "score/m7_03.mirror.notes.json", + noteAssets: "livenotes/notes.json", liveAudio: "audio/live-audio.json", ...extra }), + "score/m7_03.notes.json": JSON.stringify({ notes: [1] }), "score/m7_03.mirror.notes.json": JSON.stringify({ notes: [2] }), + "audio/live-audio.json": JSON.stringify({ + sounds: { 1: cue("bgm_7", "audio/bgm_7/bgm_7.m4a", ["LiveBgm"]), 10: cue("se1", "audio/se1/a.flac", ["LiveSe"]) }, + categories: { LiveBgm: 1, LiveSe: 1 }, react: [], music: { soundId: 1 }, noteSe: { types: { 1: 10 } }, liveSe: {}, + timeline: { startCheerDelaySec: 1 } }), + "livescene/scene.json": json(SCENE), "livenotes/notes.json": json(NOTES), + "livescene/shaders/shaders.json": JSON.stringify([{ name: SHADER, parsed: "Sprites_Default.json", variants: [ + { file: "Sprites_Default/gles3/v0.glsl", platform: "gles3", type: "GLES3", subShader: 0, pass: 0, keywords: [] }] }]), + "livescene/shaders/Sprites_Default.json": JSON.stringify({ name: SHADER, properties: [], subShaders: [] }), + "livescene/shaders/Sprites_Default/gles3/v0.glsl": "#version 300 es\n", + "livenotes/shaders/shaders.json": "[]", + }, + bytes: { "audio/bgm_7/bgm_7.m4a": new Uint8Array([7]), "audio/se1/a.flac": new Uint8Array([1]), + "livescene/textures/judge.png": new Uint8Array([0x89, 0x50, 0x4e, 0x47]) }, +}); diff --git a/tests/unit/live-settings.test.mjs b/tests/unit/live-settings.test.mjs index a6ab1f8..af625cc 100644 --- a/tests/unit/live-settings.test.mjs +++ b/tests/unit/live-settings.test.mjs @@ -8,7 +8,7 @@ import assert from "node:assert/strict"; import { test } from "node:test"; import { F } from "../../src/engine/core.js"; import { PlayerLoop } from "../../src/engine/loop.js"; -import { LIVE_LANE_EFFECT_SET, LiveLaneEffects } from "../../src/live/fx-effects.js"; +import { LiveLaneEffects } from "../../src/live/fx-effects.js"; import { LiveJudgementViewCenter } from "../../src/live/fx-ui.js"; import { NoteGeo } from "../../src/live/notegeo.js"; import { ArrowGradient, LiveBarLines, LiveNotes, NoteHeadView, musicScorePosition } from "../../src/live/noteview.js"; @@ -34,9 +34,9 @@ const notesJson = (extra = {}) => ({ optionRanges: { NoteSpeed: [100, 1200], SlideOpacity: [10, 100], GuideOpacity: [10, 100] }, liveSettings: LIVE_SETTINGS, noteDisplayTimeMs: D5, effect: "effect001", ...extra }, }); -const sceneJson = () => ({ master: { optionDefaultsPreset1: { - JudgePosition: { value: "0" }, BackgroundBrightness: { value: "70" }, LaneOpacity: { value: "80" }, - GuidelineOpacity: { value: "25" }, GuidelineCount: { value: "2" }, JudgePositionDisplay: { value: "FALSE" } } } }); +const sceneJson = (extra = {}) => ({ settings: { optionDefaults: { + JudgePosition: "0", BackgroundBrightness: "70", LaneOpacity: "80", GuidelineOpacity: "25", GuidelineCount: "2", + JudgePositionDisplay: "FALSE" } }, ...extra }); const audioJson = (extra = {}) => ({ // the fresh-profile values of the game's local sound config (AppConfigDefaultData: 1 each) categories: { LiveBgm: 1, LiveBgmConfig: 1, LiveSe: 1, LiveSeConfig: 1, LiveVoiceConfig: 1, LiveNotesSeConfig: 1 }, @@ -44,8 +44,9 @@ const audioJson = (extra = {}) => ({ volumes: Object.fromEntries(Array.from({ length: 14 }, (_, i) => [i + 1, 1])), mutes: Object.fromEntries(Array.from({ length: 14 }, (_, i) => [i + 1, false])), ...extra }, }); -const context = (o = {}) => new LiveOptionContext({ live: o.live || {}, scene: sceneJson(), notes: o.notes || notesJson(), - audio: o.audio || audioJson(), info: o.info || { quality: 1 } }); +const context = (o = {}) => new LiveOptionContext({ live: o.live || {}, scene: o.scene || sceneJson(), notes: o.notes || notesJson(), + audio: o.audio || audioJson(), info: o.info || { quality: 1 }, + skins: o.skins || [], effects: o.effects || [] }); test("option math: note timing and chart position in ms (float32 product, banker's rounding, clamp)", () => { assert.equal(M.roundEven(2.5), 2); assert.equal(M.roundEven(3.5), 4); assert.equal(M.roundEven(-2.5), -2); @@ -63,7 +64,7 @@ test("option math: note timing and chart position in ms (float32 product, banker assert.equal(M.unit01(80), F(0.8)); assert.equal(M.unit01(150), 1); }); -test("defaults come from the chart data; the derived values at the defaults are the player's former constants", () => { +test("defaults come from the chart data; the derived values at the defaults are the fixed constants", () => { const ctx = context(), s = ctx.resolve({}); assert.equal(s.NoteSpeed, 5); assert.equal(s.NoteTiming, 0); assert.equal(s.ChartPosition, 0); assert.equal(s.LaneOpacity, 80); assert.equal(s.GuidelineCount, 2); assert.equal(s.BackgroundBrightness, 70); @@ -72,6 +73,7 @@ test("defaults come from the chart data; the derived values at the defaults are // values while all of them are at their defaults assert.equal(s.LiveMusicVolume, 100); assert.equal(s.LiveSeVolume, 50); assert.equal(s.LiveVoiceVolume, 80); assert.equal(s.LiveNoteSeVolume, 70); + assert.equal(context({ scene: sceneJson({ settings: { optionDefaults: { LaneOpacity: "35" } } }) }).defaults.LaneOpacity, 35); assert.deepEqual(liveDerived(ctx, s), { noteSpeed: 5, timingAdjustmentMs: 0, chartPositionMs: 0, viewProgressOffset: 0 }); assert.equal(liveNoteSeMaps(ctx, s), null); assert.deepEqual(liveCategoryVolumes(ctx, s), { LiveBgmConfig: 1, LiveSeConfig: 1, LiveVoiceConfig: 1, LiveNotesSeConfig: 1 }); @@ -98,49 +100,54 @@ test("validation: names, types, ranges, the values the chart offers, options not assert.equal(s.NoteSpeed, F(8.25)); // floats are float32 assert.equal(ctx.resolve({ NoteTiming: 0.1 }).NoteTiming, F(0.1)); - // offered with the data: a mirrored score, skins, effect sets, note SE groups - const rich = context({ live: { notesMirror: "score/x.mirror.notes.json" }, - notes: notesJson({ skins: { 1: "skin001", 2: "skin002" }, effects: { 1: "effect001", 2: "effect001Simple" } }), - audio: audioJson({ groups: { 1: { 2: 102 }, 2: { 2: 202 } } }) }); - // a skin id offered only with its record + // offered with the data: a mirrored score, note skins and effect sets loaded with the live, note SE groups + const richOf = (effects) => context({ live: { notesMirror: "score/x.mirror.notes.json" }, + notes: notesJson({ skins: { 1: "skin001", 2: "skin002" }, effects: { 1: "effect001", 2: "effect001Simple" } }), + audio: audioJson({ groups: { 1: { 2: 102 }, 2: { 2: 202 } } }), skins: ["skin001"], effects }); + const rich = richOf(["effect001", "effect001Simple"]); + // a skin id offered only with its skin loaded assert.throws(() => rich.resolve({ NoteDesignId: 2 }), /offers 1/); const r = rich.resolve({ MirrorChart: true, NoteEffectId: 2, NoteSePatternId: 2, UseIndividualNoteSe: true }); assert.equal(r.MirrorChart, true); assert.equal(liveNoteEffectName(rich, r), "effect001Simple"); - // LiveQuality Low: the Light variant only where the data has it (ExistsAsset), else the set itself + // LiveQuality Low: the Light variant only where the loaded effect sets have it, else the set itself assert.equal(liveNoteEffectName(rich, { ...r, LiveQuality: 2 }), "effect001Simple"); - rich.notes.assets = { "Effect/Live/NoteEffect/effect001Light/LiveNoteEffectAssetSettings": {} }; - assert.equal(liveNoteEffectName(rich, { ...r, LiveQuality: 2 }), "effect001Simple"); - assert.equal(liveNoteEffectName(rich, { ...r, NoteEffectId: 1, LiveQuality: 2 }), "effect001Light"); - rich.notes.assets["Effect/Live/NoteEffect/effect001SimpleLight/LiveNoteEffectAssetSettings"] = {}; - assert.equal(liveNoteEffectName(rich, { ...r, LiveQuality: 2 }), "effect001SimpleLight"); + assert.equal(liveNoteEffectName(richOf(["effect001", "effect001Simple", "effect001SimpleLight"]), { ...r, LiveQuality: 2 }), + "effect001SimpleLight"); + assert.equal(liveNoteEffectName(richOf(["effect001", "effect001Light"]), { ...r, NoteEffectId: 1, LiveQuality: 2 }), "effect001Light"); + assert.equal(liveNoteEffectName(rich, { ...r, NoteEffectId: 1, LiveQuality: 2 }), "effect001"); // without an effects table: the default set by the data's name - const plain = context({ info: { quality: 1, options: { LiveQuality: [1, 2] } }, - notes: { ...notesJson(), assets: { "Effect/Live/NoteEffect/effect001Light/LiveNoteEffectAssetSettings": {} } } }); + const plain = context({ info: { quality: 1, options: { LiveQuality: [1, 2] } }, effects: ["effect001", "effect001Light"] }); assert.equal(liveNoteEffectName(plain, plain.resolve({ LiveQuality: 2 })), "effect001Light"); }); -test("note designs: offered with their records in notes.json noteSkins (top level)", () => { - const skins = { 1: "skin001", 2: "skin002", 3: "skin003" }; - const top = context({ notes: { ...notesJson({ skins }), noteSkins: { skin002: {}, skin003: {} } } }); - assert.deepEqual(top.values(LIVE_OPTIONS.find((o) => o.name === "NoteDesignId")), [1, 2, 3]); - assert.equal(top.resolve({ NoteDesignId: 3 }).NoteDesignId, 3); - // records under settings are not the data format's place - const misplaced = context({ notes: notesJson({ skins, noteSkins: { skin002: {} } }) }); - assert.deepEqual(misplaced.values(LIVE_OPTIONS.find((o) => o.name === "NoteDesignId")), [1]); -}); - -test("lane effects: always the effect001 set (LaneEffectLoadStep.GetAssetPath), whatever the note effect set", () => { - assert.equal(LIVE_LANE_EFFECT_SET, "effect001"); - const notes = { settings: { effect: "effect001Simple" }, - assets: { "Effect/Live/LaneEffect/effect001/LiveLaneEffectAssetSettings": {} } }; - const lane = new LiveLaneEffects(null, { notes, score: { laneCount: 24, notes: [] }, - sceneInfo: { laneWidth: 19.12, laneContainers: [] }, effect: "effect001SimpleLight" }); +test("note designs: offered when their skin is loaded with the live (settings.skins names)", () => { + const skins = { 1: "skin001", 2: "skin002", 3: "skin003" }, item = LIVE_OPTIONS.find((o) => o.name === "NoteDesignId"); + const all = context({ notes: notesJson({ skins }), skins: ["skin002", "skin003"] }); + assert.deepEqual(all.values(item), [1, 2, 3]); + assert.equal(all.resolve({ NoteDesignId: 3 }).NoteDesignId, 3); + // a design whose skin is not loaded is not offered + const some = context({ notes: notesJson({ skins }), skins: ["skin002"] }); + assert.deepEqual(some.values(item), [1, 2]); + assert.throws(() => some.resolve({ NoteDesignId: 3 }), /offers 1, 2/); + // no skin loaded: the default design only + assert.deepEqual(context({ notes: notesJson({ skins }) }).values(item), [1]); +}); + +test("lane effects: the lane set is notes.settings.laneEffect (effect001), whatever the note effect set", () => { + const notes = { laneEffect: "lane-settings", settings: { effect: "effect001Simple", laneEffect: "effect001" } }; + const looked = []; + const objects = { mono: (id) => { looked.push(id); return {}; } }; + const lanes = (laneContainers) => new LiveLaneEffects(null, { objects, notes, score: { laneCount: 24, notes: [] }, + sceneInfo: { laneWidth: 19.12, laneContainers } }); + const lane = lanes([]); + assert.deepEqual(looked, ["lane-settings"]); // the settings object is notes.laneEffect assert.equal(lane.views.size, 0); - delete notes.assets["Effect/Live/LaneEffect/effect001/LiveLaneEffectAssetSettings"]; - assert.throws(() => new LiveLaneEffects(null, { notes, score: { laneCount: 24, notes: [] }, - sceneInfo: { laneWidth: 19.12, laneContainers: [] } }), - /LaneEffect\/effect001\/LiveLaneEffectAssetSettings not exported/); + assert.throws(() => lanes([{ name: "x", type: 1, field: "_missing" }]), /effect001: _missing missing/); + // the settings object must be in the store (a stub with AssetStore.object's error) + const absent = { mono: (id) => { throw new Error(`object not found: ${id}`); } }; + assert.throws(() => new LiveLaneEffects(null, { objects: absent, notes, score: { laneCount: 24, notes: [] }, + sceneInfo: { laneWidth: 19.12, laneContainers: [] } }), /object not found: lane-settings/); }); // ---- note skins: flick arrows (LiveFlickNoteView / LiveDirectionFlickNoteView, ArrowGradientAnimator) @@ -149,7 +156,7 @@ const arrowSprite = (name, us) => ({ sprite: name, rect: { width: 20, height: 10 const flickHead = (unit) => { const v = Object.create(NoteHeadView.prototype); const geo = Object.assign(Object.create(NoteGeo.prototype), { unit: F(0.5), headPlacement: () => ({ x: 0, y: 0 }) }); - Object.assign(v, { owner: { geo, clip: (id) => ({ id }) }, unit, main: null, mark: null, left: null, right: null, subArrow: null, + Object.assign(v, { owner: { geo, arrowClip: (view, loop) => ({ id: loop.clip }) }, unit, main: null, mark: null, left: null, right: null, subArrow: null, parts: null, p: { root: { t: {} } }, arrow: { sprite: null, size: { x: -1, y: -1 }, order: 0 } }); return v; }; @@ -233,19 +240,23 @@ test("music score position: bars from the last BPM / bar change in float32; zero { bar: 0, rhythm: 0, rhythmicUnit: 0, barProgress: 0, timeMs: 0 }); }); -const barLinePrefab = () => ({ key: "bar_line_view", nodes: [{ +const barLinePrefab = () => ({ name: "LiveBarLineView", nodes: [{ path: "LiveBarLineView", name: "LiveBarLineView", active: true, layer: 25, localPosition: { x: 0, y: 0, z: 0 }, localRotation: { x: 0, y: 0, z: 0, w: 1 }, localScale: { x: 1, y: 1, z: 1 }, - components: [{ type: "MonoBehaviour", class: "LiveBarLineView", _viewProgress: 0, _renderer: { gameObject: "LiveBarLineView" } }, - { type: "SpriteRenderer", m_Enabled: 1, m_Sprite: { sprite: "bar", texture: { texture: "t" } }, m_Materials: [{ material: "m" }], - m_Size: { x: 1, y: F(0.04) }, m_SortingOrder: 4999, m_Color: { r: 1, g: 1, b: 1, a: 1 } }] }] }); + components: [{ type: "MonoBehaviour", pathId: "1", script: "Assembly-CSharp|Live|LiveBarLineView", + fields: { _viewProgress: 0, _renderer: { gameObject: "LiveBarLineView" } } }, + { type: "SpriteRenderer", pathId: "2", + fields: { m_Enabled: 1, m_Sprite: { sprite: "bar", texture: { texture: "t" } }, m_Materials: [{ material: "m" }], + m_Size: { x: 1, y: F(0.04) }, m_SortingOrder: 4999, m_Color: { r: 1, g: 1, b: 1, a: 1 } } }] }] }); test("bar lines: the bar lines after the position's bar within the display offset, views reused in order", () => { const geo = Object.assign(Object.create(NoteGeo.prototype), { laneCount: 24, unit: F(0.5), headPlacement: (c, t) => ({ x: 7, y: F(10 * t) }) }); const score = { barLineTimeMs: [0, 1000, 2000, 3000, 4000, 5000], bpmChanges: [{ bpm: 240, bar: 0, barProgress: 0, timeMs: 0 }], barChanges: [{ beatsPerBar: 4, bar: 0, barProgress: 0, timeMs: 0 }] }; // one bar per second - const owner = { score, geo, displayOffsetMs: 1500 }; - const bl = new LiveBarLines(owner, barLinePrefab(), null); + // the view prefab's stand-in store: the prefab by id, its sprite and material by the references given + const objects = { assets: { prefab: () => barLinePrefab() }, sprite: (v) => v, material: (v) => v }; + const owner = { score, geo, objects, displayOffsetMs: 1500 }; + const bl = new LiveBarLines(owner, "bar_line_view", null); bl.update({ timeMs: 1200 }); // bar 1: lines 2 (2000) within 1200 + 1500; 3 (3000) not assert.equal(bl.active.length, 1); const v = bl.active[0]; @@ -269,18 +280,22 @@ test("bar lines: the container's element root relative to screen_root; offered o const node = (path, pos = { x: 0, y: 0, z: 0 }, components = []) => ({ path, name: path.split("/").pop(), active: true, layer: 0, localPosition: pos, localRotation: { x: 0, y: 0, z: 0, w: 1 }, localScale: { x: 1, y: 1, z: 1 }, components }); const base = "LiveGameView/LiveGameCamera/screen_root/LiveGameAllNoteView"; + const script = (cls) => `Assembly-CSharp|Live|${cls}`; const scene = new Prefab({ key: "scene", nodes: [ node("LiveGameView"), node("LiveGameView/LiveGameCamera"), node("LiveGameView/LiveGameCamera/screen_root", { x: 0, y: 0, z: 10 }), - node(base, { x: 0, y: 0, z: 0 }, [{ type: "MonoBehaviour", class: "LiveAllNoteView", _barLineView: { gameObject: `${base}/LiveAllBarLineView` } }]), - node(`${base}/LiveAllBarLineView`, { x: 0, y: 1, z: 0 }, [{ type: "MonoBehaviour", class: "LiveAllBarLineView", _container: { gameObject: `${base}/LiveAllBarLineView/container` } }]), - node(`${base}/LiveAllBarLineView/container`, { x: 0, y: 0, z: 2 }, [{ type: "MonoBehaviour", class: "LiveBarLineViewContainer", - _elementRoot: { transform: `${base}/LiveAllBarLineView/container` }, _elementPrefab: { gameObject: "LiveBarLineView" } }]) ] }); + node(base, { x: 0, y: 0, z: 0 }, [{ type: "MonoBehaviour", script: script("LiveAllNoteView"), + fields: { _barLineView: { gameObject: `${base}/LiveAllBarLineView` } } }]), + node(`${base}/LiveAllBarLineView`, { x: 0, y: 1, z: 0 }, [{ type: "MonoBehaviour", script: script("LiveAllBarLineView"), + fields: { _container: { gameObject: `${base}/LiveAllBarLineView/container` } } }]), + node(`${base}/LiveAllBarLineView/container`, { x: 0, y: 0, z: 2 }, [{ type: "MonoBehaviour", script: script("LiveBarLineViewContainer"), + fields: { _elementRoot: { transform: `${base}/LiveAllBarLineView/container` }, _elementPrefab: { gameObject: "LiveBarLineView" } } }]) ] }); const m = LiveBarLines.elementRoot(scene); assert.deepEqual([m[12], m[13], m[14]], [0, 1, 2]); // screen_root's own z not included const item = LIVE_OPTIONS.find((o) => o.name === "MeasureLineDisplay"); assert.deepEqual(context().values(item), [false]); assert.throws(() => context().resolve({ MeasureLineDisplay: true }), /this chart offers false/); - const withPrefab = context({ notes: { ...notesJson(), prefabs: { bar_line_view: barLinePrefab() } } }); + // the scene's bar line view prefab (scene.json prefabs.barLineView) offers the bar lines + const withPrefab = context({ scene: sceneJson({ prefabs: { barLineView: "bar_line_view" } }) }); assert.deepEqual(withPrefab.values(item), [false, true]); assert.equal(withPrefab.resolve({ MeasureLineDisplay: true }).MeasureLineDisplay, true); assert.deepEqual(liveSettingsChanges(withPrefab.resolve({}), withPrefab.resolve({ MeasureLineDisplay: true })).boot, ["MeasureLineDisplay"]); diff --git a/tests/unit/live-store.test.mjs b/tests/unit/live-store.test.mjs index 20e17f7..a5c1dbe 100644 --- a/tests/unit/live-store.test.mjs +++ b/tests/unit/live-store.test.mjs @@ -6,6 +6,7 @@ import { AssetStore, PRODUCT_MANIFEST } from "../../src/data/assets.js"; import { LiveData } from "../../src/live/data.js"; import { isLegacyLive, legacyLive } from "../../src/live/legacy.js"; import { LIVE_PARTS, loadLiveStore, productDir } from "../../src/live/store.js"; +import { chartFiles } from "./live-legacy-fixture.mjs"; const ROOT = "https://example.org/out/"; @@ -115,22 +116,9 @@ test("LiveData: the score, and the sounds of the live.sounds product with the mo assert.throws(() => new LiveData(new AssetStore()), /not the store of a live/); }); -// a chart of the site layout: live.json, the score, live-audio.json and its waveforms -const chart = (extra = {}) => new AssetStore({ - text: { - "live.json": JSON.stringify({ musicId: 7, difficulty: "expert", scene: "livescene/scene.json", - notes: "score/m7_03.notes.json", notesMirror: "score/m7_03.mirror.notes.json", - noteAssets: "livenotes/notes.json", liveAudio: "audio/live-audio.json", ...extra }), - "score/m7_03.notes.json": JSON.stringify({ notes: [1] }), "score/m7_03.mirror.notes.json": JSON.stringify({ notes: [2] }), - "audio/live-audio.json": JSON.stringify({ - sounds: { 1: cue("bgm_7", "audio/bgm_7/bgm_7.m4a", ["LiveBgm"]), 10: cue("se1", "audio/se1/a.flac", ["LiveSe"]) }, - categories: { LiveBgm: 1, LiveSe: 1 }, react: [], music: { soundId: 1 }, noteSe: { types: { 1: 10 } }, liveSe: {}, - timeline: { startCheerDelaySec: 1 } }), - "livescene/scene.json": "{}", "livenotes/notes.json": "{}", - }, - bytes: { "audio/bgm_7/bgm_7.m4a": new Uint8Array([7]), "audio/se1/a.flac": new Uint8Array([1]) }, - info: { format: 3, chart: { title: "x" }, quality: 1 }, -}); +// a chart of the site layout (live-legacy-fixture.mjs) in memory +const chart = (extra = {}) => new AssetStore({ ...chartFiles(extra), info: { format: 3, chart: { title: "x" }, quality: 1 } }); +const refId = (v) => `${v.$ref.file}:${v.$ref.pathId}`; test("legacyLive: a chart of the site layout read as the store of a live", () => { const old = chart(); @@ -139,11 +127,71 @@ test("legacyLive: a chart of the site layout read as the store of a live", () => assert.deepEqual([data.score().notes, data.score(true).notes], [[1], [2]]); assert.equal(store.info.dir, "lives/7_expert"); assert.deepEqual([store.info.chart, store.info.quality, store.info.product], [{ title: "x" }, 1, "live"]); + assert.deepEqual([data.live.band, store.object(data.live.jacket).class], [{ band: 3 }, "Sprite"]); const a = data.audio(); assert.deepEqual([a.music, Object.keys(a.sounds)], [{ soundId: 1 }, ["1", "10"]]); assert.deepEqual([...store.bytes(a.sounds[1].layers[0].file)], [7]); assert.deepEqual([...store.bytes(a.sounds[10].layers[0].file)], [1]); - assert.deepEqual([data.sceneDoc, data.noteAssets], [{}, {}]); + assert.deepEqual([data.noteskin().name, data.noteeffect().name], ["skin001", "effect001"]); assert.equal(new LiveData(legacyLive(chart({ notesMirror: undefined }))).live.notesMirror, undefined); assert.throws(() => legacyLive(chart({ musicId: undefined })), /live\.json: no musicId/); }); + +test("legacyLive: the scene's roots, references between them, the graphics settings and the band's timeline", () => { + const store = legacyLive(chart()), scene = new LiveData(store).scene; + const [live, stage] = scene.scene.map((id) => store.prefab(id)); + assert.deepEqual([live.name, stage.name], ["Live", "Stage"]); + // a reference into another root: the component; inside the root: the node path as it is + const view = stage.nodes[0].components[0], judge = live.nodes.find((n) => n.path === "Live/judge"); + assert.deepEqual([view.type, view.script], ["MonoBehaviour", "StageView"]); + assert.equal(view.fields._judge.$ref.pathId, judge.components[0].pathId); + assert.deepEqual(view.fields._self, { gameObject: "Stage/mesh" }); + // materials name their shader by object; a shader the chart does not pack names no object + const mat = store.material(refId(judge.components[0].fields.m_Materials[0])); + assert.equal(refId(mat.shader), scene.shaders["Sprites/Default"]); + assert.ok(store.roles(scene.shaders["Sprites/Default"]).includes("program:gles3/v0.glsl")); + const mesh = stage.nodes.find((n) => n.path === "Stage/mesh"); + assert.equal(store.hasObject(refId(store.material(refId(mesh.components[0].fields.m_Materials[0])).shader)), false); + assert.deepEqual(store.mesh(refId(mesh.components[1].fields.m_Mesh)).submeshes, [[0, 1, 2]]); + // a texture has its image when the chart carries the file + const tex = (sp) => refId(store.objectJson(refId(sp)).renderData.texture); + assert.ok(store.roles(tex(judge.components[0].fields.m_Sprite)).includes("image")); + const noteJudge = store.prefab(scene.prefabs.judgeEffectView).nodes[0].components[0].fields.m_Sprite; + assert.deepEqual(store.roles(tex(noteJudge)), ["json"]); + // GraphicsSettings -> the pipeline -> its renderers -> the post-process data with the renderer's film grain + const g = store.objectJson(scene.graphics.graphics), pipe = store.mono(refId(g.m_CustomRenderPipeline)); + assert.equal(pipe.m_RendererDataList[1], null); + const post = store.mono(refId(store.mono(refId(pipe.m_RendererDataList[0])).postProcessData)); + assert.equal(store.object(refId(post.textures.filmGrainTex[0])).class, "Texture2D"); + assert.deepEqual(scene.settings.optionDefaults, { LaneOpacity: "80" }); + // the band's timeline: its track bound to the timeline prefab's Animator, the clip's bindings under it; an infinite + // extrapolation time stays infinite + const band = scene.bands[3], [trackId] = Object.keys(band.tracks); + assert.deepEqual(band.tracks[trackId], { name: "Band", prefab: band.timeline, animator: "StartTimeline/band" }); + assert.equal(store.mono(trackId).m_Clips[0].m_PostExtrapolationTime, Infinity); + assert.deepEqual(Object.values(scene.bindings[band.timeline]["StartTimeline/band"]), [[{ path: "body", attribute: "m_LocalPosition.x" }]]); + // the scene's Animator plays the note assets' controller + const ctrl = refId(live.nodes[0].components[0].fields.m_Controller); + assert.equal(store.object(ctrl).class, "AnimatorController"); + assert.deepEqual(Object.values(scene.bindings[scene.scene[0]].Live), [[{ path: "Live/judge", attribute: "m_IsActive" }]]); +}); + +test("legacyLive: the note assets, the skin's arrow clip and the effect sets' references to their prefabs", () => { + const store = legacyLive(chart()), data = new LiveData(store); + const notes = data.notes, skin = data.noteskin(), effect = data.noteeffect(); + assert.deepEqual([notes.settings.laneEffect, notes.settings.judgementSprites], ["effect001", undefined]); + const flick = notes.prefabs.flick_note_view; + // the LoopAnimationTimeSync's clip in the notes bindings, the skin's arrow clip (a sprite curve) in the skin's + const anim = "flick_note_view/arrow/anim_root"; + assert.deepEqual(Object.values(notes.bindings[flick][anim]), [[{ path: "arrow", attribute: "m_LocalPosition.y" }]]); + const unit = store.mono(refId(store.mono(skin.skin).FlickNoteAsset)); + assert.deepEqual(skin.bindings[flick][anim][refId(unit._arrowLoopAnimation)], [{ path: "arrow", attribute: "m_Sprite" }]); + // an effect set's field: the effect asset component on the root of its prefab + for (const [settings, cls] of [[notes.laneEffect, "LiveLaneEffectAsset"], [effect.settings, "LiveNoteEffectAsset"]]) { + const r = store.mono(settings)._tap, root = store.prefab(store.contained(r).prefab).nodes[0]; + assert.equal(root.components.find((c) => c.pathId === r.$ref.pathId).script, cls); + const ps = store.prefab(store.contained(r).prefab).nodes[1].components[0].fields; + assert.deepEqual(ps.m_VertexStreams, { $hex: "00010304" }); + } + assert.equal(store.mono(data.scene.spriteAssets.combo).$script, "LiveComboSpriteAsset"); +}); \ No newline at end of file diff --git a/tests/unit/read-set.test.mjs b/tests/unit/read-set.test.mjs index 965e481..6368fe3 100644 --- a/tests/unit/read-set.test.mjs +++ b/tests/unit/read-set.test.mjs @@ -14,42 +14,54 @@ import { fileURLToPath } from "node:url"; import { LiveNotes, NoteGL } from "../../src/live/noteview.js"; import { NoteGeo } from "../../src/live/notegeo.js"; import { DirStore, headlessGL, storeFromDir } from "../../scripts/lib/headless.mjs"; +import { chartFiles } from "./live-legacy-fixture.mjs"; const script = path.join(path.dirname(fileURLToPath(import.meta.url)), "..", "..", "scripts", "read-set.mjs"); // ---- synthetic note views const TRS = { localPosition: { x: 0, y: 0, z: 0 }, localRotation: { x: 0, y: 0, z: 0, w: 1 }, localScale: { x: 1, y: 1, z: 1 } }; -const mat = (name, shader = "Sprites/Default") => ({ material: name, shader: { shader }, keywords: [], floats: { _Z: 1 }, colors: {} }); -const sr = (m, enabled = 1) => ({ type: "SpriteRenderer", m_Enabled: enabled, m_Materials: [m], m_Size: { x: 1, y: 1 }, - m_Color: { r: 1, g: 1, b: 1, a: 1 }, m_SortingOrder: 0, m_SortingLayer: 0, m_DrawMode: 0 }); +// a material record as LiveObjects.material gives it: the shader by name +const mat = (name, shader = "Sprites/Default") => ({ material: name, shader, keywords: [], floats: { _Z: 1 }, colors: {} }); const sprite = (name) => ({ sprite: name, texture: { texture: `textures/${name}.png` }, rect: { x: 0, y: 0, width: 4, height: 4 } }); -// a head prefab: root (view script) with a main and a mark renderer -const head = (name, main, mark) => ({ nodes: [ - { path: name, name, active: true, ...TRS, components: [{ type: "MonoBehaviour", class: "LiveNoteView", - _mainSpriteRenderer: { gameObject: `${name}/main` }, _markSpriteRenderer: { gameObject: `${name}/mark` } }] }, +// references in prefab.json fields, resolved by the store stub below by their name +const mref = (name) => ({ $ref: { file: "materials", pathId: name } }); +const sref = (name) => ({ $ref: { file: "sprites", pathId: name } }); +const srFields = (m, enabled = 1) => ({ m_Enabled: enabled, m_Materials: [mref(m)], m_Sprite: null, m_Size: { x: 1, y: 1 }, + m_Color: { r: 1, g: 1, b: 1, a: 1 }, m_SortingOrder: 0, m_SortingLayer: 0, m_DrawMode: 0 }); +const sr = (m, enabled = 1) => ({ type: "SpriteRenderer", fields: srFields(m, enabled) }); +// a head prefab (prefab.json/2): root (view script) with a main and a mark renderer +const head = (name, main, mark) => ({ name, nodes: [ + { path: name, name, active: true, ...TRS, components: [{ type: "MonoBehaviour", script: "Assembly-CSharp|Live|LiveNoteView", + fields: { _mainSpriteRenderer: { gameObject: `${name}/main` }, _markSpriteRenderer: { gameObject: `${name}/mark` } } }] }, { path: `${name}/main`, name: "main", active: true, ...TRS, components: [sr(main)] }, { path: `${name}/mark`, name: "mark", active: true, ...TRS, components: [sr(mark)] }, ] }); const M = Object.fromEntries(["tap", "tapMark", "slide", "slideMark", "conn", "connMark", "end", "endMark", "flick", "flickMark", "pair", "body"].map((k) => [k, mat(k)])); -const ln = { - prefabs: { - tap_note_view: head("tap_note_view", M.tap, M.tapMark), - slide_note_view: head("slide_note_view", M.slide, M.slideMark), - connection_note_view: head("connection_note_view", M.conn, M.connMark), - slide_end_note_view: head("slide_end_note_view", M.end, M.endMark), - flick_note_view: head("flick_note_view", M.flick, M.flickMark), - guide_note_view: { nodes: [{ path: "guide_note_view", name: "guide_note_view", active: true, ...TRS, components: [] }] }, - pair_note_line: { nodes: [{ path: "pair_note_line", name: "pair_note_line", active: true, ...TRS, - components: [{ ...sr(M.pair), m_Sprite: sprite("pair") }] }] }, - }, - noteSkin: { // the tap and connection skins have no centre mark: no mark drawn - TapNoteAsset: { _mainSprite: sprite("tap"), _centerMarkSprite: null }, - SlideNoteAsset: { _mainSprite: sprite("slide"), _centerMarkSprite: sprite("slide_mark") }, - SlideConnectNoteAsset: { _mainSprite: sprite("conn"), _centerMarkSprite: null }, - SlideEndNoteAsset: { _mainSprite: sprite("end"), _centerMarkSprite: sprite("end_mark") }, - FlickNoteAsset: { _mainSprite: sprite("flick"), _centerMarkSprite: sprite("flick_mark") }, - }, +const PREFABS = { + tap_note_view: head("tap_note_view", "tap", "tapMark"), + slide_note_view: head("slide_note_view", "slide", "slideMark"), + connection_note_view: head("connection_note_view", "conn", "connMark"), + slide_end_note_view: head("slide_end_note_view", "end", "endMark"), + flick_note_view: head("flick_note_view", "flick", "flickMark"), + guide_note_view: { name: "guide_note_view", nodes: [{ path: "guide_note_view", name: "guide_note_view", active: true, ...TRS, components: [] }] }, + pair_note_line: { name: "pair_note_line", nodes: [{ path: "pair_note_line", name: "pair_note_line", active: true, ...TRS, + components: [{ type: "SpriteRenderer", fields: { ...srFields("pair"), m_Sprite: sref("pair") } }] }] }, +}; +// the objects the read set reads: prefab documents by id, materials and sprites by reference +const store = { + assets: { prefab: (id) => PREFABS[id] }, + material: (v) => M[v.$ref.pathId], + sprite: (v) => sprite(v.$ref.pathId), +}; +const ln = { prefabs: Object.fromEntries(Object.keys(PREFABS).map((name) => [name, name])) }; +// the note skin's units, resolved (sprites in place of their references) +const SKIN = { // the tap and connection skins have no centre mark: no mark drawn + TapNoteAsset: { _mainSprite: sprite("tap"), _centerMarkSprite: null }, + SlideNoteAsset: { _mainSprite: sprite("slide"), _centerMarkSprite: sprite("slide_mark") }, + SlideConnectNoteAsset: { _mainSprite: sprite("conn"), _centerMarkSprite: null }, + SlideEndNoteAsset: { _mainSprite: sprite("end"), _centerMarkSprite: sprite("end_mark") }, + FlickNoteAsset: { _mainSprite: sprite("flick"), _centerMarkSprite: sprite("flick_mark") }, }; const note = (id, op, extra = {}) => ({ id, op, timeMs: 1000 * id, laneStart: 2, laneEnd: 4, laneStartFloat: 2, laneEndFloat: 4, width: 3, critical: false, direction: "Normal", pairNoteId: 0, lineIds: [], ...extra }); @@ -57,7 +69,8 @@ const GEO = new NoteGeo({ laneCount: 24, laneSize: [1912, 1000], laneTopRange: 0 judgementScreenBottomPosition: 2.24 }); const planned = (notes, { showPairLines = true } = {}) => { const owner = Object.assign(Object.create(LiveNotes.prototype), { - ln, score: { notes }, geo: GEO, showPairLines, notes: new Map(notes.map((n) => [n.id, n])), material: M.body, + objects: store, ln, skin: SKIN, score: { notes }, geo: GEO, showPairLines, notes: new Map(notes.map((n) => [n.id, n])), + material: M.body, _clips: new Map(), }); return LiveNotes.prototype.plannedMaterials.call(owner).map((m) => m.material).sort(); @@ -125,18 +138,16 @@ test("read-set --serve answers each request line in order, one JSON line each, u } finally { fs.rmSync(dir, { recursive: true, force: true }); } }); -// a live directory whose session gets as far as the Live options: live.json, empty scene and note asset files, an -// audio file with the music's sound, no score (a load with accepted settings fails at the score) +// a live directory whose session gets as far as the Live options: the synthetic chart of live-legacy-fixture.mjs +// without its score (a load with accepted settings fails at the score) const settingsDir = () => { const dir = fs.mkdtempSync(path.join(os.tmpdir(), "ournotes-settings-")); - fs.writeFileSync(path.join(dir, "live.json"), JSON.stringify({ - musicId: 1, difficulty: "expert", scene: "livescene/scene.json", noteAssets: "livenotes/notes.json", - liveAudio: "audio.json", notes: "score.json" })); - for (const d of ["livescene", "livenotes"]) { - fs.mkdirSync(path.join(dir, d)); - fs.writeFileSync(path.join(dir, d, d === "livescene" ? "scene.json" : "notes.json"), "{}"); + const { text, bytes } = chartFiles({ musicId: 1, notes: "score.json", notesMirror: undefined }); + for (const [p, body] of [...Object.entries(text), ...Object.entries(bytes)]) { + if (p.startsWith("score/")) continue; + fs.mkdirSync(path.dirname(path.join(dir, p)), { recursive: true }); + fs.writeFileSync(path.join(dir, p), body); } - fs.writeFileSync(path.join(dir, "audio.json"), JSON.stringify({ music: { soundId: 1 }, sounds: { 1: { layers: [] } } })); return dir; }; From 9bc333571ab359cdf784ca27c45f7cf763805a92 Mon Sep 17 00:00:00 2001 From: nichinichisou Date: Thu, 15 Oct 2026 22:02:56 +0800 Subject: [PATCH 05/18] feat(data): draw atlas-packed sprites and read mesh submeshes by topology A sprite packed by a SpriteAtlas of another serialized file has no render data of its own: AssetStore.sprite reads it from the atlas' m_RenderDataMap entry under the sprite's m_RenderDataKey, as the engine does at run time. A Mesh's arrays document lists each submesh with its topology and indices; AssetStore.mesh gives the triangle indices of each submesh and reports a submesh of another topology as an error. Co-Authored-By: Claude Opus 5.5 --- src/data/assets.js | 27 +++++++++--- src/live/legacy.js | 3 +- tests/unit/assets.test.mjs | 68 ++++++++++++++++++++++++++++++ tests/unit/live-legacy-fixture.mjs | 2 +- 4 files changed, 93 insertions(+), 7 deletions(-) diff --git a/src/data/assets.js b/src/data/assets.js index 5a36a9a..0a0a9c5 100644 --- a/src/data/assets.js +++ b/src/data/assets.js @@ -323,16 +323,28 @@ export class AssetStore { } // A Sprite as it is drawn from its render data (sprite.json: the typetree `sprite`, its render mesh `mesh`, the - // render data `renderData`): {id, name, texture (texture() of the page it is drawn from), rect, pivot, border, + // render data `renderData`; a sprite packed by a SpriteAtlas of another file has no `renderData`: its render data is + // the atlas' m_RenderDataMap entry under the sprite's m_RenderDataKey, as the engine reads it at run time): + // {id, name, texture (texture() of the page it is drawn from), rect, pivot, border, // pixelsToUnits, offset, textureRect, textureRectOffset, atlasRectOffset, downscaleMultiplier, settingsRaw, // uvTransform, vertices, uv, indices}. The texture coordinates of a vertex p are ((p.x * uvTransform.x + // uvTransform.y) / width, (p.y * uvTransform.z + uvTransform.w) / height) of the page, in float32; vertices, uv and // indices are null for a sprite without a render mesh. Made once and shared: callers do not modify it. + // the render data of sprite `id` (typetree `t`) in its SpriteAtlas (m_SpriteAtlas): the m_RenderDataMap entry whose + // key is the sprite's m_RenderDataKey + _atlasRenderData(id, t) { + if (!t.m_SpriteAtlas) throw new Error(`sprite ${id} (${t.m_Name}): no render data and no atlas`); + const atlas = this.objectJson(this.ref(t.m_SpriteAtlas, "SpriteAtlas")), key = JSON.stringify(t.m_RenderDataKey); + const e = atlas.m_RenderDataMap.find(([k]) => JSON.stringify(k) === key); + if (!e) throw new Error(`sprite ${id} (${t.m_Name}): not in the render data of atlas ${atlas.m_Name}`); + return e[1]; + } + sprite(id) { this.object(id, "Sprite"); let s = this._sprites.get(id); if (s) return s; - const j = this.objectJson(id), t = j.sprite, rd = j.renderData, ut = rd.uvTransform; + const j = this.objectJson(id), t = j.sprite, rd = j.renderData ?? this._atlasRenderData(id, t), ut = rd.uvTransform; const tex = this.texture(this.ref(rd.texture, "Texture2D")), f = Math.fround; const m = j.mesh; if (m && m.submeshes.length !== 1) throw new Error(`sprite ${id} (${t.m_Name}): ${m.submeshes.length} submeshes`); @@ -348,14 +360,19 @@ export class AssetStore { } // A Mesh in Unity's space, from its `arrays` file: {id, name, vertices, normals, tangents, colors, uv0 .. uv7 (each - // [[...]] per vertex, absent channels null), submeshes (the triangle indices of each submesh, flat), skin ({joints, - // weights}: four per vertex, or null), bindPoses (the typetree's m_BindPose)}. Made once and shared. + // [[...]] per vertex, absent channels null), submeshes (the triangle indices of each submesh, flat; a submesh of + // another topology is an error), skin ({joints, weights}: four per vertex, or null), bindPoses (the typetree's + // m_BindPose)}. Made once and shared. mesh(id) { const o = this.object(id, "Mesh"); let m = this._meshes.get(id); if (m) return m; const a = this.objectJson(id, "arrays"), t = this.objectJson(id); - m = { id, name: o.name, vertices: a.positions, submeshes: a.submeshes.map((s) => s.flat()), + const tris = (s, i) => { + if (s.topology !== "triangles" || !s.indices) throw new Error(`mesh ${id} (${o.name}): submesh ${i} is ${s.topology}`); + return s.indices.flat(); + }; + m = { id, name: o.name, vertices: a.positions, submeshes: a.submeshes.map(tris), skin: a.joints ? { joints: a.joints, weights: a.weights } : null, bindPoses: t.m_BindPose || [] }; for (const k of ["normals", "tangents", "colors", "uv0", "uv1", "uv2", "uv3", "uv4", "uv5", "uv6", "uv7"]) m[k] = a[k] ?? null; this._meshes.set(id, m); diff --git a/src/live/legacy.js b/src/live/legacy.js index 6a26bd0..0c4683e 100644 --- a/src/live/legacy.js +++ b/src/live/legacy.js @@ -172,7 +172,8 @@ class LegacyFile { // a mesh record {mesh, vertices, normals, uv0, submeshes (flat indices), m_Skin, m_BindPose, ...} mesh(m) { return this.add("Mesh", m.mesh, () => { - const arrays = { positions: m.vertices, normals: m.normals ?? null, uv0: m.uv0 ?? null, submeshes: m.submeshes.map(triangles) }; + const arrays = { positions: m.vertices, normals: m.normals ?? null, uv0: m.uv0 ?? null, + submeshes: m.submeshes.map((s) => ({ topology: "triangles", indices: triangles(s) })) }; if (m.m_Skin) { arrays.joints = m.m_Skin.map((w) => w.boneIndex); arrays.weights = m.m_Skin.map((w) => w.weight); diff --git a/tests/unit/assets.test.mjs b/tests/unit/assets.test.mjs index 7ac37a0..71bcb1f 100644 --- a/tests/unit/assets.test.mjs +++ b/tests/unit/assets.test.mjs @@ -318,3 +318,71 @@ test("fromManifest: cancellation while reading an empty manifest still rejects", finishRead({ files: {} }); await rejected; }); + +// ------------------------------------------------------------------------------------------------ sprites and meshes +// object ids are ":" (refId); a reference value names one +const ref = (file, pathId) => ({ $ref: { file, pathId: String(pathId) } }); +const texture = (name) => ({ class: "Texture2D", name, files: { json: JSON.stringify({ m_Name: name, m_Width: 2, m_Height: 2, + m_MipCount: 1, m_ColorSpace: 1, m_TextureSettings: {} }) } }); +const KEY = [{ "data[0]": 1, "data[1]": 2, "data[2]": 3, "data[3]": 4 }, 21300000]; +const OTHER_KEY = [{ "data[0]": 9, "data[1]": 9, "data[2]": 9, "data[3]": 9 }, 21300001]; +const renderData = (texturePath, uvTransform, textureRect) => ({ + texture: ref(...texturePath), textureRect, textureRectOffset: { x: 0, y: 0 }, atlasRectOffset: { x: 0, y: 0 }, + uvTransform, downscaleMultiplier: 1, settingsRaw: 0, +}); +// a sprite packed by the SpriteAtlas "atlas:10" (two entries: the other key first, the sprite's key second) +const packedObjects = (spriteFields) => ({ + "pages:1": texture("page-a"), + "pages:2": texture("page-b"), + "atlas:10": { class: "SpriteAtlas", name: "atlas", files: { json: JSON.stringify({ m_Name: "atlas", m_RenderDataMap: [ + [OTHER_KEY, renderData(["pages", 2], { x: 1, y: 1, z: 1, w: 1 }, { x: 0, y: 0, width: 1, height: 1 })], + [KEY, renderData(["pages", 1], { x: 2, y: 0, z: 2, w: 0 }, { x: 0, y: 0, width: 2, height: 2 })], + ] }) } }, + "sprites:20": { class: "Sprite", name: "icon", files: { json: JSON.stringify({ sprite: { + m_Name: "icon", m_Rect: { x: 0, y: 0, width: 2, height: 2 }, m_Pivot: { x: 0.5, y: 0.5 }, + m_Border: { x: 0, y: 0, z: 0, w: 0 }, m_PixelsToUnits: 100, m_Offset: { x: 0, y: 0 }, + m_SpriteAtlas: ref("atlas", 10), m_RenderDataKey: KEY, ...spriteFields } }) } }, +}); + +test("sprite: a sprite packed by an atlas of another file is drawn from the atlas' render data", () => { + const s = new AssetStore({ objects: packedObjects({}) }).sprite("sprites:20"); + assert.equal(s.texture.id, "pages:1"); // the matching entry's texture, not the other entry's + assert.deepEqual(s.textureRect, { x: 0, y: 0, width: 2, height: 2 }); + assert.deepEqual(s.uvTransform, { x: 2, y: 0, z: 2, w: 0 }); + assert.equal(s.vertices, null); // no render mesh +}); + +test("sprite: a render data key missing from the atlas' map is an error", () => { + const store = new AssetStore({ objects: packedObjects({ m_RenderDataKey: [OTHER_KEY[0], 21300002] }) }); + assert.throws(() => store.sprite("sprites:20"), /not in the render data of atlas atlas/); +}); + +test("sprite: a sprite with neither render data nor an atlas is an error", () => { + const store = new AssetStore({ objects: packedObjects({ m_SpriteAtlas: null }) }); + assert.throws(() => store.sprite("sprites:20"), /no render data and no atlas/); +}); + +// a Mesh whose `arrays` file is `arrays` +const meshStore = (arrays) => new AssetStore({ objects: { + "meshes:3": { class: "Mesh", name: "quad", files: { json: "{}", arrays: JSON.stringify(arrays) } }, +} }); +const QUAD = [[0, 0, 0], [1, 0, 0], [0, 1, 0], [1, 1, 0]]; + +test("mesh: triangle submeshes are read flat, vertices are the positions, no skin without joints", () => { + const store = meshStore({ positions: QUAD, submeshes: [{ topology: "triangles", indices: [[0, 1, 2], [2, 1, 3]] }] }); + const m = store.mesh("meshes:3"); + assert.equal(m.name, "quad"); + assert.deepEqual(m.submeshes, [[0, 1, 2, 2, 1, 3]]); + assert.deepEqual(m.vertices, QUAD); + assert.equal(m.skin, null); // no joints + assert.equal(meshStore({ positions: QUAD, joints: null, submeshes: [] }).mesh("meshes:3").skin, null); + assert.equal(store.mesh("meshes:3"), m); // made once +}); + +test("mesh: a submesh of another topology is an error that names its index", () => { + const store = meshStore({ positions: QUAD, submeshes: [ + { topology: "triangles", indices: [[0, 1, 2]] }, + { topology: "lines", indices: [[0, 1]] }, + ] }); + assert.throws(() => store.mesh("meshes:3"), /submesh 1 is lines/); +}); diff --git a/tests/unit/live-legacy-fixture.mjs b/tests/unit/live-legacy-fixture.mjs index 21cd597..f26d25c 100644 --- a/tests/unit/live-legacy-fixture.mjs +++ b/tests/unit/live-legacy-fixture.mjs @@ -36,7 +36,7 @@ export const SCENE = { node("Stage", [{ type: "MonoBehaviour", class: "StageView", _judge: { gameObject: "Live/judge", component: "SpriteRenderer" }, _self: { gameObject: "Stage/mesh" } }]), node("Stage/mesh", [{ type: "MeshRenderer", m_Materials: [material("Unpacked", "Hidden/NotPacked")] }, - { type: "MeshFilter", m_Mesh: { mesh: "quad", vertices: [0, 0, 0, 1, 0, 0, 0, 1, 0], uv0: [0, 0, 1, 0, 0, 1], + { type: "MeshFilter", m_Mesh: { mesh: "quad", vertices: [[0, 0, 0], [1, 0, 0], [0, 1, 0]], uv0: [[0, 0], [1, 0], [0, 1]], submeshes: [[0, 1, 2]], m_LocalAABB: {} } }]), ] }, assets: { From f0637211e9d559c3358107682655bf913177e7dc Mon Sep 17 00:00:00 2001 From: nichinichisou Date: Thu, 15 Oct 2026 23:21:28 +0800 Subject: [PATCH 06/18] feat(tools)!: validate and test lives from their products validate-data checks a directory of live products: each lives/.json manifest, the parts its live.json names (each part once, with its errors reported at the first live that uses it), and the live read as the player reads it: the documents against the new live-scene, live-notes, live-noteskin, live-noteeffect, live-sounds and live-music schemas, the objects (Texture2D sizes and mip counts, the GLES3 programs of the shaders that materials draw with, mesh arrays), the score's lane count and the sound files. A chart of the site layout is read as a live through src/live/legacy.js and checked the same way. read-set records the object files a session reads as "#". The plan reads the prefabs of the None view and of each line view kind, as the spawns do, so plan and full mode read the same files. The data tests take a product layout as well as a chart site: a chart id names lives/.json, else charts/.json. docs/data-format.md describes products, lives and their parts, objects, shader objects and textures; the chart site format is described in its own section. schema/live.schema.json is the live.json of a live product, and the product name pattern accepts "live". Co-Authored-By: Claude Opus 5.5 --- CONTRIBUTING.md | 13 +- docs/data-format.md | 880 ++++++++++++++++++----------- docs/embedding.md | 2 +- docs/story-data-format.md | 4 +- schema/live-audio.schema.json | 100 ---- schema/live-music.schema.json | 153 +++++ schema/live-noteeffect.schema.json | 87 +++ schema/live-notes.schema.json | 235 ++++++++ schema/live-noteskin.schema.json | 87 +++ schema/live-scene.schema.json | 287 ++++++++++ schema/live-sounds.schema.json | 282 +++++++++ schema/live.schema.json | 79 ++- schema/product.schema.json | 2 +- scripts/read-set.mjs | 15 +- scripts/validate-data.mjs | 705 ++++++++++++++++------- src/live/legacy.js | 2 + src/live/noteview.js | 29 +- src/live/store.js | 15 +- tests/data/harness.mjs | 22 +- tests/data/read-set.test.mjs | 6 +- tests/unit/live-legacy-fixture.mjs | 35 +- tests/unit/read-set.test.mjs | 13 +- tests/unit/validate-live.test.mjs | 186 ++++++ 23 files changed, 2541 insertions(+), 698 deletions(-) delete mode 100644 schema/live-audio.schema.json create mode 100644 schema/live-music.schema.json create mode 100644 schema/live-noteeffect.schema.json create mode 100644 schema/live-notes.schema.json create mode 100644 schema/live-noteskin.schema.json create mode 100644 schema/live-scene.schema.json create mode 100644 schema/live-sounds.schema.json create mode 100644 tests/unit/validate-live.test.mjs diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 1da4e4f..fa148b3 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -35,10 +35,11 @@ schemas. - `tests/unit/`: `node --test`, synthetic inputs only. Test behaviour the code documents: an expected value should come from the definition the module states (an easing formula, Unity's matrix convention, a documented algorithm), not from the current output of the code. -- `tests/data/` (opt-in): runs the player in Node with a no-op WebGL2 context and a no-op AudioContext over a real - chart site, and checks that charts play to the end, that seeking, speed changes, pause and late frames keep the - chart state exact, and that a Live settings change during a chart gives the state of a session started with the - settings. It is skipped unless `OURNOTES_DATA` names a site directory: +- `tests/data/` (opt-in): runs the player in Node with a no-op WebGL2 context and a no-op AudioContext over real + data, the lives of a product layout and the charts of a chart site, and checks that charts play to the end, that + seeking, speed changes, pause and late frames keep the chart state exact, and that a Live settings change during a + chart gives the state of a session started with the settings. It is skipped unless `OURNOTES_DATA` names the data's + directory (a chart id names `lives/.json`, else `charts/.json`): ```sh OURNOTES_DATA=/path/to/site npm run test:data @@ -50,8 +51,8 @@ schemas. process for many charts) answers every chart in both modes as the chart's own process does; `OURNOTES_READSET_CHARTS=all OURNOTES_JOBS=` checks every chart of the site, `n` at a time. -- `npm run validate-data -- [chart id ...]` checks a site against the data format and the schemas in - `schema/`. +- `npm run validate-data -- [live id | chart id | model id ...]` checks a directory of products and chart + sites against the data format and the schemas in `schema/`. Rendering changes cannot be verified by the tests alone; describe how you checked them in the pull request. diff --git a/docs/data-format.md b/docs/data-format.md index 9e72140..5d98c7e 100644 --- a/docs/data-format.md +++ b/docs/data-format.md @@ -1,187 +1,85 @@ # Data format -The player does not ship any game data. It plays a chart from a set of **logical files** (paths such as -`live.json` or `livescene/scene.json`) that the embedding page supplies, usually as a static **site** served over -HTTP. This document describes that data: the site layout, the chart manifest, and every logical file as far as the -player reads it. Producing the data is outside this repository; the -[nnnotes](https://github.com/MetaSekaiLab/nnnotes) toolkit produces it from game files the user supplies. +The player does not ship any game data. It plays a live from the **products** of a product layout (a live and the +products it plays with, as the [nnnotes](https://github.com/MetaSekaiLab/nnnotes) toolkit makes them from game files +the user supplies) or from a chart of a **chart site** (the site layout of chart manifests), and shows Live2D models +from model products. The embedding page supplies the data, usually as static files served over HTTP. This document +describes that data as far as the player reads it: product manifests, their files and the objects of their object +tables, the documents of a live, Live2D models, and the chart site layout. -Machine-readable schemas are in [`schema/`](../schema), and `node scripts/validate-data.mjs [chart id ...]` -checks a whole site (see [Validation](#validation)). +Machine-readable schemas are in [`schema/`](../schema), and `node scripts/validate-data.mjs [id ...]` checks +a whole directory of products and chart sites (see [Validation](#validation)). Keys that are not described here are ignored by the player; producers may add their own. Where a structure comes from the game's Unity data (scene hierarchy, prefabs, materials, animation clips, particle systems), this document describes its shape and the conventions it follows rather than every serialized field. Contents: -[Site layout](#site-layout) · +[Products](#products) · [Lives](#lives) · -[charts.json](#chartsjson) · -[Chart manifest](#chart-manifest) · -[Logical files](#logical-files) · [live.json](#livejson) · [Score](#score) · -[Sounds](#sounds-audiolive-audiojson) · -[Live scene](#live-scene-livescenescenejson) · -[Note assets](#note-assets-livenotesnotesjson) · -[Shaders](#shaders) · +[Sounds](#sounds-soundsjson-musicjson) · +[Live scene](#live-scene-scenejson) · +[Note highway](#note-highway-notesjson-noteskinjson-noteeffectjson) · +[Bindings and shaders](#bindings-and-shaders) · +[Objects of a live](#objects-of-a-live) · +[Shader objects](#shader-objects) · [Textures](#textures) · [Live2D models](#live2d-models) · +[Chart sites](#chart-sites) · [Conventions](#conventions) · [Validation](#validation) -## Site layout +## Products -``` -/ - charts.json chart index (for listings; the player does not need it) - charts/_.json chart manifest, one per chart - charts//.json a region's own manifest of a chart (a site of several regions, see below) - assets/. file contents, content-addressed - assets/..gz the same, stored gzip-encoded (or .br: brotli-encoded) -``` - -- `` is the lowercase hex SHA-256 of the file's bytes (the decoded bytes of an encoded asset), `` the - extension of the logical file it holds (`json`, `glsl`, `png`, `flac`, `m4a`, …). Identical contents are stored - once and shared between charts, so assets can be served with long-lived, immutable caching. -- An asset may be stored encoded: gzip (`.gz`) or brotli (`.br`) after its extension, where that makes it smaller. - Only these kinds of file are encoded: `json` (split parts included), `glsl`, `moc3`, `atlas`, `skel`, `bin`, `wav`, - `glb`; PNG and audio files are stored as they are. The manifest entry gives both byte counts - ([File entries](#file-entries)); the player decodes an encoded asset itself unless the server already did (see - [docs/embedding.md](embedding.md#hosting-the-data)). -- Asset paths in a manifest are relative to the site root. When a page loads a manifest by URL, the player resolves - asset paths against the directory above the manifest's directory (`charts/..`) unless it is given another base - (see [docs/api.md](api.md)). -- Any static file server works. The page that embeds the player must be allowed to fetch the site (same origin, or - CORS headers on the site). - -A page can also hand the player the logical files directly, without a site; the files and their contents are the -same (see [docs/api.md](api.md)). - -## Lives - -A live is the product `live` of nnnotes' product layout and the products it plays with. Each has a product manifest -(`nnnotes.product/1`: `files` and the `objects` of its object table) at `/.json`, its files under -`assets/.`: +A product is one result of nnnotes' product layout: a live, a part a live plays with, or a Live2D model. Each has a +product manifest ([schema](../schema/product.schema.json)) at `/.json`, and its files are stored +content-addressed under `assets/`: ``` / - lives/_.json the live (live.json, the score) + lives/_.json a live (live) scenes/default.json the live scene (live.scene) notes/default.json the note highway (live.notes) noteskins/.json a note skin (live.noteskin) noteeffects/.json a note effect set (live.noteeffect) sounds/.json the note and live sounds (live.sounds) - music/.json the music (live.music) - assets/. + music/.json the music of a live (live.music) + models/.json a Live2D model (live2d.model) + assets/. file contents, content-addressed + assets/..gz the same, stored gzip-encoded (or .br: brotli-encoded) ``` -The player loads the live manifest, the manifests its `live.json` `parts` names (the defaults) and the manifests a page -gives as variants (the other note skins, note effect sets and note sound sets it offers), and puts each product's files -under its manifest path without `.json` (`lives/100001_expert/live.json`, `sounds/default/audio/LiveSe/.flac`). - -`live.json` (format 1), as far as the player reads it: - -| Key | Meaning | -|---|---| -| `format` | `1`. | -| `notes`, `notesMirror` | The score and the score converted with `MirrorChart`, files of the live ([Score](#score)). | -| `parts` | Part → manifest path of the product: `scene`, `notes`, `noteskin`, `noteeffect`, `sounds`, `music`. | - -`sounds.json` of `live.sounds` (format 1) holds the keys of [Sounds](#sounds-audiolive-audiojson) except `music`, with -the sounds of the note sound sets and the live sounds; `music.json` of `live.music` (format 1) holds `music` -(`{ soundId }`) and `sounds` with the music's cue. A layer's `file` is relative to its product's directory. The player -uses the `live.sounds` product that offers the most note sound sets (one with `NoteSePatternId=` holds the -default set and the others) together with the music. - -A chart manifest of the site layout ([Chart manifest](#chart-manifest)) is read as a live as well: its `live.json` -gives the score and its `audio/live-audio.json` the sounds and the music. - -## charts.json - -The index of the charts a site offers ([schema](../schema/charts.schema.json)). Chart listings read it; the player -does not. - -```json -{ "format": 2, "charts": [ { "id": "100001_expert", "musicId": 100001, "difficulty": "expert", - "manifest": "charts/100001_expert.json", "title": "…", "level": 25, "notes": 768, - "durationMs": 99989, "audio": true, "audioFormat": "aac", "flows": ["direct"], - "bytes": 42860517, "…": "…" } ] } -``` - -| Key | Type | Meaning | -|---|---|---| -| `format` | `2` | Version of this layout. | -| `charts[].id` | string | `_`. | -| `charts[].musicId`, `difficulty` | integer, string | The chart. Difficulties in the current data: `easy`, `normal`, `hard`, `expert`. | -| `charts[].manifest` | string | Path of the chart manifest, relative to the site root. | -| `charts[].title`, `bands`, `bandIds`, `stageBand`, `level`, `displayLevel`, `notes`, `fullComboCount`, `durationMs`, `sortOrder` | | The chart's facts; the same values as the manifest's `chart` object. | -| `charts[].audio`, `audioFormat`, `flows` | | Copies of the manifest keys of the same name. | -| `charts[].bytes` | integer | Sum of `size` over the manifest's `files` (the download size before shared assets are deduplicated). | -| `charts[].regions` | string[] | Optional. The regions the manifest serves (the manifest's `regions`). An entry without it serves every region. | -| `charts[].language`, `titles`, `bandNames` | | Optional. The language of `title` and `bands`, the title per language (`{ "": "…" }`), the band names per language (`{ "": ["…"] }`); the same values as the manifest's `chart` object. | -| `language` | string | Optional. The default language of a listing (the language of most entries' `title`). | -| `languages` | string[] | Optional. The languages the entries' `titles` are in. | -| `regions` | object[] | Optional. The regions the site serves, the first one the default: `id`, `name` (a label), `languages` (optional: the languages a listing offers in that region). | - -Languages are the game's text languages: `ja`, `en`, `zh-Hant`, `zh-Hans`, `ko`. - -### Several regions - -A site can serve several game regions. Where the chart data of two regions is the same, they share one manifest, -`charts/.json`, whose `regions` lists both; a region whose chart differs has its own manifest -`charts//.json`. So an id can appear in `charts.json` several times, but at most once per region, and an -entry without `regions` only once. A listing shows the entries whose `regions` include the chosen region (and those -without `regions`), with `titles[language]` and `bandNames[language]` where present, else `title` and `bands`; it -loads the manifest of the chosen region's entry. The chart list page of this repository -([examples/chart-list](../examples/chart-list)) switches with `?region=&lang=`. - -```json -{ "format": 2, "language": "zh-Hant", "languages": ["ja", "en", "zh-Hant", "zh-Hans", "ko"], - "regions": [ { "id": "tw", "name": "…", "languages": ["zh-Hant", "…"] }, { "id": "kr", "name": "…" } ], - "charts": [ { "id": "100001_expert", "manifest": "charts/100001_expert.json", "regions": ["tw", "kr"], - "title": "…", "titles": { "ja": "…", "en": "…", "…": "…" }, "bandNames": { "ja": ["…"], "…": "…" }, - "…": "…" } ] } -``` - -## Chart manifest - -`charts/_.json`, or `charts//_.json` on a site of -[several regions](#several-regions) ([schema](../schema/manifest.schema.json)), lists the logical files of one chart -and where their bytes are. +- `` is the lowercase hex SHA-256 of the file's bytes (the decoded bytes of an encoded asset), `` the + extension of the file it holds (`json`, `glsl`, `png`, `flac`, `m4a`, …). Identical contents are stored once and + shared between products, so assets can be served with long-lived, immutable caching. +- An asset may be stored encoded: gzip (`.gz`) or brotli (`.br`) after its extension, where that makes it smaller. + Only these kinds of file are encoded: `json` (split parts included), `glsl`, `moc3`, `atlas`, `skel`, `bin`, `wav`, + `glb`; PNG and audio files are stored as they are. The file entry gives both byte counts + ([File entries](#file-entries)); the player decodes an encoded asset itself unless the server already did (see + [docs/embedding.md](embedding.md#hosting-the-data)). +- Asset and manifest paths are relative to the root. When a page loads a manifest by URL, the player resolves them + against the directory above the manifest's directory unless it is given another base (see [docs/api.md](api.md)). +- Any static file server works. The page that embeds the player must be allowed to fetch the files (same origin, or + CORS headers). ```json -{ - "format": 3, - "musicId": 100001, - "difficulty": "expert", - "chart": { "title": "…", "level": 25, "notes": 768, "durationMs": 99989, "…": "…" }, - "audio": true, - "audioFormat": "aac", - "flows": ["direct"], - "quality": 1, - "files": { - "live.json": { "asset": "assets/5203…cba0.json.gz", "size": 466, "stored": 301 }, - "livescene/textures/lane_base-d14cea1c.png": { "asset": "assets/9b1e…07aa.png", "size": 181236 }, - "livenotes/notes.json": { "parts": [["settings", "assets/45be…f418.json.gz", 1691, 612], - ["prefabs", "assets/f367…e839.json.gz", 735095, 70118], "…"], "size": 4192212 }, - "…": "…" - } -} +{ "schema": "nnnotes.product/1", "product": "live.noteskin", "subject": "skin001", + "files": { "noteskin.json": { "asset": "assets/.json", "size": 1024 } }, + "objects": { + "CAB-…:7903…": { "class": "MonoBehaviour", "name": "skin001", + "files": { "json": { "asset": "assets/.json", "size": 4096 } } }, + "…": "…" } } ``` -| Key | Type | Read by the player | Meaning | -|---|---|---|---| -| `format` | `3` \| `2` | no | Version of this layout: 3, whose assets may be encoded; 2, the earlier version, every asset stored as it is. The player reads both. | -| `musicId`, `difficulty` | integer, string | no | The chart. | -| `chart` | object | yes | The chart's facts (title, bands, level, note count, duration, …; on a site of several languages also `language`, `titles`, `bandNames` as in [charts.json](#chartsjson)), handed to the page as they are (see [docs/api.md](api.md)). The player does not interpret them. | -| `regions` | string[] | no | Optional. The regions this manifest serves (see [Several regions](#several-regions)). | -| `audio` | boolean | yes | `false`: the chart has no waveform files. The player then plays no sound and runs the chart clock from game time, as with the music switched off. | -| `audioFormat` | `"aac"` \| `"flac"` | no | Format of the music file: AAC in an MP4 container (`.m4a`) or FLAC. Sound effects are FLAC. | -| `flows` | string[] | no | Start flows the files support. The player uses the direct start (the chart starts at the end of the live's intro timeline), `"direct"`, which every chart supports; other values name start sequences this player does not use. | -| `quality` | `0` \| `1` \| `2` | yes | The `LiveQuality` the files were prepared for (0 High, 1 Middle, 2 Low); the default quality of the player. The current data uses 1 (Middle, the game's default option). | -| `options` | object | yes | Optional. Live option name → the values the files support, for options that select files: `{"LiveQuality": [1, 2]}`. Absent: `LiveQuality` offers `quality` only. (The other file-selecting options are offered by the data itself: `live.json` `notesMirror`, `livenotes/notes.json` `settings.skins` / `settings.effects`, `audio/live-audio.json` `noteSe.groups`.) | -| `files` | object | yes | Logical path → file entry. | +| Key | Meaning | +|---|---| +| `schema` | `"nnnotes.product/1"`. | +| `product` | The product: `live`, `live.scene`, `live.notes`, `live.noteskin`, `live.noteeffect`, `live.sounds`, `live.music`, `live2d.model`. | +| `subject` | What the product is of: a chart (`100001/expert`), a skin or effect set name, a sound option set, a music id, `default` for a product shared by every live, a model's key. | +| `files` | The product's own files: logical path → [file entry](#file-entries). | +| `objects` | The object table: the [objects](#objects) the product uses, object id → its `class`, `name` and `files` by role, each a file entry. | ### File entries @@ -192,9 +90,10 @@ A file entry has one of these forms: `.br`). `size` is the byte count of the file, `stored` that of the asset as stored (the encoded bytes). An asset is encoded only when that makes it smaller, so `stored` is less than `size`; `stored` is present exactly when the asset name ends in `.gz` or `.br`. -- **Split JSON object**: `{ "parts": [[key, asset, size], ...], "size": }`; a part whose asset is encoded is - `[key, asset, size, stored]`. A large JSON file whose top level is an object is stored per top-level key: each - part's asset holds the raw JSON text of that key's value. The file's text is rebuilt as +- **Split JSON object** (a chart manifest's files only): `{ "parts": [[key, asset, size], ...], "size": }`; a + part whose asset is encoded is `[key, asset, size, stored]`. A large JSON file whose top level is an object is + stored per top-level key: each part's asset holds the raw JSON text of that key's value. The file's text is rebuilt + as ``` "{" + join(parts.map(([key, asset]) => JSON.stringify(key) + ":" + text(asset)), ",") + "}" @@ -215,44 +114,71 @@ Rules: `DecompressionStream("brotli")`); one whose fetched length is already `size` (the server sent it with `Content-Encoding`, and the browser decoded it) is used as it is; any other length fails the load. The file's byte length must then equal its `size`. -- The player fetches every listed file before the chart starts (several requests at a time), so a manifest should list - only what a chart needs: files the player does not read are fetched but unused. +- The player fetches every listed file before a live or a model starts (several requests at a time), so a manifest + should list only what is needed: files the player does not read are fetched but unused. -## Logical files +### Objects -The files the player reads, starting from `live.json`: +An object is one Unity object, converted. Its id is `:` (the path id as decimal digits, +possibly negative), the same in every product that uses it. An object's JSON names another object by a reference +`{ "$ref": { "file": "", "pathId": "" } }`, the object `:`; `null` names none. +A byte array field is `{ "$hex": "" }`. The roles of an object's files: -| Logical file | Content | +| Role | File | |---|---| -| `live.json` | Index: the paths of the four files below. | -| score, e.g. `score/0001_03.notes.json` | Notes and lines of the chart. | -| `audio/live-audio.json` | Sounds: cues, waveform layers, category volumes, ducking, sound ids. | -| audio files, e.g. `audio//.flac`, `.m4a` | Waveforms named by `audio/live-audio.json`. | -| `livescene/scene.json` | The live scene: cameras, lane, stage, background, UI canvas, post-processing settings. | -| `livescene/shaders/…`, `livescene/textures/…` | Shaders and textures of the scene. | -| `livenotes/notes.json` | Note views, note skin, hit effects, particles, judgement and combo UI, animation clips. | -| `livenotes/shaders/…`, `livenotes/textures/…` | Shaders and textures of the notes and effects. | +| `json` | The object as JSON: every object has it ([Objects of a live](#objects-of-a-live), [Live2D models](#model-manifest)). | +| `image` | A Texture2D's level 0 as a PNG ([Textures](#textures)). | +| `arrays` | A Mesh's vertex channels, submeshes and skin as JSON. | +| `index` | A Shader's variant index ([Shader objects](#shader-objects)). | +| `program:/` | A Shader's program of the variant whose `role` it is: GLSL ES 3.00 for `gles3`. | +| `blob:` | The bytes of a byte-array field: `blob:_bytes` of a CubismMoc is the moc3. | -The directory names `livescene/` and `livenotes/` are fixed: the player looks up shaders in `livescene/shaders/` and -`livenotes/shaders/`, and texture paths inside `livescene/scene.json` and `livenotes/notes.json` are relative to -their own directory. +## Lives + +A live is the product `live` of one chart and the products it plays with, its parts. Its `live.json` names the +manifest of each part: + +| Part | Product | Manifest | Document | +|---|---|---|---| +| — | `live` | `lives/_.json` | [`live.json`](#livejson), the [score](#score) files | +| `scene` | `live.scene` | `scenes/default.json` | [`scene.json`](#live-scene-scenejson) | +| `notes` | `live.notes` | `notes/default.json` | [`notes.json`](#note-highway-notesjson-noteskinjson-noteeffectjson) | +| `noteskin` | `live.noteskin` | `noteskins/.json` | `noteskin.json` | +| `noteeffect` | `live.noteeffect` | `noteeffects/.json` | `noteeffect.json` | +| `sounds` | `live.sounds` | `sounds/.json` | [`sounds.json`](#sounds-soundsjson-musicjson) | +| `music` | `live.music` | `music/.json` | `music.json` | + +A manifest's name is its subject with `/` as `_` (`lives/100001_expert.json` is the live of subject `100001/expert`). +The scene, the note highway, the skins, the effect sets and the sounds are shared by every live that plays them; the +music by the charts of one music. + +The player loads the live manifest, the manifests its `live.json` `parts` names (the defaults) and the manifests a page +gives as variants (the other note skins, note effect sets and note sound sets it offers), puts each product's files +under its manifest path without `.json` (`lives/100001_expert/live.json`, `sounds/default/audio/LiveSe/.flac`) +and merges their object tables into one (an object in several manifests must be the same in each). + +A chart manifest of a [chart site](#chart-sites) is read as a live as well. ## live.json -([schema](../schema/live.schema.json)) +([schema](../schema/live.schema.json)) The document of the live: format 1. ```json -{ "musicId": 100001, "difficulty": "expert", "scene": "livescene/scene.json", - "notes": "score/0001_03.notes.json", "noteAssets": "livenotes/notes.json", "liveAudio": "audio/live-audio.json" } +{ "format": 1, "musicId": 100001, "difficulty": "expert", "band": { "band": 1 }, "jacket": "CAB-…:-1723…", + "notes": "notes.json", "notesMirror": "notes.mirror.json", + "parts": { "scene": "scenes/default.json", "notes": "notes/default.json", "noteskin": "noteskins/skin001.json", + "noteeffect": "noteeffects/effect001.json", "sounds": "sounds/default.json", "music": "music/100001.json" } } ``` | Key | Meaning | |---|---| -| `scene` | Path of the live scene. | -| `notes` | Path of the score. | -| `notesMirror` | Optional. Path of the score converted mirrored (the game's `MirrorChart` option: lanes and flick directions mirrored while the score is converted), same format as `notes`. Without it the `MirrorChart` option is not offered. | -| `noteAssets` | Path of the note assets. | -| `liveAudio` | Path of the sound definitions. | +| `format` | `1`; the player refuses other values. | +| `musicId`, `difficulty` | The chart. | +| `band` | `{ band }`: the band whose start timeline and LightWeight background the live plays, a key of `scene.json` `bands`. | +| `jacket` | The jacket Sprite. | +| `notes` | The score, a file of the live ([Score](#score)). | +| `notesMirror` | Optional. The score converted mirrored (the game's `MirrorChart` option: lanes and flick directions mirrored while the score is converted), same format as `notes`. Without it the `MirrorChart` option is not offered. | +| `parts` | Part → the manifest path of the product, relative to the root ([Lives](#lives)). | ## Score @@ -310,20 +236,36 @@ With `MeasureLineDisplay` on, the player also reads `barLineTimeMs` (the chart t from which it finds the bar of each frame as the game does. Other keys of the score (fever and skill ranges, …) are not read. -## Sounds (`audio/live-audio.json`) +## Sounds (`sounds.json`, `music.json`) -([schema](../schema/live-audio.schema.json)) The sounds of one live, after the game's CRI ADX2 cue data: each sound -id names a cue made of one or more waveform layers, routed through CRI categories. +([sounds.json](../schema/live-sounds.schema.json), [music.json](../schema/live-music.schema.json)) The sounds of a +live, after the game's CRI ADX2 cue data: each sound id names a cue made of one or more waveform layers, routed +through CRI categories. `sounds.json` of `live.sounds` (format 1) holds the note and live sounds of an option set, +`music.json` of `live.music` (format 1) the music. The player uses the `live.sounds` product that offers the most note +sound sets (one of subject `NoteSePatternId=` holds the default set and the others) with the music; a sound id is +in one of the two only. + +`sounds.json`: | Key | Meaning | |---|---| +| `format` | `1`. | | `sounds` | Sound id (decimal string) → cue (below). | | `categories` | CRI category name → volume. A cue plays at the product of the volumes of its categories. `LiveBgmConfig`, `LiveSeConfig`, `LiveVoiceConfig` and `LiveNotesSeConfig` are the live volumes of a fresh profile (the game's local sound config after a fresh install, not the volume options' defaults); the volume options replace them once one of them is changed (a volume option whose category is absent is not offered). | | `react` | Ducking rules (REACT): `[{src, dest, level, decrementMs, incrementMs, holdMs}]`. While a cue of category `src` plays, category `dest` goes to `level` over `decrementMs`; after the last one ends it holds for `holdMs`, then returns to 1 over `incrementMs`. | -| `music` | `{ soundId }`: the music (BGM). | | `noteSe` | `{ types, volumes, mutes }`: `LiveNoteSeType` → sound id / volume / muted, for the default options. Optional: `patternId` (the note sound set of `types`, default 1) and `groups` (note sound set id → `{ LiveNoteSeType: sound id }`, every set the page may choose; their sounds must be in `sounds`), which offer the note sound options (`NoteSePatternId`, `UseIndividualNoteSe` and the per-type sounds). | | `liveSe` | `LiveSeType` → sound id. | | `voice` | Optional: `startVoiceSoundId`, `clearVoiceSoundId`, `fullComboVoiceSoundId`, `allPerfectVoiceSoundId` (absent or -1: none). | +| `timeline` | `startCheerDelaySec`: the delay of the start cheers after the start timeline begins. | + +`music.json`: + +| Key | Meaning | +|---|---| +| `format` | `1`. | +| `musicId` | The music. | +| `music` | `{ soundId }`: the music (BGM). | +| `sounds` | The music's cue by its sound id. | A cue (`sounds.`): @@ -340,7 +282,7 @@ A layer: | Key | Meaning | |---|---| -| `file` | Logical path of the waveform: FLAC, or AAC in MP4 (`.m4a`). | +| `file` | Path of the waveform, relative to its product's directory: FLAC, or AAC in MP4 (`.m4a`). | | `sampleRate` | 48000, the rate of the waveform file. `samples`, the loop points and `encoderDelay` count frames at this rate. Browsers decode into the rate of the `AudioContext` (the player's own runs at 48 kHz; a page may give one at another rate), and the player scales these counts to it. | | `channels` | Channel count. | | `samples` | Length of the waveform in sample frames, without encoder padding. The music's first layer defines the chart's length: `floor(samples × 1000 / sampleRate)` ms. | @@ -361,149 +303,153 @@ Types and sounds the player uses: LiveClearDirection). nnnotes writes the sounds of StartCheers (9), FinishCheers (10) and every finish direction (13–16) into `sounds`. The sounds of 10 and 16 must be in `sounds`; sounds of other types may be omitted, and the player then plays nothing for them. -- Every sound in `sounds` is decoded before the chart starts. +- Every sound of the two documents is decoded before the chart starts. -Other keys (cue sheet details, bus sends, timeline timings of the game's start sequence, …) are not read. +Other keys (cue sheet details, bus sends, the ACF, the names of the types, …) are not read. -## Live scene (`livescene/scene.json`) +## Live scene (`scene.json`) -The live scene after Unity serialization, plus the few assets and master data rows the player needs. It is usually -large and stored as a split JSON object. Keys read: +([schema](../schema/live-scene.schema.json)) The document of `live.scene` (format 1): what exists on the live screen +before the first note, and the start timeline of each band. It names objects of the product's object table: | Key | Content | |---|---| -| `scene` | `{ key, nodes }`: the live scene's GameObject hierarchy (a node list, below). | -| `assets.startTimeline` | The live's intro timeline prefab (node list). The direct start runs it to its end without drawing, so stage, lane and tap area start as the game leaves them when the music starts. | -| `assets.laneLinePrefab` | The lane line prefab (node list). | -| `laneSkin` | `{ lane_base, lane_tap_area, out_side_line }`: sprite descriptors of the lane skin. | -| `sprites` | `{ lightweightBackground, jacket }`: sprite descriptors of the LightWeight background and the music jacket. | -| `master.liveQualitySettings` | Rows of the game's live quality master data (`_quality`, `_effectRenderingScale`, …); the row of the active quality must exist. | -| `master.optionDefaultsPreset1` | Default option values: option name → `{ id, value }` (value as a string, as in the master data). | -| `postTextures.ForwardRendererLiveGameEffect.filmGrainTex` | Texture descriptors of the post-processing film grain textures. | - -Other keys are not read. - -### Node lists +| `scene` | The root GameObjects of the live scene; their prefabs are the scene's hierarchy (cameras, lane, stage, background, render canvas, live UI). A root references a component of another root with a `$ref`, which names the component inside that root's prefab. | +| `graphics.graphics` | The GraphicsSettings: its render pipeline asset, the asset's renderer data and their post-processing data (the film grain textures of the effect camera's renderer). `graphics.quality` and `graphics.player` (QualitySettings, PlayerSettings) are not read. | +| `laneSkins` | Lane skin name → `{ lane_base, lane_tap_area, out_side_line }`: Sprites. | +| `prefabs` | `laneLine` (the inner lane lines) and `judgeEffectView`: GameObjects. `barLineView`: the bar line view the scene's bar line container instantiates, a GameObject, or `null` (no `MeasureLineDisplay`). | +| `spriteAssets` | `combo` (LiveComboSpriteAsset) and `judgement` (LiveJudgementSpriteAsset): MonoBehaviours whose fields reference the digit and judgement Sprites. | +| `bands` | Band id → `{ timeline, background, tracks }`: the GameObject of the band's start timeline (its PlayableDirector), the LightWeight background Sprite, and per AnimationTrack (a MonoBehaviour id) `{ name, prefab, animator }`, the GameObject and the node path of the Animator the track plays on. The direct start runs the timeline to its end without drawing, so stage, lane and tap area start as the game leaves them when the music starts. | +| `settings` | `optionDefaults`: option name → default value as a string (option preset 1, as in the master data; `JudgePositionDisplay` and `NoteStartPosition` are required). `optionRanges`: option name → `[min, max]`. `qualitySettings`: the rows of the game's live quality master data (`_quality`, `_effectRenderingScale`, …), a row for each `LiveQuality` the live offers. `laneSkin`: the lane skin of the default options, a key of `laneSkins`. | +| `bindings`, `shaders` | [Bindings and shaders](#bindings-and-shaders). | -A scene, prefab or effect is a node list: `{ key, nodes: [node, ...] }`, nodes in hierarchy order (every parent -before its children). - -```json -{ "path": "Live/LiveGameView/Lane", "name": "Lane", "active": true, "layer": 0, - "localPosition": {"x": 0, "y": 0, "z": 0}, "localRotation": {"x": 0, "y": 0, "z": 0, "w": 1}, - "localScale": {"x": 1, "y": 1, "z": 1}, "components": [ … ] } -``` +Other keys (`renderSettings`, the lane skins by option value, the judgement sprite names, …) are not read. -- `path` is the `/`-joined chain of GameObject names from the root; it identifies the node and gives its parent. -- `components` are in GameObject order. Each has `type` (the Unity class, e.g. `Camera`, `MeshRenderer`, `Animator`, - `ParticleSystem`, or `MonoBehaviour`) and, for scripts, `class` (the script class, e.g. `Image`, `Canvas`, - `UIComboCounterView`), followed by the component's serialized fields under their Unity names. -- A reference to another component is `{ "component": , "gameObject": , "class"?: + ``` -A story: +Start with [installation and your first site](docs/getting-started.md), then read [embedding and hosting](docs/embedding.md). +Audio begins from a user gesture. Serve game data over HTTP(S). -```html - - - - -``` +## Components -A Live2D model: +| Content | Custom element | JavaScript module | API | +|---|---|---|---| +| Live charts | `` | `ournotes-player` | [ChartPlayer](docs/api.md) | +| Stories | `` | `ournotes-player/story` | [StoryPlayer](docs/story.md) | +| Live2D models | `` | `ournotes-player/live2d` | [ModelPlayer](docs/live2d.md) | +| Experimental UI preview | `` | `ournotes-player/ui` | [UIPlayer](docs/ui-preview.md) | -```html - - - -``` +Each module has an `/element` entry that registers its custom element. Browser bundles in `dist/` provide ESM, +element-registering ESM and classic scripts, with minified files and source maps. The classic script globals are +`OurnotesPlayer`, `OurnotesStory`, `OurnotesLive2D` and `OurnotesUI`. -JavaScript module: +Live2D requires Cubism Core. Story lip sync can require MotionSync Core; home conversations require the Spine +runtime. The embedding page loads these separately under their respective licenses before creating the player. +The [runtime setup](docs/getting-started.md#browser-runtimes) describes each dependency. -```js -import { ChartPlayer } from "ournotes-player"; +## Playback -const player = await ChartPlayer.create(document.getElementById("stage"), { - src: "https://example.org/site/charts/100001_expert.json", -}); -button.onclick = () => player.play(); // browsers start audio only from a user gesture -``` +- **Live charts:** auto-perfect playback, stage and note rendering, effects, judgment UI, audio, seek, speed and + supported Live settings. Simulation uses a fixed 60 Hz clock and float32 arithmetic. +- **Stories:** ADV commands, scene and camera effects, Live2D, localized text, video and audio. Home and afterlive + conversations use their host products. The language selector offers languages supplied by the product. +- **Live2D:** idle animation, expressions, blinking, breathing and physics using the model's shader data. +- **UI preview:** Canvas 2D layout and a subset of UI animation from `nnnotes export ui` products. -Example pages. Open them with any static server at the repository root: +See [fidelity](docs/fidelity.md), [story features](docs/story-features.md) and [simple stories](docs/story-simple.md) +for supported behavior. [Story authoring](docs/story-author.md) provides experimental tools for custom dialogue. -| Page | Contents | -|---|---| -| [examples/basic.html](examples/basic.html) | one `` | -| [examples/iframe.html](examples/iframe.html) | a page holding one player, ready to embed with an ` ``` -`allow="autoplay"` lets the frame start audio without a gesture inside it where the browser supports the permission. +Browser autoplay policy still applies. A player page hosted beside its data uses the same origin; an embedding +page can then place it in an iframe from another origin. ## Hosting the data -A site is a directory served over HTTP(S): +A site generated by nnnotes 0.3.x contains: +```text +charts.json, models.json, stories.json listings +charts/.json public chart entries +models/.json public model entries +stories/.json public story entries +products/.json immutable product manifests +assets/. shared file contents +assets/..gz gzip storage (.br for brotli) ``` -charts.json list of the charts (the chart-list example reads it) -charts/_.json one manifest per chart -models.json, models/.json the Live2D models (the model viewer and the stories read them) -stories.json, stories/.json the stories -assets/. every file of every chart, model and story, named by its content hash -assets/..gz the same, stored gzip-encoded (.br: brotli-encoded) -``` -A manifest lists the files of one chart and the asset that holds each one; the format is in -[data-format.md](data-format.md) (stories: [story-data-format.md](story-data-format.md)). The player fetches the -manifest, then every asset it lists, before the chart starts. A chart is about 40–45 MB; charts share most of their -assets (stage, notes, effects), so a browser that caches them downloads much less for the second chart. - -Serving notes: - -- **Paths.** Asset paths in a manifest are relative to the site root. For a chart or model manifest the player takes - the directory above the manifest's directory (`charts/..`, `models/..`). A story manifest of format `/2` names the - site root in its `root`, so `stories//.json` works too, and the model manifests it uses are found - from there; for one of format `/1` the player takes the directory above the manifest's directory. A manifest served - from somewhere else needs `AssetStore.fromManifest(url, { base })` (`loadStoryStore(url, { base })` for a story). -- **CORS.** A page on another origin than the site needs `Access-Control-Allow-Origin` (for example `*`) on the - manifests and the assets. No credentials are sent, and the requests are simple GETs (no preflight). -- **Encoded assets.** The JSON, GLSL and moc3 assets (and the other kinds listed in - [data-format.md](data-format.md#products)) are stored encoded where that makes them smaller: `.gz`, or `.br` in a - site built with brotli. The manifest gives each one's stored and decoded byte counts. Served as plain files, they - are decoded by the player with the browser's `DecompressionStream`: gzip in current browsers, brotli where - `DecompressionStream("brotli")` exists (Firefox and Safari; not Chromium-based browsers). The server may instead - send them with the matching `Content-Encoding` (`gzip` for `.gz`, `br` for `.br`) and a type of the logical file; - the browser then decodes them and the player takes them as they arrive. A brotli site that Chromium-based browsers - must play has to be served that way, with `Content-Encoding: br` on the `.br` assets. A server must not compress - the encoded assets again. -- **Compression of the other files.** Manifests, index files and plain `.json` / `.glsl` assets compress well with - the server's own `Content-Encoding`; the PNG and audio files are already compressed. The manifest's sizes are those - of the decoded files. -- **Caching.** Assets never change under their name: `Cache-Control: public, max-age=31536000, immutable`. Manifests - and the index files change when the site is rebuilt: a short max-age or revalidation. -- **Types.** `application/json`, `image/png`, `audio/flac`, `audio/mp4` (`.m4a`), `text/plain` (`.glsl`); an encoded - asset served as a plain file can be `application/octet-stream` (`application/gzip` for `.gz`). +Regional aliases can add a region segment to entry paths. Follow the paths in the listings. Products resolve their +dependencies through immutable manifests; standalone layouts also use the paths in their product references. +The [data format](data-format.md) and [story format](story-data-format.md) describe these contracts. + +| Concern | Configuration | +|---|---| +| Paths | Preserve the directory layout and resolve product files against the manifest's `root`. Format 2 chart/model sites use their documented site-relative paths. | +| CORS | For cross-origin playback, allow the page origin on every manifest and asset. Public static data can use `Access-Control-Allow-Origin: *`. | +| Immutable caching | Content-addressed assets and `products/.json` can use `Cache-Control: public, max-age=31536000, immutable`. | +| Mutable caching | Indexes, public entry aliases and HTML pages use revalidation or a short cache lifetime. | +| Publication | Upload assets and immutable products first, then public entries/pages and indexes. Retire unused files after successful publication. | + +### Encoded assets + +Manifest entries record decoded sizes and, for encoded assets, stored sizes. Plain `.gz` files can be decoded by +the player's `DecompressionStream`. A server may instead send them with `Content-Encoding: gzip` and the logical +content type. `.br` files require `Content-Encoding: br` for Chromium playback; other browsers may also support +client-side brotli decoding. Apply one encoding layer to each stored file. + +Use `application/json` for JSON, `image/png` for PNG, `audio/flac` for FLAC, `audio/mp4` for M4A and `text/plain` +for GLSL. An encoded file served without `Content-Encoding` can use `application/octet-stream` or +`application/gzip`. Mutable JSON and HTML can use ordinary server-side compression. + +### Compatibility + +ournotes-player 0.2.x reads nnnotes 0.3.x story/UI products and format 2 chart/model sites. For a manifest served +outside its normal root, use the relevant loader's `base` option as described in the API reference. +Run `node node_modules/ournotes-player/scripts/validate-data.mjs path/to/site` before serving a generated directory. ## Browser requirements -WebGL2, WebAudio, `ResizeObserver`, ES2022 modules; custom elements for the element. The audio files are AAC (MP4) and -FLAC; current desktop and mobile browsers decode both. +Use a browser with WebGL2, WebAudio, ES2022 modules, ResizeObserver and Custom Elements. It must support the audio +and video encodings selected during export, such as AAC/FLAC audio and VP9/Opus WebM video. Audio starts from a +user gesture. Load the required Cubism, MotionSync or Spine runtime before the corresponding component. ## Mobile -The player works at phone width (the control bar compacts below 480 px) and with touch: a tap shows or hides the -control bar, the buttons and the seek bar take taps. Audio starts on the first tap of the play button. The drawing -buffer follows `devicePixelRatio`; on a slow device pass `pixelRatio: 1` (or a lower value) to `ChartPlayer.create`. -A story draws at the screen the game renders at for its quality (at most 1920 pixels on the longer side at Best and -High, 1440 at Middle) rather than at every device pixel ([story.md](story.md), `resolution`); on a slow phone -`quality="middle"` also turns off what the game turns off at Middle (the character and background blur, the stage post -effect, Unity lighting, the breath motion, FXAA). +Controls support touch and adapt to narrow layouts. Live drawing follows `devicePixelRatio`; set `pixelRatio: 1` +in `ChartPlayer.create` to reduce the drawing buffer on a constrained device. Story `quality="middle"` selects +the corresponding reduced effects and resolution; see [story options](story.md). ## Several players on one page -Each player has its own WebGL context, `AudioContext` and state; players do not interfere. Browsers limit the number of -live WebGL contexts per page (16 in Chromium), and every loaded chart holds its files in memory: dispose players that -are no longer shown. +Each player owns a WebGL context, audio context and playback state. Browser context and memory limits apply; +dispose players that are no longer shown. See [performance](performance.md) for a repeatable profiling method. diff --git a/docs/getting-started.md b/docs/getting-started.md new file mode 100644 index 0000000..4970118 --- /dev/null +++ b/docs/getting-started.md @@ -0,0 +1,63 @@ +# Getting started + +ournotes-player 0.2.x consumes nnnotes 0.3.x products and format 2 chart/model sites. Use Node.js 22+ for package +tools. Browser playback requires WebGL2, WebAudio and the audio/video codecs used by your data. + +## Install + +```sh +npm install ournotes-player@0.2.0 +``` + +The npm package includes source modules, types, schemas, browser bundles and site tools. Use its directory, +`node_modules/ournotes-player`, as nnnotes' `--player` argument. The browser ZIP in a GitHub release is useful for +embedding bundles; a site build needs the complete package, including scripts, metadata and example pages. + +## Prepare a site + +Configure [nnnotes](https://metasekailab.github.io/nnnotes/quickstart/) with your own input files first: + +```sh +pip install 'nnnotes[fonts]==0.3.0' +nnnotes site build --name example -o out/site --pair 100001:expert --fonts game --player node_modules/ournotes-player +node node_modules/ournotes-player/scripts/validate-data.mjs out/site +python -m http.server 8000 --directory out/site +``` + +Open `http://localhost:8000/` for charts. Add `--story ID` to select a story, or `--live2d KEY` for a model. +The respective pages are `story/` and `live2d/`. Use IDs available in the configured catalog and master data. + +Site builds reuse the named selection's depot results. Keep its indexes, manifests and assets together when +hosting it. See [embedding](embedding.md#hosting-the-data) for paths, caching and compression headers. + +## Browser runtimes + +| Feature | Runtime loaded by the page | +|---|---| +| Live chart playback | Browser WebGL2 and WebAudio | +| Live2D models and story characters | Live2D Cubism Core | +| Models with MotionSync lip sync | Live2D MotionSync Core, in addition to Cubism Core | +| Home conversations | Spine runtime, in addition to the story's other requirements | + +Obtain these runtimes under their respective licenses and load them before creating the relevant player. +The example story page accepts `?core=URL&motionsync=URL&spine=URL`; pass the URLs needed by that story. +The model page accepts `?core=URL`. These files are provided separately from the player package. + +## Embed a chart + +```html + + +``` + +For stories, load the required runtime scripts and `ournotes-player.story.element.min.js`, then use +``. +For models, load Cubism Core and `ournotes-player.live2d.element.min.js`, then use ``. +Use a language declared by the story product. The [embedding guide](embedding.md) covers JavaScript and iframe APIs. + +## Validate and diagnose + +Run the validator against the complete exported directory. It checks manifests, object references, asset sizes, +chart facts and declared languages. In a browser, inspect failed network requests, required runtimes and decode +errors. For a cross-origin page, return CORS headers on every manifest and asset. Chromium playback of `.br` +assets requires `Content-Encoding: br`. diff --git a/docs/live2d.md b/docs/live2d.md index 6bffb70..23a9541 100644 --- a/docs/live2d.md +++ b/docs/live2d.md @@ -35,7 +35,7 @@ model viewer. ```html - + diff --git a/docs/performance.md b/docs/performance.md index f86a4c4..89b6f0b 100644 --- a/docs/performance.md +++ b/docs/performance.md @@ -1,131 +1,31 @@ # Performance -What a frame of the player costs on the CPU, which of that work the browser does in its own compiled code, and how -much is the player's JavaScript. The numbers below were measured with the version of the player this file ships -with, on the machine and charts listed; other versions, browsers, machines and charts give other numbers. They are -rounded. +Measure the player on the browser and device that will display it. Rendering cost depends on the product, +quality settings, canvas resolution, runtime dependencies and the number of active players. -The browser drew with a software rasterizer, so this page covers the CPU side only: the time the GPU (or the -rasterizer) takes to draw a frame is not measured. +## Profile a repeatable workload -## Conditions +1. Record the player version, product manifest, browser version, viewport, device pixel ratio and playback settings. +2. Load the same product once to separate initial transfer/decode work from steady playback. +3. Use browser performance tools to record a fixed playback interval. Include scenes with the effects used by + your content, and exercise seeking separately from continuous playback. +4. Inspect main-thread work, frame duration, WebGL activity and memory. Use the network panel to distinguish + transfer time from asset decoding and rendering. +5. Repeat with a cold cache when assessing first-load behavior. Compare equal viewports and settings. -| | | -|---|---| -| Browser | Chromium 154.0.8037.57, headless, on Linux (Debian 12) | -| Graphics | ANGLE on SwiftShader: WebGL2 rendered in software, in the browser's GPU process | -| Machine | 64 logical CPUs (AMD EPYC 9K65) | -| Canvas | 320 × 180 CSS pixels at a device pixel ratio of 1, so that the software rasterizer keeps up with 60 frames per second (at 1280 × 720 it draws about 8 frames per second and the main thread waits for it). The player makes the same draw calls at any canvas size (only values such as the screen size differ); at larger sizes the rasterizer's work grows | -| Player | `dist/ournotes-player.js`, `ChartPlayer` with `controls: false` and `seed: 1`; music and sound effects on, the manifest's quality | -| Charts | from a site built by nnnotes for the Taiwan server, version 1.0.1: 100015 expert (1 874 notes; of that region's 336 charts the one with the most notes and the most WebGL calls per frame), 100040 expert (1 161 notes; with its effects the second most WebGL calls per frame), 100001 expert (822 notes) | -| Span | 60 s of playback, starting 5 s after `play()` | +Chart simulation advances at 60 Hz. Seeking resimulates state, so its workload differs from drawing a frame. +Stories can also spend time preparing glyph pages, model resources and media when loading or changing language. -Each chart was played twice over the same span: once with a Chrome trace of the renderer's main thread (the thread -time of every task, summed per animation frame) and `performance.now()` around the session's simulation phases and -its `render()`; once under the V8 CPU profiler (100 µs sampling) for the split by kind of work. +## Control resource use -## Per-frame CPU time in the browser +- Set the Live `pixelRatio` option to bound drawing resolution on high-density displays. +- Choose the story quality appropriate to the device; [story options](story.md) describe its rendering effects. +- Cache content-addressed assets with immutable cache headers so related products reuse shared resources. +- Dispose hidden or removed players to release their WebGL contexts and audio resources. +- Profile a single player before adding several concurrent instances. -Main-thread time per animation frame, against the 16.7 ms of a 60 Hz frame: +## Check behavior after a change -| Chart | Frames per second | Median | 95th percentile | 99th percentile | Frames over 16.7 ms | -|---|---|---|---|---|---| -| 100015 expert | 57 | 3.5 ms | 6.1 ms | 8.9 ms | 14 of 3 428 (0.4 %) | -| 100040 expert | 58 | 3.7 ms | 6.1 ms | 11.9 ms | 21 of 3 471 (0.6 %) | -| 100001 expert | 60 | 2.7 ms | 4.4 ms | 8.6 ms | 15 of 3 578 (0.4 %) | - -Headless Chromium produced slightly fewer than 60 animation frames per second here; the player then runs the steps -that are due (1.00 to 1.05 steps per animation frame in these runs) and draws once per animation frame. - -| Chart | Simulation, per step | `render()`, per draw: JavaScript and WebGL calls | -|---|---|---| -| 100015 expert | 0.5 ms | 2.5 ms | -| 100040 expert | 0.6 ms | 2.6 ms | -| 100001 expert | 0.4 ms | 2.0 ms | - -The main thread's busy time by kind of work (CPU profile): - -| | 100015 expert | 100040 expert | 100001 expert | -|---|---|---|---| -| The player's JavaScript | 65 % | 64 % | 62 % | -| WebGL calls: the browser's validation and command encoding on the main thread | 16 % | 16 % | 17 % | -| Browser code outside JavaScript: task scheduling, frame production | 10 % | 12 % | 12 % | -| JavaScript garbage collector | 6 % | 5 % | 6 % | -| Other browser APIs the player calls: `MessageChannel` (the order of the loop's phases), timers, audio nodes | 3 % | 3 % | 3 % | - -The measurement's own code took under 1 %. Within the player's JavaScript, the particle systems take about a -quarter, looking up and setting the shaders' uniforms and render state for each draw call about a sixth, the note -views about a tenth, and the chart simulation itself (note states, judgements, combo) about 1 %. - -## What runs in the browser's compiled code - -- **WebGL2.** The main thread validates each call and encodes it into the browser's command buffer (the WebGL share - above). The GPU process executes the draw calls and compiles the game's GLSL ES 3.00 shaders, on the GPU or, as - here, in the software rasterizer; neither runs on the main thread. -- **WebAudio.** `decodeAudioData` decodes the FLAC and AAC files (at the context's sample rate) when the chart - loads; mixing and output run on the browser's audio thread. On the main thread the player only creates and - schedules sources and gains (under 1 % of its busy time). -- **Images.** `createImageBitmap` decodes the PNG textures when the chart loads or a texture is first used. -- **Live2D.** Cubism Core, which the page supplies, is compiled code. -- **JavaScript.** V8 compiles the player's frequently run code to machine code. - -## The JavaScript share - -The player's JavaScript and its garbage collection take about two thirds of the main thread's busy time: at the -median, 1.9 to 2.6 ms per frame. An implementation of that work that took no time at all would save at most that -much per frame; the WebGL calls and the browser's own work would stay. At the median the frames already leave -13 ms or more of the 16.7 ms unused. - -## Live2D model viewer - -`ModelPlayer` with the model `adv_live2d_rana_003_casual_spring_01` on a 720 × 720 canvas, measured over 60 s from -5 s after `play()`, under the same conditions otherwise: 60 animation frames per second and 30 model steps per -second (idle motion, eye blink, breath, physics). A frame that steps and draws the model takes about 5 ms of -main-thread time (99th percentile 9 ms; 3 frames of 3 599 over 16.7 ms); a frame without a step about 1 ms. Per -step, the model update (motions, physics and the Cubism Core update) takes 1.8 ms and the draw 3.3 ms. - -| The main thread's busy time | | -|---|---| -| The player's JavaScript | 41 % | -| WebGL calls | 23 % | -| Cubism Core | 22 % | -| Browser code outside JavaScript | 10 % | -| JavaScript garbage collector | 2 % | -| Other browser APIs | 2 % | - -## Node.js without a browser - -`scripts/read-set.mjs` in full mode runs a whole chart in Node.js with a WebGL2 context and an AudioContext that do -nothing: every frame is simulated and its draw calls are made, nothing is drawn or played. This measures the -player's own work without any WebGL implementation. Per step, one process at a time on the same machine, Node.js -22.23.3, two runs per chart: - -| Chart | Frames | Step: median | 95th percentile | 99th percentile | Of the step, draw submission (median) | `read-set.mjs` full mode, load and every frame | -|---|---|---|---|---|---|---| -| 100015 expert | 10 462 | 1.6 ms | 2.5 to 2.7 ms | 3.4 to 3.7 ms | 1.1 ms | 18 s | -| 100040 expert | 6 678 | 1.6 to 1.7 ms | 2.7 to 3.0 ms | 4.6 to 5.4 ms | 1.1 to 1.2 ms | 12 s | -| 100001 expert | 6 001 | 1.3 ms | 2.0 ms | 2.4 ms | 0.9 ms | 8 s | - -Over the region's 336 charts, the median chart makes about 2 900 WebGL calls and 88 draw calls per frame; 100015 -expert makes the most, 5 530 and 166. - -## Reproducing the measurement - -1. Build the player (`npm ci`, `npm run build`) and serve a site and the player's `dist/` from one origin. For - `performance.now()` at its finest resolution, serve the pages cross-origin isolated - (`Cross-Origin-Opener-Policy: same-origin`, `Cross-Origin-Embedder-Policy: require-corp`). -2. Start Chromium, for example headless through the DevTools protocol, with - `--autoplay-policy=no-user-gesture-required` (and, without a GPU, `--use-angle=swiftshader - --enable-unsafe-swiftshader`). On a page with a 320 × 180 element, create a `ChartPlayer` with - `controls: false` and `seed: 1`, call `play()`, wait 5 s, then measure 60 s between two `performance.mark()`s. -3. Per frame: record a trace (`Tracing.start` with the categories `toplevel`, `devtools.timeline` and - `blink.user_timing`) and sum the thread time (`tdur`) of the main thread's top-level tasks from one - `FireAnimationFrame` event to the next. -4. Split: over a second playback of the same span, run the CPU profiler (`Profiler.setSamplingInterval` with 100 µs, - `Profiler.start`, `Profiler.stop`) and sum each node's self time: the player's bundle (by the source module each - `// src/...` section of `dist/ournotes-player.js` names), the methods of `WebGL2RenderingContext`, - `(garbage collector)`, `(program)` and `(idle)`. -5. Node.js: `node scripts/read-set.mjs /charts/.json` prints the frames and seconds of a full run. For - per-step times, run the same loop yourself (`ChartSession.create` with `headlessGL()` and `HeadlessAudioContext` - from `scripts/lib/headless.mjs`, `setMusic(false)`, `play()`, then `step()` until the state is `"ended"`) with - `performance.now()` around each step. +Run the synthetic tests and package contracts, then replay the same workload with the same data and settings. +Compare completion, seeking, audio synchronization, selected language and visible output as well as frame cost. +[Embedding](embedding.md) covers cache and compression headers; [fidelity](fidelity.md) describes playback semantics.