diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 1932b8e..5f4dbe6 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -4,7 +4,6 @@ name: ci # step when either changes. on: pull_request: - branches: [main] push: branches: [main] diff --git a/.github/workflows/commitlint.yml b/.github/workflows/commitlint.yml index 7b0e22a..6d922d6 100644 --- a/.github/workflows/commitlint.yml +++ b/.github/workflows/commitlint.yml @@ -3,7 +3,6 @@ name: lint-commits # Enforces conventional commit titles so git-cliff can drive the release notes. See cliff.toml for the type groups. on: pull_request: - branches: [main] push: branches: [main] diff --git a/CHANGELOG.md b/CHANGELOG.md index c8a4c0e..0a46d05 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,44 @@ All notable changes to this package. Versions follow [Semantic Versioning](https://semver.org/). +## [0.2.0](https://github.com/empty-sekai/ournotes-player/compare/v0.1.8...v0.2.0) - 2026-10-19 + +### Product loading and compatibility + +- Read shared object resources from `nnnotes.product/1` manifests through typed texture, material, mesh, prefab, animation and shader accessors. +- Play Live2D model format 3, live products and UI packs through the shared object layer. +- Read format 2 chart and Live2D model sites through the same rendering interfaces. +- Story and UI inputs use the product formats generated by nnnotes 0.3.x. +- Route story cue volumes by CRI category names while retaining sound-row stop and crossfade behavior. + +### Data validation + +- Check product chart listings against manifest metadata and the complete unique asset closure. +- Validate chat status text values in story UI documents. +- Validate prefab hierarchy documents against their published transform and component schema. +- Describe room, spot, sound, video and CRI Lips documents, and accept null localization on native text records. + +### Site products + +- Select regional model aliases by region and localize standalone chart band names. +- Expose chart metadata for standalone live products and site aliases. +- Load live variants from entry manifests and accept variant paths in read-set requests. +- Validate product chart aliases, language-indexed UI references and offered variants. +- Honor audio-disabled story products and select an available playback language from story listings. + +### Story products + +- Story loading, playback and authoring use `nnnotes.product/1` manifests with shared object resources and language-specific UI and fonts. +- Home and live-result stories resolve their host prefabs, room geometry, Spine materials and animation bindings by object identity. +- Data validation checks the story product graph, font bindings, shader variants, audio files and model dependencies. `--lang` selects an explicit language scope. +- Node story loading accepts a manifest or a product layout containing exactly one story. + +### Bug fixes + +- Keep Spine index uploads from changing another renderer's vertex array. +- Keep model and shared-object validation caches independent. +- Clear loading text after language changes and display configured audio volumes in the controls. + ## [0.1.8](https://github.com/empty-sekai/ournotes-player/compare/v0.1.7..v0.1.8) — 2026-10-16 ### Features @@ -174,6 +212,3 @@ All notable changes to this package. Versions follow [Semantic Versioning](https - install from GitHub until the package is on npm - add performance.md: the CPU cost of a frame - a file holds each controller and clip once; the validator checks every reference _(story)_ - - - diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 1da4e4f..fa148b3 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -35,10 +35,11 @@ schemas. - `tests/unit/`: `node --test`, synthetic inputs only. Test behaviour the code documents: an expected value should come from the definition the module states (an easing formula, Unity's matrix convention, a documented algorithm), not from the current output of the code. -- `tests/data/` (opt-in): runs the player in Node with a no-op WebGL2 context and a no-op AudioContext over a real - chart site, and checks that charts play to the end, that seeking, speed changes, pause and late frames keep the - chart state exact, and that a Live settings change during a chart gives the state of a session started with the - settings. It is skipped unless `OURNOTES_DATA` names a site directory: +- `tests/data/` (opt-in): runs the player in Node with a no-op WebGL2 context and a no-op AudioContext over real + data, the lives of a product layout and the charts of a chart site, and checks that charts play to the end, that + seeking, speed changes, pause and late frames keep the chart state exact, and that a Live settings change during a + chart gives the state of a session started with the settings. It is skipped unless `OURNOTES_DATA` names the data's + directory (a chart id names `lives/.json`, else `charts/.json`): ```sh OURNOTES_DATA=/path/to/site npm run test:data @@ -50,8 +51,8 @@ schemas. process for many charts) answers every chart in both modes as the chart's own process does; `OURNOTES_READSET_CHARTS=all OURNOTES_JOBS=` checks every chart of the site, `n` at a time. -- `npm run validate-data -- [chart id ...]` checks a site against the data format and the schemas in - `schema/`. +- `npm run validate-data -- [live id | chart id | model id ...]` checks a directory of products and chart + sites against the data format and the schemas in `schema/`. Rendering changes cannot be verified by the tests alone; describe how you checked them in the pull request. diff --git a/README.en.md b/README.en.md index fba517f..9cc8715 100644 --- a/README.en.md +++ b/README.en.md @@ -2,249 +2,113 @@ [简体中文](README.md) | [English](README.en.md) -ournotes-player reproduces three kinds of BanG Dream! Our Notes screens in a web page: the auto play of a live chart, -story episodes (ADV, home spot talks and live result talks included), and the Live2D characters. It draws with the -game's own shaders, running the GLSL ES 3.00 programs that Unity compiled from the data in WebGL2. Sound goes -through WebAudio. Each of the three can be embedded in other pages as a custom element, as a JavaScript module or in -an iframe. - -This is an unofficial fan project, not affiliated with the game's developer, publisher or operator. The repository -and the npm package contain no game assets: the player reads data prepared in the data formats -([live and Live2D](docs/data-format.md), [stories](docs/story-data-format.md)), supplied by the user; the -[nnnotes](https://github.com/MetaSekaiLab/nnnotes) toolkit produces that data from the user's own game files. -BanG Dream! and related names and trademarks belong to their respective owners. - -An independent experimental [UI prefab preview](docs/ui-preview.md) is available through `ournotes-player/ui` -and ``. It inspects `nnnotes ui` exports with Canvas 2D and a motion subset, without pixel-fidelity or -full game-runtime claims. The [example viewer](examples/ui) loads a library supplied by the user. +Play BanG Dream! Our Notes live charts, story episodes and Live2D models in a browser. The player renders the +data's game shaders with WebGL2 and plays sound with WebAudio. Embed a custom element, use the JavaScript API, +or host a player page in an iframe. -## Components - -| | Live charts | Stories | Live2D models | -|---|---|---|---| -| Custom element | `` | `` | `` | -| Modules | `ournotes-player`, `ournotes-player/element` | `ournotes-player/story`, `ournotes-player/story/element` | `ournotes-player/live2d`, `ournotes-player/live2d/element` | -| Main class | `ChartPlayer` | `StoryPlayer` | `ModelPlayer` | -| Browser bundles (`dist/`) | `ournotes-player.*` | `ournotes-player.story.*` | `ournotes-player.live2d.*` | -| The page loads | — | Live2D Cubism Core; the MotionSync Core for lip sync; a Spine runtime for home spot talks | Live2D Cubism Core | -| Docs | [api.md](docs/api.md) | [story.md](docs/story.md) | [live2d.md](docs/live2d.md) | - -Importing a `…/element` module defines its custom element; it exports the same API as the module without -`/element`. Every browser bundle comes in three forms: the ESM API, an ESM module that defines the element, and a -classic script that defines the element and exposes a global (`OurnotesPlayer`, `OurnotesStory`, `OurnotesLive2D`). -Each form also comes as `.min.js`, with source maps. The live chart entry points do not include the story and -Live2D code. - -Live2D Cubism Core, the MotionSync Core and the Spine runtime belong to their owners and come under their own -licenses. The repository, the npm package and the bundles do not include them; the page loads them before it creates -a player. +Version **0.2.0** reads shared object products generated by **nnnotes 0.3.x**. It also supports format 2 chart and +Live2D model sites. Game data is supplied by each user; [nnnotes](https://github.com/MetaSekaiLab/nnnotes) generates +products and static sites from that user's game files. -## Quick start +## Install and embed ```sh -npm install ournotes-player +npm install ournotes-player@0.2.0 ``` -Installing from GitHub (`npm install github:empty-sekai/ournotes-player`) builds `dist/` during the install. - -A live chart. With a bundler, use `import "ournotes-player/element";`. Without one, load the bundle from a CDN: +With a bundler, import `ournotes-player/element`. For a plain HTML page: ```html - + ``` -A story: +Start with [installation and your first site](docs/getting-started.md), then read [embedding and hosting](docs/embedding.md). +Audio begins from a user gesture. Serve game data over HTTP(S). -```html - - - - -``` +## Components -A Live2D model: +| Content | Custom element | JavaScript module | API | +|---|---|---|---| +| Live charts | `` | `ournotes-player` | [ChartPlayer](docs/api.md) | +| Stories | `` | `ournotes-player/story` | [StoryPlayer](docs/story.md) | +| Live2D models | `` | `ournotes-player/live2d` | [ModelPlayer](docs/live2d.md) | +| Experimental UI preview | `` | `ournotes-player/ui` | [UIPlayer](docs/ui-preview.md) | -```html - - - -``` +Each module has an `/element` entry that registers its custom element. Browser bundles in `dist/` provide ESM, +element-registering ESM and classic scripts, with minified files and source maps. The classic script globals are +`OurnotesPlayer`, `OurnotesStory`, `OurnotesLive2D` and `OurnotesUI`. -JavaScript module: +Live2D requires Cubism Core. Story lip sync can require MotionSync Core; home conversations require the Spine +runtime. The embedding page loads these separately under their respective licenses before creating the player. +The [runtime setup](docs/getting-started.md#browser-runtimes) describes each dependency. -```js -import { ChartPlayer } from "ournotes-player"; +## Playback -const player = await ChartPlayer.create(document.getElementById("stage"), { - src: "https://example.org/site/charts/100001_expert.json", -}); -button.onclick = () => player.play(); // browsers start audio only from a user gesture -``` +- **Live charts:** auto-perfect playback, stage and note rendering, effects, judgment UI, audio, seek, speed and + supported Live settings. Simulation uses a fixed 60 Hz clock and float32 arithmetic. +- **Stories:** ADV commands, scene and camera effects, Live2D, localized text, video and audio. Home and afterlive + conversations use their host products. The language selector offers languages supplied by the product. +- **Live2D:** idle animation, expressions, blinking, breathing and physics using the model's shader data. +- **UI preview:** Canvas 2D layout and a subset of UI animation from `nnnotes export ui` products. -Example pages. Open them with any static server at the repository root: +See [fidelity](docs/fidelity.md), [story features](docs/story-features.md) and [simple stories](docs/story-simple.md) +for supported behavior. [Story authoring](docs/story-author.md) provides experimental tools for custom dialogue. -| Page | Contents | -|---|---| -| [examples/basic.html](examples/basic.html) | one `` | -| [examples/iframe.html](examples/iframe.html) | a page holding one player, ready to embed with an ` ``` -`allow="autoplay"` lets the frame start audio without a gesture inside it where the browser supports the permission. +Browser autoplay policy still applies. A player page hosted beside its data uses the same origin; an embedding +page can then place it in an iframe from another origin. ## Hosting the data -A site is a directory served over HTTP(S): +A site generated by nnnotes 0.3.x contains: +```text +charts.json, models.json, stories.json listings +charts/.json public chart entries +models/.json public model entries +stories/.json public story entries +products/.json immutable product manifests +assets/. shared file contents +assets/..gz gzip storage (.br for brotli) ``` -charts.json list of the charts (the chart-list example reads it) -charts/_.json one manifest per chart -models.json, models/.json the Live2D models (the model viewer and the stories read them) -stories.json, stories/.json the stories -assets/. every file of every chart, model and story, named by its content hash -assets/..gz the same, stored gzip-encoded (.br: brotli-encoded) -``` -A manifest lists the files of one chart and the asset that holds each one; the format is in -[data-format.md](data-format.md) (stories: [story-data-format.md](story-data-format.md)). The player fetches the -manifest, then every asset it lists, before the chart starts. A chart is about 40–45 MB; charts share most of their -assets (stage, notes, effects), so a browser that caches them downloads much less for the second chart. - -Serving notes: - -- **Paths.** Asset paths in a manifest are relative to the site root. For a chart or model manifest the player takes - the directory above the manifest's directory (`charts/..`, `models/..`). A story manifest of format `/2` names the - site root in its `root`, so `stories//.json` works too, and the model manifests it uses are found - from there; for one of format `/1` the player takes the directory above the manifest's directory. A manifest served - from somewhere else needs `AssetStore.fromManifest(url, { base })` (`loadStoryStore(url, { base })` for a story). -- **CORS.** A page on another origin than the site needs `Access-Control-Allow-Origin` (for example `*`) on the - manifests and the assets. No credentials are sent, and the requests are simple GETs (no preflight). -- **Encoded assets.** The JSON, GLSL and moc3 assets (and the other kinds listed in - [data-format.md](data-format.md#site-layout)) are stored encoded where that makes them smaller: `.gz`, or `.br` in a - site built with brotli. The manifest gives each one's stored and decoded byte counts. Served as plain files, they - are decoded by the player with the browser's `DecompressionStream`: gzip in current browsers, brotli where - `DecompressionStream("brotli")` exists (Firefox and Safari; not Chromium-based browsers). The server may instead - send them with the matching `Content-Encoding` (`gzip` for `.gz`, `br` for `.br`) and a type of the logical file; - the browser then decodes them and the player takes them as they arrive. A brotli site that Chromium-based browsers - must play has to be served that way, with `Content-Encoding: br` on the `.br` assets. A server must not compress - the encoded assets again. -- **Compression of the other files.** Manifests, index files and plain `.json` / `.glsl` assets compress well with - the server's own `Content-Encoding`; the PNG and audio files are already compressed. The manifest's sizes are those - of the decoded files. -- **Caching.** Assets never change under their name: `Cache-Control: public, max-age=31536000, immutable`. Manifests - and the index files change when the site is rebuilt: a short max-age or revalidation. -- **Types.** `application/json`, `image/png`, `audio/flac`, `audio/mp4` (`.m4a`), `text/plain` (`.glsl`); an encoded - asset served as a plain file can be `application/octet-stream` (`application/gzip` for `.gz`). +Regional aliases can add a region segment to entry paths. Follow the paths in the listings. Products resolve their +dependencies through immutable manifests; standalone layouts also use the paths in their product references. +The [data format](data-format.md) and [story format](story-data-format.md) describe these contracts. + +| Concern | Configuration | +|---|---| +| Paths | Preserve the directory layout and resolve product files against the manifest's `root`. Format 2 chart/model sites use their documented site-relative paths. | +| CORS | For cross-origin playback, allow the page origin on every manifest and asset. Public static data can use `Access-Control-Allow-Origin: *`. | +| Immutable caching | Content-addressed assets and `products/.json` can use `Cache-Control: public, max-age=31536000, immutable`. | +| Mutable caching | Indexes, public entry aliases and HTML pages use revalidation or a short cache lifetime. | +| Publication | Upload assets and immutable products first, then public entries/pages and indexes. Retire unused files after successful publication. | + +### Encoded assets + +Manifest entries record decoded sizes and, for encoded assets, stored sizes. Plain `.gz` files can be decoded by +the player's `DecompressionStream`. A server may instead send them with `Content-Encoding: gzip` and the logical +content type. `.br` files require `Content-Encoding: br` for Chromium playback; other browsers may also support +client-side brotli decoding. Apply one encoding layer to each stored file. + +Use `application/json` for JSON, `image/png` for PNG, `audio/flac` for FLAC, `audio/mp4` for M4A and `text/plain` +for GLSL. An encoded file served without `Content-Encoding` can use `application/octet-stream` or +`application/gzip`. Mutable JSON and HTML can use ordinary server-side compression. + +### Compatibility + +ournotes-player 0.2.x reads nnnotes 0.3.x story/UI products and format 2 chart/model sites. For a manifest served +outside its normal root, use the relevant loader's `base` option as described in the API reference. +Run `node node_modules/ournotes-player/scripts/validate-data.mjs path/to/site` before serving a generated directory. ## Browser requirements -WebGL2, WebAudio, `ResizeObserver`, ES2022 modules; custom elements for the element. The audio files are AAC (MP4) and -FLAC; current desktop and mobile browsers decode both. +Use a browser with WebGL2, WebAudio, ES2022 modules, ResizeObserver and Custom Elements. It must support the audio +and video encodings selected during export, such as AAC/FLAC audio and VP9/Opus WebM video. Audio starts from a +user gesture. Load the required Cubism, MotionSync or Spine runtime before the corresponding component. ## Mobile -The player works at phone width (the control bar compacts below 480 px) and with touch: a tap shows or hides the -control bar, the buttons and the seek bar take taps. Audio starts on the first tap of the play button. The drawing -buffer follows `devicePixelRatio`; on a slow device pass `pixelRatio: 1` (or a lower value) to `ChartPlayer.create`. -A story draws at the screen the game renders at for its quality (at most 1920 pixels on the longer side at Best and -High, 1440 at Middle) rather than at every device pixel ([story.md](story.md), `resolution`); on a slow phone -`quality="middle"` also turns off what the game turns off at Middle (the character and background blur, the stage post -effect, Unity lighting, the breath motion, FXAA). +Controls support touch and adapt to narrow layouts. Live drawing follows `devicePixelRatio`; set `pixelRatio: 1` +in `ChartPlayer.create` to reduce the drawing buffer on a constrained device. Story `quality="middle"` selects +the corresponding reduced effects and resolution; see [story options](story.md). ## Several players on one page -Each player has its own WebGL context, `AudioContext` and state; players do not interfere. Browsers limit the number of -live WebGL contexts per page (16 in Chromium), and every loaded chart holds its files in memory: dispose players that -are no longer shown. +Each player owns a WebGL context, audio context and playback state. Browser context and memory limits apply; +dispose players that are no longer shown. See [performance](performance.md) for a repeatable profiling method. diff --git a/docs/fidelity.md b/docs/fidelity.md index e68e9cb..f442652 100644 --- a/docs/fidelity.md +++ b/docs/fidelity.md @@ -528,7 +528,6 @@ Every `ENGINE:` note in `src/`, by file. `npm test` checks that this list matche - the ambient probe of a Flat ambient is built natively from the ambient colour; a white Flat ambient is taken as - the default reflection of a Skybox-mode scene without a skybox material (and without a custom cube) is taken - unity_LightData (native per-object light data): x the offset of the per-object light indices (0), y their -- atlas page texture import settings are not in the data; bilinear, clamped, no mipmaps here. - a Skybox clear without a skybox material clears to the background colour. **`src/story/simple/home/room.js`** diff --git a/docs/getting-started.md b/docs/getting-started.md new file mode 100644 index 0000000..4970118 --- /dev/null +++ b/docs/getting-started.md @@ -0,0 +1,63 @@ +# Getting started + +ournotes-player 0.2.x consumes nnnotes 0.3.x products and format 2 chart/model sites. Use Node.js 22+ for package +tools. Browser playback requires WebGL2, WebAudio and the audio/video codecs used by your data. + +## Install + +```sh +npm install ournotes-player@0.2.0 +``` + +The npm package includes source modules, types, schemas, browser bundles and site tools. Use its directory, +`node_modules/ournotes-player`, as nnnotes' `--player` argument. The browser ZIP in a GitHub release is useful for +embedding bundles; a site build needs the complete package, including scripts, metadata and example pages. + +## Prepare a site + +Configure [nnnotes](https://metasekailab.github.io/nnnotes/quickstart/) with your own input files first: + +```sh +pip install 'nnnotes[fonts]==0.3.0' +nnnotes site build --name example -o out/site --pair 100001:expert --fonts game --player node_modules/ournotes-player +node node_modules/ournotes-player/scripts/validate-data.mjs out/site +python -m http.server 8000 --directory out/site +``` + +Open `http://localhost:8000/` for charts. Add `--story ID` to select a story, or `--live2d KEY` for a model. +The respective pages are `story/` and `live2d/`. Use IDs available in the configured catalog and master data. + +Site builds reuse the named selection's depot results. Keep its indexes, manifests and assets together when +hosting it. See [embedding](embedding.md#hosting-the-data) for paths, caching and compression headers. + +## Browser runtimes + +| Feature | Runtime loaded by the page | +|---|---| +| Live chart playback | Browser WebGL2 and WebAudio | +| Live2D models and story characters | Live2D Cubism Core | +| Models with MotionSync lip sync | Live2D MotionSync Core, in addition to Cubism Core | +| Home conversations | Spine runtime, in addition to the story's other requirements | + +Obtain these runtimes under their respective licenses and load them before creating the relevant player. +The example story page accepts `?core=URL&motionsync=URL&spine=URL`; pass the URLs needed by that story. +The model page accepts `?core=URL`. These files are provided separately from the player package. + +## Embed a chart + +```html + + +``` + +For stories, load the required runtime scripts and `ournotes-player.story.element.min.js`, then use +``. +For models, load Cubism Core and `ournotes-player.live2d.element.min.js`, then use ``. +Use a language declared by the story product. The [embedding guide](embedding.md) covers JavaScript and iframe APIs. + +## Validate and diagnose + +Run the validator against the complete exported directory. It checks manifests, object references, asset sizes, +chart facts and declared languages. In a browser, inspect failed network requests, required runtimes and decode +errors. For a cross-origin page, return CORS headers on every manifest and asset. Chromium playback of `.br` +assets requires `Content-Encoding: br`. diff --git a/docs/live2d.md b/docs/live2d.md index ca8b324..23a9541 100644 --- a/docs/live2d.md +++ b/docs/live2d.md @@ -35,7 +35,7 @@ model viewer. ```html - + @@ -81,10 +81,11 @@ its shadow root; the div fills the host, and a host without a height gets a box its width. It resolves once the model is shown, after the `ready` event, and rejects when loading fails (after an `error` event). -Options: `src` (a model manifest URL) or `assets` (an `AssetStore`); `motion`, `expression`, `loop`, `physics`, -`breath`, `seed` as for `ModelSession`; `paused` (start paused); `pixelRatio` (device pixels per CSS pixel, default -`devicePixelRatio`); `signal` (an `AbortSignal` that cancels the loading); `on` (`{type: listener}` added before -loading starts). +Options: `src` (a model manifest URL: a product manifest of `live2d.model` or a +[model manifest of format 2](data-format.md#model-manifests-of-format-2)) or `assets` (an `AssetStore`); `motion`, +`expression`, `loop`, `physics`, `breath`, `seed` as for `ModelSession`; `paused` (start paused); `pixelRatio` (device +pixels per CSS pixel, default `devicePixelRatio`); `signal` (an `AbortSignal` that cancels the loading); `on` +(`{type: listener}` added before loading starts). | | | |---|---| @@ -101,7 +102,7 @@ loading starts). | `physics`, `breath` | Read / write. | | `hasPhysics` | The model has physics (a `CubismPhysicsController`); without it `physics` stays `false`. | | `play()`, `pause()`, `paused` | While paused no frame advances; the last frame stays shown. | -| `info` | The manifest without `files`. | +| `info` | The manifest without `files` and `objects`: `{ schema, product, subject }` (for a [model manifest of format 2](data-format.md#model-manifests-of-format-2), that form with its key as `subject`). | | `session`, `root`, `canvas` | The `ModelSession`, the appended div and the canvas. | | `dispose()` | Stops the player, releases the WebGL context and the model and removes the div. | diff --git a/docs/performance.md b/docs/performance.md index f86a4c4..89b6f0b 100644 --- a/docs/performance.md +++ b/docs/performance.md @@ -1,131 +1,31 @@ # Performance -What a frame of the player costs on the CPU, which of that work the browser does in its own compiled code, and how -much is the player's JavaScript. The numbers below were measured with the version of the player this file ships -with, on the machine and charts listed; other versions, browsers, machines and charts give other numbers. They are -rounded. +Measure the player on the browser and device that will display it. Rendering cost depends on the product, +quality settings, canvas resolution, runtime dependencies and the number of active players. -The browser drew with a software rasterizer, so this page covers the CPU side only: the time the GPU (or the -rasterizer) takes to draw a frame is not measured. +## Profile a repeatable workload -## Conditions +1. Record the player version, product manifest, browser version, viewport, device pixel ratio and playback settings. +2. Load the same product once to separate initial transfer/decode work from steady playback. +3. Use browser performance tools to record a fixed playback interval. Include scenes with the effects used by + your content, and exercise seeking separately from continuous playback. +4. Inspect main-thread work, frame duration, WebGL activity and memory. Use the network panel to distinguish + transfer time from asset decoding and rendering. +5. Repeat with a cold cache when assessing first-load behavior. Compare equal viewports and settings. -| | | -|---|---| -| Browser | Chromium 154.0.8037.57, headless, on Linux (Debian 12) | -| Graphics | ANGLE on SwiftShader: WebGL2 rendered in software, in the browser's GPU process | -| Machine | 64 logical CPUs (AMD EPYC 9K65) | -| Canvas | 320 × 180 CSS pixels at a device pixel ratio of 1, so that the software rasterizer keeps up with 60 frames per second (at 1280 × 720 it draws about 8 frames per second and the main thread waits for it). The player makes the same draw calls at any canvas size (only values such as the screen size differ); at larger sizes the rasterizer's work grows | -| Player | `dist/ournotes-player.js`, `ChartPlayer` with `controls: false` and `seed: 1`; music and sound effects on, the manifest's quality | -| Charts | from a site built by nnnotes for the Taiwan server, version 1.0.1: 100015 expert (1 874 notes; of that region's 336 charts the one with the most notes and the most WebGL calls per frame), 100040 expert (1 161 notes; with its effects the second most WebGL calls per frame), 100001 expert (822 notes) | -| Span | 60 s of playback, starting 5 s after `play()` | +Chart simulation advances at 60 Hz. Seeking resimulates state, so its workload differs from drawing a frame. +Stories can also spend time preparing glyph pages, model resources and media when loading or changing language. -Each chart was played twice over the same span: once with a Chrome trace of the renderer's main thread (the thread -time of every task, summed per animation frame) and `performance.now()` around the session's simulation phases and -its `render()`; once under the V8 CPU profiler (100 µs sampling) for the split by kind of work. +## Control resource use -## Per-frame CPU time in the browser +- Set the Live `pixelRatio` option to bound drawing resolution on high-density displays. +- Choose the story quality appropriate to the device; [story options](story.md) describe its rendering effects. +- Cache content-addressed assets with immutable cache headers so related products reuse shared resources. +- Dispose hidden or removed players to release their WebGL contexts and audio resources. +- Profile a single player before adding several concurrent instances. -Main-thread time per animation frame, against the 16.7 ms of a 60 Hz frame: +## Check behavior after a change -| Chart | Frames per second | Median | 95th percentile | 99th percentile | Frames over 16.7 ms | -|---|---|---|---|---|---| -| 100015 expert | 57 | 3.5 ms | 6.1 ms | 8.9 ms | 14 of 3 428 (0.4 %) | -| 100040 expert | 58 | 3.7 ms | 6.1 ms | 11.9 ms | 21 of 3 471 (0.6 %) | -| 100001 expert | 60 | 2.7 ms | 4.4 ms | 8.6 ms | 15 of 3 578 (0.4 %) | - -Headless Chromium produced slightly fewer than 60 animation frames per second here; the player then runs the steps -that are due (1.00 to 1.05 steps per animation frame in these runs) and draws once per animation frame. - -| Chart | Simulation, per step | `render()`, per draw: JavaScript and WebGL calls | -|---|---|---| -| 100015 expert | 0.5 ms | 2.5 ms | -| 100040 expert | 0.6 ms | 2.6 ms | -| 100001 expert | 0.4 ms | 2.0 ms | - -The main thread's busy time by kind of work (CPU profile): - -| | 100015 expert | 100040 expert | 100001 expert | -|---|---|---|---| -| The player's JavaScript | 65 % | 64 % | 62 % | -| WebGL calls: the browser's validation and command encoding on the main thread | 16 % | 16 % | 17 % | -| Browser code outside JavaScript: task scheduling, frame production | 10 % | 12 % | 12 % | -| JavaScript garbage collector | 6 % | 5 % | 6 % | -| Other browser APIs the player calls: `MessageChannel` (the order of the loop's phases), timers, audio nodes | 3 % | 3 % | 3 % | - -The measurement's own code took under 1 %. Within the player's JavaScript, the particle systems take about a -quarter, looking up and setting the shaders' uniforms and render state for each draw call about a sixth, the note -views about a tenth, and the chart simulation itself (note states, judgements, combo) about 1 %. - -## What runs in the browser's compiled code - -- **WebGL2.** The main thread validates each call and encodes it into the browser's command buffer (the WebGL share - above). The GPU process executes the draw calls and compiles the game's GLSL ES 3.00 shaders, on the GPU or, as - here, in the software rasterizer; neither runs on the main thread. -- **WebAudio.** `decodeAudioData` decodes the FLAC and AAC files (at the context's sample rate) when the chart - loads; mixing and output run on the browser's audio thread. On the main thread the player only creates and - schedules sources and gains (under 1 % of its busy time). -- **Images.** `createImageBitmap` decodes the PNG textures when the chart loads or a texture is first used. -- **Live2D.** Cubism Core, which the page supplies, is compiled code. -- **JavaScript.** V8 compiles the player's frequently run code to machine code. - -## The JavaScript share - -The player's JavaScript and its garbage collection take about two thirds of the main thread's busy time: at the -median, 1.9 to 2.6 ms per frame. An implementation of that work that took no time at all would save at most that -much per frame; the WebGL calls and the browser's own work would stay. At the median the frames already leave -13 ms or more of the 16.7 ms unused. - -## Live2D model viewer - -`ModelPlayer` with the model `adv_live2d_rana_003_casual_spring_01` on a 720 × 720 canvas, measured over 60 s from -5 s after `play()`, under the same conditions otherwise: 60 animation frames per second and 30 model steps per -second (idle motion, eye blink, breath, physics). A frame that steps and draws the model takes about 5 ms of -main-thread time (99th percentile 9 ms; 3 frames of 3 599 over 16.7 ms); a frame without a step about 1 ms. Per -step, the model update (motions, physics and the Cubism Core update) takes 1.8 ms and the draw 3.3 ms. - -| The main thread's busy time | | -|---|---| -| The player's JavaScript | 41 % | -| WebGL calls | 23 % | -| Cubism Core | 22 % | -| Browser code outside JavaScript | 10 % | -| JavaScript garbage collector | 2 % | -| Other browser APIs | 2 % | - -## Node.js without a browser - -`scripts/read-set.mjs` in full mode runs a whole chart in Node.js with a WebGL2 context and an AudioContext that do -nothing: every frame is simulated and its draw calls are made, nothing is drawn or played. This measures the -player's own work without any WebGL implementation. Per step, one process at a time on the same machine, Node.js -22.23.3, two runs per chart: - -| Chart | Frames | Step: median | 95th percentile | 99th percentile | Of the step, draw submission (median) | `read-set.mjs` full mode, load and every frame | -|---|---|---|---|---|---|---| -| 100015 expert | 10 462 | 1.6 ms | 2.5 to 2.7 ms | 3.4 to 3.7 ms | 1.1 ms | 18 s | -| 100040 expert | 6 678 | 1.6 to 1.7 ms | 2.7 to 3.0 ms | 4.6 to 5.4 ms | 1.1 to 1.2 ms | 12 s | -| 100001 expert | 6 001 | 1.3 ms | 2.0 ms | 2.4 ms | 0.9 ms | 8 s | - -Over the region's 336 charts, the median chart makes about 2 900 WebGL calls and 88 draw calls per frame; 100015 -expert makes the most, 5 530 and 166. - -## Reproducing the measurement - -1. Build the player (`npm ci`, `npm run build`) and serve a site and the player's `dist/` from one origin. For - `performance.now()` at its finest resolution, serve the pages cross-origin isolated - (`Cross-Origin-Opener-Policy: same-origin`, `Cross-Origin-Embedder-Policy: require-corp`). -2. Start Chromium, for example headless through the DevTools protocol, with - `--autoplay-policy=no-user-gesture-required` (and, without a GPU, `--use-angle=swiftshader - --enable-unsafe-swiftshader`). On a page with a 320 × 180 element, create a `ChartPlayer` with - `controls: false` and `seed: 1`, call `play()`, wait 5 s, then measure 60 s between two `performance.mark()`s. -3. Per frame: record a trace (`Tracing.start` with the categories `toplevel`, `devtools.timeline` and - `blink.user_timing`) and sum the thread time (`tdur`) of the main thread's top-level tasks from one - `FireAnimationFrame` event to the next. -4. Split: over a second playback of the same span, run the CPU profiler (`Profiler.setSamplingInterval` with 100 µs, - `Profiler.start`, `Profiler.stop`) and sum each node's self time: the player's bundle (by the source module each - `// src/...` section of `dist/ournotes-player.js` names), the methods of `WebGL2RenderingContext`, - `(garbage collector)`, `(program)` and `(idle)`. -5. Node.js: `node scripts/read-set.mjs /charts/.json` prints the frames and seconds of a full run. For - per-step times, run the same loop yourself (`ChartSession.create` with `headlessGL()` and `HeadlessAudioContext` - from `scripts/lib/headless.mjs`, `setMusic(false)`, `play()`, then `step()` until the state is `"ended"`) with - `performance.now()` around each step. +Run the synthetic tests and package contracts, then replay the same workload with the same data and settings. +Compare completion, seeking, audio synchronization, selected language and visible output as well as frame cost. +[Embedding](embedding.md) covers cache and compression headers; [fidelity](fidelity.md) describes playback semantics. diff --git a/docs/story-author.md b/docs/story-author.md index 2d91e83..9515e21 100644 --- a/docs/story-author.md +++ b/docs/story-author.md @@ -11,7 +11,7 @@ The tool is experimental: the script format and the commands may change between ## Requirements - Node.js 22 or later. -- A story site with stories in the model manifest format (`ournotes.story-manifest/2`) and its `models.json` +- A story site with stories in the story product format (`nnnotes.product/1`) and its `models.json` ([story-data-format.md](story-data-format.md)). Pass it with `--site ` or set `OURNOTES_SITE`. - For pictures (`render`, `base --out`, `model --out`): Chrome, Chromium or Edge. The tool starts a headless browser of its own with a temporary profile and ends it when the command ends. It draws with SwiftShader (WebGL2 on the @@ -181,10 +181,11 @@ long auto mode then keeps the line before the next one (`_waitTalkTextUnitTime` ### build -`build