Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
133 changes: 133 additions & 0 deletions PR_DESCRIPTION.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,133 @@
## Summary

Add a model-neutral `flashdreams-runner` shell and a `t2v-app` example
application for running FlashDreams pipelines through replay/MP4, WebRTC, or
headless I/O modes.

The application owns inference. Its runtime owns model weights and one-time
initialization; each session owns its prompt, cache, step state, and generation
logic. The runner owns mode selection, process setup, lifecycle, the main loop,
and presentation.

## High-level design

```text
uv run flashdreams-runner t2v-app {mp4 | replay | webrtc | none}
|
v
+--------------------------+ create_runtime() +--------------------+
| flashdreams_runner | ---------------------------> | t2v_app |
| | <--------------------------- | |
| select I/O mode | Runtime | Runtime |
| initialize Runtime | | model + pipeline |
| create Session | | |
| | input / output | Session |
| main loop: | ---------------------------> | prompt + cache |
| read input | <--------------------------- | generate/finalize |
| Session.step | StepResult +--------------------+
| present output |
+------------+-------------+
|
+-----+-----+----------------+
| | |
v v v
MP4/replay WebRTC None
```

## Entrypoint examples

Generate an MP4 with the packaged default preset:

```bash
uv run flashdreams-runner t2v-app mp4 \
--prompt "A waterfall" \
--output o.mp4
```

Use the explicit replay mode name and override its finite iteration count:

```bash
uv run flashdreams-runner t2v-app replay \
--steps 4 \
--prompt "A waterfall" \
--output o.mp4
```

Serve the same application through WebRTC:

```bash
uv run flashdreams-runner t2v-app webrtc \
--prompt "A waterfall"
```

Run without presentation or artifacts:

```bash
uv run flashdreams-runner t2v-app none \
--steps 2 \
--prompt "A waterfall"
```

Select a packaged preset explicitly:

```bash
uv run flashdreams-runner t2v-app mp4 \
--preset-id self-forcing-wan2.1-t2v-1.3b \
--prompt "A neon-lit city at night" \
--output outputs/city.mp4
```

When `--preset-id` is omitted, `t2v-app` uses the catalog's
`default_preset_id`. The packaged default is
`causal-forcing-wan2.1-t2v-1.3b-chunkwise`.

## Application ABI

- Require an application module to expose only
`create_runtime(ApplicationArguments) -> Runtime`.
- Let the application factory extend the selected mode parser and resolve all
application-specific command-line configuration.
- Define `Runtime.initialize()`, `Runtime.create_session()`, and
`Runtime.destroy()` for one-time model and process state.
- Keep presentation fields in application-owned `AppConfig`, exposed through
`Runtime.config` for runner modes.
- Define `Session.step()` and `Session.destroy()` for per-user prompt,
cache, world state, and main-loop logic.
- Keep compatibility methods on the base runtime/session classes so shared
FlashDreams WebRTC code consumes application runtimes directly without a
runner-specific adapter.

## Runner and modes

- Add the root-level `flashdreams-runner` workspace package and console entrypoint.
- Select and construct runner-owned I/O modes independently of applications.
- Initialize the application runtime with the selected device and I/O handler.
- Own session creation, deterministic batch input, output delivery, and cleanup.
- Add finite replay/MP4, live WebRTC, and finite headless `none` modes behind an
extensible `IOHandler` contract.
- Keep `mp4` as a compatibility name for replay-to-file behavior.

## T2V example application

- Add `t2v-app` as an implementation of the application ABI.
- Resolve pipeline object graphs from packaged YAML presets without depending
on the legacy runner registry.
- Construct and retain the FlashDreams pipeline in `T2VRuntime`.
- Create the prompt-conditioned cache and run pipeline `generate`/`finalize`
inside `T2VSession.step()`.
- Keep prompts, dimensions, caches, and step indexes isolated per session.

## Shared FlashDreams changes

- Add reusable pipeline-preset parsing and provider loading under
`flashdreams.core.pipeline_presets`.
- Share the generator checkpoint prefix-remapping helper from FlashDreams core
across causal-forcing and self-forcing configurations.

## Validation

- Affected CPU tests cover the application ABI, runner lifecycle, mode
separation, WebRTC construction, T2V runtime/session ownership, and preset
resolution.
- Ruff, `ty`, Basedpyright, lockfile validation, and CLI help checks pass.
- GPU model generation was not run as part of this change.
133 changes: 133 additions & 0 deletions apps/t2v_app/ARCHITECTURE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,133 @@
# T2V Application Architecture

The T2V application is an adapter between `flashdreams-runner` and a
FlashDreams streaming inference pipeline. The runner owns orchestration and
presentation; the application owns model configuration, model state, and
generation.

## Minimal application contract

The runner discovers the installed `t2v-app` distribution and imports its
top-level `t2v_app` module. That module exposes one public factory:

```python
create_runtime(arguments: ApplicationArguments) -> Runtime
```

The returned runtime implements the contract in
[`flashdreams_runner/contracts.py`](../../flashdreams_runner/contracts.py):

- `config` describes output identity, frame rate, layout, dimensions, and the
optional default step count.
- `initialize(device, io_handler)` constructs process-wide model state.
- `create_session(initial_input)` creates isolated generation state.
- `destroy()` releases process-wide resources.

Each session implements:

- `step_index`, the next autoregressive iteration.
- `step(inputs)`, which returns a FlashDreams `StepResult`.
- `destroy()`, which releases session state.

`Runtime` and `Session` also adapt this runner-facing API to the shared
FlashDreams `InferenceRuntime` and `InferenceSession` protocols through
`start_session`, `next_step_request`, and `close`.

## Components

### Application factory

[`t2v_app/application.py`](t2v_app/application.py) parses application
arguments, loads a pipeline preset, and creates an uninitialized `T2VRuntime`.
It does not construct model weights.

### Runtime

[`t2v_app/runtime.py`](t2v_app/runtime.py) owns the configured pipeline and
process-wide model weights. Initialization constructs the pipeline and moves it
to the selected device. The runtime creates one `T2VSession` for each isolated
generation.

### Session

[`t2v_app/session.py`](t2v_app/session.py) owns per-generation state:

- prompt and video dimensions;
- autoregressive cache;
- current block index;
- optional WebRTC recording.

Each generation step calls the pipeline's `generate` and `finalize` methods and
wraps the resulting video tensor in a `StepResult`.

### WebRTC customization

[`t2v_app/webrtc.py`](t2v_app/webrtc.py) is an optional adapter installed only
when the selected I/O handler is `WebRTCMode`. It supplies browser assets,
initial session input, prompt and duration updates, playback, and artifact
download routes.

## Control flow

### Startup

```text
flashdreams-runner
-> import t2v_app
-> t2v_app.create_runtime(arguments)
-> T2VRuntime.initialize(device, io_handler)
-> io_handler.run(runtime, drive_session)
```

### Finite modes (`mp4`, `replay`, and `none`)

```text
IO handler
-> drive_session(runtime, input_handler, output_handler)
-> runtime.create_session(initial_input)
-> session.step(step_input), repeated until input ends
-> output_handler.write(step_result)
-> session.destroy()
```

### WebRTC mode

```text
WebRTCMode
-> T2VWebRTCCustomization.prepare_initial_input()
-> shared WebRTC session manager
-> runtime.start_session(initial_input)
-> session.next_step_request()
-> session.step(step_input), repeated until complete
-> session.close()
```

Browser prompt updates call `T2VRuntime.prepare_session_input()` to replace the
initial input used by the next generation.

## Data boundary

The primary values crossing between the application and FlashDreams are:

- `ApplicationArguments`: runner mode and unparsed application arguments.
- `AppConfig`: presentation metadata consumed by runner I/O modes.
- `InferenceInput`: global conditioning at session creation and optional
per-step input.
- `StepRequest`: shared-serving request for the next iteration.
- `StepResult`: generated video chunk, layout, metadata, and metrics.
- `OutputArtifact`: persistent output returned by an I/O handler.

Pipeline presets, WAN recipe classes, autoregressive cache contents, browser
routes, and MP4 recording are implementation details rather than part of the
minimal application ABI.

## Ownership boundary

- `flashdreams-runner` owns application discovery, CLI modes, device/process
setup, lifecycle, iteration, and output presentation.
- `flashdreams.runtime` owns shared inference inputs, requests, results,
artifacts, and serving protocols.
- `flashdreams.infra` owns reusable pipeline, decoder, post-processing, and
configuration primitives.
- `t2v_app` owns T2V arguments, presets, pipeline setup, session state,
generation, and optional WebRTC behavior.
61 changes: 61 additions & 0 deletions apps/t2v_app/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
# T2V Example Application

`t2v-app` is an example implementation of the `flashdreams-runner`
application ABI. Its public module exports only `create_runtime(arguments)`.

The implementation has three layers:

- `application.py` parses T2V arguments, resolves the YAML pipeline preset, and
returns an uninitialized `T2VRuntime`.
- `runtime.py` owns pipeline construction, model weights, and one-time device
initialization.
- `session.py` owns the prompt, autoregressive cache, step counter, and the
pipeline `generate`/`finalize` calls for each main-loop iteration.

The runner owns mode selection, process setup, session lifecycle, iteration,
and presentation.

```bash
uv run flashdreams-runner t2v-app mp4 \
--preset-id causal-forcing-wan2.1-t2v-1.3b-chunkwise \
--prompt "A waterfall at sunset" \
--output outputs/waterfall.mp4

uv run flashdreams-runner t2v-app webrtc \
--preset-id self-forcing-wan2.1-t2v-1.3b

uv run flashdreams-runner t2v-app none \
--steps 2 \
--prompt "A waterfall"
```

## Pipeline presets

The application loads a YAML preset catalog through
`flashdreams.core.pipeline_presets` and asks the selected pipeline provider to
construct a `StreamInferencePipelineConfig`. The packaged catalog is
[`t2v_app/pipeline_presets.yaml`](t2v_app/pipeline_presets.yaml); pass
`--preset-config` to use another catalog.

Every preset specifies a pipeline provider, application defaults, and
provider-owned pipeline options. `total_blocks` is an optional default for
finite runner modes; `--steps` overrides it. The T2V WebRTC page lets each user
edit the prompt and video duration, keeps the connection open for subsequent
generations, plays the completed MP4, and downloads a ZIP containing the video
and prompt metadata.

FlashDreams' `ObjectGraphPipelineProvider` supports these trusted declarative
nodes:

- `_target: module:attribute` imports and calls a config class with the other
mapping entries as keyword arguments.
- `_ref: module:attribute` imports a value such as a checkpoint transform
without calling it.
- `_tuple: [...]` preserves tuple-valued config fields.

A custom package can expose a zero-argument pipeline provider class or instance
implementing `flashdreams.core.pipeline_presets.PipelineProvider` and reference
it from the YAML `provider` field.

See the [`flashdreams-runner` application ABI](../../flashdreams_runner/README.md#application-abi)
for the runtime, session, and mode lifecycle.
29 changes: 29 additions & 0 deletions apps/t2v_app/pyproject.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

[build-system]
requires = ["setuptools>=69", "wheel"]
build-backend = "setuptools.build_meta"

[project]
name = "t2v-app"
version = "0.1.0"
description = "Text-to-video example application for flashdreams-runner"
readme = "README.md"
requires-python = ">=3.10"
dependencies = ["flashdreams", "flashdreams-runner"]

[tool.uv.sources]
flashdreams = { workspace = true }
flashdreams-runner = { workspace = true }

[tool.pyright]
extraPaths = ["../..", "../../flashdreams"]
venvPath = "../.."
venv = ".venv"

[tool.setuptools.packages.find]
where = ["."]

[tool.setuptools.package-data]
t2v_app = ["pipeline_presets.yaml", "web/*.css", "web/*.js"]
8 changes: 8 additions & 0 deletions apps/t2v_app/t2v_app/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

"""Text-to-video example application for ``flashdreams-runner``."""

from .application import create_runtime

__all__ = ["create_runtime"]
Loading
Loading