Skip to content

Port remaining demos to the FlashDreams application/session API #479

Description

@jarcherNV

Port remaining demos to the FlashDreams application/session API

Summary

The newer public demo API is the flashdreams.demo application/session contract (IFlashDreamsApplication, IFlashDreamsApplicationSession, run_application) with host-owned MP4, null, local-window, and WebRTC I/O. The host invokes the unified runtime internally; application/session API coverage is the only migration status tracked by this issue.

Port the two remaining plain T2V integration families and add application adapters for the already-runtime-migrated OmniDreams and Lingbot demos. Do not include OmniDreams Interactive Drive in this issue.

Current status

Area Application/session API Evidence
Cosmos Predict2 T2V Complete t2v-cosmos-predict2 application entry point
Causal Forcing T2V Complete t2v-causal-forcing application entry point
Self Forcing T2V Complete t2v-self-forcing application entry point
Wan 2.1 T2V Missing wan21-t2v-1.3b-480p has no application entry point
FastVideo CausalWan 2.2 T2V Missing fastvideo-causal-wan2.2-t2v-14b has no application entry point
OmniDreams demo (excluding Interactive Drive) Missing No flashdreams.applications entry point
Lingbot demo Missing No flashdreams.applications entry point

The three existing T2V application packages are the intended reference: their integration owns a small t2v/app.py factory, depends on flashdreams-t2v, and registers a flashdreams.applications entry point.

Work

1. Add the missing plain T2V applications

  • Add t2v application factories and flashdreams.applications registrations for Wan 2.1 T2V and FastVideo CausalWan 2.2 T2V.
  • Reuse apps/t2v's T2VApplication / T2VApplicationSession where the model's pipeline contracts permit it; keep integration-specific defaults in the integration package.
  • Seed defaults from the existing runner configs: RUNNER_WAN21_T2V_1PT3B_480P and RUNNER_WAN22_T2V_14B.
  • Preserve the current prompt, resolution, FPS, block-count, decoder layout, compile override, and checkpoint/pipeline behavior. The new applications must stream every autoregressive block through the shared output path rather than call the legacy runner's run() method.
  • Add these slugs to apps/t2v/README.md and its integration-entry-point CPU coverage. Update the app's documented package-selection commands.

Suggested slugs: t2v-wan21 and t2v-fastvideo-causal-wan22 (confirm naming against the project's application-slug convention before landing).

2. Port the OmniDreams demo to an application adapter

  • Add a dedicated OmniDreams IFlashDreamsApplication and session implementation plus a flashdreams.applications entry point. Do not force it into the plain T2VApplication: it has world/scene conditioning and live controls, not prompt-only input.
  • Implement the application/session adapter as the supported OmniDreams demo path. Reuse useful scenario, provider, model-session, and output logic from the current implementation, but do not retain a parallel direct demo adapter as part of the target architecture.
  • Define the canonical input schema for the supported OmniDreams demo controls and map the application's canonical input windows to the current conditioning inputs. Preserve finite replay (MP4 and null) and WebRTC behavior, including model warmup and session lifecycle.
  • Keep scope to the standalone OmniDreams replay/WebRTC demo. Exclude omnidreams.interactive_drive, its local desktop loop, controller setup, and physics work.
  • Register/document the application and replace or clearly demote the temporary DemoApplication user-facing launch path only after feature and parity coverage is in place.

3. Port the Lingbot demo to an application adapter

  • Add a dedicated Lingbot IFlashDreamsApplication and session implementation plus a flashdreams.applications entry point. Lingbot is I2V with live camera pose controls, so it also should not use the prompt-only T2VApplication.
  • Implement the application/session adapter as the supported Lingbot demo path. Reuse useful example-data setup, input mapping, provider, and session logic from the current implementation, while keeping first-frame image, prompt, camera poses, intrinsics, example selection, and context-parallel setup functional.
  • Declare canonical controls for Lingbot's camera input and verify that their time windows produce the expected conditioning/pose sequence.
  • Support bounded MP4/null replay and WebRTC through the shared application host, including the current warmup and reconnect/reset semantics.
  • Register/document the application and retire the lingbot-demo development shortcut from user-facing guidance only once the application path is covered.

Acceptance criteria

  • flashdreams-run discovers five T2V-style application slugs: the existing Cosmos Predict2, Causal Forcing, and Self Forcing applications plus Wan 2.1 and FastVideo CausalWan 2.2; the OmniDreams and Lingbot application slugs are also discoverable through the same flashdreams.applications mechanism.
  • Each application owns its integration-specific factory and is independently installable with the minimum package selection documented in its README.
  • CPU tests cover entry-point registration, factory construction, argument validation, canonical schema validation, session reset/lifecycle, and MP4/null adapter routing. Extend the existing apps/t2v test style rather than adding untested registration-only code.
  • GPU/manual smoke coverage runs one finite generation for every new T2V application and exercises WebRTC startup for OmniDreams and Lingbot. For the controlled demos, include short deterministic trace/fixture coverage for canonical-input-to-conditioning mapping.
  • Update apps/t2v/README.md, the two integration READMEs, relevant model docs, and runner/application slug inventories so the documented commands use the new API.
  • No Interactive Drive files or behavior are changed as part of this work.

Notes for implementation

  • Existing T2V applications expose one integration-level default, not every legacy runner preset. Keep that precedent unless product requirements call for preset selection; do not silently drop a supported variant.
  • The application host already bridges applications onto the unified runtime internally. This port is therefore an API/ownership migration, not a request to replace the runtime engine or duplicate WebRTC/output infrastructure.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions