diff --git a/README.md b/README.md index 0e9fd81..e4681a9 100644 --- a/README.md +++ b/README.md @@ -180,6 +180,12 @@ tuirec record --mouse-pointer all --binary ./myapp --keystrokes "..." ## For AI Agents +If a GIF or PNG *looks* wrong but the live app looked fine, treat the **capture +pipeline** as guilty first — keep the `.cast`, error-grep it, and only then +change keystrokes or the app. Prefer **settled keyframes** for multi-step state +demos and continuous `record` when motion itself is the story. Full agent +semantics: `tuirec agent-guide` (and `agent/RECORDING-AGENT.md`). + tuirec provides three layers of agent guidance — enabling agents to one-shot the creation of GIFs of TUI apps doing complicated stuff: diff --git a/agent/RECORDING-AGENT.md b/agent/RECORDING-AGENT.md index 0c09360..3e740f7 100644 --- a/agent/RECORDING-AGENT.md +++ b/agent/RECORDING-AGENT.md @@ -133,6 +133,11 @@ A keystroke script is a **comma-separated** string. Each token is one of: cast to see the final printed output). Post-exit terminal noise (stderr from ConfigurationManager, shell prompt redraws) is normal during `--drain` — filter accordingly when validating. +- **Pipeline before product** — if the PNG/GIF shows wrong geometry, a false + solid color band, off-center chrome, or missing glyphs **but the live app + looked fine**, treat the capture pipeline as guilty first. Keep the `.cast`, + error-grep it, and only then change the keystroke script or the app. A green + tuirec exit code does not mean the image is faithful evidence. - **Verifying grid-anchored sixels — measure, don't eyeball** — confirming a sixel *appears* in the GIF, or that `agg` rendered it faithfully at the requested cursor cell, does **not** prove it is the right size or position: a @@ -149,6 +154,8 @@ A keystroke script is a **comma-separated** string. Each token is one of: they agree and that the rendered bounding box spans the intended columns/rows. This is the cheap check that turns "looks right" into a number, which is the only thing that catches sub-cell and few-percent errors. + The same discipline applies beyond sixels: **measure or dual-check the cast** + whenever the picture surprises you. - **Key-name collisions with literal words** — bare tokens like `delete`, `home`, `end`, `space`, `tab` resolve as **key presses**, not literal text. If you mean to type those words as text, you must backtick-quote them: `` `delete` ``, @@ -262,6 +269,19 @@ If running inside a restricted agent sandbox that blocks PTY-spawning commands: recording for 2 seconds so the final UI state is visible in the GIF. Without drain, the recording may end before the last action renders. +11. **Pick continuous vs keyframe on purpose** — one long `record` GIF is + perfect when *motion* is the story (typing cadence, scroll, animation). + For multi-step state demos (open dialog → change setting → save → quit), + a short sequence of **settled keyframes** is often clearer: after each + important action, wait for the UI to settle, capture a still (or keep a + cast frame), optionally caption the step, then assemble a short GIF. + Do not force product docs to spell out pixel-perfect chrome just so a + continuous recording looks tidy — fix waits and framing instead. + +12. **Settle before you snap** — the same rule as "wait after UI transitions," + applied to stills: inject the key, wait until the view is stable, *then* + `snapshot` / assert. Snapping mid-repaint produces false "bugs." + --- ## Example keystroke scripts @@ -564,6 +584,9 @@ When asked to "record doing X", follow this process: can error in-frame and tuirec still finishes. For CLI apps, also grep for expected output (`grep "1966-09-10" demo.cast`). For TUI apps, the GIF is the positive verification — pair it with the error-grep as the negative check. + If the GIF looks *structurally* wrong (shifted layout, solid false color + bands, missing labels) while the live session looked fine, **debug the + pipeline and cast** before rewriting the app or the product spec. 8. **If execution fails due to permissions**, output the full command for the user to run manually — do not loop retrying. 9. **Report the output paths and the exact command used** back to the user so they diff --git a/cmd/tuirec/agent-guide.md b/cmd/tuirec/agent-guide.md index 0c09360..3e740f7 100644 --- a/cmd/tuirec/agent-guide.md +++ b/cmd/tuirec/agent-guide.md @@ -133,6 +133,11 @@ A keystroke script is a **comma-separated** string. Each token is one of: cast to see the final printed output). Post-exit terminal noise (stderr from ConfigurationManager, shell prompt redraws) is normal during `--drain` — filter accordingly when validating. +- **Pipeline before product** — if the PNG/GIF shows wrong geometry, a false + solid color band, off-center chrome, or missing glyphs **but the live app + looked fine**, treat the capture pipeline as guilty first. Keep the `.cast`, + error-grep it, and only then change the keystroke script or the app. A green + tuirec exit code does not mean the image is faithful evidence. - **Verifying grid-anchored sixels — measure, don't eyeball** — confirming a sixel *appears* in the GIF, or that `agg` rendered it faithfully at the requested cursor cell, does **not** prove it is the right size or position: a @@ -149,6 +154,8 @@ A keystroke script is a **comma-separated** string. Each token is one of: they agree and that the rendered bounding box spans the intended columns/rows. This is the cheap check that turns "looks right" into a number, which is the only thing that catches sub-cell and few-percent errors. + The same discipline applies beyond sixels: **measure or dual-check the cast** + whenever the picture surprises you. - **Key-name collisions with literal words** — bare tokens like `delete`, `home`, `end`, `space`, `tab` resolve as **key presses**, not literal text. If you mean to type those words as text, you must backtick-quote them: `` `delete` ``, @@ -262,6 +269,19 @@ If running inside a restricted agent sandbox that blocks PTY-spawning commands: recording for 2 seconds so the final UI state is visible in the GIF. Without drain, the recording may end before the last action renders. +11. **Pick continuous vs keyframe on purpose** — one long `record` GIF is + perfect when *motion* is the story (typing cadence, scroll, animation). + For multi-step state demos (open dialog → change setting → save → quit), + a short sequence of **settled keyframes** is often clearer: after each + important action, wait for the UI to settle, capture a still (or keep a + cast frame), optionally caption the step, then assemble a short GIF. + Do not force product docs to spell out pixel-perfect chrome just so a + continuous recording looks tidy — fix waits and framing instead. + +12. **Settle before you snap** — the same rule as "wait after UI transitions," + applied to stills: inject the key, wait until the view is stable, *then* + `snapshot` / assert. Snapping mid-repaint produces false "bugs." + --- ## Example keystroke scripts @@ -564,6 +584,9 @@ When asked to "record doing X", follow this process: can error in-frame and tuirec still finishes. For CLI apps, also grep for expected output (`grep "1966-09-10" demo.cast`). For TUI apps, the GIF is the positive verification — pair it with the error-grep as the negative check. + If the GIF looks *structurally* wrong (shifted layout, solid false color + bands, missing labels) while the live session looked fine, **debug the + pipeline and cast** before rewriting the app or the product spec. 8. **If execution fails due to permissions**, output the full command for the user to run manually — do not loop retrying. 9. **Report the output paths and the exact command used** back to the user so they diff --git a/llms.txt b/llms.txt index 8190941..3eb46fa 100644 --- a/llms.txt +++ b/llms.txt @@ -32,6 +32,8 @@ The guide covers: - Terminal.Gui app defaults (--kitty-keyboard, --startup-delay, etc.) - Known gotchas (cursor/page/arrow literal splitting, --agg-path requirement) - Composing demo recordings step by step +- Capture integrity: pipeline-before-product, settle-then-snap, continuous vs keyframe +- Sixel size/position: measure, don't eyeball ## Key flags diff --git a/spec.md b/spec.md index f4f0b02..0ba89dc 100644 --- a/spec.md +++ b/spec.md @@ -81,6 +81,10 @@ Rewrite tuirec as a **cross-platform Go CLI** that records any terminal app and GIF passes that). Decode the GIF and assert: ≥ 2 frames, non-zero dimensions, and non-trivial inter-frame pixel variance (the screen actually changed). For the test fixture, golden-frame compare. + - **Pipeline honesty:** a valid GIF that *looks wrong* is still a + pipeline problem until the cast proves the app painted that state. + Do not treat the rendered frame as ground truth for the product under + test without a cast-side check (see *Capture integrity* below). 5. **FR-5: Max duration timeout** - Hard cap on recording duration (default 60s, configurable) @@ -426,6 +430,91 @@ Encode these so they are **not** rediscovered: UICatalog lives at `Examples/UICatalog/UICatalog.csproj`. v1 should rely on `internal/testapp` and not take this dependency. +## Capture integrity (field lessons) + +These lessons come from real host-side capture work (including non-PTY +frame dumps used next to silico/product metal demos). They apply to +**tuirec's** cast → PNG/GIF path and to agent workflows that treat those +images as evidence. + +### Pipeline before product + +If a still or GIF disagrees with what a human saw on the live terminal +(wrong geometry, false color bands, off-center chrome, missing glyphs): + +1. **Suspect the capture pipeline first** — not the product under test. +2. Keep the `.cast` (or intermediate payload) and check it for expected + text / sequences before rewriting the app or the keystroke script. +3. A green exit code and a file that "looks like an image" are not enough + (same spirit as FR-4: stronger than magic bytes). + +Sixel measurement already says "measure, don't eyeball" for size/position. +The same honesty applies to **any** surprising visual: a few-percent size +error and a transport-corrupted frame are both pipeline-class failures. + +### Transport and encoding invent bugs + +Any path that moves frame bytes through a **cooked text channel** (line +ending conversion, log interleaving, partial reads) can invent solid color +bands, horizontal shift, and "off-center" UIs that never appeared on the +real screen. tuirec's cast format is JSON-lines of UTF-8 for a reason — +it is not a raw framebuffer dump. + +When agents or tools add companion capture paths (serial snapshots, external +frame sources, base64 payloads), they must: + +- Prefer text-safe encoding for binary rasters on serial/console links. +- Silence concurrent log noise during payload emit, or delimit payloads so + interleaving is detectable. +- Carry width/height/format (and ideally a checksum) in a clear header. +- Decode with the **same** pixel packing the producer used; a wrong + RGB565 endian/swap looks like a product theme bug. + +### Scenario keyframes vs one long continuous GIF + +Continuous wall-clock recordings are excellent for "watch it happen." +Multi-step demos (boot → theme → details → play → pause) are often clearer +as a **short sequence of labeled keyframes**: inject input, **settle**, +snapshot, annotate, then assemble a GIF. + +That pattern is compatible with today's CLI: `record` for motion, or +repeated `snapshot` / retained cast frames for stills. Specs and agent +guides should not force one style; they should say when each helps. + +- **Use continuous `record`** when motion itself is the story (scroll, + animation, typing cadence). +- **Use keyframe scenarios** when the story is a sequence of states and + each state must be readable (and when full continuous capture is too + noisy or too expensive). +- After every inject, wait for the UI to settle before the next snap — + the same reason `wait:` exists after dialogs. + +### Do not optimize product contracts for capture tools + +tuirec exists so demos of *real* apps are honest. Specs for those apps +should stay human-readable. Capture tooling and agent guides absorb the +pipeline discipline; product manuals should not grow "on-screen token +tables" just so a recorder passes. + +## Companion capture (out of core v1, in-scope for honesty) + +tuirec v1 records **PTY output**, not device glass and not arbitrary +host framebuffers (constitution: not a screen recorder of pixels). Host +tooling next to silico-class metal products may still dump a product +**shadow framebuffer** over serial for agent vision without a camera. + +That companion path is **not** a tuirec v1 feature requirement. It is +listed here so maintainers do not: + +- Confuse metal frame dumps with `tuirec record` / `snapshot` semantics. +- Silently expand v1 scope into a general device screen recorder. +- Ignore the transport-integrity lessons above when those dumps exist + alongside tuirec artifacts in the same agent session. + +If a future milestone absorbs external frame sources, it starts as an +explicit design (package boundary, encoding, validation) — not as a +side effect of an agent guide tip. + ## Key Risks | Risk | Mitigation |