diff --git a/.claude/rules/opinionated-settings.md b/.claude/rules/opinionated-settings.md new file mode 100644 index 0000000000..79863b6f40 --- /dev/null +++ b/.claude/rules/opinionated-settings.md @@ -0,0 +1,45 @@ +--- +paths: + - "src/macos/**/*.swift" + - "**/*.tsx" + - "src/*/ui/**" + - "src/dashboard/**" + - "src/dev-dashboard/**" + - "src/hub/lib/widget/**" + - "src/notify/**" +--- + +# Opinionated behavior gets a setting + +Loaded by Claude Code when you read, write or edit native or web UI code, the widget's wire schema, or notifications. + +Martin's rule (2026-10-23): when a UI change picks one way where a reasonable user could want another, the choice +is a setting with that way as its default. You decide the default and ship the setting in the same change; do not +ask first, and do not ship the behavior hard-coded. + +## Opinionated means + +- **Interruptions**: a sound, which sound per kind of event, a popup, a panel that opens or expands by itself, a + notification, a focus steal. +- **Attention cues**: a flash, a pulse, a badge, their length and their colour; "flash the thing that caused it" is + the expected default, and its duration is the setting. +- **Placement and order**: where a composer or toolbar sits (top or bottom), list order, which filter a list opens + with, what a list shows (only what needs you, or everything active). +- **Hover behavior**: a tooltip or a preview on hover, hover delays, whether hover opens anything at all. +- **Timings**: auto-collapse delays, how long a popup stays. + +Not opinionated: a bug fix, a layout that was wrong, a missing hover effect, accessibility labels, performance. +Those are fixed, not made optional. + +## How + +- One preference per choice, stored where its surface already stores preferences (the widget: the zod schema in + `src/hub/lib/widget/types.ts` plus `WidgetPreferences` in GenesisKit `WidgetWireModels.swift`; a native window: + its settings store). An enum over a boolean when a third way is plausible ("auto" | "always" | "never"). +- The control goes on the settings page of that surface, in the card the behavior belongs to, with a one-line detail + that says what it does. A sound picker has a play button. A list that has a filter in place gets a settings + shortcut that opens the page at its card (`#`, `.nativeSettingsAnchor`), which flashes the card's border. +- The default is your best judgment of what Martin asked for; the option you did not pick stays reachable. +- A shared component's look change keeps the old look as a named option (the `AttentionPulse` presets: the border + flash is the default, the filled flash stays as `.filledFlash`). +- Say in the reply which settings you added and their defaults. diff --git a/.claude/skills/swift/references/gotchas.md b/.claude/skills/swift/references/gotchas.md index 0e8d1e0fa0..bb30625820 100644 --- a/.claude/skills/swift/references/gotchas.md +++ b/.claude/skills/swift/references/gotchas.md @@ -390,3 +390,67 @@ Keep measured findings separate from hypotheses; faster CPU numbers alone do not - **Driving a background app's context menu with `tools control` is not safe yet.** `AXShowMenu` on a SwiftUI tab blocked 2 s and opened the parent's menu first; the menu rows were not in `see --scope menu`; a coordinate click into the open menu closed it without choosing. Trigger the code path from a test instead. + +## Row-only controls with popovers, key modifiers, shared builds — 2026-10-23 15:17 + +- **A control that exists only on the hovered row must keep its popover open after the pointer leaves.** The + popover's `isPresented` lives in the control; remove the control on hover-out and SwiftUI closes the popover. + Keep the `.popover` modifier on a same-size `Color.clear` placeholder while the row is not live, and treat "popover + open" as live (`WidgetCmuxButton(live:)`, GenesisKit Widgets). Hover is the row's, not the control's: a hover + sensor on the button covers only the button (`WidgetHoverRow { hovering in … }`). +- **A popover from an edge panel needs its own "dialog open" flag.** Setting the host's shared `dialogOpen` from the + popover's `onChange` can release a window the host opened when the popover closes. One flag per presenter, ORed + into the getter (`sessionChooserOpen`, `cmuxPopoverOpen`). +- **`.onKeyPress(.downArrow) { … }` has no key press to read.** Its closure takes no argument; to read Shift use + `.onKeyPress(.downArrow, phases: [.down, .repeat]) { press in press.modifiers.contains(.shift) }`. A click's + modifiers come from `NSEvent.modifierFlags` inside the button action. +- **"input file … was modified during the build" is another session editing GenesisKit**, not your code. Rerun the + build through `.claude/skills/babysit/scripts/swift-slot.ts`; do not stash or touch their file. + +## A whole window ignores clicks after one exception in an action — 2026-10-23 16:58 + +- **`NSEvent.clickCount` raises for any event that is not a mouse click.** It is an Objective-C exception + (`NSInternalInconsistencyException`), not a Swift error, and `NSApp.currentEvent` inside a SwiftUI button action is + not always a mouse event: a key press, an accessibility press or a system-defined event can be current. The raise + unwound through SwiftUI's event dispatch and left the Genesis Markdown window ignoring every click, tab bar and + sidebar included, until relaunch (2026-10-23 16:36, the tab group chip's double-click check). Read the count through + GenesisKit `MouseClick.current` or `MouseClick.count(of:)`, which checks the event type first. lint-rules + `swift-current-event-click-count` fails `currentEvent?.clickCount`. Inside `mouseDown(with:)` or a monitor + filtered to mouse types, `event.clickCount` is safe. +- **When "nothing is clickable", read the app's unified log for an exception first.** `command log show --predicate + 'processID == ' --last 10m | rg -i "exception|raise|terminat"` found the line in one query; hit tests, AX + walks and layout theories had found nothing. +- **A test window's synthesized click always makes a mouse event current.** `NSApp.postEvent` down and up gives the + action a real `leftMouseUp`, so a test can never reproduce a crash that needs a key or AX event as the current + one. A unit test of the reader with a key event (`NSEvent.keyEvent(...)`) covers it; a passing click test does not. +- **An AX hit test over a `WKWebView` names the web area, not the view on top.** `tools control hittest` at a tab + that sat above the page reported `AXWebArea` (the web document's rect), while real clicks reached the tab. Judge + delivery by the app's own action log, never by the AX hit test. +- **A hidden test window has no accessibility tree, so find click targets with `FrameProbe`.** GenesisKit + `.frameProbe("name")` records a view's window frame only while `FrameProbe.enabled` (a test sets it); click the + centre of `FrameProbe.frames["name"]`. Never scan a grid of clicks to find a target: the scan pressed the toolbar's + Obsidian and Editor buttons and opened dialogs and editor tabs in the user's real apps. + +## Trigger animations, and loops no stall detector sees — 2026-10-23 17:47 + +- **An animation started from `onChange(of: trigger)` never plays in a view built after the change.** The widget's + hover list was opened by the same alert that bumped the flash counter, so its rows started with the new value and + did not flash. Pass the start time too and play the rest on appear (`AttentionPulse(since:)`). +- **A SwiftUI update loop never delays a main-queue ping.** Each pass returns to the run loop, so a queue-latency + watchdog reports nothing while the app burns a whole core (the widget, 2026-10-23 17:10, three minutes, no + log line). Read the main thread's CPU time instead: `thread_info(THREAD_BASIC_INFO)` on the port from + `mach_thread_self()` taken on the main thread (`HangWatch.SpinTracker`, `main-spin` lines). +- **Read a spin's sample for who schedules the next pass.** Inside the transaction, look for + `asyncTransaction` callers: `LazyLayoutViewCache.signalPrefetch` (a lazy stack that never settles) and + `EventBindingManager.enqueueHoverUpdateIfNeeded` (hover re-checked after every update) were the two in the widget's. +- **A test process has no `NSApp` until something makes one.** `NSApp.postEvent` or `nextEvent` then crashes + with "unexpectedly found nil" on the line that reads `NSApp`; call `_ = NSApplication.shared` first. +- **A window resize lays SwiftUI out synchronously, and content can call back into its controller from there.** The + edge panel moved its window in `begin`, the hover list measured itself in that layout and asked for its real size, + and the nested request started a reshape that `begin` then overwrote with its own transition: the panel kept the + estimate (336 instead of 489 pt). Defer a request made while a transition is being set up, and run the latest one + after it (`EdgePanelController.applyingPresentation`). +- **A `CADisplayLink` keeps a strong reference to its target.** A ticker held only by a struct that gets overwritten + never deinitialises and fires every frame for the life of the process. Stop it wherever the owner is replaced (a + `didSet` on the property), and count live ones for a test (`EdgeFrameTicker.live`). Find them in a sample of the + idle app: `EdgeFrameTicker.step` under `CA::Display::DisplayLink` with nothing animating. diff --git a/.claude/skills/ux/SKILL.md b/.claude/skills/ux/SKILL.md index 0c45b6aed0..dcc60cd6e1 100644 --- a/.claude/skills/ux/SKILL.md +++ b/.claude/skills/ux/SKILL.md @@ -26,6 +26,30 @@ thing for real, frame by frame, and comparing it with the checklist**, then fixi after sit next to each other. 6. **Write down what was wrong on the first try** in the ledger, generically, with the commit. +## What Martin asks for, before he has to (his reviews of 2026-10-23) + +Read this before a widget, hub or settings change, and do these without being asked. Each line is a request he had +to make by screenshot. + +- **Every clickable thing answers the pointer.** Round bubbles get a round hover ring, not a rectangular plate or + nothing. If he cannot tell a thing is clickable by hovering it, it is a defect. +- **Opinionated means configurable.** A sound, a popup, an auto-expand, a flash, a tooltip, a placement, an order: + ship a setting with your pick as its default (`.claude/rules/opinionated-settings.md`), and say which you added. +- **Say why something moved.** When the widget chimes or opens by itself, the session that caused it flashes + (its bubble, its row, its popup) for a configurable few seconds. A cue without its cause is noise. +- **Rows carry their times.** A session row says when it was last active and when its newest question or answer + arrived ("active 12m ago · asked 3m ago"), live (`LiveAgo`). A header says what the session is: agent, project, + model, working or idle for how long, context size. +- **No jumps.** A filter that empties a list does not shrink an open panel; a tab, filter or headline never scrolls + under a fade; content never blinks empty while a panel grows or a hover changes it. +- **The action is near the eye.** The reply box sits where the reading happens (under the tabs), not at the far + bottom of a tall panel. +- **Attention cues point, they do not cover.** A flash is the border; a tinted overlay over a whole card reads as + covered. Border flash is the `AttentionPulse` default. +- **A tooltip never stands in for the real hover behavior.** If hover should open a preview, a tooltip appearing + instead is a bug to fix, not a fallback. +- **Every count opens what it counts**, and every session offers its terminal (cmux) and its inbox. + ## Design authorities (load before a judgment call) | Question | Load | diff --git a/.claude/skills/ux/references/checklist.md b/.claude/skills/ux/references/checklist.md index 28a2f092a6..aed4d06072 100644 --- a/.claude/skills/ux/references/checklist.md +++ b/.claude/skills/ux/references/checklist.md @@ -45,6 +45,42 @@ Walk it for every view you change. Each line exists because it was missed at lea - One navigation per surface; no two controls that do the same thing side by side. - Distinct icons for distinct features (two modules once shared `waveform`). +## Widget and panel controls (Martin, 2026-10-23 14:40) +- A text field inside a padded box: the whole box focuses the field (`.contentShape` + `.onTapGesture { focus = true }` + on the box); a click beside or under the one-line field did nothing. +- A button that opened something closes it on a second click when what it opened is still on screen (a rail + session's inbox, the inbox overview). A second click that does nothing reads as a dead control. +- A count leads to the list of exactly what it counts. "35 need you" over 4 rows, and "99+ unread" that opened one + arbitrary session, are both defects: the count opens a list of every counted item (scrollable, filterable), and + each item says which session it belongs to (name, project, short id) and how to get there. +- Every session row offers a way to its terminal: the cmux button with GenesisKit `CmuxSessionPanel` (focus, open + in its last pane, choose a pane), one shared component in the hub and the widget. +- A list's filter and order live next to the list, with a settings shortcut that opens the page at the card and + pulses it (`#` + `.nativeSettingsAnchor(id)`, GenesisKit `AttentionPulse`). +- One row style for every top-level item of a list. Sessions with sub-agents drew as disclosure headers and sessions + without as plain rows, in one list: same row, a chevron only where there is something to fold, and selection, pin + and cmux inside the row's own box. +- A scroll view reused for several sections (or sessions) gets an identity per content (`.id(section + key)`): it + otherwise keeps the old offset, and a short section drew its rows over the header. + +## Widget and panel controls, round 2 (Martin, 2026-10-23 15:35) +- Hover on every control, shaped like the control: a circle bubble gets a ring (`WidgetBubbleChrome`), which pulses + once on enter and settles. Opacity only, never a repeating animation. +- A filter bar and its headline sit outside the scroll view they filter, pinned under the tabs. +- A filter change in an open panel holds the panel's height (`WidgetModel.holdAgentFit`); folding a row or switching + view releases it. An empty filter result must not collapse the panel. +- A header line under a title is the place for facts: agent · project · model · working / idle `LiveAgo` · ctx. +- Rows: last active and newest question or answer, both `LiveAgo`. +- An alert (sound, popup, auto-expand) has a setting each, a sound per kind with a play button, and flashes the + session that caused it for `alertFlashSeconds`. +- A hover tooltip that duplicates a hover preview is off by default ("auto"), and the preview must open reliably: + a control the pointer reaches inside a panel starts the panel's hover itself, since the panel's own hover fires + only on crossing its edge. +- Two views of the same sessions (the rail bubbles and the hover list) read in one order; a bubble and its row sit + at the same place. A sticky order that never moves is an option, not the default (Martin, 2026-10-23 17:47). +- A flash reaches every place the session shows, including a list the alert itself opened: such a list is built + after the trigger changed, so it needs the flash start time, not only the trigger. + ## Feedback - Every action answers within 100 ms: visible state change, sound for capture, progress for slow work. - Errors say what happened and what to do, in a dialog when the user must act. A missing grant goes through diff --git a/.claude/skills/ux/references/findings.md b/.claude/skills/ux/references/findings.md index 54640d8af8..c5c4711f94 100644 --- a/.claude/skills/ux/references/findings.md +++ b/.claude/skills/ux/references/findings.md @@ -3,6 +3,38 @@ Newest first. Each entry: date, symptom as the user saw it, cause, fix, commit. Generic lessons go to checklist.md or motion.md as well. +- 2026-10-23 16:30 — **Widget round 2 and the markdown tab bar (Martin's screenshots 23-28).** + - *Rail bubbles had no hover answer.* `.genHoverPlain` drew a rectangular plate a round bubble hid. Fix: + `WidgetBubbleChrome`, a round ring that pulses once on enter and settles (opacity only). + - *The hover list blinked and opened cut at the top.* One hover asked the panel for 336, 295 and 489 pt within 42 ms + (`edge.request` lines in `~/.genesis-tools/logs/app-perf.log`): the list is rebuilt per hover and its first layout + measured short. Fix: `FittedScroll(initialHeight:)` from the last height, shrinks wait 300 ms, side content pinned + to the top. Lesson: log every resize request with its branch before guessing at a blink. + - *A bubble showed its tooltip instead of the preview.* The panel's hover fires only on crossing its edge; after a + collapse under a resting pointer it never fired again. Fix: a bubble the pointer reaches starts the panel hover; + the bubble tooltip is off while previews are on (`bubbleTooltip`). + - *A filter that emptied the list dropped the open panel's height.* Fix: `holdAgentFit` before a filter change. + - *72 tabs wrapped into nine rows.* Fix: tab groups in GenesisKit (`MarkdownTabGrouping`), inline by default, the + wrapped bar capped at 4 rows with a fade. A first engine split big groups into singletons, so everything landed in + "Other": render the real data (`MARKDOWN_TAB_GROUPS_RENDER`) before calling a grouping rule done. +- 2026-10-23 15:17 — **The widget's inbox and session lists (Martin's review with seven screenshots).** Commit 5e0a6e328d. + - *Only the text line of the composer took a click.* The `TextField` was one line inside a padded box. Fix: the + box is the target (`LiveWidgetView.composer`, legacy `AgentWidgetView` too). + - *A second click on a rail session ("4") did nothing.* Every session button called `openInboxNotification` + again. Fix: `WidgetModel.toggleSession` / `toggleInboxOverview` collapse when that view is already open. + - *"35 need you" with four rows.* The preview took `prefix(4)` of the visible sessions. Fix: `previewRows`, every + session that needs you in a `FittedScroll`, an "Active too" filter, three orders, and a settings shortcut to + Projects / Sessions `#inbox-preview`, which pulses. + - *"99+ unread" opened one random session with no name or way to its terminal, and nothing else was reachable.* + The button picked the newest item of the whole inbox. Fix: an inbox overview over every session (the snapshot + now lists up to eight entries per session, `WidgetInboxSession.items`), each entry with its session and "Open in + its session", a session inbox filter (All / Inbox entries / Decisions), and a cmux button on every session. + - *A session opened from Sessions drew its inbox rows over the header.* The pane's one `ScrollView` kept the long + Sessions list's offset. Fix: `.id(section + "|" + selectedKey)`. + - *Sessions with and without sub-agents looked different.* Two row styles in one list. Fix: one top-level row. + - The first cut put the cmux button on every row of a list of hundreds; the swift skill's dense-list rule caught + it before commit (row-only controls, `WidgetHoverRow`). + - 2026-10-15 00:50 — **The vault approval window (GenesisTools.app `vault.unlock`).** - *"This can be probably nicer": one block of text with full paths, the raw command, two UUIDs and five buttons in a row.* It was an `NSAlert`. Now a SwiftUI panel (GenesisKit Permissions/ApprovalDialog.swift): the reason @@ -163,3 +195,63 @@ checklist.md or motion.md as well. user saw no feedback at all. Fix: the tree watches its root (FSEvents) and a reveal re-reads a folder that lacks the file; the reveal jumps a far row to the nearer edge, glides the last stretch and pulses the row. Before and after recordings in the notes vault's `GenesisTools.native/MarkdownViewer/Screenshots/2026-10-15-0618-*`. +- 2026-10-23 — **One press on a new tab group chip made the whole Genesis Markdown window ignore clicks.** The chip + read `NSApp.currentEvent?.clickCount`, which raises for a non-mouse event; the exception left SwiftUI's event + dispatch broken. I had checked the groups only as a rendered PNG and never pressed a chip in the real app, and my + synthesized test clicks always made a mouse event current, so no test could see it. Fix: GenesisKit `MouseClick`, + lint-rules `swift-current-event-click-count`, an exact-click test through `FrameProbe`, and the checklist line + "a render is not a click". My first test also swept clicks across the toolbar and opened Obsidian and the Editor in + Martin's apps; targets now come from `FrameProbe`. +- 2026-10-23 — **The alert flashed only the rail bubble, and the rail and the hover list disagreed on order.** The hover + list opened after the alert bumped the flash counter, so its rows never saw the change; the rail kept a sticky + order while the list sorted by attention. Fix: `AttentionPulse(since:)` plays the rest of a running flash on + appear; the rail follows the list order, with "Keep places" as the setting for the old one. +- 2026-10-23 — **The widget spun at 100% CPU for three minutes and nothing logged it.** A SwiftUI update loop + returns to the run loop between passes, so HangWatch's ping never ran late. HangWatch now also reads the main + thread's CPU time: 5 s above 85% logs `main-spin`, samples the stack and counts the bodies that ran (the dev + monitor shows it as `spin`). Strongest lead, fixed in the same commit: every preview open lost a resize and leaked + a display link that fired every frame for good (sampled live in the collapsed widget); not proven to be that spin, + because three hidden-window replays with the real inbox stayed idle. +- 2026-10-23 — **A hint line that appears under the focused row moves everything below it.** The Claude Code mods Git + sidebar drew a button's description under its own row while the focus ring was on it, so every Tab shifted the + rest of the pane. Fix (170c6ee191): one status bar with two fixed rows, pinned to the pane's foot; the body is built + from one-row lines, so the plugin counts them and pins the bar only when they fit. Check: read the pane's text + after each Tab and compare every line above the bar with the first read; nothing may differ but a live time. +- 2026-10-23 — **A slow call started from a render died when the next render came.** The engine aborts a replaced + render hook's in-flight process, and the plugin remembered the aborted `tools genesis-tools probe` as "no link + opens here" for 30 minutes, in every session: no hub button, no links. Fix (170c6ee191): start such work from the + session start or a timer, let renders only read it, never keep an aborted result; a test with the real abort text. +- 2026-10-23 — **A full row squeezed one of its chips to nothing.** The cache band's flex row shrank the own chip to + zero width when other sessions' chips filled it. Fix: every chip keeps its width and the row clips at its edge. +- 2026-10-23 21:13 — **Save draft chimed, flashed the session and showed nothing.** The widget treats a newer inbox item time + as a new arrival, and the server stamped decisions with their last update, so the user's own draft looked like a + new decision. Five saves in two minutes, each with a chime (decision log events 17:58 to 18:00). Fix: a decision's + item time is when it was asked (`askedTimeOf`); Save draft stages the note and the card says "Draft saved". +- 2026-10-23 21:13 — **An answer click reordered and collapsed the list under the pointer.** Every new outgoing message + scrolled the list to the bottom three times, also an answer sent from a card in the middle of the list. Only a + composer message follows now (`followsOutgoing`). Rule: a list moves only when the user's own new content lands + where the list scrolls to. +- 2026-10-23 21:13 — **A click that sends at once loses the user's chance to batch and correct.** Decision options now stage + (the decision log's `drafted` state), the inbox says how many are staged, a review shows "3 decisions will be sent + to 2 sessions", and closing with staged decisions asks Send now / Keep staged / Discard. Settings: `decisionClick`, + `stagedOnLeave`. +- 2026-10-23 23:11 — **The second recording of a session opened an invisible picker.** Every test and render made a + fresh picker panel, so all passed, while the live widget reused one: after its first dismissal the panel came back at + 0x0, the recorder waited for a choice nobody could see, and the video button stuck on Stop. Found only by running the + flow twice in the live app. Fix: a fresh hosting view per opening (`RecordingPickerPanelTests` opens it three times). + Rule: a reusable panel gets a test that opens, dismisses and opens it again. +- 2026-10-24 15:12 — **A recording stuck at "Starting" with no border, and Stop did nothing.** The first recording after a + change to the native recorder's sources compiled it for 20 to 60 s. The controls showed only a spinner, no border + was drawn (the recorder draws it once it runs), and the session CLI built synchronously, so its stdin reader was + blocked and Stop was read only after the compile. Worse, opening the picker had started that compile through the + target list, and the app's 20 s timeout killed it midway, so the recording compiled again from the start. Fix: an + async build in its own process group (a cancelled or killed caller leaves it running, the next one reuses it), a + `preparing` event that the controls show as "Preparing" with the reason, a dashed outline around the target from the + first moment, Discard answered in under a second. Rule: a start that can be slow says why, stays cancellable, and + shows what was chosen at once; never test a native flow only on a warm build (plant a one-line source change). +- 2026-10-24 15:51 — **"2 decisions were sent to 1 session" when nothing arrived.** The session ran in two Claude + processes (one resumed twice in cmux); delivery refused to pick a receiver and queued the answers, and the widget + counted every non-failure as sent. Fix: the receiver whose registry entry moved last, by 10 minutes or more, gets + the answers; a send no receiver took says "Not delivered to : " with a Send again button, for + every kind of agent (`resend-answers`). Rule: a send reports what the receiver confirmed, never what was attempted, + and every not-delivered result offers the retry in place. diff --git a/CLAUDE.md b/CLAUDE.md index 86a98a8be1..7e68071d23 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -14,6 +14,7 @@ The detailed rules live in area files. Claude Code loads an area file by itself - AI, accounts, secrets, cost, agent history (`src/utils/ai/`, `src/utils/security/`, `src/utils/agent-sessions/`, `src/ai-proxy/`, `src/ask/`, `src/claude/`, `src/codex/`): `.claude/rules/ai.md`. - Models and pricing (the catalog, the proxy ledger, Claude request rules, Codex and Grok model lists, the handoff-to rankings): `.claude/rules/models.md`. Where each model fact lives, and the checklist for a new model. - Web UI (`*.tsx`, `src/*/ui/`, `src/dashboard/`, `src/dev-dashboard/`): `.claude/rules/web-ui.md`. +- Any UI, native or web, the widget schema and notifications: `.claude/rules/opinionated-settings.md`. A behavior a reasonable user could want another way (a sound, a popup, a flash, a placement, a hover tooltip, a timing) ships as a setting with your pick as its default. - macOS app, notifications, privacy grants (`src/macos/`, `src/utils/macos/`, `src/notify/`): `.claude/rules/macos.md`. Window rules for the Swift app (`src/macos/GenesisTools/`): `.claude/rules/macos-app.md`. - The personal-data probe and the git hooks: `.claude/rules/placeholder-check.md`. - Browser automation (`src/chrome-devtools/`, `src/utils/chrome-devtools/`, which another repo vendors): `.claude/rules/chrome-devtools.md`. The plugin (`plugins/`, `.claude-plugin/`): `plugins.md`. CI, guards and git hooks (`.github/`, `scripts/ci/`, `.githooks/`): `ci.md`. Dependencies (`package.json`, `bun.lock`): `dependencies.md`. diff --git a/native/ax-tool/SnapshotSupport/FrameSampler.swift b/native/ax-tool/SnapshotSupport/FrameSampler.swift index 52becde389..9df0427006 100644 --- a/native/ax-tool/SnapshotSupport/FrameSampler.swift +++ b/native/ax-tool/SnapshotSupport/FrameSampler.swift @@ -71,11 +71,29 @@ public struct KeepPolicy { public let keep: Bool public let changePercent: Double public let reason: String + /// The frame changed past the threshold against the last kept one: what the frame rate follows. It differs + /// from `keep` only past the cap, where no still is written but motion is still measured. + public let moved: Bool + + public init(keep: Bool, changePercent: Double, reason: String, moved: Bool? = nil) { + self.keep = keep + self.changePercent = changePercent + self.reason = reason + self.moved = moved ?? keep + } } public mutating func consider(_ signature: FrameSignature) -> Decision { guard keptCount < maxFrames else { - return Decision(keep: false, changePercent: 0, reason: "cap") + // The cap stops the stills, not the motion check: a recording past it must still return to its active + // frame rate when the window moves. + let change = lastKept.map { signature.changePercent(from: $0) } ?? 100 + let moved = change >= thresholdPercent && change > 0 + if moved { + lastKept = signature + } + + return Decision(keep: false, changePercent: change, reason: "cap", moved: moved) } guard let previous = lastKept else { lastKept = signature diff --git a/native/ax-tool/Sources/Record.swift b/native/ax-tool/Sources/Record.swift index b1d5b2cd14..36f069ddd8 100644 --- a/native/ax-tool/Sources/Record.swift +++ b/native/ax-tool/Sources/Record.swift @@ -349,7 +349,7 @@ final class NativeRecorder: NSObject, SCStreamOutput, SCStreamDelegate { let signature = FrameSignature(bgra: UnsafeRawPointer(base), width: width, height: height, bytesPerRow: CVPixelBufferGetBytesPerRow(pixelBuffer), stride: 4) let decision = policy.consider(signature) - if let fps = rate.observe(kept: decision.keep) { + if let fps = rate.observe(kept: decision.moved) { applyRate(fps) } diff --git a/native/ax-tool/Tests/FrameSamplerTests.swift b/native/ax-tool/Tests/FrameSamplerTests.swift index 15394a20e7..e202d1838c 100644 --- a/native/ax-tool/Tests/FrameSamplerTests.swift +++ b/native/ax-tool/Tests/FrameSamplerTests.swift @@ -87,6 +87,28 @@ final class FrameSamplerTests: XCTestCase { XCTAssertEqual(again.reason, "cap") } + /// A one-still cap (a session recording keeps only its first PNG) must not pin the rate at idle: motion past the + /// cap still brings the active rate back. + func testMotionPastTheCapStillRestoresTheActiveRate() { + var policy = KeepPolicy(thresholdPercent: 10, maxFrames: 1) + var rate = FrameRatePolicy(activeFps: 30, idleFps: 2, stillFramesBeforeIdle: 3) + let dark = signature(frame(width: 16, height: 16) { _, _ in (0, 0, 0) }, width: 16, height: 16) + let light = signature(frame(width: 16, height: 16) { _, _ in (240, 240, 240) }, width: 16, height: 16) + XCTAssertNil(rate.observe(kept: policy.consider(dark).moved)) + for _ in 0..<3 { + _ = rate.observe(kept: policy.consider(dark).moved) + } + XCTAssertEqual(rate.currentFps, 2, "a still screen past the cap drops to idle") + + let moved = policy.consider(light) + XCTAssertFalse(moved.keep, "the cap writes no second still") + XCTAssertEqual(moved.reason, "cap") + XCTAssertTrue(moved.moved) + XCTAssertEqual(rate.observe(kept: moved.moved), 30, "motion past the cap returns to the active rate") + XCTAssertFalse(policy.consider(light).moved, "the same frame again is still") + XCTAssertEqual(policy.keptCount, 1) + } + func testTheRatePolicyDropsToIdleOnlyAfterARunOfStillFrames() { var rate = FrameRatePolicy(activeFps: 8, idleFps: 2, stillFramesBeforeIdle: 3) XCTAssertEqual(rate.currentFps, 8) diff --git a/scripts/native/widget-drive.ts b/scripts/native/widget-drive.ts index faca5aafc2..6cb125e13b 100644 --- a/scripts/native/widget-drive.ts +++ b/scripts/native/widget-drive.ts @@ -13,6 +13,7 @@ import { SafeJSON } from "@genesiscz/utils/json"; * * Commands (WidgetWindow.swift runTestCommand): `expand [group]`, `module [group]`, * `hover [group]`, `unhover [group]`, `collapse`, `select `, `settings [page]`, + * `inbox [group]` (the inbox button), `session [group]` (a rail session button), * `passive on|off`. Send `passive on` before `expand` on the user's own screen: an expanded panel otherwise takes the * keyboard from the app in front. * Edges: top, right, left. The face honours them only with staging on (`bun scripts/native/staging.ts on`) or in the diff --git a/src/control/commands/capture-session.ts b/src/control/commands/capture-session.ts new file mode 100644 index 0000000000..ce6c72ac8e --- /dev/null +++ b/src/control/commands/capture-session.ts @@ -0,0 +1,274 @@ +import { randomUUID } from "node:crypto"; +import { withInterrupt } from "@genesiscz/utils/cli/interrupt"; +import { SafeJSON } from "@genesiscz/utils/json"; +import { logger, out } from "@genesiscz/utils/logger"; +import { createBoxTable, renderCliHeader } from "@genesiscz/utils/table"; +import type { Command } from "commander"; +import { ancestorPids } from "../lib/decision/frontmost"; +import { ffmpegSegmentJoiner, nativeSegmentRecorder } from "../lib/recording-native"; +import { + parseRecordingCommand, + RECORDING_SESSION_MAX_SECONDS, + RecordingSession, + type RecordingSessionEvent, + recordingsDir, +} from "../lib/recording-session"; +import { + defaultRecordingTarget, + describeRecordingTarget, + listRecordingTargets, + recordingTargetSchema, +} from "../lib/recording-targets"; + +const { log } = logger.scoped("control-capture-session"); + +interface SessionFlags { + json?: boolean; + stopOnStdin?: boolean; + target?: string; + excludeWindowIds?: string; + fps?: string; + /** Commander's `--no-indicator`: true unless the flag is given. */ + indicator?: boolean; + maxSeconds?: string; + output?: string; +} + +function parseIds(raw: string | undefined): number[] | undefined { + if (!raw) { + return undefined; + } + + const ids = raw.split(",").map(Number); + if (ids.some((id) => !Number.isInteger(id) || id <= 0)) { + throw new Error("--exclude-window-ids takes positive CG window ids separated by commas"); + } + + return ids; +} + +function boundedNumber(raw: string | undefined, name: string, min: number, max: number): number | undefined { + if (raw === undefined) { + return undefined; + } + + const value = Number(raw); + if (!Number.isFinite(value) || value < min || value > max) { + throw new Error(`${name} must be between ${min} and ${max}`); + } + + return value; +} + +function stamp(date = new Date()): string { + const pad = (value: number) => String(value).padStart(2, "0"); + const day = `${date.getFullYear()}${pad(date.getMonth() + 1)}${pad(date.getDate())}`; + return `${day}-${pad(date.getHours())}${pad(date.getMinutes())}${pad(date.getSeconds())}`; +} + +function humanLine(event: RecordingSessionEvent): string { + switch (event.event) { + case "state": + return `${event.state} (${event.reason}) · ${describeRecordingTarget(event.target)} · ${Math.round(event.elapsedMs / 1000)} s`; + case "elapsed": + return `${Math.round(event.elapsedMs / 1000)} s`; + case "segment": + return `segment ${event.index}: ${Math.round(event.durationMs / 1000)} s`; + case "done": + return `saved ${event.path} (${Math.round(event.durationMs / 1000)} s, ${event.segments} segments)`; + case "discarded": + return "discarded"; + case "preparing": + return event.message; + case "error": + return `error: ${event.message}`; + } +} + +/** + * Reads one command per stdin line and calls `onEnd` at EOF. The returned function stops reading, so a finished + * session does not keep the process alive while its host still holds the pipe open. + */ +function readCommands({ onLine, onEnd }: { onLine: (line: string) => void; onEnd: () => void }): () => void { + const reader = Bun.stdin.stream().getReader(); + const decoder = new TextDecoder(); + let cancelled = false; + void (async () => { + let pending = ""; + try { + while (true) { + const next = await reader.read(); + if (next.done) { + break; + } + + pending += decoder.decode(next.value, { stream: true }); + let newline = pending.indexOf("\n"); + while (newline >= 0) { + const line = pending.slice(0, newline).trim(); + pending = pending.slice(newline + 1); + if (line) { + onLine(line); + } + + newline = pending.indexOf("\n"); + } + } + } catch (error) { + log.debug({ error }, "recording command input ended with an error"); + } + + if (!cancelled) { + onEnd(); + } + })(); + return () => { + cancelled = true; + void reader.cancel().catch((error: unknown) => log.debug({ error }, "recording command input cancel")); + }; +} + +export function registerCaptureSessionCommands(capture: Command): void { + capture + .command("session") + .description( + "A recording you control while it runs: JSONL commands on stdin (pause, resume, switch, restart, stop, discard), JSONL events on stdout" + ) + .option("--json", "Events as JSON lines on stdout") + .option("--stop-on-stdin", "Read commands from stdin; its end stops and keeps the recording") + .option( + "--target ", + 'What to record: {"kind":"window","windowIds":[n]}, {"kind":"app","app":"Finder"} or {"kind":"display","screenIndex":0}; default the front window' + ) + .option("--exclude-window-ids ", "CG window ids the movie never shows (the host's own controls)") + .option("--fps ", "Frame rate while something moves, 0.5–30", "15") + .option("--no-indicator", "Hide the red border around what is recorded (never in the movie either way)") + .option("--max-seconds ", `Longest recording, 1–${RECORDING_SESSION_MAX_SECONDS} s`) + .option("--output ", "Where the joined movie goes (default ~/.genesis-tools/control/recordings/)") + .action(async (flags: SessionFlags) => { + // A host that dies closes the pipe: the next write fails with EPIPE, which would end this process with + // the segments on disk and no movie. Stop instead, and keep what was recorded. + let hostGone = false; + let session: RecordingSession | undefined; + process.stdout.on("error", (error) => { + if (hostGone) { + return; + } + + hostGone = true; + log.warn({ error }, "stdout closed; the host is gone, so the recording stops and is kept"); + void session?.command({ cmd: "stop" }); + }); + const emit = (event: RecordingSessionEvent) => { + log.info({ event }, "recording event"); + if (hostGone) { + return; + } + + if (flags.json) { + out.print(`${SafeJSON.stringify(event)}\n`); + } else { + out.printlnErr(humanLine(event)); + } + }; + + try { + const target = flags.target + ? recordingTargetSchema.parse(SafeJSON.parse(flags.target, { strict: true })) + : defaultRecordingTarget(await listRecordingTargets(), new Set([...ancestorPids(), process.pid])); + if (!target) { + throw new Error("Nothing on screen to record."); + } + + session = new RecordingSession({ + target, + directory: recordingsDir(`session-${randomUUID()}`), + // The suffix keeps two sessions that start in the same second from saving over each other. + output: flags.output ?? recordingsDir(`recording-${stamp()}-${randomUUID().slice(0, 8)}.mp4`), + recorder: nativeSegmentRecorder({ + excludeWindowIds: parseIds(flags.excludeWindowIds), + fps: boundedNumber(flags.fps, "--fps", 0.5, 30), + indicator: flags.indicator, + }), + join: ffmpegSegmentJoiner, + emit, + maxSeconds: boundedNumber(flags.maxSeconds, "--max-seconds", 1, RECORDING_SESSION_MAX_SECONDS), + }); + log.info({ target, stopOnStdin: flags.stopOnStdin === true }, "recording session starting"); + } catch (error) { + emit({ event: "error", message: error instanceof Error ? error.message : String(error), fatal: true }); + process.exitCode = 1; + return; + } + + const active = session; + const outcome = await withInterrupt( + async (signal) => { + // Ctrl-C or SIGTERM keeps what was recorded: a host that quits never loses the movie. + signal.addEventListener("abort", () => void active.command({ cmd: "stop" }), { once: true }); + await active.start(); + const stopReading = flags.stopOnStdin + ? readCommands({ + onLine: (line) => { + const command = parseRecordingCommand(line); + if ("error" in command) { + emit({ event: "error", message: command.error, fatal: false }); + return; + } + + void active.command(command); + }, + onEnd: () => void active.command({ cmd: "stop" }), + }) + : undefined; + const result = await active.finished; + stopReading?.(); + return result; + }, + { handleTermination: true } + ); + log.info({ outcome }, "recording session ended"); + if (outcome === "failed") { + process.exitCode = 1; + } + }); + + capture + .command("targets") + .description("What a recording session can record: on-screen windows, their apps, and the displays") + .option("--json", "JSON on stdout") + .action(async (flags: { json?: boolean }) => { + const targets = await listRecordingTargets(); + if (flags.json) { + out.print(`${SafeJSON.stringify(targets)}\n`); + return; + } + + renderCliHeader( + "Recording targets", + `${targets.windows.length} windows · ${targets.displays.length} displays` + ); + const table = createBoxTable(["KIND", "ID", "APP", "TITLE / NAME", "SIZE"]); + for (const window of targets.windows) { + table.push([ + "window", + String(window.id), + window.app, + window.title || "—", + `${Math.round(window.bounds.width)}x${Math.round(window.bounds.height)}`, + ]); + } + + for (const display of targets.displays) { + table.push([ + "display", + String(display.index), + "", + display.name + (display.isPrimary ? " (main)" : ""), + `${display.width}x${display.height}`, + ]); + } + + out.println(table.toString()); + }); +} diff --git a/src/control/commands/capture.ts b/src/control/commands/capture.ts index 00c7271564..40bfa59e7e 100644 --- a/src/control/commands/capture.ts +++ b/src/control/commands/capture.ts @@ -9,6 +9,7 @@ import { nativeCapturePreflight } from "../lib/capture-native"; import { CAPTURE_HELP, type Plan } from "../lib/capture-plan"; import { CaptureRunError, type RunResult, runCapturePlan, runClickmap, runRecrop } from "../lib/capture-runner"; import { addCaptureFlags, type CaptureFlags, captureFromFlags, resolveCaptureEnumFlags } from "../lib/native-record"; +import { registerCaptureSessionCommands } from "./capture-session"; function fail(msg: string, exitCode = 2): never { console.error(`capture-with-actions: ${msg}`); @@ -98,6 +99,8 @@ export function registerCaptureCommands(program: Command): void { } }); + registerCaptureSessionCommands(capture); + capture .command("recrop ") .description("Re-crop frames of a PRIOR run (no re-record) — the offline reframing flow") diff --git a/src/control/lib/capture-plan.ts b/src/control/lib/capture-plan.ts index 93a383f52d..ae01faae79 100644 --- a/src/control/lib/capture-plan.ts +++ b/src/control/lib/capture-plan.ts @@ -452,6 +452,10 @@ export interface CaptureSpec { idleFps?: number; threshold?: number; videoOut?: string; + /** Most PNG frames the recorder writes beside the movie (1–5000, recorder default 800). */ + maxFrames?: number; + /** CG window ids the movie never shows, such as a host's own recording controls (native backend). */ + excludeWindowIds?: number[]; countdownSec?: number; noRemote?: boolean; captureEngine?: "cg" | "sc"; diff --git a/src/control/lib/capture-runner.ts b/src/control/lib/capture-runner.ts index 567206b411..e30c0d11db 100644 --- a/src/control/lib/capture-runner.ts +++ b/src/control/lib/capture-runner.ts @@ -58,7 +58,7 @@ import { type WindowBounds, windowShotArgv, } from "./peekaboo"; -import { ensureBinary } from "./runner"; +import { ensureBinaryAsync } from "./runner"; import { publishVitrinka } from "./vitrinka-publish"; /** Operational failure of a capture-family command; exitCode preserves the legacy script's codes. */ @@ -263,15 +263,25 @@ export function vitrinkaRefusal(input: { return undefined; } -/** `signal` is the run's cancellation; `stopGraceMs` exists for tests of the SIGINT-then-kill escalation. */ +/** + * `signal` is the run's cancellation; `stopGraceMs` exists for tests of the SIGINT-then-kill escalation. + * `onRecording` runs once the recorder wrote its first frame, which is when the movie's clock starts: arming (a + * private recorder copy, ScreenCaptureKit's enumeration, one stream per window) takes seconds, and a host that shows + * "recording" before it would count time that is not in the movie. + */ export async function runCapturePlanWithSignal({ plan, signal, stopGraceMs = RECORDER_STOP_GRACE_MS, + onRecording, + onPreparing, }: { plan: Plan; signal: AbortSignal; stopGraceMs?: number; + onRecording?: () => void; + /** The native recorder has to compile first (after a source change), which takes up to a minute. */ + onPreparing?: () => void; }): Promise { const throwIfCancelled = (stage: string) => { if (signal.aborted) { @@ -309,7 +319,12 @@ export async function runCapturePlanWithSignal({ let axTool = AX_TOOL_PATH; if (backend === "native") { validateNativeCapturePlan(plan); - axTool = ensureBinary(); + try { + axTool = await ensureBinaryAsync({ signal, onBuild: onPreparing }); + } catch (error) { + throwIfCancelled("while the recorder was being built"); + throw error; + } } using nativeControls = backend === "native" ? new NativeCaptureControls(cap) : undefined; @@ -399,6 +414,9 @@ export async function runCapturePlanWithSignal({ } // non-null: the retry block above throws when no attempt produced a dir const sessionDir = attempt.sessionDir!; + if (attempt.sessionDir && !signal.aborted) { + onRecording?.(); + } const t0 = Date.now(); const fired: FiredAction[] = []; diff --git a/src/control/lib/native-record.test.ts b/src/control/lib/native-record.test.ts index 897b8e2606..1831886ba6 100644 --- a/src/control/lib/native-record.test.ts +++ b/src/control/lib/native-record.test.ts @@ -104,6 +104,25 @@ describe("nativeCaptureArgv", () => { "/tmp/v.mp4", ]); }); + + it("passes the PNG frame cap and the windows a movie must never show", () => { + const argv = nativeCaptureArgv( + { + mode: "screen", + screenIndex: 0, + duration: 5, + maxFrames: 1, + excludeWindowIds: [71, 72], + videoOut: "/v.mp4", + }, + "/o" + ); + expect(argv.slice(-4)).toEqual(["--max-frames", "1", "--exclude-window-ids", "71,72"]); + expect(() => validateCaptureOptions({ mode: "screen", duration: 1, excludeWindowIds: [0] })).toThrow( + /excludeWindowIds/ + ); + expect(() => validateCaptureOptions({ mode: "screen", duration: 1, maxFrames: 0 })).toThrow(/maxFrames/); + }); }); describe("parseScreenList", () => { diff --git a/src/control/lib/native-record.ts b/src/control/lib/native-record.ts index 343ea56e22..fb528ce575 100644 --- a/src/control/lib/native-record.ts +++ b/src/control/lib/native-record.ts @@ -98,6 +98,14 @@ export function nativeCaptureArgv(cap: CaptureSpec, outDir: string): string[] { argv.push("--video-out", cap.videoOut); } + if (cap.maxFrames !== undefined) { + argv.push("--max-frames", String(cap.maxFrames)); + } + + if (cap.excludeWindowIds?.length) { + argv.push("--exclude-window-ids", cap.excludeWindowIds.join(",")); + } + return argv; } @@ -112,6 +120,7 @@ export function validateCaptureOptions(cap: CaptureSpec): void { ["activeFps", cap.activeFps, 0.5, 30], ["idleFps", cap.idleFps, 0.1, 5], ["threshold", cap.threshold, 0, 100], + ["maxFrames", cap.maxFrames, 1, 5000], ]; for (const [name, value, min, max] of numbers) { if (value !== undefined && (!Number.isFinite(value) || value < min || value > max)) { @@ -132,6 +141,8 @@ export function validateCaptureOptions(cap: CaptureSpec): void { cap.transparent, cap.codec, cap.indicator, + cap.maxFrames, + cap.excludeWindowIds, ]; if (cap.backend === "peekaboo" && (cap.mode === "isolated" || extended.some((value) => value !== undefined))) { throw new Error( @@ -165,6 +176,14 @@ export function validateCaptureOptions(cap: CaptureSpec): void { throw new Error("windowIds must contain positive CG window IDs"); } + if ( + cap.excludeWindowIds !== undefined && + (!Array.isArray(cap.excludeWindowIds) || + cap.excludeWindowIds.some((id) => !Number.isInteger(id) || id <= 0 || id > 0xffffffff)) + ) { + throw new Error("excludeWindowIds must contain positive CG window IDs"); + } + if ( cap.apps !== undefined && (!Array.isArray(cap.apps) || cap.apps.some((app) => typeof app !== "string" || !app.trim())) diff --git a/src/control/lib/recording-native.ts b/src/control/lib/recording-native.ts new file mode 100644 index 0000000000..4a95d5cd73 --- /dev/null +++ b/src/control/lib/recording-native.ts @@ -0,0 +1,171 @@ +import { existsSync } from "node:fs"; +import { rm, unlink } from "node:fs/promises"; +import { logger } from "@genesiscz/utils/logger"; +import { probeVideo } from "@genesiscz/utils/video/probe"; +import { runVideoCommand } from "@genesiscz/utils/video/process"; +import type { CaptureSpec } from "./capture-plan"; +import { CaptureRunError, runCapturePlanWithSignal } from "./capture-runner"; +import { captureFromFlags, validateCaptureOptions } from "./native-record"; +import { joinSegmentsArgv, type SegmentJoiner, type SegmentRecorder, type SegmentResult } from "./recording-session"; +import type { RecordingTarget } from "./recording-targets"; + +const { log } = logger.scoped("control-recording-native"); + +/** The `[screen-recording-denied]` code GenesisKit's `WidgetCaptureError` already recognizes. */ +export const SCREEN_RECORDING_DENIED = "screen-recording-denied"; + +/** + * Motion threshold for the recorder's frame-rate policy. A session keeps one PNG (`maxFrames: 1`), so the threshold + * only decides when the stream drops to the idle rate; a low one keeps typing and small edits at the active rate. + */ +const SESSION_MOTION_THRESHOLD = 0.5; + +export interface NativeRecorderOptions { + /** CG window ids no movie may show: the host's own recording controls. */ + excludeWindowIds?: number[]; + /** The recorder's frame rate while something moves (0.5–30). */ + fps?: number; + /** False hides the red border around what is recorded (it never shows in the movie either way). */ + indicator?: boolean; +} + +/** The capture spec one segment records: isolated windows or an app's windows, or a whole display. */ +export function segmentCaptureSpec({ + target, + output, + maxSeconds, + options, +}: { + target: RecordingTarget; + output: string; + maxSeconds: number; + options: NativeRecorderOptions; +}): CaptureSpec { + const duration = String(Math.max(0.1, Math.min(180, maxSeconds))); + const common = { + codec: "h264", + videoOut: output, + duration, + activeFps: options.fps === undefined ? undefined : String(options.fps), + threshold: String(SESSION_MOTION_THRESHOLD), + } as const; + let spec: CaptureSpec; + if (target.kind === "display") { + spec = { + mode: "screen", + backend: "native", + screenIndex: target.screenIndex, + duration: Number(duration), + codec: "h264", + videoOut: output, + activeFps: options.fps, + threshold: SESSION_MOTION_THRESHOLD, + }; + } else if (target.kind === "window") { + spec = captureFromFlags({ ...common, windowIds: target.windowIds.join(",") }); + } else { + spec = captureFromFlags({ ...common, includeApp: [target.app] }); + } + + spec.maxFrames = 1; + if (options.indicator === false) { + spec.indicator = false; + } + + if (options.excludeWindowIds?.length) { + spec.excludeWindowIds = options.excludeWindowIds; + } + + validateCaptureOptions(spec); + return spec; +} + +/** A failure text that names a missing Screen Recording grant carries the code a host handles itself. */ +export function refusalCode(message: string): string | undefined { + return /Screen Recording is not granted/i.test(message) ? SCREEN_RECORDING_DENIED : undefined; +} + +/** One `tools control capture record` run per segment, on the ScreenCaptureKit recorder. */ +export function nativeSegmentRecorder(options: NativeRecorderOptions = {}): SegmentRecorder { + return { + async record({ target, output, maxSeconds, signal, onRecording, onPreparing }): Promise { + const capture = segmentCaptureSpec({ target, output, maxSeconds, options }); + let sessionDir: string | undefined; + try { + const result = await runCapturePlanWithSignal({ + plan: { capture, actions: [] }, + signal, + onRecording, + onPreparing, + }); + sessionDir = result.sessionDir; + const envelope = result.capture as { + error?: unknown; + data?: { stats?: { durationMs?: unknown } }; + } | null; + const reported = envelope?.data?.stats?.durationMs; + const durationMs = typeof reported === "number" && reported > 0 ? reported : undefined; + const endedBy = signal.aborted ? "stop" : result.captureFailed ? "failure" : "deadline"; + const error = + typeof envelope?.error === "string" ? envelope.error : result.warnings.at(-1) || undefined; + // A stop can land on a recorder that already failed, or kill one that did not finish its movie in + // time, so a failed run's movie is probed whatever ended it: an unfinished one would break every save. + const ok = existsSync(output) && (!result.captureFailed || (await playable(output))); + if (!result.captureFailed) { + return { ok, endedBy, durationMs }; + } + + const code = error ? refusalCode(error) : undefined; + return { ok, endedBy, durationMs, error: code ? `[${code}] ${error}` : error, code }; + } catch (error) { + const message = + error instanceof CaptureRunError || error instanceof Error ? error.message : String(error); + const code = refusalCode(message); + return { + ok: false, + endedBy: signal.aborted ? "stop" : "failure", + error: code ? `[${code}] ${message}` : message, + code, + }; + } finally { + // The recorder writes its PNG frames and metadata into a session folder of its own; the movie is + // the product, so the folder goes with the run. + if (sessionDir) { + await rm(sessionDir, { recursive: true, force: true }); + } + } + }, + }; +} + +/** A movie a failed run left behind is kept only when ffprobe can read it. */ +async function playable(path: string): Promise { + try { + await probeVideo({ input: path }); + return true; + } catch (error) { + log.warn({ error, path }, "a failed recording left an unreadable movie; dropping it"); + try { + await unlink(path); + } catch (cleanupError) { + log.debug({ error: cleanupError, path }, "unreadable movie already gone"); + } + + return false; + } +} + +/** ffmpeg concat of the segments at the first segment's size; one H.264 MP4. */ +export const ffmpegSegmentJoiner: SegmentJoiner = async ({ segments, output }) => { + const first = await probeVideo({ input: segments[0].path }); + const argv = joinSegmentsArgv({ + inputs: segments.map((segment) => segment.path), + width: first.displayWidth, + height: first.displayHeight, + output, + }); + const totalMs = segments.reduce((sum, segment) => sum + segment.durationMs, 0); + await runVideoCommand({ command: argv, timeoutMs: Math.max(120_000, totalMs * 2) }); + const joined = await probeVideo({ input: output }); + return { durationMs: Math.round(joined.durationUs / 1000) }; +}; diff --git a/src/control/lib/recording-session.test.ts b/src/control/lib/recording-session.test.ts new file mode 100644 index 0000000000..62784802f3 --- /dev/null +++ b/src/control/lib/recording-session.test.ts @@ -0,0 +1,692 @@ +import { afterEach, beforeEach, describe, expect, it, mock, spyOn } from "bun:test"; +import { randomUUID } from "node:crypto"; +import { existsSync } from "node:fs"; +import * as fs from "node:fs/promises"; +import { mkdir, readFile, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import * as videoProbe from "@genesiscz/utils/video/probe"; +import * as captureRunner from "./capture-runner"; +import { nativeSegmentRecorder } from "./recording-native"; +import { + FIRST_FRAME_MESSAGE, + type FinishedSegment, + joinSegmentsArgv, + PREPARING_MESSAGE, + parseRecordingCommand, + RecordingSession, + type RecordingSessionEvent, + type SegmentRecorder, + type SegmentResult, +} from "./recording-session"; +import { + defaultRecordingTarget, + groupRecordingApps, + parseRecordingWindows, + type RecordingTarget, +} from "./recording-targets"; + +const finder: RecordingTarget = { kind: "app", app: "4242", name: "Finder" }; +const display: RecordingTarget = { kind: "display", screenIndex: 0, name: "Built-in" }; +const editor: RecordingTarget = { kind: "window", windowIds: [77], app: "Editor", title: "notes.txt" }; + +interface FakeRun { + target: RecordingTarget; + output: string; + maxSeconds: number; + signal: AbortSignal; + /** Ends the run by itself, as the recorder's deadline or a failure would. */ + end: (result: SegmentResult, writeMovie?: boolean) => Promise; + /** The first frame arrives (runs arm at once unless the fake was made with `armLater`). */ + arm: () => void; +} + +/** + * A recorder that writes a small file per run and finishes it when the session aborts the run. A run aborted before + * it armed writes no movie, as the real recorder cancelled while arming does. + */ +function fakeRecorder(options: { failFirst?: boolean; armLater?: boolean; prepares?: boolean } = {}) { + const runs: FakeRun[] = []; + const recorder: SegmentRecorder = { + record({ target, output, maxSeconds, signal, onRecording, onPreparing }) { + if (options.prepares) { + onPreparing?.(); + } + + return new Promise((resolve) => { + const index = runs.length; + let armed = false; + const end = async (result: SegmentResult, writeMovie = true) => { + if (writeMovie) { + await writeFile(output, `movie ${index} ${kindOf(target)}`); + } + + resolve(result); + }; + const arm = () => { + armed = true; + onRecording(); + }; + runs.push({ target, output, maxSeconds, signal, end, arm }); + if (options.failFirst && index === 0) { + void end({ ok: false, endedBy: "failure", error: "[screen-recording-denied] no grant" }, false); + return; + } + + signal.addEventListener( + "abort", + () => + void (armed + ? end({ ok: true, endedBy: "stop", durationMs: 1000 }) + : end({ ok: false, endedBy: "stop", error: "cancelled while arming" }, false)), + { once: true } + ); + if (!options.armLater) { + arm(); + } + }); + }, + }; + return { recorder, runs }; +} + +function kindOf(target: RecordingTarget): string { + return target.kind; +} + +describe("RecordingSession", () => { + let root: string; + let clock: number; + let events: RecordingSessionEvent[]; + let joined: FinishedSegment[][]; + + beforeEach(async () => { + root = join(tmpdir(), `recording-session-${randomUUID()}`); + await mkdir(root, { recursive: true }); + clock = 1_000_000; + events = []; + joined = []; + }); + + afterEach(async () => { + await rm(root, { recursive: true, force: true }); + }); + + function session( + recorder: SegmentRecorder, + extra: { + maxSeconds?: number; + segmentSeconds?: number; + slowStartMs?: number; + join?: ConstructorParameters[0]["join"]; + } = {} + ) { + return new RecordingSession({ + target: finder, + directory: join(root, "segments"), + output: join(root, "out.mp4"), + recorder, + join: async ({ segments, output }) => { + joined.push(segments); + await writeFile(output, segments.map((segment) => segment.path).join("\n")); + return { durationMs: segments.reduce((sum, segment) => sum + segment.durationMs, 0) }; + }, + emit: (event) => events.push(event), + now: () => clock, + tickMs: 0, + slowStartMs: 0, + ...extra, + }); + } + + /** The state changes past `starting`, which every run passes through while it arms. */ + const states = () => + events.flatMap((event) => + event.event === "state" && event.state !== "starting" + ? [`${event.state}:${event.reason}:${event.target.kind}`] + : [] + ); + + it("arming is not recording: no time counts until the first frame, and a pause then keeps nothing", async () => { + const { recorder, runs } = fakeRecorder({ armLater: true }); + const recording = session(recorder); + await recording.start(); + expect(events.at(-1)).toMatchObject({ event: "state", state: "starting", elapsedMs: 0 }); + clock += 2500; + expect(recording.elapsedMs()).toBe(0); + runs[0].arm(); + expect(events.at(-1)).toMatchObject({ event: "state", state: "recording", reason: "user" }); + clock += 1000; + expect(recording.elapsedMs()).toBe(1000); + await recording.command({ cmd: "pause" }); + await recording.command({ cmd: "resume" }); + clock += 3000; + await recording.command({ cmd: "pause" }); + expect(existsSync(runs[1].output)).toBe(false); + expect(recording.elapsedMs()).toBe(1000); + await recording.command({ cmd: "stop" }); + expect(await recording.finished).toBe("done"); + expect(joined).toHaveLength(0); + expect(events.find((event) => event.event === "done")).toMatchObject({ segments: 1 }); + }); + + it("pause, resume, switch and stop join every segment in order, each with its own target", async () => { + const { recorder, runs } = fakeRecorder(); + const recording = session(recorder); + await recording.start(); + clock += 4000; + await recording.command({ cmd: "pause" }); + clock += 60_000; + await recording.command({ cmd: "resume" }); + clock += 3000; + await recording.command({ cmd: "switch", target: display }); + clock += 2000; + await recording.command({ cmd: "stop" }); + + expect(await recording.finished).toBe("done"); + expect(runs.map((run) => run.target.kind)).toEqual(["app", "app", "display"]); + expect(states()).toEqual([ + "recording:user:app", + "paused:user:app", + "recording:user:app", + "recording:user:display", + "finishing:user:display", + ]); + expect(joined).toHaveLength(1); + expect(joined[0].map((segment) => segment.index)).toEqual([1, 2, 3]); + expect(joined[0].map((segment) => segment.target.kind)).toEqual(["app", "app", "display"]); + const done = events.find((event) => event.event === "done"); + expect(done).toEqual({ event: "done", path: join(root, "out.mp4"), durationMs: 3000, segments: 3 }); + expect(existsSync(join(root, "segments"))).toBe(false); + // The paused minute is not recording time. + const finishing = events.findLast((event) => event.event === "state"); + expect(finishing?.event === "state" && finishing.elapsedMs).toBe(9000); + }); + + it("one segment is moved into place without a join", async () => { + const { recorder } = fakeRecorder(); + const recording = session(recorder); + await recording.start(); + clock += 1500; + await recording.command({ cmd: "stop" }); + expect(await recording.finished).toBe("done"); + expect(joined).toHaveLength(0); + expect(await readFile(join(root, "out.mp4"), "utf8")).toBe("movie 0 app"); + }); + + it("restart drops every earlier segment and the clock", async () => { + const { recorder, runs } = fakeRecorder(); + const recording = session(recorder); + await recording.start(); + clock += 5000; + await recording.command({ cmd: "pause" }); + await recording.command({ cmd: "resume" }); + clock += 5000; + await recording.command({ cmd: "restart" }); + expect(existsSync(runs[0].output)).toBe(false); + expect(existsSync(runs[1].output)).toBe(false); + expect(recording.elapsedMs()).toBe(0); + clock += 2000; + await recording.command({ cmd: "pause" }); + await recording.command({ cmd: "resume" }); + clock += 1000; + await recording.command({ cmd: "stop" }); + expect(await recording.finished).toBe("done"); + expect(joined[0].map((segment) => segment.index)).toEqual([3, 4]); + }); + + it("a save that fails pauses the session, so a stop tries again and a discard still works", async () => { + const { recorder, runs } = fakeRecorder(); + let attempts = 0; + const recording = session(recorder, { + join: async ({ segments, output }) => { + attempts += 1; + if (attempts === 1) { + throw new Error("ffmpeg is missing"); + } + + await writeFile(output, segments.map((segment) => segment.path).join("\n")); + return { durationMs: segments.reduce((sum, segment) => sum + segment.durationMs, 0) }; + }, + }); + await recording.start(); + clock += 2000; + await recording.command({ cmd: "pause" }); + await recording.command({ cmd: "resume" }); + clock += 1000; + await recording.command({ cmd: "stop" }); + + expect(states().slice(-2)).toEqual(["finishing:user:app", "paused:recorder:app"]); + expect(events.at(-1)).toEqual({ + event: "error", + message: "The recording could not be saved: ffmpeg is missing", + fatal: false, + }); + expect(runs.every((run) => existsSync(run.output))).toBe(true); + + await recording.command({ cmd: "stop" }); + expect(await recording.finished).toBe("done"); + expect(attempts).toBe(2); + expect(existsSync(join(root, "out.mp4"))).toBe(true); + }); + + it("a join that fails after it began writing leaves no partial movie at the output", async () => { + const { recorder } = fakeRecorder(); + const recording = session(recorder, { + join: async ({ output }) => { + await writeFile(output, "half a movie"); + throw new Error("ffmpeg died"); + }, + }); + await recording.start(); + await recording.command({ cmd: "pause" }); + await recording.command({ cmd: "resume" }); + await recording.command({ cmd: "stop" }); + + expect(existsSync(join(root, "out.mp4"))).toBe(false); + await recording.command({ cmd: "discard" }); + expect(await recording.finished).toBe("discarded"); + }); + + it("a failed join keeps a movie already at the output and leaves no temporary file", async () => { + const { recorder } = fakeRecorder(); + await writeFile(join(root, "out.mp4"), "previous movie"); + const recording = session(recorder, { + join: async ({ output }) => { + await writeFile(output, "half a movie"); + throw new Error("ffmpeg died"); + }, + }); + await recording.start(); + await recording.command({ cmd: "pause" }); + await recording.command({ cmd: "resume" }); + await recording.command({ cmd: "stop" }); + + expect(await readFile(join(root, "out.mp4"), "utf8")).toBe("previous movie"); + expect((await fs.readdir(root)).filter((name) => name.startsWith(".recording-"))).toEqual([]); + await recording.command({ cmd: "discard" }); + expect(await recording.finished).toBe("discarded"); + }); + + it("a session whose folder cannot be made ends as failed instead of waiting forever", async () => { + const blocker = join(root, "blocker"); + await writeFile(blocker, "a file where the folder should go"); + const { recorder, runs } = fakeRecorder(); + const recording = new RecordingSession({ + target: finder, + directory: join(blocker, "segments"), + output: join(root, "out.mp4"), + recorder, + join: async () => ({ durationMs: 0 }), + emit: (event) => events.push(event), + tickMs: 0, + slowStartMs: 0, + }); + await recording.start(); + + expect(await recording.finished).toBe("failed"); + expect(runs).toHaveLength(0); + expect(events.some((event) => event.event === "error" && event.fatal === true)).toBe(true); + }); + + it("a single segment on another volume is copied next to the output and renamed there", async () => { + const { recorder } = fakeRecorder(); + const recording = session(recorder); + const originalRename = fs.rename; + const moves: string[] = []; + const renameSpy = spyOn(fs, "rename").mockImplementation(async (source, destination) => { + moves.push(String(source)); + if (String(source).includes("segments")) { + throw Object.assign(new Error("cross-device move"), { code: "EXDEV" }); + } + + return originalRename(source, destination); + }); + try { + await recording.start(); + await recording.command({ cmd: "stop" }); + expect(events.at(-1)).toMatchObject({ event: "done", segments: 1 }); + expect(await recording.finished).toBe("done"); + expect(moves).toHaveLength(2); + expect(moves[1]).toStartWith(join(root, ".recording-")); + expect(await readFile(join(root, "out.mp4"), "utf8")).toBe("movie 0 app"); + expect(existsSync(join(root, "segments"))).toBe(false); + } finally { + renameSpy.mockRestore(); + } + }); + + it("a folder that will not go after the save still ends the session as done", async () => { + const { recorder } = fakeRecorder(); + const recording = session(recorder); + const cleanup = spyOn(fs, "rm").mockRejectedValue(new Error("cleanup denied")); + try { + await recording.start(); + await recording.command({ cmd: "stop" }); + expect(events.at(-1)).toMatchObject({ event: "done" }); + expect(await recording.finished).toBe("done"); + expect(await readFile(join(root, "out.mp4"), "utf8")).toBe("movie 0 app"); + } finally { + cleanup.mockRestore(); + } + }); + + it("a failed cross-volume copy keeps the segment and the old output for a retry", async () => { + const { recorder, runs } = fakeRecorder(); + const recording = session(recorder); + await writeFile(join(root, "out.mp4"), "previous movie"); + const move = spyOn(fs, "rename").mockRejectedValue(Object.assign(new Error("cross-device"), { code: "EXDEV" })); + const copy = spyOn(fs, "copyFile").mockImplementation(async (_source, destination) => { + await writeFile(destination, "partial movie"); + throw new Error("disk full"); + }); + try { + await recording.start(); + await recording.command({ cmd: "stop" }); + expect(events.findLast((event) => event.event === "state")).toMatchObject({ state: "paused" }); + expect(existsSync(runs[0].output)).toBe(true); + expect(await readFile(join(root, "out.mp4"), "utf8")).toBe("previous movie"); + expect((await fs.readdir(root)).filter((name) => name.startsWith(".recording-"))).toEqual([]); + } finally { + move.mockRestore(); + copy.mockRestore(); + } + + await recording.command({ cmd: "stop" }); + expect(await recording.finished).toBe("done"); + expect(await readFile(join(root, "out.mp4"), "utf8")).toBe("movie 0 app"); + }); + + it("a discard after a failed save deletes every segment", async () => { + const { recorder, runs } = fakeRecorder(); + const recording = session(recorder, { + join: async () => { + throw new Error("disk full"); + }, + }); + await recording.start(); + await recording.command({ cmd: "pause" }); + await recording.command({ cmd: "resume" }); + await recording.command({ cmd: "stop" }); + await recording.command({ cmd: "discard" }); + + expect(await recording.finished).toBe("discarded"); + expect(runs.every((run) => !existsSync(run.output))).toBe(true); + expect(existsSync(join(root, "segments"))).toBe(false); + }); + + it("discard deletes every segment and writes no movie", async () => { + const { recorder, runs } = fakeRecorder(); + const recording = session(recorder); + await recording.start(); + await recording.command({ cmd: "pause" }); + await recording.command({ cmd: "switch", target: editor }); + await recording.command({ cmd: "discard" }); + expect(await recording.finished).toBe("discarded"); + expect(events.at(-1)).toEqual({ event: "discarded" }); + expect(runs.every((run) => !existsSync(run.output))).toBe(true); + expect(existsSync(join(root, "out.mp4"))).toBe(false); + expect(existsSync(join(root, "segments"))).toBe(false); + }); + + it("a run that reaches its deadline is followed by the next, and the limit pauses the session", async () => { + const { recorder, runs } = fakeRecorder(); + const recording = session(recorder, { maxSeconds: 5, segmentSeconds: 3 }); + await recording.start(); + expect(runs[0].maxSeconds).toBe(3); + clock += 3000; + await runs[0].end({ ok: true, endedBy: "deadline", durationMs: 3000 }); + await Bun.sleep(0); + await recording.command({ cmd: "resume" }); + expect(runs).toHaveLength(2); + expect(runs[1].target).toEqual(finder); + expect(runs[1].maxSeconds).toBe(2); + clock += 2000; + await runs[1].end({ ok: true, endedBy: "deadline", durationMs: 2000 }); + await Bun.sleep(0); + await recording.command({ cmd: "resume" }); + expect(runs).toHaveLength(2); + expect(states()).toEqual([ + "recording:user:app", + "recording:recorder:app", + "paused:limit:app", + "paused:limit:app", + ]); + const refusal = events.findLast((event) => event.event === "error"); + expect(refusal).toMatchObject({ code: "limit", fatal: false }); + await recording.command({ cmd: "stop" }); + expect(await recording.finished).toBe("done"); + expect(joined[0].map((segment) => segment.durationMs)).toEqual([3000, 2000]); + }); + + it("a first run that fails without a movie ends the session with the refusal code", async () => { + const { recorder } = fakeRecorder({ failFirst: true }); + const recording = session(recorder); + await recording.start(); + expect(await recording.finished).toBe("failed"); + expect(events.at(-1)).toMatchObject({ event: "error", fatal: true }); + expect(events.at(-1)?.event === "error" && events.at(-1)).toMatchObject({ + message: expect.stringContaining("[screen-recording-denied]"), + }); + await recording.command({ cmd: "stop" }); + expect(events.filter((event) => event.event === "state" && event.state === "finishing")).toHaveLength(0); + }); + + it("a later run that fails pauses the session and keeps what was recorded", async () => { + const { recorder, runs } = fakeRecorder(); + const recording = session(recorder); + await recording.start(); + clock += 2000; + await recording.command({ cmd: "switch", target: editor }); + await runs[1].end({ ok: false, endedBy: "failure", error: "window 77 is not visible" }, false); + await Bun.sleep(0); + await recording.command({ cmd: "pause" }); + expect(states().at(-1)).toBe("paused:recorder:window"); + expect(events.findLast((event) => event.event === "error")).toMatchObject({ + message: "window 77 is not visible", + fatal: false, + }); + await recording.command({ cmd: "stop" }); + expect(await recording.finished).toBe("done"); + expect(await readFile(join(root, "out.mp4"), "utf8")).toBe("movie 0 app"); + }); + it("a stop before the first frame keeps no movie, even when the recorder left an empty file", async () => { + const recorder: SegmentRecorder = { + record: ({ output, signal }) => + new Promise((resolve) => { + signal.addEventListener( + "abort", + () => void writeFile(output, "").then(() => resolve({ ok: true, endedBy: "stop" })), + { once: true } + ); + }), + }; + const recording = session(recorder); + await recording.start(); + await recording.command({ cmd: "stop" }); + expect(await recording.finished).toBe("failed"); + expect(events.some((event) => event.event === "done")).toBe(false); + expect(events.at(-1)).toMatchObject({ event: "error", fatal: true }); + expect(existsSync(join(root, "out.mp4"))).toBe(false); + }); + + it("says it still waits for the first frame when no compile explains a slow start", async () => { + const { recorder } = fakeRecorder({ armLater: true }); + const recording = session(recorder, { slowStartMs: 20 }); + await recording.start(); + await Bun.sleep(60); + expect(events.filter((event) => event.event === "preparing")).toEqual([ + { event: "preparing", message: FIRST_FRAME_MESSAGE }, + ]); + await recording.command({ cmd: "discard" }); + + const prepared = fakeRecorder({ armLater: true, prepares: true }); + events = []; + const compiling = session(prepared.recorder, { slowStartMs: 20 }); + await compiling.start(); + await Bun.sleep(60); + expect(events.filter((event) => event.event === "preparing").map((event) => event.message)).toEqual([ + PREPARING_MESSAGE, + ]); + await compiling.command({ cmd: "discard" }); + }); + + it("says why a start is slow, and a discard while the recorder prepares ends the session at once", async () => { + const { recorder } = fakeRecorder({ prepares: true, armLater: true }); + const recording = session(recorder); + await recording.start(); + expect(events.filter((event) => event.event === "preparing")).toEqual([ + { event: "preparing", message: PREPARING_MESSAGE }, + ]); + + await recording.command({ cmd: "discard" }); + expect(await recording.finished).toBe("discarded"); + expect(existsSync(join(root, "out.mp4"))).toBe(false); + }); +}); + +describe("nativeSegmentRecorder", () => { + let root: string; + + beforeEach(async () => { + root = join(tmpdir(), `recording-native-${randomUUID()}`); + await mkdir(root, { recursive: true }); + }); + + afterEach(async () => { + mock.restore(); + await rm(root, { recursive: true, force: true }); + }); + + /** One recorder run that a stop ends; `failed` is what the recorder reported when it exited. */ + async function stoppedRun(failed: boolean) { + const output = join(root, "segment.mp4"); + spyOn(captureRunner, "runCapturePlanWithSignal").mockImplementation(async ({ signal, onRecording }) => { + await writeFile(output, "movie"); + onRecording?.(); + if (!signal.aborted) { + await new Promise((resolve) => signal.addEventListener("abort", () => resolve(), { once: true })); + } + + const sessionDir = join(root, "session"); + await mkdir(sessionDir, { recursive: true }); + return { + ...({} as Awaited>), + captureFailed: failed, + sessionDir, + warnings: failed ? ["video writer did not finish within 10s"] : [], + capture: { error: failed ? "video writer ended without completing" : undefined }, + }; + }); + const probe = spyOn(videoProbe, "probeVideo").mockImplementation(async () => { + throw new Error("no finished container"); + }); + const controller = new AbortController(); + const run = nativeSegmentRecorder().record({ + target: display, + output, + maxSeconds: 3, + signal: controller.signal, + onRecording: () => controller.abort(), + }); + return { result: await run, output, probes: probe.mock.calls.length }; + } + + it("a stop on a recorder that failed probes its movie and drops an unfinished one", async () => { + const { result, output, probes } = await stoppedRun(true); + expect(probes).toBe(1); + expect(result.ok).toBe(false); + expect(result.endedBy).toBe("stop"); + expect(result.error).toBe("video writer ended without completing"); + expect(existsSync(output)).toBe(false); + }); + + it("a clean stop keeps its movie without a probe", async () => { + const { result, output, probes } = await stoppedRun(false); + expect(probes).toBe(0); + expect(result).toEqual({ ok: true, endedBy: "stop", durationMs: undefined }); + expect(existsSync(output)).toBe(true); + }); +}); + +describe("joinSegmentsArgv", () => { + it("scales and pads every input to the first segment's even size and concatenates them", () => { + const argv = joinSegmentsArgv({ inputs: ["/a.mp4", "/b.mp4"], width: 1281, height: 721, output: "/o.mp4" }); + expect(argv.slice(0, 6)).toEqual(["ffmpeg", "-hide_banner", "-nostdin", "-loglevel", "error", "-y"]); + expect(argv).toContain("-nostdin"); + expect(argv.filter((value) => value === "-i")).toHaveLength(2); + const graph = argv[argv.indexOf("-filter_complex") + 1]; + expect(graph).toBe( + "[0:v]scale=1280:720:force_original_aspect_ratio=decrease,pad=1280:720:(ow-iw)/2:(oh-ih)/2:color=black,setsar=1,format=yuv420p,settb=AVTB[v0];" + + "[1:v]scale=1280:720:force_original_aspect_ratio=decrease,pad=1280:720:(ow-iw)/2:(oh-ih)/2:color=black,setsar=1,format=yuv420p,settb=AVTB[v1];" + + "[v0][v1]concat=n=2:v=1:a=0[out]" + ); + expect(argv.slice(-9)).toEqual([ + "-crf", + "20", + "-pix_fmt", + "yuv420p", + "-fps_mode", + "vfr", + "-movflags", + "+faststart", + "/o.mp4", + ]); + }); +}); + +describe("recording commands and targets", () => { + it("reads one command per line and names a line it cannot use", () => { + expect(parseRecordingCommand('{"cmd":"pause"}')).toEqual({ cmd: "pause" }); + expect(parseRecordingCommand('{"cmd":"switch","target":{"kind":"display","screenIndex":1}}')).toEqual({ + cmd: "switch", + target: { kind: "display", screenIndex: 1 }, + }); + expect(parseRecordingCommand("pause")).toMatchObject({ error: expect.stringContaining("Not a JSON") }); + expect(parseRecordingCommand('{"cmd":"switch","target":{"kind":"window","windowIds":[]}}')).toMatchObject({ + error: expect.stringContaining("Unknown command"), + }); + }); + + it("lists windows without system chrome or slivers, groups apps and picks the frontmost app's window", () => { + const raw = { + ok: true, + frontmostPid: 20, + windows: [ + { pid: 1, app: "Dock", windowId: 5, bounds: { x: 0, y: 0, width: 1000, height: 80 } }, + { pid: 9, app: "Terminal", windowId: 6, title: "zsh", bounds: { x: 0, y: 0, width: 800, height: 500 } }, + { + pid: 20, + app: "Editor", + windowId: 7, + title: "a.txt", + bounds: { x: 10, y: 10, width: 600, height: 400 }, + }, + { pid: 20, app: "Editor", windowId: 8, bounds: { x: 0, y: 0, width: 30, height: 30 } }, + { + pid: 20, + app: "Editor", + windowId: 9, + title: "b.txt", + bounds: { x: 50, y: 50, width: 600, height: 400 }, + }, + { pid: 30, app: "Viewer", title: "missing id", bounds: { x: 0, y: 0, width: 600, height: 400 } }, + ], + }; + const windows = parseRecordingWindows(raw); + expect(windows.map((window) => window.id)).toEqual([6, 7, 9]); + expect(groupRecordingApps(windows)).toEqual([ + { pid: 9, name: "Terminal", windowCount: 1 }, + { pid: 20, name: "Editor", windowCount: 2 }, + ]); + const targets = { windows, apps: groupRecordingApps(windows), displays: [], frontmostPid: 9 }; + // The frontmost app is this command's own terminal, so the pick falls through to the next app. + expect(defaultRecordingTarget(targets, new Set([9]))).toEqual({ + kind: "window", + windowIds: [7], + app: "Editor", + title: "a.txt", + pid: 20, + }); + }); +}); diff --git a/src/control/lib/recording-session.ts b/src/control/lib/recording-session.ts new file mode 100644 index 0000000000..7286d14c38 --- /dev/null +++ b/src/control/lib/recording-session.ts @@ -0,0 +1,632 @@ +import { randomUUID } from "node:crypto"; +import { existsSync } from "node:fs"; +import { copyFile, mkdir, rename, rm, unlink } from "node:fs/promises"; +import { dirname, join } from "node:path"; +import { SafeJSON } from "@genesiscz/utils/json"; +import { logger } from "@genesiscz/utils/logger"; +import { toolDataDir } from "@genesiscz/utils/storage/root"; +import { z } from "zod"; +import { type RecordingTarget, recordingTargetSchema } from "./recording-targets"; + +const { log } = logger.scoped("control-recording-session"); + +/** + * The longest movie a session records, in seconds: the widget's video import (`probeVideo`) refuses anything longer + * than ten minutes, and both hosts hand the movie to it. At the limit the session pauses and waits for the person. + */ +export const RECORDING_SESSION_MAX_SECONDS = 600; + +/** The recorder stops a run after 180 s (`validateCaptureOptions`), so a longer recording is a chain of runs. */ +export const RECORDING_SEGMENT_MAX_SECONDS = 180; + +export const PREPARING_MESSAGE = + "Building the recorder after an update. This happens once and can take up to a minute. Discard cancels it."; + +/** Said when a segment has no first frame after `slowStartMs` and no compile explains it. */ +export const FIRST_FRAME_MESSAGE = + "Still waiting for the first frame. macOS sends none while the screen is locked or the window is hidden or minimized. Discard cancels it."; + +const SLOW_START_MS = 8000; + +export const recordingCommandSchema = z.discriminatedUnion("cmd", [ + z.object({ cmd: z.literal("pause") }), + z.object({ cmd: z.literal("resume") }), + z.object({ cmd: z.literal("restart") }), + z.object({ cmd: z.literal("stop") }), + z.object({ cmd: z.literal("discard") }), + z.object({ cmd: z.literal("switch"), target: recordingTargetSchema }), +]); + +export type RecordingCommand = z.infer; + +/** One stdin line as a session command, or the reason it is not one. */ +export function parseRecordingCommand(line: string): RecordingCommand | { error: string } { + let value: unknown; + try { + value = SafeJSON.parse(line, { strict: true }); + } catch (error) { + log.debug({ error, line: line.slice(0, 200) }, "recording command is not JSON"); + return { error: `Not a JSON command: ${line.slice(0, 200)}` }; + } + + const parsed = recordingCommandSchema.safeParse(value); + return parsed.success ? parsed.data : { error: `Unknown command: ${line.slice(0, 200)}` }; +} + +/** `starting`: a recorder run was asked for and has not written its first frame yet; nothing is in the movie. */ +export type RecordingState = "starting" | "recording" | "paused" | "finishing"; + +/** Why the session is in its state: the person asked, the length limit, or the recorder stopped by itself. */ +export type RecordingStateReason = "user" | "limit" | "recorder"; + +export type RecordingSessionEvent = + | { + event: "state"; + state: RecordingState; + reason: RecordingStateReason; + target: RecordingTarget; + elapsedMs: number; + segments: number; + limitMs: number; + } + | { event: "elapsed"; elapsedMs: number; limitMs: number } + | { event: "segment"; index: number; path: string; durationMs: number; target: RecordingTarget } + | { event: "done"; path: string; durationMs: number; segments: number } + | { event: "discarded" } + /** The recorder is not running yet for a known slow reason (it compiles after an update); `starting` stays. */ + | { event: "preparing"; message: string } + | { event: "error"; message: string; code?: string; fatal: boolean }; + +/** How one recorder run ended: asked to stop, reached its own deadline, or failed. */ +export type SegmentEnd = "stop" | "deadline" | "failure"; + +export interface SegmentResult { + /** True when the movie at `output` exists and is complete. */ + ok: boolean; + endedBy: SegmentEnd; + /** The recorder's own measure of the movie, when it reported one. */ + durationMs?: number; + error?: string; + /** A stable code for a refusal a host handles itself (`screen-recording-denied`). */ + code?: string; +} + +/** + * One recorder run: records `target` into `output` until `maxSeconds` or until `signal` aborts, then finalizes the + * movie. `onRecording` runs once, when the first frame is in. Resolves after the recorder exited; never rejects for + * a recording problem (that is `endedBy: "failure"`). + */ +export interface SegmentRecorder { + record(input: { + target: RecordingTarget; + output: string; + maxSeconds: number; + signal: AbortSignal; + onRecording: () => void; + /** The recorder needs a slow step before it can start (a compile after an update). */ + onPreparing?: () => void; + }): Promise; +} + +export interface FinishedSegment { + index: number; + path: string; + durationMs: number; + target: RecordingTarget; +} + +/** Joins finished segments into `output`, one H.264 MP4, and returns the joined movie's length. */ +export type SegmentJoiner = (input: { segments: FinishedSegment[]; output: string }) => Promise<{ durationMs: number }>; + +export type RecordingOutcome = "done" | "discarded" | "failed"; + +interface ActiveSegment { + index: number; + path: string; + target: RecordingTarget; + maxSeconds: number; + /** When the first frame came in; unset while the recorder arms. */ + startedAt?: number; + /** The reason the `recording` state will carry once the first frame is in. */ + reason: RecordingStateReason; + controller: AbortController; + /** Set before this session aborts the run, so its end is not mistaken for a rollover or a failure. */ + stopping: boolean; + /** The recorder said why its start is slow (a compile): the first-frame hint stays quiet. */ + preparing?: boolean; + /** Fires the first-frame hint; cleared by the first frame or the run's end. */ + slowStart?: ReturnType; + result: Promise; +} + +export function recordingsDir(...segments: string[]): string { + return toolDataDir("control", "recordings", ...segments); +} + +/** + * A recording made of segments, one recorder run each. Pause ends the current segment and resume starts a new one + * with the same target; switch ends it and starts one with the new target; restart drops every segment and starts + * again; stop joins the segments into one movie; discard deletes them. A run that ends by itself at its 180 s + * deadline is followed by the next one at once. Commands run one at a time, in arrival order. + */ +export class RecordingSession { + private state: RecordingState | "ended" = "paused"; + private target: RecordingTarget; + private readonly segments: FinishedSegment[] = []; + private current: ActiveSegment | undefined; + private counter = 0; + /** Wall time of every finished segment, capped at its run's own limit. */ + private recordedMs = 0; + private queue: Promise = Promise.resolve(); + private ticker: ReturnType | undefined; + private settle: (outcome: RecordingOutcome) => void = () => {}; + /** Resolves once with how the session ended. */ + readonly finished: Promise; + private readonly limitMs: number; + private readonly segmentSeconds: number; + private readonly now: () => number; + + constructor( + private readonly options: { + target: RecordingTarget; + /** The segment folder; created on start, removed when the session ends. */ + directory: string; + /** Where the joined movie goes. */ + output: string; + recorder: SegmentRecorder; + join: SegmentJoiner; + emit: (event: RecordingSessionEvent) => void; + maxSeconds?: number; + segmentSeconds?: number; + now?: () => number; + /** Milliseconds between `elapsed` events while recording; 0 sends none. */ + tickMs?: number; + /** A segment with no first frame after this long says so (`FIRST_FRAME_MESSAGE`); 0 never does. */ + slowStartMs?: number; + } + ) { + this.target = options.target; + this.limitMs = Math.round((options.maxSeconds ?? RECORDING_SESSION_MAX_SECONDS) * 1000); + this.segmentSeconds = Math.min(options.segmentSeconds ?? RECORDING_SEGMENT_MAX_SECONDS, 180); + this.now = options.now ?? Date.now; + this.finished = new Promise((resolve) => { + this.settle = resolve; + }); + } + + /** Starts the first segment. A first segment that fails without a movie ends the session as `failed`. */ + start(): Promise { + return this.enqueue(async () => { + try { + await mkdir(this.options.directory, { recursive: true }); + } catch (error) { + log.warn({ error }, "could not create recording directory"); + this.end("failed"); + this.options.emit({ event: "error", message: messageOf(error), fatal: true }); + return; + } + + this.beginSegment("user"); + const tick = this.options.tickMs ?? 1000; + if (tick > 0) { + this.ticker = setInterval(() => { + if (this.state === "recording") { + this.options.emit({ event: "elapsed", elapsedMs: this.elapsedMs(), limitMs: this.limitMs }); + } + }, tick); + } + }); + } + + command(command: RecordingCommand): Promise { + return this.enqueue(() => this.apply(command)); + } + + elapsedMs(): number { + return this.recordedMs + (this.current ? this.liveMs(this.current) : 0); + } + + /** Time a run has been in the movie: none while it arms, never past its own limit. */ + private liveMs(run: ActiveSegment): number { + if (run.startedAt === undefined) { + return 0; + } + + return Math.min(Math.max(0, this.now() - run.startedAt), run.maxSeconds * 1000); + } + + private enqueue(work: () => Promise | void): Promise { + const next = this.queue.then(async () => { + if (this.state === "ended") { + return; + } + + try { + await work(); + } catch (error) { + log.warn({ error }, "recording session step failed"); + this.options.emit({ event: "error", message: messageOf(error), fatal: false }); + } + }); + this.queue = next; + return next; + } + + private async apply(command: RecordingCommand): Promise { + log.info({ command: command.cmd, state: this.state, segments: this.segments.length }, "recording command"); + if (this.state === "finishing") { + this.options.emit({ event: "error", message: "The recording is already being saved.", fatal: false }); + return; + } + + switch (command.cmd) { + case "pause": + if (this.state === "recording" || this.state === "starting") { + await this.stopSegment(); + this.setState("paused", "user"); + } + + return; + case "resume": + if (this.state === "paused") { + this.resumeOrRefuse(this.target); + } + + return; + case "switch": + await this.stopSegment(); + this.target = command.target; + this.resumeOrRefuse(command.target); + return; + case "restart": + await this.stopSegment(); + await this.deleteSegments(); + this.recordedMs = 0; + this.beginSegment("user"); + return; + case "stop": + await this.stopSegment(); + await this.finish(); + return; + case "discard": + await this.stopSegment(); + await this.deleteSegments(); + await this.removeDirectory(); + this.options.emit({ event: "discarded" }); + this.end("discarded"); + return; + } + } + + private resumeOrRefuse(target: RecordingTarget): void { + if (this.recordedMs >= this.limitMs) { + this.setState("paused", "limit"); + this.options.emit({ + event: "error", + message: `The recording reached its ${formatLimit(this.limitMs)} limit. Stop to keep it, or restart.`, + code: "limit", + fatal: false, + }); + return; + } + + this.target = target; + this.beginSegment("user"); + } + + private beginSegment(reason: RecordingStateReason): void { + this.counter += 1; + const index = this.counter; + const path = join(this.options.directory, `segment-${String(index).padStart(3, "0")}.mp4`); + const maxSeconds = Math.min(this.segmentSeconds, (this.limitMs - this.recordedMs) / 1000); + const controller = new AbortController(); + const target = this.target; + const run: ActiveSegment = { + index, + path, + target, + maxSeconds, + reason, + controller, + stopping: false, + result: Promise.resolve({ ok: false, endedBy: "failure" }), + }; + this.current = run; + log.info({ index, target, maxSeconds }, "recording segment requested"); + this.setState("starting", reason); + run.result = this.options.recorder + .record({ + target, + output: path, + maxSeconds, + signal: controller.signal, + onRecording: () => this.markRecording(run), + onPreparing: () => { + run.preparing = true; + this.options.emit({ event: "preparing", message: PREPARING_MESSAGE }); + }, + }) + .catch((error: unknown) => ({ ok: false, endedBy: "failure" as const, error: messageOf(error) })); + const slowStartMs = this.options.slowStartMs ?? SLOW_START_MS; + if (slowStartMs > 0) { + run.slowStart = setTimeout(() => { + if (this.current === run && run.startedAt === undefined && !run.stopping && !run.preparing) { + this.options.emit({ event: "preparing", message: FIRST_FRAME_MESSAGE }); + } + }, slowStartMs); + } + + void run.result.then((result) => { + clearTimeout(run.slowStart); + if (!run.stopping) { + void this.enqueue(() => this.endedByItself(run, result)); + } + }); + } + + /** The run's first frame is in: from here its time is in the movie. */ + private markRecording(run: ActiveSegment): void { + if (this.current !== run || run.stopping || run.startedAt !== undefined) { + return; + } + + run.startedAt = this.now(); + clearTimeout(run.slowStart); + log.info({ index: run.index }, "recording segment started"); + this.setState("recording", run.reason); + } + + private async stopSegment(): Promise { + const run = this.current; + if (!run) { + return; + } + + run.stopping = true; + run.controller.abort(); + const result = await run.result; + this.current = undefined; + this.record(run, result); + } + + /** Keeps a finished run's movie, or says why there is none. */ + private record(run: ActiveSegment, result: SegmentResult): void { + const wallMs = this.liveMs(run); + if (run.startedAt === undefined || !existsSync(run.path)) { + if (result.endedBy === "failure" || !result.ok || run.startedAt === undefined) { + log.warn({ index: run.index, error: result.error }, "recording segment produced no movie"); + } + + // A run whose first frame never came can still leave an empty file (a stop while it waited, a locked + // screen); joined, it made a 0-byte movie reported as done. + if (existsSync(run.path)) { + void unlink(run.path).catch((error: unknown) => { + log.debug({ error, path: run.path }, "could not remove a frameless segment"); + }); + } + + return; + } + + const segment: FinishedSegment = { + index: run.index, + path: run.path, + durationMs: result.durationMs ?? wallMs, + target: run.target, + }; + this.segments.push(segment); + this.recordedMs += wallMs; + this.options.emit({ event: "segment", ...segment }); + } + + private async endedByItself(run: ActiveSegment, result: SegmentResult): Promise { + if (this.current !== run) { + return; + } + + this.current = undefined; + this.record(run, result); + if (result.endedBy === "deadline" && result.ok) { + if (this.recordedMs >= this.limitMs) { + log.info({ recordedMs: this.recordedMs }, "recording reached its limit"); + this.setState("paused", "limit"); + return; + } + + this.beginSegment("recorder"); + return; + } + + const message = result.error?.trim() || "The recorder stopped by itself."; + if (this.segments.length === 0 && this.counter === 1) { + this.options.emit({ event: "error", message, code: result.code, fatal: true }); + await this.removeDirectory(); + this.end("failed"); + return; + } + + this.setState("paused", "recorder"); + this.options.emit({ event: "error", message, code: result.code, fatal: false }); + } + + private async finish(): Promise { + this.setState("finishing", "user"); + if (this.segments.length === 0) { + this.options.emit({ event: "error", message: "Nothing was recorded.", fatal: true }); + await this.removeDirectory(); + this.end("failed"); + return; + } + + const ordered = [...this.segments].sort((a, b) => a.index - b.index); + let durationMs: number; + try { + if (ordered.length === 1) { + await this.moveMovie(ordered[0].path); + durationMs = ordered[0].durationMs; + } else { + durationMs = await this.joinMovie(ordered); + } + } catch (error) { + // The segments are still on disk: a stop tries the save again and a discard still works. + log.warn({ error, output: this.options.output, segments: ordered.length }, "recording could not be saved"); + this.setState("paused", "recorder"); + this.options.emit({ + event: "error", + message: `The recording could not be saved: ${messageOf(error)}`, + fatal: false, + }); + return; + } + + await this.removeDirectory(); + log.info({ output: this.options.output, segments: ordered.length, durationMs }, "recording saved"); + this.options.emit({ event: "done", path: this.options.output, durationMs, segments: ordered.length }); + this.end("done"); + } + + private async deleteSegments(): Promise { + for (const segment of this.segments.splice(0)) { + try { + await unlink(segment.path); + } catch (error) { + log.debug({ error, path: segment.path }, "recording segment already gone"); + } + } + } + + /** + * Puts a single segment at the output. A rename across volumes fails with EXDEV, so the movie is copied next to + * the output first and renamed there: the output is either the old file or the whole new movie, never a part. + */ + private async moveMovie(source: string): Promise { + try { + await rename(source, this.options.output); + } catch (error) { + if (!(error instanceof Error) || !("code" in error) || error.code !== "EXDEV") { + throw error; + } + + const temporary = this.temporaryOutput(); + try { + await copyFile(source, temporary); + await rename(temporary, this.options.output); + } finally { + await unlink(temporary).catch((cleanupError: unknown) => { + log.debug({ error: cleanupError, temporary }, "recording temporary file already gone"); + }); + } + } + } + + /** + * Joins the segments into a file next to the output and renames it into place, so a join that fails part way + * leaves no partial movie and never touches a file already at the output. + */ + private async joinMovie(ordered: FinishedSegment[]): Promise { + const temporary = this.temporaryOutput(); + try { + const { durationMs } = await this.options.join({ segments: ordered, output: temporary }); + await rename(temporary, this.options.output); + return durationMs; + } finally { + await unlink(temporary).catch((cleanupError: unknown) => { + log.debug({ error: cleanupError, temporary }, "recording temporary file already gone"); + }); + } + } + + private temporaryOutput(): string { + return join(dirname(this.options.output), `.recording-${randomUUID()}.mp4`); + } + + /** The movie is saved by now: a folder that will not go is logged, and the session still ends. */ + private async removeDirectory(): Promise { + try { + await rm(this.options.directory, { recursive: true, force: true }); + } catch (error) { + log.warn({ error, directory: this.options.directory }, "could not clean up recording directory"); + } + } + + private setState(state: RecordingState, reason: RecordingStateReason): void { + this.state = state; + this.options.emit({ + event: "state", + state, + reason, + target: this.target, + elapsedMs: this.elapsedMs(), + segments: this.segments.length, + limitMs: this.limitMs, + }); + } + + private end(outcome: RecordingOutcome): void { + this.state = "ended"; + clearInterval(this.ticker); + this.ticker = undefined; + this.settle(outcome); + } +} + +function messageOf(error: unknown): string { + return error instanceof Error ? error.message : String(error); +} + +function formatLimit(ms: number): string { + const seconds = Math.round(ms / 1000); + return seconds % 60 === 0 ? `${seconds / 60} min` : `${seconds} s`; +} + +/** + * ffmpeg arguments that join segments of any sizes into one H.264 MP4 at the first segment's size: each input is + * scaled to fit, padded with black and given square pixels, then the concat filter chains them. The recorder writes + * variable frame rate movies (a frame only when something moved), so the output keeps their timestamps. + */ +export function joinSegmentsArgv({ + inputs, + width, + height, + output, +}: { + inputs: string[]; + width: number; + height: number; + output: string; +}): string[] { + const w = Math.max(2, Math.floor(width / 2) * 2); + const h = Math.max(2, Math.floor(height / 2) * 2); + const chains = inputs.map( + (_, index) => + `[${index}:v]scale=${w}:${h}:force_original_aspect_ratio=decrease,` + + `pad=${w}:${h}:(ow-iw)/2:(oh-ih)/2:color=black,setsar=1,format=yuv420p,settb=AVTB[v${index}]` + ); + const labels = inputs.map((_, index) => `[v${index}]`).join(""); + return [ + "ffmpeg", + "-hide_banner", + "-nostdin", + "-loglevel", + "error", + "-y", + ...inputs.flatMap((input) => ["-i", input]), + "-filter_complex", + `${chains.join(";")};${labels}concat=n=${inputs.length}:v=1:a=0[out]`, + "-map", + "[out]", + "-c:v", + "libx264", + "-preset", + "veryfast", + "-crf", + "20", + "-pix_fmt", + "yuv420p", + "-fps_mode", + "vfr", + "-movflags", + "+faststart", + output, + ]; +} diff --git a/src/control/lib/recording-targets.ts b/src/control/lib/recording-targets.ts new file mode 100644 index 0000000000..872392745c --- /dev/null +++ b/src/control/lib/recording-targets.ts @@ -0,0 +1,207 @@ +import { logger } from "@genesiscz/utils/logger"; +import { z } from "zod"; +import { parseScreenList } from "./native-record"; +import { type AxResult, runAxAsync } from "./runner"; + +const { log } = logger.scoped("control-recording-targets"); + +/** One thing a recording session can record: exact windows, every visible window of an app, or a whole display. */ +export const recordingTargetSchema = z.discriminatedUnion("kind", [ + z.object({ + kind: z.literal("window"), + windowIds: z.array(z.number().int().positive().max(0xffffffff)).min(1).max(16), + app: z.string().optional(), + title: z.string().optional(), + pid: z.number().int().positive().optional(), + }), + z.object({ + kind: z.literal("app"), + /** A pid, an app name or a bundle id, as `--include-app` takes it. */ + app: z.string().trim().min(1), + name: z.string().optional(), + }), + z.object({ + kind: z.literal("display"), + /** NSScreen order, as `ax-tool screens` lists it. */ + screenIndex: z.number().int().nonnegative(), + name: z.string().optional(), + }), +]); + +export type RecordingTarget = z.infer; + +export interface RecordingWindow { + id: number; + pid: number; + app: string; + title: string; + bounds: { x: number; y: number; width: number; height: number }; +} + +export interface RecordingApp { + pid: number; + name: string; + windowCount: number; +} + +export interface RecordingDisplay { + index: number; + name: string; + isPrimary: boolean; + width: number; + height: number; + scaleFactor: number; +} + +export interface RecordingTargets { + /** On-screen windows, front to back. */ + windows: RecordingWindow[]; + /** Apps with at least one listed window, in the order their front window appears. */ + apps: RecordingApp[]; + displays: RecordingDisplay[]; + /** The WindowServer's frontmost application, when it reported one. */ + frontmostPid?: number; +} + +/** + * Window owners that are system chrome rather than something a person records. GenesisTools stays listed: the hub + * and the review window are worth recording, and a host filters its own panels out by pid. + */ +const SYSTEM_OWNERS = new Set([ + "Dock", + "WindowServer", + "Window Server", + "Control Center", + "Notification Center", + "NotificationCenter", + "Spotlight", + "SystemUIServer", + "loginwindow", + "Screenshot", + "TextInputMenuAgent", + "Wallpaper", +]); + +/** Smaller than this on either side is a strip, a badge or an invisible helper window, not a recording target. */ +const MIN_WINDOW_SIDE = 60; + +const frontRowSchema = z.object({ + pid: z.number().int(), + app: z.string(), + windowId: z.number().int().positive(), + title: z.string().optional(), + bounds: z.object({ x: z.number(), y: z.number(), width: z.number(), height: z.number() }), +}); + +/** `ax-tool front` rows (layer 0, on screen, front to back) as recordable windows. Malformed rows are skipped. */ +export function parseRecordingWindows(raw: unknown): RecordingWindow[] { + const rows = (raw as { windows?: unknown } | null | undefined)?.windows; + if (!Array.isArray(rows)) { + return []; + } + + const windows: RecordingWindow[] = []; + for (const row of rows) { + const parsed = frontRowSchema.safeParse(row); + if (!parsed.success) { + continue; + } + + const { pid, app, windowId, title, bounds } = parsed.data; + if (SYSTEM_OWNERS.has(app) || app.length === 0) { + continue; + } + + if (bounds.width < MIN_WINDOW_SIDE || bounds.height < MIN_WINDOW_SIDE) { + continue; + } + + windows.push({ id: windowId, pid, app, title: title ?? "", bounds }); + } + + return windows; +} + +/** One row per app, in the order its front window appears. */ +export function groupRecordingApps(windows: RecordingWindow[]): RecordingApp[] { + const apps = new Map(); + for (const window of windows) { + const app = apps.get(window.pid); + if (app) { + app.windowCount += 1; + } else { + apps.set(window.pid, { pid: window.pid, name: window.app, windowCount: 1 }); + } + } + + return [...apps.values()]; +} + +export function parseRecordingDisplays(raw: unknown): RecordingDisplay[] { + return parseScreenList(raw, "cocoa").map((screen) => ({ + index: screen.index, + name: screen.name, + isPrimary: screen.isPrimary, + width: screen.points.width, + height: screen.points.height, + scaleFactor: screen.scaleFactor, + })); +} + +/** Everything a picker offers, from one `ax-tool front` and one `ax-tool screens`. Reads only. */ +export async function listRecordingTargets( + options: { run?: (request: { args: string[]; timeoutMs: number }) => Promise } = {} +): Promise { + const run = options.run ?? runAxAsync; + const [front, screens] = await Promise.all([ + run({ args: ["front"], timeoutMs: 5000 }), + run({ args: ["screens"], timeoutMs: 5000 }), + ]); + if (!front.ok) { + throw new Error(`Cannot list windows: ${front.error ?? "ax-tool front failed"}`); + } + + if (!screens.ok) { + throw new Error(`Cannot list displays: ${screens.error ?? "ax-tool screens failed"}`); + } + + const windows = parseRecordingWindows(front); + const frontmost = (front as { frontmostPid?: unknown }).frontmostPid; + const targets: RecordingTargets = { + windows, + apps: groupRecordingApps(windows), + displays: parseRecordingDisplays(screens), + ...(typeof frontmost === "number" ? { frontmostPid: frontmost } : {}), + }; + log.debug( + { windows: targets.windows.length, apps: targets.apps.length, displays: targets.displays.length }, + "recording targets listed" + ); + return targets; +} + +/** The front window of the app the person was in: the frontmost app unless it is in `excludePids` (a terminal). */ +export function defaultRecordingTarget(targets: RecordingTargets, excludePids: Set): RecordingTarget | null { + const usable = targets.windows.filter((window) => !excludePids.has(window.pid)); + const front = + (targets.frontmostPid !== undefined && usable.find((window) => window.pid === targets.frontmostPid)) || + usable[0]; + if (front) { + return { kind: "window", windowIds: [front.id], app: front.app, title: front.title, pid: front.pid }; + } + + const display = targets.displays.find((entry) => entry.isPrimary) ?? targets.displays[0]; + return display ? { kind: "display", screenIndex: display.index, name: display.name } : null; +} + +/** A short human name for a target, for logs and the CLI. */ +export function describeRecordingTarget(target: RecordingTarget): string { + switch (target.kind) { + case "window": + return [target.app, target.title].filter(Boolean).join(" · ") || `window ${target.windowIds.join(",")}`; + case "app": + return target.name ?? target.app; + case "display": + return target.name ?? `display ${target.screenIndex}`; + } +} diff --git a/src/control/lib/runner.ts b/src/control/lib/runner.ts index 46ac5bfe38..ae78a58301 100644 --- a/src/control/lib/runner.ts +++ b/src/control/lib/runner.ts @@ -1,7 +1,8 @@ -import { spawnSync } from "node:child_process"; +import { type ChildProcess, spawn, spawnSync } from "node:child_process"; import { appendFileSync, existsSync, readFileSync } from "node:fs"; import { join, resolve } from "node:path"; import { agentSessionIds } from "@genesiscz/utils/agent/host"; +import { untilAborted } from "@genesiscz/utils/async"; import { env } from "@genesiscz/utils/env"; import { SafeJSON } from "@genesiscz/utils/json"; import { logger } from "@genesiscz/utils/logger"; @@ -311,6 +312,118 @@ export function ensureBinary(): string { ); } +interface PendingBuild { + binary: Promise; + /** Stops this process from waiting on the compile, so it can exit while the compile finishes. */ + release: () => void; +} + +let pendingBuild: PendingBuild | null = null; + +/** + * `ensureBinary` for a caller that must stay responsive while the native CLI compiles (20 to 60 s + * after a source change): a recording session that still has to read Stop and Discard, a listing + * a host cancels. `onBuild` runs once when a compile is needed, before the wait, so the caller can + * say why it is slow. The worker leads its own process group: a caller that gives up (`signal`) or + * is killed while it waits leaves the compile running, and the next caller reuses that binary + * instead of compiling again from the start. + */ +export async function ensureBinaryAsync({ + signal, + onBuild, +}: { + signal?: AbortSignal; + onBuild?: () => void; +} = {}): Promise { + refuseRealMachineInTest("ax-tool (ensureBinaryAsync hands out, and may build, the real binary)"); + if (verifiedBinary !== null) { + return verifiedBinary; + } + + if (!nativeNeedsBuild({ binary: BINARY_PATH, sourceDir: SWIFT_SOURCE })) { + verifiedBinary = BINARY_PATH; + return verifiedBinary; + } + + if (!existsSync(join(SWIFT_SOURCE, "Package.swift"))) { + return ensureBinary(); + } + + onBuild?.(); + if (!pendingBuild) { + const build = buildInOwnGroup(); + pendingBuild = build; + void build.binary.then( + () => { + pendingBuild = null; + }, + () => { + pendingBuild = null; + } + ); + } + + const build = pendingBuild; + signal?.addEventListener("abort", () => build.release(), { once: true }); + return untilAborted(build.binary, signal); +} + +function buildInOwnGroup(): PendingBuild { + logger.info({ source: SWIFT_SOURCE }, "ax-tool binary missing or stale; compiling native CLI in its own group"); + // stderr is ignored: once a cancelled caller is gone, a write to its pipe would end the compile + // with SIGPIPE. The worker's stdout result carries a failure's compiler output. + const child = spawn(process.execPath, [BUILD_WORKER, BINARY_PATH, SWIFT_SOURCE, BUILD_LOCK], { + detached: true, + stdio: ["ignore", "pipe", "ignore"], + }); + let stdout = ""; + child.stdout?.setEncoding("utf-8"); + child.stdout?.on("data", (chunk: string) => { + stdout += chunk; + }); + const deadline = setTimeout(() => child.kill("SIGTERM"), NATIVE_BUILD_WORKER_TIMEOUT_MS); + const release = () => { + child.unref(); + deadline.unref(); + // The pipe keeps the event loop alive on its own (measured on Bun 1.3: 4 s with the child unref'd alone). + const pipe = child.stdout; + if (pipe && "unref" in pipe && typeof pipe.unref === "function") { + pipe.unref(); + } + }; + return { binary: waitForBuild({ child, deadline, stdout: () => stdout }), release }; +} + +async function waitForBuild({ + child, + deadline, + stdout, +}: { + child: ChildProcess; + deadline: ReturnType; + stdout: () => string; +}): Promise { + const status = await new Promise((resolve, reject) => { + child.once("error", (error) => { + clearTimeout(deadline); + reject(error); + }); + child.once("close", (code) => { + clearTimeout(deadline); + resolve(code); + }); + }); + const outcome = parseBuildOutcome(stdout()); + if (status === 0 && outcome?.ok && existsSync(BINARY_PATH)) { + logger.info(outcome.built ? "ax-tool built successfully" : "ax-tool was built by another process; reusing it"); + verifiedBinary = BINARY_PATH; + return verifiedBinary; + } + + const details = (outcome?.error ?? stdout().trim()).slice(-4000); + throw new Error(`ax-tool build failed (${status ?? "killed"}):\n${details}`); +} + export interface AxSpawnRequest { binary: string; args: string[]; @@ -638,7 +751,6 @@ export async function runAxAsync(options: { /** Shown on the cursor overlay's label for this one call. */ note?: string; }): Promise { - const clock = new Stopwatch(); const timeoutMs = Math.floor(options.timeoutMs ?? 10000); if (options.signal?.aborted || !Number.isSafeInteger(timeoutMs) || timeoutMs < 1 || timeoutMs > 2147483647) { return { @@ -650,15 +762,22 @@ export async function runAxAsync(options: { const args = nativeArguments(options.args); let binary: string; try { - binary = ensureBinary(); + binary = await ensureBinaryAsync({ signal: options.signal }); } catch (error) { if (error instanceof RealMachineInTestError) { throw error; } + if (options.signal?.aborted) { + return { ok: false, dispatchState: "not_started", error: "Native command cancelled before dispatch." }; + } + logger.error({ error }, "Native build unavailable"); return { ok: false, dispatchState: "not_started", error: "Native build unavailable; no action dispatched." }; } + // The budget starts once the binary is ready: the one compile after a source change (20 to 60 s) is not the + // command's time, and counting it made a 5 s window listing fail right after the compile it waited for. + const clock = new Stopwatch(); const remainingMs = Math.floor(timeoutMs - clock.elapsedMs); if (remainingMs < 1 || options.signal?.aborted) { return { ok: false, dispatchState: "not_started", error: "Native command deadline reached before dispatch." }; diff --git a/src/hub/lib/composer/dispatch.ts b/src/hub/lib/composer/dispatch.ts index b6cfd527dc..bc15b0a6ef 100644 --- a/src/hub/lib/composer/dispatch.ts +++ b/src/hub/lib/composer/dispatch.ts @@ -14,6 +14,11 @@ import { boundedCommand } from "@genesiscz/utils/process/bounded-command"; import type { WidgetAsset, WidgetOutgoing } from "../widget/types"; import type { OutboxDispatcher } from "./engine"; +/** The transport the widget delivers through: the `tools` runner of the widget's own checkout. */ +export function widgetDeliverDeps(signal?: AbortSignal): DeliverDeps { + return { runTool: (args: string[]) => runWidgetDelivery({ args, signal }) }; +} + async function runWidgetDelivery({ args, signal }: { args: string[]; signal?: AbortSignal }) { const nativeTurn = (args[0] === "claude" && args[1] === "worker") || args[0] === "grok"; const result = await boundedCommand({ diff --git a/src/hub/lib/composer/outbox.test.ts b/src/hub/lib/composer/outbox.test.ts index d6fba0282e..f919f4f964 100644 --- a/src/hub/lib/composer/outbox.test.ts +++ b/src/hub/lib/composer/outbox.test.ts @@ -42,7 +42,7 @@ import { readDataVersion } from "@genesiscz/utils/workers/sqlite-wake"; import { createCanvas } from "@napi-rs/canvas"; import { z } from "zod"; import type { AgentNode } from "../agents/types"; -import { performWidgetAction } from "../widget/actions"; +import { performWidgetAction, widgetActionSchema } from "../widget/actions"; import { IMAGE_PENDING_LIMIT, type ImageFiles, @@ -70,13 +70,16 @@ import { readShelfAttachment, removeShelfItem, stageShelfImage, + stageShelfMedia, } from "../widget/shelf"; import { claudeAssistantText, discoverWidgetCatalog, + INBOX_ITEMS_PER_SESSION, inboxWindowCounts, lastAssistantTextOf, lastClaudeAssistantText, + listedInboxItems, readWidgetChanges, readWidgetDecisionEvents, realWidgetSources, @@ -87,6 +90,7 @@ import { widgetSnapshot, widgetTitle, } from "../widget/snapshot"; +import { isStaged, resendAnsweredDecisions, sendStagedDecisions, unstageDecisions } from "../widget/staged"; import { MAX_WIDGET_STATE_BYTES, mutateWidgetState, readWidgetState } from "../widget/storage"; import { parseWidgetSessionKey, @@ -99,13 +103,15 @@ import { widgetSessionKey, } from "../widget/types"; import { + addLiveVoiceNote, discardVoiceNote, editVoiceNote, + incomingClipPath, listVoiceNotes, recordVoiceNote, transcribeVoiceNote, } from "../widget/voice-notes"; -import { watchWidget } from "../widget/watch"; +import { StdinLines, WARM_SESSIONS, watchWidget, widgetSelectionLine } from "../widget/watch"; import { importWidgetAsset, reviseVideoAsset } from "./assets"; import { widgetDeliveryReceipt, widgetDispatcher } from "./dispatch"; import { processWidgetOutbox } from "./engine"; @@ -619,6 +625,110 @@ describe("widget source and delivery contracts", () => { ).rejects.toThrow("changed"); }); + test("staged decisions go one message per session; unstaging keeps them waiting", async () => { + const directory = await root(); + const files = { file: join(directory, "decisions.jsonl"), events: join(directory, "events.jsonl") }; + const post = (sessionId: string, prompts: string[]) => + postDecisions( + files.file, + files.events, + { + sessionId, + provider: "codex", + decisions: prompts.map((prompt) => ({ prompt, options: ["yes", "no"] })), + }, + { env: {} } + ); + const [a1, a2, a3] = await post("staged-a", ["First?", "Second?", "Left alone?"]); + const [b1] = await post("staged-b", ["Other?"]); + await updateDecisions(files.file, files.events, { + updates: [ + { id: a1.id, state: "drafted", draftOption: "a" }, + { id: a2.id, state: "drafted", draft: "only if it is cheap" }, + { id: a3.id, state: "drafted", draftOption: "b" }, + { id: b1.id, state: "drafted", draftOption: "b" }, + ], + }); + const sent: string[] = []; + const result = await sendStagedDecisions({ + ids: [a1.id, b1.id, a2.id], + files, + deliver: { + codexWorkerFor: () => "fixture-worker", + runTool: async (args: string[]) => { + sent.push(args.join(" ")); + return { success: true, stdout: '{"queued":false,"turnId":"fixture-turn"}', stderr: "" }; + }, + }, + }); + expect(result).toMatchObject({ sent: 3, queued: 0, failed: 0 }); + expect(result.sessions.map((entry) => [entry.session, entry.numbers])).toEqual([ + ["staged-a", [1, 2]], + ["staged-b", [1]], + ]); + expect(sent).toHaveLength(2); + const rows = readDecisions(files.file); + const row = (id: string) => rows.find((entry) => entry.id === id); + expect(row(a1.id)).toMatchObject({ option: "a" }); + expect(row(a2.id)?.answer).toBe("only if it is cheap"); + expect(row(a1.id)?.draftOption).toBeUndefined(); + // Not chosen for this send: still staged, still waiting. + expect(row(a3.id)).toMatchObject({ state: "drafted", draftOption: "b" }); + + expect(await unstageDecisions({ ids: [a3.id, a1.id], files })).toBe(1); + const after = readDecisions(files.file).find((entry) => entry.id === a3.id); + expect(after?.state).toBe("drafted"); + expect(after?.draftOption ?? "").toBe(""); + expect(after && isStaged(after)).toBe(false); + }); + + test("a staged answer no receiver took counts as queued, and Send again delivers it", async () => { + const directory = await root(); + const files = { file: join(directory, "decisions.jsonl"), events: join(directory, "events.jsonl") }; + const [row] = await postDecisions( + files.file, + files.events, + { sessionId: "resend-a", provider: "codex", decisions: [{ prompt: "Ship?", options: ["yes", "no"] }] }, + { env: {} } + ); + await updateDecisions(files.file, files.events, { + updates: [{ id: row.id, state: "drafted", draftOption: "a" }], + }); + const nowhere = await sendStagedDecisions({ + ids: [row.id], + files, + deliver: { + codexWorkerFor: () => null, + runTool: async () => { + throw new Error("No transport may run"); + }, + }, + }); + expect(nowhere).toMatchObject({ sent: 0, queued: 1, failed: 0 }); + expect(nowhere.sessions[0]?.result?.delivered).toBe(false); + expect(readDecisions(files.file).find((entry) => entry.id === row.id)?.state).toBe("answered"); + + expect(widgetActionSchema.parse({ action: "resend-answers", ids: [row.id] })).toMatchObject({ + action: "resend-answers", + }); + const sent: string[] = []; + const again = await resendAnsweredDecisions({ + ids: [row.id], + files, + deliver: { + codexWorkerFor: () => "fixture-worker", + runTool: async (args: string[]) => { + sent.push(args.join(" ")); + return { success: true, stdout: '{"queued":false,"turnId":"fixture-turn"}', stderr: "" }; + }, + }, + }); + expect(again).toMatchObject({ sent: 1, queued: 0, failed: 0 }); + expect(again.sessions[0]?.result?.delivered).toBe(true); + expect(sent).toHaveLength(1); + expect(readDecisions(files.file).find((entry) => entry.id === row.id)?.state).toBe("sent"); + }); + test("a media-only reply keeps the original session and card title in one literal message", async () => { const directory = await root(); const files = { file: join(directory, "decisions.jsonl"), events: join(directory, "events.jsonl") }; @@ -1244,7 +1354,8 @@ test("malformed stored times sort as the oldest activity instead of breaking the root: directory, sources: { ...emptySources, - decisions: () => readDecisions(file).map((row) => ({ ...row, updatedTs: "not a time" })), + decisions: () => + readDecisions(file).map((row) => ({ ...row, createdTs: "not a time", updatedTs: "not a time" })), }, }); expect(snapshot.sessions[0]?.activityAt).toBe(0); @@ -1677,10 +1788,32 @@ test("decision cards carry recorded answers and supersede links; only open decis number: 3, title: widgetTitle(replacement.prompt), }); + // The overview lists the session's entries, the waiting one first. + expect(session?.items?.[0]).toMatchObject({ id: `decision:${replacement.id}`, needsAnswer: true }); expect( snapshot.notifications?.sessions.find((entry) => entry.pendingItem?.sourceId === elsewhere.id)?.pendingItem ).toMatchObject({ number: 1, title: "Other" }); + // A saved draft updates the row but asks nothing new: the item keeps the time it was asked, so the widget + // sees no new arrival (it chimed and flashed on every Save draft, 2026-10-23). + const askedAt = session?.pendingItem?.at; + await Bun.sleep(5); + await updateDecisions(file, events, { + updates: [{ id: replacement.id, state: "drafted", draft: "back both up", expectedRevision: 1 }], + }); + const drafted = await widgetSnapshot({ root: directory, sources }); + expect(drafted.notifications?.sessions.find((entry) => entry.key === sessionKey)?.pendingItem?.at).toBe(askedAt); + expect(drafted.cards.find((entry) => entry.sourceId === replacement.id)?.at).toBe(askedAt); + // The staged list is its own, uncapped: the widget sends from it, not from a session's eight listed items. + expect(drafted.notifications?.staged.map((entry) => entry.sourceId)).toEqual([replacement.id]); + // "Mark all read" stops the badge counting it, but the staged pick is unsent input and stays. + await Bun.sleep(5); + await performWidgetAction({ root: directory, input: { action: "inbox-clear", at: Date.now() } }); + const cleared = await widgetSnapshot({ root: directory, sources }); + expect(cleared.notifications?.sessions.find((entry) => entry.key === sessionKey)?.needsAnswer ?? 0).toBe(0); + expect(cleared.notifications?.staged.map((entry) => entry.sourceId)).toEqual([replacement.id]); + expect(readDecisions(file).find((row) => row.id === replacement.id)?.updatedTs).not.toBe(replacement.createdTs); + await updateDecisions(file, events, { updates: [{ id: replacement.id, state: "superseded", supersededBy: elsewhere.id, comment: "moved" }], }); @@ -1692,6 +1825,23 @@ test("decision cards carry recorded answers and supersede links; only open decis title: "Other", }); expect(moved.notifications?.sessions.find((entry) => entry.key === sessionKey)?.needsAnswer ?? 0).toBe(0); + expect(moved.notifications?.staged).toEqual([]); +}); + +test("a session lists its waiting entries first, then the newest, at most eight", () => { + const item = (id: string, at: number, needsAnswer = false) => ({ + id, + sourceId: id, + kind: needsAnswer ? ("decision" as const) : ("answer" as const), + key: "k", + at, + needsAnswer, + }); + const items = [item("old", 1), item("ask", 2, true), ...Array.from({ length: 9 }, (_, i) => item(`n${i}`, 10 + i))]; + const listed = listedInboxItems(items); + expect(listed).toHaveLength(INBOX_ITEMS_PER_SESSION); + expect(listed.map((entry) => entry.id).slice(0, 3)).toEqual(["ask", "n8", "n7"]); + expect(listed.map((entry) => entry.id)).not.toContain("old"); }); test("alert preferences default on with the Glass sound, load from an older state, and refuse a bad sound", async () => { @@ -2298,6 +2448,36 @@ test("explicit shelf images deduplicate while general image files remain file re expect((await listWidgetShelf(directory)).items).toHaveLength(2); }); +test("a screen recording stages as a video capture and attaches to a draft as that video asset", async () => { + const directory = await root(); + const input = join(directory, "recording-20261023-221500.mp4"); + await writeFile(input, "fixture-recording"); + const probe = spyOn(videos, "probeVideo").mockImplementation(async ({ input: path }) => ({ + path: resolve(path), + durationUs: 12_000_000, + width: 1280, + height: 720, + displayWidth: 1280, + displayHeight: 720, + rotation: 0, + bytes: (await files.stat(path)).size, + codec: "fixture", + })); + try { + const item = await stageShelfMedia({ root: directory, input, name: "Screen recording.mp4", type: "video" }); + expect(item).toMatchObject({ kind: "capture", name: "Screen recording.mp4", bytes: 17 }); + const asset = (await readWidgetState(directory)).assets[item.assetId!]; + expect(asset).toMatchObject({ type: "video", durationUs: 12_000_000, status: "pending" }); + expect(await attachShelfItem({ root: directory, id: item.id, key: widgetSessionKey(target) })).toEqual({ + added: true, + mode: "image", + }); + expect((await readWidgetState(directory)).drafts[widgetSessionKey(target)].assetIds).toEqual([item.assetId!]); + } finally { + probe.mockRestore(); + } +}); + test("cancel after producing a capture cleans temporary bytes and allows restart", async () => { const directory = await root(); const controller = new AbortController(); @@ -2998,6 +3178,21 @@ describe("widget inbox notifications", () => { expect(cleared.sessions.find((session) => session.key.includes("live-fixture"))?.status).toBe( "waiting" ); + // Staging the cleared decision updates its row; it was asked before the clear, so it stays read. + decisions = decisions.map((row) => + row.sessionId === "live-fixture" + ? { + ...row, + state: "drafted", + draftOption: "yes", + createdTs: new Date(now - hour).toISOString(), + updatedTs: new Date(now + 500).toISOString(), + } + : row + ); + const afterStage = (await widgetSnapshot({ root: directory, sources, now })).notifications; + expect(afterStage?.needsAnswer).toBe(0); + expect(afterStage?.staged.map((item) => item.sourceId)).toEqual(["d_1_live-fixture"]); decisions = [...decisions, decision("new-fixture", now + 1000)]; expect((await widgetSnapshot({ root: directory, sources, now })).notifications?.needsAnswer).toBe(1); } @@ -3133,6 +3328,31 @@ describe("private Widget voice notes", () => { const note = await recordVoiceNote({ root, input }); return { root, input, note }; } + test("a live dictation from the reply box becomes a transcribed note with its audio and where it was spoken", async () => { + const root = await mkdtemp(join(tmpdir(), "widget-voice-live-")); + const clip = await incomingClipPath(root); + const audio = new Uint8Array(32_000).map((_, index) => (index % 2 === 1 ? 32 : 0)); + await Bun.write(clip, audio); + const context = { sessionKey: "local:claude:fixture:", sessionTitle: "fixture", decisionNumber: 18 }; + const note = await addLiveVoiceNote({ root, clip, text: "Fixture words", context, provider: "fixture" }); + expect(note.transcription).toBe("ready"); + expect(note.text).toBe("Fixture words"); + expect(note.context).toEqual(context); + expect(note.clip.durationMs).toBe(1000); + expect(note.clip.peakRms).toBeGreaterThan(0); + expect(await Bun.file(note.clip.path).bytes()).toEqual(audio); + expect(await Bun.file(clip).exists()).toBe(false); + expect((await files.stat(join(root, "voice-notes", "incoming"))).mode & 0o777).toBe(0o700); + expect((await files.stat(join(root, "voice-notes", "clips"))).mode & 0o777).toBe(0o700); + expect((await files.stat(note.clip.path)).mode & 0o777).toBe(0o600); + expect((await listVoiceNotes(root)).notes.map((entry) => entry.id)).toEqual([note.id]); + // Only the incoming folder may hand a clip over: any other path stays where it is. + const outside = join(root, "outside.pcm"); + await Bun.write(outside, audio); + await expect(addLiveVoiceNote({ root, clip: outside, text: "x" })).rejects.toThrow("incoming folder"); + expect(await Bun.file(outside).exists()).toBe(true); + }); + test("local record, explicit fixture transcription, edit and guarded discard use one private note", async () => { const { root, input, note } = await fixture(); expect((await files.stat(note.clip.path)).mode & 0o777).toBe(0o600); @@ -4094,6 +4314,128 @@ test("incoming answer, Decision and pending-form writes wake the Widget without }); }); +test("a snapshot carries the cards of the sessions that need you beside the selected one's", async () => { + const directory = await root(); + const file = join(directory, "decisions.jsonl"); + const events = join(directory, "events.jsonl"); + for (const sessionId of ["picked", "waiting-too"]) { + await postDecisions( + file, + events, + { sessionId, provider: "codex", decisions: [{ prompt: `Proceed in ${sessionId}?`, options: ["yes"] }] }, + { env: {} } + ); + } + await postDecisions( + file, + events, + { sessionId: "quiet", provider: "codex", decisions: [{ type: "todo", prompt: "Later", options: [] }] }, + { env: {} } + ); + const sources: WidgetSources = { ...emptySources, decisions: () => readDecisions(file) }; + const roster = await widgetSnapshot({ root: directory, sources }); + const keyOf = (sessionId: string): string => { + const key = roster.sessions.find((entry) => entry.target.sessionId === sessionId)?.key; + if (!key) { + throw new Error(`the roster has no session ${sessionId}`); + } + + return key; + }; + await mutateWidgetState(directory, (state) => { + state.selectedKey = keyOf("picked"); + state.preferences.showChanges = false; + }); + + const plain = await widgetSnapshot({ root: directory, sources }); + expect(plain.cards.map((card) => card.sessionKey)).toEqual([keyOf("picked")]); + expect(plain.warmKeys).toBeUndefined(); + + const warm = await widgetSnapshot({ root: directory, sources, warm: 4 }); + expect(warm.warmKeys).toEqual([keyOf("waiting-too")]); + expect(new Set(warm.cards.map((card) => card.sessionKey))).toEqual( + new Set([keyOf("picked"), keyOf("waiting-too")]) + ); + expect(warm.cards.some((card) => card.sessionKey === keyOf("quiet"))).toBe(false); + expect(warm.selectedKey).toBe(keyOf("picked")); +}); + +test("a session picked on the watch's stdin is the next snapshot's selection, with the warm sessions asked for", async () => { + const directory = await root(); + await env.testing.withOverrides({ GENESIS_TOOLS_HOME: directory }, async () => { + const controller = new AbortController(); + const asked: { selectedKey?: string; warm?: number }[] = []; + let pick: ((key: string) => void) | undefined; + let ready!: () => void; + const started = new Promise((resolve) => { + ready = resolve; + }); + let picked!: () => void; + const pickedSnapshot = new Promise((resolve) => { + picked = resolve; + }); + const worker = watchWidget({ + root: directory, + signal: controller.signal, + emit: () => ready(), + selections: (listener) => { + pick = listener; + return () => { + pick = undefined; + }; + }, + dependencies: { + inboxPaths: { + answerLog: join(directory, "question", "log"), + database: join(directory, "question", "qa.db"), + decisions: join(directory, "question", "decisions", "decisions.jsonl"), + }, + watchInbox: async () => ({ active: true, errorCount: 0, unsubscribe: async () => {} }), + snapshot: async (options) => { + asked.push({ selectedKey: options?.selectedKey, warm: options?.warm }); + if (options?.selectedKey === "local:codex:picked:") { + picked(); + } + return { + version: 1, + state: await readWidgetState(directory), + activity: [], + sessions: [], + cards: [], + manifests: {}, + changes: null, + errors: [], + selectedKey: options?.selectedKey ?? null, + }; + }, + dispatcher: { + validate: async () => {}, + dispatch: async () => ({ delivered: true, channel: "fixture" }), + }, + }, + }); + try { + await withTimeout(started, 3000); + expect(asked[0]).toEqual({ selectedKey: undefined, warm: WARM_SESSIONS }); + pick?.("local:codex:picked:"); + await withTimeout(pickedSnapshot, 3000); + } finally { + controller.abort(); + await withTimeout(worker, 3000); + } + expect(pick).toBeUndefined(); + }); +}); + +test("the watch's stdin control lines: whole lines only, and only a select with a key", () => { + const keys: (string | undefined)[] = []; + const lines = new StdinLines((line) => keys.push(widgetSelectionLine(line))); + lines.push('{"select":"local:claude:a:"}\n{"sel'); + lines.push('ect":"local:claude:b:"}\n\n'); + lines.push('{"select":""}\nnot json\n{"other":1}\n'); + expect(keys).toEqual(["local:claude:a:", "local:claude:b:", undefined, undefined, undefined]); +}); + test("a session queue directory that cannot be watched leaves the Widget running on its safety refresh", async () => { const directory = await root(); await env.testing.withOverrides({ GENESIS_TOOLS_HOME: directory }, async () => { diff --git a/src/hub/lib/dev-monitor.test.ts b/src/hub/lib/dev-monitor.test.ts index 42a5777a09..b9d62a6400 100644 --- a/src/hub/lib/dev-monitor.test.ts +++ b/src/hub/lib/dev-monitor.test.ts @@ -53,6 +53,18 @@ describe("classifyPerfLine", () => { ).toBe("hang"); }); + test("a main thread spinning at full CPU is an event, and its sample names the file", () => { + expect(classifyPerfLine("[17:10:32.100] mark main-spin 5s at 99% cpu: the main thread never idles")?.kind).toBe( + "spin" + ); + expect(classifyPerfLine("[17:10:34.200] mark main-spin bodies in 2s: bodies[widget.inbox.row=812]")?.kind).toBe( + "spin" + ); + expect( + classifyPerfLine("[17:10:32.101] mark main-spin 5.0s so far — sampling to hang-2026-10-23-171032.txt")?.file + ).toEndWith("logs/hangs/hang-2026-10-23-171032.txt"); + }); + test("short stalls, quick main-thread spans and routine lines are left out", () => { expect(classifyPerfLine("[01:48:30.597] mark main-stall recovered after 319ms load=6.3")).toBeNull(); expect(classifyPerfLine("[01:48:30.597] mark main-stall recovered after 819ms load=6.3")?.kind).toBe("stall"); diff --git a/src/hub/lib/dev-monitor.ts b/src/hub/lib/dev-monitor.ts index 85d7894ecc..ccc70ec504 100644 --- a/src/hub/lib/dev-monitor.ts +++ b/src/hub/lib/dev-monitor.ts @@ -16,7 +16,18 @@ const log = logger.child({ component: "hub:dev-monitor" }); * agent on each instead of the user finding a "not responding" banner first. */ export interface DevEvent { - kind: "wedge" | "hang" | "stall" | "layout-loop" | "slow-main" | "jank" | "error" | "relay" | "crash" | "slow"; + kind: + | "wedge" + | "spin" + | "hang" + | "stall" + | "layout-loop" + | "slow-main" + | "jank" + | "error" + | "relay" + | "crash" + | "slow"; /** The time the source line carries ("01:46:40.479"), or the file's time for a crash or a hang file. */ time: string; text: string; @@ -137,7 +148,7 @@ export function classifyPerfLine(line: string, options: ClassifyOptions = DEFAUL return { kind: "wedge", time, text: short(body) }; } - const sampled = body.match(/main-stall .*sampling to (\S+)/); + const sampled = body.match(/main-(?:stall|spin) .*sampling to (\S+)/); if (sampled) { return { kind: "hang", time, text: short(body), file: genesisToolsDir("logs", "hangs", sampled[1]) }; } @@ -147,6 +158,11 @@ export function classifyPerfLine(line: string, options: ClassifyOptions = DEFAUL return { kind: "hang", time, text: short(body), file: genesisToolsDir("logs", "hangs", stacks[1]) }; } + // A main thread that never idles (an update loop) never delays a ping, so HangWatch reports it by CPU time. + if (/^main-spin /.test(body)) { + return { kind: "spin", time, text: short(body) }; + } + const recovered = body.match(/main-stall recovered after (\d+)ms/); if (recovered) { return Number(recovered[1]) >= options.minStallMs ? { kind: "stall", time, text: short(body) } : null; diff --git a/src/hub/lib/widget/actions.ts b/src/hub/lib/widget/actions.ts index 84c3cfb797..54f7d8d129 100644 --- a/src/hub/lib/widget/actions.ts +++ b/src/hub/lib/widget/actions.ts @@ -9,6 +9,7 @@ import { toolDataDir } from "@genesiscz/utils/storage/root"; import { videoSettingsSchema } from "@genesiscz/utils/video/types"; import { z } from "zod"; import { confirmVideoAsset, importWidgetAsset, reviseVideoAsset } from "../composer/assets"; +import { widgetDeliverDeps } from "../composer/dispatch"; import { changeOutgoing, enqueueWidgetMessage } from "../composer/outbox"; import { createWidgetHandoff } from "./handoff"; import { @@ -19,6 +20,7 @@ import { } from "./screenshot"; import { readShelfAttachment } from "./shelf"; import { type WidgetSources, widgetProvider, widgetResultNode } from "./snapshot"; +import { resendAnsweredDecisions, sendStagedDecisions, unstageDecisions } from "./staged"; import { acknowledgeWidgetInbox, mutateWidgetState, readWidgetState, widgetRoot } from "./storage"; import { parseWidgetSessionKey, @@ -29,6 +31,18 @@ import { widgetTargetSchema, } from "./types"; +/** Where each session's answers go, by session id: the widget's own target for it. */ +const decisionTargetsSchema = z + .record( + z.string(), + z.object({ + provider: z.string().optional(), + sourceHome: z.string().optional(), + cwd: z.string().optional(), + }) + ) + .optional(); + export const widgetActionSchema = z.discriminatedUnion("action", [ z.object({ action: z.literal("selection"), key: z.string().nullable() }), z.object({ action: z.literal("handoff"), key: z.string() }), @@ -42,6 +56,9 @@ export const widgetActionSchema = z.discriminatedUnion("action", [ draft: z.string().optional(), draftOption: z.string().optional(), }), + z.object({ action: z.literal("send-staged"), ids: z.array(z.string()).min(1), targets: decisionTargetsSchema }), + z.object({ action: z.literal("resend-answers"), ids: z.array(z.string()).min(1), targets: decisionTargetsSchema }), + z.object({ action: z.literal("unstage"), ids: z.array(z.string()).min(1) }), z.object({ action: z.literal("preferences"), patch: widgetPreferencesPatchSchema }), z.object({ action: z.literal("visibility"), key: z.string(), pinned: z.boolean() }), z.object({ action: z.literal("draft"), key: z.string(), draft: widgetDraftSchema }), @@ -111,6 +128,20 @@ export async function performWidgetAction({ ...(request.draftOption !== undefined ? { draftOption: request.draftOption } : {}), }); } + case "send-staged": + return sendStagedDecisions({ + ids: request.ids, + targets: request.targets, + deliver: widgetDeliverDeps(signal), + }); + case "resend-answers": + return resendAnsweredDecisions({ + ids: request.ids, + targets: request.targets, + deliver: widgetDeliverDeps(signal), + }); + case "unstage": + return { unstaged: await unstageDecisions({ ids: request.ids }) }; case "selection": return mutateWidgetState(root, (state) => { state.selectedKey = request.key; diff --git a/src/hub/lib/widget/shelf.ts b/src/hub/lib/widget/shelf.ts index 4a3395649b..c978a0f9b8 100644 --- a/src/hub/lib/widget/shelf.ts +++ b/src/hub/lib/widget/shelf.ts @@ -221,16 +221,31 @@ export async function captureShelfImage({ ); } -export async function stageShelfImage({ +export function stageShelfImage(options: { + root?: string; + input: string; + name?: string; + signal?: AbortSignal; +}): Promise { + return stageShelfMedia({ ...options, type: "image" }); +} + +/** + * Stages a screenshot or a screen recording as a capture: a widget asset with no recipient yet. A video asset gets the + * same frame review a pasted video gets once it is attached to a draft. + */ +export async function stageShelfMedia({ root, input, name, signal, + type, }: { root?: string; input: string; name?: string; signal?: AbortSignal; + type: "image" | "video"; }): Promise { signal?.throwIfAborted(); const digest = await sha256FileAsync(resolve(input), { signal }); @@ -248,11 +263,11 @@ export async function stageShelfImage({ } } - const asset = await importWidgetAsset({ root, input, type: "image" }); + const asset = await importWidgetAsset({ root, input, type }); try { signal?.throwIfAborted(); - if (asset.type !== "image") { - throw new Error("Capture did not produce an image"); + if (asset.type !== type) { + throw new Error(`Capture did not produce ${type === "image" ? "an image" : "a video"}`); } const item: ShelfItem = { @@ -261,7 +276,7 @@ export async function stageShelfImage({ name: name ?? basename(input), path: asset.path, sha256: asset.sha256, - bytes: asset.bytes, + bytes: asset.type === "image" ? asset.bytes : (await stat(asset.path)).size, assetId: asset.id, createdAt: Date.now(), }; @@ -351,7 +366,7 @@ export async function readShelfAttachment({ root, id, key }: { root?: string; id const draft = state.drafts[key] ?? { text: "", assetIds: [] }; if (item.kind === "capture") { if (!item.assetId || !state.assets[item.assetId]) { - throw new Error("The captured image is unavailable; capture it again"); + throw new Error("The capture is unavailable; capture it again"); } return { mode: "image" as const, assetId: item.assetId, draft }; @@ -382,7 +397,7 @@ export async function attachShelfItem({ const draft = state.drafts[key] ?? { text: "", assetIds: [] }; if (item.kind === "capture") { if (!item.assetId || !state.assets[item.assetId]) { - throw new Error("The captured image is unavailable; capture it again"); + throw new Error("The capture is unavailable; capture it again"); } const added = !draft.assetIds.includes(item.assetId); diff --git a/src/hub/lib/widget/snapshot.ts b/src/hub/lib/widget/snapshot.ts index e7da1eca7d..a098379891 100644 --- a/src/hub/lib/widget/snapshot.ts +++ b/src/hub/lib/widget/snapshot.ts @@ -7,7 +7,12 @@ import { INBOX_STALE_MS } from "@app/question/lib/inbox/build"; import { renderFormAnswer } from "@app/question/lib/pending/render"; import { listFormsSnapshot, PENDING_MIGRATIONS } from "@app/question/lib/pending/store"; import type { AskForm, AskItem } from "@app/question/lib/pending/types"; -import { type QaRow, queryEntriesSnapshot, readQuestionSnapshot } from "@app/question/lib/read-model"; +import { + type QaRow, + queryEntriesOfSessionsSnapshot, + queryEntriesSnapshot, + readQuestionSnapshot, +} from "@app/question/lib/read-model"; import type { TranscriptAnchor } from "@genesiscz/utils/agent/source-anchor"; import { type ResolvedTranscript, @@ -46,6 +51,29 @@ function timeOf(value: string | null | undefined): number { return Number.isFinite(at) ? at : 0; } +/** The unsent pick and note of a `drafted` decision, each only when present. */ +function stagedOf(row: { state: string; draftOption?: string; draft?: string }): { + draftOption?: string; + draft?: string; +} { + if (row.state !== "drafted") { + return {}; + } + + return { + ...(row.draftOption ? { draftOption: row.draftOption } : {}), + ...(row.draft?.trim() ? { draft: row.draft } : {}), + }; +} + +/** + * When a decision was asked: its last post (a supersede revises it), not its last update. A saved draft or a staged + * pick updates the row, and the widget treats a newer item time as a new arrival that chimes and flashes. + */ +function askedTimeOf(row: { revisedTs?: string; createdTs?: string; updatedTs: string }): number { + return timeOf(row.revisedTs ?? row.createdTs ?? row.updatedTs); +} + export interface WidgetSession { key: string; target: WidgetTarget; @@ -102,6 +130,12 @@ export interface WidgetCard { /** Decision cards: the chosen option letters and the answer text, however they were stored. */ option?: string; answer?: string; + /** + * Decision cards: the user's unsent pick and note (state `drafted`). A staged decision has either; the widget + * sends staged decisions together, one message per session. + */ + draftOption?: string; + draft?: string; /** Decision cards: every comment, oldest first (a supersede reason, progress notes). */ comments?: string[]; /** Decision cards: who stored the answer, the user in an inbox (`inbox`) or an agent copying a chat answer. */ @@ -132,6 +166,8 @@ export interface WidgetSources { decisions(): DecisionRecord[]; forms(session?: string): AskForm[]; answers(session?: string): QaRow[]; + /** The newest `limit` answers of each session in one read (the warm sessions); else `answers` per session. */ + answersOf?(sessions: string[], limit: number): QaRow[]; agents(refresh?: boolean): Promise; events?(ids: string[]): WidgetActivityEvent[]; inboxData?(window: WidgetInboxWindow): WidgetInboxData; @@ -195,6 +231,15 @@ export interface WidgetInboxItem { title?: string; /** A decision's number. */ number?: number; + /** How many inbox items this entry stands for: an answers group lists only its newest item. */ + count?: number; + /** A decision: its options, so the inbox of every session can stage a pick without opening the session. */ + options?: string[]; + recommended?: string; + revision?: number; + /** A decision's unsent pick and note (`drafted`); see `WidgetCard.draftOption`. */ + draftOption?: string; + draft?: string; } export interface WidgetInboxSession { key: string; @@ -203,6 +248,18 @@ export interface WidgetInboxSession { latest?: WidgetInboxItem; unreadItem?: WidgetInboxItem; pendingItem?: WidgetInboxItem; + /** The session's listed entries, the ones waiting for an answer first, then newest first; at most `INBOX_ITEMS_PER_SESSION`. */ + items?: WidgetInboxItem[]; +} + +/** The entries one session lists in the widget's inbox overview; its counts still cover every item. */ +export const INBOX_ITEMS_PER_SESSION = 8; + +/** A session's entries for the overview: the ones waiting for an answer first, then newest first, capped. */ +export function listedInboxItems(items: WidgetInboxItem[]): WidgetInboxItem[] { + return [...items] + .sort((a, b) => Number(b.needsAnswer) - Number(a.needsAnswer) || b.at - a.at) + .slice(0, INBOX_ITEMS_PER_SESSION); } export interface WidgetInboxSummary { unread: number; @@ -210,6 +267,8 @@ export interface WidgetInboxSummary { complete: boolean; truncated: boolean; sessions: WidgetInboxSession[]; + /** Every staged decision, uncapped: the widget sends from this list, not from a session's capped `items`. */ + staged: WidgetInboxItem[]; profile?: { hostId: "local" }; } @@ -498,6 +557,8 @@ export const realWidgetSources: WidgetSources = { // The notified answer can sit behind 80 newer read ones; opening it must still find its card. opts: sessionId ? { sessionId, limit: 80, includeNewestUnread: true } : { limit: 100 }, }), + answersOf: (sessionIds, limit) => + queryEntriesOfSessionsSnapshot({ dbPath: toolDataDir("question", "qa.db"), sessionIds, limit }), agents: (refresh) => widgetAgents({ refresh }), inboxData: readInboxData, events: (ids) => readWidgetDecisionEvents({ ids }), @@ -690,6 +751,7 @@ export async function widgetSnapshot({ sources = realWidgetSources, now = Date.now(), onImagesDeferred, + warm = 0, }: { root?: string; selectedKey?: string; @@ -697,6 +759,11 @@ export async function widgetSnapshot({ sources?: WidgetSources; /** The clock the inbox ages items against; tests pass a fixed one. */ now?: number; + /** + * Up to this many other inbox sessions (waiting for an answer first, then unread, newest first) get their newest + * cards in the snapshot too, named in `warmKeys`, so the widget opens one of them with its cards at once. + */ + warm?: number; /** This snapshot left card images unread (its read budget was spent); the watch loop takes another one. */ onImagesDeferred?: () => void; }) { @@ -1000,6 +1067,13 @@ export async function widgetSnapshot({ if (!entry[target] || item.at > entry[target].at) { entry[target] = item; } + + const listed = count > 1 ? { ...item, count } : item; + if (entry.items) { + entry.items.push(listed); + } else { + entry.items = [listed]; + } inboxSessions.set(session.key, entry); }; // Counted per item: each SQL group carries its session's newest item and how many of its items fall after the @@ -1053,29 +1127,39 @@ export async function widgetSnapshot({ const inboxCount = (kind: "answer" | "form", groups: WidgetInboxGroup[]) => counted[kind] + Math.max(0, (groups[0]?.recentAfterClearTotal ?? 0) - listedRecent[kind]); let pendingDecisions = 0; + const staged: WidgetInboxItem[] = []; for (const row of decisions) { if (kindOf(row) !== "decision" || !["open", "drafted"].includes(row.state) || row.delivery?.uncertain) { continue; } const session = findSession(row.sessionId, row.provider); - if (!session || !inboxCounted(session, timeOf(row.updatedTs))) { + if (!session) { continue; } - pendingDecisions += 1; - putInbox( - session, - { - id: `decision:${row.id}`, - sourceId: row.id, - kind: "decision", - key: session.key, - at: timeOf(row.updatedTs), - needsAnswer: true, - number: row.number, - ...decisionTitle(row), - }, - 1 - ); + const item: WidgetInboxItem = { + id: `decision:${row.id}`, + sourceId: row.id, + kind: "decision", + key: session.key, + at: askedTimeOf(row), + needsAnswer: true, + number: row.number, + ...decisionTitle(row), + options: row.options, + ...(row.recommended ? { recommended: row.recommended } : {}), + revision: row.revision ?? 1, + ...stagedOf(row), + }; + // A staged pick is unsent input: it stays in `staged` after "Mark all read" and in a quiet session. + if (item.draftOption || item.draft) { + staged.push(item); + } + // "Mark all read" compares the time it was asked: staging an older decision updates the row, and must not + // count it as new again. A recent update still keeps the decision fresh in a quiet session. + if (item.at > (state.inboxClearedAt ?? 0) && inboxFresh(session, timeOf(row.updatedTs))) { + pendingDecisions += 1; + putInbox(session, item, 1); + } } let unreadResults = 0; const resultNodes = new Map(); @@ -1125,16 +1209,39 @@ export async function widgetSnapshot({ truncated: inboxData.truncated || inboxSessions.size > 512, sessions: [...inboxSessions.values()] .sort((a, b) => b.needsAnswer - a.needsAnswer || b.unread - a.unread) - .slice(0, 512), + .slice(0, 512) + .map((entry) => ({ ...entry, items: listedInboxItems(entry.items ?? []) })), + staged, ...(prof.enabled ? { profile: { hostId: "local" as const } } : {}), }; const inboxMetadata: { notifications?: WidgetInboxSummary } = { notifications }; const selected = selectedKey ? sessions.get(selectedKey) : undefined; const persistedTarget = selectedKey ? parseWidgetSessionKey(selectedKey) : undefined; const selectedId = selected?.target.sessionId ?? persistedTarget?.sessionId; - const [answers, forms] = await Promise.all([ + // A switch to a session used to show "Loading this session's inbox…" until a snapshot named it, every time + // (Martin, 2026-10-24: "can't it just be cached and be prepared at all times"). The sessions that need you come + // along, so the hover list and the rail open them with their cards. + const warmKeys = + selectedKey && warm > 0 ? warmInboxKeys({ notifications, sessions, selectedKey, limit: warm }) : []; + const cardKeys = selectedKey ? new Set([selectedKey, ...warmKeys]) : undefined; + const warmIds = [...new Set(warmKeys.flatMap((key) => sessions.get(key)?.target.sessionId ?? []))].filter( + (id) => id !== selectedId + ); + // The warm sessions read images from their own small budget, so the selected session's never wait for theirs. + const warmImageBudget = new ImageLiftBudget({ files: 2, paths: 32 }); + const budgetFor = (key: string) => (key === selectedKey ? imageBudget : warmImageBudget); + const [answers, forms, warmAnswers, warmForms] = await Promise.all([ selectedId ? read("answers", () => sources.answers(selectedId), []) : Promise.resolve(answerRoster), selectedId ? read("question timeline", () => sources.forms(selectedId), []) : Promise.resolve(waitingForms), + read( + "warm answers", + () => + sources.answersOf + ? sources.answersOf(warmIds, WARM_CARD_WINDOW) + : warmIds.flatMap((id) => sources.answers(id)), + [] + ), + read("warm questions", () => warmIds.flatMap((id) => sources.forms(id)), []), ]); for (const form of forms) { const id = form.sessionHint || form.id; @@ -1155,13 +1262,21 @@ export async function widgetSnapshot({ } } const cards: WidgetCard[] = []; - const formEntries = new Set(forms.flatMap((form) => (form.entryId ? [form.entryId] : []))); - for (const row of answers) { + const selectedAnswers = new Set(answers.map((row) => row.id)); + const selectedForms = new Set(forms.map((form) => form.id)); + const cardForms = [...forms, ...warmForms.filter((form) => !selectedForms.has(form.id))]; + const formEntries = new Set(cardForms.flatMap((form) => (form.entryId ? [form.entryId] : []))); + for (const row of [...answers, ...warmAnswers.filter((entry) => !selectedAnswers.has(entry.id))]) { if (formEntries.has(row.id)) { continue; } + const known = findSession(row.sessionId, row.agent === "unknown" ? undefined : row.agent); + // A warm session's answer joins a session already listed; only the selected session's may add one. + if (!known && !selectedAnswers.has(row.id)) { + continue; + } const session = - findSession(row.sessionId, row.agent === "unknown" ? undefined : row.agent) ?? + known ?? addSession( targetOf({ sessionId: row.sessionId, provider: row.agent, cwd: row.cwd }, row.id), row.sessionTitle ?? row.sessionId, @@ -1170,7 +1285,7 @@ export async function widgetSnapshot({ ); const answerImages = await liftCardImages([row.question, cleanVisibleContext(row.answerMd)], { existing: row.attachments ?? [], - budget: imageBudget, + budget: budgetFor(session.key), }); const answerTitle = liftedTitle(answerImages.texts[0], answerImages.attachments); const answerBody = answerImages.texts[1]; @@ -1226,7 +1341,7 @@ export async function widgetSnapshot({ }; for (const row of decisions) { const session = findSession(row.sessionId, row.provider); - if (!session || (selectedKey && session.key !== selectedKey)) { + if (!session || (cardKeys && !cardKeys.has(session.key))) { continue; } // The prompt is lifted too: an item's attachments are embedded there. With a title (a task's short name) the @@ -1240,7 +1355,7 @@ export async function widgetSnapshot({ ), ...(row.title ? [cleanVisibleContext(row.prompt)] : []), ], - { budget: imageBudget } + { budget: budgetFor(session.key) } ); const decisionTitle = liftedTitle(decisionImages.texts[0], decisionImages.attachments); const decisionBody = [decisionImages.texts[2], decisionImages.texts[1]] @@ -1264,7 +1379,7 @@ export async function widgetSnapshot({ }, sessionKey: session.key, sourceId: row.id, - at: timeOf(row.updatedTs), + at: askedTimeOf(row), title: decisionTitle, body: decisionBody, status: row.delivery?.uncertain ? "delivery unknown" : row.state, @@ -1280,16 +1395,17 @@ export async function widgetSnapshot({ read: true, ...(row.option ? { option: row.option } : {}), ...(row.answer?.trim() ? { answer: row.answer } : {}), + ...stagedOf(row), ...(row.comments?.length ? { comments: row.comments } : {}), ...(row.answeredVia ? { answeredVia: row.answeredVia } : {}), ...supersededBy(row), }); } - for (const form of forms) { + for (const form of cardForms) { const session = (form.sessionHint ? findSession(form.sessionHint, formProvider(form)) : undefined) ?? [...sessions.values()].find((entry) => entry.target.sessionId === form.id); - if (!session || (selectedKey && session.key !== selectedKey)) { + if (!session || (cardKeys && !cardKeys.has(session.key))) { continue; } // Each item's prompt is drawn by the form view, so the lifted images leave the prompts the widget receives. @@ -1298,7 +1414,7 @@ export async function widgetSnapshot({ form.status === "answered" ? cleanVisibleContext(renderFormAnswer(form, form.answers ?? {})) : "", ...form.items.map((item) => item.promptMarkdown), ], - { budget: imageBudget } + { budget: budgetFor(session.key) } ); const [formAnswer, ...formPrompts] = formImages.texts; const formItems = form.items.map((item, index) => ({ ...item, promptMarkdown: formPrompts[index] ?? "" })); @@ -1337,16 +1453,16 @@ export async function widgetSnapshot({ ...flattenAgents(agents.orphans), ]) { const session = findResultSession(node); - if (!session || !hasResult(node) || (selectedKey && session.key !== selectedKey)) { + if (!session || !hasResult(node) || (cardKeys && !cardKeys.has(session.key))) { continue; } - const result = selectedKey ? await read("agent result", () => widgetResult(node), "") : ""; + const result = cardKeys ? await read("agent result", () => widgetResult(node), "") : ""; const resultImages = await liftCardImages( [ result || `Task context: ${node.spawnPromptPreview ?? node.description ?? "Open the conversation to read the agent's result."}`, ], - { budget: imageBudget } + { budget: budgetFor(session.key) } ); cards.push({ id: `result:${node.harness}:${node.id}`, @@ -1395,15 +1511,24 @@ export async function widgetSnapshot({ files: [], }) : null; - // The window keeps the newest cards plus each card a notification names, so opening it can acknowledge it. + // The window keeps the newest cards plus each card a notification names, so opening it can acknowledge it. A warm + // session keeps fewer, and every card that waits for an answer; opening it brings the rest a moment later. const selectedCards = cards - .filter((card) => !selectedKey || card.sessionKey === selectedKey) + .filter((card) => !cardKeys || cardKeys.has(card.sessionKey)) .sort((a, b) => a.at - b.at); const notified = new Set( notifications.sessions.flatMap((session) => [session.unreadItem?.id, session.pendingItem?.id]) ); - const windowStart = Math.max(0, selectedCards.length - 100); - const visibleCards = selectedCards.filter((card, index) => index >= windowStart || notified.has(card.id)); + const left = new Map(); + const visibleCards = selectedCards + .reverse() + .filter((card) => { + const bucket = cardKeys ? card.sessionKey : ""; + const remaining = left.get(bucket) ?? (bucket === "" || bucket === selectedKey ? 100 : WARM_CARD_WINDOW); + left.set(bucket, remaining - 1); + return remaining > 0 || notified.has(card.id) || cardWaits(card); + }) + .reverse(); const activity = selectedId && sources.events ? await read( @@ -1432,5 +1557,38 @@ export async function widgetSnapshot({ changes, errors, selectedKey: selectedKey ?? null, + ...(warmKeys.length > 0 ? { warmKeys } : {}), }; } + +/** The newest cards a warm session brings; those that wait for an answer come on top of these. */ +export const WARM_CARD_WINDOW = 12; + +/** A card that waits for the user: an open or drafted decision, a pending form (GenesisKit `WidgetCard.needsAnswer`). */ +function cardWaits(card: WidgetCard): boolean { + return ( + (card.kind === "decision" && (card.status === "open" || card.status === "drafted")) || + (card.kind === "form" && card.status === "pending") + ); +} + +/** The inbox sessions to keep warm: those waiting for an answer, then those with unread items, each newest first. */ +export function warmInboxKeys({ + notifications, + sessions, + selectedKey, + limit, +}: { + notifications: WidgetInboxSummary; + sessions: Map; + selectedKey: string; + limit: number; +}): string[] { + return notifications.sessions + .filter((entry) => entry.key !== selectedKey && entry.needsAnswer + entry.unread > 0 && sessions.has(entry.key)) + .sort( + (a, b) => Number(b.needsAnswer > 0) - Number(a.needsAnswer > 0) || (b.latest?.at ?? 0) - (a.latest?.at ?? 0) + ) + .slice(0, limit) + .map((entry) => entry.key); +} diff --git a/src/hub/lib/widget/staged.ts b/src/hub/lib/widget/staged.ts new file mode 100644 index 0000000000..1243f95ef3 --- /dev/null +++ b/src/hub/lib/widget/staged.ts @@ -0,0 +1,204 @@ +import type { DeliverDeps } from "@app/question/lib/decisions/deliver"; +import { decisionFiles } from "@app/question/lib/decisions/read"; +import { sendAnsweredDecisions } from "@app/question/lib/decisions/send"; +import { type DecisionRecord, kindOf, readDecisions, updateDecisions } from "@app/question/lib/decisions/store"; +import { answerInboxDecisions, type InboxAnswerResult } from "@app/question/lib/inbox/answer"; +import { waitingBlock } from "@app/question/lib/inbox/load"; + +/** Where a session's answers go: the widget's target for it (the decision row alone has no source home). */ +export interface StagedTarget { + provider?: string; + sourceHome?: string; + cwd?: string; +} + +export interface StagedSendResult { + /** One entry per session, in the order the sessions first appear in `ids`. */ + sessions: Array<{ session: string; numbers: number[]; result?: InboxAnswerResult; error?: string }>; + /** Answers a receiver took. */ + sent: number; + /** Answers stored and waiting for the session's next prompt: no receiver took them. */ + queued: number; + failed: number; +} + +/** + * A staged decision: the user picked an option or wrote a note in an inbox and has not sent it. It is the decision + * log's own unsent state (`drafted` with `draftOption` or `draft`), so the hub, the widget and the CLI see one list. + */ +export function isStaged(row: DecisionRecord): boolean { + return kindOf(row) === "decision" && row.state === "drafted" && Boolean(row.draftOption || row.draft?.trim()); +} + +/** + * Sends the staged decisions `ids` as their answers: one message per session, through `answerInboxDecisions`, the + * path the inbox and the hub's Decisions pane send with. A session that fails keeps its decisions staged and is + * reported; the other sessions still go. An id that is not staged (sent meanwhile, unstaged) is skipped. + */ +export async function sendStagedDecisions({ + ids, + targets = {}, + files = decisionFiles(), + deliver = {}, +}: { + ids: readonly string[]; + targets?: Record; + files?: { file: string; events: string }; + deliver?: DeliverDeps; +}): Promise { + const wanted = new Set(ids); + const staged = readDecisions(files.file).filter((row) => wanted.has(row.id) && isStaged(row)); + const bySession = new Map(); + + for (const id of ids) { + const row = staged.find((entry) => entry.id === id); + + if (!row) { + continue; + } + + bySession.set(row.sessionId, [...(bySession.get(row.sessionId) ?? []), row]); + } + + const result: StagedSendResult = { sessions: [], sent: 0, queued: 0, failed: 0 }; + + for (const [session, rows] of bySession) { + const target = targets[session] ?? {}; + const numbers = rows.map((row) => row.number); + const provider = target.provider ?? rows[0]?.provider; + const cwd = target.cwd ?? rows[0]?.cwd; + + try { + const sent = await answerInboxDecisions( + { + session, + ...(provider ? { provider } : {}), + ...(target.sourceHome ? { sourceHome: target.sourceHome } : {}), + ...(cwd ? { cwd } : {}), + answers: rows.map((row) => ({ + number: row.number, + expectedRevision: row.revision ?? 1, + ...(row.draftOption ? { option: row.draftOption } : {}), + ...(row.draft?.trim() ? { text: row.draft.trim() } : {}), + })), + }, + { ...files, block: waitingBlock, deliver } + ); + result.sessions.push({ session, numbers, result: sent }); + + if (sent.delivered) { + result.sent += rows.length; + } else { + result.queued += rows.length; + } + } catch (error) { + result.sessions.push({ session, numbers, error: error instanceof Error ? error.message : String(error) }); + result.failed += rows.length; + } + } + + return result; +} + +/** + * Sends again the answered decisions `ids` that no receiver took (the widget's Send again): one message per session, + * through `sendAnsweredDecisions`. An answer already reserved by a session queue stays queued; its next prompt takes it. + */ +export async function resendAnsweredDecisions({ + ids, + targets = {}, + files = decisionFiles(), + deliver = {}, +}: { + ids: readonly string[]; + targets?: Record; + files?: { file: string; events: string }; + deliver?: DeliverDeps; +}): Promise { + const wanted = new Set(ids); + const bySession = new Map(); + + for (const row of readDecisions(files.file)) { + if (wanted.has(row.id) && kindOf(row) === "decision" && row.state === "answered") { + bySession.set(row.sessionId, [...(bySession.get(row.sessionId) ?? []), row]); + } + } + + const result: StagedSendResult = { sessions: [], sent: 0, queued: 0, failed: 0 }; + + for (const [session, rows] of bySession) { + const target = targets[session] ?? {}; + const numbers = rows.map((row) => row.number); + const free = rows.filter((row) => !row.delivery?.queueId); + + if (free.length === 0) { + const detail = "They are in the session's queue already."; + result.sessions.push({ + session, + numbers, + result: { session, text: "", channel: "queued", delivered: false, detail }, + }); + result.queued += rows.length; + continue; + } + + result.queued += rows.length - free.length; + + try { + const sent = await sendAnsweredDecisions({ + session, + ids: free.map((row) => row.id), + ...(target.provider ? { provider: target.provider } : {}), + ...(target.sourceHome ? { sourceHome: target.sourceHome } : {}), + files, + deps: deliver, + }); + const delivered = sent.delivered === true; + const detail = sent.target ?? sent.error; + result.sessions.push({ + session, + numbers, + result: { + session, + text: sent.text, + channel: sent.channel ?? "queued", + delivered, + ...(detail ? { detail } : {}), + ...(sent.target ? { target: sent.target } : {}), + ...(sent.error ? { error: sent.error } : {}), + }, + }); + + if (delivered) { + result.sent += free.length; + } else { + result.queued += free.length; + } + } catch (error) { + result.sessions.push({ session, numbers, error: error instanceof Error ? error.message : String(error) }); + result.failed += free.length; + } + } + + return result; +} + +/** Discards the unsent pick and note of each staged decision in `ids`; the decision still waits for an answer. */ +export async function unstageDecisions({ + ids, + files = decisionFiles(), +}: { + ids: readonly string[]; + files?: { file: string; events: string }; +}): Promise { + const wanted = new Set(ids); + const updates = readDecisions(files.file) + .filter((row) => wanted.has(row.id) && isStaged(row)) + .map((row) => ({ id: row.id, draft: "", draftOption: "" })); + + if (updates.length > 0) { + await updateDecisions(files.file, files.events, { updates }); + } + + return updates.length; +} diff --git a/src/hub/lib/widget/types.ts b/src/hub/lib/widget/types.ts index 7f19052541..db7ab9868b 100644 --- a/src/hub/lib/widget/types.ts +++ b/src/hub/lib/widget/types.ts @@ -67,6 +67,32 @@ export const widgetPreferencesSchema = z.object({ alertSound: z.string().trim().min(1).max(40).default("Glass"), /** Show the widget's own popup with an alert. */ alertPopup: z.boolean().default(true), + /** The agents hover list: only sessions that need you, or every active session too. */ + previewFilter: z.enum(["needs-you", "active"]).default("needs-you"), + /** The agents hover list's order: waiting for you first, newest activity first, or by project. */ + previewSort: z.enum(["attention", "recent", "project"]).default("attention"), + /** The sound an answer or message plays; "same" uses `alertSound`, "none" plays no sound. */ + alertAnswerSound: z.string().trim().min(1).max(40).default("same"), + /** Seconds the session that raised an alert flashes in the widget; 0 does not flash it. */ + alertFlashSeconds: z.number().int().min(0).max(30).default(5), + /** Where a session inbox puts its reply composer: under the tabs, or below the list. */ + inboxComposer: z.enum(["top", "bottom"]).default("top"), + /** A rail session bubble's tooltip: only while hover previews are off ("auto"), always, or never. */ + bubbleTooltip: z.enum(["auto", "always", "never"]).default("auto"), + /** The rail's session bubbles: in the hover list's order ("list"), or each keeps its place ("stable"). */ + railOrder: z.enum(["list", "stable"]).default("list"), + /** A click on a decision option: stages the pick to send later with others ("stage"), or sends it at once. */ + decisionClick: z.enum(["stage", "send"]).default("stage"), + /** Leaving the inbox with staged decisions: ask what to do ("ask"), keep them staged, or send them. */ + stagedOnLeave: z.enum(["ask", "keep", "send"]).default("ask"), + /** The inbox of every session: entries grouped under their session ("session"), or one list. */ + inboxGrouping: z.enum(["session", "list"]).default("session"), + /** The filter a session's inbox opens with: every entry ("all"), the inbox entries, or the decisions. */ + inboxFilter: z.enum(["all", "inbox", "decisions"]).default("all"), + /** The filter the inbox of every session opens with: every entry ("all"), the ones that need you, or unread. */ + overviewFilter: z.enum(["all", "needs", "unread"]).default("all"), + /** The composer's microphone: records a voice note and puts its transcript in the field ("note"), or live dictation. */ + composerVoice: z.enum(["ask", "note", "live"]).default("ask"), }); // Zod defaults also run inside partial objects; a patch must retain only supplied keys. diff --git a/src/hub/lib/widget/voice-notes.ts b/src/hub/lib/widget/voice-notes.ts index 40d8197012..831ed972e9 100644 --- a/src/hub/lib/widget/voice-notes.ts +++ b/src/hub/lib/widget/voice-notes.ts @@ -1,9 +1,10 @@ import { createHash, randomUUID } from "node:crypto"; import { existsSync, realpathSync, unlinkSync } from "node:fs"; -import { mkdir, unlink } from "node:fs/promises"; +import { chmod, mkdir, rename, stat, unlink } from "node:fs/promises"; import { join, resolve } from "node:path"; import { parseSttProvider } from "@genesiscz/utils/ai/stt/resolve"; import type { LiveTranscriptEvent } from "@genesiscz/utils/ai/stt/types"; +import { pcmRms } from "@genesiscz/utils/ai/stt/vad"; import { type RecordingEvent, recordPcmClip } from "@genesiscz/utils/ai/voice/record"; import { createVoiceSession, type VoiceEvent } from "@genesiscz/utils/ai/voice/session"; import { SafeJSON } from "@genesiscz/utils/json"; @@ -13,17 +14,30 @@ import { withFileLock } from "@genesiscz/utils/storage/file-lock"; import { z } from "zod"; import { widgetRoot } from "./storage"; +const MAX_CLIP_BYTES = 9_600_000; + const clipSchema = z.object({ path: z.string(), - bytes: z.number().int().positive().max(960_000), + // A recorded note stops at 30 s; one kept from live dictation may run up to five minutes. + bytes: z.number().int().positive().max(MAX_CLIP_BYTES), sampleRateHz: z.literal(16000), channels: z.literal(1), encoding: z.literal("s16le"), - durationMs: z.number().positive().max(30_000), + durationMs: z.number().positive().max(300_000), peakRms: z.number(), endedBy: z.enum(["eof", "stop", "limit"]), sha256: z.string().length(64), }); +/** Where a note was recorded: the widget's reply box of a session, for one decision or a follow-up. */ +export const voiceNoteContextSchema = z.object({ + sessionKey: z.string().max(1_000), + sessionTitle: z.string().max(500).optional(), + decisionId: z.string().max(500).optional(), + decisionNumber: z.number().int().positive().optional(), + title: z.string().max(500).optional(), +}); +export type VoiceNoteContext = z.infer; + export const voiceNoteSchema = z.object({ id: z.string().uuid(), revision: z.number().int().positive(), @@ -38,6 +52,8 @@ export const voiceNoteSchema = z.object({ language: z.string().optional(), /** The newest transcription attempt; only that attempt may write its outcome. */ operationId: z.string().uuid().optional(), + /** Recorded from a reply box: its session and decision. Absent for a note recorded in Voice Notes itself. */ + context: voiceNoteContextSchema.optional(), }); export type VoiceNote = z.infer; const indexSchema = z.object({ revision: z.number().int().nonnegative(), notes: z.array(voiceNoteSchema).max(100) }); @@ -148,11 +164,13 @@ export async function recordVoiceNote({ signal, stopSignal, onEvent, + context, }: { root?: string; id?: string; input?: string; micLauncher?: string; + context?: VoiceNoteContext; signal?: AbortSignal; stopSignal?: AbortSignal; onEvent?: (event: RecordingEvent) => void; @@ -186,6 +204,7 @@ export async function recordVoiceNote({ text: "", recognizedText: "", transcription: "none", + ...(context ? { context } : {}), }); let duplicate = false; try { @@ -223,6 +242,110 @@ export async function recordVoiceNote({ throw new Error(`The recording was discarded because it could not be saved: ${reason}`, { cause: error }); } } +/** Where `voice listen --save-audio` writes a reply box dictation before `addLiveVoiceNote` files it. */ +export function voiceNotesIncomingDirectory(root?: string): string { + return join(voiceNotesDirectory(root), "incoming"); +} + +/** A new file name in the incoming folder; the folder exists afterwards, private like every voice-note folder. */ +export async function incomingClipPath(root?: string): Promise { + await mkdir(voiceNotesIncomingDirectory(root), { recursive: true, mode: 0o700 }); + return join(voiceNotesIncomingDirectory(root), `${randomUUID()}.pcm`); +} + +/** + * Files the audio of a live dictation from the reply box as a voice note with its text, already transcribed, and + * where it was spoken (Martin, 2026-10-23: the dictation "can go into the voice notes place (with context of which + * session, which decision etc was it recorded in)"). The clip must be raw s16le 16 kHz mono in the incoming folder; + * it moves into the note's clip path. + */ +export async function addLiveVoiceNote({ + root, + clip, + text, + context, + provider, + model, + language, + signal, +}: { + root?: string; + clip: string; + text: string; + context?: VoiceNoteContext; + provider?: string; + model?: string; + language?: string; + signal?: AbortSignal; +}): Promise { + const incoming = resolve(voiceNotesIncomingDirectory(root)); + if (resolve(clip, "..") !== incoming) { + throw new Error("A live dictation clip must be in the voice notes incoming folder"); + } + + const clipStat = await stat(clip); + if (!clipStat.isFile() || clipStat.size > MAX_CLIP_BYTES) { + throw new Error("The dictation clip is not a file within the voice note size limit"); + } + + const audio = new Uint8Array(await Bun.file(clip).arrayBuffer()); + if (audio.byteLength < 2 || audio.byteLength % 2 !== 0) { + throw new Error("The dictation recorded no audio"); + } + + let peakRms = 0; + for (let offset = 0; offset < audio.byteLength; offset += 3_200) { + peakRms = Math.max(peakRms, pcmRms(audio.subarray(offset, Math.min(audio.byteLength, offset + 3_200)))); + } + + const id = randomUUID(); + const output = join(voiceNotesDirectory(root), "clips", `${id}.pcm`); + await mkdir(join(voiceNotesDirectory(root), "clips"), { recursive: true, mode: 0o700 }); + const note = voiceNoteSchema.parse({ + id, + revision: 1, + createdAt: Date.now(), + clip: { + path: output, + bytes: audio.byteLength, + sampleRateHz: 16000, + channels: 1, + encoding: "s16le", + durationMs: audio.byteLength / 32, + peakRms, + endedBy: "stop", + sha256: createHash("sha256").update(audio).digest("hex"), + }, + text, + recognizedText: text, + transcription: "ready", + ...(provider ? { provider } : {}), + ...(model ? { model } : {}), + ...(language ? { language } : {}), + ...(context ? { context } : {}), + }); + try { + await rename(clip, output); + // The recorder wrote the clip with the default mode; a voice note's audio is private (recordPcmClip's 0600). + await chmod(output, 0o600); + return await mutate({ + root, + signal, + change: (index) => { + if (index.notes.length >= 100) { + throw new Error("The local voice notebook is full. Discard a note before recording another."); + } + + index.notes.push(note); + return note; + }, + }); + } catch (error) { + await unlink(output).catch((cleanup) => logger.debug({ error: cleanup, id }, "Unsaved dictation clip cleanup")); + throw error; + } +} + export async function editVoiceNote({ root, id, diff --git a/src/hub/lib/widget/watch.ts b/src/hub/lib/widget/watch.ts index bc538f0cba..3552d986f1 100644 --- a/src/hub/lib/widget/watch.ts +++ b/src/hub/lib/widget/watch.ts @@ -34,18 +34,26 @@ const SAFETY_MS = 30_000; const DEGRADED_SAFETY_MS = 5000; /** Every this many safety ticks the whole catalog is refreshed, for changes outside the watched session roots. */ const FULL_REFRESH_TICKS = 4; +/** Inbox sessions whose cards every snapshot carries beside the selected one's (`widgetSnapshot({ warm })`). */ +export const WARM_SESSIONS = 16; export async function watchWidget({ root, selectedKey, signal, emit, + selections, dependencies = {}, }: { root?: string; selectedKey?: string; signal: AbortSignal; emit: (snapshot: Awaited>) => void; + /** + * The widget's picks as they happen (`{"select": key}` lines on stdin): the next snapshot names that session at + * once, without waiting for the `selection` action to reach `state.json` (a process and a debounce, about 0.5 s). + */ + selections?: (listener: (key: string) => void) => () => void; dependencies?: { prepare?: typeof prepareWidgetAsset; snapshot?: typeof widgetSnapshot; @@ -93,8 +101,13 @@ export async function watchWidget({ const startedAt = Date.now(); let emitted = false; const profiledInbox = new Map(); + /** The widget's latest pick, until `state.json` names it too. */ + let picked: string | undefined; const refresh = async () => { const state = await readWidgetState(directory); + if (picked && state.selectedKey === picked) { + picked = undefined; + } const referenced = new Set([ ...Object.values(state.drafts).flatMap((draft) => draft.assetIds), ...state.outgoing @@ -150,8 +163,9 @@ export async function watchWidget({ const snapshot = await prof.measureAsync("watch snapshot", () => readSnapshot({ root: directory, - selectedKey, + selectedKey: selectedKey ?? picked, refresh: false, + warm: WARM_SESSIONS, onImagesDeferred: () => void requestRefresh(), }) ); @@ -253,6 +267,10 @@ export async function watchWidget({ roster.refresh(); } const subscription = watchPath(join(directory, "state.json"), requestRefresh, { debounceMs: 180 }); + const stopSelections = selections?.((key) => { + picked = key; + void requestRefresh(); + }); let queueSubscription: WatcherSubscription | undefined; const inboxSubscriptions: WatcherSubscription[] = []; const sessionSubscriptions: WatcherSubscription[] = []; @@ -359,6 +377,7 @@ export async function watchWidget({ } }); } finally { + stopSelections?.(); cancelSafety?.(); roster?.stop(); await subscription.unsubscribe(); @@ -452,3 +471,41 @@ async function watchSessionRoots({ logger.debug({ roots: all.length, complete }, "Watching Widget session roots"); return complete; } + +/** Splits stdin chunks into lines; a partial line waits for the rest, and a runaway one is dropped. */ +export class StdinLines { + private rest = ""; + + constructor(private readonly onLine: (line: string) => void) {} + + readonly push = (chunk: Buffer | string): void => { + const lines = (this.rest + chunk.toString()).split("\n"); + const rest = lines.pop() ?? ""; + this.rest = rest.length > 4096 ? "" : rest; + for (const line of lines) { + if (line.trim()) { + this.onLine(line); + } + } + }; +} + +/** The session key of a `{"select": ""}` control line from the widget, else undefined. */ +export function widgetSelectionLine(line: string): string | undefined { + try { + const value: unknown = SafeJSON.parse(line, { strict: true }); + if ( + value && + typeof value === "object" && + "select" in value && + typeof value.select === "string" && + value.select + ) { + return value.select; + } + } catch (error) { + logger.debug({ error }, "Widget control line ignored"); + } + + return undefined; +} diff --git a/src/hub/widget-cli.ts b/src/hub/widget-cli.ts index 194d9cbda3..6ccb121029 100644 --- a/src/hub/widget-cli.ts +++ b/src/hub/widget-cli.ts @@ -18,9 +18,10 @@ import { readShelfAttachment, removeShelfItem, stageShelfImage, + stageShelfMedia, } from "./lib/widget/shelf"; import { discoverWidgetCatalog, widgetSnapshot } from "./lib/widget/snapshot"; -import { watchWidget } from "./lib/widget/watch"; +import { StdinLines, watchWidget, widgetSelectionLine } from "./lib/widget/watch"; import { registerWidgetTasks } from "./widget-tasks-cli"; import { registerWidgetVoiceNotes } from "./widget-voice-notes-cli"; @@ -132,8 +133,20 @@ widget const owner = new AbortController(); const abort = () => owner.abort(); signal.addEventListener("abort", abort, { once: true }); + const selections = new Set<(key: string) => void>(); + const lines = new StdinLines((line) => { + const key = widgetSelectionLine(line); + if (!key) { + return; + } + + for (const listener of selections) { + listener(key); + } + }); if (options.stopOnStdin) { process.stdin.on("end", abort); + process.stdin.on("data", lines.push); process.stdin.resume(); if (process.stdin.readableEnded) { owner.abort(); @@ -145,10 +158,15 @@ widget selectedKey: options.selected, signal: owner.signal, emit: (snapshot) => out.print(`${SafeJSON.stringify(snapshot)}\n`), + selections: (listener) => { + selections.add(listener); + return () => selections.delete(listener); + }, }); } finally { signal.removeEventListener("abort", abort); process.stdin.removeListener("end", abort); + process.stdin.removeListener("data", lines.push); if (options.stopOnStdin) { process.stdin.pause(); } @@ -195,6 +213,28 @@ shelf { handleTermination: true } ); }); +shelf + .command("video ") + .description( + "Stage a screen recording for later attachment; attached, it gets the frame review a pasted video gets" + ) + .option("--name ", "The name the shelf shows") + .action(async (input: string, options: { name?: string }, command) => { + await withInterrupt( + async (signal) => { + out.result( + await stageShelfMedia({ + root: command.optsWithGlobals().stateRoot, + input, + name: options.name, + signal, + type: "video", + }) + ); + }, + { handleTermination: true } + ); + }); shelf.command("capture").action(async (_options, command) => { await withInterrupt( async (signal) => { diff --git a/src/hub/widget-voice-notes-cli.ts b/src/hub/widget-voice-notes-cli.ts index e6de2dc961..5c83ba4daa 100644 --- a/src/hub/widget-voice-notes-cli.ts +++ b/src/hub/widget-voice-notes-cli.ts @@ -7,12 +7,15 @@ import { out } from "@genesiscz/utils/logger"; import type { Command } from "commander"; import { z } from "zod"; import { + addLiveVoiceNote, discardVoiceNote, editVoiceNote, + incomingClipPath, listVoiceNotes, readVoiceNote, recordVoiceNote, transcribeVoiceNote, + voiceNoteContextSchema, } from "./lib/widget/voice-notes"; export function registerWidgetVoiceNotes(widget: Command): void { @@ -23,6 +26,41 @@ export function registerWidgetVoiceNotes(widget: Command): void { .action(async (_options, command) => { out.result(await listVoiceNotes(command.optsWithGlobals().stateRoot)); }); + notes + .command("add") + .description("File a live dictation's audio and text as a voice note") + .requiredOption("--clip ", "raw s16le 16 kHz mono audio in the voice notes incoming folder") + .requiredOption("--text ", "the dictated text") + .option( + "--context ", + "where it was spoken: {sessionKey, sessionTitle?, decisionId?, decisionNumber?, title?}" + ) + .option("--provider ") + .option("--model ") + .option("--language ") + .option("--json") + .action(async (options, command) => { + const context = options.context + ? voiceNoteContextSchema.parse(SafeJSON.parse(options.context, { strict: true })) + : undefined; + const note = await addLiveVoiceNote({ + root: command.optsWithGlobals().stateRoot, + clip: options.clip, + text: options.text, + context, + provider: options.provider, + model: options.model, + language: options.language, + }); + out.result({ kind: "added", note }); + }); + notes + .command("incoming-path") + .description("A fresh path in the incoming folder for voice listen --save-audio") + .option("--json") + .action(async (_options, command) => { + out.result({ path: await incomingClipPath(command.optsWithGlobals().stateRoot) }); + }); notes .command("record") .option("--id ") @@ -30,8 +68,15 @@ export function registerWidgetVoiceNotes(widget: Command): void { .option("--mic-launcher ") .option("--wait-for-start") .option("--stop-on-stdin") + .option( + "--context ", + "where it was recorded: {sessionKey, sessionTitle?, decisionId?, decisionNumber?, title?}" + ) .option("--json") .action(async (options, command) => { + const context = options.context + ? voiceNoteContextSchema.parse(SafeJSON.parse(options.context, { strict: true })) + : undefined; if (options.input === "-" && (options.waitForStart || options.stopOnStdin)) { throw new Error("PCM stdin cannot also control recording"); } @@ -58,6 +103,7 @@ export function registerWidgetVoiceNotes(widget: Command): void { signal, stopSignal: control?.stopSignal, onEvent: (event) => out.print(`${SafeJSON.stringify(event)}\n`), + context, }); out.result({ kind: "recorded", note }); } catch (error) { diff --git a/src/macos/GenesisKit/Package.swift b/src/macos/GenesisKit/Package.swift index 11be8b017d..36fe69b12f 100644 --- a/src/macos/GenesisKit/Package.swift +++ b/src/macos/GenesisKit/Package.swift @@ -35,6 +35,7 @@ let kitProducts: [Product] = [ // Test targets only: the one copy of the `tools` disk fixtures (KitFixture). The GenesisTools bun tests // read the same files by path. An app target never links it. .library(name: "GenesisKitFixtures", targets: ["GenesisKitFixtures"]), + .library(name: "GenesisKitTestHooks", targets: ["GenesisKitTestHooks"]), ] // The markdown viewer's "Native+" renderer: diagrams and highlighted code without a web view. @@ -62,11 +63,14 @@ let kitTargets: [Target] = [ .target(name: "GenesisKitDisk", dependencies: ["GenesisKit"], path: "Sources/GenesisKitDisk"), // Disk/: the frozen CLI contracts and real `tools` JSON output the disk decoders are tested against. .target(name: "GenesisKitFixtures", path: "Sources/GenesisKitFixtures", resources: [.copy("Disk")]), + // Tests only: a window asked to become key is only ordered front, and the app never activates, so a test run + // never takes the user's keyboard focus. + .target(name: "GenesisKitTestHooks", path: "Sources/GenesisKitTestHooks", linkerSettings: [.linkedFramework("AppKit")]), .testTarget( name: "GenesisKitTests", dependencies: [ "GenesisKitBase", "GenesisKit", "GenesisKitCharts", "GenesisKitOverlay", "GenesisKitPosture", "GenesisKitDisk", - "GenesisKitFixtures", + "GenesisKitFixtures", "GenesisKitTestHooks", ], path: "Tests/GenesisKitTests" ), diff --git a/src/macos/GenesisKit/README.md b/src/macos/GenesisKit/README.md index 7af9c305bc..210fb66f67 100644 --- a/src/macos/GenesisKit/README.md +++ b/src/macos/GenesisKit/README.md @@ -66,6 +66,7 @@ A fixture an app test needs goes there under its own name, never into a test tar | `Perf/` | `PerfLog`, `HangWatch`, `MainStackSampler`, `MonitorPerf`, `RenderProbe`, `PerfConfiguration`, `MainBusy` (main-thread busy time after an event, `measureUntilSettled`, and a `Meter` for benches), `SessionOpenBench` (times one open of a session screen for the bench tests of both apps) | | `Tools/` | `ToolsBridge` (runs `tools`), `ToolsLineStream` (one long-running `tools` child, stdout as whole lines on the main queue, stdin held open so the child ends with the app), `MonitorJSON`, `TitleFormatter`, `SessionTranscriptClient` (the `tools ai sessions tail` envelope), `TranscriptPromptPart`, `DiskCache` (last answers on disk; `load` / `loadData` read off the main thread), `DirectoryWatcher` (one FSEvents stream over several folders, a path filter, main-queue callback) | | `Providers/` | `AIProviderMeta`, `AIProviders`, `AIProviderGlyph` | +| `Recording/` | screen recording for any host: `ScreenRecorder` (picker, grant, session, floating controls) with `ScreenRecordingHost`, `RecordingTargetPicker`, `RecordingControlsView`, `RecordingSessionController` (one `tools control capture session`), `RecordingTarget(s)`, `RecordingSettings` + `RecordingSettingsView` | | `Sessions/` | `SessionPalette`, `SessionFormat` | | `Sessions/Transcript/` | `SessionTranscriptList`, `TranscriptDocument`, prompt parts, `TranscriptScrollAnchor`, `TranscriptBus` / `TranscriptFilters`, `TranscriptMarkdownStyle`, `TranscriptLiveTail` (one `tools ai sessions tail --live` child, or a server subscription, per open screen; its `TranscriptLiveActivity` says what the agent does now, the list's last row through a `TranscriptActivityFeed` only that row observes) | | `Sessions/ToolCalls/` | `SessionToolCallView` + `TranscriptServices`, `SessionNativeLog` + `SessionNativeLogStore`, `SessionToolChanges`, `BatchedToolChangeSource` + `ToolChangeBatcher` (rows that ask within 100 ms share one `tools agents changes` run) | @@ -101,6 +102,7 @@ count of the section. - [Pose](#pose) (8) - [Posture](#posture) (9) - [Providers](#providers) (3) +- [Recording](#recording) (7) - [Sessions](#sessions) (65) - [Settings](#settings) (49) - [Style](#style) (32) @@ -6686,6 +6688,102 @@ HStack(spacing: 6) { } ``` +### Recording + +Screen recording for any host. `ScreenRecorder` is the entry point: it shows `RecordingTargetPicker` in a floating panel (a window, an app or a display), checks the Screen Recording grant, runs `tools control capture session` through `RecordingSessionController`, and floats `RecordingControlsView` next to what is recorded: pause or resume, switch what is recorded, restart, the host's "stop and use" action and stop and discard. The host says what "use" means through `ScreenRecordingHost` (the widget pastes the movie into a session's draft as "Stop and paste", the Capture shelf stages it as "Stop and save"). The recorder records in segments and joins them into one H.264 MP4 of at most ten minutes; the controls panel is left out of every movie by its window id, because ScreenCaptureKit ignores `sharingType = .none` on macOS 15 and later. One recording runs at a time in a process. `RecordingSettings` holds the preferences and `RecordingSettingsView` is their settings page. + +#### `ScreenRecorder` + +The whole flow for one host. It is `@MainActor` and an `ObservableObject`. + +- `bridge: ToolsBridge`: runs `tools control capture targets --json` and `tools control capture session`. +- `settings: RecordingSettings? = nil`: the preferences; `RecordingSettings.shared` when nil. +- `permissionGate: ((PermissionNeed) async -> Bool)? = nil`: the Screen Recording check; `PermissionCenter.shared.ensure` when nil. + +Properties and methods: + +- `isActive: Bool`: true from the picker's first appearance until the recording ended. +- `begin(host: ScreenRecordingHost, anchor: CGPoint? = nil, preferredPid: Int32? = nil)`: opens the picker near `anchor` (a Cocoa screen point, the pointer when nil), starting on the front window of `preferredPid` (else the frontmost app that is not this process), then records the choice. While another recording runs it brings that one's controls forward instead. +- `stopAndUse()`, `discard()`: what the controls' buttons do, for a host control (the widget's video button stops and pastes on a second press). +- `showControls()`: brings the controls forward. + +Switching target pauses the recording while the picker is open, so the movie never shows the picker; a cancel resumes. + +```swift +let recorder = ScreenRecorder(bridge: bridge) +recorder.begin(host: ScreenRecordingHost( + useTitle: "Stop and save", + permissionReason: "Records the window, app or display you choose.", + onUse: { movie in try await library.add(movie) }, + onError: { message in notice = message })) +``` + +#### `ScreenRecordingHost` + +What a host does with a recording. + +- `useTitle: String`: the name of "stop and keep" in the controls, such as "Stop and paste". +- `useSymbol: String = "stop.fill"`: its symbol. +- `permissionReason: String`: why the recording needs Screen Recording, shown in the permission dialog. +- `onUse: (URL) async throws -> Void`: gets the finished movie; the file is the host's from then on. A throw is reported through `onError` with the movie's path. +- `onDiscard: () -> Void = {}`: the person discarded the recording. +- `onError: (String) -> Void`: the recording could not start or failed, or `onUse` threw. +- `onStart: () -> Void = {}`: the controls are up and the recorder starts; a host can step aside (the widget collapses). + +#### `RecordingTargetPicker` + +Picks what to record: tabs for windows (grouped under their app, with titles and sizes), apps and displays, a filter field, and the rows as buttons. It starts on `initial`. ↑ and ↓ move, Return records, Esc cancels, a click on a row records it. Values and closures only. + +- `load: RecordingTargetPicker.Load`: `.loading`, `.failed(String)` (with a Try again button) or `.loaded(RecordingTargets)`. +- `title: String = "Choose what to record"` +- `initial: RecordingTarget?`: the highlighted row and the first tab. +- `onChoose: (RecordingTarget) -> Void`, `onCancel: () -> Void`, `onRetry: () -> Void = {}` + +```swift +RecordingTargetPicker(load: .loaded(targets), initial: targets.defaultTarget(preferredPid: pid), + onChoose: { start($0) }, onCancel: { close() }) +``` + +#### `RecordingControlsView` + +The floating controls of a running recording: a grip, the recorded time with a red dot (a spinner while starting or saving, a pause glyph while paused, the time left in the last minute), what is recorded as a chip that switches it, pause or resume, restart, the use button and stop and discard. The grip, dot and time drag the window. With `confirmsDiscard`, discard and restart ask inside the controls once something was recorded ("Discard 1:23 of recording?" with Discard and Keep). + +- `state: RecordingControlsState`: `phase`, `elapsedMs`, `limitMs`, `target`, `pauseReason` (`user`, `limit`, `recorder`), `notice`, `useTitle`, `useSymbol`, `confirmsDiscard`. +- `actions: RecordingControlsActions`: `pause`, `resume`, `switchTarget`, `restart`, `use`, `discard`, `dismissNotice`, `moved`. + +```swift +RecordingControlsView( + state: RecordingControlsState(phase: .recording, elapsedMs: 83_000, target: target, useTitle: "Stop and paste"), + actions: RecordingControlsActions(pause: pause, resume: resume, switchTarget: pick, restart: restart, + use: stop, discard: discard)) +``` + +#### `RecordingSessionController` + +One `tools control capture session --json --stop-on-stdin` child on `ToolsLineStream`. It publishes `phase` (`RecordingPhase`: idle, starting, recording, paused, finishing, done, discarded, failed), `elapsedMs` (time in the movie: no arming, no paused time), `limitMs`, `target`, `pauseReason`, `segments` and `notice`, and calls `onDone(URL, durationMs)`, `onDiscarded()` or `onFailed(message, code)` (`screen-recording-denied` for a missing grant). + +- `start(target:excludeWindowIDs:frameRate:showsBorder:) throws`: `excludeWindowIDs` are windows no movie may show. +- `pause()`, `resume()`, `switchTo(_:)`, `restart()`, `stop()` (keeps the movie), `discard()`, `terminate()` (SIGTERM: the session stops and keeps what it recorded). + +#### `RecordingTarget`, `RecordingTargets` + +`RecordingTarget` is `.window(ids:app:title:pid:)`, `.app(app:name:)` (a pid, name or bundle id) or `.display(index:name:)` (NSScreen order); `json` is the CLI's form, and `title`, `subtitle` and `symbol` describe it. `RecordingTargets` decodes `tools control capture targets --json`: `windows` (`RecordingWindow`: id, pid, app, title, bounds in CG points), `apps` (`RecordingApp`), `displays` (`RecordingDisplay`) and `frontmostPid`; `excluding(pids:)` drops a host's own process and `defaultTarget(preferredPid:)` is the front window of that app, else of the frontmost app, else the main display. + +#### `RecordingSettings`, `RecordingSettingsView` + +The recorder's preferences in the app's defaults (`recording.*`), with their page (the widget shows it as Widgets › Screen recording): + +- `placement: RecordingPanelPlacement = .nearTarget`: where the controls appear (under the recorded window, or at the bottom of a recorded display), or a corner of that display. A drag moves them for the rest of the recording. +- `frameRate: Int = 15`: frames per second while something moves (5, 10, 15, 24 or 30); a still screen records at 2. +- `showsBorder: Bool = true`: the pulsing red border around what is recorded. It is never in the movie. +- `confirmsDiscard: Bool = true`: discard and restart ask first. + +```swift +NativeSettingsPage(id: "app.recording", title: "Screen recording", symbol: "record.circle", tint: .red) { + RecordingSettingsView() +} +``` + ### Sessions Sessions shows one AI coding session (Claude, Codex or Grok) as a screen. `SessionDetailScreen` is the main entry point: a header, a transcript (`SessionTranscriptList`) and a 300 pt sidebar of usage, files, commits and sub-agents, and `SessionSidebarSplit` lets the sidebar cover the transcript in a narrow pane. A host turns `tools ai sessions tail` turns into a `TranscriptDocument` and a `SessionActivityDigest`, fills `SessionDetailInfo` for the header and passes its buttons as closures in `SessionDetailActions`. The host hooks are `TranscriptServices` (the session file, change log, row actions, notice buttons and the search callback for the tool rows), `SessionDetailActions`, `TranscriptBus` (jump and reveal messages from the sidebar to the transcript) and `TranscriptFilters` (the tool filter of each session). `TranscriptLiveTail` follows a running session, `TranscriptActivityFeed` carries its live activity row, and `SessionPalette` and `SessionFormat` hold the shared look and number formats. The entries below start with the transcript, its document model, the live tail and the shared tokens. diff --git a/src/macos/GenesisKit/Sources/GenesisKit/Audio/VoiceLevelBars.swift b/src/macos/GenesisKit/Sources/GenesisKit/Audio/VoiceLevelBars.swift new file mode 100644 index 0000000000..6e57888dee --- /dev/null +++ b/src/macos/GenesisKit/Sources/GenesisKit/Audio/VoiceLevelBars.swift @@ -0,0 +1,44 @@ +import SwiftUI + +/// Seven bars that move with the voice level while someone talks: the Flow pill's bars, shared so the widget's reply +/// box shows the same waves (Martin, 2026-10-23: "you should show me the "waves" as i am talking"). +/// +/// The recognisers expose one RMS level, not a spectrum, so these are one amplitude shaped by fixed per-bar weights: +/// a motion cue, not a frequency display. +public struct VoiceLevelBars: View { + /// The level, 0 to 1. + public var level: Double + /// Listening: full height range and opacity. Otherwise the bars rest at their base height, dimmed. + public var active: Bool + public var tint: Color + public var reduceMotion: Bool + + public init(level: Double, active: Bool = true, tint: Color, reduceMotion: Bool = false) { + self.level = level + self.active = active + self.tint = tint + self.reduceMotion = reduceMotion + } + + static let weights: [Double] = [0.45, 0.7, 0.95, 1.0, 0.9, 0.65, 0.4] + + public var body: some View { + HStack(spacing: 3) { + ForEach(0.. CGFloat { + let base: CGFloat = 4 + guard active else { return base } + return base + CGFloat(max(0, min(1, level)) * Self.weights[index]) * 16 + } +} diff --git a/src/macos/GenesisKit/Sources/GenesisKit/Clicky/ClickyAudio.swift b/src/macos/GenesisKit/Sources/GenesisKit/Clicky/ClickyAudio.swift index 6837d0bb06..70a0cd7b8c 100644 --- a/src/macos/GenesisKit/Sources/GenesisKit/Clicky/ClickyAudio.swift +++ b/src/macos/GenesisKit/Sources/GenesisKit/Clicky/ClickyAudio.swift @@ -36,6 +36,24 @@ final class ClickyAudio { private var buffers: [String: AVAudioPCMBuffer] = [:] private var cursor = 0 private var idleStop: DispatchWorkItem? + /// The engine starts here, never on the main thread: the start ran inside the keyboard event tap and took 0.4 to + /// 1 s after each 3 s pause on 2026-10-23 (HangWatch stalls in the settings face), and a tap that waits delays + /// every key the user types. + private let startQueue = DispatchQueue(label: "genesiskit.clicky.audio-start", qos: .userInteractive) + private var starting = false + /// Whether the engine runs, as this class started and stopped it. `AVAudioEngine.isRunning` takes the engine's + /// lock, which a start on the start queue holds for its whole length: reading it there blocked the key tap again + /// (HangWatch, 2026-10-23 21:14). + private var running = false + /// The click that arrived while the engine starts; it plays once the engine runs (a newer one replaces it). + private var pendingClick: (() -> Void)? + private var stopAfterStart = false + /// A hardware change stopped the engine while it was starting; the start then finished on a stopped engine. + private var configChangedWhileStarting = false + private final class EngineBox: @unchecked Sendable { + let engine: AVAudioEngine + init(_ engine: AVAudioEngine) { self.engine = engine } + } private struct PackBank { let reference: ClickyPackReference let manifestHash: String @@ -133,6 +151,13 @@ final class ClickyAudio { } } engine.prepare() + // A hardware change stops the engine without telling `running`; the next click must start it again. + NotificationCenter.default.addObserver(forName: .AVAudioEngineConfigurationChange, object: engine, queue: .main) { [weak self] _ in + MainActor.assumeIsolated { + guard let self else { return } + if self.starting { self.configChangedWhileStarting = true } else { self.running = false } + } + } } func play(profile: ClickySwitch, release: Bool, preferences: ClickyPreferences, pan: Float) throws { @@ -144,9 +169,63 @@ final class ClickyAudio { private func schedule(_ buffer: AVAudioPCMBuffer, preferences: ClickyPreferences, pan: Float, gain: Float, pitchVariation: Float) throws { guard preferences.volume > 0 else { return } idleStop?.cancel() - if !engine.isRunning { - try PerfLog.span("clicky.audio.start") { try engine.start() } + // Offline rendering (tests) starts at once: no hardware to wake. + if !running, !starting, engine.isInManualRenderingMode { + try engine.start() + running = true + } + guard running else { + pendingClick = { [weak self] in + try? self?.play(buffer, preferences: preferences, pan: pan, gain: gain, pitchVariation: pitchVariation) + } + startEngine() + return } + try play(buffer, preferences: preferences, pan: pan, gain: gain, pitchVariation: pitchVariation) + } + + private func startEngine() { + // A click after a stop request cancels the stop, also while a start is already in flight. + stopAfterStart = false + guard !starting else { return } + starting = true + let box = EngineBox(engine) + startQueue.async { + let begun = Date() + var failure: Error? + do { try box.engine.start() } catch { failure = error } + let ms = Date().timeIntervalSince(begun) * 1000 + DispatchQueue.main.async { [weak self] in + MainActor.assumeIsolated { + guard let self else { return } + self.starting = false + self.running = failure == nil + PerfLog.mark(String(format: "clicky.audio.start %.1fms off-main", ms) + + (failure.map { " failed: \($0.localizedDescription)" } ?? "")) + if self.configChangedWhileStarting { + // The engine stopped during the start: start it again before the waiting click plays. + self.configChangedWhileStarting = false + if !self.stopAfterStart { + self.running = false + self.startEngine() + return + } + } + if self.stopAfterStart { + self.pendingClick = nil + self.stopVoices() + return + } + let click = self.pendingClick + self.pendingClick = nil + if failure == nil { click?() } + } + } + } + } + + private func play(_ buffer: AVAudioPCMBuffer, preferences: ClickyPreferences, pan: Float, gain: Float, + pitchVariation: Float) throws { let voice = voices[cursor] cursor = (cursor + 1) % voices.count voice.0.volume = min(1, max(0, Float(preferences.volume) * gain)) @@ -169,6 +248,12 @@ final class ClickyAudio { private func stopVoices() { idleStop?.cancel() idleStop = nil + // A start in flight finishes first; the engine stops right after it. + if starting { + stopAfterStart = true + return + } + running = false for voice in voices { voice.0.stop() } engine.stop() } diff --git a/src/macos/GenesisKit/Sources/GenesisKit/Cmux/CmuxCommandRunner.swift b/src/macos/GenesisKit/Sources/GenesisKit/Cmux/CmuxCommandRunner.swift new file mode 100644 index 0000000000..18e6d23cc6 --- /dev/null +++ b/src/macos/GenesisKit/Sources/GenesisKit/Cmux/CmuxCommandRunner.swift @@ -0,0 +1,121 @@ +import Foundation +import GenesisKitBase + +/// The cmux CLI for every part that opens a session or a command in a pane: the hub's `CmuxHost` and the widget's cmux +/// button. One binary lookup, the new workspace / pane / tab / type-into-a-tab moves, and the reading of +/// `tools cmux focus --json`. Blocking (each call spawns a process): call it off the main thread. +public enum CmuxCommandRunner { + public static var binary: String? { + ["/Applications/cmux.app/Contents/Resources/bin/cmux", "~/.local/bin/cmux", "/usr/local/bin/cmux", "/opt/homebrew/bin/cmux"] + .map { ($0 as NSString).expandingTildeInPath } + .first { FileManager.default.isExecutableFile(atPath: $0) } + } + + @discardableResult + public static func cmux(_ args: [String]) -> (ok: Bool, out: String) { + guard let binary else { return (false, "cmux is not installed") } + let process = Process() + process.executableURL = URL(fileURLWithPath: binary) + process.arguments = args + var environment = ProcessInfo.processInfo.environment + environment["CMUX_QUIET"] = "1" + process.environment = environment + guard let result = try? process.runCapturing() else { return (false, "cmux failed to start") } + let text = String(decoding: result.stdout, as: UTF8.self) + return (result.status == 0, result.status == 0 ? text : String(decoding: result.stderr, as: UTF8.self) + text) + } + + /// Runs `command` in `cwd` at `target`. An error message, or nil when it opened. Words are shell-quoted here because + /// cmux takes one command string. + public static func run(_ command: [String], cwd: String, name: String, at target: CmuxTarget) -> String? { + let words = command.map(ShellWords.word).joined(separator: " ") + let line = cwd.isEmpty ? words : "cd \(ShellWords.word(cwd)) && \(words)" + switch target { + case .newWorkspace(let window): + let result = cmux(["new-workspace", "--name", name, "--cwd", cwd, "--command", words, "--focus", "true"] + + (window.map { ["--window", $0] } ?? [])) + return result.ok ? nil : result.out.trimmed + case .newPane(let workspace): + return typeInto(cmux(["new-pane", "--type", "terminal", "--direction", "right", "--workspace", workspace, + "--focus", "true"]), workspace: workspace, line: line) + case .newTab(let workspace, let pane): + return typeInto(cmux(["new-surface", "--type", "terminal", "--workspace", workspace, "--pane", pane, + "--focus", "true"]), workspace: workspace, line: line) + case .surface(let workspace, let surface): + return send(line, workspace: workspace, surface: surface) + } + } + + /// new-surface / new-pane print the new surface ref; the command is typed into it, then Enter. + private static func typeInto(_ created: (ok: Bool, out: String), workspace: String, line: String) -> String? { + guard created.ok else { return created.out.trimmed } + guard let range = created.out.range(of: #"surface:\d+"#, options: .regularExpression) else { + return "cmux did not report the new surface: \(created.out.trimmed)" + } + return send(line, workspace: workspace, surface: String(created.out[range])) + } + + private static func send(_ line: String, workspace: String, surface: String) -> String? { + let sent = cmux(["send", "--workspace", workspace, "--surface", surface, line]) + guard sent.ok else { return sent.out.trimmed } + let enter = cmux(["send-key", "--workspace", workspace, "--surface", surface, "Enter"]) + return enter.ok ? nil : enter.out.trimmed + } + + /// The command that resumes an agent session in a terminal, or nil for an agent with none. + public static func resumeCommand(provider: String, sessionId: String) -> [String]? { + switch provider { + case "claude": return ["tools", "claude", "resume", sessionId] + case "codex": return ["codex", "resume", sessionId] + case "grok": return ["grok", "--resume", sessionId] + default: return nil + } + } + + /// The `tools ` that owns a session's cmux resolver: `tools codex cmux` and `tools grok cmux` know their own + /// history (Grok has no journal and its own fallback); anything else goes through Claude's. + public static func tool(provider: String?) -> String { + switch provider { + case "codex", "grok": return provider ?? "claude" + default: return "claude" + } + } + + public static let paneGone = "That cmux pane is gone. Choose a pane to resume the session in." + + /// What `tools cmux focus --json` said, in words: nil when it focused. A recorded tab that was closed + /// answers `gone`; before, the hub and Genesis showed cmux's raw "[socket] RPC error". + public static func focusFailure(status: Int32, stdout: Data, stderr: Data) -> String? { + guard status != 0 else { return nil } + struct Opaque: Decodable { + init(from decoder: Decoder) throws {} + } + struct Result: Decodable { + let gone: Bool? + let matches: [Opaque]? + } + if let result = try? JSONDecoder().decode(Result.self, from: stdout) { + if result.gone == true { + return paneGone + } + if result.matches?.isEmpty == true { + return "No cmux pane shows this session. Choose a pane to resume it in." + } + } + let lines = String(decoding: stderr, as: UTF8.self).split(whereSeparator: \.isNewline) + let said = lines.map { $0.trimmingCharacters(in: CharacterSet(charactersIn: "│■▲● ").union(.whitespaces)) } + .last { !$0.isEmpty } + return said.map { String($0) } ?? "tools claude cmux focus exited \(status)" + } + + /// `open-session --json` reports a failure as {ok: false, error} on stdout, with stderr empty. + public static func openSessionFailure(status: Int32, stdout: Data, stderr: Data) -> String? { + guard status != 0 else { return nil } + struct Failure: Decodable { let error: String } + if let failure = try? JSONDecoder().decode(Failure.self, from: stdout) { + return failure.error + } + let text = String(decoding: stderr, as: UTF8.self).trimmed + return text.isEmpty ? "tools claude cmux open-session exited \(status)" : String(text.suffix(300)) + } +} diff --git a/src/macos/GenesisKit/Sources/GenesisKit/Controls/AgentRosterRow.swift b/src/macos/GenesisKit/Sources/GenesisKit/Controls/AgentRosterRow.swift index 99831d3cba..c7d5783178 100644 --- a/src/macos/GenesisKit/Sources/GenesisKit/Controls/AgentRosterRow.swift +++ b/src/macos/GenesisKit/Sources/GenesisKit/Controls/AgentRosterRow.swift @@ -106,9 +106,12 @@ public struct AgentRosterGroupLabel: View { public let running: Int public var live: Bool public var lastAt: Date? + /// Inbox items waiting in the session, shown beside its name (the widget's session list). + public var unread: Int public init(title: String, provider: String, project: String? = nil, account: String? = nil, - total: Int, running: Int, live: Bool, lastAt: Date? = nil) { + total: Int, running: Int, live: Bool, lastAt: Date? = nil, unread: Int = 0) { + self.unread = unread self.title = title self.provider = provider self.project = project @@ -129,7 +132,13 @@ public struct AgentRosterGroupLabel: View { } }.padding(.top, 1) VStack(alignment: .leading, spacing: 3) { - Text(title).font(.system(size: 12.5, weight: .semibold)).lineLimit(2) + HStack(alignment: .firstTextBaseline, spacing: 6) { + Text(title).font(.system(size: 12.5, weight: .semibold)).lineLimit(2) + if unread > 0 { + Spacer(minLength: 0) + CountBadge(unread, tooltip: "\(Plural.count(unread, "inbox item"))", color: KitPalette.modified) + } + } HStack(spacing: 6) { if let project { Text(project).layoutPriority(-2) } if let account { diff --git a/src/macos/GenesisKit/Sources/GenesisKit/Controls/FittedScroll.swift b/src/macos/GenesisKit/Sources/GenesisKit/Controls/FittedScroll.swift index 13c6a889d9..97bd0a00bb 100644 --- a/src/macos/GenesisKit/Sources/GenesisKit/Controls/FittedScroll.swift +++ b/src/macos/GenesisKit/Sources/GenesisKit/Controls/FittedScroll.swift @@ -11,14 +11,23 @@ import SwiftUI public struct FittedScroll: View { public var maxHeight: CGFloat public var minHeight: CGFloat + /// The height until the content is first measured, and every measurement goes to `measured`. A view that is + /// built again each time it shows (the widget's hover list) passes its last height here: from 0, the first layout + /// pass reported a list-less height, its window shrank and grew back within 40 ms, and the panel blinked + /// (edge.request log, 2026-10-23: 336, then 295, then 489 pt for one hover). + public var initialHeight: CGFloat + private let measured: ((CGFloat) -> Void)? private let content: () -> Content @State private var height: CGFloat = 0 @State private var expansion = FittedExpansionNote() @GenReduceMotion private var reduceMotion - public init(maxHeight: CGFloat, minHeight: CGFloat = 0, @ViewBuilder content: @escaping () -> Content) { + public init(maxHeight: CGFloat, minHeight: CGFloat = 0, initialHeight: CGFloat = 0, + measured: ((CGFloat) -> Void)? = nil, @ViewBuilder content: @escaping () -> Content) { self.maxHeight = maxHeight self.minHeight = minHeight + self.initialHeight = initialHeight + self.measured = measured self.content = content } @@ -29,11 +38,12 @@ public struct FittedScroll: View { .scrollOverflowContent() } .scrollOverflowHints() - .frame(height: min(maxHeight, max(minHeight, height))) + .frame(height: min(maxHeight, max(minHeight, height > 0 ? height : initialHeight))) .onGenExpansion { expansion.note($0) } } private func apply(_ value: CGFloat) { + measured?(value) guard height > 0, let expanding = expansion.recent() else { height = value return diff --git a/src/macos/GenesisKit/Sources/GenesisKit/Controls/FrameProbe.swift b/src/macos/GenesisKit/Sources/GenesisKit/Controls/FrameProbe.swift new file mode 100644 index 0000000000..bd9b0acc8f --- /dev/null +++ b/src/macos/GenesisKit/Sources/GenesisKit/Controls/FrameProbe.swift @@ -0,0 +1,29 @@ +import SwiftUI + +/// Where a named control sits in its window, for a test that clicks it. A hidden test window has no accessibility tree, +/// so a test cannot find a control by its label; a click scan over points presses whatever lies there (it opened +/// Obsidian dialogs on the user's screen, 2026-10-23). Off unless a test turns it on; then `.frameProbe(name)` records +/// the view's frame in window coordinates (top-left origin, as SwiftUI's global space). +public enum FrameProbe { + @MainActor public static var enabled = false + @MainActor public private(set) static var frames: [String: CGRect] = [:] + + @MainActor static func record(_ name: String, _ frame: CGRect) { + frames[name] = frame + } + + @MainActor public static func reset() { + frames = [:] + } +} + +public extension View { + /// Records this view's window frame under `name` while `FrameProbe.enabled`; nothing at all otherwise. + @MainActor @ViewBuilder func frameProbe(_ name: String) -> some View { + if FrameProbe.enabled { + onGeometryChange(for: CGRect.self, of: { $0.frame(in: .global) }) { FrameProbe.record(name, $0) } + } else { + self + } + } +} diff --git a/src/macos/GenesisKit/Sources/GenesisKit/Controls/NoticePill.swift b/src/macos/GenesisKit/Sources/GenesisKit/Controls/NoticePill.swift index 3b86575263..48bb925fad 100644 --- a/src/macos/GenesisKit/Sources/GenesisKit/Controls/NoticePill.swift +++ b/src/macos/GenesisKit/Sources/GenesisKit/Controls/NoticePill.swift @@ -1,23 +1,91 @@ import SwiftUI -/// A short status line that fades after six seconds: icon, message, optional dim detail. An error -/// stays until it is dismissed. +/// One way to fix what a notice reports. A button says in its label what it does; a button that changes something +/// outside the view (quits an app, stops a process) asks first, inside the notice, so the question also works in a +/// panel that never becomes key. A command is for the person's terminal: the notice shows it and copies it. +public struct NoticeFix: Identifiable { + /// The question a button asks before it acts, and what its confirming button says. + public struct Confirmation { + public let question: String + public let consequence: String? + public let confirmLabel: String + + public init(_ question: String, consequence: String? = nil, confirmLabel: String) { + self.question = question + self.consequence = consequence + self.confirmLabel = confirmLabel + } + } + + public enum Kind { + case action(confirm: Confirmation?, perform: @MainActor () -> Void) + case command(String) + } + + public let label: String + public let symbol: String? + public let kind: Kind + public var id: String { + switch kind { + case .action: return "action:" + label + case .command(let command): return "command:" + command + } + } + + /// `label` says what a press does ("Quit the older Genesis (pid 1234)"); `confirm` makes it ask first. + public static func button(_ label: String, symbol: String? = nil, confirm: Confirmation? = nil, + perform: @escaping @MainActor () -> Void) -> NoticeFix { + NoticeFix(label: label, symbol: symbol, kind: .action(confirm: confirm, perform: perform)) + } + + /// `label` says what the command does ("Or quit it from a terminal"). + public static func command(_ label: String, _ command: String) -> NoticeFix { + NoticeFix(label: label, symbol: nil, kind: .command(command)) + } +} + +/// A status line: icon, message, optional dim detail, the fixes, and the full message behind "Details". A success +/// notice without fixes fades after six seconds; an error, a warning or a notice with a fix stays until it is +/// dismissed or no longer applies. public struct NoticePill: View { + public enum Tone: Sendable { + case success, warning, error + + var color: Color { + switch self { + case .success: return KitPalette.added + case .warning: return KitPalette.modified + case .error: return KitPalette.removed + } + } + + var symbol: String { self == .success ? "checkmark.circle.fill" : "exclamationmark.triangle.fill" } + } + let text: String var detail: String? - /// The full message behind a short `text` (a CLI's raw error), shown only after a click on "Details". + /// The full message behind a short `text` (a CLI's raw error, what was found), shown only after a click on "Details". var more: String? - var isError: Bool - let dismiss: () -> Void + var tone: Tone + var fixes: [NoticeFix] + /// Nil: no close icon, for a notice that stays while what it reports is true. + let dismiss: (() -> Void)? @State private var showsMore = false + @State private var confirming: String? @Environment(\.accessibilityReduceMotion) private var reduceMotion public init(text: String, detail: String? = nil, more: String? = nil, isError: Bool = false, - dismiss: @escaping () -> Void) { + fixes: [NoticeFix] = [], dismiss: (() -> Void)? = nil) { + self.init(text: text, detail: detail, more: more, tone: isError ? .error : .success, fixes: fixes, dismiss: dismiss) + } + + public init(text: String, detail: String? = nil, more: String? = nil, tone: Tone, + fixes: [NoticeFix] = [], dismiss: (() -> Void)? = nil) { self.text = text self.detail = detail self.more = more - self.isError = isError + self.tone = tone + self.fixes = fixes self.dismiss = dismiss } @@ -26,8 +94,7 @@ public struct NoticePill: View { // get a rounded box of the same look, and the text can be selected and copied. public var body: some View { HStack(alignment: .firstTextBaseline, spacing: 8) { - Image(systemName: isError ? "exclamationmark.triangle.fill" : "checkmark.circle.fill") - .foregroundColor(isError ? KitPalette.removed : KitPalette.added) + Image(systemName: tone.symbol).foregroundColor(tone.color) VStack(alignment: .leading, spacing: 2) { Text(text).font(.system(size: 12, weight: .medium)) .fixedSize(horizontal: false, vertical: true) @@ -35,6 +102,13 @@ public struct NoticePill: View { Text(detail).font(.system(size: 11.5)).foregroundColor(KitPalette.dim) .fixedSize(horizontal: false, vertical: true) } + if !fixes.isEmpty { + VStack(alignment: .leading, spacing: 6) { + ForEach(fixes) { fix in fixRow(fix) } + } + .padding(.top, 5) + .padding(.bottom, 2) + } if let more { Button { GenMotion.perform(expanding: !showsMore, reduceMotion: reduceMotion) { showsMore.toggle() } @@ -56,16 +130,94 @@ public struct NoticePill: View { } .textSelection(.enabled) .multilineTextAlignment(.leading) - IconButton(systemName: "xmark", tooltip: "Dismiss", size: 9, action: dismiss) + if let dismiss { + IconButton(systemName: "xmark", tooltip: "Dismiss", size: 9, action: dismiss) + } } .padding(.leading, 10) - .padding(.trailing, 4) - .padding(.vertical, 3) + .padding(.trailing, dismiss == nil ? 10 : 4) + .padding(.vertical, fixes.isEmpty ? 3 : 7) .background(RoundedRectangle(cornerRadius: 11, style: .continuous).fill(Color.white.opacity(0.06))) .overlay(RoundedRectangle(cornerRadius: 11, style: .continuous).stroke(Color.white.opacity(0.1))) .task(id: text) { try? await Task.sleep(nanoseconds: 6_000_000_000) - if !isError { dismiss() } + if tone == .success, fixes.isEmpty { dismiss?() } + } + } + + @ViewBuilder + private func fixRow(_ fix: NoticeFix) -> some View { + switch fix.kind { + case .action(let confirm, let perform): + VStack(alignment: .leading, spacing: 5) { + Button { + if confirm == nil { perform() } else { toggleConfirmation(fix.id) } + } label: { + fixLabel(fix.label, symbol: fix.symbol ?? "wrench.and.screwdriver", tint: tone == .success ? KitPalette.renamed : tone.color) + } + .buttonStyle(.genHoverPlain()) + .disabled(confirming == fix.id) + .accessibilityIdentifier("notice-fix-" + fix.label) + if let confirm, confirming == fix.id { + confirmation(confirm, perform: perform) + .transition(.genReveal) + } + } + case .command(let command): + VStack(alignment: .leading, spacing: 3) { + Text(fix.label).font(.system(size: 11)).foregroundColor(KitPalette.dim) + HStack(alignment: .firstTextBaseline, spacing: 4) { + Text(verbatim: command).font(.system(size: 11, design: .monospaced)) + .foregroundColor(Color.white.opacity(0.85)) + .fixedSize(horizontal: false, vertical: true) + .padding(.horizontal, 6).padding(.vertical, 3) + .background(RoundedRectangle(cornerRadius: 5).fill(Color.black.opacity(0.25))) + IconButton(systemName: "doc.on.doc", tooltip: "Copy the command", size: 10) { + Clipboard.copy(command, what: "command") + } + } + } + .accessibilityIdentifier("notice-command-" + fix.label) } } + + private func confirmation(_ confirm: NoticeFix.Confirmation, perform: @escaping @MainActor () -> Void) -> some View { + VStack(alignment: .leading, spacing: 5) { + Text(confirm.question).font(.system(size: 11.5, weight: .medium)) + .fixedSize(horizontal: false, vertical: true) + if let consequence = confirm.consequence { + Text(consequence).font(.system(size: 11)).foregroundColor(KitPalette.dim) + .fixedSize(horizontal: false, vertical: true) + } + HStack(spacing: 10) { + Button { + toggleConfirmation(nil) + perform() + } label: { + fixLabel(confirm.confirmLabel, symbol: "checkmark", tint: KitPalette.removed) + } + .buttonStyle(.genHoverPlain()) + .accessibilityIdentifier("notice-confirm-" + confirm.confirmLabel) + Button("Cancel") { toggleConfirmation(nil) } + .font(.system(size: 11)) + .foregroundColor(KitPalette.dim) + .buttonStyle(.genHoverPlain()) + } + } + .padding(8) + .background(RoundedRectangle(cornerRadius: 7, style: .continuous).fill(Color.white.opacity(0.05))) + } + + private func fixLabel(_ text: String, symbol: String, tint: Color) -> some View { + Label(text, systemImage: symbol) + .font(.system(size: 11, weight: .medium)) + .foregroundColor(tint) + .padding(.horizontal, 8).padding(.vertical, 3) + .background(Capsule().fill(tint.opacity(0.14))) + .contentShape(Capsule()) + } + + private func toggleConfirmation(_ id: String?) { + GenMotion.perform(expanding: id != nil, reduceMotion: reduceMotion) { confirming = id } + } } diff --git a/src/macos/GenesisKit/Sources/GenesisKit/Controls/ScrollOverflowHints.swift b/src/macos/GenesisKit/Sources/GenesisKit/Controls/ScrollOverflowHints.swift index cfd23936c5..461e0a483e 100644 --- a/src/macos/GenesisKit/Sources/GenesisKit/Controls/ScrollOverflowHints.swift +++ b/src/macos/GenesisKit/Sources/GenesisKit/Controls/ScrollOverflowHints.swift @@ -37,10 +37,17 @@ private struct ScrollOverflowFrameKey: PreferenceKey { public extension View { /// Inside a `ScrollView`, on its content: reports the content frame to `scrollOverflowHints()` on that scroll view. - func scrollOverflowContent() -> some View { - background(GeometryReader { proxy in - Color.clear.preference(key: ScrollOverflowFrameKey.self, value: proxy.frame(in: .named(ScrollOverflowSpace.name))) - }) + /// macOS 15 and later need nothing here: the hints read the scroll view's own geometry. The background reader + /// measured the whole content (every row of a lazy list) on each layout pass, a quarter of the 2026-10-23 widget + /// spins. + @ViewBuilder func scrollOverflowContent() -> some View { + if #available(macOS 15, *) { + self + } else { + background(GeometryReader { proxy in + Color.clear.preference(key: ScrollOverflowFrameKey.self, value: proxy.frame(in: .named(ScrollOverflowSpace.name))) + }) + } } /// On a `ScrollView` whose content carries `scrollOverflowContent()`: a soft fade at each edge that has more @@ -67,6 +74,23 @@ private struct ScrollOverflowHints: ViewModifier { } func body(content: Content) -> some View { + if #available(macOS 15, *) { + masked(content.onScrollGeometryChange(for: ScrollOverflow.self) { geometry in + let visible = geometry.visibleRect + return axis == .vertical + ? ScrollOverflow(above: visible.minY > 0.5, below: visible.maxY < geometry.contentSize.height - 0.5) + : ScrollOverflow(above: visible.minX > 0.5, below: visible.maxX < geometry.contentSize.width - 0.5) + } action: { _, next in + guard next != overflow else { return } + overflow = next + changed?(next) + }) + } else { + masked(measuredByPreference(content)) + } + } + + private func measuredByPreference(_ content: Content) -> some View { content .coordinateSpace(.named(ScrollOverflowSpace.name)) .onPreferenceChange(ScrollOverflowFrameKey.self) { frame in @@ -87,6 +111,10 @@ private struct ScrollOverflowHints: ViewModifier { update() } }) + } + + private func masked(_ view: some View) -> some View { + view .mask { if axis == .vertical { VStack(spacing: 0) { diff --git a/src/macos/GenesisKit/Sources/GenesisKit/Diff/Views/GitCommitPicker.swift b/src/macos/GenesisKit/Sources/GenesisKit/Diff/Views/GitCommitPicker.swift index b641b0c0cb..1a1fde58af 100644 --- a/src/macos/GenesisKit/Sources/GenesisKit/Diff/Views/GitCommitPicker.swift +++ b/src/macos/GenesisKit/Sources/GenesisKit/Diff/Views/GitCommitPicker.swift @@ -10,6 +10,7 @@ // `git log --no-walk` for all of its commits. // +import AppKit import SwiftUI /// How much a commit changed: every file, and the chooser's own file when it names one. @@ -218,10 +219,19 @@ public extension GitCommitPicker where Label == GitPickerChip { public extension View { /// The commit list as a popover from this view. Its rows are built when it opens, never while it is closed. + /// With `onPickRange`, Shift with the arrows or a Shift-click picks several commits in a row: the callback gets + /// them in list order, every commit between the two ends included, even ones the filter hides. func gitCommitPopover(isPresented: Binding, choices: [GitCommitChoice], selection: String?, title: String? = nil, - details: GitCommitDetailsLoader? = nil, onPick: @escaping (GitCommitChoice) -> Void) -> some View { + details: GitCommitDetailsLoader? = nil, onPick: @escaping (GitCommitChoice) -> Void, + onPickRange: (([GitCommitChoice]) -> Void)? = nil) -> some View { popover(isPresented: isPresented, arrowEdge: .bottom) { - GitCommitPickerList(choices: choices, selection: selection, title: title, details: details) { choice in + GitCommitPickerList(choices: choices, selection: selection, title: title, details: details, + pickRange: onPickRange.map { pickRange in + { range in + isPresented.wrappedValue = false + pickRange(range) + } + }) { choice in isPresented.wrappedValue = false onPick(choice) } @@ -246,6 +256,8 @@ struct GitCommitPickerState { private(set) var rows: [GitCommitChoice] = [] /// The row the keyboard or the pointer is on: Return picks it. var highlighted: String? + /// Where a Shift-extended range started; nil while one commit is chosen. + var anchor: String? init(choices: [GitCommitChoice], selection: String?, query: String = "") { self.choices = choices @@ -254,8 +266,35 @@ struct GitCommitPickerState { highlighted = selection.flatMap { sha in rows.first { $0.sha == sha }?.id } ?? rows.first?.id } - /// Moves the highlight `step` rows, stopping at either end; from no highlight, down starts at the top. + /// Moves the highlight `step` rows, stopping at either end; from no highlight, down starts at the top. A plain move + /// ends a range. mutating func move(_ step: Int) { + anchor = nil + advance(step) + } + + /// Shift with an arrow: the range grows from where it started. + mutating func extend(_ step: Int) { + if anchor == nil { anchor = highlighted } + advance(step) + } + + /// Shift-click: the range runs from where it started (or the highlighted row) to `id`. + mutating func extend(to id: String) { + if anchor == nil { anchor = highlighted } + highlighted = id + } + + /// The commits of a range, in the list's order and with every commit between its ends (also the ones the filter + /// hides, so the diff never skips one); nil while one commit is chosen. + var range: [GitCommitChoice]? { + guard let anchor, let highlighted, anchor != highlighted, + let start = choices.firstIndex(where: { $0.id == anchor }), + let end = choices.firstIndex(where: { $0.id == highlighted }) else { return nil } + return Array(choices[min(start, end)...max(start, end)]) + } + + private mutating func advance(_ step: Int) { guard !rows.isEmpty else { return } let index = highlighted.flatMap { id in rows.firstIndex { $0.id == id } } ?? (step > 0 ? -1 : rows.count) highlighted = rows[min(max(index + step, 0), rows.count - 1)].id @@ -280,6 +319,8 @@ struct GitCommitPickerList: View { let selection: String? let title: String? let details: GitCommitDetailsLoader? + /// Several commits in a row; nil offers one commit only. + let pickRange: (([GitCommitChoice]) -> Void)? let pick: (GitCommitChoice) -> Void @State private var state: GitCommitPickerState @@ -296,10 +337,11 @@ struct GitCommitPickerList: View { static let page = 8 init(choices: [GitCommitChoice], selection: String?, title: String?, details: GitCommitDetailsLoader?, query: String = "", - pick: @escaping (GitCommitChoice) -> Void) { + pickRange: (([GitCommitChoice]) -> Void)? = nil, pick: @escaping (GitCommitChoice) -> Void) { self.selection = selection self.title = title self.details = details + self.pickRange = pickRange self.pick = pick _state = State(initialValue: GitCommitPickerState(choices: choices, selection: selection, query: query)) } @@ -344,11 +386,9 @@ struct GitCommitPickerList: View { .textFieldStyle(.plain) .font(.system(size: 12.5)) .focused($searchFocused) - .onSubmit { - if let choice = state.highlightedChoice { pick(choice) } - } - .onKeyPress(.upArrow) { move(-1) } - .onKeyPress(.downArrow) { move(1) } + .onSubmit { submit() } + .onKeyPress(.upArrow, phases: [.down, .repeat]) { press in move(-1, extending: press.modifiers.contains(.shift)) } + .onKeyPress(.downArrow, phases: [.down, .repeat]) { press in move(1, extending: press.modifiers.contains(.shift)) } .onKeyPress(.pageUp) { move(-Self.page) } .onKeyPress(.pageDown) { move(Self.page) } .accessibilityLabel("Filter commits") @@ -374,13 +414,17 @@ struct GitCommitPickerList: View { ScrollViewReader { proxy in FittedScroll(maxHeight: Self.maxListHeight) { LazyVStack(spacing: 1) { + let inRange = Set(state.range?.map(\.id) ?? []) ForEach(state.rows) { choice in - GitCommitPickerRow(choice: choice, current: choice.sha == selection, highlighted: choice.id == state.highlighted) { - pick(choice) + GitCommitPickerRow(choice: choice, current: choice.sha == selection, + highlighted: choice.id == state.highlighted || inRange.contains(choice.id)) { + click(choice) } .id(choice.id) .onHover { inside in - guard inside, state.highlighted != choice.id, Date().timeIntervalSince(keyMovedAt) > 0.4 else { return } + // A range keeps its ends while the pointer crosses other rows. + guard inside, state.anchor == nil, state.highlighted != choice.id, + Date().timeIntervalSince(keyMovedAt) > 0.4 else { return } state.highlighted = choice.id } } @@ -413,7 +457,7 @@ struct GitCommitPickerList: View { .accessibilityLabel("Reading the line counts") } Spacer(minLength: 8) - Text("↑↓ move · ↩ show · esc close") + Text(verbatim: keysHint) .accessibilityHidden(true) } .font(.system(size: 10.5)) @@ -422,8 +466,40 @@ struct GitCommitPickerList: View { .padding(.vertical, 6) } - private func move(_ step: Int) -> KeyPress.Result { - state.move(step) + /// The keys, and with a range how many commits Return shows. + private var keysHint: String { + if let range = state.range { + return "\(Plural.count(range.count, "commit")) chosen · ↩ show them · ↑↓ back to one" + } + return pickRange == nil ? "↑↓ move · ↩ show · esc close" + : "↑↓ move · ⇧↑↓ or ⇧-click more · ↩ show · esc close" + } + + private func submit() { + if pickRange != nil, let range = state.range { + pickRange?(range) + } else if let choice = state.highlightedChoice { + pick(choice) + } + } + + /// A click shows that one commit; with Shift (when ranges are offered) it extends the range instead. + private func click(_ choice: GitCommitChoice) { + if pickRange != nil, NSEvent.modifierFlags.contains(.shift) { + state.extend(to: choice.id) + keyMovedAt = Date() + return + } + + pick(choice) + } + + private func move(_ step: Int, extending: Bool = false) -> KeyPress.Result { + if extending, pickRange != nil { + state.extend(step) + } else { + state.move(step) + } scrollTarget = state.highlighted keyMovedAt = Date() return .handled diff --git a/src/macos/GenesisKit/Sources/GenesisKit/Flow/FlowPillView.swift b/src/macos/GenesisKit/Sources/GenesisKit/Flow/FlowPillView.swift index 41f2e3acea..457c6cc590 100644 --- a/src/macos/GenesisKit/Sources/GenesisKit/Flow/FlowPillView.swift +++ b/src/macos/GenesisKit/Sources/GenesisKit/Flow/FlowPillView.swift @@ -96,26 +96,8 @@ public struct FlowPillView: View { /// not a frequency display. A real band split would need the FFT that /// BridgeVoice does in `compute_band_levels_into`. private var bars: some View { - HStack(spacing: 3) { - ForEach(0..<7, id: \.self) { index in - Capsule() - .fill(accent) - .frame(width: 3, height: barHeight(index)) - .opacity(session.phase == .listening ? 1 : 0.35) - } - } - .frame(width: 44, height: 22, alignment: .center) - .animation(reduceMotion ? nil : .easeOut(duration: 0.08), value: recognizer.micLevel) - } - - private static let barWeights: [Double] = [0.45, 0.7, 0.95, 1.0, 0.9, 0.65, 0.4] - - private func barHeight(_ index: Int) -> CGFloat { - let base: CGFloat = 4 - guard session.phase == .listening else { return base } - let level = max(0, min(1, recognizer.micLevel)) - let weighted = level * Self.barWeights[index] - return base + CGFloat(weighted) * 16 + VoiceLevelBars(level: recognizer.micLevel, active: session.phase == .listening, tint: accent, + reduceMotion: reduceMotion) } private func stopBreathing() { diff --git a/src/macos/GenesisKit/Sources/GenesisKit/Flow/FlowWidget.swift b/src/macos/GenesisKit/Sources/GenesisKit/Flow/FlowWidget.swift index e4fac9bf28..05c1ea810b 100644 --- a/src/macos/GenesisKit/Sources/GenesisKit/Flow/FlowWidget.swift +++ b/src/macos/GenesisKit/Sources/GenesisKit/Flow/FlowWidget.swift @@ -65,9 +65,14 @@ public struct FlowWidget: View { } windowLinks if !flow.history.isEmpty { recentDictation } - if let error = runtime.lastError ?? flow.lastError ?? focus.lastError { - Text(error).font(.system(size: 11)).foregroundStyle(Color.genError) - .fixedSize(horizontal: false, vertical: true) + if !runtime.legacyHosts.isEmpty { + // Not an error: this host leaves dictation and Focus to an older Genesis while it runs. + let notice = runtime.legacyNotice() + NoticePill(text: notice.text, detail: notice.detail, more: notice.found, tone: .warning, + fixes: notice.fixes) + .accessibilityIdentifier("flow-widget-legacy") + } else if let error = runtime.lastError ?? flow.lastError ?? focus.lastError { + NoticePill(text: error, tone: .error) } }.padding(18) } @@ -123,6 +128,7 @@ public struct FlowWidget: View { } private var dictationStatus: String { + if !runtime.legacyHosts.isEmpty { return "In the older Genesis while it runs" } if !ready { return "Waiting for Flow" } if !flow.labEnabled || !flow.config.enabled { return "Dictation is off in Settings" } switch flow.phase { diff --git a/src/macos/GenesisKit/Sources/GenesisKit/Flow/Runtime/FlowFocusAudio.swift b/src/macos/GenesisKit/Sources/GenesisKit/Flow/Runtime/FlowFocusAudio.swift index 7eaac86d71..dc78b4a850 100644 --- a/src/macos/GenesisKit/Sources/GenesisKit/Flow/Runtime/FlowFocusAudio.swift +++ b/src/macos/GenesisKit/Sources/GenesisKit/Flow/Runtime/FlowFocusAudio.swift @@ -3,27 +3,44 @@ import Foundation /// Admission to the elected runtime's microphone. Attach a gated recorder before allowing it /// to open input, then release after the recorder has closed input and exited. +/// +/// A standalone lease has no owner behind it: the host steps back for an older Genesis, so no current host owns audio +/// to coordinate with. Attach and release then only check the recorder, and the recording goes ahead. @MainActor public final class FlowFocusAudioLease { private weak var runtime: FlowFocusRuntime? private let token: UUID + let standalone: Bool private var released = false init(runtime: FlowFocusRuntime, token: UUID) { self.runtime = runtime self.token = token + standalone = false + } + + init(standaloneFor runtime: FlowFocusRuntime) { + self.runtime = runtime + token = UUID() + standalone = true } public func attachRecorder(pid: Int32) async throws { - guard !released, let runtime, let identity = FlowAudioProcess.read(pid: pid) else { + guard !released, let identity = FlowAudioProcess.read(pid: pid) else { throw FlowFocusMailbox.Failure.unavailable("The recorder is no longer available.") } + if standalone { return } + guard let runtime else { throw FlowFocusMailbox.Failure.unavailable("The recorder is no longer available.") } let value = FlowAudioAttachment(token: token, recorder: identity) _ = try await runtime.send(action: "audio.attach", payload: JSONEncoder().encode(value)) } public func release() async throws { guard !released else { return } + if standalone { + released = true + return + } guard let runtime else { throw FlowFocusMailbox.Failure.unavailable("The audio owner is no longer available.") } diff --git a/src/macos/GenesisKit/Sources/GenesisKit/Flow/Runtime/FlowFocusPeers.swift b/src/macos/GenesisKit/Sources/GenesisKit/Flow/Runtime/FlowFocusPeers.swift new file mode 100644 index 0000000000..313d57effd --- /dev/null +++ b/src/macos/GenesisKit/Sources/GenesisKit/Flow/Runtime/FlowFocusPeers.swift @@ -0,0 +1,73 @@ +import AppKit +import Darwin +import Foundation + +/// A running Genesis.app, as the runtime sees it when it decides whether that copy speaks the Flow and Focus protocol. +/// Genesis links this package too, so a current copy joins the same election; a copy built before the protocol runs its +/// own dictation shortcut, capture, Focus timer and Do Not Disturb. +public struct FlowFocusPeerApp: Equatable, Sendable { + public static let genesisBundleID = "dev.foltyn.genesis" + + public let pid: Int32 + public let bundlePath: String? + public let version: String? + /// LaunchServices' launch date. Nil for an app started another way (a script, a relaunch through exec). + public let launchDate: Date? + /// The kernel's start time of the process, which every process has. + public let startedAt: Date? + + public init(pid: Int32, bundlePath: String? = nil, version: String? = nil, launchDate: Date? = nil, startedAt: Date? = nil) { + self.pid = pid + self.bundlePath = bundlePath + self.version = version + self.launchDate = launchDate + self.startedAt = startedAt + } + + /// Every running Genesis.app. + @MainActor static func running() -> [FlowFocusPeerApp] { + NSRunningApplication.runningApplications(withBundleIdentifier: genesisBundleID) + .filter { !$0.isTerminated } + .map { app in + let bundle = app.bundleURL.flatMap(Bundle.init(url:)) + let short = bundle?.object(forInfoDictionaryKey: "CFBundleShortVersionString") as? String + let build = (bundle?.object(forInfoDictionaryKey: "CFBundleVersion") as? String).map { "(\($0))" } + let version = [short, build].compactMap { $0 }.joined(separator: " ") + return FlowFocusPeerApp(pid: app.processIdentifier, bundlePath: app.bundleURL?.path, + version: version.isEmpty ? nil : version, launchDate: app.launchDate, + startedAt: processStart(app.processIdentifier)) + } + } + + static func processStart(_ pid: Int32) -> Date? { + FlowAudioProcess.read(pid: pid).map { + Date(timeIntervalSince1970: Double($0.seconds) + Double($0.microseconds) / 1_000_000) + } + } + + /// The process's short name from the kernel ("Genesis", "bun"), for diagnostics. + static func processName(_ pid: Int32) -> String? { + var buffer = [CChar](repeating: 0, count: 256) + guard proc_name(pid, &buffer, UInt32(buffer.count)) > 0 else { return nil } + return String(cString: buffer) + } +} + +/// What the runtime tells a person about a state it is in or a request it refused: the message, a dim line, what was +/// found (behind "Details"), and the fixes. Drawn by GenesisKit's `NoticePill`. +public struct FlowFocusNotice { + public let text: String + public let detail: String? + public let found: String + public let fixes: [NoticeFix] +} + +enum FlowFocusDiagnostics { + static func stamp(_ date: Date?) -> String { + guard let date else { return "unknown" } + let formatter = DateFormatter() + formatter.locale = Locale(identifier: "en_US_POSIX") + formatter.dateFormat = "yyyy-MM-dd HH:mm:ss" + return formatter.string(from: date) + } +} diff --git a/src/macos/GenesisKit/Sources/GenesisKit/Flow/Runtime/FlowFocusRuntime.swift b/src/macos/GenesisKit/Sources/GenesisKit/Flow/Runtime/FlowFocusRuntime.swift index 297d9acb7e..53c8d1e9de 100644 --- a/src/macos/GenesisKit/Sources/GenesisKit/Flow/Runtime/FlowFocusRuntime.swift +++ b/src/macos/GenesisKit/Sources/GenesisKit/Flow/Runtime/FlowFocusRuntime.swift @@ -21,6 +21,21 @@ public final class FlowFocusRuntime: ObservableObject { @Published public private(set) var role: Role = .stopped @Published public private(set) var lastError: String? + /// The older Genesis copies this host steps back for. They run their own dictation shortcut, capture, Focus timer + /// and Do Not Disturb, so this host owns none of those while they run. An explicit recording still works + /// (`acquireExternalAudio`), and the host takes its part back when they quit. + @Published public private(set) var legacyHosts: [FlowFocusPeerApp] = [] + /// Why the last `acquireExternalAudio` was refused, with what was found and the fixes. The Voice Notes module + /// shows it for the error text it matches (its store keeps only the text). + public private(set) static var lastAudioRefusal: FlowFocusNotice? + /// The running Genesis.app copies to check before this host owns anything. Only the shared data root is the one + /// an older Genesis also uses, so a runtime on an explicit root sees none. Tests replace it. + var peerApps: @MainActor () -> [FlowFocusPeerApp] + /// Asks a peer app to quit (the legacy notice's fix). Tests replace it. + var terminatePeer: @MainActor (Int32) -> Bool = { pid in + NSRunningApplication(processIdentifier: pid)?.terminate() ?? false + } + private var legacyExits: [DispatchSourceProcess] = [] public let flow: FlowSession public let focus: FocusController public let configuration: FlowFocusConfiguration @@ -66,7 +81,9 @@ public final class FlowFocusRuntime: ObservableObject { public init(dataRoot: URL? = nil, hostID: String? = nil, liveServices: Bool = true, presentsWindows: Bool = true, sharedModels: Bool = false) { let root = dataRoot ?? Self.defaultDataRoot - let usesSharedModels = sharedModels && root.standardizedFileURL == Self.defaultDataRoot.standardizedFileURL + let sharedRoot = root.standardizedFileURL == Self.defaultDataRoot.standardizedFileURL + let usesSharedModels = sharedModels && sharedRoot + peerApps = sharedRoot ? { FlowFocusPeerApp.running() } : { [] } self.dataRoot = root directory = root.appendingPathComponent("feature-runtime", isDirectory: true) self.hostID = hostID ?? Bundle.main.bundleIdentifier ?? "dev.genesis.feature-host" @@ -102,7 +119,7 @@ public final class FlowFocusRuntime: ObservableObject { do { try registerParticipant() installDiscovery() - try rejectLegacyOwner() + let legacy = legacyPeers() for _ in 0 ..< 20 { guard isCurrentStart() else { return } if Task.isCancelled { @@ -111,17 +128,33 @@ public final class FlowFocusRuntime: ObservableObject { return } if !settingsOnly, let acquired = try FlowFocusLease.acquire(directory: directory, hostID: hostID) { + guard legacy.isEmpty else { + // The lock was free, so no current host owns Flow and Focus, and an older Genesis runs them + // itself. Owning them here would run its shortcut, capture, timer and DND twice: this host + // steps back from them, and from nothing else. + acquired.release() + stepBack(for: legacy) + return + } + clearLegacy() lease = acquired try becomeOwner(acquired) return } + // A live owner already runs every service, so joining it as a client duplicates nothing, whatever + // else runs. if let owner = try? FlowFocusLease.readOwner(directory: directory) { + clearLegacy() try becomeClient(owner) return } try await Task.sleep(nanoseconds: 100_000_000) } guard isCurrentStart() else { return } + guard legacy.isEmpty else { + stepBack(for: legacy) + return + } throw FlowFocusMailbox.Failure.unavailable("The Flow and Focus owner is still starting. Try again shortly.") } catch is CancellationError { // The host withdrew the start while it waited for another owner to finish starting. That is not a @@ -200,6 +233,7 @@ public final class FlowFocusRuntime: ObservableObject { lease = nil if let applicationObserver { NSWorkspace.shared.notificationCenter.removeObserver(applicationObserver) } applicationObserver = nil + clearLegacy() unregisterParticipant() } @@ -209,6 +243,12 @@ public final class FlowFocusRuntime: ObservableObject { } public func acquireExternalAudio() async throws -> FlowFocusAudioLease { + // While this host steps back for an older Genesis, no current host owns audio to admit a recorder, and the + // older copy captures without admissions anyway: an explicit recording goes ahead on its own lease. + if mailbox == nil, !legacyHosts.isEmpty { + PerfLog.mark("flow-focus.audio standalone legacy=\(legacyHosts.map { String($0.pid) }.joined(separator: ","))") + return FlowFocusAudioLease(standaloneFor: self) + } guard let holder = FlowAudioProcess.read(pid: ProcessInfo.processInfo.processIdentifier) else { throw FlowFocusMailbox.Failure.unavailable("The recording host could not be identified.") } @@ -219,6 +259,7 @@ public final class FlowFocusRuntime: ObservableObject { _ = try await send(action: "audio.acquire", payload: JSONEncoder().encode(request)) return FlowFocusAudioLease(runtime: self, token: token) } catch { + Self.lastAudioRefusal = audioRefusal(error) do { _ = try await send(action: "audio.release", payload: JSONEncoder().encode(token)) } catch { FlowFocusLog.flow.warning("Unacknowledged audio admission cleanup: \(error.localizedDescription)") } throw error @@ -605,31 +646,169 @@ public final class FlowFocusRuntime: ObservableObject { } } - private var participantURL: URL { directory.appendingPathComponent("participant-\(participant.pid).json") } + private func participantURL(_ pid: Int32) -> URL { directory.appendingPathComponent("participant-\(pid).json") } private func registerParticipant() throws { try FileManager.default.createDirectory(at: directory, withIntermediateDirectories: true, attributes: [.posixPermissions: 0o700]) - try FlowFocusLease.writePrivate(try JSONEncoder().encode(participant), to: participantURL) + try FlowFocusLease.writePrivate(try JSONEncoder().encode(participant), to: participantURL(participant.pid)) } private func unregisterParticipant() { - guard let data = try? Data(contentsOf: participantURL), + guard let data = try? Data(contentsOf: participantURL(participant.pid)), let current = try? JSONDecoder().decode(Participant.self, from: data), current.nonce == participant.nonce else { return } - do { try FileManager.default.removeItem(at: participantURL) } + do { try FileManager.default.removeItem(at: participantURL(participant.pid)) } catch { FlowFocusLog.focus.warning("participant cleanup failed: \(error.localizedDescription)") } } - private func rejectLegacyOwner() throws { - guard dataRoot.standardizedFileURL == Self.defaultDataRoot.standardizedFileURL else { return } - for app in NSRunningApplication.runningApplications(withBundleIdentifier: "dev.foltyn.genesis") - where app.processIdentifier != participant.pid { - let record = directory.appendingPathComponent("participant-\(app.processIdentifier).json") - if let data = try? Data(contentsOf: record), - let peer = try? JSONDecoder().decode(Participant.self, from: data), - peer.version == 1, peer.launchDate != nil, peer.launchDate == app.launchDate { continue } - throw FlowFocusMailbox.Failure.unavailable( - "An older Genesis is running its own Flow and Focus services. Update or quit that copy before enabling this host.") + /// The running Genesis copies that do not speak this protocol. + private func legacyPeers() -> [FlowFocusPeerApp] { + peerApps().filter { $0.pid != participant.pid && !isParticipant($0) } + } + + /// A peer speaks this protocol when its participant record was written by the process running now: the launch + /// dates match, or the record is not older than the process. A record left by an earlier process with the same + /// pid predates it. LaunchServices gives no launch date to an app started another way, so the date alone called a + /// current Genesis "older" and refused this host (2026-10-23). + private func isParticipant(_ peer: FlowFocusPeerApp) -> Bool { + let record = participantURL(peer.pid) + guard let data = try? Data(contentsOf: record), + let found = try? JSONDecoder().decode(Participant.self, from: data), + found.version == 1, found.pid == peer.pid else { return false } + if let launched = found.launchDate, launched == peer.launchDate { return true } + guard let started = peer.startedAt, + let written = (try? FileManager.default.attributesOfItem(atPath: record.path))?[.modificationDate] as? Date + else { return false } + // One second of slack for a file system that stores whole seconds. + return written >= started.addingTimeInterval(-1) + } + + private func stepBack(for hosts: [FlowFocusPeerApp]) { + let message = Self.legacyMessage(hosts) + legacyHosts = hosts + role = .unavailable(message) + // The runtime's own error only (a command sent meanwhile names the reason; the next start clears it). Not the + // Flow and Focus models' errors: nothing clears those when the host takes its part back. + lastError = message + watchLegacyExits(hosts) + PerfLog.mark("flow-focus.legacy pids=\(hosts.map { String($0.pid) }.joined(separator: ","))") + } + + static func legacyMessage(_ hosts: [FlowFocusPeerApp]) -> String { + let pids = hosts.map { String($0.pid) }.joined(separator: ", ") + return "An older Genesis (\(hosts.count == 1 ? "pid" : "pids") \(pids)) runs its own dictation shortcut, Focus " + + "timer and Do Not Disturb, so this host leaves them to it. Voice Notes still record." + } + + /// When an older copy quits, this host takes its part back: it starts again, as owner or client. + private func watchLegacyExits(_ hosts: [FlowFocusPeerApp]) { + legacyExits.forEach { $0.cancel() } + legacyExits = hosts.map { host in + let source = DispatchSource.makeProcessSource(identifier: host.pid, eventMask: .exit, queue: .main) + source.setEventHandler { [weak self] in + MainActor.assumeIsolated { + guard let self, case .unavailable = self.role, self.legacyHosts.contains(host) else { return } + Task { await self.start() } + } + } + source.resume() + return source + } + } + + private func clearLegacy() { + legacyExits.forEach { $0.cancel() } + legacyExits = [] + if !legacyHosts.isEmpty { legacyHosts = [] } + } + + /// The notice the Flow module shows while this host steps back: each older copy, how it was found, and how to quit it. + public func legacyNotice() -> FlowFocusNotice { + let found = legacyHosts.map { host in + [ + "Genesis, pid \(host.pid)", + "App: \(host.bundlePath.map(PathDisplay.tilde) ?? "unknown")", + "Version: \(host.version ?? "unknown")", + "Started: \(FlowFocusDiagnostics.stamp(host.startedAt ?? host.launchDate))", + "Why it counts as older: since it started, it wrote no Flow and Focus participant record " + + "(\(PathDisplay.tilde(participantURL(host.pid).path))).", + ].joined(separator: "\n") + }.joined(separator: "\n\n") + var fixes: [NoticeFix] = [] + for host in legacyHosts { + let confirm = NoticeFix.Confirmation( + "Quit Genesis (pid \(host.pid))?", + consequence: "Genesis may ask to save open work. Dictation and Focus then start in this host.", + confirmLabel: "Quit Genesis") + fixes.append(.button("Quit the older Genesis (pid \(host.pid))", symbol: "power", confirm: confirm) { [weak self] in + let asked = self?.terminatePeer(host.pid) ?? false + PerfLog.mark("flow-focus.legacy quit pid=\(host.pid) asked=\(asked)") + }) + fixes.append(.command("Or quit it from a terminal", "kill \(host.pid)")) + } + return FlowFocusNotice( + text: "An older Genesis runs dictation and Focus", + detail: "This host leaves the dictation shortcut, the Focus timer and Do Not Disturb to that copy until it " + + "quits or is updated. Voice Notes record as usual.", + found: found, fixes: fixes) + } + + /// What was found when the owner refused audio, and the fixes that apply: the owner, a dictation in progress, the + /// process that holds the microphone. + private func audioRefusal(_ error: Error) -> FlowFocusNotice { + var lines = ["This host: \(hostID), pid \(participant.pid), \(Self.describe(role))"] + var fixes: [NoticeFix] = [] + if let owner = try? FlowFocusLease.readOwner(directory: directory) { + lines.append("Flow and Focus owner: \(owner.hostID), pid \(owner.pid), since \(FlowFocusDiagnostics.stamp(owner.startedAt))") + } else { + lines.append("Flow and Focus owner: none (\(PathDisplay.tilde(directory.appendingPathComponent("owner.json").path)))") + } + if [.listening, .transcribing, .injecting].contains(flow.phase) { + lines.append("Dictation: \(flow.phase.rawValue)") + fixes.append(.button("Cancel the dictation in progress", symbol: "xmark.circle", + confirm: .init("Cancel the dictation in progress?", + consequence: "The words heard so far are not inserted.", + confirmLabel: "Cancel dictation")) { [weak self] in self?.flow.cancelTurn() }) + } + let admissionURL = directory.appendingPathComponent("audio-admission.json") + if let data = try? Data(contentsOf: admissionURL), + let admission = try? JSONDecoder().decode(FlowAudioAdmission.self, from: data) { + let process = admission.recorder ?? admission.holder + let name = FlowFocusPeerApp.processName(process.pid) ?? "a process" + lines.append("Microphone admitted to: \(name), pid \(process.pid), " + + (admission.recorder == nil ? "recorder not started yet" : "recording")) + if admission.recorder != nil, process.isAlive, process.pid != participant.pid { + fixes.append(.button("Stop the other recording (\(name), pid \(process.pid))", symbol: "stop.circle", + confirm: .init("Stop \(name) (pid \(process.pid))?", + consequence: "That recording ends now; what it captured may not be saved.", + confirmLabel: "Stop it")) { + // The notice can outlive the recorder; the kernel start time catches a reused pid. + guard process.isAlive else { + PerfLog.mark("flow-focus.audio stop pid=\(process.pid) skipped=exited") + return + } + let sent = kill(process.pid, SIGTERM) == 0 + PerfLog.mark("flow-focus.audio stop pid=\(process.pid) sent=\(sent)") + }) + fixes.append(.command("Or stop it from a terminal", "kill \(process.pid)")) + } + } + if case .unavailable = role { + fixes.append(.button("Connect to Flow and Focus again", symbol: "arrow.clockwise") { [weak self] in + guard let self, case .unavailable = self.role else { return } + Task { await self.start() } + }) + } + return FlowFocusNotice(text: error.localizedDescription, detail: nil, found: lines.joined(separator: "\n"), fixes: fixes) + } + + private static func describe(_ role: Role) -> String { + switch role { + case .stopped: return "stopped" + case .starting: return "starting" + case .owner: return "owner" + case .client(let owner): return "client of \(owner)" + case .unavailable(let why): return "unavailable: \(why)" } } diff --git a/src/macos/GenesisKit/Sources/GenesisKit/Flow/Support/AttentionPulse.swift b/src/macos/GenesisKit/Sources/GenesisKit/Flow/Support/AttentionPulse.swift index daf5f60405..969e9ff481 100644 --- a/src/macos/GenesisKit/Sources/GenesisKit/Flow/Support/AttentionPulse.swift +++ b/src/macos/GenesisKit/Sources/GenesisKit/Flow/Support/AttentionPulse.swift @@ -28,8 +28,15 @@ struct AttentionPulseStyle: Equatable { /// Number of blinks. Never zero, and a zero period cannot divide by zero. var cycles: Int { max(1, Int((duration / max(period, 0.05)).rounded())) } - /// Three quick flashes of border and fill over about 1.4 s. Reads as "look here, now". - static func flash(color: Color, cornerRadius: CGFloat) -> Self { + /// Three quick flashes of the border over about 1.4 s. Reads as "look here, now" without covering the content, + /// so it is the default (Martin, 2026-10-23: a settings card under a tinted overlay read as covered, not pointed at). + static func borderFlash(color: Color, cornerRadius: CGFloat) -> Self { + Self(color: color, cornerRadius: cornerRadius, lineWidth: 2, + duration: 1.44, period: 0.48, fadeIn: 0.12, fadeOut: 0.2) + } + + /// The same three flashes with a tinted fill over the whole view, for a target too small for a border to read. + static func filledFlash(color: Color, cornerRadius: CGFloat) -> Self { Self(color: color, cornerRadius: cornerRadius, fillOpacity: 0.30, duration: 1.44, period: 0.48, fadeIn: 0.12, fadeOut: 0.2) } @@ -42,9 +49,14 @@ struct AttentionPulseStyle: Equatable { /// Plays one `AttentionPulseStyle` over the view each time `trigger` changes. A new trigger /// while a pulse runs restarts it rather than stacking a second one on top. +/// +/// `since` is when the flash began. A view that appears while it runs plays the rest of it: a list built after the +/// trigger changed (the widget's hover list, opened by the alert that flashed the session) never saw the change, so +/// only the rail bubble flashed (Martin, 2026-10-23). struct AttentionPulse: ViewModifier { let trigger: Trigger let style: AttentionPulseStyle + var since: Date? @State private var isOn = false @State private var running: Task? @@ -70,6 +82,13 @@ struct AttentionPulse: ViewModifier { } } + /// What is left of a flash that began at `since`, or nil when none runs. + static func remaining(since: Date?, now: Date = Date(), duration: TimeInterval) -> TimeInterval? { + guard let since else { return nil } + let left = duration - now.timeIntervalSince(since) + return left > 0.05 && left <= duration ? left : nil + } + func body(content: Content) -> some View { content // Drawn over everything and never hit-testable: a pulse must not eat the next click. @@ -85,14 +104,20 @@ struct AttentionPulse: ViewModifier { .accessibilityHidden(true) } .onChange(of: trigger) { _, _ in play() } + .onAppear { + guard running == nil, let left = Self.remaining(since: since, duration: style.duration) else { return } + var rest = style + rest.duration = left + play(rest) + } // Turning Reduce Motion on mid-pulse replaces the blinking with the static border at once. .onChange(of: reduceMotion) { _, _ in if running != nil { play() } } .onDisappear { running?.cancel() } } - private func play() { + private func play(_ style: AttentionPulseStyle? = nil) { running?.cancel() - let steps = Self.steps(style: style, reduceMotion: reduceMotion) + let steps = Self.steps(style: style ?? self.style, reduceMotion: reduceMotion) running = Task { @MainActor in for step in steps { if let duration = step.animation { @@ -109,8 +134,12 @@ struct AttentionPulse: ViewModifier { } extension View { - /// Blinks this view for attention each time `trigger` changes. Pass a counter you bump. - func attentionPulse(trigger: Trigger, style: AttentionPulseStyle) -> some View { - modifier(AttentionPulse(trigger: trigger, style: style)) + /// Blinks this view for attention each time `trigger` changes. Pass a counter you bump. The default is a + /// border-only flash in the accent colour; `.filledFlash` and `.slowBorder` are the other presets. + func attentionPulse( + trigger: Trigger, style: AttentionPulseStyle = .borderFlash(color: .accentColor, cornerRadius: 12), + since: Date? = nil + ) -> some View { + modifier(AttentionPulse(trigger: trigger, style: style, since: since)) } } diff --git a/src/macos/GenesisKit/Sources/GenesisKit/Recording/RecordingControlsView.swift b/src/macos/GenesisKit/Sources/GenesisKit/Recording/RecordingControlsView.swift new file mode 100644 index 0000000000..84f5534759 --- /dev/null +++ b/src/macos/GenesisKit/Sources/GenesisKit/Recording/RecordingControlsView.swift @@ -0,0 +1,411 @@ +import AppKit +import SwiftUI + +/// What the recording controls show. A value, so a host (or a test) can draw any moment of a recording. +public struct RecordingControlsState: Equatable { + public var phase: RecordingPhase + public var elapsedMs: Double + public var limitMs: Double + public var target: RecordingTarget? + /// `user`, `limit` or `recorder` while paused. + public var pauseReason: String? + public var notice: String? + /// Why the start takes long: the recorder compiles after an update, or no first frame came yet. + public var preparing: String? + /// The host's name for stopping and keeping the movie: "Stop and paste", "Stop and save". + public var useTitle: String + public var useSymbol: String + /// Discard and Restart ask once inside the controls first (`RecordingSettings.confirmsDiscard`). + public var confirmsDiscard: Bool + + public init(phase: RecordingPhase, elapsedMs: Double = 0, limitMs: Double = 600_000, target: RecordingTarget? = nil, + pauseReason: String? = nil, notice: String? = nil, preparing: String? = nil, useTitle: String, + useSymbol: String = "stop.fill", confirmsDiscard: Bool = true) { + self.phase = phase + self.elapsedMs = elapsedMs + self.limitMs = limitMs + self.target = target + self.pauseReason = pauseReason + self.notice = notice + self.preparing = preparing + self.useTitle = useTitle + self.useSymbol = useSymbol + self.confirmsDiscard = confirmsDiscard + } +} + +/// What the recording controls do. Each closure is one of Martin's actions: pause or resume, switch what is recorded, +/// restart, stop and use the movie, stop and discard it. +public struct RecordingControlsActions { + public var pause: () -> Void + public var resume: () -> Void + public var switchTarget: () -> Void + public var restart: () -> Void + public var use: () -> Void + public var discard: () -> Void + public var dismissNotice: () -> Void + /// The person dragged the controls somewhere else. + public var moved: () -> Void + + public init(pause: @escaping () -> Void, resume: @escaping () -> Void, switchTarget: @escaping () -> Void, + restart: @escaping () -> Void, use: @escaping () -> Void, discard: @escaping () -> Void, + dismissNotice: @escaping () -> Void = {}, moved: @escaping () -> Void = {}) { + self.pause = pause + self.resume = resume + self.switchTarget = switchTarget + self.restart = restart + self.use = use + self.discard = discard + self.dismissNotice = dismissNotice + self.moved = moved + } +} + +/// A recording's elapsed time as `m:ss`, or `h:mm:ss` from an hour on. +public enum RecordingClock { + public static func format(_ milliseconds: Double) -> String { + let total = max(0, Int(milliseconds / 1000)) + let hours = total / 3600 + let minutes = (total % 3600) / 60 + let seconds = total % 60 + return hours > 0 + ? String(format: "%d:%02d:%02d", hours, minutes, seconds) + : String(format: "%d:%02d", minutes, seconds) + } +} + +/// The controls' state as something to observe, so a host sets the view up once and only the values change (its +/// own state, such as an open discard question, survives the clock's ticks). +@MainActor +final class RecordingControlsModel: ObservableObject { + @Published var state: RecordingControlsState + + init(state: RecordingControlsState) { + self.state = state + } +} + +struct RecordingControlsHost: View { + @ObservedObject var model: RecordingControlsModel + let actions: RecordingControlsActions + + var body: some View { + RecordingControlsView(state: model.state, actions: actions) + } +} + +/// The icon of a running app by pid, looked up once per pid: the controls redraw every second while recording. +@MainActor +enum RecordingAppIcon { + private static var cache: [Int32: NSImage] = [:] + + static func image(pid: Int32?) -> NSImage? { + guard let pid, pid > 0 else { return nil } + if let cached = cache[pid] { return cached } + guard let icon = NSRunningApplication(processIdentifier: pid)?.icon else { return nil } + cache[pid] = icon + return icon + } +} + +/// The small floating controls of a running recording: the recorded time with a red dot, what is recorded (a click +/// switches it), pause or resume, restart, the host's "stop and use" action and stop and discard. The leading part +/// (grip, dot and time) moves the window. Generic: the host names the use action and decides what it does. +public struct RecordingControlsView: View { + private let state: RecordingControlsState + private let actions: RecordingControlsActions + @State private var confirming: Confirmation? + @GenReduceMotion private var reduceMotion + + private enum Confirmation: Equatable { case discard, restart } + + public init(state: RecordingControlsState, actions: RecordingControlsActions) { + self.state = state + self.actions = actions + } + + private var busy: Bool { state.phase == .finishing || !state.phase.isActive } + private var paused: Bool { state.phase == .paused } + /// Under a minute left before the length limit, the clock says so. + private var remainingMs: Double? { + let left = state.limitMs - state.elapsedMs + return state.phase == .recording && left < 60_000 ? max(0, left) : nil + } + + public var body: some View { + VStack(alignment: .leading, spacing: 6) { + HStack(spacing: 8) { + status + if let confirming { + confirmation(confirming) + .transition(.opacity) + } else { + targetChip + controls + .transition(.opacity) + } + } + if state.phase == .starting, let preparing = state.preparing { + preparingRow(preparing) + } + if let notice = state.notice { + noticeRow(notice) + } + } + .padding(.leading, 6) + .padding(.trailing, 8) + .padding(.vertical, 6) + .frame(minHeight: 44) + .modifier(NativeGlassShapeSurface(shape: RoundedRectangle(cornerRadius: 14, style: .continuous))) + .environment(\.colorScheme, .dark) + .animation(reduceMotion ? nil : .easeOut(duration: 0.16), value: confirming) + .onChange(of: state.phase) { _, phase in + if !phase.isActive || phase == .finishing { confirming = nil } + } + .accessibilityElement(children: .contain) + .accessibilityLabel("Recording controls") + } + + // MARK: Status (the drag area) + + private var status: some View { + HStack(spacing: 7) { + RecordingGrip() + indicator + VStack(alignment: .leading, spacing: 0) { + Text(verbatim: RecordingClock.format(state.elapsedMs)) + .font(.system(size: 13, weight: .semibold).monospacedDigit()) + .foregroundStyle(paused ? .secondary : .primary) + if let caption = statusCaption { + Text(caption).font(.system(size: 9.5, weight: .medium)) + .foregroundStyle(captionColor) + .lineLimit(1) + } + } + .frame(minWidth: 48, alignment: .leading) + } + .padding(.leading, 4) + .padding(.trailing, 2) + .frame(maxHeight: .infinity) + .contentShape(Rectangle()) + .allowsHitTesting(false) + .background(RecordingWindowDragArea(dragged: actions.moved)) + .accessibilityElement(children: .combine) + .accessibilityLabel(accessibilityStatus) + .frameProbe("recording.status") + } + + @ViewBuilder private var indicator: some View { + switch state.phase { + case .starting, .finishing: + ProgressView().controlSize(.mini).frame(width: 12, height: 12) + case .paused: + Image(systemName: "pause.circle.fill").font(.system(size: 12)).foregroundStyle(.orange) + case .recording: + Circle().fill(Color.red).frame(width: 9, height: 9) + .shadow(color: .red.opacity(0.7), radius: 3) + default: + Circle().fill(Color.secondary).frame(width: 9, height: 9) + } + } + + private var statusCaption: String? { + switch state.phase { + case .starting: return "Starting" + case .finishing: return "Saving" + case .paused: + return state.pauseReason == "limit" ? "Limit reached" : "Paused" + case .recording: + return remainingMs.map { "\(RecordingClock.format($0)) left" } + default: + return nil + } + } + + private var captionColor: Color { + state.phase == .paused || remainingMs != nil ? .orange : .secondary + } + + private var accessibilityStatus: String { + let time = RecordingClock.format(state.elapsedMs) + switch state.phase { + case .starting: return state.preparing.map { "Starting the recording. \($0)" } ?? "Starting the recording" + case .recording: return "Recording, \(time)" + case .paused: return "Paused at \(time)" + case .finishing: return "Saving the recording" + default: return "Recording ended" + } + } + + // MARK: Target + + private var targetChip: some View { + Button(action: actions.switchTarget) { + HStack(spacing: 6) { + if let icon = RecordingAppIcon.image(pid: state.target?.pid) { + Image(nsImage: icon).resizable().interpolation(.high).frame(width: 16, height: 16) + } else { + Image(systemName: state.target?.symbol ?? "rectangle.dashed").font(.system(size: 11)) + .frame(width: 16, height: 16) + } + Text(state.target?.title ?? "Choose what to record") + .font(.system(size: 12, weight: .medium)) + .lineLimit(1).truncationMode(.middle) + .frame(maxWidth: 180, alignment: .leading) + Image(systemName: "chevron.down").font(.system(size: 8, weight: .bold)).foregroundStyle(.secondary) + } + .padding(.horizontal, 8) + .padding(.vertical, 5) + .background(.white.opacity(0.07), in: RoundedRectangle(cornerRadius: 8, style: .continuous)) + .contentShape(RoundedRectangle(cornerRadius: 8, style: .continuous)) + } + .buttonStyle(.genHoverPlain()) + .disabled(busy) + .instantTooltip("Record something else") + .accessibilityLabel("Recording \(state.target?.title ?? "nothing"). Switch what is recorded") + .accessibilityIdentifier("recording.switch") + .frameProbe("recording.switch") + } + + // MARK: Controls + + private var controls: some View { + HStack(spacing: 6) { + Group { + if paused { + IconButton(systemName: "play.fill", tooltip: state.pauseReason == "limit" ? "The length limit is reached" : "Resume", + size: 13, action: actions.resume) + .disabled(state.pauseReason == "limit") + .accessibilityIdentifier("recording.resume") + .frameProbe("recording.resume") + } else { + IconButton(systemName: "pause.fill", tooltip: "Pause", size: 13, action: actions.pause) + .disabled(busy) + .accessibilityIdentifier("recording.pause") + .frameProbe("recording.pause") + } + } + IconButton(systemName: "arrow.counterclockwise", tooltip: "Restart: drop what was recorded and start again", + size: 13) { + ask(.restart, then: actions.restart) + } + .disabled(busy) + .accessibilityIdentifier("recording.restart") + .frameProbe("recording.restart") + Rectangle().fill(.white.opacity(0.14)).frame(width: 1, height: 20).padding(.horizontal, 2) + Button(action: actions.use) { + HStack(spacing: 5) { + Image(systemName: state.useSymbol).font(.system(size: 10, weight: .bold)) + Text(state.useTitle).font(.system(size: 12, weight: .semibold)).lineLimit(1) + } + .foregroundStyle(.white) + .padding(.horizontal, 10) + .padding(.vertical, 6) + .background(Color.red.opacity(busy ? 0.4 : 0.88), in: Capsule()) + .contentShape(Capsule()) + } + .buttonStyle(.genHoverPlain(brighten: 0.12)) + .disabled(busy) + .fixedSize() + .accessibilityLabel(state.useTitle) + .accessibilityIdentifier("recording.use") + .frameProbe("recording.use") + IconButton(systemName: "trash", tooltip: "Stop and discard", size: 13) { + ask(.discard, then: actions.discard) + } + .disabled(busy) + .accessibilityIdentifier("recording.discard") + .frameProbe("recording.discard") + } + // While the movie is saved nothing can be pressed, and it shows. + .opacity(busy ? 0.45 : 1) + } + + private func ask(_ confirmation: Confirmation, then action: @escaping () -> Void) { + // Nothing recorded yet means nothing to lose. + if state.confirmsDiscard && state.elapsedMs >= 1000 { + confirming = confirmation + } else { + action() + } + } + + private func confirmation(_ kind: Confirmation) -> some View { + HStack(spacing: 8) { + Text(kind == .discard + ? "Discard \(RecordingClock.format(state.elapsedMs)) of recording?" + : "Restart and drop \(RecordingClock.format(state.elapsedMs))?") + .font(.system(size: 12, weight: .medium)) + .lineLimit(1) + .fixedSize() + Button { + confirming = nil + kind == .discard ? actions.discard() : actions.restart() + } label: { + Text(kind == .discard ? "Discard" : "Restart") + .font(.system(size: 12, weight: .semibold)).foregroundStyle(.white) + .padding(.horizontal, 10).padding(.vertical, 5) + .background(Color.red.opacity(0.88), in: Capsule()) + .contentShape(Capsule()) + } + .buttonStyle(.genHoverPlain(brighten: 0.12)) + .fixedSize() + .accessibilityIdentifier("recording.confirm") + .frameProbe("recording.confirm") + Button { + confirming = nil + } label: { + Text("Keep").font(.system(size: 12, weight: .medium)) + .padding(.horizontal, 10).padding(.vertical, 5) + .background(.white.opacity(0.1), in: Capsule()) + .contentShape(Capsule()) + } + .buttonStyle(.genHoverPlain()) + .fixedSize() + .accessibilityIdentifier("recording.keep") + .frameProbe("recording.keep") + } + } + + private func preparingRow(_ text: String) -> some View { + HStack(alignment: .top, spacing: 6) { + Image(systemName: "hourglass").font(.system(size: 10)).foregroundStyle(.secondary) + Text(text).font(.system(size: 11)).foregroundStyle(.secondary) + .lineLimit(3).fixedSize(horizontal: false, vertical: true) + .frame(maxWidth: 420, alignment: .leading) + Spacer(minLength: 0) + } + .padding(.leading, 6) + .accessibilityElement(children: .combine) + .accessibilityIdentifier("recording.preparing") + } + + private func noticeRow(_ notice: String) -> some View { + HStack(alignment: .top, spacing: 6) { + Image(systemName: "exclamationmark.triangle.fill").font(.system(size: 10)).foregroundStyle(.orange) + Text(notice).font(.system(size: 11)).foregroundStyle(.secondary) + .lineLimit(3).fixedSize(horizontal: false, vertical: true) + .textSelection(.enabled) + .frame(maxWidth: 420, alignment: .leading) + Spacer(minLength: 0) + IconButton(systemName: "xmark", tooltip: "Dismiss", size: 9, action: actions.dismissNotice) + } + .padding(.leading, 6) + } +} + +/// Six dots: the grip that says the controls can be dragged. +private struct RecordingGrip: View { + var body: some View { + Grid(horizontalSpacing: 2.5, verticalSpacing: 2.5) { + ForEach(0..<3, id: \.self) { _ in + GridRow { + Circle().frame(width: 2.5, height: 2.5) + Circle().frame(width: 2.5, height: 2.5) + } + } + } + .foregroundStyle(.white.opacity(0.35)) + .accessibilityHidden(true) + } +} diff --git a/src/macos/GenesisKit/Sources/GenesisKit/Recording/RecordingModels.swift b/src/macos/GenesisKit/Sources/GenesisKit/Recording/RecordingModels.swift new file mode 100644 index 0000000000..0e79c9ef8d --- /dev/null +++ b/src/macos/GenesisKit/Sources/GenesisKit/Recording/RecordingModels.swift @@ -0,0 +1,222 @@ +import AppKit +import Foundation + +/// What a screen recording records: exact windows, every visible window of an app, or a whole display. The JSON is +/// the `--target` of `tools control capture session` and the `target` of its `switch` command. +public enum RecordingTarget: Hashable, Sendable { + case window(ids: [UInt32], app: String?, title: String?, pid: Int32?) + case app(app: String, name: String?) + case display(index: Int, name: String?) + + /// The line a person reads: the window's title (or its app), the app's name, the display's name. + public var title: String { + switch self { + case let .window(ids, app, title, _): + if let title, !title.isEmpty { return title } + return app ?? "Window \(ids.map(String.init).joined(separator: ", "))" + case let .app(app, name): + return name ?? app + case let .display(index, name): + return name ?? "Display \(index + 1)" + } + } + + /// The smaller line under the title: whose window it is, or what kind of target it is. + public var subtitle: String { + switch self { + case let .window(_, app, title, _): + return (title?.isEmpty == false ? app : nil) ?? "Window" + case .app: + return "Every window of the app" + case .display: + return "The whole display" + } + } + + public var symbol: String { + switch self { + case .window: return "macwindow" + case .app: return "square.stack" + case .display: return "display" + } + } + + /// The pid of the recorded app when the target names one, so a picker can show its icon. + public var pid: Int32? { + switch self { + case let .window(_, _, _, pid): return pid + case let .app(app, _): return Int32(app) + case .display: return nil + } + } + + var wire: [String: Any] { + switch self { + case let .window(ids, app, title, pid): + var object: [String: Any] = ["kind": "window", "windowIds": ids.map(Int.init)] + if let app { object["app"] = app } + if let title { object["title"] = title } + if let pid { object["pid"] = Int(pid) } + return object + case let .app(app, name): + var object: [String: Any] = ["kind": "app", "app": app] + if let name { object["name"] = name } + return object + case let .display(index, name): + var object: [String: Any] = ["kind": "display", "screenIndex": index] + if let name { object["name"] = name } + return object + } + } + + /// The target as one line of JSON. + public var json: String { + guard let data = try? JSONSerialization.data(withJSONObject: wire, options: [.sortedKeys]) else { return "{}" } + return String(decoding: data, as: UTF8.self) + } +} + +extension RecordingTarget: Decodable { + private enum Keys: String, CodingKey { case kind, windowIds, app, title, pid, name, screenIndex } + + public init(from decoder: Decoder) throws { + let container = try decoder.container(keyedBy: Keys.self) + switch try container.decode(String.self, forKey: .kind) { + case "window": + self = .window( + ids: try container.decode([UInt32].self, forKey: .windowIds), + app: try container.decodeIfPresent(String.self, forKey: .app), + title: try container.decodeIfPresent(String.self, forKey: .title), + pid: try container.decodeIfPresent(Int32.self, forKey: .pid)) + case "app": + self = .app(app: try container.decode(String.self, forKey: .app), + name: try container.decodeIfPresent(String.self, forKey: .name)) + case "display": + self = .display(index: try container.decode(Int.self, forKey: .screenIndex), + name: try container.decodeIfPresent(String.self, forKey: .name)) + case let other: + throw DecodingError.dataCorruptedError(forKey: .kind, in: container, + debugDescription: "unknown recording target kind \(other)") + } + } +} + +/// One on-screen window, as `tools control capture targets --json` lists it (CG points, top-left origin). +public struct RecordingWindow: Decodable, Identifiable, Hashable, Sendable { + public struct Bounds: Decodable, Hashable, Sendable { + public let x: Double + public let y: Double + public let width: Double + public let height: Double + + public init(x: Double, y: Double, width: Double, height: Double) { + self.x = x + self.y = y + self.width = width + self.height = height + } + } + + public let id: UInt32 + public let pid: Int32 + public let app: String + public let title: String + public let bounds: Bounds + + public init(id: UInt32, pid: Int32, app: String, title: String, bounds: Bounds) { + self.id = id + self.pid = pid + self.app = app + self.title = title + self.bounds = bounds + } + + public var target: RecordingTarget { .window(ids: [id], app: app, title: title, pid: pid) } +} + +public struct RecordingApp: Decodable, Identifiable, Hashable, Sendable { + public let pid: Int32 + public let name: String + public let windowCount: Int + public var id: Int32 { pid } + + public init(pid: Int32, name: String, windowCount: Int) { + self.pid = pid + self.name = name + self.windowCount = windowCount + } + + public var target: RecordingTarget { .app(app: String(pid), name: name) } +} + +public struct RecordingDisplay: Decodable, Identifiable, Hashable, Sendable { + public let index: Int + public let name: String + public let isPrimary: Bool + public let width: Double + public let height: Double + public var id: Int { index } + + public init(index: Int, name: String, isPrimary: Bool, width: Double, height: Double) { + self.index = index + self.name = name + self.isPrimary = isPrimary + self.width = width + self.height = height + } + + public var target: RecordingTarget { .display(index: index, name: name) } +} + +/// Everything a picker offers. +public struct RecordingTargets: Decodable, Hashable, Sendable { + public let windows: [RecordingWindow] + public let apps: [RecordingApp] + public let displays: [RecordingDisplay] + public let frontmostPid: Int32? + + public init(windows: [RecordingWindow], apps: [RecordingApp], displays: [RecordingDisplay], frontmostPid: Int32? = nil) { + self.windows = windows + self.apps = apps + self.displays = displays + self.frontmostPid = frontmostPid + } + + /// The same lists without the windows and apps of `pids` (a host's own process). + public func excluding(pids: Set) -> RecordingTargets { + RecordingTargets(windows: windows.filter { !pids.contains($0.pid) }, apps: apps.filter { !pids.contains($0.pid) }, + displays: displays, frontmostPid: frontmostPid) + } + + /// The front window of the app the person was in (`preferredPid`, else the frontmost app), else the main display. + public func defaultTarget(preferredPid: Int32? = nil) -> RecordingTarget? { + for pid in [preferredPid, frontmostPid].compactMap({ $0 }) { + if let window = windows.first(where: { $0.pid == pid }) { return window.target } + } + if let window = windows.first { return window.target } + return (displays.first(where: \.isPrimary) ?? displays.first)?.target + } +} + +/// Where a recording is: arming (no frame yet), recording, paused, saving, or over. +public enum RecordingPhase: String, Sendable { + case idle, starting, recording, paused, finishing, done, discarded, failed + + public var isActive: Bool { [.starting, .recording, .paused, .finishing].contains(self) } +} + +/// One JSON line of `tools control capture session --json`. +struct RecordingSessionLine: Decodable { + let event: String + let state: String? + let reason: String? + let target: RecordingTarget? + let elapsedMs: Double? + let limitMs: Double? + let segments: Int? + let path: String? + let durationMs: Double? + let message: String? + let code: String? + let fatal: Bool? +} diff --git a/src/macos/GenesisKit/Sources/GenesisKit/Recording/RecordingPanels.swift b/src/macos/GenesisKit/Sources/GenesisKit/Recording/RecordingPanels.swift new file mode 100644 index 0000000000..ff4220b809 --- /dev/null +++ b/src/macos/GenesisKit/Sources/GenesisKit/Recording/RecordingPanels.swift @@ -0,0 +1,366 @@ +import AppKit +import SwiftUI + +/// A hosting view that acts on the first click: a panel that never becomes key gets every click as a first click, +/// and AppKit drops it before SwiftUI sees it unless the view accepts it. +final class FirstMouseHostingView: NSHostingView { + override func acceptsFirstMouse(for event: NSEvent?) -> Bool { true } +} + +/// The recording controls' window: floats over every app and Space, never takes the keyboard or activates the app. +final class RecordingControlsWindow: NSPanel { + override var canBecomeKey: Bool { false } + override var canBecomeMain: Bool { false } +} + +/// The target picker's window: takes the keyboard (arrows, Return, Esc, the filter) without activating the app, so +/// the app the person was in stays active and gets the keyboard back when the picker closes. +final class RecordingPickerWindow: NSPanel { + override var canBecomeKey: Bool { true } + override var canBecomeMain: Bool { false } +} + +/// Where the controls go, in Cocoa screen coordinates. Pure, so the placement rules are tested without a screen. +enum RecordingPanelLayout { + struct Screen { + let frame: CGRect + let visible: CGRect + } + + static let margin: CGFloat = 16 + static let gap: CGFloat = 12 + + /// `targetCG` is what is recorded in CoreGraphics points (top-left origin of the primary display), nil for a + /// display; `displayIndex` names the recorded display in `screens` order (NSScreen order). + static func frame(size: CGSize, placement: RecordingPanelPlacement, targetCG: CGRect?, displayIndex: Int?, + screens: [Screen]) -> CGRect { + guard let primary = screens.first else { return CGRect(origin: .zero, size: size) } + let target = targetCG.map { cocoa($0, primaryHeight: primary.frame.height) } + let screen = displayIndex.flatMap { screens.indices.contains($0) ? screens[$0] : nil } + ?? target.flatMap { rect in screens.first { $0.frame.contains(CGPoint(x: rect.midX, y: rect.midY)) } } + ?? primary + let visible = screen.visible + var origin: CGPoint + switch placement { + case .topLeft: + origin = CGPoint(x: visible.minX + margin, y: visible.maxY - margin - size.height) + case .topRight: + origin = CGPoint(x: visible.maxX - margin - size.width, y: visible.maxY - margin - size.height) + case .bottomLeft: + origin = CGPoint(x: visible.minX + margin, y: visible.minY + margin) + case .bottomRight: + origin = CGPoint(x: visible.maxX - margin - size.width, y: visible.minY + margin) + case .nearTarget: + if let target, displayIndex == nil { + let below = target.minY - gap - size.height + // Under the window when the screen has room, else inside its bottom edge. + let y = below >= visible.minY + margin ? below : target.minY + gap + origin = CGPoint(x: target.midX - size.width / 2, y: y) + } else { + origin = CGPoint(x: visible.midX - size.width / 2, y: visible.minY + margin * 2) + } + } + origin.x = min(max(origin.x, visible.minX + margin), visible.maxX - margin - size.width) + origin.y = min(max(origin.y, visible.minY + margin), visible.maxY - margin - size.height) + return CGRect(origin: origin, size: size).integral + } + + static func cocoa(_ rect: CGRect, primaryHeight: CGFloat) -> CGRect { + CGRect(x: rect.minX, y: primaryHeight - rect.maxY, width: rect.width, height: rect.height) + } + + static var liveScreens: [Screen] { + NSScreen.screens.map { Screen(frame: $0.frame, visible: $0.visibleFrame) } + } +} + +/// A drag area that moves its window. It tracks the drag itself from each event's location: `performDrag(with:)` +/// moved a non-activating panel only on the first drag of a show (ux ledger, 2026-10-14). +struct RecordingWindowDragArea: NSViewRepresentable { + var dragged: () -> Void + + func makeNSView(context: Context) -> RecordingWindowDragView { + let view = RecordingWindowDragView() + view.dragged = dragged + return view + } + + func updateNSView(_ view: RecordingWindowDragView, context: Context) { view.dragged = dragged } +} + +final class RecordingWindowDragView: NSView { + var dragged: (() -> Void)? + private var last: CGPoint? + + override init(frame: NSRect) { + super.init(frame: frame) + setAccessibilityElement(true) + setAccessibilityRole(.handle) + setAccessibilityIdentifier("recording.drag") + setAccessibilityLabel("Move the recording controls") + } + + required init?(coder: NSCoder) { fatalError("init(coder:) is not used") } + override var mouseDownCanMoveWindow: Bool { false } + override func acceptsFirstMouse(for event: NSEvent?) -> Bool { true } + override func resetCursorRects() { addCursorRect(bounds, cursor: last == nil ? .openHand : .closedHand) } + + /// Global CoreGraphics points (y down), from the event itself, so a drag delivered in the background moves it too. + private func location(_ event: NSEvent) -> CGPoint { + event.cgEvent?.location ?? CGPoint(x: NSEvent.mouseLocation.x, y: -NSEvent.mouseLocation.y) + } + + override func mouseDown(with event: NSEvent) { + last = location(event) + window?.invalidateCursorRects(for: self) + } + + override func mouseDragged(with event: NSEvent) { + guard let window, let last else { return } + let point = location(event) + window.setFrameOrigin(CGPoint(x: window.frame.minX + point.x - last.x, y: window.frame.minY - (point.y - last.y))) + self.last = point + dragged?() + } + + override func mouseUp(with event: NSEvent) { + last = nil + window?.invalidateCursorRects(for: self) + } +} + +/// The floating window that holds the recording controls. Out of every movie by window id (the session passes it to +/// the recorder); `sharingType = .none` also keeps it out of legacy window captures. +@MainActor +final class RecordingControlsPanel { + let window: RecordingControlsWindow + private let hosting: FirstMouseHostingView + /// True once the person dragged the controls: a target switch then leaves them where they are. + private(set) var moved = false + + init() { + window = RecordingControlsWindow(contentRect: CGRect(x: 0, y: 0, width: 520, height: 52), + styleMask: [.borderless, .nonactivatingPanel], backing: .buffered, defer: false) + window.title = "Recording controls" + window.identifier = NSUserInterfaceItemIdentifier("recording.controls") + RecordingPanelChrome.apply(to: window) + hosting = FirstMouseHostingView(rootView: AnyView(EmptyView())) + // The panel sizes itself from its content's measured size (below); a hosting view that also reports an + // intrinsic size makes AppKit lay the whole tree out on every update. + hosting.sizingOptions = [] + window.contentView = hosting + } + + /// The CG window id the recorder leaves out. + var windowID: Int { window.windowNumber } + + /// `view` is drawn at its ideal size; the window follows that size, keeping its centre and bottom edge. + func setContent(_ view: some View) { + hosting.rootView = AnyView(view.fixedSize().onGeometryChange(for: CGSize.self, of: { $0.size }) { [weak self] size in + // Never resize the window inside the layout pass that measured it. + DispatchQueue.main.async { self?.resize(to: size) } + }) + } + + func noteMoved() { moved = true } + + private func resize(to size: CGSize) { + let size = CGSize(width: size.width.rounded(.up), height: size.height.rounded(.up)) + guard size.width > 0, size.height > 0, size != window.frame.size else { return } + let frame = window.frame + window.setFrame(CGRect(x: (frame.midX - size.width / 2).rounded(), y: frame.minY, width: size.width, + height: size.height), display: true) + } + + func place(_ placement: RecordingPanelPlacement, targetCG: CGRect?, displayIndex: Int?) { + guard !moved else { return } + hosting.layoutSubtreeIfNeeded() + let size = hosting.fittingSize + window.setFrame(RecordingPanelLayout.frame(size: size, placement: placement, targetCG: targetCG, + displayIndex: displayIndex, screens: RecordingPanelLayout.liveScreens), + display: true) + } + + func show() { window.orderFrontRegardless() } + + func close() { + window.orderOut(nil) + window.close() + } +} + +/// A steady dashed border around what is about to be recorded, shown while the recorder starts. The recorder draws its +/// own pulsing border once it runs; this one covers the gap (a compile after an update can take a minute), so the +/// person sees at once what they chose. One clear panel per display draws every rect on it: a panel per window made +/// 50 windows for a long window list and stalled the main thread 1.6 s. Out of every movie by window id, like the +/// controls, and on screen only while a segment starts. +@MainActor +final class RecordingTargetOutline { + private struct Overlay { + let screen: CGRect + let panel: NSPanel + let view: RecordingOutlineView + } + + private var overlays: [Overlay] + + var windowIDs: [Int] { overlays.map(\.panel.windowNumber) } + /// The rects each display's panel draws, in that panel's own points (tests). + var drawn: [[CGRect]] { overlays.map(\.view.rects) } + + /// Makes the panels up front, so their window ids exist before the recorder is told what to leave out. + init(screens: [CGRect] = NSScreen.screens.map(\.frame)) { + overlays = screens.map { frame in + let view = RecordingOutlineView(frame: CGRect(origin: .zero, size: frame.size)) + return Overlay(screen: frame, panel: Self.makePanel(frame: frame, view: view), view: view) + } + } + + /// `rects` in Cocoa screen points; a display with none of them hides its panel. + func show(_ rects: [CGRect]) { + for overlay in overlays { + let local = rects.map { $0.insetBy(dx: -3, dy: -3) } + .filter { $0.intersects(overlay.screen) } + .map { $0.offsetBy(dx: -overlay.screen.minX, dy: -overlay.screen.minY) } + overlay.view.rects = local + if local.isEmpty { + overlay.panel.orderOut(nil) + } else { + overlay.panel.orderFrontRegardless() + } + } + } + + func hide() { overlays.forEach { $0.panel.orderOut(nil) } } + + func close() { + overlays.forEach { $0.panel.orderOut(nil); $0.panel.close() } + overlays = [] + } + + private static func makePanel(frame: CGRect, view: NSView) -> NSPanel { + let panel = NSPanel(contentRect: frame, styleMask: [.borderless, .nonactivatingPanel], backing: .buffered, + defer: false) + panel.title = "Recording target" + panel.level = .screenSaver + panel.isOpaque = false + panel.backgroundColor = .clear + panel.hasShadow = false + panel.ignoresMouseEvents = true + panel.hidesOnDeactivate = false + panel.isReleasedWhenClosed = false + panel.collectionBehavior = [.canJoinAllSpaces, .fullScreenAuxiliary, .ignoresCycle] + panel.sharingType = .none + panel.contentView = view + panel.setFrame(frame, display: false) + return panel + } +} + +/// Draws its rects as dashed red rounded borders, once per change, no animation. +final class RecordingOutlineView: NSView { + var rects: [CGRect] = [] { + didSet { if rects != oldValue { needsDisplay = true } } + } + + override init(frame: NSRect) { + super.init(frame: frame) + setAccessibilityElement(false) + } + + @available(*, unavailable) + required init?(coder: NSCoder) { fatalError("init(coder:) is not used") } + + override func draw(_ dirtyRect: NSRect) { + NSColor.systemRed.withAlphaComponent(0.9).setStroke() + for rect in rects { + let path = NSBezierPath(roundedRect: rect.insetBy(dx: 1.5, dy: 1.5), xRadius: 9, yRadius: 9) + path.lineWidth = 3 + path.setLineDash([9, 6], count: 2, phase: 0) + path.stroke() + } + } +} + +/// What both recording windows share: floating over every app, Space and full-screen app, above the widget's panels, +/// clear behind their own rounded content, out of legacy window captures. +@MainActor +enum RecordingPanelChrome { + static func apply(to window: NSPanel) { + window.isFloatingPanel = true + // After isFloatingPanel, whose setter resets the level: above the widget's edge panels. + window.level = NSWindow.Level(rawValue: NSWindow.Level.statusBar.rawValue + 1) + window.collectionBehavior = [.canJoinAllSpaces, .fullScreenAuxiliary, .ignoresCycle] + window.hidesOnDeactivate = false + window.isReleasedWhenClosed = false + window.backgroundColor = .clear + window.isOpaque = false + window.hasShadow = true + window.sharingType = .none + } +} + +/// The floating window that holds the target picker. It closes, as a cancel, when it loses the keyboard (a click in +/// another app or window). +@MainActor +final class RecordingPickerPanel: NSObject, NSWindowDelegate { + let window: RecordingPickerWindow + private var onDismiss: (() -> Void)? + + override init() { + window = RecordingPickerWindow(contentRect: CGRect(x: 0, y: 0, width: 400, height: 420), + styleMask: [.borderless, .nonactivatingPanel], backing: .buffered, defer: false) + super.init() + window.title = "Choose what to record" + window.identifier = NSUserInterfaceItemIdentifier("recording.picker") + RecordingPanelChrome.apply(to: window) + window.delegate = self + } + + var isVisible: Bool { window.isVisible } + + /// Shows the picker with its top centred under `anchor` (Cocoa screen point), inside that screen, and gives it + /// the keyboard without activating the app. + func present(_ view: some View, near anchor: CGPoint, onDismiss: @escaping () -> Void) { + self.onDismiss = onDismiss + // A fresh hosting view each time: a reused one that had been on screen measured 0x0 after a new root view, so + // the second picker of a session opened invisible and the recorder waited for a choice nobody could see. + let hosting = FirstMouseHostingView(rootView: AnyView(view.fixedSize().onGeometryChange(for: CGSize.self, of: { $0.size }) { [weak self] size in + DispatchQueue.main.async { self?.resize(to: size) } + })) + hosting.sizingOptions = [] + window.contentView = hosting + hosting.layoutSubtreeIfNeeded() + let size = hosting.fittingSize + let screen = NSScreen.screens.first { $0.frame.contains(anchor) } ?? NSScreen.screens.first + let visible = screen?.visibleFrame ?? CGRect(origin: .zero, size: size) + var origin = CGPoint(x: anchor.x - size.width / 2, y: anchor.y - 8 - size.height) + if origin.y < visible.minY + 8 { origin.y = anchor.y + 8 } + origin.x = min(max(origin.x, visible.minX + 8), visible.maxX - 8 - size.width) + origin.y = min(max(origin.y, visible.minY + 8), visible.maxY - 8 - size.height) + window.setFrame(CGRect(origin: origin, size: size).integral, display: true) + window.makeKeyAndOrderFront(nil) + } + + /// Follows the content (a tab that lists more or fewer rows), keeping the top edge. + private func resize(to size: CGSize) { + let size = CGSize(width: size.width.rounded(.up), height: size.height.rounded(.up)) + guard window.isVisible, size.width > 0, size.height > 0, size != window.frame.size else { return } + let frame = window.frame + window.setFrame(CGRect(x: frame.minX, y: frame.maxY - size.height, width: size.width, height: size.height), + display: true) + } + + func dismiss() { + onDismiss = nil + window.orderOut(nil) + } + + func windowDidResignKey(_ notification: Notification) { + guard window.isVisible, let onDismiss else { return } + PerfLog.mark("recording.picker cancelled: lost the keyboard to \(NSApp.keyWindow?.title ?? "another app")") + self.onDismiss = nil + window.orderOut(nil) + onDismiss() + } +} diff --git a/src/macos/GenesisKit/Sources/GenesisKit/Recording/RecordingSessionController.swift b/src/macos/GenesisKit/Sources/GenesisKit/Recording/RecordingSessionController.swift new file mode 100644 index 0000000000..76be55b432 --- /dev/null +++ b/src/macos/GenesisKit/Sources/GenesisKit/Recording/RecordingSessionController.swift @@ -0,0 +1,190 @@ +import Foundation + +/// The command side of a running `tools control capture session`: one JSON line per command, EOF to stop. +@MainActor +public protocol RecordingCommandChannel: AnyObject { + func send(_ line: String) throws + func finishInput() + /// SIGTERM: the session stops and keeps what it recorded. + func stop() +} + +extension ToolsLineStream: RecordingCommandChannel { + public func send(_ line: String) throws { try sendInput(line) } +} + +/// Drives one recording session through `tools control capture session --json --stop-on-stdin` and publishes what it +/// reports. It knows nothing about who records or what happens to the movie: the host gets `onDone` with the movie, +/// `onDiscarded`, or `onFailed`. +@MainActor +public final class RecordingSessionController: ObservableObject { + @Published public private(set) var phase: RecordingPhase = .idle + /// Recorded time: what is in the movie so far, without arming or paused time. + @Published public private(set) var elapsedMs: Double = 0 + @Published public private(set) var limitMs: Double = 600_000 + @Published public private(set) var target: RecordingTarget? + /// Why the session paused: `user`, `limit` (the length limit) or `recorder` (it stopped by itself). + @Published public private(set) var pauseReason: String? + @Published public private(set) var segments = 0 + /// The last problem the session reported that did not end it; cleared by the next state change. + @Published public var notice: String? + /// Why `starting` takes long (the recorder compiles after an update, or no first frame came yet); cleared once + /// the session leaves `starting`. + @Published public private(set) var preparing: String? + + public var onDone: ((URL, Double) -> Void)? + public var onDiscarded: (() -> Void)? + /// The message and its code (`screen-recording-denied`), when the session could not record at all. + public var onFailed: ((String, String?) -> Void)? + + private let bridge: ToolsBridge? + private let openChannel: (([String], @escaping @MainActor ([String]) -> Void, + @escaping @MainActor (Int32, String) -> Void) throws -> RecordingCommandChannel)? + private var channel: RecordingCommandChannel? + private var ended = false + + public init(bridge: ToolsBridge) { + self.bridge = bridge + openChannel = nil + } + + /// Tests: `open` gets the CLI arguments and the line and exit callbacks, and returns the command channel. + init(open: @escaping ([String], @escaping @MainActor ([String]) -> Void, + @escaping @MainActor (Int32, String) -> Void) throws -> RecordingCommandChannel) { + bridge = nil + openChannel = open + } + + /// The arguments after `tools control`. + static func arguments(target: RecordingTarget, excludeWindowIDs: [Int], frameRate: Int, showsBorder: Bool) -> [String] { + var args = ["capture", "session", "--json", "--stop-on-stdin", "--target", target.json, "--fps", String(frameRate)] + if !excludeWindowIDs.isEmpty { + args += ["--exclude-window-ids", excludeWindowIDs.map(String.init).joined(separator: ",")] + } + if !showsBorder { args.append("--no-indicator") } + return args + } + + /// Starts recording `target`. `excludeWindowIDs` are windows the movie never shows (the controls panel). + public func start(target: RecordingTarget, excludeWindowIDs: [Int], frameRate: Int, showsBorder: Bool) throws { + guard channel == nil, !ended else { return } + let args = Self.arguments(target: target, excludeWindowIDs: excludeWindowIDs, frameRate: frameRate, + showsBorder: showsBorder) + self.target = target + phase = .starting + let lines: @MainActor ([String]) -> Void = { [weak self] in self?.receive($0) } + let exit: @MainActor (Int32, String) -> Void = { [weak self] status, stderr in self?.exited(status: status, stderr: stderr) } + do { + if let openChannel { + channel = try openChannel(args, lines, exit) + } else if let bridge { + channel = try ToolsLineStream( + bridge: bridge, subcommand: "control", args: args, + onLines: lines, onExit: { exit($0.status, $0.stderr) }) + } + PerfLog.mark("recording.session started target=\(target.json)") + } catch { + phase = .failed + ended = true + throw error + } + } + + public func pause() { send(["cmd": "pause"]) } + public func resume() { send(["cmd": "resume"]) } + public func restart() { send(["cmd": "restart"]) } + /// Stops and keeps the movie: `onDone` follows once it is joined. + public func stop() { send(["cmd": "stop"]) } + public func discard() { send(["cmd": "discard"]) } + + public func switchTo(_ target: RecordingTarget) { + var wire = target.wire + // The channel takes one line of at most 512 bytes. A window's title and a display's name are labels only, so + // a long one is left out rather than the switch refused. + let line = try? JSONSerialization.data(withJSONObject: ["cmd": "switch", "target": wire], options: [.sortedKeys]) + if (line?.count ?? 0) >= Self.commandLimit { + wire["title"] = nil + wire["name"] = nil + } + send(["cmd": "switch", "target": wire]) + } + + /// `ToolsLineStream.sendInput`'s limit; the newline takes one byte of it. + static let commandLimit = 512 + + /// Ends the child at once (an app that quits): it stops and keeps what it recorded on SIGTERM. + public func terminate() { + channel?.stop() + } + + private func send(_ command: [String: Any]) { + guard let channel, !ended else { return } + do { + let data = try JSONSerialization.data(withJSONObject: command, options: [.sortedKeys]) + try channel.send(String(decoding: data, as: UTF8.self) + "\n") + } catch { + notice = "The recorder did not take the command: \(error.localizedDescription)" + PerfLog.mark("recording.session send failed \(error.localizedDescription)") + } + } + + func receive(_ lines: [String]) { + for line in lines { + guard let event = try? JSONDecoder().decode(RecordingSessionLine.self, from: Data(line.utf8)) else { + PerfLog.mark("recording.session unreadable line \(line.prefix(200))") + continue + } + apply(event) + } + } + + private func apply(_ event: RecordingSessionLine) { + switch event.event { + case "state": + if let state = event.state, let next = RecordingPhase(rawValue: state) { phase = next } + if let target = event.target { self.target = target } + if let elapsed = event.elapsedMs { elapsedMs = elapsed } + if let limit = event.limitMs { limitMs = limit } + if let segments = event.segments { self.segments = segments } + pauseReason = phase == .paused ? event.reason : nil + if phase != .paused { notice = nil } + if phase != .starting { preparing = nil } + case "preparing": + preparing = event.message + case "elapsed": + if let elapsed = event.elapsedMs { elapsedMs = elapsed } + case "segment": + segments += 1 + case "done": + guard let path = event.path else { return } + phase = .done + ended = true + onDone?(URL(fileURLWithPath: path), event.durationMs ?? elapsedMs) + case "discarded": + phase = .discarded + ended = true + onDiscarded?() + case "error": + let message = event.message ?? "The recorder failed." + if event.fatal == true { + phase = .failed + ended = true + onFailed?(message, event.code) + } else { + notice = message + } + default: + break + } + } + + private func exited(status: Int32, stderr: String) { + channel = nil + PerfLog.mark("recording.session exited status=\(status) phase=\(phase.rawValue)") + guard !ended else { return } + ended = true + phase = .failed + let tail = stderr.trimmingCharacters(in: .whitespacesAndNewlines).suffix(600) + onFailed?(tail.isEmpty ? "The recorder stopped unexpectedly (exit \(status))." : String(tail), nil) + } +} diff --git a/src/macos/GenesisKit/Sources/GenesisKit/Recording/RecordingSettings.swift b/src/macos/GenesisKit/Sources/GenesisKit/Recording/RecordingSettings.swift new file mode 100644 index 0000000000..5b8b443691 --- /dev/null +++ b/src/macos/GenesisKit/Sources/GenesisKit/Recording/RecordingSettings.swift @@ -0,0 +1,115 @@ +import Foundation +import SwiftUI + +/// Where the recording controls float when a recording starts or switches target. A drag moves them for the rest of +/// that recording. +public enum RecordingPanelPlacement: String, CaseIterable, Identifiable, Sendable { + /// Under the recorded window (inside its bottom edge when the screen has no room below), or at the bottom of the + /// recorded display. + case nearTarget + case topLeft + case topRight + case bottomLeft + case bottomRight + + public var id: String { rawValue } + + public var title: String { + switch self { + case .nearTarget: return "Near what is recorded" + case .topLeft: return "Top left corner" + case .topRight: return "Top right corner" + case .bottomLeft: return "Bottom left corner" + case .bottomRight: return "Bottom right corner" + } + } +} + +/// The screen recorder's preferences, shared by every host (the widget composer, the Capture shelf). Stored in the +/// app's defaults under `recording.*`; every value applies to the next recording. +@MainActor +public final class RecordingSettings: ObservableObject { + public static let shared = RecordingSettings() + + public static let frameRates = [5, 10, 15, 24, 30] + + public enum Key { + public static let placement = "recording.panelPlacement" + public static let frameRate = "recording.frameRate" + public static let showsBorder = "recording.showsBorder" + public static let confirmsDiscard = "recording.confirmsDiscard" + } + + /// Default: near what is recorded, so the controls sit where the person is looking. + @Published public var placement: RecordingPanelPlacement { + didSet { defaults.set(placement.rawValue, forKey: Key.placement) } + } + + /// Frames per second while something moves (the recorder drops to 2 while the screen is still). Default 15. + @Published public var frameRate: Int { + didSet { defaults.set(frameRate, forKey: Key.frameRate) } + } + + /// The pulsing red border around what is recorded. It never appears in the movie. Default on. + @Published public var showsBorder: Bool { + didSet { defaults.set(showsBorder, forKey: Key.showsBorder) } + } + + /// "Stop and discard" and "Restart" ask once inside the controls before they drop what was recorded. Default on. + @Published public var confirmsDiscard: Bool { + didSet { defaults.set(confirmsDiscard, forKey: Key.confirmsDiscard) } + } + + private let defaults: UserDefaults + + public init(defaults: UserDefaults = .standard) { + self.defaults = defaults + placement = defaults.string(forKey: Key.placement).flatMap(RecordingPanelPlacement.init(rawValue:)) ?? .nearTarget + let storedRate = defaults.integer(forKey: Key.frameRate) + frameRate = Self.frameRates.contains(storedRate) ? storedRate : 15 + showsBorder = defaults.object(forKey: Key.showsBorder) as? Bool ?? true + confirmsDiscard = defaults.object(forKey: Key.confirmsDiscard) as? Bool ?? true + } +} + +/// The settings page for screen recording: placement, frame rate, the border and the discard question. +public struct RecordingSettingsView: View { + @ObservedObject private var settings: RecordingSettings + + public init(settings: RecordingSettings? = nil) { + self.settings = settings ?? .shared + } + + public var body: some View { + VStack(spacing: 18) { + NativeSettingsCard("Controls", subtitle: "The small panel that floats beside a recording.") { + NativeSettingsRow("Placement", detail: "Where the controls appear when a recording starts. Drag them anywhere.") { + Picker("Placement", selection: $settings.placement) { + ForEach(RecordingPanelPlacement.allCases) { placement in + Text(placement.title).tag(placement) + } + }.labelsHidden().frame(width: 220, alignment: .trailing) + .accessibilityIdentifier("recording.placement") + } + Divider() + NativeSettingsToggle( + "Ask before discarding", detail: "Stop and discard, and Restart, ask once inside the controls.", + identifier: "recording.confirmsDiscard", isOn: $settings.confirmsDiscard) + } + NativeSettingsCard("Recording") { + NativeSettingsRow("Frame rate", detail: "While something moves. A still screen records at 2 per second.") { + Picker("Frame rate", selection: $settings.frameRate) { + ForEach(RecordingSettings.frameRates, id: \.self) { rate in + Text("\(rate) fps").tag(rate) + } + }.labelsHidden().frame(width: 120, alignment: .trailing) + .accessibilityIdentifier("recording.frameRate") + } + Divider() + NativeSettingsToggle( + "Red border around the recording", detail: "Marks what is recorded. It never appears in the movie.", + identifier: "recording.showsBorder", isOn: $settings.showsBorder) + } + } + } +} diff --git a/src/macos/GenesisKit/Sources/GenesisKit/Recording/RecordingTargetPicker.swift b/src/macos/GenesisKit/Sources/GenesisKit/Recording/RecordingTargetPicker.swift new file mode 100644 index 0000000000..a3b192417e --- /dev/null +++ b/src/macos/GenesisKit/Sources/GenesisKit/Recording/RecordingTargetPicker.swift @@ -0,0 +1,331 @@ +import AppKit +import SwiftUI + +/// The three kinds of target the picker offers, one tab each. +public enum RecordingPickerTab: String, CaseIterable, Hashable, Sendable { + case windows, apps, displays + + public var title: String { + switch self { + case .windows: return "Windows" + case .apps: return "Apps" + case .displays: return "Displays" + } + } + + static func of(_ target: RecordingTarget?) -> RecordingPickerTab { + switch target { + case .app: return .apps + case .display: return .displays + default: return .windows + } + } +} + +/// One line of the picker: an app header over its windows, or a target that a click records. +struct RecordingPickerRow: Identifiable, Equatable { + enum Kind: Equatable { + case header + case target(RecordingTarget) + } + + let id: String + let kind: Kind + let title: String + let detail: String + let pid: Int32? + let symbol: String + var badge: String? + + var target: RecordingTarget? { + if case let .target(target) = kind { return target } + return nil + } + + /// The rows of one tab: windows grouped under their app in the order the apps' front windows stand, apps, or + /// displays. `query` keeps the windows whose title or app contains it (apps by name). + static func rows(_ targets: RecordingTargets, tab: RecordingPickerTab, query: String) -> [RecordingPickerRow] { + let needle = query.trimmingCharacters(in: .whitespacesAndNewlines) + func matches(_ values: String...) -> Bool { + needle.isEmpty || values.contains { $0.localizedCaseInsensitiveContains(needle) } + } + switch tab { + case .windows: + var order: [Int32] = [] + var byApp: [Int32: [RecordingWindow]] = [:] + for window in targets.windows where matches(window.title, window.app) { + if byApp[window.pid] == nil { order.append(window.pid) } + byApp[window.pid, default: []].append(window) + } + return order.flatMap { pid -> [RecordingPickerRow] in + let windows = byApp[pid] ?? [] + let header = RecordingPickerRow( + id: "app-\(pid)", kind: .header, title: windows.first?.app ?? "App", + detail: Plural.count(windows.count, "window"), pid: pid, symbol: "app") + return [header] + windows.map { window in + RecordingPickerRow( + id: "window-\(window.id)", kind: .target(window.target), + title: window.title.isEmpty ? "Untitled window" : window.title, + detail: "\(Int(window.bounds.width)) × \(Int(window.bounds.height))", + pid: nil, symbol: "macwindow") + } + } + case .apps: + return targets.apps.filter { matches($0.name) }.map { app in + RecordingPickerRow(id: "apps-\(app.pid)", kind: .target(app.target), title: app.name, + detail: Plural.count(app.windowCount, "window"), pid: app.pid, symbol: "app") + } + case .displays: + return targets.displays.filter { matches($0.name) }.map { display in + RecordingPickerRow(id: "display-\(display.index)", kind: .target(display.target), title: display.name, + detail: "\(Int(display.width)) × \(Int(display.height))", pid: nil, symbol: "display", + badge: display.isPrimary ? "Main" : nil) + } + } + } +} + +/// Lets the person pick what to record: one window (grouped by app, with titles), every window of an app, or a whole +/// display. Starts on `initial` (the front window of the app the person was in). ↑ ↓ move, Return records, Esc +/// cancels; a click on a row records it. Values and closures only. +public struct RecordingTargetPicker: View { + public enum Load: Equatable { + case loading + case failed(String) + case loaded(RecordingTargets) + } + + private let load: Load + private let title: String + private let initial: RecordingTarget? + private let onChoose: (RecordingTarget) -> Void + private let onCancel: () -> Void + private let onRetry: () -> Void + private let onFilterChange: (RecordingPickerTab, String) -> Void + @State private var tab: RecordingPickerTab + @State private var query = "" + @State private var highlighted: String? + /// The list took over 4 s: the hint names the known slow cause. + @State private var slowLoad = false + @FocusState private var filterFocused: Bool + + public init(load: Load, title: String = "Choose what to record", initial: RecordingTarget?, + onChoose: @escaping (RecordingTarget) -> Void, onCancel: @escaping () -> Void, + onRetry: @escaping () -> Void = {}, restoring: (tab: RecordingPickerTab, query: String)? = nil, + onFilterChange: @escaping (RecordingPickerTab, String) -> Void = { _, _ in }) { + self.load = load + self.title = title + self.initial = initial + self.onChoose = onChoose + self.onCancel = onCancel + self.onRetry = onRetry + self.onFilterChange = onFilterChange + _tab = State(initialValue: restoring?.tab ?? RecordingPickerTab.of(initial)) + _query = State(initialValue: restoring?.query ?? "") + } + + private var rows: [RecordingPickerRow] { + guard case let .loaded(targets) = load else { return [] } + return RecordingPickerRow.rows(targets, tab: tab, query: query) + } + + private var targetRows: [RecordingPickerRow] { rows.filter { $0.target != nil } } + + public var body: some View { + VStack(alignment: .leading, spacing: 10) { + header + GenSegmentedTabs("What to record", items: RecordingPickerTab.allCases.map { .init($0, $0.title) }, + selection: $tab, minSegmentWidth: 114, identifier: "recording.picker.tabs") + filterField + content + footer + } + .padding(14) + .frame(width: 380) + .modifier(NativeGlassShapeSurface(shape: RoundedRectangle(cornerRadius: 16, style: .continuous))) + .environment(\.colorScheme, .dark) + .onAppear { + highlight(initial: true) + filterFocused = true + } + .onChange(of: load) { _, _ in highlight(initial: true) } + .onChange(of: tab) { _, _ in onFilterChange(tab, query); highlight(initial: false) } + .onChange(of: query) { _, _ in onFilterChange(tab, query); highlight(initial: false) } + .accessibilityElement(children: .contain) + .accessibilityLabel(title) + } + + private var header: some View { + HStack(spacing: 8) { + Image(systemName: "record.circle").font(.system(size: 14, weight: .semibold)).foregroundStyle(.red) + Text(title).font(.system(size: 13, weight: .semibold)) + Spacer(minLength: 0) + IconButton(systemName: "xmark", tooltip: "Cancel (Esc)", size: 10, action: onCancel) + .accessibilityIdentifier("recording.picker.cancel") + } + } + + private var filterField: some View { + HStack(spacing: 6) { + Image(systemName: "magnifyingglass").font(.system(size: 11)).foregroundStyle(.secondary) + TextField("Filter \(tab.title.lowercased())", text: $query) + .textFieldStyle(.plain) + .font(.system(size: 12)) + .focused($filterFocused) + .onSubmit(chooseHighlighted) + .onExitCommand(perform: onCancel) + .onKeyPress(.escape) { + onCancel() + return .handled + } + .onKeyPress(.downArrow) { + move(1) + return .handled + } + .onKeyPress(.upArrow) { + move(-1) + return .handled + } + .accessibilityIdentifier("recording.picker.filter") + } + .padding(.horizontal, 8) + .padding(.vertical, 6) + .background(.white.opacity(0.07), in: RoundedRectangle(cornerRadius: 8, style: .continuous)) + } + + @ViewBuilder private var content: some View { + switch load { + case .loading: + VStack(spacing: 6) { + HStack(spacing: 8) { + ProgressView().controlSize(.small) + Text("Looking for windows and displays…").font(.system(size: 12)).foregroundStyle(.secondary) + } + if slowLoad { + Text("The first time after an update, the recorder is built first. This can take up to a minute.") + .font(.system(size: 11)).foregroundStyle(.tertiary) + .multilineTextAlignment(.center).fixedSize(horizontal: false, vertical: true) + .frame(maxWidth: 300) + } + } + .frame(maxWidth: .infinity, minHeight: 120) + .task { + try? await Task.sleep(for: .seconds(4)) + slowLoad = true + } + case let .failed(message): + VStack(spacing: 8) { + Text(message).font(.system(size: 12)).foregroundStyle(.secondary) + .multilineTextAlignment(.center).fixedSize(horizontal: false, vertical: true) + Button("Try again", action: onRetry).buttonStyle(.genHover()) + } + .frame(maxWidth: .infinity, minHeight: 120) + case .loaded: + if rows.isEmpty { + Text(query.isEmpty ? "Nothing to record here." : "Nothing matches “\(query)”.") + .font(.system(size: 12)).foregroundStyle(.secondary) + .frame(maxWidth: .infinity, minHeight: 80) + } else { + ScrollViewReader { proxy in + FittedScroll(maxHeight: 330) { + LazyVStack(alignment: .leading, spacing: 2) { + ForEach(rows) { row in + rowView(row).id(row.id) + } + } + } + .onChange(of: highlighted) { _, id in + guard let id else { return } + proxy.scrollTo(id) + } + } + } + } + } + + @ViewBuilder private func rowView(_ row: RecordingPickerRow) -> some View { + if let target = row.target { + let isHighlighted = highlighted == row.id + Button { + onChoose(target) + } label: { + HStack(spacing: 8) { + icon(row).frame(width: 18, height: 18) + Text(row.title).font(.system(size: 12, weight: isHighlighted ? .semibold : .regular)) + .lineLimit(1).truncationMode(.middle) + if let badge = row.badge { + Text(badge).font(.system(size: 9, weight: .semibold)) + .padding(.horizontal, 5).padding(.vertical, 1) + .background(.white.opacity(0.12), in: Capsule()) + } + Spacer(minLength: 8) + Text(verbatim: row.detail).font(.system(size: 10.5).monospacedDigit()).foregroundStyle(.secondary) + } + .padding(.leading, row.symbol == "macwindow" ? 22 : 6) + .padding(.trailing, 8) + .frame(height: 28) + .background(isHighlighted ? Color.accentColor.opacity(0.32) : .clear, + in: RoundedRectangle(cornerRadius: 6, style: .continuous)) + .contentShape(Rectangle()) + } + .buttonStyle(.genHoverPlain()) + .onHover { inside in + if inside { highlighted = row.id } + } + .accessibilityLabel("Record \(row.title)") + .accessibilityIdentifier("recording.picker.\(row.id)") + .frameProbe("recording.picker.\(row.id)") + } else { + HStack(spacing: 6) { + icon(row).frame(width: 14, height: 14) + Text(row.title).font(.system(size: 11, weight: .semibold)).foregroundStyle(.secondary) + Text(verbatim: row.detail).font(.system(size: 10)).foregroundStyle(.tertiary) + } + .padding(.horizontal, 6) + .padding(.top, 6) + .padding(.bottom, 2) + .accessibilityAddTraits(.isHeader) + } + } + + @ViewBuilder private func icon(_ row: RecordingPickerRow) -> some View { + if let image = RecordingAppIcon.image(pid: row.pid) { + Image(nsImage: image).resizable().interpolation(.high) + } else { + Image(systemName: row.symbol).font(.system(size: 12)).foregroundStyle(.secondary) + } + } + + private var footer: some View { + HStack(spacing: 4) { + Text("↑ ↓ to choose · Return records · Esc cancels") + Spacer(minLength: 0) + } + .font(.system(size: 10)) + .foregroundStyle(.tertiary) + } + + private func highlight(initial useInitial: Bool) { + let candidates = targetRows + if useInitial, let initial, let row = candidates.first(where: { $0.target == initial }) { + highlighted = row.id + } else if let current = highlighted, candidates.contains(where: { $0.id == current }) { + return + } else { + highlighted = candidates.first?.id + } + } + + private func move(_ step: Int) { + let candidates = targetRows + guard !candidates.isEmpty else { return } + let index = candidates.firstIndex { $0.id == highlighted } ?? -1 + highlighted = candidates[min(max(index + step, 0), candidates.count - 1)].id + } + + private func chooseHighlighted() { + guard let row = targetRows.first(where: { $0.id == highlighted }) ?? targetRows.first, + let target = row.target else { return } + onChoose(target) + } +} diff --git a/src/macos/GenesisKit/Sources/GenesisKit/Recording/ScreenRecorder.swift b/src/macos/GenesisKit/Sources/GenesisKit/Recording/ScreenRecorder.swift new file mode 100644 index 0000000000..5ad9a39799 --- /dev/null +++ b/src/macos/GenesisKit/Sources/GenesisKit/Recording/ScreenRecorder.swift @@ -0,0 +1,391 @@ +import AppKit +import Combine +import SwiftUI + +/// What a host does with a recording. The recorder itself is generic: the widget pastes the movie into a session's +/// draft, the Capture shelf stages it, any other host does what it likes. +public struct ScreenRecordingHost { + /// The name of "stop and keep" in the controls: "Stop and paste", "Stop and save". + public var useTitle: String + public var useSymbol: String + /// Says why the recording asks for Screen Recording, in the permission dialog. + public var permissionReason: String + /// Gets the finished movie. The file is the host's from here on; a throw is reported through `onError`. + public var onUse: (URL) async throws -> Void + public var onDiscard: () -> Void + /// A recording that could not start or failed, or a use that threw. + public var onError: (String) -> Void + /// The recording has started (the picker closed and the controls are up): a host can step aside here. + public var onStart: () -> Void + + public init(useTitle: String, useSymbol: String = "stop.fill", permissionReason: String, + onUse: @escaping (URL) async throws -> Void, onDiscard: @escaping () -> Void = {}, + onError: @escaping (String) -> Void, onStart: @escaping () -> Void = {}) { + self.useTitle = useTitle + self.useSymbol = useSymbol + self.permissionReason = permissionReason + self.onUse = onUse + self.onDiscard = onDiscard + self.onError = onError + self.onStart = onStart + } +} + +/// The whole screen recording flow for one host: the target picker, the Screen Recording grant, the session in +/// `tools control capture session`, the floating controls, switching target (the recording pauses while the picker is +/// open, so the movie never shows it) and the end. One recording runs at a time across every recorder in the process; +/// `begin` while another runs brings that one's controls forward. +@MainActor +public final class ScreenRecorder: ObservableObject { + /// True from the picker's first appearance until the recording ended. + @Published public private(set) var isActive = false + + public let settings: RecordingSettings + private let bridge: ToolsBridge + private let permissionGate: @MainActor (PermissionNeed) async -> Bool + private let loadTargets: @MainActor () async throws -> RecordingTargets + private var host: ScreenRecordingHost? + private var controller: RecordingSessionController? + private var controllerWatch: AnyCancellable? + /// Waits for the recorder to confirm a pause before the Switch picker opens (`switchTarget`). + private var pauseWatch: AnyCancellable? + private var settingsWatch: AnyCancellable? + private var controls: RecordingControlsPanel? + private var controlsModel: RecordingControlsModel? + private var outline: RecordingTargetOutline? + /// The target the outline is drawn around now; nil while it is hidden. + private var outlinedTarget: RecordingTarget? + private let picker = RecordingPickerPanel() + private var targets: RecordingTargets? + private var pickerLoad: RecordingTargetPicker.Load = .loading + private var pickerInitial: RecordingTarget? + /// The picker's tab and filter text, kept here because every `presentPicker` builds a new view; nil until the person + /// changes either, so a list that arrives later does not reset what they typed while it loaded. + private var pickerFilter: (tab: RecordingPickerTab, query: String)? + private var pickerTitle = "Choose what to record" + private var pickerChoose: ((RecordingTarget) -> Void)? + private var pickerCancel: (() -> Void)? + private var pickerAnchor = CGPoint.zero + + private static weak var running: ScreenRecorder? + + public init(bridge: ToolsBridge, settings: RecordingSettings? = nil, + permissionGate: (@MainActor (PermissionNeed) async -> Bool)? = nil) { + self.bridge = bridge + self.settings = settings ?? .shared + self.permissionGate = permissionGate ?? { await PermissionCenter.shared.ensure($0) } + loadTargets = { + // Long enough for the one compile after an update (about 30 s, more on a busy Mac); the compile goes on + // without this call if it is cut, and the recording then reuses its binary. + let result = try await bridge.run(subcommand: "control", args: ["capture", "targets", "--json"], + timeoutSeconds: 120) + guard result.exitCode == 0 else { + throw ToolsBridgeError.refused(String(result.stderr.suffix(600))) + } + return try JSONDecoder().decode(RecordingTargets.self, from: Data(result.stdout.utf8)) + } + } + + /// Opens the picker near `anchor` (a Cocoa screen point; the pointer when nil), starting on the front window of + /// `preferredPid` (the app the person was in), then records what they choose. + public func begin(host: ScreenRecordingHost, anchor: CGPoint? = nil, preferredPid: Int32? = nil) { + if let running = Self.running, running.isActive { + running.controls?.show() + if running !== self { host.onError("A screen recording is already running. Use its controls to end it.") } + return + } + Self.running = self + self.host = host + isActive = true + let preferred = preferredPid ?? NSWorkspace.shared.frontmostApplication.flatMap { app in + app.processIdentifier == ProcessInfo.processInfo.processIdentifier ? nil : app.processIdentifier + } + openPicker(title: "Choose what to record", anchor: anchor ?? NSEvent.mouseLocation, preferredPid: preferred, + choose: { [weak self] target in self?.start(target) }, + cancel: { [weak self] in self?.finish() }) + } + + /// Stops and hands the movie to the host, as the controls' use button does. + public func stopAndUse() { controller?.stop() } + + public func discard() { controller?.discard() } + + /// Brings the controls of a running recording forward. + public func showControls() { controls?.show() } + + /// Chooses in the open picker as a click on its row does (the widget's driver and tests). False when no picker is + /// open or `pick` finds nothing in the listed targets. + @discardableResult + public func choose(_ pick: (RecordingTargets) -> RecordingTarget?) -> Bool { + guard picker.isVisible, let targets, let target = pick(targets) else { return false } + picker.dismiss() + pickerChoose?(target) + return true + } + + /// One line for logs and the driver: the phase, why a start is slow, and whether the target outline shows. + public var stateDescription: String { + guard let controller else { return "idle picker=\(picker.isVisible)" } + return "phase=\(controller.phase.rawValue) preparing=\(controller.preparing ?? "-") " + + "outline=\(outlinedTarget?.title ?? "-") elapsedMs=\(Int(controller.elapsedMs))" + } + + // MARK: Picker + + private func openPicker(title: String, anchor: CGPoint, preferredPid: Int32?, + choose: @escaping (RecordingTarget) -> Void, cancel: @escaping () -> Void) { + pickerTitle = title + pickerAnchor = anchor + pickerChoose = choose + pickerCancel = cancel + pickerLoad = targets.map { .loaded($0) } ?? .loading + pickerInitial = controller?.target ?? targets?.defaultTarget(preferredPid: preferredPid) + pickerFilter = nil + presentPicker() + Task { [weak self] in await self?.refreshTargets(preferredPid: preferredPid) } + } + + private func refreshTargets(preferredPid: Int32?) async { + let previousLoad = pickerLoad + let previousInitial = pickerInitial + do { + let own = ProcessInfo.processInfo.processIdentifier + let loaded = try await loadTargets().excluding(pids: [own]) + targets = loaded + pickerLoad = .loaded(loaded) + if controller?.target == nil { pickerInitial = loaded.defaultTarget(preferredPid: preferredPid) } + } catch { + PerfLog.mark("recording.targets failed \(error.localizedDescription)") + pickerLoad = .failed("Could not list what to record: \(error.localizedDescription)") + } + // A new picker view resets the person's query, tab and arrow selection: skip it when nothing changed. + guard picker.isVisible, pickerLoad != previousLoad || pickerInitial != previousInitial else { return } + presentPicker() + } + + private func presentPicker() { + let view = RecordingTargetPicker( + load: pickerLoad, title: pickerTitle, initial: pickerInitial, + onChoose: { [weak self] target in + guard let self else { return } + self.picker.dismiss() + self.pickerChoose?(target) + }, + onCancel: { [weak self] in + guard let self else { return } + self.picker.dismiss() + self.pickerCancel?() + }, + onRetry: { [weak self] in + guard let self else { return } + self.pickerLoad = .loading + self.presentPicker() + Task { await self.refreshTargets(preferredPid: nil) } + }, + restoring: pickerFilter, + onFilterChange: { [weak self] tab, query in self?.pickerFilter = (tab, query) }) + picker.present(view, near: pickerAnchor) { [weak self] in self?.pickerCancel?() } + } + + // MARK: Session + + private func start(_ target: RecordingTarget) { + guard let host else { return } + Task { [weak self] in + guard let self else { return } + let need = PermissionNeed(.screenRecording, reason: host.permissionReason, grantWorksInNewProcess: true) + guard await self.permissionGate(need) else { + self.finish() + return + } + self.record(target) + } + } + + private func record(_ target: RecordingTarget) { + guard let host else { return } + let controller = RecordingSessionController(bridge: bridge) + let controls = RecordingControlsPanel() + self.controller = controller + self.controls = controls + controller.onDone = { [weak self] url, _ in self?.use(url) } + controller.onDiscarded = { [weak self] in + self?.host?.onDiscard() + self?.finish() + } + controller.onFailed = { [weak self] message, code in self?.failed(message, code: code) } + controllerWatch = controller.objectWillChange.sink { [weak self] _ in + DispatchQueue.main.async { self?.renderControls() } + } + settingsWatch = settings.objectWillChange.sink { [weak self] _ in + DispatchQueue.main.async { self?.renderControls() } + } + let outline = RecordingTargetOutline() + self.outline = outline + renderControls() + place(target) + // The controls are on screen before the recorder lists windows, so it can leave them out by id. + controls.show() + do { + try controller.start(target: target, excludeWindowIDs: [controls.windowID] + outline.windowIDs, + frameRate: settings.frameRate, showsBorder: settings.showsBorder) + host.onStart() + } catch { + failed("Could not start the recorder: \(error.localizedDescription)", code: nil) + } + } + + /// The view goes in once per recording; each tick only changes its state, so what the view holds itself (an open + /// discard question) survives the clock. + private func renderControls() { + guard let controller, let controls, let host else { return } + renderOutline(phase: controller.phase, target: controller.target) + let state = RecordingControlsState( + phase: controller.phase, elapsedMs: controller.elapsedMs, limitMs: controller.limitMs, + target: controller.target, pauseReason: controller.pauseReason, notice: controller.notice, + preparing: controller.preparing, useTitle: host.useTitle, useSymbol: host.useSymbol, + confirmsDiscard: settings.confirmsDiscard) + if let controlsModel { + if controlsModel.state != state { controlsModel.state = state } + return + } + let model = RecordingControlsModel(state: state) + controlsModel = model + controls.setContent(RecordingControlsHost(model: model, actions: RecordingControlsActions( + pause: { [weak controller] in controller?.pause() }, + resume: { [weak controller] in controller?.resume() }, + switchTarget: { [weak self] in self?.switchTarget() }, + restart: { [weak controller] in controller?.restart() }, + use: { [weak controller] in controller?.stop() }, + discard: { [weak controller] in controller?.discard() }, + dismissNotice: { [weak controller] in controller?.notice = nil }, + moved: { [weak controls] in controls?.noteMoved() }))) + } + + /// Pauses (so the picker never shows in the movie), lets the person choose, then switches or resumes. The picker + /// is a window the recorder was not told to leave out, so it opens only once the recorder confirms the pause: a + /// display recording would otherwise capture it in the frames before the pause takes effect. + private func switchTarget() { + guard let controller, !picker.isVisible, pauseWatch == nil else { return } + let wasRecording = controller.phase == .recording || controller.phase == .starting + guard wasRecording else { return openSwitchPicker(resumeOnCancel: false) } + controller.pause() + pauseWatch = controller.$phase + .first { $0 != .recording && $0 != .starting } + .timeout(.seconds(Self.pauseConfirmSeconds), scheduler: DispatchQueue.main) + .receive(on: DispatchQueue.main) + .sink(receiveCompletion: { [weak self] _ in + guard let self, self.pauseWatch != nil else { return } + self.pauseWatch = nil + guard let controller = self.controller, controller.phase != .paused else { return } + if controller.phase == .recording || controller.phase == .starting { + controller.notice = "The recording did not pause, so the picker stayed closed. Try Switch again." + } + }, receiveValue: { [weak self] phase in + guard let self else { return } + self.pauseWatch = nil + if phase == .paused { self.openSwitchPicker(resumeOnCancel: true) } + }) + } + + /// How long `switchTarget` waits for the recorder's pause before it gives up and keeps the picker closed. + static let pauseConfirmSeconds = 3 + + private func openSwitchPicker(resumeOnCancel: Bool) { + guard let controls else { return } + let wasRecording = resumeOnCancel + let frame = controls.window.frame + openPicker(title: "Switch to", anchor: CGPoint(x: frame.midX, y: frame.minY), preferredPid: nil, + choose: { [weak self] target in + self?.controller?.switchTo(target) + self?.place(target) + }, + cancel: { [weak self] in + if wasRecording { self?.controller?.resume() } + }) + } + + /// The dashed outline stands in for the recorder's own border until the first frame: it shows what was chosen + /// at once, also through a slow start. It follows the border setting. + private func renderOutline(phase: RecordingPhase, target: RecordingTarget?) { + guard let outline else { return } + let wanted = phase == .starting && settings.showsBorder ? target : nil + guard wanted != outlinedTarget else { return } + outlinedTarget = wanted + guard let wanted else { return outline.hide() } + outline.show(outlineRects(wanted)) + } + + private func outlineRects(_ target: RecordingTarget) -> [CGRect] { + guard let windows = listedWindows(target) else { + guard case let .display(index, _) = target, NSScreen.screens.indices.contains(index) else { return [] } + return [NSScreen.screens[index].frame] + } + let primaryHeight = NSScreen.screens.first?.frame.height ?? 0 + return windows.map { RecordingPanelLayout.cocoa(rect($0), primaryHeight: primaryHeight) } + } + + private func place(_ target: RecordingTarget) { + guard let controls else { return } + var display: Int? + if case let .display(index, _) = target { display = index } + let bounds = listedWindows(target).flatMap { windows in + windows.map(rect).reduce(nil) { partial, rect in partial.map { $0.union(rect) } ?? rect } + } + controls.place(settings.placement, targetCG: bounds, displayIndex: display) + } + + /// The listed windows a target records (CG points); nil for a display. + private func listedWindows(_ target: RecordingTarget) -> [RecordingWindow]? { + switch target { + case let .window(ids, _, _, _): return targets?.windows.filter { ids.contains($0.id) } ?? [] + case let .app(app, _): return targets?.windows.filter { String($0.pid) == app } ?? [] + case .display: return nil + } + } + + private func rect(_ window: RecordingWindow) -> CGRect { + CGRect(x: window.bounds.x, y: window.bounds.y, width: window.bounds.width, height: window.bounds.height) + } + + private func use(_ url: URL) { + guard let host else { return finish() } + Task { [weak self] in + do { + try await host.onUse(url) + } catch { + host.onError("The recording is saved at \(url.path), but it could not be used: \(error.localizedDescription)") + } + self?.finish() + } + } + + private func failed(_ message: String, code: String?) { + let host = self.host + finish() + if code == WidgetCaptureError.screenRecordingDenied, let host { + let need = PermissionNeed(.screenRecording, reason: host.permissionReason, grantWorksInNewProcess: true) + Task { [permissionGate] in + if await permissionGate(need) { host.onError(message) } + } + return + } + host?.onError(message) + } + + private func finish() { + picker.dismiss() + controls?.close() + controls = nil + outline?.close() + outline = nil + outlinedTarget = nil + controlsModel = nil + controllerWatch = nil + settingsWatch = nil + controller = nil + host = nil + isActive = false + if Self.running === self { Self.running = nil } + } +} diff --git a/src/macos/GenesisKit/Sources/GenesisKit/Settings/NativeGlass.swift b/src/macos/GenesisKit/Sources/GenesisKit/Settings/NativeGlass.swift index cb6d0c11ed..d42fcd2f18 100644 --- a/src/macos/GenesisKit/Sources/GenesisKit/Settings/NativeGlass.swift +++ b/src/macos/GenesisKit/Sources/GenesisKit/Settings/NativeGlass.swift @@ -53,7 +53,11 @@ public struct NativeGlassShapeSurface: ViewModifier { content.background(NativeThemePalette.graphiteCard, in: shape) .overlay(shape.stroke(.white.opacity(0.08), lineWidth: 1)) } else if #available(macOS 26, *) { + // SwiftUI does not hit-test Liquid Glass itself, so a gap between two rows on a glass panel was not + // part of it: hover there read as "the pointer left", and the widget's hover list closed under a + // scrolling pointer each time a gap passed (Martin, 2026-10-24). The solid form's fill always counted. content.foregroundStyle(Color.white).glassEffect(.regular.tint(NativeThemePalette.glassCardTint), in: shape) + .contentShape(shape) } else { content.background { ZStack { @@ -130,12 +134,15 @@ public struct NativeGlassControl: ViewModifier { } else if opaqueControl { content.modifier(NativeGlassSurface(radius: radius, opaque: true)) } else if #available(macOS 26, *) { + // The whole control answers the pointer, as its opaque form does (glass itself is not hit-tested). if systemStill || requestedStill { content.foregroundStyle(.white).glassEffect( .regular.tint(tint), in: RoundedRectangle(cornerRadius: radius)) + .contentShape(RoundedRectangle(cornerRadius: radius)) } else { content.foregroundStyle(.white).glassEffect( .regular.tint(tint).interactive(), in: RoundedRectangle(cornerRadius: radius)) + .contentShape(RoundedRectangle(cornerRadius: radius)) } } else { content.modifier(NativeGlassSurface(radius: radius)) diff --git a/src/macos/GenesisKit/Sources/GenesisKit/Settings/NativeSettingsModel.swift b/src/macos/GenesisKit/Sources/GenesisKit/Settings/NativeSettingsModel.swift index 065b17b104..3d9651aec7 100644 --- a/src/macos/GenesisKit/Sources/GenesisKit/Settings/NativeSettingsModel.swift +++ b/src/macos/GenesisKit/Sources/GenesisKit/Settings/NativeSettingsModel.swift @@ -56,6 +56,65 @@ public enum NativeSettingsCatalogError: Error, Equatable, CustomStringConvertibl } } +/// A settings link can name a card on its page: `widgets.sessions#inbox-preview` selects the page and pulses the card +/// that carries `.nativeSettingsAnchor("inbox-preview")`. One settings window per process, so one shared center. +@MainActor +public final class NativeSettingsHighlightCenter: ObservableObject { + public static let shared = NativeSettingsHighlightCenter() + @Published public private(set) var anchor: String? + /// Bumped on every request, so the same card asked for twice pulses twice. + @Published public private(set) var token = 0 + /// The last request a card pulsed for, so going back to the page does not replay an old one. + private var deliveredToken = 0 + + public func request(_ anchor: String) { + self.anchor = anchor + token += 1 + } + + /// True once per request, for the card that carries `anchor`. + func consume(_ anchor: String) -> Bool { + guard self.anchor == anchor, deliveredToken != token else { return false } + deliveredToken = token + return true + } + + /// "page#anchor" into its page id and the anchor; a plain page id has no anchor. + public nonisolated static func split(_ requested: String) -> (page: String, anchor: String?) { + guard let mark = requested.firstIndex(of: "#") else { return (requested, nil) } + let anchor = String(requested[requested.index(after: mark)...]) + return (String(requested[.. some View { + content + .attentionPulse(trigger: pulses) + // A window that opens on the page asks before the card exists: the card pulses once it is on screen. + .onAppear { if center.consume(id) { pulseSoon() } } + .onChange(of: center.token) { _, _ in if center.consume(id) { pulseSoon() } } + } + + private func pulseSoon() { + Task { @MainActor in + try? await Task.sleep(for: .milliseconds(350)) + pulses += 1 + } + } +} + +extension View { + /// Names this card for a settings link (`#`): following the link pulses it. + public func nativeSettingsAnchor(_ id: String) -> some View { + modifier(NativeSettingsAnchor(id: id)) + } +} + @MainActor public final class NativeSettingsStore: ObservableObject { @Published public private(set) var sections: [NativeSettingsSection] @@ -78,25 +137,30 @@ public final class NativeSettingsStore: ObservableObject { catalogError = error let acceptedSections = error == nil ? sections.sorted { $0.order < $1.order } : [] self.sections = acceptedSections - let requested = initialPageID ?? defaults.string(forKey: selectionKey) + let link = initialPageID.map(NativeSettingsHighlightCenter.split) + let requested = link?.page ?? defaults.string(forKey: selectionKey) let pages = acceptedSections.flatMap(\.pages) selectedPageID = pages.first(where: { $0.id == requested })?.id ?? pages.first?.id pendingPageID = requested != selectedPageID ? requested : nil + if let anchor = link?.anchor { NativeSettingsHighlightCenter.shared.request(anchor) } } public var selectedPage: NativeSettingsPage? { sections.lazy.flatMap(\.pages).first { $0.id == selectedPageID } } + /// `pageID` may name a card too (`#`): the page opens and the card pulses. @discardableResult - public func select(pageID: String) -> Bool { + public func select(pageID requested: String) -> Bool { + let (pageID, anchor) = NativeSettingsHighlightCenter.split(requested) guard sections.contains(where: { $0.pages.contains(where: { $0.id == pageID }) }) else { - pendingPageID = pageID + pendingPageID = requested return false } pendingPageID = nil selectedPageID = pageID defaults.set(pageID, forKey: selectionKey) + if let anchor { NativeSettingsHighlightCenter.shared.request(anchor) } return true } @@ -115,7 +179,9 @@ public final class NativeSettingsStore: ObservableObject { catalogError = nil candidate.sort { $0.order < $1.order } sections = candidate - if let pendingPageID, candidate.contains(where: { $0.pages.contains(where: { $0.id == pendingPageID }) }) { + if let pendingPageID, candidate.contains(where: { + $0.pages.contains(where: { $0.id == NativeSettingsHighlightCenter.split(pendingPageID).page }) + }) { return select(pageID: pendingPageID) } if selectedPage == nil { selectedPageID = candidate.lazy.flatMap(\.pages).first?.id } diff --git a/src/macos/GenesisKit/Sources/GenesisKit/Widgets/AgentWidgetViews.swift b/src/macos/GenesisKit/Sources/GenesisKit/Widgets/AgentWidgetViews.swift index 84dfe48e62..1de5fc56d8 100644 --- a/src/macos/GenesisKit/Sources/GenesisKit/Widgets/AgentWidgetViews.swift +++ b/src/macos/GenesisKit/Sources/GenesisKit/Widgets/AgentWidgetViews.swift @@ -405,6 +405,8 @@ public struct AgentWidgetView: View { action: actions.submit) } .padding(11).background(.white.opacity(0.065), in: RoundedRectangle(cornerRadius: 12)) + .contentShape(RoundedRectangle(cornerRadius: 12)) + .onTapGesture { composerFocused = true } HStack(spacing: 6) { if sending { ProgressView().controlSize(.mini) diff --git a/src/macos/GenesisKit/Sources/GenesisKit/Widgets/LiveWidgetView.swift b/src/macos/GenesisKit/Sources/GenesisKit/Widgets/LiveWidgetView.swift index c42b03d0ba..9d6bbb3302 100644 --- a/src/macos/GenesisKit/Sources/GenesisKit/Widgets/LiveWidgetView.swift +++ b/src/macos/GenesisKit/Sources/GenesisKit/Widgets/LiveWidgetView.swift @@ -25,6 +25,8 @@ public struct LiveWidgetView: View { } @State private var retry: WidgetOutgoing? + /// The session inbox's filter picked here; until then the one the `inboxFilter` setting opens with. + @State private var inboxFilterPick: String? @State private var choosingSession = false @State private var sourcesExpanded = false @Environment(\.accessibilityReduceMotion) private var systemReduceMotion @@ -57,6 +59,7 @@ public struct LiveWidgetView: View { } public var body: some View { + let _ = RenderProbe.hit("widget.live") Group { if embedded { content @@ -103,7 +106,7 @@ public struct LiveWidgetView: View { draft: Binding(get: { model.draft.text }, set: model.setText), actions: AgentWidgetActions( expand: { model.open(edge) }, collapse: model.collapse, - select: { model.select($0, edge: edge) }, choose: { model.submit(choice: $0) }, + select: { model.select($0, edge: edge) }, choose: { model.choose(option: $0) }, submit: { model.submit() }, settings: { model.showSettings?() }, next: { model.next() }), liveContent: AnyView(content) ) @@ -117,9 +120,16 @@ public struct LiveWidgetView: View { VStack(alignment: .leading, spacing: 12) { header if let notice = model.notice { - HStack { + HStack(alignment: .top) { Text(notice).font(.caption).foregroundStyle(.secondary) + .fixedSize(horizontal: false, vertical: true) Spacer() + if let action = model.noticeAction { + Button(action.title) { action.run() } + .buttonStyle(.genHover()) + .controlSize(.small) + .accessibilityIdentifier("widget.notice.action") + } IconButton(systemName: "xmark", tooltip: "Dismiss this note") { model.notice = nil } } } @@ -147,14 +157,30 @@ public struct LiveWidgetView: View { .textSelection(.enabled) }.font(.caption).foregroundStyle(.orange) } - GenSegmentedTabs("View", items: sectionItems, selection: Binding(get: { section }, set: { section = $0 })) - .accessibilityIdentifier("widget.sections") + if !model.inboxOverview { + GenSegmentedTabs("View", items: sectionItems, selection: Binding(get: { section }, set: { section = $0 })) + .accessibilityIdentifier("widget.sections") + } if model.snapshot == nil || (model.selected == nil && model.snapshot?.rosterLoading == true) { VStack(spacing: 12) { ProgressView() Text("Connecting to your local agents…").font(.callout).foregroundStyle(.secondary) if model.error != nil || model.connectionLost { Button("Reconnect", action: model.start) } }.frame(maxWidth: .infinity, maxHeight: .infinity) + } else if model.inboxOverview { + if model.stagedView == nil { WidgetStagedChip(model: model) } + ScrollView { + Group { + if model.stagedView != nil { WidgetStagedList(model: model) } else { WidgetInboxOverview(model: model) } + } + .frame(maxWidth: .infinity, alignment: .leading).padding(.vertical, 2) + .scrollOverflowContent() + .modifier(DocumentHeight(onScrollView: false) { fit.document = $0; reportFit() }) + } + .modifier(DocumentHeight(onScrollView: true) { fit.document = $0; reportFit() }) + .onGeometryChange(for: CGFloat.self, of: \.size.height) { fit.viewport = $0; reportFit() } + .onDisappear { fit.viewport = nil; fit.document = nil; reportFit() } + .scrollOverflowHints() } else if model.selected == nil { if model.inbox.sessions.isEmpty { VStack(spacing: 10) { @@ -168,10 +194,17 @@ public struct LiveWidgetView: View { waitingSessions } } else { + if composerOnTop { composer } + // Pinned above the list: inside it, a session that opened at its selected card scrolled the filter + // under the top fade (Martin's image 24, 2026-10-23). + if section == "Inbox", !model.cards.isEmpty { inboxFilterBar } + if section == "Inbox", model.stagedView == nil { WidgetStagedChip(model: model) } ScrollViewReader { reader in ScrollView { LazyVStack(alignment: .leading, spacing: 14) { - if section == "Inbox" { + if model.stagedView != nil { + WidgetStagedList(model: model) + } else if section == "Inbox" { inbox } else if section == "Sessions" { WidgetSessionBrowser(model: model) @@ -187,24 +220,35 @@ public struct LiveWidgetView: View { } }.frame(maxWidth: .infinity, alignment: .leading).padding(.vertical, 2) .scrollOverflowContent() - .onGeometryChange(for: CGFloat.self, of: \.size.height) { fit.document = $0; reportFit() } + .modifier(DocumentHeight(onScrollView: false) { fit.document = $0; reportFit() }) } .modifier(LatestScrollAnchor(enabled: section == "Conversation", identity: model.selectedKey)) + // A new view (section or session) starts at its own top: the Sessions list's deep offset, + // kept for the much shorter inbox, drew its rows over the header (Martin, 2026-10-23). + .id(section + "|" + model.selectedKey) .onChange(of: WidgetOutgoingPosition(session: model.selectedKey, messageID: model.outgoing.last?.id)) { previous, next in - guard section == "Inbox", previous.session == next.session, let id = next.messageID else { return } + guard section == "Inbox", previous.session == next.session, let id = next.messageID, + Self.followsOutgoing(model.outgoing.last) else { return } ScrollViewPositioning.scroll(reader, to: "outgoing-" + id, anchor: .bottom) } + .modifier(DocumentHeight(onScrollView: true) { fit.document = $0; reportFit() }) .onGeometryChange(for: CGFloat.self, of: \.size.height) { fit.viewport = $0; reportFit() } .onDisappear { fit.viewport = nil; fit.document = nil; reportFit() } // Outermost, so the scroll view's own anchor modifiers still sit directly on it. .scrollOverflowHints() } // While a decision card holds the composer for an own answer, the bottom one would edit the same draft. - if section != "Sessions" && !(section == "Inbox" && model.ownAnswerCard != nil) { composer } + if showsComposer && !composerOnTop { + composer + } } } .mediaPreviewHost() .padding(18).frame(maxWidth: .infinity, maxHeight: .infinity) + .overlay { + if model.stagedLeavePrompt { WidgetStagedLeavePrompt(model: model) } + } + .animation(reduceMotion ? nil : .easeOut(duration: 0.15), value: model.stagedLeavePrompt) .onGeometryChange(for: CGFloat.self, of: \.size.height) { fit.outer = $0; reportFit() } .onDrop(of: [.fileURL], isTargeted: nil) { providers in for provider in providers { @@ -232,7 +276,7 @@ public struct LiveWidgetView: View { fit.scheduled = false guard let outer = fit.outer else { return } // The same condition that chooses the list over the connecting / nothing-selected placeholders. - if model.snapshot != nil, model.selected != nil { + if model.snapshot != nil, model.selected != nil || model.inboxOverview { // The list is shown but not measured yet: wait instead of reporting the placeholder for a moment. guard let viewport = fit.viewport, let document = fit.document else { return } model.reportAgentFit(outer - viewport + document) @@ -248,13 +292,15 @@ public struct LiveWidgetView: View { choosingSession = true } label: { VStack(alignment: .leading, spacing: 3) { - Text(model.selected?.title ?? "Agent inbox").font(.system(size: 13, weight: .semibold)) - .lineLimit(1) - Text( - [model.selected?.target.provider.capitalized, model.selected?.project].compactMap { $0 } - .joined(separator: " · ") - ) - .font(.system(size: 10)).foregroundStyle(.secondary).lineLimit(1) + Text(model.inboxOverview ? "Inbox" : model.selected?.title ?? "Agent inbox") + .font(.system(size: 13, weight: .semibold)).lineLimit(1) + if model.inboxOverview { + Text("Every session").font(.system(size: 10)).foregroundStyle(.secondary).lineLimit(1) + } else if let selected = model.selected { + WidgetSessionFacts(session: selected, + contextTokens: WidgetModel.contextTokens(model.transcript) ?? model.contextTokens[selected.key], + animate: !reduceMotion) + } } .padding(.horizontal, 6).padding(.vertical, 3) } @@ -273,17 +319,34 @@ public struct LiveWidgetView: View { .onChange(of: choosingSession) { _, value in model.sessionChooserOpen = value } .onDisappear { model.sessionChooserOpen = false } Spacer(minLength: 0) - IconButton(systemName: "pin.slash", tooltip: "Unpin this session from the widget", action: model.unpinSelected) - .accessibilityLabel("Unpin selected session") - IconButton(systemName: "rectangle.on.rectangle", tooltip: "Open this session in Hub") { - model.openHub?(model.selected) + if model.inboxOverview { + if model.selected != nil { + IconButton(systemName: "chevron.left", tooltip: "Back to " + (model.selected?.title ?? "the session")) { + model.inboxOverview = false + }.accessibilityLabel("Back to the session") + } + } else { + IconButton(systemName: "tray.full", tooltip: "Inbox entries from every session") { + model.inboxOverview = true + }.accessibilityLabel("Inbox from every session") + if let selected = model.selected { + WidgetCmuxButton(model: model, session: selected, size: 12) + } + } + // Both act on one session, which the overview of every session does not show. + if !model.inboxOverview { + IconButton(systemName: "pin.slash", tooltip: "Unpin this session from the widget", action: model.unpinSelected) + .accessibilityLabel("Unpin selected session") + IconButton(systemName: "rectangle.on.rectangle", tooltip: "Open this session in Hub") { + model.openHub?(model.selected) + } + .accessibilityLabel("Open session in Hub") } - .accessibilityLabel("Open session in Hub") IconButton(systemName: "slider.horizontal.3", tooltip: "Widget settings and filters") { model.showSettings?() } .accessibilityLabel("Widget settings") - IconButton(systemName: "xmark", tooltip: WidgetKeymap.hint("Collapse", WidgetKeymap.collapse), action: model.collapse) + IconButton(systemName: "xmark", tooltip: WidgetKeymap.hint("Collapse", WidgetKeymap.collapse), action: model.requestCollapse) .accessibilityLabel("Collapse widget") } } @@ -297,6 +360,16 @@ public struct LiveWidgetView: View { private var reduceMotion: Bool { systemReduceMotion || model.effectiveReduceMotion } + /// While a decision card holds the composer for an own answer, a second one would edit the same draft. + private var showsComposer: Bool { + !model.inboxOverview && section != "Sessions" && !(section == "Inbox" && model.ownAnswerCard != nil) + } + + /// The inbox's composer sits under the tabs unless the user moved it to the bottom (Widgets → Sessions). + private var composerOnTop: Bool { + showsComposer && section == "Inbox" && (model.snapshot?.state.preferences.inboxComposer ?? "top") == "top" + } + /// No session picked: the sessions that hold inbox items, so nothing waits out of sight. private var waitingSessions: some View { ScrollView { @@ -353,14 +426,34 @@ public struct LiveWidgetView: View { /// Every item of the session as a row; the open row carries the whole card in place. Several answers from one /// session read as a list, not as one card behind a drop-down. - @ViewBuilder private var inbox: some View { - if !model.cards.isEmpty { + /// The session inbox's filter: every entry, the inbox entries (answers, results, questions) or the decisions. + private var inboxFilter: String { inboxFilterPick ?? model.snapshot?.state.preferences.inboxFilter ?? "all" } + + private var inboxFilterBar: some View { + VStack(alignment: .leading, spacing: 6) { HStack(spacing: 8) { - Text(inboxHeadline).font(.system(size: 11)).foregroundStyle(.secondary) + GenSegmentedTabs( + "Inbox filter", + items: [.init("all", "All"), .init("inbox", "Inbox entries"), .init("decisions", "Decisions")], + selection: Binding(get: { inboxFilter }, set: { value in + model.holdAgentFit() + inboxFilterPick = value + }), minSegmentWidth: 64, identifier: "widget.inbox.filter") Spacer() markAllReadButton } - ForEach(model.cards.reversed()) { card in + Text(inboxHeadline).font(.system(size: 11)).foregroundStyle(.secondary) + } + } + + @ViewBuilder private var inbox: some View { + if !model.cards.isEmpty { + let shown = Self.filtered(model.cards, by: inboxFilter) + if shown.isEmpty { + Text(inboxFilter == "decisions" ? "No decisions in this session." : "No inbox entries in this session.") + .font(.callout).foregroundStyle(.secondary).padding(.vertical, 6) + } + ForEach(shown.reversed()) { card in inboxRow(card).id("card-" + card.id) } } else if model.inboxLoading { @@ -391,6 +484,7 @@ public struct LiveWidgetView: View { /// Opens or folds a row in one `GenMotion` animation; the panel around it grows or shrinks to fit (W3). private func toggle(_ card: WidgetCard) { let opening = !isOpen(card) + model.releaseAgentFit() GenMotion.perform(expanding: opening, reduceMotion: reduceMotion) { if opening { model.openCard(card) @@ -402,6 +496,7 @@ public struct LiveWidgetView: View { private func inboxRow(_ card: WidgetCard) -> some View { let open = isOpen(card) + let _ = RenderProbe.hit("widget.inbox.row") return VStack(alignment: .leading, spacing: 0) { Button { toggle(card) } label: { HStack(alignment: .top, spacing: 9) { @@ -414,7 +509,7 @@ public struct LiveWidgetView: View { // A one-question form's title is its prompt, which the open card does not repeat: open, it // shows whole and as Markdown, the way the form drew it. let isPrompt = Self.asksOneQuestion(card) - (isPrompt ? Text(.init(card.title)) : Text(card.title)) + (isPrompt ? Text(.init(card.title)) : Text(card.numberedTitle)) .font(.system(size: 12.5, weight: open ? .semibold : .medium)) .lineLimit(open ? (isPrompt ? nil : 4) : 2).fixedSize(horizontal: false, vertical: true) if !open, let preview = Self.previewLine(card.body) { @@ -437,7 +532,7 @@ public struct LiveWidgetView: View { .contentShape(Rectangle()) } .buttonStyle(RowButtonStyle(cornerRadius: 11)) - .accessibilityLabel(Self.kindLabel(card) + ": " + card.title) + .accessibilityLabel(Self.kindLabel(card) + ": " + card.numberedTitle) .accessibilityValue(open ? "Expanded" : "Collapsed") .accessibilityIdentifier("widget.inbox.row." + card.id) if open { @@ -452,8 +547,32 @@ public struct LiveWidgetView: View { .strokeBorder(card.needsAnswer ? Color.orange.opacity(0.28) : .clear, lineWidth: 1)) } + /// A card whose pick or note is staged, here or in the snapshot. + private func staged(_ card: WidgetCard) -> Bool { + card.kind == "decision" && card.needsAnswer + && (model.stagedOption(card.sourceId, stored: card.draftOption) != nil || card.draft != nil) + } + + static func choiceTooltip(_ choice: WidgetChoice, index: Int, stages: Bool, picked: Bool) -> String { + let key = index < 9 ? " · key \(index + 1)" : "" + if !stages { return "Answer \(choice.id)" + (choice.recommended ? " (recommended)" : "") + key } + return (picked ? "Staged; click again to unstage" : "Stage \(choice.id), sent later with your other staged decisions") + + (choice.recommended ? " (recommended)" : "") + key + } + @ViewBuilder private func cardDetail(_ card: WidgetCard) -> some View { VStack(alignment: .leading, spacing: 10) { + if let saved = model.draftSavedAt[card.id] ?? (card.draft != nil ? Date(timeIntervalSince1970: card.at / 1000) : nil), + card.kind == "decision", card.needsAnswer { + HStack(spacing: 4) { + Image(systemName: "checkmark.circle.fill").foregroundStyle(SessionPalette.blue) + Text(card.draft.map { "Draft saved: " + $0 } ?? "Draft saved").lineLimit(2) + if model.draftSavedAt[card.id] != nil { + Text("·") + LiveAgo(date: saved, style: .brief) + } + }.font(.system(size: 11)).foregroundStyle(.secondary) + } WidgetReceiptContextView(card: card, model: model) .id(card.id + "|" + String(card.at)) if !card.body.isEmpty { @@ -475,9 +594,11 @@ public struct LiveWidgetView: View { .buttonStyle(.borderedProminent).pointerCursor(!model.cardPending) .disabled(model.importing > 0 || model.cardPending) } else { + let staged = model.stagedOption(card.sourceId, stored: card.draftOption) ForEach(Array(card.choices.enumerated()), id: \.element.id) { index, choice in + let picked = staged == choice.id Button { - model.submit(choice: choice.id) + model.choose(card, option: choice.id) } label: { HStack(alignment: .top, spacing: 8) { Text(verbatim: choice.id).font(.system(size: 10.5, weight: .semibold, design: .rounded)) @@ -487,17 +608,26 @@ public struct LiveWidgetView: View { horizontal: false, vertical: true) .padding(.top, 2) Spacer(minLength: 0) + if picked { + Text("Staged").font(.system(size: 10, weight: .semibold)) + .foregroundStyle(SessionPalette.blue).padding(.top, 3) + } if choice.recommended { Image(systemName: "sparkle").foregroundStyle(.orange) } }.padding(9).frame(maxWidth: .infinity, alignment: .leading) - .background(.white.opacity(0.065), in: RoundedRectangle(cornerRadius: 9)) + .background(picked ? SessionPalette.blue.opacity(0.16) : .white.opacity(0.065), + in: RoundedRectangle(cornerRadius: 9)) + .overlay(RoundedRectangle(cornerRadius: 9) + .strokeBorder(picked ? SessionPalette.blue.opacity(0.7) : .clear, lineWidth: 1)) .contentShape(RoundedRectangle(cornerRadius: 9)) } .buttonStyle(RowButtonStyle(cornerRadius: 9)).disabled(model.importing > 0) - .instantTooltip("Answer \(choice.id)" + (choice.recommended ? " (recommended)" : "") - + (index < 9 ? " · key \(index + 1)" : "")) - .accessibilityLabel("Answer \(choice.id): " + Self.choiceText(choice)) + .instantTooltip(Self.choiceTooltip(choice, index: index, stages: model.decisionClickStages, + picked: picked)) + .accessibilityLabel((model.decisionClickStages ? "Stage " : "Answer ") + "\(choice.id): " + + Self.choiceText(choice)) + .accessibilityValue(picked ? "Staged" : "") } ownAnswer(card) } @@ -510,7 +640,13 @@ public struct LiveWidgetView: View { Button("Mark complete") { model.ledger(card, state: "implemented") } Button("Dismiss") { model.ledger(card, state: "dismissed") } } else if card.kind == "decision" && card.needsAnswer { - Button("Save draft") { model.ledger(card, state: "drafted") } + Button("Save draft") { model.saveDraft(card) } + .instantTooltip("Keeps your text as this decision's note, staged with your other decisions.") + if staged(card) { + Button("Send now") { model.sendStaged([card.sourceId]) } + .instantTooltip("Sends this staged decision alone, without the others.") + Button("Unstage") { model.discardStaged([card.sourceId]) } + } Button("Dismiss") { model.ledger(card, state: "dismissed") } } if card.kind == "result" || card.kind == "answer" { @@ -555,11 +691,30 @@ public struct LiveWidgetView: View { .onAppear { editing = true } } else { WidgetOwnAnswerRow { - GenMotion.perform(expanding: true, reduceMotion: reduceMotion) { model.ownAnswerCardID = card.id } + GenMotion.perform(expanding: true, reduceMotion: reduceMotion) { model.openOwnAnswer(card) } }.disabled(model.importing > 0 || model.cardPending) } } + /// "inbox": answers, results and questions; "decisions": decisions and tasks; anything else: every card. + static func filtered(_ cards: [WidgetCard], by filter: String) -> [WidgetCard] { + switch filter { + case "inbox": return cards.filter { !["decision", "todo"].contains($0.kind) } + case "decisions": return cards.filter { ["decision", "todo"].contains($0.kind) } + default: return cards + } + } + + /// A composer message lands under the list, and the list follows it. An answer sent from a card leaves the list + /// where it is: three scrolls to the bottom moved the answered card away (Martin, 2026-10-23: "the whole widget + /// jumped around … i totally lost idea about which one was it"). + static func followsOutgoing(_ message: WidgetOutgoing?) -> Bool { + guard let message, case .object(let payload) = message.payload, case .string(let kind) = payload["kind"] else { + return false + } + return kind == "followup" + } + static func kindSymbol(_ kind: String) -> String { switch kind { case "answer": return "text.bubble" @@ -689,7 +844,7 @@ public struct LiveWidgetView: View { VStack(alignment: .leading, spacing: 6) { Text(card.kind.capitalized + " · " + card.status).font(.caption2).foregroundStyle( .secondary) - Text(card.title).font(.system(size: 12, weight: .semibold)).textSelection(.enabled) + Text(card.numberedTitle).font(.system(size: 12, weight: .semibold)).textSelection(.enabled) if !card.body.isEmpty { WidgetMarkdown(text: card.body) } @@ -753,7 +908,7 @@ public struct LiveWidgetView: View { id: "out:" + $0.id, date: Date(timeIntervalSince1970: $0.createdAt / 1000), event: .outgoing($0)) } - let activity = (model.snapshot?.activity ?? []).map { + let activity = model.sessionActivity.map { TimelineItem(id: "event:" + $0.id, date: Date(timeIntervalSince1970: $0.at / 1000), event: .activity($0)) } return (turns + cards + outgoing + activity).sorted { @@ -762,7 +917,7 @@ public struct LiveWidgetView: View { } @ViewBuilder private var changes: some View { - if let changes = model.snapshot?.changes, changes.available { + if let changes = model.sessionChanges, changes.available { ForEach(changes.files) { file in Button { openReference(file.path) @@ -852,15 +1007,19 @@ public struct LiveWidgetView: View { attachments(model.draft.assetIds, tile: CGSize(width: 88, height: 60), removable: true) } if model.voiceActive { - HStack(spacing: 8) { - Image(systemName: "waveform").foregroundStyle(.red) - Text( - model.voiceText.isEmpty ? "Listening… tap Stop to edit the transcript" : model.voiceText - ) - .font(.caption).lineLimit(2) - Spacer() - ProgressView(value: min(1, model.voiceLevel * 5)).frame(width: 40) + HStack(alignment: .center, spacing: 9) { + VoiceLevelBars(level: min(1, model.voiceLevel * 5), tint: .red, reduceMotion: model.effectiveReduceMotion) + VStack(alignment: .leading, spacing: 2) { + Text(model.voiceMode == .note ? "Voice note" : "Dictation") + .font(.system(size: 10, weight: .semibold)).foregroundStyle(.secondary) + let transcript = model.voiceTranscript + // The newest words stay in view: a long dictation drops its beginning, not its end. + Text(transcript.isEmpty ? "Listening… the words appear here as you talk" : transcript) + .font(.caption).lineLimit(3).truncationMode(.head) + } + Spacer(minLength: 0) } + .accessibilityElement(children: .combine) } if model.importing > 0 { ProgressView("Saving attachment…").controlSize(.small).font(.caption) @@ -882,6 +1041,7 @@ public struct LiveWidgetView: View { .onSubmit { model.send() } .accessibilityLabel(ownAnswer == nil ? "Reply to selected agent" : "Your own answer") .onGeometryChange(for: CGFloat.self, of: { $0.size.height.rounded() }) { composerFieldHeight = $0 } + .frameProbe("widget.composer.field") } Button { model.send() @@ -902,18 +1062,49 @@ public struct LiveWidgetView: View { }.instantTooltip(WidgetKeymap.hint("Paste image or video", WidgetKeymap.paste)).accessibilityLabel("Paste media") Button(action: model.capture) { Image(systemName: "camera") }.instantTooltip("Capture a screenshot") .accessibilityLabel("Capture screenshot") - Button(action: model.toggleVoice) { - Image(systemName: model.voiceActive ? "stop.circle.fill" : "mic").padding(.horizontal, 8) - .padding(.vertical, 4) + Button(action: model.toggleVideoCapture) { + Image(systemName: model.recordingVideo ? "stop.circle.fill" : "video") } - .modifier(WidgetGlassControl()) - .foregroundStyle(model.voiceActive ? .red : .secondary) - .instantTooltip(model.voiceActionLabel).accessibilityLabel(model.voiceActionLabel) + .foregroundStyle(model.recordingVideo ? .red : .secondary) + .instantTooltip(model.recordingVideo ? "Stop the recording and paste it into the draft" + : "Record a window, an app or a display (up to 10 min)") + .accessibilityLabel(model.recordingVideo ? "Stop and paste the recording" : "Record the screen") + voiceButton Spacer() Text(WidgetKeymap.mediaFooter).font(.system(size: 9)).foregroundStyle(.tertiary) }.font(.system(size: 12)).foregroundStyle(.secondary).buttonStyle(.genHoverIcon()) } .padding(11).background(.white.opacity(0.065), in: RoundedRectangle(cornerRadius: 13)) + // The whole box is the field's target: a click beside the one-line field (or under it, in the padding) starts + // typing. The buttons inside keep their own clicks, because a child's gesture wins over this one. + .contentShape(RoundedRectangle(cornerRadius: 13)) + .onTapGesture { editing = true } + .frameProbe("widget.composer") + } + + /// The microphone: Stop while a dictation runs; otherwise a menu of the two modes when the setting asks, else the + /// set mode at once. + @ViewBuilder private var voiceButton: some View { + let icon = Image(systemName: model.voiceActive ? "stop.circle.fill" : "mic") + .padding(.horizontal, 8).padding(.vertical, 4) + if model.voiceActive || model.composerVoiceChoice != "ask" { + Button(action: model.toggleVoice) { icon } + .modifier(WidgetGlassControl()) + .foregroundStyle(model.voiceActive ? .red : .secondary) + .instantTooltip(model.voiceActionLabel).accessibilityLabel(model.voiceActionLabel) + } else { + Menu { + Button("Dictate into the reply box", systemImage: "text.cursor") { model.startVoice(.dictate) } + Button("Voice note into the reply box (keeps the recording)", systemImage: "waveform") { + model.startVoice(.note) + } + .disabled(model.voiceNoteClip == nil) + } label: { icon } + .menuStyle(.borderlessButton).menuIndicator(.hidden).fixedSize() + .modifier(WidgetGlassControl()) + .foregroundStyle(.secondary) + .instantTooltip(model.voiceActionLabel).accessibilityLabel(model.voiceActionLabel) + } } private func attachments(_ ids: [String], tile: CGSize, removable: Bool) -> some View { @@ -959,6 +1150,30 @@ public struct LiveWidgetView: View { } } +/// The scroll document's height for the pane's fit. macOS 15 and later read it from the scroll view itself +/// (`onScrollView: true`); before, a geometry query of the content (`onScrollView: false`) measures it, which lays out +/// every row of a lazy list on each pass. Each scroll view carries both; one of them reports. +private struct DocumentHeight: ViewModifier { + let onScrollView: Bool + let report: (CGFloat) -> Void + + func body(content: Content) -> some View { + if #available(macOS 15, *) { + if onScrollView { + content.onScrollGeometryChange(for: CGFloat.self, of: { $0.contentSize.height.rounded() }) { _, height in + report(height) + } + } else { + content + } + } else if onScrollView { + content + } else { + content.onGeometryChange(for: CGFloat.self, of: \.size.height) { report($0) } + } + } +} + /// Measured heights of LiveWidgetView's pane, its scroll viewport and the scroll document. final class WidgetContentFit { static let placeholderHeight: CGFloat = 360 diff --git a/src/macos/GenesisKit/Sources/GenesisKit/Widgets/Shelf/WidgetShelfModules.swift b/src/macos/GenesisKit/Sources/GenesisKit/Widgets/Shelf/WidgetShelfModules.swift index da74f72bbf..4665196df8 100644 --- a/src/macos/GenesisKit/Sources/GenesisKit/Widgets/Shelf/WidgetShelfModules.swift +++ b/src/macos/GenesisKit/Sources/GenesisKit/Widgets/Shelf/WidgetShelfModules.swift @@ -88,6 +88,13 @@ struct WidgetShelfModuleView: View { } .disabled(store.isBusy && !store.isCapturing) .accessibilityIdentifier("shelf-capture") + Button(action: store.isRecording ? store.showRecordingControls : store.recordScreen) { + Label(store.isRecording ? "Recording…" : "Record screen…", systemImage: "record.circle") + } + .disabled(store.isCapturing || !store.canRecord) + .instantTooltip(store.isRecording ? "Show the recording's controls" + : "Record a window, an app or a display, then keep the movie here") + .accessibilityIdentifier("shelf-record") } else { Button { store.chooseFiles() } label: { Label("Add files", systemImage: "plus") } .disabled(store.isBusy) @@ -188,7 +195,8 @@ private struct WidgetShelfRow: View { @State private var recipientSelection: WidgetShelfRecipientSelection? static func preview(_ item: WidgetShelfItem) -> MediaPreviewItem { - MediaPreviewItem(id: item.id, path: item.path, name: item.name, kind: item.kind == .capture ? .image : nil) + MediaPreviewItem(id: item.id, path: item.path, name: item.name, + kind: item.kind == .capture ? (item.isVideo ? .video : .image) : nil) } var body: some View { @@ -208,7 +216,8 @@ private struct WidgetShelfRow: View { Spacer(minLength: 0) IconButton( systemName: "tray.and.arrow.down", - tooltip: item.kind == .capture ? "Attach image to an inbox draft" : "Add file reference to an inbox draft" + tooltip: item.kind == .capture ? "Attach \(item.isVideo ? "video" : "image") to an inbox draft" + : "Add file reference to an inbox draft" ) { recipientSelection = WidgetShelfRecipientSelection(item: item, recipients: store.recipients) } @@ -258,7 +267,8 @@ private struct WidgetShelfRecipientPicker: View { var body: some View { VStack(alignment: .leading, spacing: 12) { - Text(selection.item.kind == .capture ? "Attach image to inbox" : "Add file reference to inbox") + Text(selection.item.kind == .capture ? "Attach \(selection.item.isVideo ? "video" : "image") to inbox" + : "Add file reference to inbox") .font(.system(size: 13, weight: .semibold)) Text("This adds to the draft. It does not send.") .font(.system(size: 11)).foregroundStyle(.secondary) diff --git a/src/macos/GenesisKit/Sources/GenesisKit/Widgets/Shelf/WidgetShelfStore.swift b/src/macos/GenesisKit/Sources/GenesisKit/Widgets/Shelf/WidgetShelfStore.swift index f4082aaafb..a160945c7d 100644 --- a/src/macos/GenesisKit/Sources/GenesisKit/Widgets/Shelf/WidgetShelfStore.swift +++ b/src/macos/GenesisKit/Sources/GenesisKit/Widgets/Shelf/WidgetShelfStore.swift @@ -1,4 +1,5 @@ import AppKit +import Combine import CoreGraphics import SwiftUI @@ -13,6 +14,9 @@ public struct WidgetShelfItem: Codable, Identifiable, Equatable, Sendable { public let bytes: Int64 public let createdAt: Double public let assetId: String? + + /// A capture that is a screen recording rather than a screenshot (by its file's extension). + public var isVideo: Bool { MediaKind.of(path: path) == .video } } struct WidgetShelfAttachment: Codable, Sendable { @@ -57,6 +61,11 @@ public final class WidgetShelfStore: ObservableObject { @Published public private(set) var notice: String? /// True once the first inventory read finished, so a view can tell a first load from an item arriving. @Published public private(set) var loaded = false + /// A screen recording started from the Capture shelf runs (from its picker until it ends). + @Published public private(set) var isRecording = false + /// "Record screen…": GenesisKit's generic recorder; "Stop and save" stages the movie here as a capture. + private let screenRecorder: ScreenRecorder? + private var recorderActivity: AnyCancellable? private let request: ([String], Int) async throws -> Data private let recipientSource: () -> [WidgetSession] private let didAttach: (WidgetSession) -> Void @@ -103,6 +112,7 @@ public final class WidgetShelfStore: ObservableObject { } return Data(result.stdout.utf8) } + screenRecorder = ScreenRecorder(bridge: bridge) recipientSource = recipients self.didAttach = didAttach self.onCaptureWillBegin = onCaptureWillBegin @@ -115,6 +125,7 @@ public final class WidgetShelfStore: ObservableObject { request: @escaping ([String], Int) async throws -> Data, screenCaptureAccess: @escaping () -> Bool = { true }, permissionGate: @escaping @MainActor (PermissionNeed) async -> Bool = { _ in true }, + screenRecorder: ScreenRecorder? = nil, recipients: @escaping () -> [WidgetSession] = { [] }, didAttach: @escaping (WidgetSession) -> Void = { _ in }, onCaptureWillBegin: @escaping () -> Void = {}, @@ -123,6 +134,7 @@ public final class WidgetShelfStore: ObservableObject { attachToDraft: ((WidgetShelfItem, WidgetSession) async throws -> Void)? = nil ) { self.request = request + self.screenRecorder = screenRecorder self.screenCaptureAccess = screenCaptureAccess self.permissionGate = permissionGate recipientSource = recipients @@ -296,6 +308,45 @@ public final class WidgetShelfStore: ObservableObject { operation?.cancel() } + /// Whether "Record screen…" can run here (the store has a recorder; previews and tests may not). + public var canRecord: Bool { screenRecorder != nil && !stopped } + + /// "Record screen…": the person picks a window, an app or a display, records it with the floating controls, and + /// "Stop and save" stages the movie here as a capture, ready for any inbox with the frame review of a video. + public func recordScreen() { + guard !stopped, let screenRecorder else { return } + recorderActivity = screenRecorder.$isActive.removeDuplicates().sink { [weak self] active in + self?.isRecording = active + } + screenRecorder.begin(host: ScreenRecordingHost( + useTitle: "Stop and save", + permissionReason: "Record screen records the window, app or display you choose and keeps the movie on the shelf until you choose an inbox.", + onUse: { [weak self] movie in try await self?.stageRecording(movie) }, + onDiscard: { [weak self] in self?.notice = "Recording discarded." }, + onError: { [weak self] message in self?.report(ToolsBridgeError.refused(message)) }, + onStart: { [weak self] in self?.onCaptureWillBegin() })) + } + + public func showRecordingControls() { + screenRecorder?.showControls() + } + + /// Stages a finished recording as a capture, then removes the recorder's file (the shelf keeps its own copy). + func stageRecording(_ movie: URL) async throws { + guard !stopped else { return } + let stamp = Date().formatted(.dateTime.hour().minute().second()) + let data = try await request(["video", movie.path, "--name", "Screen recording \(stamp).mp4"], 300) + let item = try JSONDecoder().decode(WidgetShelfItem.self, from: data) + guard item.kind == .capture else { throw ToolsBridgeError.refused("The recording was not staged as a capture.") } + do { try FileManager.default.removeItem(at: movie) } catch { + GenesisKit.log("widget shelf: recording cleanup \(error.localizedDescription)") + } + guard !stopped else { return } + notice = "Recording staged. Choose an inbox when you are ready." + refresh() + onCaptureStaged() + } + public func importFiles(_ urls: [URL], asImages: Bool = false) { guard !stopped, !isBusy, !urls.isEmpty else { return } guard urls.allSatisfy(\.isFileURL) else { diff --git a/src/macos/GenesisKit/Sources/GenesisKit/Widgets/VoiceNotes/WidgetVoiceNote.swift b/src/macos/GenesisKit/Sources/GenesisKit/Widgets/VoiceNotes/WidgetVoiceNote.swift index 10ea174929..4fe991d830 100644 --- a/src/macos/GenesisKit/Sources/GenesisKit/Widgets/VoiceNotes/WidgetVoiceNote.swift +++ b/src/macos/GenesisKit/Sources/GenesisKit/Widgets/VoiceNotes/WidgetVoiceNote.swift @@ -15,6 +15,31 @@ public struct WidgetVoiceNote: Decodable, Identifiable, Equatable, Sendable { public let transcription: String public let error: String? public let provider: String? + /// Recorded from a reply box: its session and decision. + public var context: WidgetVoiceNoteContext? = nil +} + +/// Where a voice note was recorded: a session's reply box, for one decision or a follow-up. +public struct WidgetVoiceNoteContext: Codable, Equatable, Sendable { + public var sessionKey: String + public var sessionTitle: String? + public var decisionId: String? + public var decisionNumber: Int? + public var title: String? + + public init(sessionKey: String, sessionTitle: String? = nil, decisionId: String? = nil, decisionNumber: Int? = nil, + title: String? = nil) { + self.sessionKey = sessionKey + self.sessionTitle = sessionTitle + self.decisionId = decisionId + self.decisionNumber = decisionNumber + self.title = title + } + + /// "DECISION 19 · gt-native-fixes" or the session's title. + public var label: String { + [decisionNumber.map { "DECISION \($0)" }, sessionTitle].compactMap { $0 }.joined(separator: " · ") + } } struct WidgetVoiceNoteSnapshot: Decodable { @@ -26,6 +51,16 @@ struct WidgetVoiceNoteSnapshot: Decodable { struct WidgetVoiceNoteResult: Decodable { let note: WidgetVoiceNote } public struct WidgetVoiceNoteSettings: Sendable { + /// The `transcribe` command line. An empty setting is left out: it passed `--language ""`, which the vault key + /// window then showed as a bare `--language.` (Martin's screenshot, 2026-10-23). + func transcribeArguments(id: String, revision: Int) -> [String] { + var args = ["transcribe", id, "--revision", String(revision), "--provider", provider] + for (flag, value) in [("--account", account), ("--model", model), ("--language", language)] { + if let value, !value.isEmpty { args += [flag, value] } + } + return args + } + public var provider: String public var account: String? public var model: String? diff --git a/src/macos/GenesisKit/Sources/GenesisKit/Widgets/VoiceNotes/WidgetVoiceNotesModule.swift b/src/macos/GenesisKit/Sources/GenesisKit/Widgets/VoiceNotes/WidgetVoiceNotesModule.swift index 9ba01e9db0..8b4115e896 100644 --- a/src/macos/GenesisKit/Sources/GenesisKit/Widgets/VoiceNotes/WidgetVoiceNotesModule.swift +++ b/src/macos/GenesisKit/Sources/GenesisKit/Widgets/VoiceNotes/WidgetVoiceNotesModule.swift @@ -11,9 +11,62 @@ public extension WidgetVoiceNotesStore { } } +/// The notice for an error the store reports: the message, what was found, and the fixes for that kind of failure. +/// The store keeps only the text, so the kinds are told apart by their exact messages. +@MainActor +enum WidgetVoiceNotesNotice { + struct Content { + let text: String + var detail: String? + var more: String? + var fixes: [NoticeFix] = [] + } + + static let failedRunsCommand = "tools genesis-tools logs failures --tool hub --grep voice-notes --limit 5" + + static func content(for error: String, permission: VoiceMicrophonePermission, bundleID: String?, + reviewMicrophone: @escaping @MainActor () -> Void, + recordAgain: @escaping @MainActor () -> Void) -> Content { + if let refusal = FlowFocusRuntime.lastAudioRefusal, refusal.text == error { + return Content(text: error, detail: refusal.detail, more: refusal.found, + fixes: refusal.fixes + [.button("Record again", symbol: "mic.fill", perform: recordAgain)]) + } + let reset = "tccutil reset Microphone \(bundleID ?? "com.genesiscz.genesistools")" + let kinds: [VoiceCommandFailure] = [.microphonePermission, .microphonePromptOpen, .noAudio, .interrupted, + .captureFailed, .cancelled, .operationFailed, .timedOut] + switch kinds.first(where: { $0.errorDescription == error }) { + case .microphonePermission: + return Content(text: error, detail: "macOS answers \"\(describe(permission))\" for this app's microphone access.", + fixes: [.button("Review microphone access", symbol: "mic", perform: reviewMicrophone), + .command("Or reset the grant from a terminal, so macOS asks again on the next Record", reset)]) + case .microphonePromptOpen: + return Content(text: error, fixes: [.button("Review microphone access", symbol: "mic", perform: reviewMicrophone)]) + case .noAudio, .captureFailed, .interrupted: + return Content(text: error, fixes: [.button("Record again", symbol: "mic.fill", perform: recordAgain), + .command("See why the recorder failed", failedRunsCommand)]) + case .operationFailed, .timedOut: + return Content(text: error, fixes: [.command("See why it failed", failedRunsCommand)]) + case .cancelled, nil: + return Content(text: error, fixes: [.command("See recent failed voice runs", failedRunsCommand)]) + } + } + + private static func describe(_ permission: VoiceMicrophonePermission) -> String { + switch permission { + case .notDetermined: return "not asked yet" + case .authorized: return "allowed" + case .denied: return "denied" + case .restricted: return "restricted" + case .unknown: return "unknown" + } + } +} + private struct WidgetVoiceNotesView: View { @ObservedObject var store: WidgetVoiceNotesStore let presentation: WidgetModulePresentation + /// The error the person dismissed; it stays hidden until the next operation starts (the store clears its error then). + @State private var dismissedError: String? var body: some View { Group { @@ -44,7 +97,14 @@ private struct WidgetVoiceNotesView: View { .onReceive(NotificationCenter.default.publisher(for: NSApplication.didBecomeActiveNotification)) { _ in store.refreshMicrophonePermission() } + .onChange(of: store.phase) { _, phase in + if phase != nil { dismissedError = nil } + } + } + private var shownError: String? { + guard let error = store.error, error != dismissedError else { return nil } + return error } @ViewBuilder private var expanded: some View { @@ -69,7 +129,9 @@ private struct WidgetVoiceNotesView: View { if phase == "Recording" { WidgetVoiceNoteMeterView(meter: store.meter) } } } - if let guidance = store.microphoneGuidance { + // The permission notice below carries the same "Review microphone access" fix, so the box steps aside for it. + if let guidance = store.microphoneGuidance, + shownError != VoiceCommandFailure.microphonePermission.errorDescription { VStack(alignment: .leading, spacing: 6) { Label(guidance, systemImage: "mic.slash").font(.system(size: 11)) .fixedSize(horizontal: false, vertical: true) @@ -80,8 +142,14 @@ private struct WidgetVoiceNotesView: View { .background(.orange.opacity(0.10), in: RoundedRectangle(cornerRadius: 8)) .accessibilityIdentifier("voice-microphone-permission") } - if let error = store.error { - Text(error).font(.system(size: 11)).foregroundStyle(KitPalette.removed).textSelection(.enabled) + if let error = shownError { + let notice = WidgetVoiceNotesNotice.content( + for: error, permission: store.microphonePermission, bundleID: Bundle.main.bundleIdentifier, + reviewMicrophone: store.reviewMicrophoneAccess, recordAgain: { store.record() }) + NoticePill(text: notice.text, detail: notice.detail, more: notice.more, tone: .error, fixes: notice.fixes) { + dismissedError = error + } + .accessibilityIdentifier("voice-notes-error") } else if let receipt = store.receipt { Label(receipt, systemImage: "checkmark.circle").font(.system(size: 11)).foregroundStyle(.secondary) } diff --git a/src/macos/GenesisKit/Sources/GenesisKit/Widgets/VoiceNotes/WidgetVoiceNotesStore.swift b/src/macos/GenesisKit/Sources/GenesisKit/Widgets/VoiceNotes/WidgetVoiceNotesStore.swift index 5ede1b8384..7e6bfa4066 100644 --- a/src/macos/GenesisKit/Sources/GenesisKit/Widgets/VoiceNotes/WidgetVoiceNotesStore.swift +++ b/src/macos/GenesisKit/Sources/GenesisKit/Widgets/VoiceNotes/WidgetVoiceNotesStore.swift @@ -31,6 +31,7 @@ public final class WidgetVoiceNotesStore: ObservableObject { private let cancelCommand: () -> Void private let acquireAudio: () async throws -> any VoiceRecordingLease private let settings: () -> WidgetVoiceNoteSettings + private var incomingDirectory: String? private let sessionsSource: () -> [WidgetSession] private let attachDraft: (WidgetSession, WidgetVoiceNote) async throws -> String private let micLauncher: String @@ -193,6 +194,42 @@ public final class WidgetVoiceNotesStore: ObservableObject { } } + /// Where the reply box's live dictation writes its audio (`voice listen --save-audio`) before `fileLiveNote` + /// keeps it. The CLI names the folder once (it follows GENESIS_TOOLS_HOME); later clips are named here. + public func newIncomingClipPath() async throws -> String { + if let incomingDirectory { + return (incomingDirectory as NSString).appendingPathComponent(UUID().uuidString.lowercased() + ".pcm") + } + struct Path: Decodable { let path: String } + // A one-shot command: `execute` is the recorder's transport and answers only a recording's own events. + let data = try await request(["incoming-path", "--json"]) + let path = try JSONDecoder().decode(Path.self, from: data).path + incomingDirectory = (path as NSString).deletingLastPathComponent + return path + } + + /// Keeps a live dictation from the reply box as a voice note: its audio, its text, and where it was spoken + /// (Martin, 2026-10-23: it "can go into the voice notes place (with context of which session, which decision")). + public func fileLiveNote(clip: String, text: String, context: WidgetVoiceNoteContext, provider: String, + model: String?, language: String?) { + var args = ["add", "--clip", clip, "--text", text, "--provider", provider, "--json"] + if let json = try? JSONEncoder().encode(context), let encoded = String(data: json, encoding: .utf8) { + args += ["--context", encoded] + } + for (flag, value) in [("--model", model), ("--language", language)] { + if let value, !value.isEmpty { args += [flag, value] } + } + Task { [weak self] in + guard let self else { return } + do { + let data = try await self.request(args) + self.put(try JSONDecoder().decode(WidgetVoiceNoteResult.self, from: data).note) + } catch { + self.report(error) + } + } + } + public func record(input: String = "mic") { begin(input == "mic" ? "Checking microphone access" : "Starting recording") { [self] in meter.reset() @@ -209,8 +246,8 @@ public final class WidgetVoiceNotesStore: ObservableObject { try await lease.release() throw CancellationError() } - let data = try await execute(["record", "--input", input, "--mic-launcher", micLauncher, - "--wait-for-start", "--stop-on-stdin", "--json"], lease) { [weak self, weak meter] event in + let args = ["record", "--input", input, "--mic-launcher", micLauncher, "--wait-for-start", "--stop-on-stdin", "--json"] + let data = try await execute(args, lease) { [weak self, weak meter] event in meter?.receive(event) if event.kind == "recording", let self, self.phase == "Starting recording" { self.phase = "Recording" } } @@ -232,11 +269,7 @@ public final class WidgetVoiceNotesStore: ObservableObject { let configuration = settings() begin("Transcribing") { [self] in let saved = try await save(note) - var args = ["transcribe", saved.id, "--revision", String(saved.revision), "--provider", configuration.provider] - if let account = configuration.account { args += ["--account", account] } - if let model = configuration.model { args += ["--model", model] } - if let language = configuration.language { args += ["--language", language] } - let data = try await execute(args, nil) { _ in } + let data = try await execute(configuration.transcribeArguments(id: saved.id, revision: saved.revision), nil) { _ in } try Task.checkCancellation() let updated = try JSONDecoder().decode(WidgetVoiceNoteResult.self, from: data).note put(updated) diff --git a/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetAgentPreview.swift b/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetAgentPreview.swift index fc245684e9..fc7b5aed5d 100644 --- a/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetAgentPreview.swift +++ b/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetAgentPreview.swift @@ -63,9 +63,9 @@ struct WidgetSessionActivity: View { struct AgentWidgetPreview: View { @ObservedObject var model: WidgetModel let surface: WidgetSurfaceID - private var waiting: Int { model.waitingSessionCount } var body: some View { + let _ = RenderProbe.hit("widget.preview") VStack(alignment: .leading, spacing: 8) { HStack(spacing: 8) { Image(systemName: "tray").font(.system(size: 12, weight: .medium)) @@ -73,22 +73,22 @@ struct AgentWidgetPreview: View { Text("Inbox").font(.system(size: 12, weight: .semibold)) Spacer() if model.inbox.needsAnswer > 0 { - Button { model.openInboxNotification(on: surface, needsAnswer: true) } label: { + Button { model.toggleInboxOverview(on: surface) } label: { HStack(spacing: 4) { WidgetInboxCount(count: model.inbox.needsAnswer, needsAnswer: true, pulse: model.inboxPulse, reduceMotion: model.effectiveReduceMotion, complete: model.inbox.complete) Text("need you").font(.system(size: 10)) } - }.buttonStyle(.genHoverPlain()).instantTooltip("Open need you inbox items") + }.buttonStyle(.genHoverPlain()).instantTooltip("Every inbox entry that waits for your answer") } if model.inbox.unread > 0 { - Button { model.openInboxNotification(on: surface, needsAnswer: false) } label: { + Button { model.toggleInboxOverview(on: surface) } label: { HStack(spacing: 4) { WidgetInboxCount(count: model.inbox.unread, needsAnswer: false, pulse: model.inboxPulse, reduceMotion: model.effectiveReduceMotion, complete: model.inbox.complete) Text("unread").font(.system(size: 10)) } - }.buttonStyle(.genHoverPlain()).instantTooltip("Open unread inbox items") + }.buttonStyle(.genHoverPlain()).instantTooltip("Every unread inbox entry, from all sessions") } if model.inboxCount == 0 { Text(model.inbox.complete ? "All caught up" : "Inbox status unavailable") @@ -98,75 +98,59 @@ struct AgentWidgetPreview: View { size: 11, action: model.markAllRead) .accessibilityLabel("Mark all inbox items read") } - }.padding(.horizontal, 5).padding(.bottom, 5) + }.padding(.horizontal, 5) - if model.previewSessions.isEmpty { + HStack(spacing: 6) { + GenSegmentedTabs( + "Show", items: WidgetPreviewChoices.filters.map { .init($0.value, $0.title) }, + selection: Binding(get: { model.previewFilter }, set: { model.setPreview(filter: $0) }), + minSegmentWidth: 70, identifier: "widget.preview.filter") + Spacer(minLength: 4) + IconButton(systemName: "arrow.up.arrow.down", tooltip: sortTooltip, size: 10.5) { + let sorts = WidgetPreviewChoices.sorts.map(\.value) + let index = sorts.firstIndex(of: model.previewSort) ?? 0 + model.setPreview(sort: sorts[(index + 1) % sorts.count]) + }.accessibilityLabel("Order: " + WidgetPreviewChoices.title(sort: model.previewSort)) + IconButton(systemName: "slider.horizontal.3", tooltip: "Choose what this list shows and its order", + size: 10.5) { + model.showSettingsPage?("widgets.sessions#inbox-preview") + }.accessibilityLabel("Inbox preview settings") + }.padding(.horizontal, 5).padding(.bottom, 3) + + if model.previewList.isEmpty { HStack(spacing: 12) { GenesisWidgetMark() VStack(alignment: .leading, spacing: 4) { - Text("Nothing needs you.").font(.system(size: 13, weight: .medium)) + Text(model.previewFilter == "active" ? "Nothing is running." : "Nothing needs you.") + .font(.system(size: 13, weight: .medium)) Text("New questions and finished work appear here.") .font(.system(size: 11)).foregroundStyle(.secondary) } - }.frame(maxWidth: .infinity, maxHeight: .infinity, alignment: .leading).padding(.horizontal, 5) + }.frame(maxWidth: .infinity, alignment: .leading).padding(.horizontal, 5).padding(.vertical, 6) } else { - ForEach(model.previewSessions) { session in - Button { - model.openInboxNotification(on: surface, key: session.key) - } label: { - HStack(spacing: 9) { - AIProviderGlyph(meta: AIProviders.meta(for: session.target.provider), size: 21) - VStack(alignment: .leading, spacing: 4) { - Text(session.title).font(.system(size: 11.5, weight: .medium)).lineLimit(1) - HStack(spacing: 5) { - Text( - session.project.isEmpty ? session.target.provider.capitalized : session.project - ) - .lineLimit(1) - Text("·") - Text(session.visualStatus.label).lineLimit(1) - }.font(.system(size: 9.5)).foregroundStyle(.white.opacity(0.45)) - } - Spacer(minLength: 2) - if let inbox = model.inboxFor(session.key), inbox.unread + inbox.needsAnswer > 0 { - WidgetInboxCount(count: inbox.unread + inbox.needsAnswer, needsAnswer: inbox.needsAnswer > 0, - pulse: model.inboxPulseFor(session.key), reduceMotion: model.effectiveReduceMotion, - complete: model.inbox.complete) - } else { - WidgetActivityIndicator(status: session.visualStatus, animate: !model.effectiveReduceMotion) - .frame(width: 12, height: 12) - } - } - .padding(.horizontal, 9).padding(.vertical, 9) - .frame(maxWidth: .infinity, alignment: .leading) - .background( - session.status == "waiting" ? Color.orange.opacity(0.065) : .white.opacity(0.025), - in: RoundedRectangle(cornerRadius: 10)) + FittedScroll(maxHeight: 360, initialHeight: model.previewListHeight, + measured: { if $0 > 0 { model.previewListHeight = $0 } }) { + LazyVStack(spacing: 6) { + ForEach(model.previewList) { session in row(session) } } - .buttonStyle(.genHoverRow()) - .accessibilityLabel("Open " + session.title + ", " + session.visualStatus.label) } } - Spacer(minLength: 0) Button { model.section = "Sessions" + model.inboxOverview = false model.openModule("agents", on: surface) } label: { HStack { Image(systemName: "rectangle.stack").font(.system(size: 10)) Text("Projects & sessions").font(.system(size: 10, weight: .medium)) Spacer() - if model.activeSessionCount > 0 { - Text("\(model.activeSessionCount) active").font(.system(size: 9.5)).monospacedDigit() - } + Text(Plural.count(model.previewList.count, "session")).font(.system(size: 9.5)).monospacedDigit() Image(systemName: "chevron.right").font(.system(size: 8, weight: .semibold)) } .foregroundStyle(.white.opacity(0.5)).padding(.horizontal, 5).padding(.vertical, 5) .contentShape(Rectangle()) }.buttonStyle(.genHoverPlain()) - .instantTooltip(model.activeSessionCount > 0 - ? "\(model.activeSessionCount) sessions are working or wait for you. Browse every project and session." - : "Browse every project and session") + .instantTooltip("Browse every project and session") .accessibilityLabel("Browse all projects and sessions") } .padding(.horizontal, 13).padding(.vertical, 14) @@ -176,4 +160,132 @@ struct AgentWidgetPreview: View { .onGeometryChange(for: CGFloat.self, of: \.size.height) { model.reportPreviewFit($0) } .frame(maxWidth: .infinity, maxHeight: .infinity, alignment: .top) } + + private var sortTooltip: String { + let sorts = WidgetPreviewChoices.sorts + let index = sorts.firstIndex { $0.value == model.previewSort } ?? 0 + return "Order: \(sorts[index].title). Click for \(sorts[(index + 1) % sorts.count].title)." + } + + private func row(_ session: WidgetSession) -> some View { + WidgetHoverRow { hovering in HStack(spacing: 4) { + Button { + model.openInboxNotification(on: surface, key: session.key) + } label: { + HStack(spacing: 9) { + AIProviderGlyph(meta: AIProviders.meta(for: session.target.provider), size: 21) + VStack(alignment: .leading, spacing: 3) { + Text(session.title).font(.system(size: 11.5, weight: .medium)).lineLimit(1) + HStack(spacing: 5) { + Text(session.project.isEmpty ? session.target.provider.capitalized : session.project) + .lineLimit(1) + Text("·") + Text(session.visualStatus.label).lineLimit(1) + }.font(.system(size: 9.5)).foregroundStyle(.white.opacity(0.45)) + WidgetSessionTimes(session: session, inbox: model.inboxFor(session.key)) + } + Spacer(minLength: 2) + if let inbox = model.inboxFor(session.key), inbox.unread + inbox.needsAnswer > 0 { + WidgetInboxCount(count: inbox.unread + inbox.needsAnswer, needsAnswer: inbox.needsAnswer > 0, + pulse: model.inboxPulseFor(session.key), reduceMotion: model.effectiveReduceMotion, + complete: model.inbox.complete) + } else { + WidgetActivityIndicator(status: session.visualStatus, animate: !model.effectiveReduceMotion) + .frame(width: 12, height: 12) + } + } + .padding(.horizontal, 9).padding(.vertical, 9) + .frame(maxWidth: .infinity, alignment: .leading) + .contentShape(Rectangle()) + } + .buttonStyle(RowButtonStyle(cornerRadius: 10)) + .accessibilityLabel("Open " + session.title + ", " + session.visualStatus.label) + WidgetCmuxButton(model: model, session: session, live: hovering) + } } + .padding(.trailing, 4) + .background( + model.inboxFor(session.key).map { $0.needsAnswer > 0 } == true || session.status == "waiting" + ? Color.orange.opacity(0.065) : .white.opacity(0.025), + in: RoundedRectangle(cornerRadius: 10)) + .attentionPulse(trigger: model.attentionTrigger(session.key), style: rowAttention, since: model.attentionStart(session.key)) + } + + private var rowAttention: AttentionPulseStyle { + var style = model.attentionStyle + style.cornerRadius = 10 + return style + } +} + +/// When a session was last active and when its newest question or answer arrived: "working now · asked 3m ago", +/// "active 12m ago · answered 1h ago". Both live, so a row left open keeps counting. +struct WidgetSessionTimes: View { + let session: WidgetSession + let inbox: WidgetInboxSession? + + var body: some View { + HStack(spacing: 5) { + if session.visualStatus == .working { + Text("working now") + } else if session.activityAt > 0 { + LiveAgo(date: Date(timeIntervalSince1970: session.activityAt / 1000), style: .brief) { "active " + $0 } + } + if let entry = Self.latest(inbox) { + Text("·") + LiveAgo(date: Date(timeIntervalSince1970: entry.at / 1000), style: .brief) { entry.verb + " " + $0 } + .foregroundStyle(entry.waiting ? Color.orange.opacity(0.8) : .white.opacity(0.45)) + } + } + .font(.system(size: 9.5)).foregroundStyle(.white.opacity(0.45)).lineLimit(1) + } + + /// The newest entry: a question still waiting wins, since that is what the row is about. + static func latest(_ inbox: WidgetInboxSession?) -> (verb: String, at: Double, waiting: Bool)? { + if let item = inbox?.pendingItem, item.at.isFinite, item.at > 0 { + return (item.kind == "decision" ? "decision asked" : "asked", item.at, true) + } + guard let item = inbox?.unreadItem ?? inbox?.latest, item.at.isFinite, item.at > 0 else { return nil } + switch item.kind { + case "answer": return ("answered", item.at, false) + case "result": return ("finished", item.at, false) + default: return ("new entry", item.at, false) + } + } +} + +/// The header line under a session's title: agent, project and model, then whether it works now or how long it +/// has been idle, and its context size (Martin, 2026-10-23: "the header probably can have way more info"). +struct WidgetSessionFacts: View { + let session: WidgetSession + let contextTokens: Int? + let animate: Bool + + var body: some View { + HStack(spacing: 5) { + Text(Self.origin(session)).lineLimit(1).truncationMode(.middle) + Text("·") + if session.visualStatus == .working { + WidgetActivityIndicator(status: .working, animate: animate) + Text("working").foregroundStyle(session.visualStatus.color) + } else if session.activityAt > 0 { + LiveAgo(date: Date(timeIntervalSince1970: session.activityAt / 1000), style: .brief) { "idle " + $0 } + } else { + Text(session.visualStatus.label) + } + if let contextTokens { + Text("·") + // The header is the session chooser's button, whose tooltip covers this line; VoiceOver says it. + Text("ctx " + SessionFormat.tokens(contextTokens)).monospacedDigit() + .accessibilityLabel("Context size of the last model call, \(contextTokens) tokens") + } + } + .font(.system(size: 10)).foregroundStyle(.secondary).lineLimit(1) + } + + /// "Claude · my-project · opus-5-5": the agent, the project, and the model without its vendor prefix. + static func origin(_ session: WidgetSession) -> String { + let model = session.model.map { $0.replacingOccurrences(of: "claude-", with: "") } + return [AIProviders.meta(for: session.target.provider).displayName, session.project, model] + .compactMap { $0 }.filter { !$0.isEmpty }.joined(separator: " · ") + } } diff --git a/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetAlerts.swift b/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetAlerts.swift index ec37a3c67e..1949eacaff 100644 --- a/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetAlerts.swift +++ b/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetAlerts.swift @@ -35,6 +35,19 @@ public enum WidgetAlerts { } } + /// The sound for a burst: `alertSound` when anything in it waits for an answer, else the answers' own sound + /// (`alertAnswerSound`, where "same" means `alertSound`). nil plays the default sound. + public static func sound(for items: [WidgetInboxItem], preferences: WidgetPreferences?) -> String? { + let asks = items.contains { $0.needsAnswer || $0.kind == "decision" || $0.kind == "form" } + guard !asks, let answer = preferences?.alertAnswerSound, answer != "same" else { return preferences?.alertSound } + return answer + } + + /// Seconds the session behind an alert flashes in the widget; 0 is off. + public static func flashSeconds(_ preferences: WidgetPreferences?) -> Double { + max(0, min(30, preferences?.alertFlashSeconds ?? 5)) + } + private static var lastChime = Date.distantPast /// Plays `name` once; a burst of arrivals within 1.5 s chimes once. "none" or an unknown name is silent. @@ -42,16 +55,31 @@ public enum WidgetAlerts { let sound = name ?? defaultSound guard sound != "none", now.timeIntervalSince(lastChime) >= 1.5 else { return } lastChime = now - guard let player = NSSound(named: NSSound.Name(sound)) else { + guard NSSound(named: NSSound.Name(sound)) != nil else { PerfLog.mark("widget.alert unknown sound \(sound)") return } - player.play() + play(sound) } + /// `NSSound.play` waits for the output device: about 0.8 s on the main thread on 2026-10-23 (HangWatch), long + /// enough to freeze the widget the moment an alert arrives. It plays from a background queue instead. + private static func play(_ name: String) { + soundQueue.async { + guard let player = NSSound(named: NSSound.Name(name)) else { return } + player.play() + playing.append(player) + if playing.count > 4 { playing.removeFirst() } + } + } + + private static let soundQueue = DispatchQueue(label: "genesiskit.widget.chime", qos: .userInitiated) + /// Keeps the last few players alive until they finish; a released NSSound stops at once. + private nonisolated(unsafe) static var playing: [NSSound] = [] + /// Plays `name` now, for the settings page's preview button. @MainActor public static func preview(_ name: String) { guard name != "none" else { return } - NSSound(named: NSSound.Name(name))?.play() + play(name) } } diff --git a/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetBubble.swift b/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetBubble.swift new file mode 100644 index 0000000000..02762308b7 --- /dev/null +++ b/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetBubble.swift @@ -0,0 +1,57 @@ +import SwiftUI + +/// The chrome of a session bubble on the rail and the top strip: a ring that answers the pointer, and the flash of +/// a session that just asked for you. +/// +/// Hover: the ring fades in strong, then settles to a quieter ring while the pointer stays, so a bubble reads as a +/// control the moment it is reached (Martin, 2026-10-23: "missing hover effects (pulsing around)"). Opacity only, +/// two timed steps, never a repeating animation: a `repeatForever` lays the whole window out every frame (swift skill). +struct WidgetBubbleChrome: ViewModifier { + let trigger: Int + let attention: AttentionPulseStyle + let reduceMotion: Bool + /// When the session's flash began; a bubble rebuilt mid-flash (the rail reordered) plays the rest of it. + var since: Date? + @State private var ring = 0.0 + @State private var settle: Task? + + func body(content: Content) -> some View { + content + .background(Circle().fill(Color.white.opacity(ring > 0 ? 0.07 : 0)).padding(2)) + // The whole circle takes clicks and hover at every ring value: the count is not hit-testable, and SwiftUI + // does not hit-test the fill at opacity 0, so a bubble whose attention ended took no click at all. + .contentShape(Circle()) + .overlay(Circle().strokeBorder(Color.white.opacity(ring), lineWidth: 1.5).padding(2).allowsHitTesting(false)) + .attentionPulse(trigger: trigger, style: attention, since: since) + .onHover { inside in + settle?.cancel() + guard inside else { + withAnimation(reduceMotion ? nil : .easeOut(duration: 0.18)) { ring = 0 } + return + } + guard !reduceMotion else { + ring = 0.5 + return + } + withAnimation(.easeOut(duration: 0.14)) { ring = 0.9 } + settle = Task { @MainActor in + try? await Task.sleep(for: .milliseconds(220)) + guard !Task.isCancelled else { return } + withAnimation(.easeInOut(duration: 0.35)) { ring = 0.45 } + } + } + .onDisappear { settle?.cancel() } + } +} + +extension WidgetModel { + /// A rail bubble's tooltip under `bubbleTooltip`: "auto" shows it only while hover previews are off, because + /// the preview already names the session and an extra label over it read as the preview failing to open. + func bubbleTooltip(_ content: TooltipContent) -> TooltipContent { + switch snapshot?.state.preferences.bubbleTooltip ?? "auto" { + case "always": return content + case "never": return TooltipContent() + default: return layout.hoverPreviews ? TooltipContent() : content + } + } +} diff --git a/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetCmuxButton.swift b/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetCmuxButton.swift new file mode 100644 index 0000000000..2828eb0995 --- /dev/null +++ b/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetCmuxButton.swift @@ -0,0 +1,138 @@ +import SwiftUI + +/// The cmux button every widget session carries: a popover with the shared `CmuxSessionPanel` (Focus, open in a pane, +/// choose a pane), the same block the hub's session sidebar shows. The widget drives it through its own `tools` +/// bridge; the pane moves go through GenesisKit's `CmuxCommandRunner`, which the hub's `CmuxHost` uses too. +struct WidgetCmuxButton: View { + @ObservedObject var model: WidgetModel + let session: WidgetSession + var size: CGFloat = 11 + /// In a list, false on every row but the hovered or selected one: the row then carries no button, tooltip or + /// hover sensor, because an accessibility client walks every control of every row per update (swift skill, + /// "Dense lists and accessibility clients"). The popover stays attached, so an open one survives the pointer leaving. + var live = true + @State private var open = false + + var body: some View { + Group { + if live || open { + IconButton(systemName: "terminal", tooltip: "cmux: focus this session, or resume it in a pane", size: size) { + open = true + } + .accessibilityLabel("cmux for " + session.title) + .accessibilityIdentifier("widget.cmux." + session.key) + } else { + Color.clear.frame(width: size + 13, height: size + 13).accessibilityHidden(true) + } + } + .popover(isPresented: $open, arrowEdge: .trailing) { + WidgetCmuxPanel(model: model, session: session).padding(14).frame(width: 380) + } + // The popover takes the pointer out of the edge panel; while it is open the panel must not collapse. Its own + // flag, as the session chooser has one: closing it must never release a window the host opened. + .onChange(of: open) { _, value in model.cmuxPopoverOpen = value } + .onDisappear { if open { model.cmuxPopoverOpen = false } } + } +} + +/// A list row that knows whether the pointer is on it, for `WidgetCmuxButton(live:)` and its other row-only controls. +struct WidgetHoverRow: View { + @ViewBuilder let content: (Bool) -> Content + @State private var hovering = false + + var body: some View { + content(hovering).onHover { hovering = $0 } + } +} + +struct WidgetCmuxPanel: View { + @ObservedObject var model: WidgetModel + let session: WidgetSession + @State private var tree: CmuxTree? + @State private var loading = false + @State private var busy = false + @State private var choosing = false + @State private var notice: CmuxSessionPanel.Notice? + + var body: some View { + VStack(alignment: .leading, spacing: 10) { + VStack(alignment: .leading, spacing: 2) { + Text(session.title).font(.system(size: 12.5, weight: .semibold)).lineLimit(2) + Text([AIProviders.meta(for: session.target.provider).displayName, session.project, + String(session.target.sessionId.prefix(8))].filter { !$0.isEmpty }.joined(separator: " · ")) + .font(.system(size: 10.5)).foregroundStyle(.secondary).lineLimit(1).textSelection(.enabled) + } + CmuxSessionPanel( + sessionId: session.target.sessionId, + window: nil, workspace: nil, pane: nil, surface: nil, + tree: tree, loading: loading, busy: busy, lastPane: nil, + choosing: $choosing, notice: $notice, + modeKey: "widget.cmux.pickerMode", + focus: { Task { await perform(focus) } }, + open: { target in Task { await perform { await open(at: target) } } }, + reload: load) + } + .task { load() } + } + + private func load() { + guard !loading else { return } + loading = true + Task { + // Not the exit status: with cmux unreachable the command prints {available: false, error} and exits 1. + let result = try? await model.bridge.run(subcommand: "ai", args: ["cmux", "tree", "--json"], timeoutSeconds: 20) + tree = result.flatMap { try? CmuxTree.decode(Data($0.stdout.utf8)) } + loading = false + } + } + + private func focus() async -> String? { + let tool = CmuxCommandRunner.tool(provider: session.target.provider) + do { + let result = try await model.bridge.run( + subcommand: tool, args: ["cmux", "focus", session.target.sessionId, "--first", "--json"], timeoutSeconds: 30) + return CmuxCommandRunner.focusFailure( + status: result.exitCode, stdout: Data(result.stdout.utf8), stderr: Data(result.stderr.utf8)) + } catch { + return error.localizedDescription + } + } + + /// Claude resumes through `tools claude cmux open-session`; another agent's resume command is typed into the pane. + private func open(at target: CmuxTarget) async -> String? { + let sessionId = session.target.sessionId + if session.target.provider == "claude", + let args = target.openSessionArgs ?? tree?.windows.first.map({ ["--window", $0.target] }) { + do { + let result = try await model.bridge.run( + subcommand: "claude", args: ["cmux", "open-session", sessionId] + args + ["--json"], timeoutSeconds: 60) + return CmuxCommandRunner.openSessionFailure( + status: result.exitCode, stdout: Data(result.stdout.utf8), stderr: Data(result.stderr.utf8)) + } catch { + return error.localizedDescription + } + } + guard let command = CmuxCommandRunner.resumeCommand(provider: session.target.provider, sessionId: sessionId) else { + return "No resume command for \(session.target.provider)" + } + let cwd = session.target.cwd + let name = session.title + return await Task.detached(priority: .userInitiated) { + CmuxCommandRunner.run(command, cwd: cwd, name: name, at: target) + }.value + } + + private func perform(_ action: @escaping () async -> String?) async { + busy = true + let error = await action() + busy = false + if error == CmuxCommandRunner.paneGone { + // The tab it ran in is closed: say so in words and show where it can go instead. + notice = CmuxSessionPanel.Notice(text: error ?? "", isError: true) + choosing = true + } else { + notice = CmuxSessionPanel.Notice(text: error.map { "cmux: \($0)" } ?? "Done", isError: error != nil) + } + load() + } +} diff --git a/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetCoordinator.swift b/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetCoordinator.swift index da49b5fb3f..c829e5c707 100644 --- a/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetCoordinator.swift +++ b/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetCoordinator.swift @@ -127,7 +127,17 @@ public final class WidgetCoordinator: NSObject, NSWindowDelegate { self.model.dialogOpen = visible || self.settings?.isVisible == true || self.mediaWindow?.isVisible == true } self.voiceNotes = voiceNotes - model.recordVoiceNote = { [weak self] key in self?.openVoiceNotes(for: key) } + // The reply box's microphone dictates live in place; a voice note keeps its audio in Voice Notes without + // opening that panel (Martin, 2026-10-23: "should not open another widget but just let me dictate"). + model.voiceNoteClip = { [weak voiceNotes] in + guard let voiceNotes else { throw CancellationError() } + return try await voiceNotes.newIncomingClipPath() + } + model.fileVoiceNote = { [weak voiceNotes, weak model] clip, text, context in + guard let voiceNotes, let prefs = model?.snapshot?.state.preferences else { return } + voiceNotes.fileLiveNote(clip: clip, text: text, context: context, provider: prefs.voiceProvider, + model: prefs.voiceModel, language: prefs.voiceLanguage) + } do { try modules.register( WidgetModuleDescriptor( @@ -165,11 +175,32 @@ public final class WidgetCoordinator: NSObject, NSWindowDelegate { guard let motion = self?.panels[surface]?.motion, motion.pointerInside != inside else { return } motion.pointerInside = inside } + // SwiftUI's hover no longer moves the panel (the window's tracking area does); a disagreement is logged with + // what lies under the pointer, which is how a hover list that closed under a scroll gets explained. + model.swiftUIHoverChanged = { [weak self] surface, inside in + guard !inside, let self, let panel = self.panels[surface]?.panel, panel.isVisible else { return } + let point = NSEvent.mouseLocation + guard panel.frame.contains(point), let primary = NSScreen.screens.first?.frame else { return } + let cg = CGPoint(x: point.x, y: primary.maxY - point.y) + PerfLog.mark("widget.hover swiftui-left \(surface.key) pointer still on the panel; under it: " + + self.hitTestDescription(cgPoint: cg)) + } model.showSettings = { [weak self] in self?.showSettings() } + model.showSettingsPage = { [weak self] in self?.showSettings(pageID: $0) } model.showMedia = { [weak self] selection in self?.showMedia(selection) } model.inboxArrived = { [weak self] items in self?.alert(items) } } + /// A main-spin with a panel open: the panel closes, so the app answers again, and says why. The spin's stack and + /// the bodies it evaluated are already in app-perf.log (HangWatch). Not a setting: it recovers from a bug, and the + /// other way is an app that does not respond. + func recoverFromSpin(_ seconds: CFAbsoluteTime) { + guard model.expanded != nil else { return } + PerfLog.mark("widget.spin-recovered after \(Int(seconds))s section=\(model.section) overview=\(model.inboxOverview)") + model.collapse() + model.notice = "The panel closed: it redrew for \(Int(seconds)) s without a pause. The details are in app-perf.log." + } + /// Plays the alert chime. Tests replace it. var playChime: (String?) -> Void = { WidgetAlerts.chime($0) } @@ -180,7 +211,10 @@ public final class WidgetCoordinator: NSObject, NSWindowDelegate { let wanted = WidgetAlerts.wanted(items, preferences: preferences) guard let item = WidgetAlerts.popupItem(wanted) else { return } PerfLog.mark("widget.alert items=\(wanted.count) id=\(item.id) kind=\(item.kind)") - playChime(preferences?.alertSound) + playChime(WidgetAlerts.sound(for: wanted, preferences: preferences)) + // The session that asked flashes on its rail bubble and in the lists, popup or not: it says why the widget + // chimed or opened (Martin, 2026-10-23). + model.flashAttention(item.key, seconds: WidgetAlerts.flashSeconds(preferences)) // Checked before the module switch: a popup the user's interaction refuses must not swap the pane they use. guard preferences?.alertPopup ?? true, let surface = peekSurface(), model.canPeek(on: surface) else { return } // The peek shows the agents module and gives the user's own module back when it ends. @@ -202,6 +236,7 @@ public final class WidgetCoordinator: NSObject, NSWindowDelegate { public func start(showSettings: Bool = false, settingsWhenOff: Bool = true) { guard !started, !stopped else { return } started = true + HangWatch.onSpin = { [weak self] seconds in self?.recoverFromSpin(seconds) } runtimeStart = Task { [flowRuntime] in await flowRuntime.start() } rebuildPanels() screenObserver = NotificationCenter.default.addObserver( @@ -429,6 +464,11 @@ public final class WidgetCoordinator: NSObject, NSWindowDelegate { controller.updateContent(content) nextPanels[surface] = controller } else { + defer { + nextPanels[surface]?.pointerChanged = { [weak self] inside in + self?.model.hover(surface, inside: inside) + } + } nextPanels[surface] = EdgePanelController( placement: surface.edge, screen: screen, compactSize: compact, expandedSize: CGSize(width: top ? 432 : 476, height: 600), @@ -627,6 +667,8 @@ public final class WidgetCoordinator: NSObject, NSWindowDelegate { /// within a second (2026-10-11). func collapseIfOutside(_ event: NSEvent) { guard !passivePanels, event.timestamp > model.openedAt, model.expanded != nil, !model.dialogOpen else { return } + // A dictation keeps the panel, so its words stay in view; Esc or the close button still closes it. + guard !model.voiceActive else { return } var window = event.window while let candidate = window { if panels.values.contains(where: { $0.panel === candidate }) { return } @@ -636,7 +678,51 @@ public final class WidgetCoordinator: NSObject, NSWindowDelegate { event.window?.convertPoint(toScreen: event.locationInWindow) ?? event.locationInWindow guard !panels.values.contains(where: { $0.panel.isVisible && $0.panel.frame.contains(point) }) else { return } - model.collapse() + // A click on another GenesisTools window keeps the panel: the vault key window that a dictation or a send + // raises took its Allow click as "outside" and closed the panel under it (Martin, 2026-10-23: "allow with touch + // id does literally hide the widget"). + if Self.ownWindowOwnsPoint(point) { return } + model.requestCollapse() + } + + /// The view chain AppKit's hit test picks at a CoreGraphics screen point (top-left origin) in each visible panel, for + /// the test driver's `hittest` command: accessibility frames can name a control that AppKit never delivers a click + /// to (a nested hosting view placed outside its parent's bounds). + public func hitTestDescription(cgPoint: CGPoint) -> String { + guard let primary = NSScreen.screens.first?.frame else { return "no screen" } + let point = NSPoint(x: cgPoint.x, y: primary.maxY - cgPoint.y) + var lines: [String] = [] + for (surface, controller) in panels where controller.panel.isVisible && controller.panel.frame.contains(point) { + guard let content = controller.panel.contentView else { continue } + let local = content.convert(controller.panel.convertPoint(fromScreen: point), from: nil) + var chain: [String] = [] + var view = content.hitTest(local) + while let current = view, chain.count < 12 { + chain.append("\(type(of: current))\(current.frame.integral)") + view = current.superview + } + lines.append("\(surface.key): " + (chain.isEmpty ? "nothing" : chain.joined(separator: " < "))) + } + return lines.isEmpty ? "no panel at the point" : lines.joined(separator: " | ") + } + + /// Whether the topmost on-screen window at `point` (AppKit screen coordinates) belongs to a GenesisTools process. + static func ownWindowOwnsPoint(_ point: NSPoint) -> Bool { + guard let primary = NSScreen.screens.first?.frame, + let rows = CGWindowListCopyWindowInfo([.optionOnScreenOnly, .excludeDesktopElements], kCGNullWindowID) + as? [[CFString: Any]] else { return false } + // CoreGraphics measures from the primary screen's top edge, AppKit from its bottom edge. + let flipped = CGPoint(x: point.x, y: primary.maxY - point.y) + let bundle = Bundle.main.bundleIdentifier + for row in rows { + guard (row[kCGWindowLayer] as? Int ?? 0) < 1000, + let bounds = row[kCGWindowBounds] as? NSDictionary, + let frame = CGRect(dictionaryRepresentation: bounds), frame.contains(flipped) else { continue } + guard let pid = row[kCGWindowOwnerPID] as? pid_t else { return false } + return pid == ProcessInfo.processInfo.processIdentifier + || (bundle != nil && NSRunningApplication(processIdentifier: pid)?.bundleIdentifier == bundle) + } + return false } private func openVoiceNotes(for key: String) { @@ -692,10 +778,18 @@ public final class WidgetCoordinator: NSObject, NSWindowDelegate { answering: model.answerCard)) { case .pass: return event - case .collapse: model.collapse() + case .collapse: + // The leave question first (Esc keeps the staged decisions), then the staged view, then the list. + if model.stagedLeavePrompt { + model.resolveStagedLeave("keep") + } else if model.stagedView != nil { + model.stagedView = nil + } else if !model.escapeToList() { + model.requestCollapse() + } case .previous: model.next(-1) case .next: model.next() - case .choose(let choice): model.submit(choice: choice) + case .choose(let choice): model.choose(option: choice) } return nil } diff --git a/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetFeatureSettings.swift b/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetFeatureSettings.swift index ff311cf3dc..9356792351 100644 --- a/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetFeatureSettings.swift +++ b/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetFeatureSettings.swift @@ -93,6 +93,12 @@ public enum WidgetFeatureSettings { ) { WidgetModuleSettings(model: model, modules: modules) }, + NativeSettingsPage( + id: "widgets.recording", title: "Screen recording", symbol: "record.circle", + tint: .red, subtitle: "Where the recording controls float and how a recording looks." + ) { + RecordingSettingsView() + }, ], order: 30), NativeSettingsSection( id: "dictation", title: "Dictation", @@ -173,6 +179,21 @@ private struct WidgetGeneralSettings: View { NativeSettingsToggle( "Hover previews", detail: "See a preview without changing keyboard focus.", isOn: boolean("hoverPreviews", prefs?.hoverPreviews ?? true)) + Divider() + NativeSettingsRow("Session bubble tooltips", detail: "Automatic shows them only while hover previews are off.") { + Picker("Session bubble tooltips", selection: string("bubbleTooltip", prefs?.bubbleTooltip ?? "auto")) { + Text("Automatic").tag("auto") + Text("Always").tag("always") + Text("Never").tag("never") + }.labelsHidden().pickerStyle(.segmented).frame(width: 250, alignment: .trailing) + } + Divider() + NativeSettingsRow("Session bubble order", detail: "Same as the list follows the hover list's order. Keep places holds each bubble where it appeared.") { + Picker("Session bubble order", selection: string("railOrder", prefs?.railOrder ?? "list")) { + Text("Same as the list").tag("list") + Text("Keep places").tag("stable") + }.labelsHidden().pickerStyle(.segmented).frame(width: 250, alignment: .trailing) + } } NativeSettingsCard("Shape") { NativeSettingsRow("Side style", detail: "Classic keeps a compact monochrome rail with its handle below.") { @@ -219,6 +240,44 @@ private struct WidgetGeneralSettings: View { NativeSettingsToggle( "Changed files", detail: "Show a session's working changes alongside its conversation.", isOn: boolean("showChanges", prefs?.showChanges ?? true)) + Divider() + NativeSettingsRow("Reply box", detail: "Where a session's inbox puts the reply box.") { + Picker("Reply box", selection: string("inboxComposer", prefs?.inboxComposer ?? "top")) { + Text("Under the tabs").tag("top") + Text("Below the list").tag("bottom") + }.labelsHidden().pickerStyle(.segmented).frame(width: 250, alignment: .trailing) + } + Divider() + NativeSettingsRow("Decision click", detail: "Stage keeps your pick to send later with the others, one message per session.") { + Picker("Decision click", selection: string("decisionClick", prefs?.decisionClick ?? "stage")) { + Text("Stage").tag("stage") + Text("Send at once").tag("send") + }.labelsHidden().pickerStyle(.segmented).frame(width: 250, alignment: .trailing) + } + Divider() + NativeSettingsRow("Closing with staged decisions", detail: "What happens to staged decisions when the inbox closes.") { + Picker("Closing with staged decisions", selection: string("stagedOnLeave", prefs?.stagedOnLeave ?? "ask")) { + Text("Ask").tag("ask") + Text("Keep staged").tag("keep") + Text("Send them").tag("send") + }.labelsHidden().pickerStyle(.segmented).frame(width: 250, alignment: .trailing) + } + Divider() + NativeSettingsRow("Inbox of every session", detail: "The inbox button's list: grouped under each session, or one list.") { + Picker("Inbox of every session", selection: string("inboxGrouping", prefs?.inboxGrouping ?? "session")) { + Text("By session").tag("session") + Text("One list").tag("list") + }.labelsHidden().pickerStyle(.segmented).frame(width: 250, alignment: .trailing) + } + openingFilters + Divider() + NativeSettingsRow("Reply box microphone", detail: "Both put the words in the reply box as you talk; a voice note also keeps the recording in Voice Notes.") { + Picker("Reply box microphone", selection: string("composerVoice", prefs?.composerVoice ?? "ask")) { + Text("Ask each time").tag("ask") + Text("Voice note").tag("note") + Text("Dictate").tag("live") + }.labelsHidden().pickerStyle(.segmented).frame(width: 300, alignment: .trailing) + } } NativeSettingsCard("Alerts", subtitle: "When an agent asks or answers while the widget is closed.") { NativeSettingsToggle( @@ -229,26 +288,74 @@ private struct WidgetGeneralSettings: View { "Answers and messages", detail: "A new logged answer or a message an agent sent you.", identifier: "widget.alertAnswers", isOn: boolean("alertAnswers", prefs?.alertAnswers ?? true)) Divider() - NativeSettingsRow("Sound", detail: "One chime per burst of new items.") { - HStack(spacing: 6) { - Picker("Alert sound", selection: string("alertSound", prefs?.alertSound ?? WidgetAlerts.defaultSound)) { - Text("None").tag("none") - ForEach(WidgetAlerts.sounds, id: \.self) { Text($0).tag($0) } - }.labelsHidden().frame(width: 140, alignment: .trailing) - IconButton(systemName: "speaker.wave.2", tooltip: "Play the alert sound") { - WidgetAlerts.preview(prefs?.alertSound ?? WidgetAlerts.defaultSound) - }.disabled((prefs?.alertSound ?? WidgetAlerts.defaultSound) == "none") - } + NativeSettingsRow("Sound for decisions", detail: "Decisions and questions that wait for you. One chime per burst.") { + soundPicker("alertSound", prefs?.alertSound ?? WidgetAlerts.defaultSound, same: false) + } + Divider() + NativeSettingsRow("Sound for answers", detail: "Answers and messages, when nothing in the burst waits for you.") { + soundPicker("alertAnswerSound", prefs?.alertAnswerSound ?? "same", same: true) } Divider() NativeSettingsToggle( - "Popup", detail: "The widget opens a short preview of the new item; a click on it answers.", + "Open the widget", detail: "The widget opens a short preview of the new item; a click on it answers.", identifier: "widget.alertPopup", isOn: boolean("alertPopup", prefs?.alertPopup ?? true)) + Divider() + NativeSettingsRow("Flash the session", detail: "Its rail bubble and its row flash, so you see which session asked.") { + Picker( + "Flash the session", + selection: Binding( + get: { Int(WidgetAlerts.flashSeconds(prefs)) }, + set: { patch(["alertFlashSeconds": .number(Double($0))]) }) + ) { + Text("Off").tag(0) + Text("3 seconds").tag(3) + Text("5 seconds").tag(5) + Text("10 seconds").tag(10) + Text("20 seconds").tag(20) + }.labelsHidden().frame(width: 150, alignment: .trailing) + } } + .nativeSettingsAnchor("alerts") + WidgetAgentHooksCard(bridge: model.bridge) } }.task { model.refreshSettings() } } + /// A system sound picker with a play button. `same` adds "Same as decisions", stored as "same". + private func soundPicker(_ key: String, _ value: String, same: Bool) -> some View { + let played = value == "same" ? prefs?.alertSound ?? WidgetAlerts.defaultSound : value + return HStack(spacing: 6) { + Picker(key == "alertSound" ? "Decision sound" : "Answer sound", selection: string(key, value)) { + if same { Text("Same as decisions").tag("same") } + Text("None").tag("none") + ForEach(WidgetAlerts.sounds, id: \.self) { Text($0).tag($0) } + }.labelsHidden().frame(width: 160, alignment: .trailing) + IconButton(systemName: "speaker.wave.2", tooltip: "Play this sound") { + WidgetAlerts.preview(played) + }.disabled(played == "none") + } + } + + /// The filter each inbox list opens with. + @ViewBuilder private var openingFilters: some View { + Divider() + NativeSettingsRow("Inbox of every session opens with", detail: "The filter the inbox button's list shows first.") { + Picker("Inbox of every session opens with", selection: string("overviewFilter", prefs?.overviewFilter ?? "all")) { + Text("All").tag("all") + Text("Needs you").tag("needs") + Text("Unread").tag("unread") + }.labelsHidden().pickerStyle(.segmented).frame(width: 250, alignment: .trailing) + } + Divider() + NativeSettingsRow("Session inbox opens with", detail: "The filter a session's inbox shows first.") { + Picker("Session inbox opens with", selection: string("inboxFilter", prefs?.inboxFilter ?? "all")) { + Text("All").tag("all") + Text("Inbox entries").tag("inbox") + Text("Decisions").tag("decisions") + }.labelsHidden().pickerStyle(.segmented).frame(width: 300, alignment: .trailing) + } + } + private func string(_ key: String, _ value: String) -> Binding { Binding(get: { value }, set: { patch([key: .string($0)]) }) } @@ -303,6 +410,7 @@ private struct WidgetSessionSettings: View { var body: some View { VStack(alignment: .leading, spacing: 18) { + previewCard NativeSettingsCard("Your sessions", subtitle: "Pin a session to keep it in the widget's rail. Click one to open it in the widget.") { WidgetSessionBrowser(model: model, onOpen: openSession) } @@ -311,6 +419,22 @@ private struct WidgetSessionSettings: View { }.task { model.refreshSettings() } } + /// The agents hover list. Its settings button opens this page at `#inbox-preview`, and the card pulses. + private var previewCard: some View { + NativeSettingsCard("Inbox preview", subtitle: "The list that opens when the pointer rests on the agents button.") { + NativeSettingsSegmented( + "Show", identifier: "widget.preview.filter", + options: WidgetPreviewChoices.filters, + selection: Binding(get: { model.previewFilter }, set: { model.setPreview(filter: $0) }), + minSegmentWidth: 96) + NativeSettingsPicker( + "Order", identifier: "widget.preview.sort", + options: WidgetPreviewChoices.sorts, + selection: Binding(get: { model.previewSort }, set: { model.setPreview(sort: $0) })) + } + .nativeSettingsAnchor("inbox-preview") + } + private var filterCard: some View { let selected = prefs?.sessions ?? [] let projects = prefs?.projects ?? [] @@ -678,6 +802,78 @@ private struct WidgetDictationSettings: View { } } +/// What the Agents card reads from `tools agents hooks doctor --json`. +private struct AgentHooksDoctor: Decodable { + struct Config: Decodable { + struct Decisions: Decodable { var questionNudge: String? } + var decisions: Decisions + } + var config: Config + var stopHookWired: Bool +} + +/// The agents hooks' question nudge (`decisions.questionNudge` in `~/.genesis-tools/agents/hooks.json`), read and +/// written through `tools agents hooks`. On writes `block`; a `warn` set from the CLI shows as on. +private struct WidgetAgentHooksCard: View { + let bridge: ToolsBridge + @State private var nudge: String? + @State private var stopWired = true + @State private var busy = false + @State private var failure: String? + + var body: some View { + NativeSettingsCard("Agents", subtitle: "Checks that run when a coding agent ends its turn.") { + NativeSettingsToggle( + "Remind agents to log decisions", + detail: "An agent that asks you to decide in its reply, without sending the decision to your inbox, is sent back once to send it.", + identifier: "widget.agentsQuestionNudge", + isOn: Binding( + get: { (nudge ?? "off") != "off" }, + set: { on in Task { await run(["config", "set", "decisions.questionNudge", on ? "block" : "off", "--write"]) } })) + .disabled(nudge == nil || busy) + if (nudge ?? "off") != "off" && !stopWired { + Divider() + NativeSettingsRow("Stop hook not installed", detail: "The reminder runs in the agents' Stop hook, which is not in your Claude Code settings yet.") { + Button("Install") { Task { await run(["install", "--write"]) } } + .disabled(busy).nativeSettingsPointer() + } + } + if let failure { + Text(failure).font(.caption).foregroundStyle(.orange).textSelection(.enabled) + .frame(maxWidth: .infinity, alignment: .leading) + } + } + .task { await load() } + } + + private func load() async { + do { + let result = try await bridge.run(subcommand: "agents", args: ["hooks", "doctor", "--json"], timeoutSeconds: 20) + guard result.exitCode == 0 else { throw ToolsBridgeError.refused(result.stderr) } + let doctor = try JSONDecoder().decode(AgentHooksDoctor.self, from: Data(result.stdout.utf8)) + nudge = doctor.config.decisions.questionNudge ?? "off" + stopWired = doctor.stopHookWired + failure = nil + } catch { + if !Task.isCancelled { failure = "Could not read the agents hooks: \(error.localizedDescription)" } + } + } + + /// One `tools agents hooks …` write, then a fresh read so the card shows what the file now holds. + private func run(_ args: [String]) async { + busy = true + defer { busy = false } + do { + let result = try await bridge.run(subcommand: "agents", args: ["hooks"] + args, timeoutSeconds: 30) + guard result.exitCode == 0 else { throw ToolsBridgeError.refused(result.stderr) } + } catch { + failure = error.localizedDescription + return + } + await load() + } +} + /// Stands in for the controls until the first snapshot with the stored preferences arrives, so a page never shows /// the defaults as if they were your settings. A failed load offers a retry. private struct WidgetSettingsLoading: View { @@ -721,3 +917,18 @@ private struct WidgetSettingsError: View { } } } + +/// The hover list's filter and order choices, shared by the settings card and the list's own controls. +enum WidgetPreviewChoices { + static let filters: [NativeSettingsOption] = [ + .init("needs-you", "Needs you", detail: "Sessions with a question or decision waiting for your answer."), + .init("active", "Active too", detail: "Also the sessions that work now or hold unread answers."), + ] + static let sorts: [NativeSettingsOption] = [ + .init("attention", "Waiting first", detail: "Sessions that wait for you first, then working ones, newest first."), + .init("recent", "Newest activity", detail: "The session that did something last comes first."), + .init("project", "By project", detail: "Grouped by project name, newest first inside a project."), + ] + + static func title(sort: String) -> String { sorts.first { $0.value == sort }?.title ?? sort } +} diff --git a/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetHostView.swift b/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetHostView.swift index 04a118c664..15d1879264 100644 --- a/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetHostView.swift +++ b/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetHostView.swift @@ -58,8 +58,11 @@ struct WidgetHostView: View { joined: joined) } + /// Side panels pin their content to the top: content taller than its window (a measurement still on its way, a + /// screen too short) then loses its footer, never its header. Centred, the hover list opened with its header and + /// its last row both cut off (Martin's screenshot, 2026-10-23). private var edgeAlignment: Alignment { - surface.edge == .top ? .top : (surface.edge == .right ? .trailing : .leading) + surface.edge == .top ? .top : (surface.edge == .right ? .topTrailing : .topLeading) } private var topPadding: CGFloat { @@ -69,6 +72,7 @@ struct WidgetHostView: View { private var keepsExpandedPane: Bool { warm && selected?.id == "agents" } var body: some View { + let _ = RenderProbe.hit("widget.host") Color.clear .overlay(alignment: .topLeading) { panelLayers } // The side rail anchors itself to the screen (ScreenAnchoredWidgetRail), so it needs the whole window @@ -90,7 +94,8 @@ struct WidgetHostView: View { ) .nativeSettingsAppearance(model.appearance) .widgetAccessibility(reduceMotion: model.reduceMotion, reduceTransparency: model.reduceTransparency) - .onHover { model.hover(surface, inside: $0) } + // The panel's window tracks the pointer (EdgePanelController.pointerChanged); this only logs a disagreement. + .onHover { model.swiftUIHoverChanged?(surface, $0) } .onChange(of: motion.pointerInside) { _, inside in if inside { warm = true } } @@ -132,7 +137,7 @@ struct WidgetHostView: View { } }) { Button { - if selected?.id == "agents", model.inboxCount > 0 { model.openInboxNotification(on: surface) } + if selected?.id == "agents" { model.toggleInboxOverview(on: surface) } else { expand() } } label: { HStack(spacing: 7) { @@ -162,12 +167,13 @@ struct WidgetHostView: View { HStack(spacing: 5) { ForEach(Array(model.railActivitySessions.prefix(3))) { session in Button { - model.openInboxNotification(on: surface, key: session.key) + model.toggleSession(session.key, on: surface) } label: { sessionActivity(session) - .frame(minWidth: 24, minHeight: 30).fixedSize() - }.buttonStyle(.genHoverPlain()) - .instantTooltip(sessionTooltip(session)) + .frame(minWidth: 30, minHeight: 30).fixedSize() + .modifier(bubbleChrome(session)) + }.buttonStyle(.plain) + .instantTooltip(model.bubbleTooltip(sessionTooltip(session))) .accessibilityLabel(sessionSummary(session)) .accessibilityIdentifier("widget.agent." + session.key) .onHover { model.hoverSession(session.key, on: surface, inside: $0) } @@ -270,18 +276,19 @@ struct WidgetHostView: View { VStack(spacing: WidgetSideStripMetrics.sessionSpacing) { ForEach(Array(model.railActivitySessions.prefix(sideMetrics.sessionCount))) { session in Button { - model.openInboxNotification(on: surface, key: session.key) + model.toggleSession(session.key, on: surface) } label: { sessionActivity(session) .frame(width: WidgetSideStripMetrics.sessionWidth, height: WidgetSideStripMetrics.sessionHeight) + .modifier(bubbleChrome(session)) .overlay(alignment: .bottomTrailing) { // Which agent the count belongs to, at a glance; the tooltip names the session. AIProviderGlyph(meta: AIProviders.meta(for: session.target.provider), size: 11) .opacity(0.85).allowsHitTesting(false) } } - .buttonStyle(.genHoverPlain()) - .instantTooltip(sessionTooltip(session)) + .buttonStyle(.plain) + .instantTooltip(model.bubbleTooltip(sessionTooltip(session))) .accessibilityLabel(sessionSummary(session)) .accessibilityIdentifier("widget.agent." + session.key) .onHover { model.hoverSession(session.key, on: surface, inside: $0) } @@ -368,7 +375,9 @@ struct WidgetHostView: View { } } .frame(width: selected.previewSize.width) - .frame(maxHeight: .infinity) + // Pinned to the top: while the window still has the last measured height, an overflow hides the footer, never + // the header (the hover list opened on its middle rows, 2026-10-23). + .frame(maxHeight: .infinity, alignment: .top) } /// Inbox counts describe only the Agents module; every other module button names its module. @@ -378,7 +387,7 @@ struct WidgetHostView: View { private func moduleButton(_ id: String, size: CGFloat, location: String = "content") -> some View { Button { - if id == "agents", model.inboxCount > 0 { model.openInboxNotification(on: surface) } + if id == "agents" { model.toggleInboxOverview(on: surface) } else { model.openModule(id, on: surface) } } label: { let layout = surface.edge == .top ? AnyLayout(HStackLayout(spacing: 3)) : AnyLayout(VStackLayout(spacing: 4)) @@ -405,6 +414,14 @@ struct WidgetHostView: View { .accessibilityIdentifier("widget." + location + "." + surface.key + "." + id) } + /// The bubble's hover ring and the alert flash, in place of the rectangular hover plate of `.genHoverPlain`. + private func bubbleChrome(_ session: WidgetSession) -> WidgetBubbleChrome { + var style = model.attentionStyle + style.cornerRadius = 17 + return WidgetBubbleChrome(trigger: model.attentionTrigger(session.key), attention: style, + reduceMotion: model.effectiveReduceMotion, since: model.attentionStart(session.key)) + } + private func sessionActivity(_ session: WidgetSession) -> some View { let inbox = model.inboxFor(session.key) return WidgetSessionActivity(status: session.visualStatus, @@ -422,7 +439,7 @@ struct WidgetHostView: View { if inbox.needsAnswer > 0 { lines.append("\(inbox.needsAnswer) waiting for your answer") } if inbox.unread > 0 { lines.append("\(inbox.unread) unread") } } - lines.append("Click to open its inbox") + lines.append("Click to open its inbox, again to hide it") return TooltipContent(title: session.title, bullets: lines) } @@ -480,7 +497,7 @@ struct WidgetPeekView: View { LiveAgo(date: Date(timeIntervalSince1970: item.at / 1000), style: .brief) .font(.system(size: 10)).foregroundStyle(.secondary) } - Text(item.title ?? "Open the inbox to see it.").font(.system(size: 12.5, weight: .medium)) + Text(item.numberedTitle ?? "Open the inbox to see it.").font(.system(size: 12.5, weight: .medium)) .lineLimit(3).multilineTextAlignment(.leading).fixedSize(horizontal: false, vertical: true) if let session = model.session(forKey: item.key) { Text(session.title + " · " + session.project).font(.system(size: 10.5)).foregroundStyle(.secondary) @@ -493,7 +510,14 @@ struct WidgetPeekView: View { .contentShape(Rectangle()) } .buttonStyle(RowButtonStyle(cornerRadius: 10)) - .accessibilityLabel(WidgetAlerts.kicker(item) + ": " + (item.title ?? "")) + .attentionPulse(trigger: model.attentionTrigger(item.key), style: popupAttention, since: model.attentionStart(item.key)) + .accessibilityLabel(WidgetAlerts.kicker(item) + ": " + (item.numberedTitle ?? "")) + } + + private var popupAttention: AttentionPulseStyle { + var style = model.attentionStyle + style.cornerRadius = 10 + return style } } @@ -511,7 +535,7 @@ struct AgentWidgetModuleView: View { Text(model.selected?.title ?? "Your agent inbox").font(.system(size: 13, weight: .semibold)) .lineLimit(2) if let card = model.card { - Text(card.title).font(.system(size: 12)).lineLimit(3) + Text(card.numberedTitle).font(.system(size: 12)).lineLimit(3) if !card.body.isEmpty { Text(card.body).font(.system(size: 11)).foregroundStyle(.secondary).lineLimit(3) } diff --git a/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetInboxOverview.swift b/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetInboxOverview.swift new file mode 100644 index 0000000000..7d9a40cb03 --- /dev/null +++ b/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetInboxOverview.swift @@ -0,0 +1,274 @@ +import SwiftUI + +/// The inbox button's view: inbox entries from every session in one list, the waiting ones first. A row opens in place +/// to show its session and the way to that session's whole inbox (with the entry open) and its cmux pane. +struct WidgetInboxOverview: View { + @ObservedObject var model: WidgetModel + /// The filter picked here; until then the one the `overviewFilter` setting opens with. + @State private var filterPick: String? + @State private var openID: String? + @GenReduceMotion private var reduceMotion + + private var filter: String { filterPick ?? model.snapshot?.state.preferences.overviewFilter ?? "all" } + + private var items: [WidgetInboxItem] { + let all = model.inboxOverviewItems + switch filter { + case "needs": return all.filter(\.needsAnswer) + case "unread": return all.filter { !$0.needsAnswer } + default: return all + } + } + + /// Entries under their session ("session", the default) or one list (Widgets → Inbox, and the header button). + private var grouped: Bool { (model.snapshot?.state.preferences.inboxGrouping ?? "session") == "session" } + + var body: some View { + let items = items + let _ = RenderProbe.hit("widget.overview") + VStack(alignment: .leading, spacing: 10) { + HStack(spacing: 8) { + GenSegmentedTabs( + "Inbox entries", + items: [.init("all", "All"), .init("needs", "Needs you"), .init("unread", "Unread")], + selection: Binding(get: { filter }, set: { value in + model.holdAgentFit() + filterPick = value + }), minSegmentWidth: 70, identifier: "widget.overview.filter") + Spacer(minLength: 4) + IconButton(systemName: grouped ? "list.bullet" : "rectangle.3.group", + tooltip: grouped ? "Show one list" : "Group the entries by session") { + model.holdAgentFit() + model.updatePreferences(["inboxGrouping": .string(grouped ? "list" : "session")]) + }.accessibilityLabel(grouped ? "Show one list" : "Group by session") + if model.inboxCount > 0 { + Button("Mark all read", action: model.markAllRead) + .buttonStyle(.genHover()).font(.system(size: 11, weight: .medium)) + .instantTooltip("Answers become read; older questions and decisions stop counting. Nothing is deleted.") + } + } + if items.isEmpty { + VStack(spacing: 10) { + GenesisWidgetMark() + Text(filter == "needs" ? "Nothing waits for your answer." : "Nothing new in the inbox.") + .font(.headline) + Text("Questions, screenshots and finished work from every session land here.") + .font(.caption).foregroundStyle(.secondary) + }.frame(maxWidth: .infinity).padding(.vertical, 28) + } else { + Text(Self.summary(items)).font(.system(size: 11)).foregroundStyle(.secondary) + // Hundreds of entries: lazy, so only the rows on screen exist (swift skill, dense lists). + LazyVStack(alignment: .leading, spacing: 10) { + if grouped { + ForEach(Self.groups(items), id: \.key) { group in + Section { + ForEach(group.items, id: \.id) { item in row(item) } + } header: { + header(group.key, items: group.items) + } + } + } else { + ForEach(items, id: \.id) { item in row(item) } + } + } + if model.inbox.truncated { + Text("The newest entries are shown. Open a session for its whole inbox.") + .font(.caption2).foregroundStyle(.tertiary) + } + } + } + } + + /// The sessions in the order their first entry appears (the waiting ones first), each with its entries. + static func groups(_ items: [WidgetInboxItem]) -> [(key: String, items: [WidgetInboxItem])] { + var order: [String] = [] + var byKey: [String: [WidgetInboxItem]] = [:] + for item in items { + if byKey[item.key] == nil { order.append(item.key) } + byKey[item.key, default: []].append(item) + } + return order.map { ($0, byKey[$0] ?? []) } + } + + private func header(_ key: String, items: [WidgetInboxItem]) -> some View { + let session = model.session(forKey: key) + let waiting = items.filter(\.needsAnswer).count + return HStack(spacing: 7) { + if let session { + AIProviderGlyph(meta: AIProviders.meta(for: session.target.provider), size: 14) + } + Text(session?.title ?? "Unknown session").font(.system(size: 11.5, weight: .semibold)).lineLimit(1) + Text([Plural.count(items.count, "entry", "entries"), waiting > 0 ? "\(waiting) need you" : nil] + .compactMap { $0 }.joined(separator: " · ")) + .font(.system(size: 10)).foregroundStyle(waiting > 0 ? Color.orange : .secondary) + Spacer(minLength: 0) + if let session { + Button("Whole inbox") { + model.select(session.key) + model.section = "Inbox" + }.buttonStyle(.genHoverPlain()).font(.system(size: 10.5)) + } + } + .padding(.top, 6).padding(.horizontal, 2) + .accessibilityElement(children: .combine) + } + + private static func summary(_ items: [WidgetInboxItem]) -> String { + let waiting = items.filter(\.needsAnswer).count + let sessions = Set(items.map(\.key)).count + return [Plural.count(items.count, "entry", "entries"), waiting > 0 ? "\(waiting) need your answer" : nil, + "from " + Plural.count(sessions, "session")].compactMap { $0 }.joined(separator: " · ") + } + + private func toggle(_ item: WidgetInboxItem) { + let opening = openID != item.id + model.releaseAgentFit() + GenMotion.perform(expanding: opening, reduceMotion: reduceMotion) { openID = opening ? item.id : nil } + } + + private func row(_ item: WidgetInboxItem) -> some View { + let open = openID == item.id + let session = model.session(forKey: item.key) + return VStack(alignment: .leading, spacing: 0) { + Button { toggle(item) } label: { + HStack(alignment: .top, spacing: 9) { + Circle().fill(item.needsAnswer ? Color.orange : SessionPalette.blue) + .frame(width: 7, height: 7).padding(.top, 5) + Image(systemName: LiveWidgetView.kindSymbol(item.kind)).font(.system(size: 11)) + .foregroundStyle(.secondary).frame(width: 14).padding(.top, 1) + VStack(alignment: .leading, spacing: 3) { + Text(Self.title(item)).font(.system(size: 12.5, weight: open ? .semibold : .medium)) + .lineLimit(open ? 6 : 2).fixedSize(horizontal: false, vertical: true) + if let staged = stagedLine(item) { + Text("Staged: " + staged).font(.system(size: 11)).foregroundStyle(SessionPalette.blue) + .lineLimit(1) + } + HStack(spacing: 4) { + Text(Self.kindLabel(item)) + Text("·") + // Under its session header the row does not repeat the session. + if !grouped { + Text(session?.title ?? "Unknown session").lineLimit(1) + Text("·") + } + LiveAgo(date: Date(timeIntervalSince1970: item.at / 1000), style: .brief) + }.font(.system(size: 10)).foregroundStyle(item.needsAnswer ? Color.orange : .secondary) + } + Spacer(minLength: 4) + if let count = item.count, count > 1 { + Text(verbatim: "×\(count)").font(.system(size: 10, weight: .semibold)).monospacedDigit() + .foregroundStyle(.secondary).padding(.top, 3) + .instantTooltip("\(count) entries like this one in this session") + } + Image(systemName: "chevron.right").font(.system(size: 9, weight: .semibold)) + .foregroundStyle(.secondary).rotationEffect(.degrees(open ? 90 : 0)).padding(.top, 4) + } + .padding(.horizontal, 10).padding(.vertical, 9) + .frame(maxWidth: .infinity, alignment: .leading) + .contentShape(Rectangle()) + } + .buttonStyle(RowButtonStyle(cornerRadius: 11)) + .accessibilityLabel(Self.kindLabel(item) + ": " + Self.title(item)) + .accessibilityValue(open ? "Expanded" : "Collapsed") + .accessibilityIdentifier("widget.overview.row." + item.id) + if open { + detail(item, session: session).padding(.horizontal, 12).padding(.top, 2).padding(.bottom, 12) + .transition(.genReveal) + } + } + .clipShape(RoundedRectangle(cornerRadius: 11)) + .background(Color.white.opacity(open ? 0.055 : 0.03), in: RoundedRectangle(cornerRadius: 11)) + .overlay(RoundedRectangle(cornerRadius: 11) + .strokeBorder(item.needsAnswer ? Color.orange.opacity(0.28) : .clear, lineWidth: 1)) + } + + private func stagedLine(_ item: WidgetInboxItem) -> String? { + let option = model.stagedOption(item.sourceId, stored: item.draftOption) + guard item.kind == "decision", item.needsAnswer, option != nil || item.draft != nil else { return nil } + return WidgetStagedText.line(item, option: option) + } + + /// An open decision's options, each staging (or sending, by the setting) that pick from here. + @ViewBuilder private func choices(_ item: WidgetInboxItem) -> some View { + if item.kind == "decision", item.needsAnswer, let options = item.options, !options.isEmpty { + let staged = model.stagedOption(item.sourceId, stored: item.draftOption) + VStack(alignment: .leading, spacing: 6) { + ForEach(Array(options.enumerated()), id: \.offset) { index, title in + let letter = String(UnicodeScalar(97 + index).map(Character.init) ?? "?") + let picked = staged == letter + Button { model.choose(item, option: letter) } label: { + HStack(alignment: .top, spacing: 8) { + Text(verbatim: letter).font(.system(size: 10.5, weight: .semibold, design: .rounded)) + .frame(width: 20, height: 20).background(.white.opacity(0.07), in: RoundedRectangle(cornerRadius: 5)) + Text(title).font(.system(size: 12)).fixedSize(horizontal: false, vertical: true).padding(.top, 2) + Spacer(minLength: 0) + if picked { + Text("Staged").font(.system(size: 10, weight: .semibold)).foregroundStyle(SessionPalette.blue) + .padding(.top, 3) + } + if item.recommended == letter { Image(systemName: "sparkle").foregroundStyle(.orange) } + } + .padding(8).frame(maxWidth: .infinity, alignment: .leading) + .background(picked ? SessionPalette.blue.opacity(0.16) : .white.opacity(0.06), + in: RoundedRectangle(cornerRadius: 8)) + .overlay(RoundedRectangle(cornerRadius: 8) + .strokeBorder(picked ? SessionPalette.blue.opacity(0.7) : .clear, lineWidth: 1)) + .contentShape(RoundedRectangle(cornerRadius: 8)) + } + .buttonStyle(RowButtonStyle(cornerRadius: 8)) + .instantTooltip(LiveWidgetView.choiceTooltip( + WidgetChoice(id: letter, title: title, recommended: item.recommended == letter), + index: 10, stages: model.decisionClickStages, picked: picked)) + .accessibilityLabel((model.decisionClickStages ? "Stage " : "Answer ") + letter + ": " + title) + .accessibilityValue(picked ? "Staged" : "") + } + } + } + } + + private func detail(_ item: WidgetInboxItem, session: WidgetSession?) -> some View { + VStack(alignment: .leading, spacing: 10) { + choices(item) + if let session { + HStack(spacing: 8) { + AIProviderGlyph(meta: AIProviders.meta(for: session.target.provider), size: 18) + VStack(alignment: .leading, spacing: 2) { + Text(session.title).font(.system(size: 11.5, weight: .medium)).lineLimit(1) + Text([session.project, session.visualStatus.label, String(session.target.sessionId.prefix(8))] + .filter { !$0.isEmpty }.joined(separator: " · ")) + .font(.system(size: 10)).foregroundStyle(.secondary).lineLimit(1) + } + } + } + HStack(spacing: 8) { + Button(item.needsAnswer ? "Answer in its session" : "Open in its session") { model.openInboxItem(item) } + .buttonStyle(.genHover(accent: item.needsAnswer ? .orange : KitTheme.accent)) + .accessibilityIdentifier("widget.overview.open." + item.id) + if let session { + Button("Whole inbox") { + model.select(session.key) + model.section = "Inbox" + }.buttonStyle(.genHover()) + Spacer(minLength: 0) + WidgetCmuxButton(model: model, session: session, size: 12) + } + }.font(.caption) + } + } + + static func title(_ item: WidgetInboxItem) -> String { + if let title = item.numberedTitle, item.title?.isEmpty == false { + return title + } + return kindLabel(item) + } + + static func kindLabel(_ item: WidgetInboxItem) -> String { + if item.needsAnswer { return item.kind == "decision" ? "Decision · needs your answer" : "Needs your answer" } + switch item.kind { + case "answer": return "New answer" + case "result": return "Agent result" + default: return item.kind.capitalized + } + } +} diff --git a/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetModel+Staging.swift b/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetModel+Staging.swift new file mode 100644 index 0000000000..d1499c890f --- /dev/null +++ b/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetModel+Staging.swift @@ -0,0 +1,264 @@ +import Foundation + +/// Staged decisions: a pick or a note kept to send later, together with others (Martin, 2026-10-23: "click on the +/// decision option should not send it immediatelly, but stage that change"). They are the decision log's own unsent +/// state (`drafted` with `draftOption` or `draft`), so every inbox, the hub and the CLI see one list, and nothing is +/// lost when the widget restarts. +extension WidgetModel { + /// A click on a decision option or its number key stages it ("stage", the default) or sends it at once ("send"). + public var decisionClickStages: Bool { (snapshot?.state.preferences.decisionClick ?? "stage") == "stage" } + + /// The pick a card shows as staged: the one staged here a moment ago, else the snapshot's. + public func stagedOption(_ sourceId: String, stored: String?) -> String? { + if let pending = pendingStage[sourceId] { return pending.isEmpty ? nil : pending } + return stored + } + + /// The selected session's staged decisions. + public var sessionStaged: [WidgetInboxItem] { stagedItems.filter { $0.key == selectedKey } } + + /// A click on a decision option in a session's inbox. A second click on the staged option clears the pick. + public func choose(_ card: WidgetCard, option: String) { + guard decisionClickStages else { + // `submit` answers the selected card; another open row's click must answer its own decision. + selectedCardID = card.id + submit(choice: option) + return + } + guard let selected, let revision = card.revision else { return } + let current = stagedOption(card.sourceId, stored: card.draftOption) + stage(id: card.sourceId, sessionId: selected.target.sessionId, revision: revision, + option: current == option ? nil : option) + } + + /// The number keys: the open decision's option, as a click on it. + public func choose(option: String) { + guard let card = answerCard else { + submit(choice: option) + return + } + choose(card, option: option) + } + + /// A click on a decision option in the inbox of every session. With "Send at once" the pick is staged on the + /// item's own session and sent once the hub has it. + public func choose(_ item: WidgetInboxItem, option: String) { + guard let session = session(forKey: item.key), let revision = item.revision else { return } + guard decisionClickStages else { + stage(id: item.sourceId, sessionId: session.target.sessionId, revision: revision, option: option, + completed: { [weak self] in + self?.pendingStage[item.sourceId] = nil + self?.deliverDecisions([item], action: "send-staged") + }) + return + } + let current = stagedOption(item.sourceId, stored: item.draftOption) + stage(id: item.sourceId, sessionId: session.target.sessionId, revision: revision, + option: current == option ? nil : option) + } + + /// Save draft on a decision card: the composer's text becomes that decision's note, staged with the others. Once + /// the hub has it, the composer is cleared, so another decision's own answer does not open with this text; a + /// refused save keeps the text. + public func saveDraft(_ card: WidgetCard) { + guard let selected, let revision = card.revision else { return } + let key = selectedKey + let typed = draft.text + let text = typed.trimmingCharacters(in: .whitespacesAndNewlines) + stage(id: card.sourceId, sessionId: selected.target.sessionId, revision: revision, + option: stagedOption(card.sourceId, stored: card.draftOption), note: text, + completed: { [weak self] in + guard let self else { return } + self.draftSavedAt[card.id] = Date() + // Only the text that was saved: a session switch or new typing keeps the composer as it is. + if !text.isEmpty, self.selectedKey == key, self.draft.text == typed { self.setText("") } + }) + } + + /// The own answer composer of a decision with a saved note opens with that note. + public func openOwnAnswer(_ card: WidgetCard) { + ownAnswerCardID = card.id + if draft.text.isEmpty, let note = card.draft, !note.isEmpty { setText(note) } + } + + private func stage( + id: String, sessionId: String, revision: Int, option: String?, note: String? = nil, + completed: (() -> Void)? = nil + ) { + pendingStage[id] = option ?? "" + var fields: [String: WidgetJSON] = [ + "action": "ledger", "id": .string(id), "sessionId": .string(sessionId), + "expectedRevision": .number(Double(revision)), "state": "drafted", "draftOption": .string(option ?? ""), + ] + if let note { fields["draft"] = .string(note) } + action(.object(fields), failed: { [weak self] in self?.pendingStage[id] = nil }, completed: completed) + } + + /// The staged items as the snapshot shows them, and the picks staged here that it now shows. + func reconcileStaged(_ items: [WidgetInboxItem]) { + let staged = items.filter(\.isStaged) + if staged != stagedItems { stagedItems = staged } + guard !pendingStage.isEmpty else { return } + let shown = Dictionary(items.map { ($0.sourceId, $0.draftOption ?? "") }, uniquingKeysWith: { first, _ in first }) + // A decision the list no longer holds shows no pick, so a pending unstage ("") is done. + for (id, option) in pendingStage where (shown[id] ?? "") == option { + pendingStage[id] = nil + } + } + + /// "3 decisions will be sent to 2 sessions". + nonisolated public static func stagedSummary(_ items: [WidgetInboxItem], future: Bool = true) -> String { + let sessions = Set(items.map(\.key)).count + let verb = future ? "will be sent" : (items.count == 1 ? "was sent" : "were sent") + return Plural.count(items.count, "decision") + " \(verb) to " + Plural.count(sessions, "session") + } + + /// Sends the staged decisions `ids`: one message per session. + public func sendStaged(_ ids: [String]) { + let chosen = stagedItems.filter { ids.contains($0.sourceId) } + guard !chosen.isEmpty else { return } + deliverDecisions(chosen, action: "send-staged") + } + + /// Sends again answered decisions that no session took (the notice's Send again), for every kind of agent. + public func resendAnswers(_ items: [WidgetInboxItem]) { + guard !items.isEmpty else { return } + notice = "Sending again…" + deliverDecisions(items, action: "resend-answers") + } + + /// One message per session through the hub, then a notice that says what really happened: delivered, waiting + /// for the session's next prompt (with why, and Send again), or not sent at all. It waits for the hub writes + /// queued before it, so a pick changed a moment ago is the one that goes. + private func deliverDecisions(_ items: [WidgetInboxItem], action: String) { + var targets: [String: WidgetJSON] = [:] + var titles: [String: String] = [:] + for item in items { + guard let session = session(forKey: item.key) else { continue } + let target = session.target + var fields: [String: WidgetJSON] = ["provider": .string(target.provider)] + if !target.sourceHome.isEmpty { fields["sourceHome"] = .string(target.sourceHome) } + if !target.cwd.isEmpty { fields["cwd"] = .string(target.cwd) } + targets[target.sessionId] = .object(fields) + titles[target.sessionId] = session.title + } + afterMutations { [weak self] in + guard let self else { return } + do { + let result = try await self.call([ + "action": .string(action), "ids": .array(items.map { .string($0.sourceId) }), + "targets": .object(targets), + ]) + let outcome = WidgetDecisionSendOutcome(result, titles: titles) + self.notice = outcome.notice(items: items, staged: action == "send-staged") + let waiting = items.filter { item in + self.session(forKey: item.key).map { outcome.queuedSessions.contains($0.target.sessionId) } ?? false + } + if !waiting.isEmpty { + self.noticeAction = WidgetNoticeAction(title: "Send again") { [weak self] in self?.resendAnswers(waiting) } + } + } catch { + if action == "resend-answers" { self.notice = nil } + self.report(error) + } + } + } + + /// Discards the picks and notes of the staged decisions `ids`; they still wait for an answer. + public func discardStaged(_ ids: [String]) { + guard !ids.isEmpty else { return } + for id in ids { pendingStage[id] = "" } + action(["action": "unstage", "ids": .array(ids.map { .string($0) })], failed: { [weak self] in + for id in ids { self?.pendingStage[id] = nil } + }) + } + + /// A close the user asked for (Esc, the close button, a click outside, the idle timer): with staged decisions it + /// asks first, as `stagedOnLeave` says ("ask" by default; "keep" closes, "send" sends them and closes). + public func requestCollapse() { + let ids = Set(stagedItems.map(\.sourceId)) + guard expanded != nil, !ids.isEmpty else { + collapse() + return + } + switch snapshot?.state.preferences.stagedOnLeave ?? "ask" { + case "keep": + collapse() + case "send": + sendStaged(Array(ids)) + collapse() + default: + guard !ids.isSubset(of: keptStaged) else { + collapse() + return + } + stagedLeavePrompt = true + } + } + + /// The answer to the leave question: "send", "keep" or "discard". The panel closes after any of them. + public func resolveStagedLeave(_ choice: String) { + let ids = stagedItems.map(\.sourceId) + stagedLeavePrompt = false + switch choice { + case "send": sendStaged(ids) + case "discard": discardStaged(ids) + default: keptStaged = Set(ids) + } + stagedView = nil + collapse() + } +} + +/// A button beside the widget's notice line. +public struct WidgetNoticeAction { + public let title: String + public let run: @MainActor () -> Void + + public init(title: String, run: @escaping @MainActor () -> Void) { + self.title = title + self.run = run + } +} + +/// What a decision send did, read from the hub's answer: how many went out, waited or failed, and why a session +/// took nothing. +struct WidgetDecisionSendOutcome { + var sent = 0 + var queued = 0 + var failed = 0 + /// Session ids whose answers no receiver took, with the hub's one-line reason and the session's title. + var waiting: [(session: String, title: String, reason: String)] = [] + + var queuedSessions: Set { Set(waiting.map(\.session)) } + + init(_ result: WidgetJSON, titles: [String: String]) { + guard case .object(let fields) = result else { return } + if case .number(let value) = fields["sent"] { sent = Int(value) } + if case .number(let value) = fields["queued"] { queued = Int(value) } + if case .number(let value) = fields["failed"] { failed = Int(value) } + guard case .array(let sessions) = fields["sessions"] else { return } + for case .object(let entry) in sessions { + guard case .string(let session) = entry["session"], case .object(let answer) = entry["result"], + answer["delivered"] != .bool(true) else { continue } + var reason = "No live receiver took them." + if case .string(let detail) = answer["detail"], !detail.isEmpty { reason = detail } + waiting.append((session, titles[session] ?? "the session", reason)) + } + } + + /// The notice line: a plain summary when everything went out, else what did not and why. + func notice(items: [WidgetInboxItem], staged: Bool) -> String { + var parts: [String] = [] + if sent > 0 { parts.append(Plural.count(sent, "decision") + " delivered.") } + if let first = waiting.first { + let reason = first.reason.hasSuffix(".") ? first.reason : first.reason + "." + let place = waiting.count == 1 ? first.title : Plural.count(waiting.count, "session") + parts.append("Not delivered to \(place): \(reason) The answers wait for its next prompt.") + } + if failed > 0 { + parts.append("\(Plural.count(failed, "decision")) could not be sent" + (staged ? " and stay staged." : ".")) + } + return parts.isEmpty ? WidgetModel.stagedSummary(items, future: false) + "." : parts.joined(separator: " ") + } +} diff --git a/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetModel.swift b/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetModel.swift index 5613ee104b..bad90cd196 100644 --- a/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetModel.swift +++ b/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetModel.swift @@ -26,10 +26,18 @@ public final class WidgetModel: ObservableObject { /// The height the agents pane needs to show everything without scrolling (LiveWidgetView measures it), or nil /// before it was laid out. The expanded panel fits it, up to its maximum (W3). public private(set) var agentFitHeight: CGFloat? + /// Each session's cards as last seen, so a switch never waits for the next snapshot (`WidgetSessionCache`). + private(set) var sessionCache = WidgetSessionCache() + /// The session whose first load keeps the pane at its height: the loading row is short, and the pane shrank to it + /// and grew back a moment later (app-perf.log, 2026-10-24: 1004x900, 1004x321, 1004x900 on each switch). + private var loadingFitHold: String? public var railSessions: [WidgetSession] { sessionRoster.rail } /// Rail sessions that are working or hold unread items. Stored, because every layout pass of every edge panel reads /// it: filtering ~1000 sessions there cost a quarter of the main thread while panels animated. public private(set) var railActivitySessions: [WidgetSession] = [] + /// The agents hover list (`previewRows`): every session that needs you, with the "active" filter also the + /// sessions that work now. Stored with the inbox, because every hover layout reads it. + public private(set) var previewList: [WidgetSession] = [] @Published public private(set) var inbox = WidgetInboxSummary.empty @Published public private(set) var inboxPulse = 0 private var inboxSessions: [String: WidgetInboxSession] = [:] @@ -40,6 +48,9 @@ public final class WidgetModel: ObservableObject { private var openingInbox: WidgetInboxItem? private var inboxReadRequests: Set = [] @Published public var selectedKey = "" + /// The expanded agents pane shows the inbox entries of every session instead of one session (the inbox button). + /// Opening a session ends it. + @Published public var inboxOverview = false { didSet { if inboxOverview != oldValue { presentationChanged?() } } } @Published public var selectedCardID: String? { didSet { presentationChanged?() } } @Published public var section = "Inbox" { didSet { @@ -91,6 +102,10 @@ public final class WidgetModel: ObservableObject { @Published public var transcriptLoading = false @Published public var importing = 0 @Published public var voiceText = "" + /// Everything said in the running dictation: the finished sentences and the one still being recognised. + public var voiceTranscript: String { + (voiceFinals + [voiceText]).filter { !$0.isEmpty }.joined(separator: " ") + } @Published public var voiceLevel = 0.0 @Published public var voiceActive = false @Published public var reading = false @@ -109,26 +124,60 @@ public final class WidgetModel: ObservableObject { @Published private var hostDialogOpen = false /// Set by the session chooser alone, so closing it (or its view going away) cannot release a window still open. @Published public var sessionChooserOpen = false + /// Set by a session's cmux popover alone, for the same reason. + @Published public var cmuxPopoverOpen = false /// Hover previews and the outside-click collapse stay off while a window, picker or the chooser is open. public var dialogOpen: Bool { - get { hostDialogOpen || sessionChooserOpen } + get { hostDialogOpen || sessionChooserOpen || cmuxPopoverOpen } set { hostDialogOpen = newValue } } private var mediaPicker: NSOpenPanel? public var presentationChanged: (() -> Void)? /// The pointer entered or left an edge panel, before any hover delay. public var pointerChanged: ((WidgetSurfaceID, Bool) -> Void)? + /// SwiftUI's own hover of a panel, for the log only (the coordinator names what is under the pointer). + public var swiftUIHoverChanged: ((WidgetSurfaceID, Bool) -> Void)? public var showSettings: (() -> Void)? + /// Opens Settings at a page, or at a card on it (`widgets.sessions#inbox-preview`), which then pulses. + public var showSettingsPage: ((String) -> Void)? public var showMedia: ((WidgetMediaSelection) -> Void)? public var openHub: ((WidgetSession?) -> Void)? - public var recordVoiceNote: ((String) -> Void)? + /// A path for a voice note's audio; nil where no voice notes run (the preview studio), so a note is only dictated. + public var voiceNoteClip: (() async throws -> String)? + /// Counts voice starts, so a Stop during a clip-path request keeps that start from opening the microphone later. + private var voiceStart = 0 + /// Keeps a finished dictation as a voice note: its audio file, its text, and where it was spoken. + public var fileVoiceNote: ((_ clip: String, _ text: String, _ context: WidgetVoiceNoteContext) -> Void)? + /// The mode of the dictation that runs, or nil. + @Published public private(set) var voiceMode: WidgetVoiceMode? + private var voiceNotePlan: (clip: String, context: WidgetVoiceNoteContext)? + static let lastVoiceModeKey = "widget.voice.last-mode" + + /// `composerVoice`: "ask" (default) offers both modes on every press of the microphone (Martin, 2026-10-23: "i want + /// to be able to pick when i click the "record" every time"); "note" or "live" starts that one at once. + public var composerVoiceChoice: String { snapshot?.state.preferences.composerVoice ?? "ask" } + /// What a press starts without asking: the setting, or the mode picked last when it asks. + public var preferredVoiceMode: WidgetVoiceMode { + if let fixed = WidgetVoiceMode(rawValue: composerVoiceChoice) { return fixed } + return defaults.string(forKey: Self.lastVoiceModeKey).flatMap(WidgetVoiceMode.init) ?? .note + } public var voiceActionLabel: String { - voiceActive ? "Stop dictation" : (recordVoiceNote == nil ? "Dictate" : "Record voice note") + if voiceActive { return voiceMode == .note ? "Stop and keep the voice note" : "Stop dictation" } + switch composerVoiceChoice { + case "note": return "Record a voice note into the reply box" + case "live": return "Dictate into the reply box" + default: return "Dictate, or record a voice note, into the reply box" + } } public var openDestination: ((WidgetSession, String, String?) -> Void)? /// Called with each preference patch once the hub has stored it. public var preferencesSaved: (([String: WidgetJSON]) -> Void)? - @Published public var notice: String? + /// One line under the header; a new line drops the button the old one had. + @Published public var notice: String? { + didSet { if notice != oldValue { noticeAction = nil } } + } + /// A button beside `notice`: "Send again" after a send no session took. + @Published public var noticeAction: WidgetNoticeAction? public private(set) var openedAt: TimeInterval = 0 public let bridge: ToolsBridge public let appearance: NativeSettingsAppearance @@ -231,34 +280,105 @@ public final class WidgetModel: ObservableObject { /// The agents preview's measured height. The old estimate (100 + 54 per row) was 11 pt plus 2 pt per row short, /// so the centred content lost its header and the top of its first row (2026-10-10, Martin's screenshot). public private(set) var previewFitHeight: CGFloat? + /// The hover list's last measured height. The list is built again on every hover; it starts at this height so its + /// first layout is already the real one (`FittedScroll.initialHeight`). Not published: it must not re-render. + var previewListHeight: CGFloat = 0 public var previewHeight: CGFloat { if let previewFitHeight { return previewFitHeight } return sessionRoster.preview.isEmpty ? 180 : 112 + CGFloat(sessionRoster.preview.count) * 56 } /// Whole points and a 2 pt hysteresis, as `reportAgentFit`, so the measurement cannot chase its own resize. + /// + /// A taller measurement applies at once; a shorter one only after it held for `previewShrinkDelay`. The list is + /// built again on every hover, and its first layout passes measured it short (header and footer, then the rows): + /// one hover asked for 336, 295 and 489 pt within 42 ms, the panel shrank and grew back, and its content blinked + /// or opened cut at the top (edge.request log and Martin's screenshots, 2026-10-23). + /// + /// Before the first measurement the panel opens at the estimate (`previewHeight`), so a short first pass waits as + /// well: the first alert after a launch asked for 336 (the estimate), 295 and 489 pt (2026-10-23 17:10). func reportPreviewFit(_ height: CGFloat) { let rounded = ceil(height) - if let current = previewFitHeight, abs(rounded - current) < 2 { return } + let current = previewFitHeight ?? previewHeight + previewShrink?.cancel() + if abs(rounded - current) < 2 { + if previewFitHeight == nil { previewFitHeight = rounded } + return + } + if rounded < current { + previewShrink = Task { [weak self] in + do { try await Task.sleep(for: Self.previewShrinkDelay) } catch { return } + guard let self else { return } + self.previewFitHeight = rounded + self.presentationChanged?() + } + return + } previewFitHeight = rounded presentationChanged?() } + static let previewShrinkDelay = Duration.milliseconds(300) + private var previewShrink: Task? + public var selected: WidgetSession? { sessionIndex[selectedKey] ?? (lastSelected?.key == selectedKey ? lastSelected : nil) } + /// The height a filter change may not shrink the pane below, for the view (overview, section, session) it was + /// taken in. A filter that shows fewer rows, or none, used to drop the open panel by hundreds of points and grow + /// it back on the next click (Martin, 2026-10-23: "it probably doesn't need to lower the height"). + private var agentFitFloor: (view: String, height: CGFloat)? + private var fitView: String { "\(inboxOverview)|\(section)|\(selectedKey)" } + + /// Called just before a filter changes what an open pane lists: the pane keeps at least its current height. + public func holdAgentFit() { + guard expanded != nil, let height = agentFitHeight else { return } + agentFitFloor = (fitView, max(height, agentFitFloor?.view == fitView ? agentFitFloor?.height ?? 0 : 0)) + } + + /// A row folding or another view: the pane fits its content again. + public func releaseAgentFit() { + agentFitFloor = nil + } + /// Whole points, and only a change of 2 pt or more, so a measurement cannot chase its own resize. func reportAgentFit(_ height: CGFloat?) { - let rounded = height.map { ceil($0) } + if loadingFitHold != nil, loadingFitHold != selectedKey || !inboxLoading { + loadingFitHold = nil + agentFitFloor = nil + } + var rounded = height.map { ceil($0) } + if let floor = agentFitFloor { + if floor.view == fitView, let value = rounded { rounded = max(value, floor.height) } else { agentFitFloor = nil } + } if let rounded, let current = agentFitHeight, abs(rounded - current) < 2 { return } guard rounded != agentFitHeight else { return } agentFitHeight = rounded presentationChanged?() } - public var cards: [WidgetCard] { snapshot?.cards.filter { $0.sessionKey == selectedKey } ?? [] } + /// The selected session's cards: from the snapshot that names it, else as last seen (`sessionCache`). + public var cards: [WidgetCard] { + guard let snapshot else { return [] } + if snapshot.selectedKey != selectedKey, let cached = sessionCache[selectedKey] { return cached.cards } + return snapshot.cards.filter { $0.sessionKey == selectedKey } + } + /// Nothing is known about the selected session yet: no snapshot named it, and none carried its cards. public var inboxLoading: Bool { - !selectedKey.isEmpty && snapshot?.selectedKey != selectedKey && cards.isEmpty && error == nil + !selectedKey.isEmpty && snapshot?.selectedKey != selectedKey && sessionCache[selectedKey] == nil + && cards.isEmpty && error == nil + } + /// For the test driver's `state` command and logs. + public var selectionDescription: String { + "selected=\(selectedKey) snapshot=\(snapshot?.selectedKey ?? "-") loading=\(inboxLoading) cards=\(cards.count) " + + "cached=\(sessionCache.count) warm=\(snapshot?.warmKeys?.count ?? 0)" + } + /// The selected session's decision activity and changes, under the same rule as `cards`. + public var sessionActivity: [WidgetActivityEvent] { + snapshot?.selectedKey == selectedKey ? snapshot?.activity ?? [] : sessionCache[selectedKey]?.activity ?? [] + } + public var sessionChanges: WidgetSnapshot.Changes? { + snapshot?.selectedKey == selectedKey ? snapshot?.changes : sessionCache[selectedKey]?.changes } public var card: WidgetCard? { cards.first { $0.id == selectedCardID } ?? cards.last(where: \.needsAnswer) ?? cards.last @@ -478,6 +598,7 @@ public final class WidgetModel: ObservableObject { savedPreferences.removeAll { entry in entry.fence.map { decoded.state.revision >= $0 } ?? false } var next = decoded next.state.preferences = try overlaid(next.state.preferences) + sessionCache.remember(next) snapshot = next if !draggingSide, let position = draggedSidePosition, abs((next.state.preferences.sidePosition ?? 0.5) - position) < 0.0001 @@ -509,6 +630,7 @@ public final class WidgetModel: ObservableObject { persisted: next.state.selectedKey, visibleKeys: next.sessions.filter(\.visible).map(\.key)) selectedCardID = nil if !selectedKey.isEmpty && !settingsOnly { + pickOnWatch(selectedKey) action(["action": "selection", "key": .string(selectedKey)]) resumeTranscript() } @@ -582,6 +704,10 @@ public final class WidgetModel: ObservableObject { return } guard layout.hoverPreviews, !dialogOpen, !draggingSide, expanded == nil else { return } + // The panel's own hover fires only when the pointer crosses its edge. After a click collapsed the panel + // under a resting pointer it never fired again, so the next bubble showed its tooltip and no preview + // (Martin, 2026-10-23). A bubble the pointer reaches starts the panel's hover itself. + if hoveredSurface != surface { hover(surface, inside: true) } cancelHoverPrewarm() hoveredSession = (key, surface) transcriptHoverTask = Task { [weak self] in @@ -718,6 +844,7 @@ public final class WidgetModel: ObservableObject { expanded = edge prewarmTranscript() resumeTranscript() + refreshContext() presentationChanged?() scheduleQuietReduction() if openingInbox == nil, let card, card.kind == "answer", !card.read { @@ -772,14 +899,111 @@ public final class WidgetModel: ObservableObject { inboxSessionPulses = inboxSessionPulses.filter { activeKeys.contains($0.key) } inbox = value inboxSessions = Dictionary(value.sessions.map { ($0.key, $0) }, uniquingKeysWith: { _, last in last }) - railActivitySessions = railSessions.filter { session in + let railActive = railSessions.filter { session in let inbox = inboxSessions[session.key] return session.visualStatus == .working || (inbox?.unread ?? 0) + (inbox?.needsAnswer ?? 0) > 0 } + // The rail reads in the hover list's order, so a bubble and its row sit at the same place (Martin, + // 2026-10-23: "the order should be the same in the right and the expanded list"); "stable" keeps places. + railActivitySessions = snapshot?.state.preferences.railOrder == "stable" + ? railActive : railActive.sorted(by: Self.previewOrder(inbox: inboxSessions, sort: previewSort)) + inboxOverviewItems = Self.overviewItems(value.sessions) + reconcileStaged(value.staged ?? value.sessions.flatMap { $0.items ?? [] }) + previewList = Self.previewRows( + sessions: snapshot?.sessions ?? [], inbox: inboxSessions, + filter: previewFilter, sort: previewSort) + } + + public var previewFilter: String { snapshot?.state.preferences.previewFilter ?? "needs-you" } + public var previewSort: String { snapshot?.state.preferences.previewSort ?? "attention" } + + /// The hover list. "Needs you": a session with an unanswered item, or a shown session that waits. With the + /// "active" filter, shown top-level sessions that work or hold unread items join it. Orders: "attention" (waiting + /// for you, then working, then newest), "recent" (newest activity) and "project" (by project, then newest). + nonisolated static func previewRows( + sessions: [WidgetSession], inbox: [String: WidgetInboxSession], filter: String, sort: String + ) -> [WidgetSession] { + let rows = sessions.filter { session in + if needsYou(session, inbox: inbox) { return true } + guard filter == "active", session.visible, session.parentKey == nil, session.agentId == nil else { return false } + return session.status == "working" || (inbox[session.key]?.unread ?? 0) > 0 + } + return rows.sorted(by: previewOrder(inbox: inbox, sort: sort)) + } + + nonisolated static func needsYou(_ session: WidgetSession, inbox: [String: WidgetInboxSession]) -> Bool { + (inbox[session.key]?.needsAnswer ?? 0) > 0 || (session.visible && session.status == "waiting") + } + + /// The hover list's order, which the rail's session bubbles share. Equal times fall back to the key, so the two + /// lists never disagree on a tie. + nonisolated static func previewOrder( + inbox: [String: WidgetInboxSession], sort: String + ) -> (WidgetSession, WidgetSession) -> Bool { + func rank(_ session: WidgetSession) -> Int { + needsYou(session, inbox: inbox) ? 0 : session.status == "working" ? 1 : 2 + } + func newer(_ lhs: WidgetSession, _ rhs: WidgetSession) -> Bool { + lhs.activityAt == rhs.activityAt ? lhs.key < rhs.key : lhs.activityAt > rhs.activityAt + } + return { lhs, rhs in + switch sort { + case "recent": + return newer(lhs, rhs) + case "project": + let order = lhs.project.localizedCaseInsensitiveCompare(rhs.project) + return order == .orderedSame ? newer(lhs, rhs) : order == .orderedAscending + default: + let (left, right) = (rank(lhs), rank(rhs)) + return left == right ? newer(lhs, rhs) : left < right + } + } + } + + /// Stores the hover list's filter or order. + public func setPreview(filter: String? = nil, sort: String? = nil) { + var patch: [String: WidgetJSON] = [:] + if let filter { patch["previewFilter"] = .string(filter) } + if let sort { patch["previewSort"] = .string(sort) } + guard !patch.isEmpty else { return } + updatePreferences(patch) } public func inboxFor(_ key: String) -> WidgetInboxSession? { inboxSessions[key] } public func inboxPulseFor(_ key: String) -> Int { inboxSessionPulses[key] ?? 0 } + + /// The session an alert came from, flashed for `attentionSeconds` on its rail bubble, its hover row and its + /// popup. The counter restarts a running flash (`AttentionPulse`). + /// One counter per session, so flashing one session never changes another session's trigger. + @Published public private(set) var attentionCounts: [String: Int] = [:] + public private(set) var attentionSeconds: Double = 5 + + public func flashAttention(_ key: String, seconds: Double) { + guard seconds > 0 else { return } + attentionSeconds = seconds + attentionStarts[key] = Date() + attentionCounts[key, default: 0] += 1 + } + + /// When each session's flash began, so a row built while it runs flashes for the rest (`AttentionPulse.since`). + /// Not published: the counter already re-renders the views that flash. + private var attentionStarts: [String: Date] = [:] + + /// The pulse trigger for one session: changes only when that session is flashed. + public func attentionTrigger(_ key: String) -> Int { attentionCounts[key] ?? 0 } + + /// When the session's current flash began. + public func attentionStart(_ key: String) -> Date? { attentionStarts[key] } + + /// The style every flash of a session that needs you uses. + var attentionStyle: AttentionPulseStyle { + var style = AttentionPulseStyle.borderFlash(color: .orange, cornerRadius: 12) + style.duration = attentionSeconds + style.period = 0.8 + style.fadeIn = 0.18 + style.fadeOut = 0.3 + return style + } public var inboxCount: Int { max(0, inbox.unread) + max(0, inbox.needsAnswer) } /// Expands one inbox card in place. An unread answer or result counts as read once the user opens it, the same @@ -821,6 +1045,7 @@ public final class WidgetModel: ObservableObject { public func openInboxNotification(on surface: WidgetSurfaceID, key: String? = nil, needsAnswer: Bool? = nil) { guard !stopping else { return } + listOrigin = .overview let candidates: [WidgetInboxSession] if let key { candidates = inboxSessions[key].map { [$0] } ?? [] @@ -847,6 +1072,85 @@ public final class WidgetModel: ObservableObject { acknowledgeOpenedInbox() } + /// The overview's entries from every session: the waiting ones first, then newest first. A hub from before the + /// per-session list gives each session's pending and unread item. Stored when the inbox changes: up to 512 + /// sessions × 8 entries, sorted, is too much to derive in a body that reads it several times per pass. + public private(set) var inboxOverviewItems: [WidgetInboxItem] = [] + /// Every staged decision of every session (the decision log's unsent picks and notes), stored with the inbox. + @Published public internal(set) var stagedItems: [WidgetInboxItem] = [] + /// Picks staged or cleared here that the next snapshot has not shown yet, by decision id ("" is a cleared pick). + @Published var pendingStage: [String: String] = [:] + /// The question about staged decisions before the panel closes (`requestCollapse`). + @Published public var stagedLeavePrompt = false + /// The staged decisions view: nil closed, "session" for the selected session's, "all" for every session's. + @Published public var stagedView: String? + /// The staged decisions the user kept at the last leave; the prompt asks again only for a newly staged one. + var keptStaged: Set = [] + /// When this widget last saved a decision's draft, by card id, for the card's "Draft saved" line. + @Published public var draftSavedAt: [String: Date] = [:] + + nonisolated static func overviewItems(_ sessions: [WidgetInboxSession]) -> [WidgetInboxItem] { + sessions.flatMap { entry in entry.items ?? [entry.pendingItem, entry.unreadItem].compactMap { $0 } } + .sorted { lhs, rhs in + lhs.needsAnswer == rhs.needsAnswer ? (lhs.at == rhs.at ? lhs.id < rhs.id : lhs.at > rhs.at) : lhs.needsAnswer + } + } + + /// An overview entry's "Open in session": that session's inbox with the entry open. + /// The list a session was opened from, which the first Esc goes back to (Martin, 2026-10-23: "it should first go + /// to the list and the second escape should close the expansion entirely"). + public enum ListOrigin: Sendable { case overview, sessions } + public var listOrigin: ListOrigin = .overview + + /// Esc in a session: back to the list it came from. False when a list (or no session) is on screen, so Esc then + /// closes the panel. + @discardableResult + public func escapeToList() -> Bool { + guard activeModuleID == "agents", selected != nil, !inboxOverview, section != "Sessions" else { return false } + switch listOrigin { + case .overview: inboxOverview = true + case .sessions: section = "Sessions" + } + return true + } + + public func openInboxItem(_ item: WidgetInboxItem) { + guard !stopping else { return } + listOrigin = .overview + openingInbox = item + select(item.key) + selectedCardID = item.id + foldedCardIDs.remove(item.id) + section = "Inbox" + acknowledgeOpenedInbox() + } + + /// The inbox button: every session's entries in one list. A second click while that list is open hides the panel. + public func toggleInboxOverview(on surface: WidgetSurfaceID) { + guard !stopping else { return } + if presentation(for: surface) == .expanded, activeModuleID == "agents", inboxOverview { + requestCollapse() + return + } + + inboxOverview = true + section = "Inbox" + openModule("agents", on: surface) + } + + /// A session button on the rail or the top strip: opens that session's inbox, or hides the panel when the same + /// session is already open there (Martin, 2026-10-23: a second click did nothing). + public func toggleSession(_ key: String, on surface: WidgetSurfaceID) { + guard !stopping else { return } + PerfLog.mark("widget.toggle-session surface=\(surface.key) presentation=\(presentation(for: surface)) module=\(activeModuleID) overview=\(inboxOverview) same=\(selectedKey == key) group=\(activeSideGroup)") + if presentation(for: surface) == .expanded, activeModuleID == "agents", !inboxOverview, selectedKey == key { + requestCollapse() + return + } + + openInboxNotification(on: surface, key: key) + } + private func acknowledgeOpenedInbox() { guard !stopping, let item = openingInbox, expanded != nil, activeModuleID == "agents", section == "Inbox", selectedKey == item.key, let displayed = cards.first(where: { $0.id == item.id }), displayed.at == item.at @@ -864,6 +1168,7 @@ public final class WidgetModel: ObservableObject { public func collapse() { cancelHoverPrewarm() + agentFitFloor = nil openingInbox = nil followedTranscript = nil hoveredSurface = nil @@ -885,15 +1190,33 @@ public final class WidgetModel: ObservableObject { public func select(_ key: String, edge: EdgePanelPlacement? = nil) { if openingInbox?.key != key { openingInbox = nil } + inboxOverview = false selectedKey = key lastSelected = snapshot?.sessions.first { $0.key == key } + if expanded != nil { refreshContext() } selectedCardID = nil + if inboxLoading { + holdAgentFit() + loadingFitHold = key + } + // How often a switch still has to wait: `loading=true` means no snapshot carried this session yet. + PerfLog.mark("widget.select loading=\(inboxLoading) cards=\(cards.count) cached=\(sessionCache.count)") + pickOnWatch(key) action(["action": "selection", "key": .string(key)]) if let edge { open(edge) } if expanded != nil || hoveredSurface != nil { prewarmTranscript(on: expanded == nil ? hoveredSurface : nil) } resumeTranscript() } + /// Tells the running watch at once, so its next snapshot names this session (about 0.1 s) instead of after the + /// `selection` action reaches its state file (a process and a 180 ms debounce, about 0.5 s). The action still runs: + /// it keeps the choice for the next launch. + private func pickOnWatch(_ key: String) { + guard let watcher, let data = try? JSONEncoder().encode(["select": key]), + let line = String(data: data, encoding: .utf8) else { return } + do { try watcher.sendInput(line + "\n") } catch { PerfLog.mark("widget.pick not sent: \(error.localizedDescription)") } + } + public func unpinSelected() { let old = selectedKey action(["action": "visibility", "key": .string(old), "pinned": false]) @@ -1152,8 +1475,17 @@ public final class WidgetModel: ObservableObject { } } + /// Runs `work` after every hub write queued before it, and holds the later ones until it is done. + func afterMutations(_ work: @escaping @MainActor () async -> Void) { + let previous = mutationTask + mutationTask = Task { + await previous?.value + await work() + } + } + @discardableResult - private func call(_ value: WidgetJSON) async throws -> WidgetJSON { + func call(_ value: WidgetJSON, timeoutSeconds: Int = 120) async throws -> WidgetJSON { if let actionRunner { return try await actionRunner(value) } let file = journal.appendingPathComponent("action-" + UUID().uuidString + ".tmp") try JSONEncoder().encode(value).write(to: file, options: .atomic) @@ -1162,12 +1494,12 @@ public final class WidgetModel: ObservableObject { PerfLog.mark("widget.action cleanup \(error.localizedDescription)") } } - return try await runAction(file) + return try await runAction(file, timeoutSeconds: timeoutSeconds) } - private func runAction(_ file: URL) async throws -> WidgetJSON { + private func runAction(_ file: URL, timeoutSeconds: Int = 120) async throws -> WidgetJSON { let result = try await bridge.run( - subcommand: "hub", args: widgetArgs + ["call", "--input", file.path], timeoutSeconds: 120) + subcommand: "hub", args: widgetArgs + ["call", "--input", file.path], timeoutSeconds: timeoutSeconds) guard result.exitCode == 0 else { throw ToolsBridgeError.refused(String(result.stderr.suffix(1200))) } @@ -1489,6 +1821,61 @@ public final class WidgetModel: ObservableObject { static let screenRecordingRefused = "macOS refused the screenshot: Screen Recording is off for this app. Turn it on " + "in System Settings → Privacy & Security → Screen & System Audio Recording, then try again." + /// A screen recording for the reply box runs (from the picker until it ends): the video button shows Stop. + @Published public var recordingVideo = false + + /// The recorder behind the video button: GenesisKit's generic target picker, floating controls and session. + public private(set) lazy var screenRecorder = ScreenRecorder(bridge: bridge) + private var screenRecorderActivity: AnyCancellable? + + /// The video button: lets the person pick a window, an app or a display, records it with floating controls next to + /// it (pause, switch what is recorded, restart, "Stop and paste", stop and discard), and adds the movie to this + /// session's draft as a video, with the frame review a pasted video gets (Martin, 2026-10-23: "it should let me + /// pick what i want to record ... a widget somewhere around the recording panel"). A second press stops and pastes. + public func toggleVideoCapture() { + if recordingVideo { + screenRecorder.stopAndUse() + return + } + guard !stopping, !selectedKey.isEmpty else { return } + let key = selectedKey + let recorder = screenRecorder + screenRecorderActivity = recorder.$isActive.removeDuplicates().sink { [weak self] active in + self?.recordingVideo = active + } + recorder.begin(host: ScreenRecordingHost( + useTitle: "Stop and paste", + permissionReason: "The video button records the window, app or display you choose and adds the movie to this session's draft.", + onUse: { [weak self] movie in + guard let self, !self.stopping else { return } + try await self.importRecording(movie, to: key) + }, + onDiscard: { [weak self] in self?.notice = "Recording discarded. Nothing was added to the draft." }, + onError: { [weak self] message in self?.report(ToolsBridgeError.refused(message)) }, + onStart: { [weak self] in self?.collapse() })) + } + + /// Imports a finished recording into the draft of `key` behind every pending mutation, and removes the recorder's + /// file once the draft holds its copy. A failed import keeps the file and throws, so the recorder names where it is. + func importRecording(_ movie: URL, to key: String) async throws { + importing += 1 + defer { importing -= 1 } + let previous = mutationTask + let imported = Task { [weak self] in + await previous?.value + guard let self else { throw CancellationError() } + _ = try await self.call([ + "action": "import", "key": .string(key), "input": .string(movie.path), "type": "video", + ], timeoutSeconds: 300) + } + mutationTask = Task { + do { try await imported.value } catch { PerfLog.mark("widget.recording import \(error.localizedDescription)") } + } + try await imported.value + removeStaging(movie) + notice = "The recording is in the draft. Review its frames before you send it." + } + /// The panel steps aside for the selection, then reopens with the new attachment in the draft. Escape returns /// `{cancelled: true, reason: "user"}`, which is not an error and reopens nothing. A missing Screen Recording grant /// (an error whose text starts with `WidgetCaptureError.screenRecordingDenied`) opens the permission dialog. @@ -1525,39 +1912,75 @@ public final class WidgetModel: ObservableObject { } } + /// The microphone key and a press without a menu: stops the dictation that runs, else starts the preferred mode. public func toggleVoice() { - if let recordVoiceNote { - guard !selectedKey.isEmpty else { return } - recordVoiceNote(selectedKey) - return - } if voiceActive { - voice?.finishInput() + if let voice { + voice.finishInput() + } else { + // Stop while a voice note's clip path is still coming: no microphone runs yet, and none may start. + voiceStart += 1 + finishVoice(fileClip: false) + } return } - guard let prefs = snapshot?.state.preferences, !selectedKey.isEmpty else { return } + startVoice(preferredVoiceMode) + } + + /// Live dictation into the reply box: the words appear while the user talks, with the voice level as waves. In + /// `.note` mode the audio is kept too and becomes a voice note with this session and decision when it stops. + public func startVoice(_ mode: WidgetVoiceMode) { + guard !voiceActive, let prefs = snapshot?.state.preferences, !selectedKey.isEmpty else { return } + defaults.set(mode.rawValue, forKey: Self.lastVoiceModeKey) + let card = ownAnswerCard + let context = WidgetVoiceNoteContext( + sessionKey: selectedKey, sessionTitle: selected?.title, decisionId: card?.sourceId, + decisionNumber: card?.number, title: card?.title) voiceKey = selectedKey voiceFinals = [] voiceText = "" + voiceLevel = 0 voiceActive = true + voiceMode = mode + voiceStart += 1 + let start = voiceStart var args = [ "listen", "--provider", prefs.voiceProvider, "--language", prefs.voiceLanguage, "--input", "mic", "--json", "--stop-on-stdin", ] if let account = prefs.voiceAccount, !account.isEmpty { args += ["--account", account] } if let model = prefs.voiceModel, !model.isEmpty { args += ["--model", model] } - do { - voice = try ToolsLineStream( - bridge: bridge, subcommand: "voice", args: args, - onLines: { [weak self] lines in self?.receiveVoice(lines) }, - onExit: { [weak self] exit in - guard let self else { return } - self.finishVoice() - if exit.status != 0 && !exit.stopped { self.error = String(exit.stderr.suffix(600)) } - }) - } catch { - voiceActive = false - report(error) + let clipSource = mode == .note ? voiceNoteClip : nil + Task { [weak self] in + var args = args + if let clipSource { + do { + let clip = try await clipSource() + guard self?.voiceStart == start else { return } + args += ["--save-audio", clip] + self?.voiceNotePlan = (clip, context) + } catch { + // The words still go into the reply box; only the recording is not kept. + self?.report(error) + } + } + guard let self, self.voiceActive, self.voice == nil, self.voiceStart == start else { return } + do { + self.voice = try ToolsLineStream( + bridge: self.bridge, subcommand: "voice", args: args, + onLines: { [weak self] lines in self?.receiveVoice(lines) }, + onExit: { [weak self] exit in + guard let self else { return } + // Only a clean exit proves the audio writer closed the clip. + self.finishVoice(fileClip: exit.status == 0) + if exit.status != 0 && !exit.stopped { self.error = String(exit.stderr.suffix(600)) } + }) + } catch { + self.voiceActive = false + self.voiceMode = nil + self.voiceNotePlan = nil + self.report(error) + } } } @@ -1580,11 +2003,11 @@ public final class WidgetModel: ObservableObject { voiceText = "" } case "complete": + // The listener still flushes the audio after this line; the exit finishes the note. if let text = event.text { voiceFinals = [text] voiceText = "" } - finishVoice() case "error": error = event.error ?? event.text default: break } @@ -1592,11 +2015,23 @@ public final class WidgetModel: ObservableObject { } } - private func finishVoice() { + /// `fileClip` false: the recording may be incomplete, so it is removed instead of filed as a voice note. + private func finishVoice(fileClip: Bool = true) { guard voiceActive else { return } voiceActive = false + voiceMode = nil voice = nil let text = (voiceFinals + [voiceText]).filter { !$0.isEmpty }.joined(separator: " ") + if let plan = voiceNotePlan { + voiceNotePlan = nil + if fileClip, !text.isEmpty, let fileVoiceNote { + fileVoiceNote(plan.clip, text, plan.context) + } else { + do { try FileManager.default.removeItem(atPath: plan.clip) } catch { + PerfLog.mark("widget.voice clip cleanup \(error.localizedDescription)") + } + } + } if let key = voiceKey, !text.isEmpty { let existing = drafts[key]?.text ?? "" drafts[key] = WidgetDraft( @@ -1647,8 +2082,18 @@ public final class WidgetModel: ObservableObject { } } + /// The nearby messages last read for a card, however old: its section shows them at once while a fresh read runs, + /// instead of "Loading nearby messages…" each time a session opens again. + public func cachedReceiptContext(for card: WidgetCard) -> WidgetReceiptContext? { + receiptContextCache[Self.receiptContextKey(card)]?.value + } + + private static func receiptContextKey(_ card: WidgetCard) -> String { + card.sessionKey + "|" + card.id + "|" + String(card.at) + } + public func receiptContext(for card: WidgetCard) async throws -> WidgetReceiptContext { - let key = card.sessionKey + "|" + card.id + "|" + String(card.at) + let key = Self.receiptContextKey(card) if let cached = receiptContextCache[key], Date().timeIntervalSince(cached.at) < 15 { return cached.value } @@ -1659,7 +2104,7 @@ public final class WidgetModel: ObservableObject { try Task.checkCancellation() guard result.exitCode == 0 else { throw ToolsBridgeError.refused(String(result.stderr.suffix(1000))) } let value = try JSONDecoder().decode(WidgetReceiptContext.self, from: Data(result.stdout.utf8)) - if receiptContextCache.count >= 12, let oldest = receiptContextCache.min(by: { $0.value.at < $1.value.at }) { + if receiptContextCache.count >= 48, let oldest = receiptContextCache.min(by: { $0.value.at < $1.value.at }) { receiptContextCache.removeValue(forKey: oldest.key) } receiptContextCache[key] = (Date(), value) @@ -1671,6 +2116,31 @@ public final class WidgetModel: ObservableObject { provider: session.target.provider) } + /// Context size per session key, for the header (`WidgetSessionFacts`). Read from the transcript cache when a + /// session opens: the conversation itself loads only on the Conversation tab, and the header shows on every tab. + @Published public private(set) var contextTokens: [String: Int] = [:] + private var contextTask: Task? + + private func refreshContext() { + guard let session = selected, session.target.provider != "unknown" else { return } + let key = session.key + let query = transcriptQuery(session) + contextTask?.cancel() + contextTask = Task { [weak self] in + guard let self, let envelope = try? await self.transcriptCache.value(for: query), !Task.isCancelled, + let tokens = Self.contextTokens(envelope.turns) else { return } + if self.contextTokens[key] != tokens { self.contextTokens[key] = tokens } + } + } + + /// The context the session's last model call carried: its fresh input plus its cache reads. nil when the turns + /// have no call with usage (a provider that reports none, or nothing loaded yet). + nonisolated static func contextTokens(_ turns: [TranscriptTurn]) -> Int? { + guard let usage = turns.last(where: { $0.role == "assistant" && $0.usage != nil })?.usage else { return nil } + let total = (usage.inputTokens ?? 0) + (usage.cacheReadTokens ?? 0) + return total > 0 ? total : nil + } + private func prewarmTranscript(on surface: WidgetSurfaceID? = nil) { let module = surface.map { moduleSelections[$0.key] ?? "agents" } ?? activeModuleID let hovered = surface.flatMap { surface in @@ -1740,7 +2210,8 @@ public final class WidgetModel: ObservableObject { let delay = max(5, snapshot?.state.preferences.quietSeconds ?? 15) quietTask = Task { [weak self] in do { try await Task.sleep(for: .seconds(delay)) } catch { return } - self?.collapse() + // With staged decisions this asks first and the panel stays open (Martin, 2026-10-23). + self?.requestCollapse() } } @@ -1750,7 +2221,7 @@ public final class WidgetModel: ObservableObject { } } - private func report(_ error: Error) { + func report(_ error: Error) { self.error = error.localizedDescription PerfLog.mark("widget.error \(error.localizedDescription)") } @@ -1830,3 +2301,11 @@ final class WidgetSnapshotDecoder: @unchecked Sendable { } } } + +/// The reply box microphone's two live modes. The raw values are the `composerVoice` setting's. +public enum WidgetVoiceMode: String, Sendable { + /// Words into the reply box only. + case dictate = "live" + /// Words into the reply box, and the audio kept as a voice note with its session and decision. + case note +} diff --git a/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetMotion.swift b/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetMotion.swift index 2f9f269499..d93572ab71 100644 --- a/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetMotion.swift +++ b/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetMotion.swift @@ -44,7 +44,8 @@ struct WidgetGlassControl: ViewModifier { if systemOpaque || requestedOpaque { content.background(Color(white: 0.16), in: Capsule()) } else if #available(macOS 26.0, *) { - content.glassEffect(.regular.tint(.black.opacity(0.4)).interactive(), in: Capsule()) + // Glass is not hit-tested; the capsule is, as the opaque form's fill is. + content.glassEffect(.regular.tint(.black.opacity(0.4)).interactive(), in: Capsule()).contentShape(Capsule()) } else { content.background(.ultraThinMaterial, in: Capsule()) } diff --git a/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetReceiptContext.swift b/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetReceiptContext.swift index 1cb9806702..62eeaeda5a 100644 --- a/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetReceiptContext.swift +++ b/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetReceiptContext.swift @@ -81,6 +81,7 @@ struct WidgetReceiptContextView: View { hasPrewarmed = true let requestID = UUID() loadingID = requestID + if context == nil { context = model.cachedReceiptContext(for: card) } loading = context == nil failure = nil defer { diff --git a/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetSessionBrowser.swift b/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetSessionBrowser.swift index 17004a8dbe..77bdd01878 100644 --- a/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetSessionBrowser.swift +++ b/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetSessionBrowser.swift @@ -8,6 +8,7 @@ public struct WidgetSessionBrowser: View { @State private var limit = 40 @State private var expanded = Set() @State private var collapsed = Set() + @GenReduceMotion private var reduceMotion public init(model: WidgetModel, onOpen: ((WidgetSession) -> Void)? = nil) { self.model = model @@ -15,6 +16,7 @@ public struct WidgetSessionBrowser: View { } public var body: some View { + let _ = RenderProbe.hit("widget.sessions") let groups = WidgetAgentTree.groups(model.snapshot?.sessions ?? [], query: query, onlyPinned: onlyPinned) LazyVStack(alignment: .leading, spacing: 10) { HStack(spacing: 8) { @@ -52,29 +54,15 @@ public struct WidgetSessionBrowser: View { .frame(maxWidth: .infinity) } else { ForEach(Array(groups.prefix(limit))) { group in - if group.children.isEmpty { - row(group.parent, title: group.parent.title, depth: 0) - } else { - GenDisclosure(isExpanded: Binding(get: { - if !query.isEmpty { return true } - if collapsed.contains(group.id) { return false } - return expanded.contains(group.id) || group.id == model.selectedKey - || group.children.contains { $0.id == model.selectedKey } - }, set: { value in - if value { expanded.insert(group.id); collapsed.remove(group.id) } - else { expanded.remove(group.id); collapsed.insert(group.id) } - }), minHeight: 34, identifier: "widget.sessions.group." + group.id) { + // Every top-level session is the same row, with or without sub-agents; a chevron folds them. + VStack(alignment: .leading, spacing: 2) { + topRow(group) + if !group.children.isEmpty && isOpen(group) { LazyVStack(spacing: 2) { - row(group.parent, title: "Main", depth: 0) - ForEach(group.children) { child in row(child.session, title: child.session.title, depth: child.depth) } - }.padding(.top, 6) - } label: { - AgentRosterGroupLabel(title: group.parent.title, provider: group.parent.target.provider, - project: group.parent.project, account: group.parent.account, - total: group.children.count, running: group.running, - live: group.parent.status == "working" || group.running > 0, - lastAt: Date(timeIntervalSince1970: group.parent.activityAt / 1000)) - .padding(.vertical, 4) + ForEach(group.children) { child in + row(child.session, title: child.session.title, depth: child.depth + 1) + } + }.transition(.genReveal) } } } @@ -111,12 +99,75 @@ public struct WidgetSessionBrowser: View { .fixedSize() } + private func isOpen(_ group: WidgetAgentTree.Group) -> Bool { + if !query.isEmpty { return true } + if collapsed.contains(group.id) { return false } + return expanded.contains(group.id) || group.children.contains { $0.id == model.selectedKey } + } + + private func toggle(_ group: WidgetAgentTree.Group) { + let opening = !isOpen(group) + GenMotion.perform(expanding: opening, reduceMotion: reduceMotion) { + if opening { expanded.insert(group.id); collapsed.remove(group.id) } + else { expanded.remove(group.id); collapsed.insert(group.id) } + } + } + + private func open(_ session: WidgetSession) { + if let onOpen { onOpen(session) } + else { + model.select(session.key) + model.section = "Inbox" + model.listOrigin = .sessions + } + } + + /// A top-level session: its name, project, account and sub-agent count, the same for every session. + private func topRow(_ group: WidgetAgentTree.Group) -> some View { + let session = group.parent + let selected = session.key == model.selectedKey + return WidgetHoverRow { hovering in HStack(alignment: .center, spacing: 2) { + if group.children.isEmpty { + Color.clear.frame(width: 18, height: 18) + } else { + Button { toggle(group) } label: { + Image(systemName: "chevron.right").font(.system(size: 9, weight: .semibold)) + .foregroundStyle(.secondary).rotationEffect(.degrees(isOpen(group) ? 90 : 0)) + .frame(width: 18, height: 26).contentShape(Rectangle()) + } + .buttonStyle(.genHoverPlain()) + .instantTooltip(isOpen(group) ? "Hide its sub-agents" : "Show its \(Plural.count(group.children.count, "sub-agent"))") + .accessibilityLabel((isOpen(group) ? "Hide sub-agents of " : "Show sub-agents of ") + session.title) + .accessibilityIdentifier("widget.sessions.group." + group.id) + } + Button { open(session) } label: { + VStack(alignment: .leading, spacing: 3) { + AgentRosterGroupLabel(title: session.title, provider: session.target.provider, + project: session.project, account: session.account, + total: group.children.count, running: group.running, + live: session.status == "working" || group.running > 0, + lastAt: Date(timeIntervalSince1970: session.activityAt / 1000), + unread: model.inboxFor(session.key).map { $0.unread + $0.needsAnswer } ?? 0) + if session.hiddenByFilter { + Text("Hidden by widget filters").font(.caption2).foregroundStyle(.orange).padding(.leading, 31) + } + } + .padding(.vertical, 7).padding(.horizontal, 4) + .frame(maxWidth: .infinity, alignment: .leading).contentShape(Rectangle()) + } + .buttonStyle(RowButtonStyle(cornerRadius: 8)) + .accessibilityLabel("Open session " + session.title + ", " + session.visualStatus.label) + trailing(session, hovering: hovering) + } } + .padding(.horizontal, 4) + .background(selected ? Color.blue.opacity(0.13) : .clear, in: RoundedRectangle(cornerRadius: 10)) + .overlay(RoundedRectangle(cornerRadius: 10).strokeBorder(selected ? Color.blue.opacity(0.35) : .clear)) + } + private func row(_ session: WidgetSession, title: String, depth: Int) -> some View { - HStack(alignment: .top, spacing: 5) { - Button { - if let onOpen { onOpen(session) } - else { model.select(session.key); model.section = "Inbox" } - } label: { + let selected = session.key == model.selectedKey + return WidgetHoverRow { hovering in HStack(alignment: .center, spacing: 2) { + Button { open(session) } label: { VStack(alignment: .leading, spacing: 3) { AgentRosterRow(title: title, provider: session.target.provider, role: session.role ?? "lead", model: session.model, account: session.account, @@ -125,28 +176,42 @@ public struct WidgetSessionBrowser: View { lastAt: Date(timeIntervalSince1970: session.activityAt / 1000), toolCalls: session.toolCalls ?? 0, unread: model.inboxFor(session.key).map { $0.unread + $0.needsAnswer } ?? 0, - selected: session.key == model.selectedKey, showsRunningLabel: session.agentId == nil) + selected: selected, showsRunningLabel: session.agentId == nil) if session.hiddenByFilter { Text("Hidden by widget filters").font(.caption2).foregroundStyle(.orange).padding(.leading, 15) } - }.frame(maxWidth: .infinity, alignment: .leading) - }.buttonStyle(.genHoverRow()) - .accessibilityLabel("Open session " + session.title + ", " + session.visualStatus.label) + } + .padding(.vertical, 6).padding(.horizontal, 6) + .frame(maxWidth: .infinity, alignment: .leading).contentShape(Rectangle()) + } + .buttonStyle(RowButtonStyle(cornerRadius: 8)) + .accessibilityLabel("Open session " + session.title + ", " + session.visualStatus.label) + trailing(session, hovering: hovering) + } } + .padding(.leading, CGFloat(min(depth, 4)) * 14).padding(.trailing, 4) + .background(selected ? Color.blue.opacity(0.13) : .clear, in: RoundedRectangle(cornerRadius: 10)) + .overlay(RoundedRectangle(cornerRadius: 10).strokeBorder(selected ? Color.blue.opacity(0.35) : .clear)) + } + + /// cmux and pin, inside the row's own box so its selection and hover cover them. cmux is live only on the + /// hovered or selected row. + private func trailing(_ session: WidgetSession, hovering: Bool) -> some View { + HStack(spacing: 0) { + WidgetCmuxButton(model: model, session: session, size: 10.5, + live: hovering || session.key == model.selectedKey) Button { model.action(["action": "visibility", "key": .string(session.key), "pinned": .bool(!session.pinned)]) } label: { Image(systemName: session.pinned ? "pin.fill" : "pin") .font(.system(size: 10)).foregroundStyle(session.pinned ? Color.blue : .secondary) - .frame(width: 22, height: 25) + .frame(width: 24, height: 24).contentShape(Rectangle()) }.buttonStyle(.genHoverPlain()).instantTooltip(session.pinned ? "Unpin from widget" : "Pin to widget") .accessibilityLabel((session.pinned ? "Unpin " : "Pin ") + session.title) } - .padding(.leading, CGFloat(min(depth, 4)) * 12).padding(.vertical, 5).padding(.horizontal, 6) - .background(session.key == model.selectedKey ? Color.blue.opacity(0.12) : .clear, - in: RoundedRectangle(cornerRadius: 9)) } private func setProjects(_ projects: [String]) { model.action(["action": "preferences", "patch": ["projects": .array(projects.map(WidgetJSON.string))]]) } } + diff --git a/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetSessionCache.swift b/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetSessionCache.swift new file mode 100644 index 0000000000..cf9fdf99da --- /dev/null +++ b/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetSessionCache.swift @@ -0,0 +1,48 @@ +import Foundation + +/// What the widget last saw of each session, so a switch shows the session's cards at once instead of "Loading this +/// session's inbox…" until a snapshot names it (Martin, 2026-10-24: "can't it just be cached and be prepared at all +/// times and the new decisions would be just appended"). Every snapshot refreshes the sessions it carries: the selected +/// one in full (cards, decision activity, changes), the warm ones (`warmKeys`, the sessions that need you) with their +/// newest cards. The snapshot that names a session after the switch replaces what is shown, so new cards just appear. +struct WidgetSessionCache { + struct Detail: Equatable { + var cards: [WidgetCard] + var activity: [WidgetActivityEvent]? + var changes: WidgetSnapshot.Changes? + } + + /// Sessions kept; the least recently refreshed goes first. + static let limit = 48 + private var details: [String: Detail] = [:] + private var order: [String] = [] + + subscript(key: String) -> Detail? { details[key] } + + var count: Int { details.count } + + mutating func remember(_ snapshot: WidgetSnapshot) { + let warm = snapshot.warmKeys ?? [] + let selected = snapshot.selectedKey.flatMap { $0.isEmpty ? nil : $0 } + guard !warm.isEmpty || selected != nil else { return } + let grouped = Dictionary(grouping: snapshot.cards, by: \.sessionKey) + for key in warm where key != selected { + // Activity and changes come only with a selection; the last ones seen stay until the next. + var detail = details[key] ?? Detail(cards: []) + detail.cards = grouped[key] ?? [] + store(key, detail) + } + if let selected { + store(selected, Detail(cards: grouped[selected] ?? [], activity: snapshot.activity, changes: snapshot.changes)) + } + } + + private mutating func store(_ key: String, _ detail: Detail) { + details[key] = detail + order.removeAll { $0 == key } + order.append(key) + while order.count > Self.limit { + details.removeValue(forKey: order.removeFirst()) + } + } +} diff --git a/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetStagedViews.swift b/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetStagedViews.swift new file mode 100644 index 0000000000..034bf23988 --- /dev/null +++ b/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetStagedViews.swift @@ -0,0 +1,181 @@ +import SwiftUI + +/// A decision's staged pick or note, as one line: "a) Back both up" or the note. +enum WidgetStagedText { + static func line(_ item: WidgetInboxItem, option: String?) -> String { + if let option, let index = option.unicodeScalars.first.map({ Int($0.value) - 97 }), + let options = item.options, options.indices.contains(index) { + let note = item.draft.map { " · " + $0 } ?? "" + return "\(option)) " + options[index] + note + } + return item.draft ?? "Nothing picked" + } +} + +/// The row under a session inbox's filter that says what is staged and opens the staged view. +struct WidgetStagedChip: View { + @ObservedObject var model: WidgetModel + @GenReduceMotion private var reduceMotion + + var body: some View { + let here = model.sessionStaged.count + let all = model.stagedItems.count + if all > 0 { + HStack(spacing: 8) { + Image(systemName: "tray.and.arrow.up").font(.system(size: 11)).foregroundStyle(SessionPalette.blue) + Text(here > 0 && here < all ? "\(Plural.count(here, "decision")) staged here · \(all) in all" + : "\(Plural.count(all, "decision")) staged") + .font(.system(size: 11, weight: .medium)) + Spacer(minLength: 4) + Button("Review and send") { + GenMotion.perform(expanding: true, reduceMotion: reduceMotion) { + model.stagedView = here > 0 ? "session" : "all" + } + } + .buttonStyle(.genHover(accent: SessionPalette.blue)).font(.system(size: 11, weight: .semibold)) + .instantTooltip("The picks and notes you staged. Choose which go now; each session gets one message.") + } + .padding(.horizontal, 10).padding(.vertical, 7) + .background(SessionPalette.blue.opacity(0.12), in: RoundedRectangle(cornerRadius: 9)) + .accessibilityElement(children: .combine) + } + } +} + +/// The staged decisions, with a checkbox each: the checked ones go now, one message per session. "This session" +/// shows the selected session's; "All sessions" every one (Martin, 2026-10-23: "an easy button which shows all of +/// them not just the filtered by that specific session"). +struct WidgetStagedList: View { + @ObservedObject var model: WidgetModel + @GenReduceMotion private var reduceMotion + /// Unchecked decision ids; everything else is checked, so a newly staged one goes by default. + @State private var skipped: Set = [] + + private var scopeAll: Bool { model.stagedView == "all" || model.sessionStaged.isEmpty } + private var items: [WidgetInboxItem] { scopeAll ? model.stagedItems : model.sessionStaged } + private var chosen: [WidgetInboxItem] { items.filter { !skipped.contains($0.sourceId) } } + + var body: some View { + let items = items + let chosen = chosen + VStack(alignment: .leading, spacing: 10) { + HStack(spacing: 8) { + IconButton(systemName: "chevron.left", tooltip: "Back to the inbox") { + GenMotion.perform(expanding: false, reduceMotion: reduceMotion) { model.stagedView = nil } + }.accessibilityLabel("Back to the inbox") + Text("Staged decisions").font(.system(size: 13, weight: .semibold)) + Spacer(minLength: 4) + if !model.sessionStaged.isEmpty && model.sessionStaged.count < model.stagedItems.count { + GenSegmentedTabs( + "Staged scope", + items: [.init("session", "This session"), .init("all", "All sessions (\(model.stagedItems.count))")], + selection: Binding(get: { scopeAll ? "all" : "session" }, set: { model.stagedView = $0 }), + minSegmentWidth: 90, identifier: "widget.staged.scope") + } + } + if items.isEmpty { + Text("Nothing is staged. A click on a decision's option stages it here.") + .font(.caption).foregroundStyle(.secondary).padding(.vertical, 20).frame(maxWidth: .infinity) + } else { + ForEach(groups(items), id: \.key) { group in + VStack(alignment: .leading, spacing: 6) { + if scopeAll { + Text(model.session(forKey: group.key)?.title ?? "Unknown session") + .font(.system(size: 11, weight: .semibold)).foregroundStyle(.secondary) + } + ForEach(group.items, id: \.sourceId) { item in row(item) } + } + } + Text(chosen.isEmpty ? "Nothing is checked." : WidgetModel.stagedSummary(chosen)) + .font(.system(size: 11)).foregroundStyle(.secondary) + HStack(spacing: 8) { + Button("Send \(chosen.count) now") { + model.sendStaged(chosen.map(\.sourceId)) + model.stagedView = nil + } + .buttonStyle(.borderedProminent).disabled(chosen.isEmpty) + .keyboardShortcut(.return, modifiers: [.command]) + Button("Discard checked") { model.discardStaged(chosen.map(\.sourceId)) } + .buttonStyle(.genHover()).disabled(chosen.isEmpty) + .instantTooltip("Clears the picks and notes; the decisions still wait for an answer.") + Spacer(minLength: 0) + Button("Keep staged") { + GenMotion.perform(expanding: false, reduceMotion: reduceMotion) { model.stagedView = nil } + }.buttonStyle(.genHover()) + }.font(.caption) + } + } + .padding(12) + .background(.white.opacity(0.05), in: RoundedRectangle(cornerRadius: 12)) + } + + private func groups(_ items: [WidgetInboxItem]) -> [(key: String, items: [WidgetInboxItem])] { + var order: [String] = [] + var byKey: [String: [WidgetInboxItem]] = [:] + for item in items { + if byKey[item.key] == nil { order.append(item.key) } + byKey[item.key, default: []].append(item) + } + return order.map { ($0, byKey[$0] ?? []) } + } + + private func row(_ item: WidgetInboxItem) -> some View { + let checked = !skipped.contains(item.sourceId) + return Button { + if checked { skipped.insert(item.sourceId) } else { skipped.remove(item.sourceId) } + } label: { + HStack(alignment: .top, spacing: 9) { + Image(systemName: checked ? "checkmark.square.fill" : "square") + .foregroundStyle(checked ? SessionPalette.blue : .secondary).font(.system(size: 13)) + VStack(alignment: .leading, spacing: 2) { + Text(WidgetInboxOverview.title(item)).font(.system(size: 12, weight: .medium)).lineLimit(2) + Text(WidgetStagedText.line(item, option: model.stagedOption(item.sourceId, stored: item.draftOption))) + .font(.system(size: 11)).foregroundStyle(.secondary).lineLimit(2) + } + Spacer(minLength: 0) + } + .padding(8).frame(maxWidth: .infinity, alignment: .leading).contentShape(Rectangle()) + } + .buttonStyle(RowButtonStyle(cornerRadius: 8)) + .accessibilityLabel(WidgetInboxOverview.title(item)) + .accessibilityValue(checked ? "Will be sent" : "Stays staged") + } +} + +/// The question before the panel closes with staged decisions: "You have 3 decisions staged. What to do with them?" +struct WidgetStagedLeavePrompt: View { + @ObservedObject var model: WidgetModel + + var body: some View { + let items = model.stagedItems + ZStack { + Color.black.opacity(0.35).allowsHitTesting(true) + VStack(alignment: .leading, spacing: 10) { + Text("You have \(Plural.count(items.count, "decision")) staged. What to do with \(items.count == 1 ? "it" : "them")?") + .font(.system(size: 13, weight: .semibold)).fixedSize(horizontal: false, vertical: true) + Text(WidgetModel.stagedSummary(items) + (items.count == 1 ? " if you send it now." : " if you send them now.")) + .font(.system(size: 11)).foregroundStyle(.secondary) + HStack(spacing: 8) { + Button("Send now") { model.resolveStagedLeave("send") } + .buttonStyle(.borderedProminent).keyboardShortcut(.return, modifiers: []) + Button("Keep staged (send later)") { model.resolveStagedLeave("keep") } + .buttonStyle(.genHover()) + Button("Discard") { model.resolveStagedLeave("discard") } + .buttonStyle(.genHover(accent: .red)) + .instantTooltip("Clears the picks and notes; the decisions still wait for an answer.") + }.font(.caption) + Button("Review them first") { + model.stagedLeavePrompt = false + model.stagedView = "all" + }.buttonStyle(.genHoverPlain()).font(.caption) + } + .padding(16).frame(maxWidth: 420) + .background(.regularMaterial, in: RoundedRectangle(cornerRadius: 14)) + .overlay(RoundedRectangle(cornerRadius: 14).strokeBorder(.white.opacity(0.12))) + .shadow(radius: 18) + } + .transition(.opacity) + .accessibilityElement(children: .contain) + .accessibilityLabel("Staged decisions before closing") + } +} diff --git a/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetWireModels.swift b/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetWireModels.swift index 7eed9bbd47..4b8241e733 100644 --- a/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetWireModels.swift +++ b/src/macos/GenesisKit/Sources/GenesisKit/Widgets/WidgetWireModels.swift @@ -167,6 +167,10 @@ public struct WidgetCard: Codable, Identifiable, Equatable, Sendable { public var answeredVia: String? = nil public var comments: [String]? = nil public var supersededBy: SupersededBy? = nil + /// A staged decision's unsent pick and note (the decision log's `drafted` state); sent with the other staged ones. + public var draftOption: String? = nil + public var draft: String? = nil + public var isStaged: Bool { kind == "decision" && needsAnswer && (draftOption != nil || draft != nil) } public var needsAnswer: Bool { (kind == "decision" && ["open", "drafted"].contains(status)) || (kind == "form" && status == "pending") @@ -286,6 +290,32 @@ public struct WidgetPreferences: Codable, Equatable, Sendable { public var alertAnswers: Bool? public var alertSound: String? public var alertPopup: Bool? + /// The agents hover list: "needs-you" (default) or "active" (sessions that work now too). + public var previewFilter: String? + /// The agents hover list's order: "attention" (default), "recent" or "project". + public var previewSort: String? + /// An answer's sound: "same" (default, `alertSound`), "none" or a system sound name. + public var alertAnswerSound: String? + /// Seconds the session that raised an alert flashes; 0 does not flash it. Default 5. + public var alertFlashSeconds: Double? + /// A session inbox's composer: "top" (default, under the tabs) or "bottom". + public var inboxComposer: String? + /// A rail bubble's tooltip: "auto" (default, only without hover previews), "always" or "never". + public var bubbleTooltip: String? + /// The rail's session bubbles: "list" (default, the hover list's order) or "stable" (each keeps its place). + public var railOrder: String? + /// A click on a decision option: "stage" (default, sent later with the other staged ones) or "send" (at once). + public var decisionClick: String? + /// Leaving the inbox with staged decisions: "ask" (default), "keep" or "send". + public var stagedOnLeave: String? + /// The inbox of every session: "session" (default, grouped under each session) or "list". + public var inboxGrouping: String? + /// The filter a session's inbox opens with: "all" (default), "inbox" or "decisions". + public var inboxFilter: String? + /// The filter the inbox of every session opens with: "all" (default), "needs" or "unread". + public var overviewFilter: String? + /// The composer's microphone: "note" (default, a voice note whose transcript fills the field) or "live". + public var composerVoice: String? public var layout: WidgetLayoutConfiguration { WidgetLayoutConfiguration( @@ -375,6 +405,16 @@ public struct WidgetInboxItem: Codable, Equatable, Sendable { /// A short title (a decision's title or prompt, an answer's question) and a decision's number, for alerts. public var title: String? = nil public var number: Int? = nil + /// How many items this entry stands for: an answers group lists only its newest item. + public var count: Int? = nil + /// A decision's options and recommended letter, so the inbox of every session can stage a pick in place. + public var options: [String]? = nil + public var recommended: String? = nil + public var revision: Int? = nil + /// A staged decision's unsent pick and note; see `WidgetCard.draftOption`. + public var draftOption: String? = nil + public var draft: String? = nil + public var isStaged: Bool { kind == "decision" && needsAnswer && (draftOption != nil || draft != nil) } } public struct WidgetInboxSession: Codable, Equatable, Sendable { public var key: String @@ -383,6 +423,8 @@ public struct WidgetInboxSession: Codable, Equatable, Sendable { public var latest: WidgetInboxItem? public var unreadItem: WidgetInboxItem? public var pendingItem: WidgetInboxItem? + /// The session's listed entries for the inbox overview, waiting ones first, then newest; nil from an older hub. + public var items: [WidgetInboxItem]? = nil } public struct WidgetInboxSummary: Codable, Equatable, Sendable { public struct Profile: Codable, Equatable, Sendable { public var hostId: String } @@ -391,6 +433,8 @@ public struct WidgetInboxSummary: Codable, Equatable, Sendable { public var complete: Bool public var truncated: Bool public var sessions: [WidgetInboxSession] + /// Every staged decision, uncapped (a session's `items` stop at eight); nil from an older hub. + public var staged: [WidgetInboxItem]? = nil public var profile: Profile? public static let empty = WidgetInboxSummary(unread: 0, needsAnswer: 0, complete: false, truncated: false, sessions: []) } @@ -417,6 +461,8 @@ public struct WidgetSnapshot: Codable, Equatable, Sendable { public var changes: Changes? public var errors: [String] public var selectedKey: String? + /// Other sessions whose cards this snapshot carries too (the ones that need you), so opening one waits for nothing. + public var warmKeys: [String]? = nil } public enum WidgetSelection { @@ -469,3 +515,20 @@ public struct WidgetReceiptContext: Codable, Equatable, Sendable { public var transcript: Window? public var error: String? } + +/// A numbered decision shows its number wherever its title shows: "DECISION 20: Which mods to build next" (Martin, +/// 2026-10-23: "i dont see decision number anywhere around this and i should"). The chat, the log and the widget +/// then name it the same way. +public enum WidgetDecisionTitle { + public static func numbered(_ title: String, number: Int?) -> String { + number.map { "DECISION \($0): " + title } ?? title + } +} + +public extension WidgetCard { + var numberedTitle: String { WidgetDecisionTitle.numbered(title, number: number) } +} + +public extension WidgetInboxItem { + var numberedTitle: String? { title.map { WidgetDecisionTitle.numbered($0, number: number) } } +} diff --git a/src/macos/GenesisKit/Sources/GenesisKit/Window/EdgePanelController.swift b/src/macos/GenesisKit/Sources/GenesisKit/Window/EdgePanelController.swift index bfeb413c40..dc6e6098e3 100644 --- a/src/macos/GenesisKit/Sources/GenesisKit/Window/EdgePanelController.swift +++ b/src/macos/GenesisKit/Sources/GenesisKit/Window/EdgePanelController.swift @@ -45,9 +45,16 @@ public extension EnvironmentValues { /// outline. The window itself changes size once per transition, so no frame of the motion re-lays out SwiftUI. private final class EdgePanelRootView: NSView { let maskLayer = CAShapeLayer() + /// The pointer entered or left the window, by geometry. SwiftUI's hover is a hit test, and it said "left" with the + /// pointer still on the panel when a scroll moved a part it does not count (the overlay scroller, an AppKit view) + /// under it, so the hover list closed under a scrolling pointer (Martin, 2026-10-24). + var pointerChanged: ((Bool) -> Void)? override init(frame frameRect: NSRect) { super.init(frame: frameRect) + // Follows the bounds by itself (`inVisibleRect`); `activeAlways`, because the panel never activates the app. + addTrackingArea(NSTrackingArea(rect: .zero, options: [.mouseEnteredAndExited, .activeAlways, .inVisibleRect], + owner: self, userInfo: nil)) wantsLayer = true maskLayer.fillColor = NSColor.black.cgColor // A standalone layer animates every property change by default; only the explicit path animation may move it. @@ -59,6 +66,9 @@ private final class EdgePanelRootView: NSView { override var isFlipped: Bool { false } + override func mouseEntered(with event: NSEvent) { pointerChanged?(true) } + override func mouseExited(with event: NSEvent) { pointerChanged?(false) } + var clipping = false { didSet { guard clipping != oldValue else { return } @@ -75,23 +85,31 @@ private final class EdgePanelRootView: NSView { /// Wakes once per display frame while a transition runs: it keeps the window shadow on the moving outline and records /// the callback timing. The motion itself runs in the render server and does not depend on these callbacks. -private final class EdgeFrameTicker: NSObject { +/// +/// A display link keeps a strong reference to its target, so a ticker nobody stops fires every frame for the life +/// of the process: an overwritten transition left one behind per lost reshape (2026-10-23, `EdgeFrameTicker.step` +/// in a sample of the collapsed widget). `live` counts the running ones for the tests. +final class EdgeFrameTicker: NSObject { + @MainActor static var live = 0 private var link: CADisplayLink? private let tick: () -> Void - init(view: NSView, tick: @escaping () -> Void) { + @MainActor init(view: NSView, tick: @escaping () -> Void) { self.tick = tick super.init() let link = view.displayLink(target: self, selector: #selector(step)) link.add(to: .main, forMode: .common) self.link = link + Self.live += 1 } @objc private func step(_ link: CADisplayLink) { tick() } - func stop() { - link?.invalidate() - link = nil + @MainActor func stop() { + guard let link else { return } + link.invalidate() + self.link = nil + Self.live -= 1 } } @@ -131,12 +149,18 @@ public final class EdgePanelController { private var sideCenterY: CGFloat? private var lastTarget: CGRect? private var lastPresentation: WidgetModulePresentation? - private var transition: Transition? + /// A transition that is replaced stops its ticker, whoever replaces it. + private var transition: Transition? { + didSet { + if let old = oldValue, old.token != transition?.token { old.ticker.stop() } + } + } private var generation = 0 public private(set) var isExpanded = false /// An expanded panel takes the keyboard (its composer). Off only for scripted motion checks on the user's own - /// screen, which must not take the keyboard from the app they are typing in. - public var becomesKeyOnExpand = true + /// screen, which must not take the keyboard from the app they are typing in. Off under XCTest: every expand in + /// a test run made the test process the key window, which took the user's focus about ten times per run. + public var becomesKeyOnExpand = NSClassFromString("XCTestCase") == nil /// Driver callback timing only; neither callback count nor gaps establish compositor frame delivery. public private(set) var lastTransitionTiming: AnimationTiming.Summary? /// Slows every transition by this factor (`GENESIS_WIDGET_MOTION_SCALE`), for frame-by-frame review. @@ -191,6 +215,12 @@ public final class EdgePanelController { hosting.rootView = content() } + /// The pointer entered (true) or left (false) the panel's window. The host's "pointer on the panel" signal. + public var pointerChanged: ((Bool) -> Void)? { + get { root.pointerChanged } + set { root.pointerChanged = newValue } + } + public func setCompactSize(_ size: CGSize) { compactSize = size } public func setExpandedSize(_ size: CGSize) { expandedSize = size } public func setPreviewSize(_ size: CGSize) { previewSize = size } @@ -211,6 +241,7 @@ public final class EdgePanelController { public func hide() { lastTarget = nil + deferredPresentation = nil generation += 1 end(outcome: "interrupted") if let presentation = lastPresentation, motion.presentation != presentation { @@ -224,7 +255,29 @@ public final class EdgePanelController { setPresentation(expanded ? .expanded : .compact, reduceMotion: reduceMotion) } + /// Set while a request sets up its transition. Moving the window lays SwiftUI out at once, and content that + /// measures itself there (the widget's hover list) asks for its real size from inside that layout: the nested + /// request then started a reshape that the outer one overwrote with its own transition, and the panel stayed at + /// the estimate (2026-10-23 17:48, a reshape to 489 pt logged, the window kept 336 pt). + private var applyingPresentation = false + private var deferredPresentation: (presentation: WidgetModulePresentation, reduceMotion: Bool)? + public func setPresentation(_ presentation: WidgetModulePresentation, reduceMotion: Bool) { + if applyingPresentation { + deferredPresentation = (presentation, reduceMotion) + return + } + applyingPresentation = true + applyPresentation(presentation, reduceMotion: reduceMotion) + applyingPresentation = false + // The latest nested request runs against the transition that is now in place. + if let next = deferredPresentation { + deferredPresentation = nil + setPresentation(next.presentation, reduceMotion: next.reduceMotion) + } + } + + private func applyPresentation(_ presentation: WidgetModulePresentation, reduceMotion: Bool) { let expanded = presentation == .expanded let becameExpanded = expanded && !isExpanded let size: CGSize @@ -249,6 +302,9 @@ public final class EdgePanelController { let wasReshaping = transition?.reshape == true end(outcome: "interrupted") isExpanded = expanded + let branch = reduceMotion || !panel.isVisible || from == target ? "settle" + : motion.presentation == presentation && (!wasTransitioning || wasReshaping) ? "reshape" : "begin" + PerfLog.mark("edge.request window=\(panel.windowNumber) edge=\(placement.rawValue) shown=\(motion.presentation) to=\(presentation) \(branch) from=\(Int(from.width))x\(Int(from.height)) target=\(Int(target.width))x\(Int(target.height)) transitioning=\(wasTransitioning)") if reduceMotion || !panel.isVisible || from == target { settle(presentation, frame: target) } else if motion.presentation == presentation, !wasTransitioning || wasReshaping { diff --git a/src/macos/GenesisKit/Sources/GenesisKitBase/Perf/HangWatch.swift b/src/macos/GenesisKit/Sources/GenesisKitBase/Perf/HangWatch.swift index a3458f996d..6533e13968 100644 --- a/src/macos/GenesisKit/Sources/GenesisKitBase/Perf/HangWatch.swift +++ b/src/macos/GenesisKit/Sources/GenesisKitBase/Perf/HangWatch.swift @@ -11,6 +11,10 @@ // A block on the main queue only runs late when the main thread is actually // busy or starved. // +// A loop that returns to the run loop between its passes never delays the ping, so the same watchdog also +// reads the main thread's own CPU time: busy above 85% for 5 s is a "main-spin", sampled the same way (the +// widget sat at 100% CPU for three minutes in a SwiftUI update loop and logged nothing, 2026-10-23 17:10). +// // Stalls > 0.5 s land in PerfLog (the app's perf file + unified log). // When one crosses 1 s the watchdog runs `sample` on this process and writes // the main-thread call graph to `/hangs/