[Feature]: Add first-class native picker actions to winapp ui

[nmetulev]

Is your feature request related to a problem? Please describe.

Automating native Windows file and folder pickers currently requires a brittle sequence of generic winapp ui commands:

1. Trigger the picker in the app.
2. Discover its HWND.
3. Target it with -w.
4. Inspect for controls such as FileNameControlHost / AutomationId 1148.
5. Set a path.
6. Find and invoke Open, Save, Select Folder, or Cancel.
7. Separately verify the app result.

This is unreliable for agents because picker implementations vary. Modern pickers may run in a separate PickerHost process, while classic common dialogs commonly use class #32770. Process identity, ownership, UIA control structure, and selectors vary across Windows versions, picker kinds, locales, and app frameworks.

A blind-agent evaluation demonstrated the impact: native picker UIA inspection became nonresponsive, requiring keyboard recovery and picker-specific knowledge. A purpose-built picker action in another automation tool also failed and required retries. The missing capability is not another low-level set-path operation; it is a safe, opinionated, verified picker transaction.

Current repository behavior confirms the gap:

• UiCommand.cs has no picker-specific command.
• UiListWindowsCommand filters -a results by app PID and may miss a separate PickerHost process.
• Cross-process owned-window discovery exists in UiAutomationService and RealOwnedWindowFinder, but follows only a direct owner edge and is duplicated.
• Current UIA calls are synchronous and do not honor cancellation while blocked in native UIA/COM calls.
• Current docs describe a manual file-dialog workaround and cite AutomationId 1148, which is not a sufficient cross-provider contract.

Describe the solution you'd like

Add a first-class picker command group:

winapp ui picker select <absolute-path> --kind <open|save|folder> (-a <app> | -w <picker-hwnd>) [-t <ms>] [--json]
winapp ui picker cancel (-a <app> | -w <picker-hwnd>) [-t <ms>] [--json]
winapp ui picker list -a <app> [--kind <open|save|folder>] [--json]

picker select should perform the complete workflow in one invocation:

1. Validate the path before touching UI.
2. Wait for a picker associated with the target app.
3. Detect a supported provider and verify the requested kind.
4. Fail closed on missing or ambiguous candidates.
5. Fingerprint and revalidate the target picker before every mutation/action.
6. Resolve provider-specific path, confirm, and cancel roles without localized captions.
7. Set the exact path using non-injecting UIA/Legacy value mechanisms.
8. Read the value back and require semantic equality.
9. Invoke only the role-specific confirm action.
10. Detect validation or overwrite prompts.
11. Verify that the original picker HWND is dismissed.

Success should mean accepted_and_dismissed: the picker accepted the requested input and its original window disappeared. It must not claim that the application consumed the result or persisted data.

picker cancel should invoke the resolved semantic cancel role and verify dismissal. picker list should be diagnostic recovery for ambiguity, not a prerequisite for the normal workflow.

Command design decisions

• Use the nested ui picker namespace for discoverability and future diagnostic operations.
• Keep the path positional to match existing ui command conventions.
• Require --kind in the MVP as a safety assertion and to select correct validation behavior. Automatic kind detection can be considered only after it is proven reliable.
• Do not add public set-path and confirm verbs; they recreate the non-atomic multi-command workflow.
• Keep all existing generic ui verbs available as explicit escape hatches.
• Do not add keyboard, mouse, SendInput, PostMessage, coordinate, English-caption, or address-bar fallback in the MVP.

Three-level commands are viable: the CLI schema and npm command traversal are already recursive. The two-level coverage loop in scripts/generate-llm-docs.ps1 must be made recursive.

Supported MVP providers

Support at minimum:

• Classic Windows common dialogs (#32770) for open, save, and folder selection where verified.
• Modern Windows picker / PickerHost dialogs, including separate-process cases.
• Modern IFileDialog hosted in the caller process where it matches a verified profile.

Custom application pickers should fail with unsupported_picker rather than being acted on heuristically.

Each provider should have a versioned profile defining:

• Structural window evidence.
• Path-input role and supported value/readback patterns.
• Confirm and cancel roles.
• Allowed UIA patterns.
• Terminal validation/overwrite modal signatures.

Do not classify or act based only on process name, class name, title text, English labels, or one AutomationId.

Discovery and safety

Consolidate and extend the existing owned-window enumeration instead of adding a second implementation:

• Enumerate target app top-level HWNDs.
• Enumerate visible top-level desktop windows.
• Walk complete owner/root-owner chains with cycle/depth guards.
• Require the chain to terminate at a target app HWND when targeting by -a.
• Never choose a candidate solely by foreground status, title, size, z-order, or recency.
• Fail with picker_ambiguous and candidate details when multiple candidates match.
• When -w is supplied, still require a supported picker profile; if -a is also supplied, validate association.

Fingerprint candidates with HWND, PID, process-start identity, class, owner chain, provider profile, and UIA root runtime identity where available. Revalidate immediately before path mutation, confirmation, or cancellation to avoid acting on a recycled HWND.

Exact path contract

The MVP accepts one absolute Unicode filesystem path:

• open: existing regular file required.
• folder: existing directory required.
• save: existing parent and non-empty leaf required; leaf may not exist.
• Accept drive-rooted and UNC paths, with probes bounded by the transaction deadline.
• Reject relative paths, URIs, shell namespace paths, wildcards, embedded literal quotes, and multiple selection.
• Preserve requestedPath and report a lexical normalizedPath.
• Do not resolve reparse targets or promise filesystem-object identity; report a warning when a reparse point is present.
• Do not manipulate picker filters in the MVP.
• Require exact readback before confirmation.
• Do not silently truncate long paths or accept extension mutation.

For save:

• Do not automatically consent to destructive overwrite.
• Return overwrite_confirmation_required before UI mutation when the exact requested target exists.
• Also detect a related overwrite modal after confirmation because a selected filter may append an extension and produce a different effective path.
• Leave the prompt and picker intact and report their HWNDs.
• Treat effectivePath as nullable when it cannot be independently observed.

Timeout and cancellation

Use one monotonic transaction deadline, recommended default 15 seconds. Every filesystem, discovery, UIA, and dismissal operation receives only the remaining budget.

Before implementation, prove the bounded-execution mechanism in this order:

1. Query the existing CUIAutomation8 instance for IUIAutomation2 and test ConnectionTimeout / TransactionTimeout.
2. Run picker UIA calls on a dedicated MTA worker with wall-clock supervision.
3. Prove repeated blocked-provider calls return within the deadline without accumulating unusable workers or COM state.
4. If that cannot be proven, use a hidden self-hosted worker process and terminate it on deadline.

The feature should commit to the deadline behavior and structured failure contract, not prematurely to one isolation implementation.

Cancellation behavior must be phase-aware:

• Before mutation: return operation_cancelled; picker untouched.
• After path mutation but before confirm: return operation_cancelled, leave picker open, and report the mutation.
• After confirm starts: observe terminal state briefly; if unknown, return indeterminate_commit.
• Never attempt implicit rollback or cancellation after a potentially delivered commit.

Architecture

Add focused picker orchestration over the existing UIA implementation:

• UiPickerCommand parent with select, cancel, and list.
• IPickerResolver for enumeration, association, provider classification, ambiguity handling, and fingerprint revalidation.
• IPickerWorkflow for path validation and the phase state machine.
• Provider-specific classic and modern profiles/adapters.
• IBoundedUiaExecutor, selected by the timeout spike.
• Extended ISystemUiQuery / consolidated IOwnedWindowFinder seams for enumeration, liveness, root-owner chains, and process identity.
• Precise UIA primitives for role search, value set/readback, and InvokePattern-only actions.

Do not use generic confirmation behavior that silently falls through from Invoke to Toggle, SelectionItem, ExpandCollapse, or an invokable ancestor.

JSON and errors

Success JSON should be camelCase and include:

• schemaVersion, action, kind, provider.
• App and picker PID/HWND/class identity.
• ownerChain and picker fingerprint/profile.
• requestedPath, normalizedPath, observedInput, nullable effectivePath.
• selectionMethod and confirmation role/control/method.
• phase, outcome, dismissal state, original-picker liveness, remaining dialogs.
• elapsedMs and warnings.

Exit 0 only after verified completion. Operational failures should use the existing JSON error envelope on stderr and exit 1, extended additively with optional phase, retryable, recoveryHint, candidates, picker, and sideEffects.

Stable picker errors:

• picker_not_found
• picker_ambiguous
• picker_unresponsive
• unsupported_picker
• kind_mismatch
• invalid_path
• path_not_found
• path_not_selectable
• picker_target_changed
• confirmation_failed
• overwrite_confirmation_required
• dialog_not_dismissed
• operation_cancelled
• indeterminate_commit

Testing and acceptance

Add:

• Unit tests for ownership chains/cycles, ambiguity, provider profiles, fingerprints, path rules, state transitions, timeout propagation, cancellation, and modal classification.
• Command tests for parsing/help, required --kind, target precedence, JSON/stdout/stderr/exit behavior, and all stable errors.
• A deterministic interactive fixture opening classic and modern in-process open/save/folder pickers. The fixture must independently record the returned result in app-visible UI and a JSON event file; save tests should write a marker to disk.
• A packaged Windows App SDK sample using WinRT pickers to exercise the actual separate-process PickerHost path.
• Tests for non-English/custom commit captions, ambiguity, stale HWNDs, a blocked UIA provider, cancellation in each phase, overwrite refusal with unchanged data, extension mutation, unsupported custom pickers, and no-dialog cleanup.

Acceptance matrix:

• Windows 10 19041 and supported Windows 11 builds.
• x64 and ARM64.
• Classic #32770, modern in-process IFileDialog, and separate-process PickerHost.
• English plus at least one non-English display language.
• Open, save, folder, cancel, ambiguity, overwrite, and provider-hang scenarios.

Real picker tests must run on a known interactive test host; an inconclusive non-interactive run does not satisfy the gate.

Delivery plan: one PR

Deliver the complete feature in one pull request, including:

1. The timeout/isolation and PickerHost behavior spikes performed during development, with the chosen implementation and regression coverage included in the PR.
2. Consolidated owner enumeration, resolver, profiles, fingerprinting, and diagnostic picker list.
3. Complete open/save/folder/cancel semantic workflows.
4. Unit, command, deterministic fixture, and separate-PickerHost integration tests.
5. CLI schema, npm generated wrappers, documentation, skill source fragments, generated GitHub skill, and Claude mirrors.

The PR may use internal commits to keep review organized, but the feature should not be split across multiple PRs and should not merge or announce discovery-only behavior before the semantic actions are complete.

Additional context

Relevant repository evidence and history:

• No picker command exists: src/winapp-CLI/WinApp.Cli/Commands/UiCommand.cs.
• Current manual workflow: docs/ui-automation.md under “File dialog interaction”.
• Current agent guidance labels the process “File dialog workaround”: docs/fragments/skills/winapp-cli/ui-automation.md.
• PID-only filtered listing: UiListWindowsCommand.cs.
• Existing direct-owner cross-process discovery: UiAutomationService.GetAllAppWindowsCore and RealOwnedWindowFinder.
• Existing UIA methods are synchronous despite Task-returning interfaces, and cancellation is only checked between polling calls.
• JSON conventions are defined in UiJsonContext and UiJsonError.

Related history:

• PR UI automation: multi-window inspect, cross-window search, screenshot compositing #419: multi-window UIA and cross-window search.
• PR Improve winapp ui --json output: stable envelopes, no leaks, no cycles #511: stable UI JSON envelopes.
• Issue [Bug]: list-windows inconsistency, missing windows without a title #570 / PR List windows will show visible windows without a title by default, added a --show-hidden flag to force showing all windows #571: window-list visibility/title filtering.
• Issues [Bug]: winapp ui send-keys --via post-message reports success but delivers no input to target controls #655, [Bug]: winapp ui send-keys reports success for ctrl+alt+del with --allow-system-keys but Windows always blocks it #656, [Bug]: winapp ui send-keys --via send-input silently drops characters when sending long text #657 and PRs Hard-block ctrl+alt+del in ui send-keys (#656) #665, Fix ui send-keys --via post-message silently dropping input #666, Fix ui send-keys --via send-input silently dropping characters on long text (#657) #667: silent-success input-delivery failures, reinforcing the no-input-fallback requirement.

Agent-facing happy path should be short:

winapp ui picker select "C:\fixtures\import.json" --kind open -a MyApp --json
winapp ui picker select "C:\output\result.json" --kind save -a MyApp --json
winapp ui picker select "C:\fixtures" --kind folder -a MyApp --json
winapp ui picker cancel -a MyApp --json

Ambiguity recovery only:

winapp ui picker list -a MyApp --json
winapp ui picker select "C:\fixtures\import.json" --kind open -w 393410 --json