Skip to content

Define provider-native Android text injection for remote sessions #539

@thymikee

Description

@thymikee

Problem

Android text entry currently depends on different mechanisms across local and remote execution. Local sessions use Android platform fallbacks, while remote sessions may have a stronger provider-owned text primitive available. Without an explicit capability boundary, fill and type can drift in behavior for IME-owned input, long ASCII strings, Unicode text, and React Native TextInput verification.

Proposed direction

Introduce a provider-neutral Android text injection capability that remote session adapters can implement. agent-device should choose text backends in a clear order:

  1. Provider-native text injection when the active remote session exposes it.
  2. Device-side helper or controlled local text backend when available.
  3. Chunk-safe ADB shell input as an ASCII-only fallback.
  4. Fast, non-retryable IME ownership failure when text is being captured by the keyboard/IME instead of the target app and no stronger backend intentionally bypasses it.

Acceptance criteria

  • Android text entry has a provider-neutral capability contract for remote sessions.
  • fill and type prefer provider-native text injection when the remote session exposes it.
  • Local Android helper/ADB fallback behavior remains separate from remote provider behavior.
  • Diagnostics identify only generic backend kinds such as provider-native, helper, or adb-shell.
  • Agent-facing docs do not recommend raw adb, clipboard, or paste as manual fallbacks.
  • Public code, tests, docs, and issue text remain provider-neutral.

Testing

  • Unit tests for backend selection and diagnostics.
  • Remote-session adapter tests proving provider-native injection is preferred when advertised.
  • Local Android test-app coverage for normal keyboard, IME-owned focus, long ASCII, spaces/percent signs, and Unicode handling.
  • React Native test-app coverage that verifies text reaches TextInput and triggers expected app-side state.
  • Manual New Expensify Android verification for composer text entry with a keyboard/IME overlay state enabled.

Related: #530, #531, #532

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions