Skip to content
This repository was archived by the owner on Oct 13, 2026. It is now read-only.
This repository was archived by the owner on Oct 13, 2026. It is now read-only.

Start Chromeria when T3 is down instead of only waiting #140

Closed

Description

@lukemaj

Outcome

When a job finds T3 unreachable, the runner starts Chromeria itself and continues once the server answers, instead of only waiting (up to 24 hours) for a person to open it.

Why (Elon record)

  • Wanted result: a quit or crashed Chromeria costs a job about a minute, not hours.
  • Evidence:
    • Wait for T3 instead of blocking when Prism is unreachable #136 (0.39.0) made jobs wait up to 24h for T3 without starting it.
    • This Mac already starts Chromeria on demand: ~/.local/bin/agent-observer-review runs /usr/bin/open -g -a Chromeria when 127.0.0.1:3773 is down, then waits up to 3 minutes (runbook cavallo/SERVICES.md, Agent Observer review).
    • The install-on-quit script (/tmp/chromeria-install.sh) waits for the app to quit, sleeps 3 seconds, deletes and replaces /Applications/Chromeria.app, then reopens it itself. Relaunching inside that window would delete a running app.
  • Cuts: no launchd service, no health daemon, no new Chromeria code.
  • Smallest surviving solution: inside the existing wait loop (_await_prism in runner/controller.py), once T3 has been unreachable for 60 continuous seconds, run the launch command once, and again every 5 minutes while it stays down. Keep the existing wait and 24h cap as the backstop.

Acceptance criteria

  • The launch command defaults to /usr/bin/open -g -a Chromeria on macOS. T3_LAUNCH_CMD overrides it, and an empty value turns launching off. On other platforms the default is off.
  • No launch before 60 seconds of continuous unreachability. Then one launch, repeated every 5 minutes while still unreachable. A launch failure is recorded and never ends the wait.
  • Each launch is recorded as a ledger event (launching_t3) that shows in status.
  • T3 answering with an error, an empty Prism list, and cancel all behave exactly as in 0.39.0.
  • Tests use a fake clock, a fake client and a fake launcher: launch after 60s, not before; relaunch every 5 minutes; launch disabled when empty; the job continues once the read succeeds.

Non-goals

  • Starting T3 on other machines, or through anything other than one configurable command.
  • Changing the install script.

Contribution to the Objective

Jobs reach an open PR across Chromeria quits and crashes without waiting on a person.

Proof

Full suite (uv run --with pytest python3 -m pytest or scripts/test.py), CI proof green.

Activity

  1. lukemaj commented on Sep 28, 2026

    @lukemaj
    ContributorAuthor

    PR: #141 (release 0.40.0, not merged).

    • Launch runs at the first reread after 60s unreachable (75s with the existing backoff), then every 5 minutes (375, 675, ...). Each launch is a launching_t3 event in status (at, command, result); a failed launch never ends the wait. T3_LAUNCH_CMD overrides, empty turns it off; default /usr/bin/open -g -a Chromeria on macOS, off elsewhere; no shell, 15s timeout.
    • Tests: uv run --with pytest python3 -m pytest -q 328 passed, 30 subtests passed; scripts/test.py 328 OK; CI proof pass; versionctl release-check OK for v0.40.0 at 619b92b.
    • Unchecked: no live Chromeria launch through the runner (the app was in use). Tests replace the launcher with a fake and set T3_LAUNCH_CMD="".
  2. added a commit that references this issue on Sep 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions