Skip to content

fix(api): account for worker-interrupted usage and block unconfirmed spend - #452

Open
shuntianyifang wants to merge 3 commits into
theam:mainfrom
shuntianyifang:fix/worker-interrupted-usage
Open

shuntianyifang wants to merge 3 commits into
theam:mainfrom
shuntianyifang:fix/worker-interrupted-usage

Conversation

@shuntianyifang

@shuntianyifang shuntianyifang commented Oct 7, 2026 •

Copy link
Copy Markdown

When a worker died during an agent call, recovery marked the turn failed without recording provider usage. Project spending stayed understated and later calls could pass a budget that should have blocked them.

This change retains per-turn numerical usage in the workspace independently of the worker's observation connection, reconciles confirmed charges once, and records incomplete spending as unpriced. Enabled budgets enter unconfirmed and block new agent and title-generation calls until final retained usage is reconciled. Raising the limit, resolving attention, or a calendar rollover does not clear uncertainty. Recovery also handles missing bills left by older workers, without waking compute or rerunning a model. Confirmed charges count when settled, consistent with live-worker accounting.

API/OpenAPI/SDK types, spend presentation, and operator documentation describe the new state. No database migration or dependency change is required.

Closes #399.

Validation

Follow-up validation at 45dc057 (2026-10-07): pnpm test:critical passed 865 tests across 79 isolated processes, no skips, using a fresh dedicated PostgreSQL 16.15 database. API typecheck and repository lint passed. Two additional regressions verify that every unknown bill must settle before admission resumes, and competing final reports settle exactly once without replay changing the charge. Only test code changed after the full validation below.

Full validation at ea4da90 (2026-10-07), against upstream main 38ec556:

  • Standard pnpm verify passed lint, typecheck (14 tasks), clean full workspace build (8 tasks, no cache hits), 863 critical tests and 320 remaining tests, unused-reference checks, and both guards. No normal-suite skips. The final dependency audit exits nonzero; see below.
  • Critical integration tests use a fresh PostgreSQL 16.15 database / Linux / Node 24.18.1 / pnpm 11.28.2. Earlier PostgreSQL 18 results are historical.
  • 46 new backend tests cover durable usage, lost observers, partial and malformed evidence, unavailable/replaced workspaces, cross-tenant/project rejection, leases, replay, late settlement, legacy missing bills, unknown phases, title admission, and month boundaries. External engines use deterministic local CLI fakes and FakeWorkspaceRuntime; no live credentials or provider calls.
  • Full workspace E2E: 11 passed, no skips (2 real Docker tests and 9 deterministic journey tests), using official Linux amd64 runner ghcr.io/theam/facility/runner@sha256:9aad7af6b96b36c31f2e20806c19fea08e066af9d9aa412a961b4c3c5c1032a0.
  • The new production-workspace regression verifies both engine journals after the observer exits, partial Claude usage, retained usage across compute replacement, and no automatic wake on read. The other Docker test covers nested Compose, authenticated preview, Chromium screenshot/DOM/trace, orphan cleanup, persistent storage, and backup/restore.
  • Frontend spend tests: 21 passed; documentation contracts: 11 passed (included in remaining tests). Diff whitespace check passed.
  • Replacing recovery with the original upstream implementation makes the regression fail at the missing usage row; the fixed implementation passes. All 579 tracked files match the tested commit byte-for-byte. Runner preview script hash matches source; Node 24.20.0, Chromium 154.0.8037.92, Claude Code 2.1.284, and Codex 0.144.6 were checked in the immutable image.
  • A repeated run on the reused database hit an existing project-list pagination assertion; final full-suite results use a fresh dedicated test database and retain the original assertion.
  • Early full-build attempts hit unavailable font proxy transport and exit 139; the successful clean build used the original font downloads through the existing host proxy. Exporting Git with Windows autocrlf initially caused script failures; final validation uses a canonical LF Git export. No fonts, assertions, source behavior, or dependency versions were changed to obtain passing results.

Dependency audit remains failing

The lockfile, manifests, and audit policy match upstream 38ec556; this PR adds no exceptions. The current audit reports 3 unignored findings, plus 3 existing ignored high findings:

Limits and coordination

  • Missing final evidence cannot reconstruct the true provider bill. It remains a lower bound and blocks enforced-budget calls. There is no manual billing reconciliation endpoint; a budget administrator can explicitly disable enforcement, accepting incomplete spend. Deleting workspace evidence does not clear the block.
  • feat(api): reserve monthly budget before a turn runs #440 addresses concurrent budget reservations; this PR does not incorporate that implementation. Its current head c7473f9 conflicts with this branch in README, costs.ts, story service, and titles.ts. Both heads require an explicit resolution; choosing either side wholesale is unsafe. See the tested integration requirements below.
  • Live Vercel/provider calls and other image architectures were not tested. Audit exceptions and lockfiles are unchanged. Local results do not claim upstream CI success; the PR remains draft.

Compatibility with #440

A separate local combined candidate passed 79 related API tests across six files, no skips, and API typecheck, with the same local PostgreSQL 16.15 / deterministic engine setup. This is validation of an explicit conflict resolution, not of the unmodified heads. The required resolution is:

  • Keep this PR's all-history unpriced-row detection and budget_usage_unconfirmed policy alongside feat(api): reserve monthly budget before a turn runs #440's measured usage + open holds + settled title totals.
  • Check uncertainty inside reserveTurn after locking the budget row, using the claim transaction. Check it in reserveTitle before either creating or reusing an existing hold; propagate an unconfirmed outcome to title fallback.
  • Keep this PR's scoped conditional upsert and complete flag. Write measured/unknown usage and release the turn reservation in one transaction. Final usage must replace the placeholder once; feat(api): reserve monthly budget before a turn runs #440's conflict-ignore insertion cannot do that.
  • Recovery must atomically book uncertainty and release the reservation before admitting successors. Proven pre-engine failures release without unknown usage; keep cancellation's hold release.

Four combined-candidate regressions verify unknown recovery blocks turn/title reservation after releasing the hold, title hold reuse cannot bypass uncertainty, confirmed journal replaces a hold exactly once, and pre-engine recovery releases without a bill. Removing the admission guards causes two tests to fail; restoring conflict-ignore causes the late-final-report regression to fail. The normal candidate suite has no skips; filtered negative controls intentionally skip unrelated tests. The candidate and its extra reservation tests are retained separately and are not part of this PR's code or migration scope. Revalidate the combined implementation when either PR changes.

Upstream CI and review status

At 45dc057, the pull-request ci and conventional-subject workflow runs report action_required; the conventional-subject run has zero jobs. The account has upstream pull: true, push: false, maintain: false, admin: false. Maintainer approval is required before upstream workflows can execute. There are no reviews or comments to resolve yet. The new test-only commit does not claim successful remote CI, and this PR remains draft. No workflow permission changes or audit exceptions were made.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Turns interrupted by a dead worker are never counted against the project budget

1 participant