Make limits runaway guards (supersedes #295, with review fixes)

[egeozin]

Supersedes #295: demegire's commits unchanged, merged with main (one conflict, the authoring guide's bindings text in private_workflows.py against #296: both kept), plus the review fixes below.

Before → after

Area	Limit	Before	After
Model output	Default max_output_tokens	4,096	32,000
	Keyword screening	5,000 (v6); 8,000 + retry 16,000 (v7)	32,000 + retry 64,000 (v8)
	Content plan, style capture, X style, X revise, growth plan, character design, site health	6,000–24,000	32,000
	Organic audit, paid ads, keyword seeds/review	2,000–24,000	unchanged
Model time	Default provider wait (TIN_LITE_LUNA_TIMEOUT)	90 s	600 s
Code workflows	timeout_seconds max / activity	60 s / 4 min	900 s / 20 min
	Model calls per run / per route	8 / 4	32 / 16
	Route input / output	32 KB (~9k tokens) / 4,096 tokens	256 KB (~70k tokens) / 32,000 tokens
	Model input per run	none (only call count)	soft gate: warning logged past 700 KB (~200k tokens)
	Cost estimate	1 token per input byte	1 token per 3 input bytes, every declared call at full size
	File read / output / model response	64 KB	1 MB
	File calls / bridge calls	64 / 32	256 / 128
	Service calls / response / wait	8 / 64 KB / 25 s	32 / 1 MB / 60 s
Codex	Contract for new runs	v4 session (1.05M context); v1/v3 for non-session work (8–64 requests, 0.1–2M tokens, 4–8k output, 128k context)	v5 everywhere: 256k context, compaction at 200k, 128k output; non-session runaway stops at 256 requests / 8M tokens
	Root session ceiling (default)	$5	$10 where the project and wallet can cover it, else $5
	Child inside a parent pool	$5	$5
	Declared sandbox max / design and task default	3,600 s / 900 s	7,200 s / 1,800 s
	Compute activities	2 h / 4 h	3 h / 6 h
Repository	Files / snapshot / tarball / per file	20,000 / 100 MB / 500 MB / 2 MB	100,000 / 250 MB / 1 GB / 10 MB
	Selected-file archive	30 MB / 5,000 files	100 MB / 20,000 files
	Content delivery activity	5 min	15 min
Memory	Sources / index	200 KB (fail when over) / 100 KB	700 KB each (~200k tokens); older sources dropped
Hosted defaults (new projects)	Per run / monthly / scheduled run	$10 / $10 / $10	$25 / $100 / $50

Replaces the approach of #274. Limits become runaway guards set well above what production runs use, instead of estimates that stop ordinary runs. Model context stays bounded: no single call over about 200k tokens, and no code-workflow run over about 200k tokens of model input.

Why

Over the last 30 days in production:

• Limits stopped about 13 runs and cut short several more.
• Keyword screening stopped at exactly 5,000 output tokens in a59cded8 and de1a4d3e, with 3.5k–4.2k of them reasoning. Six of 27 screening calls reached 90% of the cap.
• Codex sessions stopped at exactly $5: code map 2c31289d and email shortlist 05a4dfc2.
• The v3 contract stopped runs at 64 requests (signup walkthrough), at 2M tokens and on 128k-context 413s (product demo).
• The v1 contract refused competitor.watch and mention_backlinks with 422s.
• Three repository workflows were refused as too large, and project memory d711b841 failed on "memory sources exceed 200000 bytes".
• 75 scheduled runs and 4 starts were refused by the $10 hosted defaults.

Actual spend is far below the ceilings: audit at most $0.90, keyword plan at most $0.50, Codex p90 at most $2.30 except the code map. Billing charges verified usage, so the higher limits change worst-case reservations, not what a normal run costs.

Context sizes use the o200k tokenizer: 4.4–4.7 bytes per token on prose, code and JSON, about 3.5 on real workflow evidence. So 256 KB is about 70k tokens and 700 KB about 200k.

What changes

Model output and time

• The default max_output_tokens goes from 4,096 to 32,000, and the default provider wait from 90 s to 600 s (TIN_LITE_LUNA_TIMEOUT too).
• Activities wrapping calls that use the default wait get longer timeouts: memory, scan, weekly brief, visibility, answer page, character design, the organic panel, observe and brand steps, and keyword collect.
• Keyword plan v8 (catalog 0.8.0) screens with a 32,000-token cap and retries once at 64,000. No low reasoning effort. The worst-case run is $2.00, which fits the existing $2 floor. v2–v7 are unchanged.
• These steps now have a 32,000-token cap, each through its own versioning: content plan (content-editorial-v8), style capture, X style, X revise and growth plan steps. Character design and site health use module constants, not pinned definitions, so their caps change without a version bump; billing stays within each run's pinned $2 ceiling.
• Organic audit, paid ads, keyword seeds and keyword review are left alone. Raising them needs repricing, and none came near its cap in production.
• Receipts now record max_output_tokens, the provider's stop reason and output_truncated. An empty answer at its cap is labelled output_truncated instead of invalid_result.

Code workflows

• timeout_seconds maximum 60 → 900 s; the activity's timeout goes from 4 to 20 minutes.
• Model calls per run go from 8 to 32, and per route from 4 to 16. Per call: 256 KB input and 32,000 output tokens. Package code's model calls now pass the package's timeout to the provider.
• New soft gate: when a run's model calls pass 700,000 bytes of input in total (about 200k tokens), the call is still sent and Tin logs one warning for the run (run ID, workflow, step, bytes). Each receipt records its input bytes.
• The worst-case cost assumes 3 bytes per input token instead of 1, for every declared call at full size (largest package on Sol: $17.39, under the new $25 per-run default). Admitted runs keep their pinned terms; a quote issued before deploy returns stale_quote once.
• File reads, outputs and model responses go from 64 KB to 1 MB. File and bridge calls go from 64 and 32 to 256 and 128.
• Services allow 32 calls with 1 MB responses. The gateway waits 60 s per call, up from 25; managed Google calls 55 s, custom APIs 50/55 s.

Codex

• New admissions pin contract v5: a 256k context with compaction from 200k, 128k output per response and 8 MiB requests.
• Work that isn't session-funded moves from v1/v3 to v5 with runaway stops of 256 requests and 8M tokens. That covers Start here children, parent children, Studio, diagrams, design and tasks.
• Start here and its approved setup children stay included and Tin-funded through the same path as before.
• A root Codex run's default ceiling goes from $5 to $10 when the project's per-run limit, monthly remainder and wallet all cover $10; otherwise it stays $5, with today's refusals. Children inside a parent pool keep $5, so the traffic system pool is unchanged.
• The maximum declared sandbox time goes from 3,600 to 7,200 s. The design and task sandbox default goes from 900 to 1,800 s. Compute activities go from 2 h / 4 h to 3 h / 6 h, and proxy grants live up to 7,260 s.
• A relay rejection's cause is now saved on the attempt receipt as relay_rejection: context_bound, request_limit, token_limit and so on. The run's failure message names it.

Sizes and billing

• Repository limits: 100,000 files (was 20,000), 250 MB snapshots (was 100 MB), 1 GB tarballs (was 500 MB) and 10 MB per file (was 2 MB).
• Files a workflow reads out of a snapshot: 100 MB and 20,000 files (was 30 MB and 5,000). Up to 500 files fetched one by one when missing from the tarball.
• Snapshots are now assembled through temp files instead of in memory. The byte limits are set by the switchboard VM: an e2-small with about 4.7 GB of free disk.
• Content delivery's activity timeout goes from 5 to 15 minutes.
• Memory sources and the memory index are both capped at 700 KB, since that text goes into a single model call. Older sources are dropped instead of failing the run.
• New hosted projects get $25 per run, $100 a month and $50 per scheduled run. Existing saved limits are untouched; credits and Stripe are unchanged.

Deploy notes (not done in this PR)

1. Rebuild and verify the isolated, browser-API and Studio-API sandbox templates before deploying the switchboard.
   • v5's context settings live in sandbox/codex_api_config.py, and the runtime refuses a v5 procedure unless the image answers --check-v5. Tasks have no such check and fail on an old image before any model call.
   • The new 400,000-entry workspace bound is also in the image.
   • Keep a v4-capable image for runs already pinned to v4.
   • The opt-in live isolation probe has a new bounded_context scenario for checking a rebuilt image.
2. The forward-proxy host carries its own copy of proxy_grants.py. Reinstall it before any procedure declares more than 3,600 s; none does today.
3. If /etc/tin-lite/runtime.env sets TIN_LITE_LUNA_TIMEOUT, it overrides the new 600 s default.
4. The growth plan module's contract digest changes. A growth plan run started before the deploy whose first activity runs after it is refused.

Follow-ups

• Procedures that don't declare a timeout still get 900 s. That default is written into built-in definitions, so changing it needs per-workflow declarations and version bumps.
• Organic panel prep, character design and keyword collect have no heartbeat, so a crashed worker now waits up to their longer timeouts (85 min for panel prep) before Temporal retries. These stages should get heartbeats.
• Organic audit, paid ads and keyword seeds/review caps need repricing before they can rise.
• Existing hosted projects keep $10 / $10 / $10 until an admin raises them.

Tests

• Full suite locally, with a disposable Postgres 16 and the Temporal test server: 5,163 passed, 65 skipped.
• ruff format --check, ruff check, lint-imports, dump_catalog.py --check and validate-community (32/32) pass.
• New tests cover:
  • keyword plan v8;
  • the soft-gate warning (sent, logged once), and the run-maximum formula;
  • the raised validator maximums and the IPC buffers;
  • a pre-v5 quote being honoured, and a pinned v4 session still running;
  • the relay rejection cause being recorded;
  • the proxy grant lifetime;
  • memory dropping older sources;
  • the new hosted defaults.

Review fixes (on top of #295)

1. The $10 Codex ceiling no longer locks out existing projects or welcome-credit wallets.
   • The problem. A session run's admission estimate is its full ceiling. Projects still on the old $10/$10 defaults were therefore refused for every Codex procedure after any spend that month ("estimated at up to $10.00, which would exceed this month's $10.00 project limit"). A $10 welcome wallet was refused after its first charge ($9.60 < $10).
   • The rule. default_maximum(room) gives $10 when the room covers $10, otherwise the earlier $5. The room is the smallest of available credits, the per-run limit and the month's remainder.
   • Nobody new is refused. A project that can't cover $5 gets today's refusal.
   • Preview matches admission. Preview (quote) and admission compute the room the same way, so the preview shows what admission pins. A quote keeps its own ceiling.
   • Scope. Any root run on the default ceiling, including diagram and Studio runs, which had the same lockout. Children and per-validator ceilings are unchanged.
2. Repository snapshots can't exhaust the e2-small switchboard.
   • At most two snapshots are built at a time (IntegrationService semaphore around download, blob fallbacks and archive build).
   • The finished archive stays in its temp file and is streamed to E2B (files.write with a file object) instead of being read into memory.
   • technical_build_profile.archive_files can still hold up to 100 MB briefly, as before (capped by ARCHIVE_MAX_BYTES).
3. Project memory keeps the newest sources together. newest_sources stops at the first source that doesn't fit, so an older source is never kept while a newer one is dropped. A single source larger than the whole budget is skipped. memory_gardened records source_count (kept) and dropped_source_count.
4. The raised caps take effect. Style capture, X style and X revise waited 165–180 s, which bound long before 32,000 tokens.
   • Their model waits are now the provider default of 600 s, with 615 s in-step budgets.
   • A new 15-minute MODEL_STEP_TIMEOUT applies only to style_extract, x_style_extract and x_feedback_generate.
   • Activity options aren't workflow commands, and the existing replay test passes.
5. OpenAI failures inside a 200 stream are named. On 2026-10-01, three runs ended "Codex usage limit reached", and only a failed response status remained. A streamed response that ends failed now records relay_rejection as rate_limited (OpenAI's rate_limit_exceeded) or upstream_failed. The run says "OpenAI rate-limited this run's model calls." or "OpenAI failed one of this run's model requests." Only those codes are stored. incomplete isn't recorded, since Codex often recovers from it.
6. Docs. docs/forward-proxy-security.md now says grants last 7,260 s (3,660 before Oct 2), with the proxy host reinstall.

Left as is:

• The $50 scheduled-run default sits above the $25 per-run default. The per-run limit still applies to scheduled runs.
• Receipts can now hold 1 MB code model responses.
• Included Start here runs can cost Tin more in the worst case (256 requests and 8M tokens per child).

Deploy order (required)

1. Merge.
2. Rebuild tin-lite-codex-isolated, tin-lite-codex-browser-api and tin-lite-codex-studio-api from the merged main: uv run python sandbox/template.py --only <alias>.
   • New admissions pin v5, and the runtime refuses a v5 procedure on an old image. Tasks fail on an old image too, and nothing is charged.
   • The rebuild also activates Give isolated Codex its run's work order, and let "no change" finish cleanly #282's bridge change: a no_change outcome no longer needs a PR title.
3. Reinstall proxy_grants.py on the proxy host before any procedure declares more than 3,600 s. None does today.
4. Check that /etc/tin-lite/runtime.env doesn't pin TIN_LITE_LUNA_TIMEOUT.
5. Deploy the switchboard. A growth plan already running at deploy time is refused once.

Verification (this branch)

• New tests for the fixes fail on Make limits runaway guards: larger model, code, Codex and repository limits, with model input bounded near 200k tokens #295's head:
  • Codex ceiling: three cases, plus the can't-cover-$5 refusal;
  • snapshots: concurrency and streaming;
  • memory: newest kept and drops counted;
  • waits: wait against cap;
  • in-stream failures: rate-limited and upstream-failed.
• Full suite: 5,218 passed, 22 skipped against a disposable Postgres. Only the 2 macOS-only localtime tests fail.
• Checks: ruff, format, lint-imports, dump_catalog.py --check and validate-community (32/32) are clean.

🤖 Generated with Claude Code