fix(api): stop rejecting long chat histories by default

[nguyenha935]

Summary

admitChatStructure() rejects any POST /v1/chat/completions whose messages array is longer than 800 with a terminal 413 chat_history_too_large. The rejection happens before compression, translation, and provider dispatch, and without consulting any memory signal. This PR makes that cap opt-in (default 0 = disabled) and leaves every other admission control untouched.

Where the 800 came from

The threshold entered the codebase as one deployment's local operating policy. #8276 states it explicitly:

Current local thresholds are deployment policy, not proposed universal defaults: 200 messages, 64 tools, conservative 32k-token estimate, 800-message hard history cap […]

The originating incident (#7849) was a V8 OOM in a 16 GiB container running with --max-old-space-size=12288. #8296 then shipped those deployment-specific numbers as universal defaults, so every OmniRoute instance — including ones with far more headroom, or far less — inherited a ceiling calibrated for a single environment.

Why a message count is the wrong gate

1. It pre-empts the component that exists to solve the problem. In the reporter's own case the compression pipeline reduced 749,942 tokens to 262,052. The cap rejects before that pipeline runs, so a conversation OmniRoute could have served is refused instead.
2. The metric is partly manufactured by OmniRoute. messages.length is read after protocol translation. A Responses- or Anthropic-format request whose input[] holds a modest number of turns expands into many more messages[] entries during translation, so a client can cross a limit it never approached in its own representation.
3. 413 is terminal. There is no retry a client can perform that makes it succeed. The same module already has the right shape for capacity problems — a retryable 503 with Retry-After — and the message cap bypasses it.
4. It is not effective as OOM protection. The guard is wired only into src/app/api/v1/chat/completions/route.ts; src/app/api/v1/responses/route.ts and src/app/api/v1/messages/route.ts have no admission wiring at all. The same process and the same heap are reachable through those routes with no message-count limit, so the cap does not close the hole it was added for while it does refuse legitimate traffic on the one route it covers.
5. It ignores the memory state it is nominally protecting. checkHeapPressureGuard() (open-sse/utils/heapPressure.ts, used by open-sse/handlers/chatCore.ts) already auto-calibrates to 85% of the actual V8 heap ceiling and adapts across 1 GB / 2 GB / large hosts. The message cap consults none of it and is a fixed number regardless of available heap — the same class of mistake as the fixed 200 MB threshold that caused the v3.8.8 "resource pressure" outage documented in that module's own comment.

Observed impact

On an instance with 15 GB RAM, no cgroup limit, no --max-old-space-size, and ~1.66 GB RSS — nowhere near the OOM condition the cap was designed for — 81 requests over a ~2h45m window were rejected with this 413. All of them arrived through a gateway that translates Responses-format traffic into /v1/chat/completions, i.e. the translation-inflation path in point 2. Rejected bodies were 819 KB–1.06 MB, well under the 50 MB byte cap; successful requests on the same route reached 652 KB.

Change

CHAT_HARD_MAX_MESSAGES now defaults to 0, and the check is skipped when it is not positive:

const maxMessages = options.maxMessages ?? CHAT_HARD_MAX_MESSAGES;
if (maxMessages > 0 && messages.length > maxMessages) {
  return { admit: false, response: structuralRejectionResponse(413, maxMessages) };
}

Deployments that want a hard ceiling set OMNIROUTE_CHAT_HARD_MAX_MESSAGES to a positive value and get byte-identical previous behaviour, including the same 413 body and reason: "message_limit".

Nothing else about admission changes. Large conversations are still classified heavyweight by OMNIROUTE_CHAT_HEAVY_MESSAGE_COUNT / OMNIROUTE_CHAT_HEAVY_TOOL_COUNT / the conservative token estimate, still require an atomic heavyweight lease, still get a retryable 503 + Retry-After when concurrency capacity is occupied, and are still subject to the 50 MB actual-byte cap and the heap-pressure shed. The concurrency bound plus the heap-pressure shed are what actually keep the allocation-heavy path from exhausting the heap, and both remain in force for every request.

Related Issues

• fix(backend): long-context compression can exhaust V8 heap and terminate OmniRoute #7849 — the originating V8 OOM report (16 GiB container, --max-old-space-size=12288)
• feat(backend): structure- and cgroup-aware long-context admission queue #8276 — states the thresholds are deployment policy, not universal defaults
• fix(backend): add structure-aware chat admission #8296 — shipped them as universal defaults

Validation

• Targeted unit suite: tests/unit/chat-body-admission.test.ts — 25/25 pass
• Deployed to a production instance and exercised end to end: a 1,001-message POST /v1/chat/completions now returns 200 through two independent providers (prompt_tokens 10,129 and 5,974) where it previously returned 413 chat_history_too_large. No 413 chat_history_too_large in the service log since the deploy.
• npx prettier --check — the two changed TypeScript files report the same formatting deviations on the pristine base commit as they do with this patch applied (1 hunk in the middleware, 3 in the test file), so this change introduces none. There is no prettier gate in CI.
• npm run lint — cannot run in this environment (eslint-config-next unavailable); deferred to CI
• npm run test:unit — deferred to CI
• npm run test:coverage — deferred to CI

Tests Added Or Updated

tests/unit/chat-body-admission.test.ts:

• new — no history cap is enforced by default; long conversations are admitted: asserts CHAT_HARD_MAX_MESSAGES === 0 and that a 5,000-message conversation is admitted through heavyweight capacity rather than rejected.
• new — an uncapped oversized conversation still yields to occupied heavyweight capacity: the same oversized conversation gets a retryable 503 + Retry-After: 1 + chat_admission_busy when the lease is held, proving backpressure still applies without the cap.
• new — maxMessages: 0 explicitly disables the history cap.
• updated — the existing hard-cap test is retitled an opt-in history cap still returns the structured compact-required 413 and keeps asserting the 413 / chat_history_too_large / reason: "message_limit" contract via an explicit maxMessages, so the opt-in path stays locked.

Coverage Notes

src/shared/middleware/chatBodyAdmission.ts is the only production file changed; the new and retained tests cover both branches of the guard (cap disabled → admit, cap configured → 413) plus the backpressure path.

Reviewer Notes

This is deliberately scoped to the default value and the guard condition — it does not attempt to redesign structural admission, add heap-awareness to admitChatStructure(), or extend admission to /v1/responses and /v1/messages. Points 4 and 5 above describe real gaps but are separate changes.

Suggested follow-ups, if wanted: wire admission into the other two chat-ingress routes so the OOM protection is actually process-wide, and let structural admission consult HEAP_PRESSURE_THRESHOLD_MB so heavy-request classification scales with the live heap ceiling instead of fixed counts.

[nguyenha935]

Closing: not yet validated on a live deployment. Will reopen after runtime verification.

[nguyenha935]

CI note: the 4 red checks are pre-existing on the base branch, not caused by this PR

Unit Tests fast-path (2/4), (3/4), (4/4) and Build (advisory) are failing. They are not attributable to this change. Evidence:

1. The failing tests are byte-identical to those on an unrelated open PR.

Comparing this PR's last completed run (31024835790) against #9513 (fix/classify429-gemini-retry-delay, run 31024808627, attempt 1) — same three shards fail, with the same failing-test sets, diff-identical:

shard	failing tests
2/4	13 tests: VB-S01/S03/S07/S10/S12/S12b/S13 (vision bridge), D1/D2/Bonus (IP filter), claude-web validator: 429 → valid, gemini-web validator: 302 redirect…, repro-8956
3/4	9 tests: the 8 callVisionModel … cases + execute (non-stream) parses bare JSON reply into OpenAI tool_calls
4/4	1 test: issue #8189: 'always' mode is unaffected…

#9510 shows the same three shards red as well.

1b. The failures are deterministic, not flaky. The three shards were re-run on the same head. All three failed again with diff-identical failing-test sets, and this PR's own four tests passed again in shard 3/4. Same inputs, same outputs — this is a reproducible state of the branch, not scheduling noise.

2. None of the failing tests touch anything this PR changes.

This branch's own commits (2cb7567d6..5da13b2b1) modify exactly five files:

.env.example
changelog.d/fixes/9494-chat-history-cap-opt-in.md
docs/reference/ENVIRONMENT.md
src/shared/middleware/chatBodyAdmission.ts
tests/unit/chat-body-admission.test.ts

tests/unit/chat-body-admission.test.ts is the only test file in the repo that references chatBodyAdmission / CHAT_HARD_MAX_MESSAGES / admitChatStructure. It lands in shard 3/4 and every one of its cases passes there, including all four added/updated by this PR:

✔ no history cap is enforced by default; long conversations are admitted
✔ an uncapped oversized conversation still yields to occupied heavyweight capacity
✔ maxMessages: 0 explicitly disables the history cap
✔ an opt-in history cap still returns the structured compact-required 413

So shard 3/4 is red around this change, not because of it. There is no import path from the changed middleware to the vision bridge, the IP filter, the provider validators, or the Qwen config.

3. The shard-3 failures trace to #8430 on the base branch.

Every callVisionModel failure is the same error:

Error: No vision-capable provider connected, cannot process image request
  at callVisionModel (src/lib/guardrails/visionBridgeHelpers.ts:224:11)

That throw was added by 7e55abbc4 — "fix(vision-bridge): do not select unreachable describe-model when no vision provider is connected (#8430)", 2026-08-04 — which is an ancestor of the current release/v3.8.50 head (9fcefcce9f77). The commit added the guard to visionBridgeHelpers.ts and updated visionBridgeRouter.test.ts and vision-bridge-preserve-on-failure-4012.test.ts, but not visionBridgeHelpers.callVisionModel.test.ts, whose fixtures still don't connect a vision provider. So those tests now hit the new guard before reaching the fetch mock they were written against — which is exactly why throws on HTTP error reports The input did not match /Vision API error 500/.

4. "The base branch is green" is not evidence to the contrary.

Base-branch CI runs report success, but they do not run these tests. In run 31024782001 on 9fcefcce9f77, every test job is skipped — Unit Tests (…/8), Vitest, Integration Tests, E2E Tests, Coverage, Security Tests, Lint, Build. Only Change Classification and CI Dashboard actually executed. A green base run here means the fast-path skipped the suite, not that the suite passes.

I've deliberately left these alone rather than fixing them inside this PR — #8430's test-fixture gap is a separate change and folding it in would mix two unrelated concerns. Happy to open a follow-up for the callVisionModel fixtures if you'd like.

One note on scope overlap: 157297ca6 merged release/v3.8.50 into this branch, bringing in a61020153 feat(admission): add adaptive overload and pressure controls and 8ca40e797 feat(api): wire shared admission across LLM routes. Those add a new adaptive-admission subsystem under open-sse/services/admission/ and wire admission across the LLM routes — which addresses point 4 of the description (admission was previously only on /v1/chat/completions). This PR's change is still needed and does not conflict: it is the default value of CHAT_HARD_MAX_MESSAGES and the guard condition in src/shared/middleware/chatBodyAdmission.ts, which the new subsystem doesn't alter. If anything, wiring the structural guard across more routes makes an 800-message default more consequential, not less.