Skip to content

D5: serve over device KV (4096) returns 500 — refuse with 400 'context exceeds N'; --context-length sizes KV for gate-named cells #4616

Closed

Description

@noahgift

OPERATOR ruling D5 (ruling-3715, 2026-09-28), verbatim: "Serve > 4096 tokens → rc500: a 500 is a crash-class lie. Minimum for rc.2: over-length requests return a clear 4xx ("context exceeds 4096"), and serve cells > 4096 are withdrawn by data with a warning, UNLESS gate-named (then fixed: KV size = the model's context)."

Child of #3715: its serve × consumer-context cells above 4096 tokens fail with 500.

Defect: dense CUDA apr serve sends a turn longer than the device KV (max_kv_len(), 4096) to generation. Generation fails with UnsupportedOperation, and the server returns HTTP 500.

Fix (PR #4614):

  • serving_context = the model's context, capped by the device KV.
  • The chat, completions and stream handlers refuse an over-length turn before generating, with 400 "context exceeds N".
  • ContextLimitExceeded maps to 400.
  • --context-length N sizes the KV for gate-named cells.

Acceptance:

Activity

  1. noahgift commented on Oct 4, 2026

    @noahgift
    MemberAuthor

    Fixed on main by #4655 (merged 2026-10-02). #4614 is closed. (ISSUE-TREE-001 cleanup, C297 §4, aprender-wtix-even)

  2. added a commit that references this issue on Oct 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions