Skip to content

Tool-output limits for Grok Build, ZCode and Kimi Code (2.2.2) - #59

Closed
darko-mijic wants to merge 1 commit into
mainfrom
docs/harness-output-limits
Closed

darko-mijic wants to merge 1 commit into
mainfrom
docs/harness-output-limits

Conversation

@darko-mijic

@darko-mijic darko-mijic commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Relay 2.2.1 gives every Codex job a tool-output budget and carries it into code mode. This draft collects what is known about the other three harnesses, so 2.2.2 can give Grok Build, ZCode and Kimi Code jobs the whole evidence too. It holds the findings and a probe; the implementation follows once the open decisions below are made.

What is in this draft

  • docs/harness-output-limits.md: per harness, what a 128 KB synthetic result became, every limit with its default, cut position and setting, and what Relay can change. Measured on 30 October 2026 with random start, middle and end markers; each claim is marked measured, on disk, official or recalled in the source reports, and official sources are linked.
  • scripts/tool-output-probe.mjs: writes the probe file and prints its markers, for a check after a harness upgrade.

Findings

Harness Shell result Settable by Relay Full output kept?
Grok Build 1.0.70 20 KB, middle cut MCP: GROK_MAX_MCP_OUTPUT_BYTES per job. Shell: [toolset.bash] output_byte_limit in the home's config.toml only yes, a log path
ZCode (runtime 0.17.0) 2 KB preview of the start nothing: budgets are constants yes, a log path
Kimi Code 2.3.1 4 KB start and 1 KB end file reads only: [read] default_max_chars in the home's config.toml yes, output_path

Unlike Codex before 2.2.1, none of the three discards the cut text: each saves it and names the file. The risk is a job that reasons from the preview.

Proposed changes

  1. Codex, after the 2.2.1 trial: a firmer code-mode instruction (see below).
  2. Grok MCP budget per job: set GROK_MAX_MCP_OUTPUT_BYTES in the job's environment, with Codex's presets in bytes (deep 262,144 for a 256k window).
  3. An instruction for saved output: tell Grok, ZCode and Kimi jobs to read a saved output path in pages before concluding. Codex carries its instruction in developerInstructions; the other three have no confirmed channel (see decisions).
  4. Settings that live in a home's config.toml (Grok's shell preview, Kimi's file reads): document them, or manage them for jobs-only homes.
  5. The probe as a smoke step: run it on each backend after an upgrade and compare markers.

From a 2.2.1 trial

Six Codex jobs (five at once, gpt-6.1-sol and gpt-6-astra at high effort, a configured budget of 65,536, no per-job override) all completed with one attempt each and no recoveries, stalls or relay violations. Findings:

  • Models ask for small budgets. The code-mode instruction reached every job, yet 30 results in 23 of 67 calls carried Warning: truncated output: models often set max_output_tokens far below the budget, some to 2,000. Three whole-file reads (about 93k, 101k and 117k tokens) exceeded 65,536, and one lane read about a quarter of a large diff. Proposed: make the instruction firmer, asking for the full budget on every exec that may print file contents, a diff, a log or search results, never less, and telling the model to read large files and diffs by line range and to read any part a truncation notice names before relying on the result.
  • Codex's bundled computer-use plugin (cua_repl) is not an mcp_servers entry, so job-mcp-off cannot turn it off. Open: whether a -c override or feature flag can.
  • Setup and wrapper binaries: a Codex binary that is a wrapper script reports binaryIsScript: true with realBinary: null; Grok's probe names the real binary, Codex's does not.

Decisions needed

  • Instruction channel. ACP's session/new _meta and session/set_config_option carry only model, effort or mode on Grok and Kimi, and ZCode's session takes none found so far. Options: prefix the brief with a labelled Relay note (changes "the brief reaches the job byte for byte"), use a protocol field if research finds one, or rely on each harness's AGENTS.md (writes into the working directory).
  • Home configuration. Relay does not write a user's configuration. Should it manage config.toml keys for a jobs-only home (grok-home: jobs, kimi-home: jobs), or only document them?

Open questions

  • Grok: auto-compaction at 80% (catalog) or 85% (guide); whether file reads also have a 25k-token cap; whether GROK_CONFIG can carry output_byte_limit; selecting the 500k window over ACP.
  • ZCode: which tools several budget blocks in the bundle belong to; whether truncate-strategy tools keep their full output.
  • Kimi: server-side limits of its hosted search and fetch; the cut position of the ACP client terminal's 4 MiB limit.
  • Codex, beyond 2.2.1: the 1 MiB capture buffer per command, per-MCP-tool output_token_limit, and whether a resumed thread takes a new budget.

🤖 Generated with Claude Code

What each harness Relay runs does to a large tool result, measured on 30 October 2026 with a
synthetic file carrying random start, middle and end markers, and what Relay can change: Codex
(handled in 2.2.1), Grok Build, ZCode and Kimi Code. scripts/tool-output-probe.mjs writes the
probe file for checks after a harness upgrade.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Oct 30, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@darko-mijic

Copy link
Copy Markdown
Contributor Author

Note

🤖 Claude Opus 5.5 responding on behalf of darko-mijic

Closing as folded into #60. This draft's head commit a0616ca is an ancestor of release/2.2.2, and #60 carries docs/harness-output-limits.md and the probe, corrected against the installed harnesses. The two decisions left open here are settled there: no instruction channel for Grok, ZCode or Kimi, and no managed home configuration. The branch stays in place.

Two items from the 2.2.1 trial are not fixed by #60: Codex setup reports no real binary for a wrapper script, and job-mcp-off cannot turn off cua_repl (documented in #60).

@darko-mijic
darko-mijic deleted the docs/harness-output-limits branch November 3, 2026 14:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant