Repository navigation
Tool-output limits for Grok Build, ZCode and Kimi Code (2.2.2) - #59
darko-mijic wants to merge 1 commit into
Conversation
What each harness Relay runs does to a large tool result, measured on 30 October 2026 with a synthetic file carrying random start, middle and end markers, and what Relay can change: Codex (handled in 2.2.1), Grok Build, ZCode and Kimi Code. scripts/tool-output-probe.mjs writes the probe file for checks after a harness upgrade. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Note 🤖 Claude Opus 5.5 responding on behalf of darko-mijic Closing as folded into #60. This draft's head commit Two items from the 2.2.1 trial are not fixed by #60: Codex setup reports no real binary for a wrapper script, and |
Relay 2.2.1 gives every Codex job a tool-output budget and carries it into code mode. This draft collects what is known about the other three harnesses, so 2.2.2 can give Grok Build, ZCode and Kimi Code jobs the whole evidence too. It holds the findings and a probe; the implementation follows once the open decisions below are made.
What is in this draft
docs/harness-output-limits.md: per harness, what a 128 KB synthetic result became, every limit with its default, cut position and setting, and what Relay can change. Measured on 30 October 2026 with random start, middle and end markers; each claim is marked measured, on disk, official or recalled in the source reports, and official sources are linked.scripts/tool-output-probe.mjs: writes the probe file and prints its markers, for a check after a harness upgrade.Findings
GROK_MAX_MCP_OUTPUT_BYTESper job. Shell:[toolset.bash] output_byte_limitin the home'sconfig.tomlonly[read] default_max_charsin the home'sconfig.tomloutput_pathUnlike Codex before 2.2.1, none of the three discards the cut text: each saves it and names the file. The risk is a job that reasons from the preview.
Proposed changes
GROK_MAX_MCP_OUTPUT_BYTESin the job's environment, with Codex's presets in bytes (deep262,144 for a 256k window).developerInstructions; the other three have no confirmed channel (see decisions).config.toml(Grok's shell preview, Kimi's file reads): document them, or manage them for jobs-only homes.From a 2.2.1 trial
Six Codex jobs (five at once, gpt-6.1-sol and gpt-6-astra at high effort, a configured budget of 65,536, no per-job override) all completed with one attempt each and no recoveries, stalls or relay violations. Findings:
Warning: truncated output: models often setmax_output_tokensfar below the budget, some to 2,000. Three whole-file reads (about 93k, 101k and 117k tokens) exceeded 65,536, and one lane read about a quarter of a large diff. Proposed: make the instruction firmer, asking for the full budget on everyexecthat may print file contents, a diff, a log or search results, never less, and telling the model to read large files and diffs by line range and to read any part a truncation notice names before relying on the result.cua_repl) is not anmcp_serversentry, sojob-mcp-offcannot turn it off. Open: whether a-coverride or feature flag can.binaryIsScript: truewithrealBinary: null; Grok's probe names the real binary, Codex's does not.Decisions needed
session/new_metaandsession/set_config_optioncarry only model, effort or mode on Grok and Kimi, and ZCode's session takes none found so far. Options: prefix the brief with a labelled Relay note (changes "the brief reaches the job byte for byte"), use a protocol field if research finds one, or rely on each harness'sAGENTS.md(writes into the working directory).config.tomlkeys for a jobs-only home (grok-home: jobs,kimi-home: jobs), or only document them?Open questions
GROK_CONFIGcan carryoutput_byte_limit; selecting the 500k window over ACP.truncate-strategy tools keep their full output.output_token_limit, and whether a resumed thread takes a new budget.🤖 Generated with Claude Code