Repository navigation
fix(bench): qwen35 bench forgets the session prefix before every turn (#4445) - #4577
Conversation
…#4445) `apr bench` on a Qwen3.5 GGUF refused every run with warmup+iterations > 1: the warmup and each timed turn repeat the same prompt on one Session, and since #4214 the session resumes a repeated prompt from its checkpoint, so turn 2+ reported `reused != 0` and the prefill guard (correctly) refused. Session::forget_prefix drops the held positions and the checkpoint; the bench calls it before each warmup and each timed turn, so every turn prefills whole. Proof: bench_on_qwen35_times_session_generate (warmup 1, iterations 2, real Qwen3.5-0.8B) goes red without the calls with the issue's own error ("16 prompt tokens were served from a cached prefix") and green with them; forget_prefix_makes_the_same_prompt_prefill_whole_again goes red when forget_prefix keeps the checkpoint. Refs #4445 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#4445) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
quorum-review (AD-04): NOT agreed (auto_merge: checked=true was_armed=false disarmed=false) {
"ticket": "PMAT-4445",
"head": "4877908462588896d4e313d1ba2fbc44610dba3f",
"width": 3,
"executor": "agy",
"agreed": false,
"auto_merge": {
"checked": true,
"was_armed": false,
"disarmed": false,
"note": "auto-merge not armed"
},
"lanes": [
{
"lane": 1,
"verdict": "PASS",
"findings": 2
},
{
"lane": 2,
"verdict": "FAIL",
"findings": 3
},
{
"lane": 3,
"verdict": "FAIL",
"findings": 1
}
]
} |
…ide resolution Quorum round 1 on #4577 (1 PASS / 2 FAIL) held the diff to criteria written for the car version: a repeated-prompt test and a checkpoint reset main cannot have. The criteria now say what this PR does on main, that it is inert until the car lands, and exactly what the car merge adds (9c34ca6). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
quorum-review (AD-04): three PASS — agreed (auto_merge: checked=true was_armed=false disarmed=false) {
"ticket": "PMAT-4445",
"head": "874891dfe80afe9c0409c1b3beda4071d1713455",
"width": 3,
"executor": "agy",
"agreed": true,
"auto_merge": {
"checked": true,
"was_armed": false,
"disarmed": false,
"note": "auto-merge not armed"
},
"lanes": [
{
"lane": 1,
"verdict": "PASS",
"findings": 0
},
{
"lane": 2,
"verdict": "PASS",
"findings": 0
},
{
"lane": 3,
"verdict": "PASS",
"findings": 0
}
]
} |
|
§13.11 rung 1 — quorum shadow verdict Shadow mode: this records a verdict and merges nothing. A refusal |
…4445) check_roadmap_fragment_required.sh [run] was RED on x86-main: the fragment was added without `make roadmap-aggregate`. The aggregate diff is the one PMAT-4445 entry. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
quorum-review (AD-04): NOT agreed (auto_merge: checked=true was_armed=false disarmed=false) {
"ticket": "PMAT-4445",
"head": "51c41b92e2ee6bc54867fd87885632a842ecc040",
"width": 3,
"executor": "agy",
"agreed": false,
"auto_merge": {
"checked": true,
"was_armed": false,
"disarmed": false,
"note": "auto-merge not armed"
},
"lanes": [
{
"lane": 1,
"verdict": "NO-VERDICT",
"findings": 0
},
{
"lane": 2,
"verdict": "PASS",
"findings": 0
},
{
"lane": 3,
"verdict": "PASS",
"findings": 0
}
]
} |
|
quorum-review (AD-04): three PASS — agreed (auto_merge: checked=true was_armed=false disarmed=false) {
"ticket": "PMAT-4445",
"head": "51c41b92e2ee6bc54867fd87885632a842ecc040",
"width": 3,
"executor": "agy",
"agreed": true,
"auto_merge": {
"checked": true,
"was_armed": false,
"disarmed": false,
"note": "auto-merge not armed"
},
"lanes": [
{
"lane": 1,
"verdict": "PASS",
"findings": 0
},
{
"lane": 2,
"verdict": "PASS",
"findings": 0
},
{
"lane": 3,
"verdict": "PASS",
"findings": 0
}
]
} |
|
Degraded quorum (PMAT-360). same-family (policy: agy reserved for tier-1 diffs) — withheld by policy: gemini (policy); Claude Code lanes: 2. Armed under the default of the operator, verbatim: "we are STILL draining agy too quick. we need to switch default to sonnet, agy, haiku", with agy reserved for tier-1 diffs (quorum.agy_tier1_only, cop ruling 2026-09-24). This is a same-family verdict, not an independent one. |
Closes #4445
apr benchon a Qwen3.5 GGUF refuses every run with warmup+iterations > 1: warmup and each timed turn repeat one prompt on oneSession, the session resumes the repeated prefix, the turn reportsreused != 0, and the prefill guard (correctly) refuses.Fix.
Session::forget_prefix()drops every prefix the session could resume from.bench_qwen35calls it before each warmup turn and insession_timed_turn, so every timed turn prefills its whole prompt.Opened against main ahead of car #4429 (operator: "make it faster"). On main,
Sessionhas no #4214checkpoint, soforget_prefixclearsprocessedonly. It destructuresSessionexhaustively, so when the car'scheckpointfield meets this code the merge fails to compile until the checkpoint is reset too. It cannot merge silently with a resumable prefix left behind. The car-side version (clears the checkpoint too; its test goes red on aforget_prefixthat keeps the checkpoint, measured) is9c34ca648donbb/4445-bench-fresh-prefill, quorum 3/3 on that branch. Whichever order lands second takes that one-line resolution.Tests.
forget_prefix_makes_an_extending_prompt_prefill_whole_again: on main an extending prompt normally resumes (reused == 2). Afterforget_prefixit must prefill from 0.session::31/31 pass on this branch.🤖 Generated with Claude Code