Repository navigation
Failed analysis keeps no trace of the reply ("analysis returned no parseable JSON") and is retried every run #193
Description
Problem
When extractJson(result.text) finds nothing, analysis throws analysis returned no parseable JSON (src/analyze.js). The evidence record then keeps only status: "failed" and that error string. The reply is dropped.
A failed record is never fresh (isEvidenceFresh needs status === "ok"), so the transcript is analyzed again on every run, with no limit.
Measured
In one repo, 27 of 211 evidence records failed this way (13%). Analysis agent: claude, Sonnet, auto-picked, effort medium, jobs: 10. The failures do not track transcript size: median 0.62 MB for failed records against 0.79 MB for ok records.
From the record alone, the user cannot tell the cause: an empty turn, prose instead of JSON, JSON cut off at a length limit, or a fence the extractor does not accept. Each run pays for the same failures again.
Proposal
- On this failure, save in the evidence record, after redaction: the reply length in characters, the first and last ~300 characters, and the harness stop reason if acpx gives one.
- Show a short class in
status:empty,prose,truncatedorunparsed. - Optional: after N failures of the same transcript at the same key, skip it and say so, instead of retrying it forever.
VISION: "Failure is loud and named."
backpass 0.1.30 (main d1e7550).
Activity
kunchenguid commentedon Sep 30, 2026
Speaking as Kun's firstmate:
Outcome: ready-for-pr — real failure-path gap on main (
d1e7550). No open PR that fixes provenance or the retry storm. #183 (blank-turn retry-once) may reduce some empty replies but does not keep the failed reply, classify it, or bound retries — notexisting-pr.On main:
extractJsonmiss →analysis returned no parseable JSON(src/analyze.js); catch writes{ status: "failed", error }only.isEvidenceFreshrequiresstatus === "ok", so every run re-pays the same failures. Comment in-tree already says “retried next run” with no cap.contract-class: restore — VISION “Failure is loud and named” plus measured artifacts (reply shape / stop reason) are the promised failure path; today’s mute
failed+ infinite retry is that path broken. Optional skip-after-N is fine only if the skip is named instatus(not a quieter degraded path).VISION (per rule):
status: empty/prose/truncated/unparsed; resist silent skip).Alignment criteria: resist a quieter degraded path — if skipping after N, say so and keep the last failure trace.
ready-for-pr+bug. I will not open a competing PR.kunchenguid commentedon Oct 14, 2026
Speaking as Kun's firstmate:
Still-real pass (14-day restamp): ready-for-pr. Rechecked latest main (
29c46e7):src/analyze.jsstill throwsanalysis returned no parseable JSONand the catch still writes only{ status: "failed", error }with the comment "retried next run", so the reply is dropped and the transcript is re-paid on every run with no cap. #183 (blank-turn retry) is still open and awaiting a decision, and it doesn't keep the failed reply, classify it, or bound retries, so it isn't the fix here. No other open PR helps.contract-class: restore. VISION's "Failure is loud and named" is the promised failure path; a mute
failedrecord with unbounded retries is that path broken. A skip-after-N is fine only ifstatusnames the skip.VISION (per rule):
empty/prose/truncated/unparsedclasses instatus).Leaving
ready-for-pr. I won't open a competing PR.Pantoria commentedon Oct 15, 2026
PR #212 is up for review (raised through no-mistakes). Failed records keep a redacted head/tail, the reply length, and a class (
empty/prose/truncated/unparsed) shown instatus. After 3 unparseable replies at the same key the transcript is skipped, and the skip is named instatus, the run summary, and--json, including for nested files.kunchenguid commentedon Oct 15, 2026
Speaking as Kun's firstmate:
existing-pr: #212 (thanks @Pantoria). It keeps a redacted head/tail,
charsand a class on the failed record and shows them instatus. That part is the restore this issue asked for. The skip after 3 unparseable replies is on by default, so #212 is waiting on Kun's decision about that default (details on the PR). Keepingready-for-pruntil it lands.VISION: evidence, human-owns-weights, nothing-on-faith, budget, owns-nothing, loud failure (strongly) and scope all align.
fix(analyze): keep a classed trace of unparseable replies and stop re…