Skip to content

triage: unparseable model output is swallowed, dropping a whole batch's triage while reporting success #120

Description

@thejesh23

What happened

triageBatch in packages/processor/src/triage.ts parses the model's verdict array with a bare JSON.parse wrapped in an empty catch:

const jsonMatch = resultText.match(/```json\s*([\s\S]*?)```/);
const jsonStr = jsonMatch ? jsonMatch[1].trim() : resultText.trim();
let verdicts: TriageVerdict[] = [];
try {
  verdicts = JSON.parse(jsonStr);
} catch {}

On any parse error verdicts stays [], the apply loop runs zero times, and the batch still emits a success message (Batch N/M: 0 triaged). A common trailing-comma or truncated array therefore silently loses the triage for the entire (up to 30-finding) batch — indistinguishable from a clean "nothing to triage" run. There's also no Array.isArray check, so a JSON object throws "not iterable" and is handled inconsistently.

The investigate/revalidate paths were specifically hardened against exactly this: agents/shared.ts routes through jsonrepair and writeParseFailureDebug and fails loud, with a comment noting "a malformed response is otherwise indistinguishable from a clean 'found nothing' run." Triage does neither.

Reproduction

(Found by code audit; regression tests are included in the linked PR.)

  1. Run deepsec triage where the model returns a verdict array with a trailing comma (e.g. [{"title":"X","priority":"P0",...,}]) — a routine LLM formatting slip.
  2. JSON.parse throws, the catch swallows it, and the batch reports "0 triaged" as success. All findings in the batch stay un-triaged with no error.
  3. jsonrepair (already a dependency, used by the investigate path) would have recovered it.

Expected vs actual

  • Expected: minor malformations are recovered via the tolerant parser; genuinely unparseable output fails the batch loudly (and is dumped for debugging), rather than being reported as a clean zero-result run.
  • Actual: any parse error silently drops the batch's triage and reports success.

Environment

  • Found against main (commit around 97ebd04); the code path is present in the current triage.ts.
  • Backend: claude-agent-sdk (the triage path).

Proposed fix

Route triage through the existing parseAgentJsonArray (jsonrepair-backed) + writeParseFailureDebug path used by the other agents, rethrowing on unrecoverable output so the batch is marked failed. PR to follow.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions