Skip to content

Add: non-generative-judgment-routing - #155

Open
shitianfang wants to merge 1 commit into
nibzard:mainfrom
shitianfang:add-non-generative-judgment-routing
Open

shitianfang wants to merge 1 commit into
nibzard:mainfrom
shitianfang:add-non-generative-judgment-routing

Conversation

@shitianfang

Copy link
Copy Markdown

Disclosure up front: the primary reference for this pattern is my own project, and I am the author of both. That relationship is stated in the front matter (authors, based_on), in the Known implementations subsection, and beside each self-reference in References. I have written this up as the general technique, not as a description of my implementation, and the implementation is listed as an example rather than a recommendation.

Adds one file: patterns/non-generative-judgment-routing.md (109 lines).

The pattern

Partition an agent loop's steps by a single test: does this step's output have to be text?

Steps whose output is the deliverable — prose, code, a patch, an answer — stay on the LLM. Steps that only produce a decision the control flow consumes — is the build done, which of these thirty elements do I click, is this shell command safe, keep or drop this message — are expressed as typed questions over an explicit state (yes/no, pick-one over an enum, score over a range), batched so that every question about the same state goes in one call, and answered by a non-generative judgment model in a single forward pass instead of an autoregressive decode.

The other half of the pattern is the escalation contract. The judge may decline with a typed reason — writing, open_ended, oversized, unsure, unreachable — and every decline routes the step back to the LLM. The contract deliberately has no default branch: a declined question never resolves to false, to the first enum member, or to a midpoint score, because a silent default reintroduces exactly the guessing the pattern exists to remove.

Trade-offs are in the body, not softened

The write-up says plainly that it only pays inside loops that repeat the same kind of decision, that it is worth little when nearly every step generates text, that the judge's accuracy is decision-type-specific and must be measured on your own traffic against a majority-class baseline before anyone runs it unsupervised, and that speedups measured against unconstrained generation overstate the gain.

The Evidence section is graded low and reports the losses alongside the wins from the one public measurement: 82.2% agreement with a claude-opus-5 reference over 454 judgments (89.5% among non-escalated verdicts) against a 68.7% majority-class baseline — but context compaction at 56.3%, below a constant answerer, and a fair-baseline rerun putting the latency advantage at roughly 3x rather than 14x once the generative baselines were enum-constrained. One implementation, one study, no replication.

Novelty relative to existing patterns

scripts/pattern_similarity_checker.py reports no pattern above 0.5 similarity across all 194 existing patterns. The nearest neighbour conceptually is budget-aware-model-routing-with-hard-cost-caps, which routes among generative models by cost and task complexity; this pattern routes by whether generation is needed at all, and adds the typed-escalation contract. I have linked it as related and drawn the distinction explicitly in References.

Checks run locally

python3 scripts/pattern_validator.py patterns/non-generative-judgment-routing.md

Validating 1 file(s)...

✅ non-generative-judgment-routing.md: 0 issue(s)

tsx scripts/validate-patterns.ts patterns/non-generative-judgment-routing.md --check-content — no output, exit code 0. Run over the whole patterns directory the error count is 1 both with and without this change (a pre-existing missing references section in cross-agent-lesson-sharing.md), so this file adds no errors or warnings.

scripts/pattern_similarity_checker.py✅ No patterns found with similarity >= 0.5.

I could not run bun run validate:patterns or bun run build:data as written because bun is not available in my environment; I ran the same TypeScript validator through tsx instead, and the Python validator in scripts/. I did not commit generated artifacts under apps/web/public or touch the README auto-generated section, matching the recent merged pattern PRs (#142, #134, #121), which each add only their patterns/*.md file.

Notes on the rules

  • External links are limited to github.com plus one dev.to write-up, which is a neutral publishing platform rather than a vendor site; both are disclosed as mine.
  • Non-self references are thin, and honestly so: there is one public implementation of this today, which is also why status is emerging rather than established. I did not want to pad the references with papers that do not actually describe this partitioning.
  • No CTA or promotional language; happy to cut anything that still reads that way.

@vercel

vercel Bot commented Sep 19, 2026

Copy link
Copy Markdown

Someone is attempting to deploy a commit to the nkkko's projects Team on Vercel.

A member of the Team first needs to authorize it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant