Add: non-generative-judgment-routing - #155
Open
shitianfang wants to merge 1 commit into
Open
shitianfang wants to merge 1 commit into
shitianfang wants to merge 1 commit into
Conversation
|
Someone is attempting to deploy a commit to the nkkko's projects Team on Vercel. A member of the Team first needs to authorize it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Disclosure up front: the primary reference for this pattern is my own project, and I am the author of both. That relationship is stated in the front matter (
authors,based_on), in theKnown implementationssubsection, and beside each self-reference inReferences. I have written this up as the general technique, not as a description of my implementation, and the implementation is listed as an example rather than a recommendation.Adds one file:
patterns/non-generative-judgment-routing.md(109 lines).The pattern
Partition an agent loop's steps by a single test: does this step's output have to be text?
Steps whose output is the deliverable — prose, code, a patch, an answer — stay on the LLM. Steps that only produce a decision the control flow consumes — is the build done, which of these thirty elements do I click, is this shell command safe, keep or drop this message — are expressed as typed questions over an explicit state (yes/no, pick-one over an enum, score over a range), batched so that every question about the same state goes in one call, and answered by a non-generative judgment model in a single forward pass instead of an autoregressive decode.
The other half of the pattern is the escalation contract. The judge may decline with a typed reason —
writing,open_ended,oversized,unsure,unreachable— and every decline routes the step back to the LLM. The contract deliberately has no default branch: a declined question never resolves tofalse, to the first enum member, or to a midpoint score, because a silent default reintroduces exactly the guessing the pattern exists to remove.Trade-offs are in the body, not softened
The write-up says plainly that it only pays inside loops that repeat the same kind of decision, that it is worth little when nearly every step generates text, that the judge's accuracy is decision-type-specific and must be measured on your own traffic against a majority-class baseline before anyone runs it unsupervised, and that speedups measured against unconstrained generation overstate the gain.
The
Evidencesection is gradedlowand reports the losses alongside the wins from the one public measurement: 82.2% agreement with aclaude-opus-5reference over 454 judgments (89.5% among non-escalated verdicts) against a 68.7% majority-class baseline — but context compaction at 56.3%, below a constant answerer, and a fair-baseline rerun putting the latency advantage at roughly 3x rather than 14x once the generative baselines were enum-constrained. One implementation, one study, no replication.Novelty relative to existing patterns
scripts/pattern_similarity_checker.pyreports no pattern above 0.5 similarity across all 194 existing patterns. The nearest neighbour conceptually isbudget-aware-model-routing-with-hard-cost-caps, which routes among generative models by cost and task complexity; this pattern routes by whether generation is needed at all, and adds the typed-escalation contract. I have linked it asrelatedand drawn the distinction explicitly inReferences.Checks run locally
python3 scripts/pattern_validator.py patterns/non-generative-judgment-routing.mdtsx scripts/validate-patterns.ts patterns/non-generative-judgment-routing.md --check-content— no output, exit code 0. Run over the wholepatternsdirectory the error count is 1 both with and without this change (a pre-existing missingreferencessection incross-agent-lesson-sharing.md), so this file adds no errors or warnings.scripts/pattern_similarity_checker.py—✅ No patterns found with similarity >= 0.5.I could not run
bun run validate:patternsorbun run build:dataas written because bun is not available in my environment; I ran the same TypeScript validator throughtsxinstead, and the Python validator inscripts/. I did not commit generated artifacts underapps/web/publicor touch the README auto-generated section, matching the recent merged pattern PRs (#142, #134, #121), which each add only theirpatterns/*.mdfile.Notes on the rules
github.complus one dev.to write-up, which is a neutral publishing platform rather than a vendor site; both are disclosed as mine.statusisemergingrather thanestablished. I did not want to pad the references with papers that do not actually describe this partitioning.