System-One Model Benchmark: Open-Weight vs. TypeSafe Jev

Comprehensive empirical evaluation comparing open-weight Local System-One decision models against TypeSafe Jev 1.13.0 on an unseen holdout benchmark across 174 realistic software engineering tasks.

Executive Summary

System-One models act as fast, zero-to-low-token semantic routers and risk scorers in agentic architectures. They classify intent (task_class), predict required reasoning depth (reasoning_need), gauge execution blast radius (execution_risk), and evaluate prerequisite binary conditions (such as repo_understanding_required or small_local_model_sufficient).

This benchmark evaluated six models under an identical canonical 7-question contract:

1. TypeSafe Jev 1.13.0 (Specialized Cloud API)
2. Decider-0.8B (Mapika/decider-0.8b, Qwen3.5-0.8B backbone, 752M params)
3. Decider-2B (Mapika/decider-2b, Qwen3.5-2B backbone, 1.88B params)
4. Local Qwen System-One V2 (Qwen3.6-35B-A3B MoE logit scoring)
5. Laya Typed-Decisions (convaiinnovations/laya, ModernBERT-large 421M)
6. Laya Base (convaiinnovations/laya, ModernBERT-large 421M)

Key Findings

• Decider-0.8B is the standout open-weight model: Achieving 85.1% task_class accuracy, it is statistically on par with TypeSafe Jev (89.7%, $p = 0.1859$ via McNemar test). It matches Jev on debugging recall (88.2%) and research recall (100.0%), while beating Jev on simple coding recall (85.7% vs 78.6%), requiring only ~1.8 GB RAM.
• Decider-2B offers top calibration: Demonstrates the lowest Expected Calibration Error (ECE 0.0314) and beats Jev on binary gate Brier scores (web_research_required Brier = 0.0756 vs 0.0901).
• Substantial Local MoE Replacement: Both Decider models outperform the 35B Local Qwen V2 baseline (80.5%) while reducing memory requirements from ~21 GB down to 1.8–3.8 GB.
• Where TypeSafe Jev Remains Essential: Operations task classification (100.0% vs 76.7%–80.0%), scalar score calibration (MAE 0.47 vs 0.75 on reasoning need), and sub-300ms multi-question cloud fan-out latency.

---

1. Six-System Empirical Comparison Matrix

Tested on a frozen holdout benchmark of 174 software engineering tasks (unseen_holdout_v1.json, SHA256: d8f5e8a9c0eae5885307e54beae403629b21d389adea54dfe813d10779bb4d44).

Metric / Dimension	TypeSafe Jev 1.13.0	Decider-0.8B	Decider-2B	Local Qwen V2 (35B)	Laya Typed (421M)	Laya Base (421M)
task_class Accuracy	89.7% (156/174)	85.1% (148/174)	83.3% (145/174)	80.5% (140/174)	57.5% (100/174)	52.3% (91/174)
Macro F1 Score	0.9005	0.8563	0.8367	0.8125	0.5037	0.4758
Simple Coding Recall	78.6% (33/42)	85.7% (36/42)	85.7% (36/42)	50.0% (21/42)	47.6% (20/42)	71.4% (30/42)
Debugging Recall	88.2% (45/51)	88.2% (45/51)	80.4% (41/51)	78.4% (40/51)	80.4% (41/51)	37.2% (19/51)
Architecture Recall	90.3% (28/31)	77.4% (24/31)	87.1% (27/31)	100.0% (31/31)	90.3% (28/31)	87.1% (27/31)
Research Recall	100.0% (20/20)	100.0% (20/20)	85.0% (17/20)	95.0% (19/20)	15.0% (3/20)	15.0% (3/20)
Operations Recall	100.0% (30/30)	76.7% (23/30)	80.0% (24/30)	96.7% (29/30)	26.7% (8/30)	40.0% (12/30)
Reasoning Need MAE	0.4724	0.7965	0.7461	0.5617	0.7605	0.9741
Execution Risk MAE	0.6861	0.7855	0.7820	0.7874	0.8290	0.9115
Ambiguity Level MAE	1.5682	0.8813	0.8543	1.5717	0.8506	0.7950
repo_understanding Brier	0.1529	0.2371	0.1569	0.2096	0.2708	0.2920
small_local_model Brier	0.1483	0.1499	0.1462	0.1968	0.2358	0.2526
web_research Brier	0.0901	0.2329	0.0756	0.1243	0.1983	0.1826
Calibration Error (ECE)	0.3297	0.0747	0.0314	0.3762	0.1008*	0.0387*
$\ge 0.90$ Conf Wrong Rate	1.4% (2/142)	6.4% (9/140)	5.8% (15/257)	10.1% (16/159)	0.0% (0/0)*	0.0% (0/0)*
Repeatability Determinism	100.0%	100.0%	100.0%	100.0%	100.0%	100.0%
1-Question Latency (Local)	~120 ms (API)	239.3 ms	214.9 ms	~1,450 ms	52.1 ms	48.2 ms
7-Question Fan-out Latency	~306 ms (API)	~2,589 ms	~3,408 ms	~1,562 ms	208.0 ms	186.1 ms
Max Native Context	>4,096 tok	32,768 tok	32,768 tok	~4,096 tok	1,024 tok	512 tok
Resident Memory	0 MB (Cloud)	~1.8 GB	~3.8 GB	~21.0 GB	~1.6 GB	~1.6 GB
Offline Autonomous	No	Yes	Yes	Yes	Yes	Yes

*Note on Laya Calibration: Laya's zero high-confidence error rate reflects temperature flattening (choice:3-5 = 1.76) where no cases achieved $\ge 0.90$ confidence on out-of-domain engineering tasks.

---

2. Statistical Significance (McNemar Tests)

Paired contingency analysis on the identical 174 holdout cases ($N=174$, $\alpha=0.05$):

Comparison	Discordant	System A Only	System B Only	$\chi^2$ (continuity)	$p$-value	Conclusion
Decider-2B vs. Laya Base	66	60 (2B)	6 (Laya)	42.561	$6.85 \times 10^{-11}$	Decider-2B significantly superior
Decider-0.8B vs. Local Qwen V2	44	26 (0.8B)	18 (Qwen)	1.114	0.2913	Decider-0.8B numerically superior
Decider-2B vs. Local Qwen V2	39	22 (2B)	17 (Qwen)	0.410	0.5218	Decider-2B numerically superior
TypeSafe Jev vs. Decider-0.8B	28	18 (Jev)	10 (0.8B)	1.750	0.1859	No statistically significant difference
TypeSafe Jev vs. Decider-2B	27	19 (Jev)	8 (2B)	3.704	0.0543	Borderline significance

Key Statistical Takeaway: At $p = 0.1859$, Decider-0.8B's 85.1% accuracy cannot be statistically distinguished from TypeSafe Jev (89.7%) on this holdout size.

---

3. Detailed Failure Mode Analysis

3.1 TypeSafe Jev (Cloud Baseline)

• Strengths: Perfect recognition of Operations (100%) and Research (100%); superior continuous calibration on reasoning depth (MAE 0.4724) and execution risk (MAE 0.6861); lowest overconfidence rate (1.4%).
• Weaknesses: Occasional confusion on subtle simple coding vs. debugging boundaries (simple coding recall 78.6%). Commercial API dependency with per-call cost and latency overhead.

3.2 Mapika Decider Family (0.8B & 2B)

• Strengths:
  • Robust semantic invariance (1.0000 correlation on score reversal; minimal choice permutation drift).
  • High recall on simple coding (85.7%) and debugging (88.2% on 0.8B).
  • Outstanding Brier scores on binary gates (web_research_required = 0.0756 on 2B, outperforming Jev).
  • Full 32,768-token native context window with zero silent truncation.
  • Extremely compact memory footprints (~1.8 GB for 0.8B, ~3.8 GB for 2B).
• Weaknesses:
  • Operations recall (76.7%–80.0%) lags behind Jev (100.0%) and Qwen V2 (96.7%), occasionally routing deployment/infrastructure tasks into debugging or architecture.
  • Multi-question fan-out latency (~2.6s on 0.8B, ~3.4s on 2B) when running on platforms without fused non-autoregressive decision kernels.

3.3 Local Qwen System-One V2 (35B MoE)

• Strengths: Perfect architecture recall (100.0%), strong operations (96.7%).
• Weaknesses: Severe under-prediction of simple coding (50.0% recall); overconfidence on wrong classifications (10.1% wrong at $\ge 0.90$ confidence); substantial memory footprint (~21 GB).

3.4 Laya (ModernBERT-421M Family)

• Strengths: Fast inference (<200 ms fan-out), lightweight (~1.6 GB).
• Weaknesses: Severe out-of-domain failure on complex engineering prompts (52.3% accuracy); 85% blindspot on research tasks; slot-ordering permutation flips (36%); strict 512–1,024 token context limit.

---

4. Qualitative Benchmarks & Edge Cases

Concurrency Race Diagnosis (Lock-Free Rust)

Prompt: "Diagnose a lock-free concurrency race in a Rust ring buffer, modify implementation, validate correctness, run tests and submit a PR."

• Decider-2B Evaluation:
  • task_class: debugging (51.9% debugging, 29.2% simple coding, 17.2% architecture)
  • reasoning_need: 1.89 / 3.0 (deep reasoning required)
  • repo_understanding_required: 0.9696 (97.0% YES)
  • small_local_model_sufficient: 0.0866 (91.3% NO)
  • execution_risk: 2.25 / 3.0 (high risk)
• Result: Successfully flags subtle concurrency race debugging, demands high reasoning and repository context, and rejects small-model delegation.

---

5. Recommended Deployment Architecture

For cost-effective, highly reliable agent routing, a two-tier hybrid dispatch strategy is recommended:

                      [ Incoming Agent State ]
                                 │
                                 ▼
                     [ Decider-0.8B (Local) ]
                      ~1.8 GB RAM | ~239 ms
                                 │
                 ┌───────────────┴───────────────┐
      Confidence >= 0.70                Confidence < 0.70
      AND class != "operations"         OR class == "operations"
                 │                               │
                 ▼                               ▼
        [ Local Fast-Path ]            [ TypeSafe Jev (Cloud) ]
        (80-85% of queries)            (High-risk / Escalation)
        Cost: $0                       Cost: Minimal

1. Primary Local Router: Mapika/decider-0.8b handles 80–85% of incoming routing decisions locally with zero marginal API cost.
2. Escalation Oracle: TypeSafe Jev is invoked selectively when confidence is low or when critical infrastructure/operations tasks are detected.
3. Offline Fallback: Decider-0.8B serves as a fully offline, privacy-preserving fallback that outperforms previous 35B local baselines.

---

6. Verification and Reproduction

The evaluation scripts and raw score outputs are structured as:

• docs/system-one-six-model-comparison.md: Detailed comparative report.
• docs/decider-evaluation.md: Technical report on Decider-2B and Decider-0.8B.
• results/comparison/six_systems_table.json: Machine-readable results table.
• results/comparison/mcnemar_stats.json: Statistical significance test outputs.

All benchmark results were generated with deterministic execution (100% repeatability across multiple runs).