Skip to content

XY-1163: R1 define model ladder and local background organizer benchmark gate #352

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
yvette-carlisle merged 1 commit into from
Jul 4, 2026
Merged

XY-1163: R1 define model ladder and local background organizer benchmark gate #352

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 16 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -340,10 +340,21 @@ provider-backed ELF evidence was required.
non-pass rows; PageIndex/OpenKB use existing fixture/source provenance, while
Honcho remains a source-provenance research gate until product-runtime evidence is
checked in.
The full ELF fixture run scores 75 pass, 0 wrong_result, 0 unsupported claims, and
The full ELF fixture run scores 77 pass, 0 wrong_result, 0 unsupported claims, and
7 blocked jobs with aggregate evidence/source-ref/quote coverage at 1.000, while
every public product row remains non-comparable until held-out, leakage-audited,
digest-identified product-runtime evidence exists.
- R1 model ladder and local background organizer gate after XY-1163: the July 4
follow-up adds `cargo make real-world-memory-r1-local-organizer`, defines the
L0/L1/L2/L3/L4 model ladder, and scores one fixture-backed proposal-only local
organizer job. The gate reports extraction F1 as explicitly `not_encoded` until a
labeled extraction set exists, validates JSON/schema, citation/source-ref coverage,
unsupported-claim rate, stale/correction/delete behavior, zero source mutation,
zero silent Memory Authority mutation, escalation rate, mean/p95 latency, and
tier-level cost/resource/runtime provenance. This is not evidence that small/local
models are safe defaults for Memory Authority writes or that ELF has Dreaming,
Letta, LangMem/LangChain, mem0/OpenMemory, Graphiti/Zep, or hosted managed-memory
parity.
- Operator-approved public-proxy addendum after XY-930: the June 19 follow-up runs
`cargo make baseline-production-private-addendum` with a simulated/public-proxy
production corpus manifest approved for this stage. The run records 12 documents,
Expand Down Expand Up @@ -493,6 +504,7 @@ Detailed evidence and interpretation:
- [Source-Backed Memory Quality Benchmark Harness - July 3, 2026](docs/evidence/benchmarking/2026-07-03-source-backed-quality-benchmark-harness.md)
- [qmd Candidate-Replay Comparability Gate - July 3, 2026](docs/evidence/benchmarking/2026-07-03-qmd-candidate-replay-comparability-gate.md)
- [Final Source-Backed Project Memory Closeout Report - July 3, 2026](docs/evidence/benchmarking/2026-07-03-final-source-backed-project-memory-closeout-report.md)
- [R1 Local Background Organizer Benchmark Gate - July 4, 2026](docs/evidence/benchmarking/2026-07-04-r1-local-background-organizer-benchmark-gate.md)
- [Live Baseline Benchmark Runbook](docs/runbook/benchmarking/live_baseline_benchmark.md)
- [Single-User Production Runbook](docs/runbook/single_user_production.md)
- Benchmark contract:
Expand Down Expand Up @@ -603,7 +615,7 @@ Detailed comparison, mechanism-level analysis, and source map:
- [Derived Knowledge Page Follow-Up Research](docs/research/derived_knowledge_page_followup.md)
- [Dreaming Product Surface Follow-Up Research](docs/research/dreaming_product_surface_followup.md)

Latest real-world benchmark report: July 3, 2026. Latest external research refresh:
Latest real-world benchmark report: July 4, 2026. Latest external research refresh:
June 11, 2026; June 20 adds the Agent Knowledge OS Closeout Benchmark Report,
the Graph Topic-Map Report - June 20, 2026, Knowledge Workspace Version-Diff
Report - June 20, 2026, and the Live Knowledge-Page Rebuild/Lint Report - June 20,
Expand All @@ -622,6 +634,8 @@ June 27 adds the public quantitative competitor scoreboard report with row-level
comparability gates and no universal leaderboard claim. July 3 adds the
source-backed memory quality benchmark harness, the qmd candidate-replay
comparability gate, and the final source-backed project memory closeout report.
July 4 adds the R1 model ladder and proposal-only local background organizer
benchmark gate.

## Documentation

Expand Down
317 changes: 317 additions & 0 deletions apps/elf-eval/fixtures/real_world_memory/local_background_organizer/proposal_only_gate.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,317 @@
{
"schema": "elf.real_world_job/v1",
"job_id": "local-background-organizer-proposal-only-001",
"suite": "local_background_organizer",
"title": "Local background organizer drafts proposals without authority mutation",
"corpus": {
"corpus_id": "real-world-memory-r1-local-background-organizer-2026-07-04",
"profile": "synthetic",
"items": [
{
"evidence_id": "local-organizer-source-current",
"kind": "work_journal_entry",
"text": "Current session: the operator wants a local background organizer to draft low-risk memory proposals only, with source refs and no automatic Memory Authority write.",
"source_ref": {
"schema": "source_ref/v1",
"resolver": "real_world_memory_fixture/v1",
"ref": {
"fixture": "local_background_organizer_proposal_only_gate",
"evidence_id": "local-organizer-source-current"
}
},
"created_at": "2026-07-04T10:00:00Z"
},
{
"evidence_id": "local-organizer-conflict-policy",
"kind": "policy",
"text": "Policy: conflict resolution, multi-source synthesis, promotion review, important corrections, benchmark judging, and public claims require a strong model or explicit policy gate.",
"source_ref": {
"schema": "source_ref/v1",
"resolver": "real_world_memory_fixture/v1",
"ref": {
"fixture": "local_background_organizer_proposal_only_gate",
"evidence_id": "local-organizer-conflict-policy"
}
},
"created_at": "2026-07-04T10:01:00Z"
},
{
"evidence_id": "local-organizer-stale-delete",
"kind": "audit",
"text": "Audit: stale, correction, and delete observations are scored before promotion and must stay reviewable proposals until a reviewer or stronger gate accepts the change.",
"source_ref": {
"schema": "source_ref/v1",
"resolver": "real_world_memory_fixture/v1",
"ref": {
"fixture": "local_background_organizer_proposal_only_gate",
"evidence_id": "local-organizer-stale-delete"
}
},
"created_at": "2026-07-04T10:02:00Z"
},
{
"evidence_id": "local-organizer-decoy",
"kind": "decoy",
"text": "Decoy: the local background organizer already promoted every extracted preference directly into current Memory Authority.",
"source_ref": {
"schema": "source_ref/v1",
"resolver": "real_world_memory_fixture/v1",
"ref": {
"fixture": "local_background_organizer_proposal_only_gate",
"evidence_id": "local-organizer-decoy"
}
},
"created_at": "2026-07-04T10:03:00Z"
}
],
"adapter_response": {
"adapter_id": "fixture_local_background_organizer",
"answer": {
"content": "The local background organizer drafted one reviewable proposal with source refs and no Memory Authority write. It escalated conflict resolution, multi-source synthesis, promotion review, important corrections, benchmark judging, and public claims to a strong model or explicit policy gate. Stale, correction, and delete observations were scored before promotion and kept as reviewable proposals.",
"claims": [
{
"claim_id": "proposal_only",
"text": "The local background organizer drafted one reviewable proposal with source refs and no Memory Authority write.",
"evidence_ids": [
"local-organizer-source-current"
],
"confidence": "high"
},
{
"claim_id": "strong_gate_escalation",
"text": "Conflict resolution, multi-source synthesis, promotion review, important corrections, benchmark judging, and public claims were escalated to a strong model or explicit policy gate.",
"evidence_ids": [
"local-organizer-conflict-policy"
],
"confidence": "high"
},
{
"claim_id": "stale_correction_delete_reviewable",
"text": "Stale, correction, and delete observations were scored before promotion and kept as reviewable proposals.",
"evidence_ids": [
"local-organizer-stale-delete"
],
"confidence": "high"
}
],
"evidence_ids": [
"local-organizer-source-current",
"local-organizer-conflict-policy",
"local-organizer-stale-delete"
],
"latency_ms": 42.0,
"cost": {
"currency": "USD",
"amount": 0.0,
"input_tokens": 320,
"output_tokens": 180
}
},
"consolidation": {
"proposals": [
{
"proposal_id": "proposal-local-organizer-review-only",
"proposal_kind": "local_background_organization_candidate",
"source_refs": [
"local-organizer-source-current",
"local-organizer-conflict-policy",
"local-organizer-stale-delete"
],
"expected_source_refs": [
"local-organizer-source-current",
"local-organizer-conflict-policy",
"local-organizer-stale-delete"
],
"usefulness_score": 0.91,
"min_usefulness_score": 0.8,
"expected_review_action": "defer",
"actual_review_action": "defer",
"source_mutations": [],
"unsupported_claim_count": 0,
"unsupported_claim_flags": [],
"diff": {
"summary": "Stage a local organizer proposal for human or strong-gate review without applying it.",
"before": {},
"after": {
"target": "reviewable_local_background_organizer_proposal",
"apply_intent": "no_op",
"authority_write_allowed": false
}
}
}
]
},
"local_organizer": {
"model_tier": "L2",
"model_profile_id": "local_background_organizer_l2",
"runtime_id": "fixture_schema_constrained_local_runtime",
"runtime_commit": "fixture-runtime@2026-07-04",
"reproducibility_provenance": [
"apps/elf-eval/fixtures/real_world_memory/local_background_organizer/proposal_only_gate.json",
"cargo make real-world-memory-r1-local-organizer"
],
"extraction_f1": null,
"extraction_f1_blocker": "not_encoded: no labeled extraction set is checked in for this first R1 local organizer slice.",
"json_schema_valid": true,
"citation_source_ref_coverage": 1.0,
"unsupported_claim_rate": 0.0,
"stale_correction_delete_score": 1.0,
"source_mutation_count": 0,
"silent_memory_authority_mutation_count": 0,
"required_escalation_count": 2,
"completed_escalation_count": 2,
"resource_footprint": {
"runtime": "fixture",
"device": "local_cpu",
"peak_memory_mb": 64,
"network_required": false
}
}
}
},
"timeline": [
{
"event_id": "local-organizer-r1-scope",
"ts": "2026-07-04T10:00:00Z",
"actor": "operator",
"action": "set_scope",
"evidence_ids": [
"local-organizer-source-current",
"local-organizer-conflict-policy",
"local-organizer-stale-delete"
],
"summary": "The R1 slice was constrained to reviewable proposals with strong-gate escalation for authority-changing work."
}
],
"prompt": {
"role": "user",
"content": "Run the local background organizer gate and report only reviewable proposals; do not mutate Memory Authority.",
"job_mode": "local_background_organizer",
"constraints": [
"proposal_only",
"cite_evidence",
"escalate_authority_changes",
"report_schema_and_resource_metrics"
]
},
"expected_answer": {
"must_include": [
{
"claim_id": "proposal_only",
"text": "The local background organizer drafted one reviewable proposal with source refs and no Memory Authority write."
},
{
"claim_id": "strong_gate_escalation",
"text": "Conflict resolution, multi-source synthesis, promotion review, important corrections, benchmark judging, and public claims were escalated to a strong model or explicit policy gate."
},
{
"claim_id": "stale_correction_delete_reviewable",
"text": "Stale, correction, and delete observations were scored before promotion and kept as reviewable proposals."
}
],
"must_not_include": [
"already promoted every extracted preference directly into current Memory Authority"
],
"evidence_links": {
"proposal_only": "local-organizer-source-current",
"strong_gate_escalation": "local-organizer-conflict-policy",
"stale_correction_delete_reviewable": "local-organizer-stale-delete"
},
"answer_type": "proposal_gate_report",
"accepted_alternates": [],
"requires_caveat": false,
"requires_refusal": false
},
"required_evidence": [
{
"evidence_id": "local-organizer-source-current",
"claim_id": "proposal_only",
"requirement": "cite",
"quote": "draft low-risk memory proposals only"
},
{
"evidence_id": "local-organizer-conflict-policy",
"claim_id": "strong_gate_escalation",
"requirement": "cite",
"quote": "require a strong model or explicit policy gate"
},
{
"evidence_id": "local-organizer-stale-delete",
"claim_id": "stale_correction_delete_reviewable",
"requirement": "cite",
"quote": "scored before promotion"
}
],
"negative_traps": [
{
"trap_id": "local-organizer-authority-write-decoy",
"type": "stale_fact",
"evidence_ids": [
"local-organizer-decoy"
],
"failure_if_used": true
}
],
"scoring_rubric": {
"dimensions": {
"answer_correctness": {
"weight": 0.2,
"max_points": 1.0,
"criteria": {
"requires": [
"proposal_only",
"strong_gate_escalation",
"stale_correction_delete_reviewable"
]
}
},
"evidence_grounding": {
"weight": 0.2,
"max_points": 1.0,
"criteria": {
"requires_source_refs": true
}
},
"lifecycle_behavior": {
"weight": 0.2,
"max_points": 1.0,
"criteria": {
"stale_correction_delete_score": 1.0
}
},
"source_immutability": {
"weight": 0.2,
"max_points": 1.0,
"criteria": {
"max_source_mutations": 0,
"max_silent_memory_authority_mutations": 0
}
},
"latency_resource": {
"weight": 0.2,
"max_points": 1.0,
"criteria": {
"max_latency_ms": 5000.0,
"requires_resource_footprint": true
}
}
},
"pass_threshold": 1.0,
"hard_fail_rules": [
"source mutation count must remain zero",
"silent Memory Authority mutation count must remain zero",
"strong model or explicit policy gate required for authority-changing work"
]
},
"allowed_uncertainty": {
"can_answer_unknown": false,
"acceptable_phrases": [],
"fallback_action": "block_or_escalate"
},
"tags": [
"r1_model_ladder",
"local_background_organizer",
"proposal_only",
"fixture_backed"
]
}
5 changes: 5 additions & 0 deletions apps/elf-eval/src/bin/real_world_job_benchmark/artifacts.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ mod answer;
mod consolidation;
mod cost;
mod knowledge;
mod local_organizer;
mod memory;
mod proactive;
mod recovery;
Expand All @@ -13,6 +14,10 @@ pub(super) use self::{
consolidation::{ConsolidationFixture, ConsolidationProposalFixture},
cost::CostReport,
knowledge::{DerivedPageArtifact, DerivedPageRebuild, DerivedPageSection},
local_organizer::{
LocalOrganizerFixture, LocalOrganizerJobReport, LocalOrganizerSummaryReport,
LocalOrganizerTierReport,
},
memory::{MemorySummaryArtifact, MemorySummaryEntry, MemorySummarySourceTrace},
proactive::{ProactiveBriefArtifact, ProactiveSuggestion},
recovery::{
Expand Down
Loading