Skip to content

Nous v2: one session, enforced validation, light method (#330) - #333

Merged
mtoslalibu merged 79 commits into
nous-v2from
nous-v2-wip
Oct 23, 2026
Merged

mtoslalibu merged 79 commits into
nous-v2from
nous-v2-wip

Conversation

@mtoslalibu

Copy link
Copy Markdown
Collaborator

Summary

This PR delivers Nous v2 from issue #330 into the nous-v2 branch. It does not change main. main keeps old Nous until a separate, later decision.

Nous v2 is Claude Code plus a light scientific method plus checks enforced by code:

  • One session, four phases. One Claude Code session (Agent SDK, claude-agent-sdk==0.2.172) runs BASELINE → DESIGN → EXECUTE → REPORT. Nous switches the phases.
  • In-process hooks record every Bash command to runs.jsonl, run the validators when a phase ends (Stop), and write a digest before compaction. Nous never blocks or judges a command.
  • JSON artifacts with schemas. A failed check goes back to the agent to repair; nothing is thrown away.
  • Locked predictions. They are hashed before EXECUTE.
  • Honest claims. Every claim value must match its evidence file, and claims must beat measured noise.
  • Light, task-aware method. A method prompt of about 5 KB plus a checklist per task type (debug, verify, optimize, policy, explain).
  • Stopping, resume and effort. The run stops when the question is answered or when it stalls (2 quiet iterations). It has a failure breaker and nous resume, and it handles effort when output hits the cap.
  • Code isolation. All code changes happen in a separate git worktree at <repo>/.nous-experiments/<run_id>/ on branch nous/<run_id>, so the user's checkout is never touched.
  • Cost is measured and reported, but it drives no decision.

Old Nous is deleted: the planner/executor split, the dispatchers, the shell hooks, gate summaries, the critic and complexity tiers, and the five v1 CLI flags. orchestrator/ went from about 15,900 lines to about 3,400; the v2 runtime is about 2,700.

Closes nothing yet. Refs #330.

How it was built and checked

  • 21 plan items were built by a Ralph-style /goal session, with a reviewer session alongside. Each item went test-first, then /pr-review-toolkit:review-pr, then commit and push. The plan and full log are in docs/plans/nous-v2/ (SPEC, plan.json, progress.txt, LEAN_AUDIT.md).
  • A full-branch review found 6 blocking bugs (P1–P6). Each was fixed with a test that fails before the fix. Details are in progress.txt.
  • 669 tests pass. There are no live LLM calls; the conftest.py guard is kept and now extended to ClaudeSDKClient.connect.
  • 4 live smoke runs on the BLIS debugging task all pass on mechanics. Smoke 3 and 4 run on the final code: exit 0, repro recorded, claims matched 1.0. In smoke 4 the user's checkout stayed clean, and the commits landed on nous/<run_id> in the campaign worktree.

Merge bar (R13): NOT met after 3 improvement rounds

See docs/plans/nous-v2/EVAL_REPORT.md. These are 3 oracle tasks compared against the recorded Claude + 4-line prompt (L2). Frontier-CS p5 and p225 were deferred because there is no Docker judge on the build machine.

task accuracy vs L2 cost vs L2 trail C1
debugging pass (2/2) ~1.3× 1.0
formula check fail (1/2 vs 2/3) 0.8–1.0× 1.0
config search fail ~4× 1.0

Across every run: 0 lost rounds, 0 crashes, 0 pre-registration violations. The merge bar runs found and fixed 7 real Nous bugs. This PR delivers a working, tested base. Closing the remaining accuracy and cost gaps is follow-up work in PRs into nous-v2.

Known follow-ups

Test plan

  • .venv/bin/python -m pytest -q: 669 passed
  • Live smoke on the final code: smoke/3.json and smoke/4.json, exit 0
  • Docs and example checks: tests/v2/test_docs_examples.py
  • Grep: no v1 terms in README, docs, prompts or skills; no removed flags in the code

🤖 Generated with Claude Code

toslali-ibm and others added 30 commits October 22, 2026 12:08
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…guard

- Pin claude-agent-sdk==0.2.172 (CLI 2.1.308) in pyproject.toml
- orchestrator/v2/session.py: Session with client_factory, builds
  ClaudeAgentOptions per spec 3.1 (preset system prompt with append,
  max_buffer_size 32MB, CLAUDE_CODE_MAX_OUTPUT_TOKENS env, hooks passthrough)
- orchestrator/v2/errors.py: TransientError hierarchy, is_transient()
  classifies ProcessError exit 143, "Fatal error in message reader",
  API 5xx, and timeouts as transient (B9)
- tests/fakes/sdk_client.py: FakeClient plays scripted message sequences,
  invokes hook callbacks, writes files — no live LLM calls
- tests/conftest.py: ClaudeSDKClient.connect blocked in all tests
- 39 new tests (1393 total), all green

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
FakeClient.connect/query/interrupt/disconnect are now async def,
receive_response is an async generator yielding each message.
All test methods in test_session.py and test_fake_client.py converted
to async def (pytest-asyncio auto mode). Added signature tests
verifying FakeClient methods are coroutines/async generators.
Corrected progress.txt SDK API note.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Session.connect now awaits client.connect(), turn() awaits
client.query() and iterates receive_response() as async for,
disconnect() awaits client.disconnect(). Completes the async
mismatch fix started in 2f2f9fd.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
8 JSON schemas (baseline, problem, bundle, plan, findings,
principle_updates, answer, ledger_entry) with B5 extra-field
handling, evidence rules for JSON/JSONL/CSV/TSV, noise honesty
validation, paired-seed check, locked_parameters/workload checks,
task_limits (B11), bundle hash, secret masking helper (3.14).

222 tests across test_schemas_v2, test_validators_v2,
test_evidence_rules, test_secrets. Full suite: 1617 passed.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Fixes from pr-review-toolkit review:
- Path traversal guard in check_evidence_item (is_relative_to check)
- RFC 6901 JSON Pointer "/" now resolves to doc[""] not root
- Paired-seed check compares all run pairs, not just first
- Baseline schema noise field wired to oneOf $defs
- Silent JSONL corruption in _check_task_limits now reported
- Regex no-capture-group now errors instead of silent pass
- Empty path/key in evidence items rejected early
- IsADirectoryError added to except clause

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|effect.value| <= 2 * noise_sd now triggers repair message for
CONFIRMED/REFUTED hypotheses. Added effect field to findings
schema. 6 new tests covering: effect inside noise rejected,
outside passes, deterministic zero sd, unmeasured exempt,
no effect field skips check.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add _get_baseline_noise_sd() helper that reads baseline.json and
returns noise sd (0.0 for deterministic, sd for measured, None
otherwise). CONFIRMED/REFUTED hypotheses without noise_sd now fall
back to baseline noise before requiring noise_unmeasured.

Missing effect.value with noise_sd present is now an error, not a
silent pass. Four new tests cover the fallback paths.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
PostToolUse records Bash commands to runs.jsonl with secret masking.
PostToolUseFailure records failed commands with exit code parsing.
Stop hook validates phase artifacts, blocks up to 6 consecutive times.
PreCompact hook writes state/digest.json for context recovery.

build_hooks() returns the hooks dict with per-event timeouts.

39 tests covering all four hooks, experiment command matching,
consecutive block escalation, digest content, and secret masking.

Review fixes (code-reviewer + silent-failure-hunter):
- Fixed output_bytes in failure hook: .encode("utf-8") for byte count
- Widened stop hook exception catch to include OSError/UnicodeDecodeError
- Injected deterministic now=lambda in test helper per tests/CLAUDE.md
- cache key uses tuple(patterns) not id() (pre-review fix)
- _append_runs_jsonl wrapped in try/except OSError with logging
- digest write wrapped in try/except with logging
- Skipped: runs.jsonl write propagation (logging is the fix per prior
  review; propagating would crash the hook framework)
- Skipped: eager regex validation (patterns from campaign config, crash
  at first use is acceptable fail-fast)
- Skipped: error field in failure rows (exit_code is sufficient;
  error text may contain secrets)
- Skipped: _PHASE_ARTIFACTS plan in EXECUTE (plan belongs to DESIGN)
- Skipped: corrupt-JSON markers in digest (best-effort, warning is fine)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
B1 recurred: build_hooks() returned dicts instead of HookMatcher objects,
causing _hooks_to_internal_format to produce empty hook lists. Also:
callbacks must be async def returning dict, tool_response must be
stringified before masking, and FakeClient now uses the real SDK contract
(_hooks_to_internal_format + _convert_hook_output_for_cli + TypeError on
sync callbacks).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
New modules:
- orchestrator/v2/stop_rule.py (139 lines): check_stop() with
  halt > stop_sentinel > answered > max_iterations > stall priority,
  claims_needed gating, claims_changed/best_changed/principles_changed
  helpers.
- orchestrator/v2/loop.py (285 lines): CampaignConfig dataclass,
  parse_campaign_config(), Campaign class driving
  BASELINE → (DESIGN → EXECUTE)* → REPORT lifecycle,
  bundle hash lock with pre-registration violation detection,
  per-iteration ledger writes, principle accumulation.

Tests: 42 new tests (22 stop_rule + 17 loop + 3 review-fix tests).
Review fixes applied: 9 accepted (None guards, baseline validation,
ledger-per-iteration, claims_needed gating, stall uses current_*
params). 5 skipped with rationale (no-ops or over-engineering).

Runtime: 1,694 lines (target ~2,500).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Reviewer findings addressed (from cross-session review):
1. Hooks wired into loop: HookState built in Campaign.run(),
   phase/iteration set before each turn. build_hooks() provides
   the hooks dict to Session options.
2. Campaign worktree commit per phase: _commit_phase() runs
   git add -A && git commit after each phase turn. Skips if
   no .git or no changes.
3. Phase messages completed per spec 3.2: BASELINE asks for
   task_type when not configured, DESIGN says to read digest
   and baseline and frame predictions relative to baseline,
   EXECUTE mentions wait_for placeholder.
4. Bundle lock raises on missing bundle.json after DESIGN
   (was silently returning "").
5. question_answered moved to findings.json top level per spec
   3.4 (was per-hypothesis). Schema, stop_rule, and all tests
   updated.
6. hooks.py _iter_dir fixed: removed erroneous "runs/" subdir
   to match loop.py's iter-N path.

Code review fix:
- _commit_phase guard simplified from wrong conjunction to
  single .git existence check.
- experiment_commands sourced from CampaignConfig field (was
  incorrectly reading from prompts dict).

Runtime: 1,758 lines (target ~2,500). Tests: 1692 passed.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
SPEC.md line 271: "iter-N/... is short for <work_dir>/runs/iter-N/..."
Added iter_dir() helper in loop.py. Updated hooks.py _iter_dir and
phase messages. All tests updated to use runs/iter-N/ layout.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Repair: 2 tries then status: partial, never drops results.
Failure breaker: 3 identical fingerprints → halt.json → REPORT.
wait_for: polls file/command every 60s in executor (no event loop block).
Preflight: validates hooks + validators on fixtures at campaign start.

Review fixes: poll_wait_for logs command errors and timeout warnings,
wait_for polling runs via run_in_executor to avoid blocking async loop.

1712 tests pass, 2086 runtime lines.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
smoke/1.json missing. See BLOCKED.md for instructions.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
1. Agent receives "Runs X finished; continue EXECUTE." after poll.
2. Stop hook allows stop when pending_wait_for is set in HookState.
3. All wait_for conditions polled together in one 60s loop.
4. Timeout counts as a breaker failure.

3 new integration tests. 1715 tests pass, 2159 runtime lines.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ation, simplify

- pending_wait_for changed from bool to list[str]; Stop hook checks
  conditions live via check_wait_condition
- Campaign passes cwd=target_system.path to Session per spec 3.1
- Secrets read from os.environ instead of literal "***"
- plan.json added to DESIGN phase artifacts in Stop hook
- Consolidated check_wait_condition into robustness.py (was duplicated
  in loop.py and hooks.py)
- Extracted _build_row helper in hooks.py
- Removed dead _EVIDENCE_EXTENSIONS, FailureBreaker.reset(), unused imports

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
loop.py:171 and loop.py:474 read target_system["path"] but the campaign
schema uses repo_path. cwd was None for every real campaign, and
_commit_phase silently did nothing. Tests used "path" which masked it.

Also adds parse_campaign_config validation: missing repo_path fails
early with a clear error.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
CostMeter deduplicates by message_id, tracks per-phase totals, prices
from defaults.yaml. Wired into Campaign.run via session.turn(on_message=).
Ledger entries record cumulative cost_meter_usd and token counts.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…er 2 cuts

EffortManager tracks consecutive and total cuts, persists cut_offs to
state.json. Ladder: max > xhigh > high > medium > low. Phase reset
restores configured effort.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
toslali-ibm and others added 29 commits October 22, 2026 18:57
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…p-rule requirements, safe phase commits

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dge on this Mac)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…c 3.5)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ode (orchestrator 6104->3076 lines)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…B12)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… crash)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…wer evidence exported into the repo

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ady validate

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t as evidence_detail

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… evidence without crashing

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… repo root, schema-exact method prompt, absolute artifact paths

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ric-string match, append-only claims_needed, repro must run as written, export path fix

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rk done

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… loop; drop v1 wiki tooling

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… --max-cli-retries, --sandbox)

They were accepted, warned about and ignored; nothing in v2 read them.
Passing one is now an argparse error. README and pyproject comment updated.
653 tests pass. The s390x deployment still passes these flags; tracked in a
separate issue.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…esume iter-1 prev, campaign worktree, persisted breaker, fake fires Stop/PreCompact/PostToolUseFailure)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mtoslalibu
mtoslalibu merged commit 25c0b12 into nous-v2 Oct 23, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants