Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .claude-plugin/plugin.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "clodex",
"version": "0.1.1",
"version": "0.2.0",
"description": "Claude Code and Codex CLI dual-agent workflow orchestration.",
"author": {
"name": "9thLevelSoftware",
Expand Down
2 changes: 1 addition & 1 deletion .codex-plugin/plugin.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "clodex",
"version": "0.1.1",
"version": "0.2.0",
"description": "Claude Code and Codex CLI dual-agent workflow orchestration.",
"author": {
"name": "9thLevelSoftware",
Expand Down
78 changes: 78 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
# Changelog

## 0.2.0

A large reliability and capability release. The headline: Clodex could not complete a real run
against current `claude` / `codex` CLIs before 0.1.1, and the default Codex model retires on
2026-10-14. 0.2.0 fixes that and builds on it.

### Upgrading from 0.1.x

1. `clodex doctor`: checks your CLIs (login, supported flags), the live Codex model catalog and
your `CLODEX.md`, with a fix for every problem it finds. Add `--strict` to fail on warnings.
2. `clodex init --migrate --dry-run`, then without `--dry-run`: rewrites retired Codex models and
unsupported efforts in the `CLODEX.md` front matter (comments and line endings are kept).
Add `--split-claude` to move the old flat `claude.model` / `claude.effort` onto the new
`claude.plan` / `claude.audit` roles (the old layout keeps working without it).
3. `clodex init --force`: refreshes the managed blocks in `CLAUDE.md` / `AGENTS.md` with the new
delegation instructions. If you used `clodex init --global`, run it again: it now writes to
`~/.claude/CLAUDE.md` and `~/.codex/AGENTS.md`, and registers the Claude server through
`claude mcp`. You can delete the old `~/CLAUDE.md`, `~/AGENTS.md`, `~/CLODEX.md` and
`~/.mcp.json` that earlier versions created.
4. `clodex hooks install` if you used the hooks: the hook settings were rewritten (the old ones
used the wrong shape and some event names that do not exist).

Behavior changes to be aware of:

- Default Codex model is `gpt-6.1-sol` at `xhigh`. Claude planning defaults to `opus` / `max` and
audits to `opus` / `high` (separate settings now).
- `clodex build`, `audit` and `task start` refuse to run on a retired Codex model
(`CLODEX_ALLOW_RETIRED_MODEL=1` overrides).
- `clodex apply` only applies approved runs whose patch still matches the approved diff.
- Exit codes: `0` ok, `1` blocked/refused, `2` usage or configuration error, `3` failed,
`4` cancelled. Output without `--json` is now plain text.
- The MCP `tasks/update` method never existed in the spec and is gone; `tasks/*` now follow the
2025-11-25 Tasks utility. `actor` and `owner` on handoffs must be `claude` or `codex`.
- State database schema v4 (additive; old databases upgrade in place).

### Added

- **Native delegation:** `clodex_delegate` runs Codex (implement / fix / audit) in the handoff's
isolated worktree and records the result; Codex can ask questions instead of guessing
(`clodex_clarify` / `clodex_answer` / `clodex_messages`), and answers reach its next prompt.
`clodex_handoff_decide` follows your configured reviewers and quorum
(`unanimous` / `majority` / N) and refuses while a delegation runs or a question is open.
- **Structured output:** plan and audit verdicts use JSON schemas via `claude --json-schema` and
`codex exec --output-schema`, validated and retried once.
- **MCP:** protocol version negotiation, spec Tasks (`tasks/get|result|list|cancel`, task-augmented
`clodex_build`), correct JSON-RPC errors, ordered handling with non-blocking long work.
- **CLI:** `audit --base REF | --commit SHA`, `clean <run>`, `doctor --strict`, `init --migrate`,
`hooks install|uninstall --scope`, plain-text output, commands work from any subdirectory.
- **Config:** nested YAML (vendored PyYAML for npm installs), `claude.plan` / `claude.audit`,
`codex.audit`, `audit.max_diff_bytes`, `workspace.apply_mode: auto`, `mcp.async_tasks`.
- `clodex eval run` is now an offline self-test of the harness (no agents, nothing written to your repo).
- GitHub Actions CI on Ubuntu and Windows across Python 3.12 - 3.14.

### Fixed

- Real runs could not reach `approved`: `codex exec` dropped `--ask-for-approval`, Claude's JSON
result envelope was never unwrapped, `codex review` rejects a prompt, and `package.json` was
invalid JSON.
- An audit with no required reviewers counted as approved; a failing optional reviewer aborted the
whole run; unexpected errors left runs `running` forever with their worktree behind.
- Diffs were read and written with newline translation, corrupting CRLF and non-UTF-8 content.
- Cancelling or timing out killed only a shim while the agent kept running; workers that died were
never noticed; a late worker could overwrite a cancellation.
- The Claude plugin manifest was invalid and could not load; hooks used the wrong settings shape,
silently wrote into Claude's context and could exit with the "block" code; the installer nested
directories on re-run and left half-installs; launchers acted on the Clodex checkout instead of
your repo; user-scope MCP servers could not find the project.

## 0.1.1

Hotfix: unblock real runs (`codex exec` flags, Claude result envelope, `package.json`), default to
`gpt-6.1-sol`, and warn when `CLODEX.md` still pins the retiring `gpt-5.5`.

## 0.1.0

Initial release.
1 change: 0 additions & 1 deletion CLODEX.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,6 @@ codex:
approval_profile: ci
audit:
quorum: unanimous
personas: [security, performance, portability, test-gap]
reviewers: [{"id": "claude-plan", "backend": "claude", "persona": "plan-adherence", "required": true, "timeout": 600}, {"id": "codex-architecture", "backend": "codex", "persona": "architecture", "required": true, "timeout": 600}, {"id": "security", "backend": "codex", "persona": "security", "required": false, "timeout": 600}, {"id": "performance", "backend": "codex", "persona": "performance", "required": false, "timeout": 600}, {"id": "portability", "backend": "codex", "persona": "portability", "required": false, "timeout": 600}, {"id": "test-gap", "backend": "claude", "persona": "test-gap", "required": false, "timeout": 600}]
mcp:
async_tasks: true
Expand Down
9 changes: 6 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,8 @@ clodex status
| Git | diff hashing and repository state |
| Claude Code CLI | planning and Claude audit |
| Codex CLI | implementation and Codex audit |
| Node 18+ (optional) | the npm launcher and the Claude Code plugin's MCP server |
| PyYAML (bundled) | parses `CLODEX.md`; pip installs depend on it, npm installs ship a vendored copy |

Subscription CLI auth is the default:

Expand Down Expand Up @@ -171,7 +173,7 @@ PowerShell:
| `clodex task start/get/cancel/list` | Manage durable async runs |
| `clodex trace export <run-id>` | Print a run trace as JSONL |
| `clodex hooks print/install/uninstall/ingest` | Manage the Claude Code hook integration (`install --scope local\|project\|user`) |
| `clodex eval run` | Run local harness smoke evals |
| `clodex eval run` | Offline self-test of the harness (config, command lines, schemas, state, worktrees, native setup, hooks); calls no agents and writes nothing to your repo |
| `clodex queue add/list/update` | Manage the local task ledger |
| `clodex status` | Show recent tasks and runs |
| `clodex mcp-server` | Run the stdio MCP server |
Expand All @@ -197,7 +199,7 @@ codex:
approval_profile: ci
workspace:
backend: git-worktree
apply_mode: manual
apply_mode: manual # auto: apply an approved worktree build to your checkout automatically
max_fix_loops: 2
```

Expand Down Expand Up @@ -307,7 +309,8 @@ The server speaks MCP `2025-11-25` (and negotiates down to `2025-06-18`, `2025-0
`queued`/`running`/`planning`/`auditing`/`needs-fix` = `working`, `approved`/`applied` =
`completed`, `blocked`/`failed` = `failed`, `cancelled` = `cancelled`. Task ids are not
access-controlled: this is a local, single-user server, so only expose it to clients you trust.
The pre-spec `tasks/update` method is gone.
The pre-spec `tasks/update` method is gone. Set `mcp.async_tasks: false` in `CLODEX.md` to
turn tasks (and `clodex_task_start`) off.
- `clodex_audit` accepts `base` (everything since that ref diverged) and `commit` (one commit) as
well as the default uncommitted diff.

Expand Down
2 changes: 1 addition & 1 deletion clodex/__init__.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
"""Clodex local Claude Code + Codex CLI orchestration."""

__version__ = "0.1.1"
__version__ = "0.2.0"
8 changes: 8 additions & 0 deletions clodex/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -391,6 +391,8 @@ def print_output(data: dict[str, Any], as_json: bool) -> None:
print(json.dumps(data, indent=2, sort_keys=True))
elif "diagnostics" in data and "python" in data:
print("\n".join(format_doctor(data)))
elif "scenarios" in data and "passed" in data:
print("\n".join(format_evals(data)))
else:
print("\n".join(format_human(data)) or "(nothing to show)")

Expand Down Expand Up @@ -427,6 +429,12 @@ def format_human(data: Any, indent: int = 0) -> list[str]:
return lines


def format_evals(data: dict[str, Any]) -> list[str]:
lines = [f"{'[ok] ' if item['passed'] else '[FAIL]'} {item['name']}: {item['detail']}" for item in data["scenarios"]]
failed = sum(1 for item in data["scenarios"] if not item["passed"])
return [*lines, "", "All checks passed." if data["passed"] else f"{failed} check(s) failed."]


def format_doctor(data: dict[str, Any]) -> list[str]:
def mark(ok: bool) -> str:
return "[ok] " if ok else "[FAIL]"
Expand Down
1 change: 0 additions & 1 deletion clodex/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,6 @@
"audit": {
"quorum": "unanimous",
"max_diff_bytes": 200000,
"personas": ["security", "performance", "portability", "test-gap"],
"reviewers": [
{"id": "claude-plan", "backend": "claude", "persona": "plan-adherence", "required": True, "timeout": 600},
{"id": "codex-architecture", "backend": "codex", "persona": "architecture", "required": True, "timeout": 600},
Expand Down
177 changes: 170 additions & 7 deletions clodex/evals.py
Original file line number Diff line number Diff line change
@@ -1,18 +1,181 @@
"""`clodex eval run`: an offline self-test of the harness itself.

Everything runs in a throwaway directory and never calls Claude or Codex, so it is fast, free and
safe to run anywhere (CI included). It checks that the pieces Clodex relies on still fit together:
your CLODEX.md, the command lines it builds, the JSON schemas, the state store, workspaces, native
setup and hooks.
"""

from __future__ import annotations

import json
import subprocess
import tempfile
from collections.abc import Callable
from pathlib import Path
from typing import Any

from .config import load_config
from .commands import claude_audit_command, claude_plan_command, codex_exec_command, codex_review_command
from .config import ClodexConfig, load_config
from .hooks import HOOK_EVENTS, hook_config, merge_hooks
from .jsonutil import AgentEnvelopeError, extract_json_object
from .models import validate
from .native import plan_native_install
from .quorum import evaluate, quorum_met
from .schemas import SCHEMA_NAMES, load_schema, schema_path
from .state import StateStore
from .workspace import WorkspaceManager


class EvalFailure(Exception):
pass


def _check(condition: bool, message: str) -> None:
if not condition:
raise EvalFailure(message)


def _config_valid(config: ClodexConfig, _tmp: Path) -> str:
errors = [d for d in validate(config) if d.level == "error"]
_check(not errors, "; ".join(f"{d.where}: {d.message}" for d in errors))
return "CLODEX.md settings are valid"


def _reviewers_and_quorum(config: ClodexConfig, _tmp: Path) -> str:
required = [r for r in config.reviewers if r.get("required", True)]
_check(bool(required), "no required reviewers configured")
_check(not quorum_met(0, len(required), config.audit.get("quorum", "unanimous")), "an empty approval set must never satisfy the quorum")
_check(quorum_met(len(required), len(required), config.audit.get("quorum", "unanimous")), "all required reviewers approving must satisfy the quorum")
verdicts = [{"reviewer_id": str(r["id"]), "approved": True, "diff_hash": "h", "_required": bool(r.get("required", True))} for r in config.reviewers]
_check(evaluate(verdicts, "h", 0, config.audit.get("quorum", "unanimous"))["approved"], "unanimous approval was not accepted")
return f"{len(required)} required reviewer(s), quorum {config.audit.get('quorum', 'unanimous')}"


def _commands(config: ClodexConfig, tmp: Path) -> str:
plan, audit = claude_plan_command(config), claude_audit_command(config)
build, review = codex_exec_command(config, tmp), codex_review_command(config, tmp)
for command in (plan, audit):
_check(command.argv[:2] == ["claude", "-p"] and "--effort" in command.argv and "--output-format" in command.argv, f"{command.name}: unexpected argv")
_check("--ask-for-approval" not in build.argv, "codex exec no longer accepts --ask-for-approval")
_check(build.argv[:2] == ["codex", "exec"] and build.argv[-1] == "-", "codex-build: unexpected argv")
_check(review.argv[:2] == ["codex", "exec"] and "read-only" in review.argv, "codex-audit must run read-only")
if "--output-schema" in review.argv:
_check(Path(review.argv[review.argv.index("--output-schema") + 1]).is_file(), "the audit schema file is missing")
return "claude plan/audit and codex build/audit command lines are well-formed"


def _strict(schema: dict[str, Any], path: str = "$") -> None:
if schema.get("type") == "object":
_check(schema.get("additionalProperties") is False, f"{path}: additionalProperties must be false")
_check(set(schema.get("required", [])) == set(schema.get("properties", {})), f"{path}: every property must be required")
for name, sub in schema.get("properties", {}).items():
_strict(sub, f"{path}.{name}")
if schema.get("type") == "array":
_strict(schema["items"], f"{path}[]")


def _schemas(_config: ClodexConfig, _tmp: Path) -> str:
for name in SCHEMA_NAMES:
_check(schema_path(name).is_file(), f"{name}: schema file missing")
schema = load_schema(name)
_check("$schema" not in schema, f"{name}: Claude rejects the draft 2020-12 $schema URI")
_strict(schema, name)
return f"{len(SCHEMA_NAMES)} schemas are strict-mode compatible"


def _output_parsing(_config: ClodexConfig, _tmp: Path) -> str:
envelope = {"type": "result", "is_error": False, "result": '{"ok": true}'}
_check(extract_json_object(json.dumps(envelope)) == {"ok": True}, "a Claude result envelope was not unwrapped")
_check(extract_json_object(json.dumps({**envelope, "structured_output": {"ok": 1}})) == {"ok": 1}, "structured_output is not preferred")
try:
extract_json_object(json.dumps({"type": "result", "is_error": True, "result": "Not logged in"}))
except AgentEnvelopeError:
pass
else:
raise EvalFailure("an error envelope was accepted")
return "Claude result envelopes and fenced JSON are handled"


def _state_roundtrip(_config: ClodexConfig, tmp: Path) -> str:
store = StateStore(tmp / "state.sqlite3")
store.create_handoff("h", "task")
question = store.add_clarification("h", "codex", "Which one?")
store.answer_clarification("h", question["id"], "This one", "claude")
store.update_handoff("h", actor="claude", diff_hash="d", report={"approved": True})
delegation = store.start_delegation("h", "implement", "go")
store.update_delegation(delegation["id"], status="completed")
data = store.get_handoff("h")
_check(store.schema_version() >= 4, "state schema is out of date")
_check(bool(data and data["delegations"] and not data["open_clarifications"]), "the handoff ledger did not round-trip")
return f"schema v{store.schema_version()}: handoff, clarification and delegation round-trip"


def _git(repo: Path, *args: str) -> None:
subprocess.run(["git", *args], cwd=repo, capture_output=True, check=True)


def _workspaces(config: ClodexConfig, tmp: Path) -> str:
repo = tmp / "ws-repo"
repo.mkdir()
_git(repo, "init", "-q")
_git(repo, "config", "user.email", "eval@example.com")
_git(repo, "config", "user.name", "eval")
(repo / "a.txt").write_text("a\n", encoding="utf-8")
_git(repo, "add", "a.txt")
_git(repo, "commit", "-q", "-m", "seed")
scratch = ClodexConfig(repo_root=repo, raw={"workspace_root": ".clodex/workspaces"})
manager = WorkspaceManager(repo, scratch)
workspace = manager.prepare("eval-run", "git-worktree")
_check(workspace.path.is_dir() and workspace.is_worktree, "git worktree was not created")
_check(manager.release(workspace.path) and not workspace.path.exists(), "git worktree was not removed")
return "git worktrees can be created and removed"


def _native_setup(_config: ClodexConfig, tmp: Path) -> str:
repo = tmp / "native-repo"
repo.mkdir()
plan = plan_native_install(repo, dry_run=True)
errors = [item for item in plan["files"] if item["action"] == "error"]
_check(not errors, "; ".join(item.get("error", "") for item in errors))
_check(len(plan["files"]) >= 5, "expected CLAUDE.md, AGENTS.md, CLODEX.md, .mcp.json and .codex/config.toml")
return f"`clodex init` would manage {len(plan['files'])} files"


def _hooks(_config: ClodexConfig, _tmp: Path) -> str:
config = hook_config()
_check(set(config["hooks"]) == set(HOOK_EVENTS), "hook config does not cover the hook events")
once = merge_hooks({"theme": "dark"})
_check(merge_hooks(once) == once and once["theme"] == "dark", "merging hooks is not idempotent or lost settings")
_check(merge_hooks(once, remove=True) == {"theme": "dark"}, "removing hooks left something behind")
return f"{len(HOOK_EVENTS)} hook events, merge/remove are idempotent"


SCENARIOS: list[tuple[str, Callable[[ClodexConfig, Path], str]]] = [
("config-valid", _config_valid),
("reviewers-and-quorum", _reviewers_and_quorum),
("command-lines", _commands),
("json-schemas", _schemas),
("agent-output-parsing", _output_parsing),
("state-roundtrip", _state_roundtrip),
("git-worktrees", _workspaces),
("native-setup", _native_setup),
("hooks", _hooks),
]


def run_local_evals(repo_root: Path | None = None) -> dict[str, Any]:
"""Run every scenario; one failing scenario never stops the others."""
config = load_config(repo_root)
store = StateStore(config.state_path)
scenarios = [
{"name": "config-loads", "passed": config.workspace["backend"] in {"git-worktree", "local"}},
{"name": "state-schema-v2", "passed": store.schema_version() >= 2},
{"name": "reviewers-configured", "passed": len(config.reviewers) >= 2},
]
scenarios: list[dict[str, Any]] = []
with tempfile.TemporaryDirectory(prefix="clodex-eval-") as tmp:
for name, check in SCENARIOS:
scratch = Path(tmp) / name
scratch.mkdir()
try:
scenarios.append({"name": name, "passed": True, "detail": check(config, scratch)})
except EvalFailure as exc:
scenarios.append({"name": name, "passed": False, "detail": str(exc)})
except Exception as exc: # noqa: BLE001 - report the scenario as failed, with why
scenarios.append({"name": name, "passed": False, "detail": f"{type(exc).__name__}: {exc}"})
return {"passed": all(item["passed"] for item in scenarios), "scenarios": scenarios}
Loading
Loading