From cda2963b47a68a5689a0186c1af2d49681941029 Mon Sep 17 00:00:00 2001 From: ikem Peter Date: Thu, 15 Oct 2026 16:25:27 +0100 Subject: [PATCH 1/2] Support newer GPT models with isolated text-only catalogs --- README.md | 6 +-- harvey/codex.py | 68 ++++++++++++++++++++--------- harvey/web/index.html | 2 +- tests/fake_codex.py | 14 +++++- tests/test_codex.py | 43 ++++++++++++++++--- tests/test_codex_cli_surface.py | 76 +++++++++++++++++++-------------- tests/test_codex_e2e.py | 8 ++-- 7 files changed, 148 insertions(+), 69 deletions(-) diff --git a/README.md b/README.md index 3035270..4849593 100644 --- a/README.md +++ b/README.md @@ -432,7 +432,7 @@ Then either pick **codex** in `harvey setup` (it suggests `gpt-5.5`), run `harve ```yaml brain: provider: codex - model: gpt-5.5 # the verified text-only Codex model (see Models) + model: gpt-5.5 # or an available Luna, Sol, or Terra model ID reasoning_effort: "" # optional: minimal | low | medium | high | xhigh ``` @@ -443,7 +443,7 @@ What to know: - **No tools, verified before every prompt.** Prompts contain untrusted text (prospect replies, scraped pages), so the model must have nothing to act with. Harvey disables every tool-providing Codex feature, web search, sub-agents, hooks, skills and project instructions, starts the thread with no execution environment, and then checks Codex's own reports (effective config, account type, thread settings, MCP status, hooks) before sending anything. Anything it can't confirm stops the call. During the turn, any attempted tool call, approval request or non-text item kills the runtime. A read-only sandbox is also set, as extra protection only. - **Your login only goes to OpenAI.** Before your login is put into the runtime at all, Harvey starts Codex without it and checks that requests would go to the built-in OpenAI provider and the default ChatGPT endpoint. Any custom provider, overridden built-in provider, base URL, telemetry exporter or other endpoint in any Codex config layer (for example `/etc/codex/config.toml` or a managed config) stops the call there. - **Login handling.** Harvey supports Codex's file credential storage (`~/.codex/auth.json`, or `$CODEX_HOME/auth.json`, mode `600`): the default unless you've configured an OS keyring. Keyring storage isn't supported. On every call Harvey reads that file (read-only, never writing or moving it) and gives the runtime a private copy *without the refresh token*, deleted after the call. Codex therefore can't change or renew your real login. When the login has expired, or is due for Codex's routine renewal (after about 8 days), Codex calls stop with "Your Codex login has expired or is due for renewal". Use Codex once yourself (it renews the login automatically) or run `codex login`, and Harvey picks up the renewed login on its next call. Harvey never renews it for you. -- **Models.** Use `model: gpt-5.5`. It's the model Harvey has verified as text-only with Codex 0.159.3 (no code mode, no extra tools), and it works on ChatGPT plans even though Codex's live catalog hides it from the model picker. Codex's newer "code mode" models (`gpt-6-*`, `gpt-5.6-*`) always carry built-in code-execution tools that no setting removes, so Harvey refuses them. If you leave `model` blank, Harvey only auto-selects a model the catalog *lists* and that has no code mode or extra tools. On current ChatGPT accounts no listed model qualifies, so a blank `model` fails with an error naming the hidden models you can set explicitly, plus each model's reason. If you set `model`, Harvey uses exactly that model or fails; it never substitutes another. Naming a code-mode model gives an error listing the models that work. +- **Models.** Set `brain.model` to an exact model ID from your account's catalog, including Luna, Sol, and Terra models such as `gpt-5.6-luna`, `gpt-6.1-sol`, and `gpt-5.6-terra`. `gpt-5.5` remains supported when present in your account's catalog and is the setup suggestion. Harvey creates a private catalog copy that preserves model capabilities while selecting direct tool mode and removing experimental tools. The model receives no tools under Harvey's lockdown. A blank `model` selects the highest-priority listed model that can use this configuration; hidden models require an explicit ID. An explicit model is used exactly or the call fails. Unknown model IDs and unrecognized tool modes are rejected. Availability depends on your ChatGPT account. - **Quota.** The budget gate and dashboard read your ChatGPT plan's real rate-limit windows from Codex, for the limit bucket that applies to your model, and compare them to `usage.max_daily_claude_percent`. If Codex reports that usage is blocked, or a call fails on the plan's usage limit, Harvey treats the plan as full (even when no window is reported) and stops Codex calls for an hour. If the limits can't be read, the dashboard and `harvey usage` say "Quota unavailable" and the budget falls back to Harvey's own daily call count. Usage is recorded per provider; Codex calls are logged with tokens but no dollar cost, because the plan isn't billed per token. - **When Codex can't be used, Harvey pauses rather than guessing.** Before each heartbeat, Harvey checks everything a call needs except the prompt itself: the Codex CLI version, the isolation, your login (expired or due for renewal), your ChatGPT account, and that your configured model can run with no tools. No prompt is sent, so nothing is used from your plan beyond a quick account check. If something is wrong, the whole cycle is skipped: no reply is read or marked handled, nothing is drafted or sent. The log line, `harvey status`, and the dashboard's Actions card say "Codex paused" and why. If Codex stops working part-way through a cycle (for example the plan hits its usage limit), the reply being handled is left exactly as it was for a clean retry, not escalated, half-recorded or lost, and that cycle sends nothing. Fix the cause (usually `codex login`), and Harvey resumes on its next heartbeat: a renewed login or reinstalled CLI is noticed immediately, a usage-limit pause ends after an hour, and other fixes are picked up within 30 minutes. `harvey status --check` runs the same check on demand and exits non-zero if it fails. Brief failures (a crash or timeout) just skip one cycle. Harvey never falls back to Claude or an API key. - **Requirements.** **macOS or Linux**, Codex CLI **0.159.3** exactly, and Node.js for its npm package. **Windows isn't supported for Codex:** Harvey refuses it there before reading your login; use the Claude provider on Windows. Harvey checks the Codex version before every call and refuses any other version, including newer ones, because a different Codex can add tools Harvey's lockdown doesn't know about. Support for a new version comes with a Harvey update. Live use with a real ChatGPT account has been verified on macOS and ARM64 Linux. The offline suites also passed on x86-64 Linux under CPU emulation; CI covers Linux and macOS. The Docker image ships the Claude CLI only, so Codex isn't supported in Docker. To re-verify the lockdown offline against your installed CLI, with no credentials or network: `HARVEY_CODEX_CLI_TESTS=1 pytest tests/test_codex_cli_surface.py`. @@ -487,7 +487,7 @@ Harvey automates outreach, but **you are the sender.** Cold email is legal in mo **Claude headless mode fails** — `claude login`, and confirm the subscription is active. Test with `claude -p "say hi"`. In Docker, mount `~/.claude` into the container. -**Codex calls fail** — Read the log line; every Codex error says what to do. "Not logged in" or "expired or is due for renewal" → use Codex once or run `codex login` with ChatGPT. "Hasn't been verified with Harvey" → install the supported Codex version named in the message. "Isolation could not be verified" → a Codex config layer outside your `~/.codex` (for example a managed or `/etc/codex` config) forces a tool, hook, MCP server, instructions or custom endpoint back on; Harvey refuses to send prompts until it's gone. "Always exposes tools" or "no Codex model is available" → set `brain.model: gpt-5.5` (the message lists each model and why). Harvey never switches to Claude on its own. +**Codex calls fail** — Read the log line; every Codex error says what to do. "Not logged in" or "expired or is due for renewal" → use Codex once or run `codex login` with ChatGPT. "Hasn't been verified with Harvey" → install the supported Codex version named in the message. "Isolation could not be verified" → a Codex config layer outside your `~/.codex` (for example a managed or `/etc/codex` config) forces a tool, hook, MCP server, instructions or custom endpoint back on; Harvey refuses to send prompts until it's gone. Model catalog errors → choose an available model ID from the catalog shown in the message, or leave `brain.model` blank to select a listed model. Harvey never switches to Claude on its own. **Discovery finds nothing** — Confirm signals first (`harvey signals --confirm free`); nothing is collected until you do. If you're on the free OpenStreetMap source, it only covers mapped trades and is thin for service-area businesses — `harvey discover --providers` shows the alternatives. diff --git a/harvey/codex.py b/harvey/codex.py index 98dfd9d..4fe3b97 100644 --- a/harvey/codex.py +++ b/harvey/codex.py @@ -19,9 +19,9 @@ endpoint, ChatGPT-only login, file credential storage, no custom provider definitions and no other endpoint anywhere in the config. Only then is the login exposed: the runtime is restarted with it, and checked again. -2. The model must not be one of Codex's "code mode" models, which always - carry code-execution tools no setting removes (checked against the model - catalog ``codex debug models`` reports for this runtime). +2. The model must exist in the runtime catalog. A private copy keeps its + capabilities while forcing direct tool mode and no experimental tools. + The restarted runtime must report that exact private catalog path. 3. ``account/read`` must report a ChatGPT account (never an API key, so a call is never billed per token). 4. The thread starts with no environments (no exec or file-edit tools), no @@ -94,9 +94,8 @@ def require_supported_platform() -> None: # tool Harvey's lockdown doesn't know about. SUPPORTED_CODEX_VERSIONS = ("0.159.3",) CODEX_INSTALL_HINT = f"npm install -g @openai/codex@{SUPPORTED_CODEX_VERSIONS[-1]}" -# The model setup suggests: verified text-only (no code mode, no extra tools) -# with 0.159.3, and callable on ChatGPT plans even where the live catalog -# hides it from the picker, so blank auto-selection can't find it. +# Keep the setup suggestion stable. Explicit newer models and catalog-based +# auto-selection use the same private text-only catalog. SUGGESTED_MODEL = "gpt-5.5" # An access token closer than this to expiry is treated as expired: the @@ -528,7 +527,7 @@ def build_child_env(private_root: str, codex_home: str, base_env=None) -> dict: return env -def attest_effective_config(response) -> list[str]: +def attest_effective_config(response, model_catalog_path: str = "") -> list[str]: """Check config/read output; return inherited MCP server names.""" if not isinstance(response, dict) or not isinstance(response.get("config"), dict): raise _isolation_error("config/read returned no effective config") @@ -544,6 +543,10 @@ def attest_effective_config(response) -> list[str]: if config.get("chatgpt_base_url") not in (None, DEFAULT_CHATGPT_BASE_URL): raise _isolation_error("a custom ChatGPT endpoint is configured") for key in MUST_BE_EMPTY: + if key == "model_catalog_json" and model_catalog_path: + if config.get(key) != model_catalog_path: + raise _isolation_error("the private model catalog was overridden") + continue if config.get(key) not in (None, "", [], {}): raise _isolation_error(f"setting '{key}' is set by another config layer") endpoint = _find_custom_endpoint(config) @@ -666,21 +669,33 @@ def parse_model_catalog(raw: bytes) -> dict: def model_tools_removable(entry) -> bool: - """Whether Harvey's lockdown leaves this model with no tools at all. + """Whether an effective catalog entry respects the text-only lockdown. - Models in Codex's code mode (``tool_mode``) always get its JavaScript - ``exec``/``wait``/user-input tools, and some get extra experimental - tools; no supported setting removes them. Verified against the Codex - CLI with a local mock model server: other models receive an empty tool - list under Harvey's lockdown. + Direct mode adds no execution wrapper. With no environments and all + tool-providing features disabled, the pinned CLI sends no tools. """ return ( isinstance(entry, dict) - and entry.get("tool_mode") in (None, "") + and entry.get("tool_mode") in (None, "", "direct") and not entry.get("experimental_supported_tools") ) +def text_only_catalog(catalog: dict) -> dict: + """Keep model capabilities, but remove model-forced tools in our runtime. + + The pinned CLI honors a private model_catalog_json before building its + tool registry. Direct mode respects the existing no-environments and + disabled-feature lockdown. Unknown tool modes still fail closed. + """ + return { + slug: {**entry, "tool_mode": "direct", "experimental_supported_tools": []} + if entry.get("tool_mode") in (None, "", "direct", "code_mode", "code_mode_only") + else dict(entry) + for slug, entry in catalog.items() + } + + def _safe_name(value) -> str: """Catalog text (from the backend) made safe to log.""" return re.sub(r"[^A-Za-z0-9._:-]", "?", str(value))[:40] @@ -690,7 +705,7 @@ def describe_model(entry) -> str: """Why a catalog entry can or can't be used, in a few words.""" if not isinstance(entry, dict): return "unreadable" - if entry.get("tool_mode") not in (None, ""): + if entry.get("tool_mode") not in (None, "", "direct"): return f"code mode ({_safe_name(entry['tool_mode'])})" extra = entry.get("experimental_supported_tools") if extra: @@ -937,6 +952,7 @@ def __init__(self, codex_path: str): self.codex_home = "" self.workspace = "" self.version = "" + self.model_catalog_path = "" self.proc: asyncio.subprocess.Process | None = None self._next_id = 1 self._stderr_tail = b"" @@ -976,6 +992,17 @@ def install_auth(self, snapshot: dict): kind="runtime", ) from None + def install_model_catalog(self, catalog: dict) -> None: + """Install Harvey's generated catalog in this disposable runtime only.""" + self.model_catalog_path = os.path.join(self.codex_home, "text-models.json") + with open(self.model_catalog_path, "x", encoding="utf-8") as f: + json.dump({"models": list(catalog.values())}, f) + path = os.path.join(self.codex_home, "config.toml") + with open(path, encoding="utf-8") as f: + config = f.read() + with open(path, "w", encoding="utf-8") as f: + f.write(f"model_catalog_json = {_toml_value(self.model_catalog_path)}\n" + config) + def _child_kwargs(self) -> dict: # Own process group, so the whole tree can be killed. return {"cwd": self.workspace, "env": build_child_env(self.root, self.codex_home), @@ -1307,12 +1334,14 @@ async def _open_session(self, server: _AppServer, deadline: float, server.install_auth(snapshot) model = self.model if toolless_model or not model: - catalog = await server.read_catalog(deadline) + catalog = text_only_catalog(await server.read_catalog(deadline)) model = model or self._default_model(catalog) if toolless_model: require_toolless_model(catalog, model) + server.install_model_catalog(catalog) await server.launch(deadline) - mcp_names = attest_effective_config(await server.read_config(deadline)) + mcp_names = attest_effective_config( + await server.read_config(deadline), server.model_catalog_path) await self._require_chatgpt(server, deadline) return model, mcp_names @@ -1321,9 +1350,10 @@ async def _attested_thread(self, server: _AppServer, deadline: float) -> tuple[s the configured model; returns (thread id, model). Nothing is sent to the model.""" model, mcp_names = await self._open_session(server, deadline, toolless_model=True) + params = build_thread_params(server.workspace, model, mcp_names) + params["config"]["model_catalog_json"] = server.model_catalog_path thread_id, model = attest_thread(await server.request( - "thread/start", build_thread_params(server.workspace, model, mcp_names), - deadline), model) + "thread/start", params, deadline), model) attest_mcp_status( await self._mcp_status_pages(server, thread_id, deadline), mcp_names) attest_hooks(await server.request( diff --git a/harvey/web/index.html b/harvey/web/index.html index b092374..bc5da05 100644 --- a/harvey/web/index.html +++ b/harvey/web/index.html @@ -403,7 +403,7 @@

Common Issues

Codex calls fail -
With brain.provider: codex, set brain.model: gpt-5.5 (the verified text-only model; a blank model fails on current ChatGPT accounts and code-mode models are refused), run codex login and choose "Sign in with ChatGPT" (API-key logins are refused). Harvey never renews your Codex login: if the log says it has expired or is due for renewal, use Codex once or run codex login. Harvey needs Codex CLI 0.159.3 on macOS or Linux (Codex isn't supported on Windows; use Claude there) and refuses other versions. While Codex can't be used, Harvey pauses (the Actions card shows "Codex paused" and why), leaves replies and queued emails untouched, and resumes once it's fixed; harvey status --check checks on demand. If the log says isolation could not be verified, a Codex config layer is forcing a tool, hook, MCP server or custom endpoint back on — Harvey refuses to send prompts until it is removed. Harvey never falls back to Claude on its own.
+
With brain.provider: codex, set brain.model to an available model ID, including Luna, Sol, or Terra (for example gpt-6.1-sol), or leave it blank to select a listed model. Harvey runs these models without tools; gpt-5.5 also remains supported when present in your account's catalog. Run codex login and choose "Sign in with ChatGPT" (API-key logins are refused). Harvey never renews your Codex login: if the log says it has expired or is due for renewal, use Codex once or run codex login. Harvey needs Codex CLI 0.159.3 on macOS or Linux (Codex isn't supported on Windows; use Claude there) and refuses other versions. While Codex can't be used, Harvey pauses (the Actions card shows "Codex paused" and why), leaves replies and queued emails untouched, and resumes once it's fixed; harvey status --check checks on demand. If the log says isolation could not be verified, a Codex config layer is forcing a tool, hook, MCP server or custom endpoint back on — Harvey refuses to send prompts until it is removed. Harvey never falls back to Claude on its own.
diff --git a/tests/fake_codex.py b/tests/fake_codex.py index 301f9c2..5353c38 100644 --- a/tests/fake_codex.py +++ b/tests/fake_codex.py @@ -11,6 +11,7 @@ import os import sys import time +import tomllib SCENARIO, LOG = sys.argv[1], sys.argv[2] ARGS = sys.argv[3:] @@ -57,15 +58,20 @@ def read_auth(): "home": os.environ.get("HOME"), "codex_home": os.environ.get("CODEX_HOME")}) catalog = dict(CATALOG) if SCENARIO == "no_toolless_model": - catalog = {"models": [CATALOG["models"][0]]} + catalog = {"models": [{**CATALOG["models"][0], "tool_mode": "future_mode"}]} if SCENARIO == "unlisted_toolless": # The tool-less model is in the catalog but not listed in the picker. - catalog = {"models": [CATALOG["models"][0], {**CATALOG["models"][1], "visibility": "hide"}]} + catalog = {"models": [{**m, "visibility": "hide"} for m in CATALOG["models"]]} sys.stdout.write(json.dumps(catalog)) sys.exit(0) assert ARGS[:1] == ["app-server"], ARGS SNAPSHOT = read_auth() +with open(os.path.join(os.environ["CODEX_HOME"], "config.toml"), "rb") as f: + catalog_path = tomllib.load(f).get("model_catalog_json") +if catalog_path: + with open(catalog_path) as f: + log({"private_catalog": json.load(f)}) log({"cmd": "app-server", "pid": os.getpid(), "cwd": os.getcwd(), "argv": ARGS, "env_keys": sorted(os.environ), "home": os.environ.get("HOME"), "codex_home": os.environ.get("CODEX_HOME"), @@ -104,6 +110,10 @@ def effective_config(): "skills": {"include_instructions": False, "bundled": {"enabled": False}}, "agents": {"enabled": False}, "mcp_servers": {}, } + with open(os.path.join(os.environ["CODEX_HOME"], "config.toml"), "rb") as f: + config["model_catalog_json"] = tomllib.load(f).get("model_catalog_json") + if SCENARIO == "catalog_override" and SNAPSHOT is not None: + config["model_catalog_json"] = "/tmp/foreign-models.json" if SCENARIO == "feature_on": config["features"]["shell_tool"] = True if SCENARIO == "custom_provider": diff --git a/tests/test_codex.py b/tests/test_codex.py index ea61c08..3440ef5 100644 --- a/tests/test_codex.py +++ b/tests/test_codex.py @@ -125,7 +125,7 @@ async def test_isolated_turn_returns_final_answer_and_usage(fake, native_home): result = await _client(native_home).run(PROMPT, timeout=30) assert result.text == '{"intent": "interested"}' # final_answer, not commentary - assert result.model == "gpt-5.5" # Harvey's tool-less default, not Codex's + assert result.model == "gpt-6.1-sol" # highest-priority listed model assert result.thread_id == "thr-1" assert result.duration_ms == 850 # The turn total (two model responses), not just the last response (5/1). @@ -331,6 +331,7 @@ def test_cli_version_parsing(): ("apikey", "setup"), ("logged_out", "auth"), ("no_toolless_model", "isolation"), + ("catalog_override", "isolation"), ]) async def test_unsafe_or_unverifiable_state_never_sends_the_prompt(fake, native_home, scenario, kind): fake.use(scenario) @@ -368,12 +369,19 @@ async def test_rate_limit_reads_attest_the_route_too(fake, native_home): @pytest.mark.asyncio -async def test_code_mode_model_is_refused_before_the_logged_in_launch(fake, native_home): - with pytest.raises(hc.CodexError) as err: - await _client(native_home, model="gpt-6.1-sol").run(PROMPT, timeout=30) - assert err.value.kind == "isolation" - assert "gpt-5.5" in str(err.value) # names a model that works - assert [s["auth"] for s in fake.servers()] == [None] # only the credential-less check ran +async def test_code_mode_model_runs_with_a_private_text_catalog(fake, native_home): + result = await _client(native_home, model="gpt-6.1-sol").run(PROMPT, timeout=30) + assert result.model == "gpt-6.1-sol" + first, second = fake.servers() + assert first["auth"] is None and second["auth"] is not None + config = tomllib.loads(second["config_toml"]) + assert config["model_catalog_json"].startswith(second["codex_home"] + "/") + assert not Path(config["model_catalog_json"]).exists() # cleaned up after call + catalog = next(e["private_catalog"] for e in fake.entries() if "private_catalog" in e) + assert all(m["tool_mode"] == "direct" and m["experimental_supported_tools"] == [] + for m in catalog["models"]) + params = next(e["params"] for e in fake.entries() if e.get("method") == "thread/start") + assert params["config"]["model_catalog_json"] == config["model_catalog_json"] @pytest.mark.asyncio @@ -861,3 +869,24 @@ def test_parse_rate_limits(): assert hc.parse_rate_limits({"primary": {"usedPercent": 5, "windowDurationMins": 60}}) == { "primary": {"utilization": 5.0, "resets_at": "", "window_minutes": 60}} assert hc.parse_rate_limits({}) is None + + +@pytest.mark.parametrize("model", ["gpt-5.6-luna", "gpt-6.1-sol", "gpt-5.6-terra", "gpt-9-future"]) +@pytest.mark.parametrize("mode", [None, "direct", "code_mode", "code_mode_only"]) +def test_text_only_catalog_preserves_model_capabilities(model, mode): + entry = {"slug": model, "visibility": "list", "priority": 1, "tool_mode": mode, + "experimental_supported_tools": ["clock"], "context_window": 200000, + "supported_reasoning_levels": [{"effort": "high"}]} + catalog = {model: entry} + effective = hc.text_only_catalog(catalog) + assert effective[model] == {**entry, "tool_mode": "direct", "experimental_supported_tools": []} + assert entry["experimental_supported_tools"] == ["clock"] + assert entry["tool_mode"] == mode + assert hc.pick_default_model(effective) == model + hc.require_toolless_model(effective, model) + + +def test_text_only_catalog_rejects_unknown_tool_mode(): + catalog = {"future": {"slug": "future", "tool_mode": "future_mode", "visibility": "list"}} + with pytest.raises(hc.CodexError): + hc.require_toolless_model(hc.text_only_catalog(catalog), "future") diff --git a/tests/test_codex_cli_surface.py b/tests/test_codex_cli_surface.py index bcc54d7..3083ee8 100644 --- a/tests/test_codex_cli_surface.py +++ b/tests/test_codex_cli_surface.py @@ -25,6 +25,7 @@ import time from datetime import datetime, timezone from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer +from pathlib import Path import pytest @@ -189,9 +190,10 @@ async def test_real_cli_model_sees_no_tools_and_no_inherited_instructions(mock_c request = mock_codex.requests[0] assert _model_visible_tools(request) == [] assert not request.get("instructions") - roles = [(i.get("type"), i.get("role")) for i in request["input"]] + messages = [i for i in request["input"] if i.get("type") == "message"] + roles = [(i.get("type"), i.get("role")) for i in messages] assert roles == [("message", "developer"), ("message", "user")] - assert request["input"][0]["content"][0]["text"] == hc.DEVELOPER_INSTRUCTIONS + assert messages[0]["content"][0]["text"] == hc.DEVELOPER_INSTRUCTIONS @pytest.mark.asyncio @@ -222,23 +224,37 @@ async def test_real_cli_inherited_mcp_server_never_starts(mock_codex, monkeypatc @pytest.mark.asyncio -async def test_real_cli_code_mode_models_are_refused(mock_codex): - catalog = None - client = hc.CodexClient(native_home=mock_codex.native_home) - server = hc._AppServer(client._resolve_codex()) - try: - server.prepare() - catalog = await server.read_catalog(hc.time.monotonic() + 30) - finally: - await server.close() - code_mode = [slug for slug, m in catalog.items() if not hc.model_tools_removable(m)] - if not code_mode: - pytest.skip("this Codex catalog has no code-mode models") - with pytest.raises(hc.CodexError) as err: - await hc.CodexClient(model=code_mode[0], native_home=mock_codex.native_home).run( - "ping", timeout=60) - assert err.value.kind == "isolation" - assert mock_codex.requests == [] +@pytest.mark.parametrize("selection", ["explicit", "omitted", "blank", "null"]) +async def test_real_cli_all_catalog_models_and_config_defaults( + mock_codex, monkeypatch, tmp_path, selection): + import yaml + + from harvey.brain import Brain + from harvey.config import load_config + from harvey.state import StateManager + + monkeypatch.setattr(hc, "DISABLED_FEATURES", + hc.DISABLED_FEATURES + ("enable_request_compression",)) + catalog = hc.parse_model_catalog(json.dumps(_bundled_catalog(tmp_path)).encode()) + expected_default = hc.pick_default_model(hc.text_only_catalog(catalog)) + models = list(catalog) if selection == "explicit" else [None] + for model in models: + mock_codex.requests.clear() + config_data = yaml.safe_load((Path(__file__).parents[1] / "harvey.yaml").read_text()) + config_data["brain"] = {"provider": "codex"} + if selection != "omitted": + config_data["brain"]["model"] = model if selection == "explicit" else ( + "" if selection == "blank" else None) + config_path = tmp_path / "selection.yaml" + config_path.write_text(yaml.safe_dump(config_data)) + config = load_config(str(config_path)) + brain = Brain(StateManager(str(tmp_path / "unused.db")), config.brain) + result = await brain.codex.run("ping", timeout=60) + expected = model if selection == "explicit" else expected_default + assert result.text == "PONG" and result.model == expected + assert len(mock_codex.requests) == 1 + assert mock_codex.requests[0]["model"] == expected + assert _model_visible_tools(mock_codex.requests[0]) == [] # ── route: hostile endpoints never receive the login ── @@ -346,7 +362,7 @@ async def test_real_cli_control_attacker_would_see_the_login(attacker, monkeypat deliver the (fake) login to the attacker — so the test above is real.""" top, tables = _hostile_routes(attacker.base)["chatgpt_base_url"] attacker.configure(top, tables) - monkeypatch.setattr(hc, "attest_effective_config", lambda response: []) + monkeypatch.setattr(hc, "attest_effective_config", lambda response, model_catalog_path="": []) with pytest.raises(hc.CodexError): await hc.CodexClient(native_home=attacker.native_home).read_rate_limits(timeout=60) assert attacker.saw_login() @@ -391,15 +407,11 @@ async def test_real_cli_unlisted_toolless_model_only_runs_when_explicit( if not toolless: pytest.skip("this Codex catalog has no tool-less model") for m in catalog["models"]: - m["visibility"] = "hide" if hc.model_tools_removable(m) else m.get("visibility") - path = tmp_path / "catalog.json" - path.write_text(json.dumps(catalog)) - # Test-only stand-in for a remote catalog; real configs may not set it. - monkeypatch.setattr(hc, "MUST_BE_EMPTY", tuple( - k for k in hc.MUST_BE_EMPTY if k != "model_catalog_json")) - with_mock = hc.build_private_config_toml - monkeypatch.setattr(hc, "build_private_config_toml", - lambda: f"model_catalog_json = {json.dumps(str(path))}\n" + with_mock()) + m["visibility"] = "hide" + # Stand in for the backend catalog without relaxing config attestation. + async def read_catalog(self, deadline): + return hc.parse_model_catalog(json.dumps(catalog).encode()) + monkeypatch.setattr(hc._AppServer, "read_catalog", read_catalog) with pytest.raises(hc.CodexError) as err: await hc.CodexClient(native_home=mock_codex.native_home).run("ping", timeout=60) @@ -421,8 +433,6 @@ async def test_real_cli_preflight_checks_the_model_without_inference(mock_codex, hc.DISABLED_FEATURES + ("enable_request_compression",)) await hc.CodexClient(model="gpt-5.5", native_home=mock_codex.native_home).preflight(timeout=60) assert mock_codex.requests == [] - with pytest.raises(hc.CodexError) as err: - await hc.CodexClient(model="gpt-6.1-sol", - native_home=mock_codex.native_home).preflight(timeout=60) - assert err.value.kind == "isolation" + await hc.CodexClient(model="gpt-6.1-sol", + native_home=mock_codex.native_home).preflight(timeout=60) assert mock_codex.requests == [] diff --git a/tests/test_codex_e2e.py b/tests/test_codex_e2e.py index bd8968e..a1f9c17 100644 --- a/tests/test_codex_e2e.py +++ b/tests/test_codex_e2e.py @@ -348,13 +348,13 @@ async def check_paused(): @pytest.mark.asyncio async def test_rejected_model_pauses_with_an_actionable_reason(monkeypatch, tmp_path): - harvey = Harness(monkeypatch, tmp_path, model="gpt-6.1-sol") # code mode + harvey = Harness(monkeypatch, tmp_path, model="missing-model") # absent from catalog await harvey.state.init_db() await _open_thread(harvey.state) harvey.mail.inbound = [_reply("Could you share pricing details?")] await harvey.run_cycles(1) status = await harvey.status() - assert status["status"] == "paused" and "gpt-5.5" in status["reason"] + assert status["status"] == "paused" and "missing-model" in status["reason"] assert not await harvey.state.is_reply_processed("in-1") assert sum(1 for e in harvey.entries() if "prompt_chars" in e) == 0 @@ -633,7 +633,7 @@ async def active(status): @pytest.mark.asyncio async def test_rejected_model_never_sends_queued_mail(monkeypatch, tmp_path): - harvey = Harness(monkeypatch, tmp_path, model="gpt-6.1-sol") # code mode + harvey = Harness(monkeypatch, tmp_path, model="missing-model") # absent from catalog await harvey.state.init_db() await _open_thread(harvey.state) await _approved_item(harvey.state) @@ -642,7 +642,7 @@ async def test_rejected_model_never_sends_queued_mail(monkeypatch, tmp_path): assert harvey.mail.sent == [] status = await harvey.status() assert status["status"] == "paused" and status["kind"] == "isolation" - assert "gpt-5.5" in status["reason"] + assert "missing-model" in status["reason"] assert not await harvey.state.is_reply_processed("in-1") assert not [e for e in harvey.entries() if "prompt_chars" in e] # Caught by the heartbeat preflight, before any agent ran: the mailbox From 08e4fb441a9e21b176a16423b8860d72c866819c Mon Sep 17 00:00:00 2001 From: Ethan Date: Thu, 15 Oct 2026 14:27:54 -0400 Subject: [PATCH 2/2] Keep real-CLI model selection tests on the fake Codex login --- tests/test_codex_cli_surface.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/tests/test_codex_cli_surface.py b/tests/test_codex_cli_surface.py index 3083ee8..740a997 100644 --- a/tests/test_codex_cli_surface.py +++ b/tests/test_codex_cli_surface.py @@ -125,6 +125,9 @@ def mock_codex(monkeypatch, tmp_path): port = server.server_address[1] native = _fake_native_login(tmp_path) + # Brain constructs its own CodexClient: keep those clients on the fake + # login too, never the operator's real CODEX_HOME or ~/.codex. + monkeypatch.setenv("CODEX_HOME", str(native)) extra = {"toml": ""} lockdown = hc.build_private_config_toml