Skip to content

Test the secret-path gate the README's security model rests on - #3

Merged
Erfouni merged 1 commit into
mainfrom
test/secret-path-gate
Sep 20, 2026
Merged

Erfouni merged 1 commit into
mainfrom
test/secret-path-gate

Conversation

@Erfouni

@Erfouni Erfouni commented Sep 20, 2026 •

Copy link
Copy Markdown
Owner

What

Tests for the request gate and for secret-path persistence, plus the repo's first CI workflow. No production code changes — server.py and config.py are untouched.

  • tests/test_secret_path_gate.py — 14 new cases for server.SecretPathAuth
  • tests/test_config.py — 8 new cases for config._load_or_create_secret
  • .github/workflows/ci.yml — new

Why

server.py states the entire security model in its module docstring:

the transport is mounted at an unguessable URL path (ChatGPT's connectors cannot send custom headers, so the URL itself has to carry the secret), and the socket only listens on 127.0.0.1

That is asserted, and it was never tested. This server is deliberately exposed through an ngrok tunnel, so the path check is the only thing standing between the open internet and a COM handle on AutoCAD. A gate that quietly stops gating looks exactly like one that works.

The case most worth pinning down is prefix confusion. The current check is correct:

if not (path == self.prefix or path.startswith(self.prefix + "/")):

but the natural "simplification" to a bare path.startswith(self.prefix) would admit /<secret>x, /<secret>-extra and /<secret>.json — turning the secret from a password into a guessable stem. Nothing would have caught that.

Secret persistence matters for a less dramatic but more likely failure: the connector URLs pasted into ChatGPT and Manus embed the secret path. If a restart regenerated it, every connector would start 404ing with no obvious cause.

Coverage

SecretPathAuth — driven as plain ASGI middleware with a stub downstream app, so no uvicorn, no COM, no AutoCAD:

  • exact secret path and paths beneath it pass through
  • unrelated paths, /, and the secret appearing later in the path → 404
  • /<secret>x, /<secret>-extra, /<secret>.json → 404
  • non-http scopes (lifespan, websocket) pass through without being path-checked
  • wrong bearer → 401; correct → through; missing → through, by design (ChatGPT cannot send headers)
  • bearer without the Bearer scheme → 401; surrounding whitespace tolerated
  • wrong bearer on a wrong path → still 404, so a bad path cannot be used to probe the token

_load_or_create_secret — generated once and written; reused on the next start; read back without the trailing newline; blank file replaced; env override wins and writes no file; override beats a stored secret; generated values do not repeat across 5 runs; MCP_PATH derives from the secret.

How it was verified

python -m unittest discover -s tests -t . → 37 tests, OK (Python 3.13, Windows).

These are new tests for existing correct code, so "fails on the old code" does not apply directly. Instead each was checked against a deliberately broken copy, rather than just run green:

Mutation Result
prefix check → bare startswith(self.prefix) 4 failures
bearer check removed 2 failures
path check skipped entirely 8 failures
.secret reuse removed 2 failures
env override ignored 3 failures

server.py and config.py were restored byte-identical afterwards (git diff clean).

Every CI step was also dry-run locally: pip install -r requirements.txt deps already present, compileall clean, the test suite green, and the PowerShell parse step run in a real PowerShell — ok: run.ps1, ok: start_ngrok.ps1. Both pinned action SHAs were confirmed to be exactly the current v4/v5 tags via the GitHub API.

What was not verified

  • No live AutoCAD. Nothing in these tests touches COM; tools.py and acad.py are only byte-compiled, not exercised.
  • The gate is tested in isolation, not end-to-end. build_app() is never called, so the real FastMCP transport, the /health route, and uvicorn's own request handling are out of scope here.
  • CI has not run on windows-latest yet — this is the first workflow in the repo. Local verification was Windows 11 / Python 3.13, which is the same shape.
  • No tunnel was stood up; the 127.0.0.1-only binding claim is not tested.

Follow-ups, not folded in

  • The bearer token is compared with !=, which is not timing-safe; hmac.compare_digest would be better. Left out deliberately — it is a behaviour-identical hardening change, so no test can fail on the old code, and mixing it in would make this PR two things. Happy to send it separately.
  • There is no LICENSE file. That is the owner's choice, not mine to make.

server.py states the whole security model in its docstring: the transport is
mounted at an unguessable URL path, and that path is the gate because ChatGPT's
connectors cannot send an Authorization header. Nothing tested it. A gate that
quietly stops gating looks exactly like one that works, and this server is
deliberately exposed through an ngrok tunnel.

tests/test_secret_path_gate.py drives SecretPathAuth as plain ASGI middleware
with a stub downstream app -- no uvicorn, no COM, no AutoCAD. 14 cases:

  - the exact secret path and paths beneath it pass through;
  - unrelated paths, the root, and the secret appearing later in the path 404;
  - `/<secret>x`, `/<secret>-extra` and `/<secret>.json` 404. A bare
    startswith(prefix) check would admit all three and turn the secret into a
    guessable stem, so this is the case worth pinning down;
  - non-http scopes (lifespan, websocket) pass without being path-checked;
  - a wrong bearer is 401, a right one passes, a missing one passes by design;
  - a wrong bearer on a wrong path is still 404, so a bad path cannot be used
    to probe the token.

tests/test_config.py gains 8 cases for _load_or_create_secret. The connector
URLs pasted into ChatGPT and Manus embed the secret path, so if a restart
regenerated it every connector would start 404ing with no obvious cause:
generated once and written, reused on the next start, read back without the
trailing newline, blank file replaced, env override wins and writes no file,
override beats a stored secret, generated values do not repeat, and MCP_PATH
derives from the secret.

The tests were checked against deliberately broken copies rather than just run
green. Replacing the prefix check with a bare startswith fails 4; removing the
bearer check fails 2; skipping the path check entirely fails 8; dropping the
.secret reuse fails 2; ignoring the env override fails 3.

The repo had no CI, so none of this would run on its own. The workflow uses
windows-latest -- config and tools deal in Windows paths and server imports
pywin32 -- and adds compileall and a PowerShell parse check. Actions are pinned
by SHA, matching helios-llm-orchestrator.

No production code changes in this commit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@Erfouni
Erfouni merged commit 195da76 into main Sep 20, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant