Skip to content

Latest commit

 

History

History
244 lines (217 loc) · 15 KB

File metadata and controls

244 lines (217 loc) · 15 KB

What you bought, and what is free

GuardStack is assembled from five of my own MIT-licensed projects. All five are free on GitHub and will stay free. Nothing here is hidden from you:

Layer Source repo Free at
Input / output guard llm-guardrails github.com/jigonyoo/llm-guardrails
Permission grants + audit mcp-permission-server github.com/jigonyoo/mcp-permission-server
Approval gate agent-approval-gate github.com/jigonyoo/agent-approval-gate
Retry / circuit breaker agent-reliability-kit github.com/jigonyoo/agent-reliability-kit
Read-only enforcement readonly-guard github.com/jigonyoo/readonly-guard

So what is the paid part?

The five repos are five separate demos. Each has its own config, its own audit log, its own idea of what "denied" means, and they do not compose. Wiring them together is a real job, and I did it:

  1. One lifecycle. Five gates in a fixed order, one call each, one shared hash-chained audit log. check_input → check_tool → execute → check_output.
  2. The demo world is configuration. Both agent-approval-gate and mcp-permission-server hardcode a support-desk / filesystem demo, and any tool outside those worlds silently produced nonsense. The tool taxonomy, the resource mapping, the order-id shape, the order lookup the staleness rule needs and the exact-scope tool list are all settings here. Correction: 1.0.0 and 1.1.x said "the demo world is gone" and that was not true — the order-id regex, the staleness lookup and one tool-name special case were still hardcoded. A reviewer found it; 1.1.2 fixed it. docs/LIMITS.md lists what is still not configurable.
  3. Adapters for the frameworks you are actually using.
  4. A test suite for the composition, not for five demos — offline, no API key, pytest -q in about three seconds. The count is deliberately not printed here. This file has now carried a stale one twice: "under a second" survived the suite growing, then "337 tests, ~2.3 s" was copied into 1.2.3 and 1.3.0, and the correction that replaced it went stale in its turn. A number that has to be copied by hand into four files is a number that will be wrong in at least one of them. The current count is in CHANGELOG.md's re-measurement table for this release, where tests/test_documented_counts.py fails the build if it drifts from what pytest actually collects.
  5. The benchmark, reproducible on your machine, plus an ablation that prices all five gates — and says, per section, which numbers came from your corpus and which came from the shipped fixtures.
  6. docs/LIMITS.md — what this does not do, measured. The input guard blocks 27/27 of our corpus and 5/42 of the corpus written to break it — and that corpus ships in corpus/reviewer/, so the number is reproducible rather than quoted. Budgets are per-process; the log assumes one writer; readonly() has five measured bypasses; the approval gate's holds live in one process's memory. "…and nothing in this package deduplicates a side effect (§9)." That clause stood here through 1.2.3 and is false as of 1.3.0 — see item 7. Read §9 before you trust any of it.
  7. Gate 3c — side-effect deduplication, new in 1.3.0. The item this file spent four releases listing as a limitation is now a feature, and the section below says exactly what was taken from the free repos and what had to be built. Measured, four attempts at one $24.00 refund on one broken order: 11 duplicate refunds ($264.00) without it, 0 with it; 40 concurrent OS processes racing one identical refund: 1 executed, 39 deduplicated. python3 bench/dup_probe.py prints both. It is off until you set dedup_dir, and §9 prices what it costs — because it does cost something, and the cost is a missed refund.

If you would rather assemble the five yourself, you can, and the free repos are a fine place to start. This is the two weeks you skip.

The idempotency contradiction, and how 1.3.0 resolves it

Two of the free repos advertise idempotency on their front page. Through 1.2.3 the paid product denied it in four documents. That is not a small inconsistency to leave standing, and a buyer who read both was entitled to ask which one was lying.

free repo its front-page number
agent-reliability-kit dup-charges retry-only 5 / reliable 0
agent-approval-gate duplicate side effects 4 → 0

The four denials were in README.md ("Nothing in this package deduplicates a side effect"), QUICKSTART.md §4, NOTICE.md item 6, and docs/LIMITS.md §9 ("There is no side-effect deduplication anywhere in this package"). All four are corrected in 1.3.0, and all four are corrected in place, with the old sentence left visible, rather than quietly deleted.

But do not read that as "the free repos had it and we forgot to port it." They are not the same mechanism, and the honest description is that they are two halves of one problem:

  • What the free repos do — and what 1.3.0 takes from them. Both derive a stable key from a call and hand it to the provider, inside one process. dup-charges 5 → 0 is a retry loop that carries the same idempotency_key on all five attempts, so Stripe refuses four of them. The dedup is done by the provider; the repo's contribution is that the key does not change between attempts. That half is in 1.3.0 as gs.idempotency_key(tool, args) — a pure function, no state, no filesystem, works with the gate switched off. This is the half that was portable, and saying otherwise would be taking credit for someone else's idea.
  • What is new here, and was in neither repo. gs.once(tool, args) refuses the second execution across processes, before anything reaches the provider. A provider key cannot help when the duplicate is a second support ticket arriving at a second gunicorn worker forty minutes later — that is four different calls to your code, and only the provider ever sees they were the same, after you have already made the call. The claim file, the O_CREAT|O_EXCL race, the TTL, the stale-claim break, dedup_key_args, and holding one claim across a whole execute() retry loop are all new in this package.
  • The keys the free repos generate are per-task; this one is not, on purpose. See the changelog entry for 1.3.0: the first implementation here copied the free repos' scoping and folded task_id into the side effect's identity, and that made the gate miss the exact incident it was built for.

So: the free repos let your provider recognise your retry. Gate 3c stops the call being made a second time at all. You want both, and 1.3.0 ships both — but neither one returns you the first call's result, and §9 says so.

Defects found in our own documentation, in v1.2.2

The list above is what the free repos got wrong. This is what this bundle got wrong, found by attacking 1.2.1 the same way. The approval gate itself held — all four ways into 1.2.0 were re-run across 46 attempts, and the reviewer measured 45 of the 46 failing closed. Nothing was allowed in any of them; the odd one out raised TypeError instead of denying, and an exception is not a denial. That case has since been fixed and its nine-attempt family re-run at 9/9 denied — the other 37 attempts have not been re-run, so 45 of 46 is still the measured figure for the round, and 46 of 46 is not a number anyone has produced. docs/LIMITS.md §9 and §11 carry both halves. What did not hold was the documentation, and two of the fixes 1.2.1 shipped:

  • The changelog table titled "Re-measured — every number in the docs" was not re-measured. It said the test count went 293 → 316; the real figure was 317, and the same file says 317 correctly in two other places. That table sat directly under the sentence "Every number here was produced by running the command next to it against this build; nothing was carried forward." Corrected, and no release of this package makes that blanket claim again — each number now names its own command.
  • docs/LIMITS.md §9 documented a hole that 1.2.1 had closed. It said pending_approvals was a shallow copy you could write a held call through, with a reproduction log attached. It is a deepcopy, and the changelog entry taking credit for making it one is 150 lines above. The section also opened with "Every item below was reproduced against 1.2.1 before it was written down" — which that one item disproves. Both are gone. A limitations document that invents a limitation costs the reader exactly what a README that invents a feature costs them.
  • The shipped adversarial report called our own corpus "your corpus", 11 times. corpus/reviewer/ was written by us, to break our own detector, and ships in the box; the benchmark labelled every number from it as a measurement of the reader's traffic — the same defect 1.1.2 fixed on the fixture side of the same report. Fixed in bench/run_bench.py; both reports regenerated.
  • 1.2.1's memory cap was a control bypass. MAX_TRACKED_TASKS evicted the oldest task, and a task's entry is its spent budget — so rotating task_id 4,096 times reset a victim task's exhausted 8-step budget to zero. Its docstring named guardstack check as the compensating control; no such warning existed. It refuses instead of evicting now.
  • 1.2.1's loop window could not see a cycle longer than the window. Period 8 and up ran 60/60 unstopped — 1.2.1's own docstring records that figure, and a reconstruction of its algorithm reproduces it; that code is not in this tree to re-run. On 1.2.2, measured directly: period 8 stops at call 17, period 9 at 19, period 20 at 41. It is a per-task per-signature counter now.
  • A second self-contradiction in the same release notes. 1.2.1 said the two dead modules were "not fixed in 1.2.1, because the fix is a code change" in one bullet and "_world.py and _clock.py are gone" in another. They are gone. Corrected in both files.
  • Two live defects were recorded as NOT fixed — and then both were fixed. A non-string resource argument (scope=['orders']) raised TypeError out of check_tool() instead of denying, and the loop counter was not bounded by the step ceiling despite its docstring saying it was (50,000 entries measured on one exhausted task). Both were written up as open here and in four other files while the code was frozen; both were fixed before 1.2.2 shipped — the wrong-typed argument is denied by the scope gate now, and MAX_LOOP_SIGNATURES = 4096 caps the counter. Two tests pin them (tests/test_stack.py). The "not fixed" wording outlived the fixes by one documentation pass, which is precisely the defect this release is largest about, pointed the other way. docs/LIMITS.md §9 keeps the whole sequence: found → recorded open → fixed, with the re-run output for each.

Defects found in our own documentation, in v1.2.1

Found by reviewing 1.2.0 the same way:

  • The QUICKSTART denied 100% of your traffic. §2 mapped payments.refund: order_id, §3 granted scope="orders", and scope containment is path-based — A-1001 is never inside orders. That is the same symptom listed under "the demo world is configuration" above, put back by our own getting-started guide. Fixed, and the corrected wiring was run end to end before it was written down.
  • docs/LIMITS.md credited a mechanism that does not exist. It said both zero-width evasions were blocked "because normalisation strips the invisible characters". There are three of them, two are blocked for unrelated reasons, and there is no normalisation anywhere in the package. An error in our favour, in the document whose job is to find errors against us.
  • The README quoted four lines the example never printed, including a fabricated approval-token value — with a test that claimed to prevent exactly that and checked three substrings. Both fixed; the test now walks every line.
  • amount_cents and order_id are hardcoded argument names and appeared in no document. If your tool calls the order id something else, the pattern check you configured is skipped in silence.
  • Two modules shipped and were imported by nothing_clock.py and _world.py, the second being the demo order dictionary this file takes credit for removing two sections above. Recorded in docs/LIMITS.md §10. Correction (1.2.2): this bullet said they were "not fixed in 1.2.1, because the fix is a code change". They were deleted in 1.2.1 — the changelog entry for it is in that release and guardstack/layers/ contains neither file. The second self-contradiction inside the 1.2.1 notes, corrected here.

Defects found while assembling it

These were live in the free repos and are fixed here. Reported honestly because you would find them on day one anyway:

  • resource() returned None for any tool outside the filesystem/db demo. Every scope check on your own tools then compared None to a scope string and denied with the reason "None is not inside orders". GuardStack refuses with an actionable message instead, and Config.resource_args fixes it.
  • The two injection detectors disagree. llm-guardrails scores prompts; agent-approval-gate scores tool arguments. Attack phrases the argument detector catches passed the prompt detector cleanly. v1.1.0 closes the common cases in the default detector at no false-positive cost; what remains of the seam is one narrow class, measured and documented in docs/TUNING.md.
  • needs_approval crashed on a partially-specified policy (KeyError: refund_stale_after_days). Config now emits the complete schema.

Found by assembling it, in v1.1.0

Two of these were only reachable because the layers sit in one place and get measured together. Neither would surface from wiring the five repos yourself.

  • The use-vs-mention discount could never fire on "summarize". The stem was written summariz\b, and "summarize" continues with a word character, so the boundary never matched. Every "summarize this attack text" request silently lost its discount.
  • Fixing that let an indirect injection through. An instruction hidden in a pasted document was discounted by the user's own "Summarize it." — two authors, one string. Signals that fire on embedded content are now exempt.
  • audit_path was documented and never implemented. The config promised a file and the log lived only in memory. It writes now, and guardstack verify checks the chain with an exit code.
  • The shipped guardstack.yaml failed its own preflight. It declared send_email and cancel_order side-effecting and mapped neither, so anyone who copied it had every call to both refused. Found by guardstack check, on the day guardstack check was written.

All four are fixed and pinned by tests.

MIT licence applies to this bundle too. Resell it if you want.