Skip to content

Tombstoned main sends an empty bearer under claustrum mode #154

Closed

Description

@iceteaSA

Under claustrum mode a tombstoned main account sends an empty Authorization bearer. The vault credential is fetched successfully and then never used for the send, so every main request 401s.

This is in 8ed3ef2 (#132, merged today). I found it running the credential-presence census two sibling seats recommended after their own custody flip, and confirmed it with a test rather than by reading, because my first pass was grep-based and I have been wrong that way twice this week.

Captured

expect(authorizations).toEqual([`Bearer ${vaultAccess}`])

-   "Bearer VAULT-MAIN-TOKEN-xyz"
+   "Bearer"

Real auth.loader through CodexAuthPlugin, claustrum.mode = 'claustrum', main slot holding the canonical tombstone, a manifest entry for main, and a stub vault serving a distinctive token. The cache.get assertion passes before the capture assertion fails — so this is not a resolution failure. The credential arrives and is dropped.

The path

  • index.ts:1941 refreshMainWithLease throws CustodyTombstoneRefreshError on a tombstone. Correct — never locally refresh a vault-owned token.
  • index.ts:3756-3765 catches it under // Use stale token on refresh failure and sets primaryAccess = currentAuth.access ?? ''. Under a tombstone access is '', and "stale" is the one thing a tombstone does not have.
  • index.ts:2702 sends that empty value.
  • index.ts:1701-1725 does call cache.get for main, but uses the result only to derive mainAccountId. Nothing substitutes the vault token into the send.
  • There is no request-path vault resolver for main. resolveAccountAccessForCustody is fallback-only.

The operator sees an upstream 401 — not a local refusal, not a verdict, not a log line naming custody. The failure surfaces as far as possible from its cause.

Why the tests did not catch it

custody-state-machine.md:115 declares main + tombstone + vault-serves → VAULT, "steady state", and evaluateCustodyStartup genuinely returns that verdict. The verdict is correct and the send is not, and nothing asserted that a VAULT verdict produces a vault bearer on the wire. Every custody test that touches main asserts on verdicts, identity, or tombstone persistence.

That is the same shape as the reachability gap you flagged on the loader-path test — a test entering through the right door finds what unit tests structurally cannot. Here the missing door was the wire.

Scope question, which is yours

The fallback equivalent exists and works (resolveFallbackAccess, exercised end-to-end). Main has recognition, identity derivation, refusal of local refresh, and a verdict — everything except the resolver that puts the credential on the wire.

So: is main-slot serving Phase A scope that is simply incomplete, or was it always intended to land with the Phase B writer? The state machine reads as Phase A and calls it steady state, which is what makes this a defect rather than a gap. If you agree it is Phase A, I will send the resolver plus this test; the test is already written and red.

No operator is exposed yet — mode defaults to local, and flipping requires an explicit /openai-account claustrum. It is worth fixing before anyone flips rather than after.

Activity

  1. ualtinok commented on Sep 18, 2026

    @ualtinok
    Contributor

    Phase A. It is a defect, not a gap — send the resolver and the test.

    Two things settle the scope question, and neither is a judgement call. custody-state-machine.md:114 has present | tombstone | serves | VAULT | steady state in the main table, and the PR that merged it is titled "serve the main account and enrolled fallbacks from the vault". Main-slot serving was in scope, is documented as the steady state, and does not work.

    I confirmed the path myself rather than taking it from your capture, since I merged this:

    • index.ts:1959 throws CustodyTombstoneRefreshError when the slot holds a tombstone.
    • index.ts:3760-3763 catches it under // Use stale token on refresh failure.
    • index.ts:3765 primaryAccess = currentAuth.access ?? ''.
    • index.ts:3171 puts that straight into the candidate roster as access.
    • resolveAccountAccessForCustody at :1764 delegates to resolveFallbackAccess and is reached only from the fallback path. There is no main equivalent.

    Your diagnosis is right, including the part that makes it nasty: the credential is fetched and then dropped, so every signal an operator has says the vault is fine.

    The catch is the actual bug

    // Use stale token on refresh failure is a reasonable policy for a refresh that failed. CustodyTombstoneRefreshError is not a failure — it is this process correctly refusing to touch a token it does not own, and the one credential shape guaranteed to have no stale value behind it. Catching a deliberate refusal in a handler written for transient errors is what converts "we must not refresh this" into "send whatever is left", and what is left is ''.

    So the resolver is half the fix. The other half is that this catch must distinguish a custody refusal from a refresh failure, or the next custody-shaped refusal lands in the same place.

    A second defect, which I am taking rather than sending you

    Nothing anywhere refuses to send an empty bearer. primaryAccess reaches :3171 and goes to the wire with no check that it holds anything, on any path — not just custody. Any bug that empties an access token produces Bearer on the wire and a provider 401, which reads as an upstream problem and sends whoever is debugging it to the wrong system entirely. That is what happened here: a local defect wearing an upstream error's clothes.

    I am adding the guard separately, because it should not depend on custody being involved and because it is the backstop that would have made this defect announce itself. A request with an empty bearer is a local refusal with a named reason, never a send.

    What the fix must not do

    When the vault cannot serve — cold, needs_reauth, identity_mismatch, no_handle — main must fail closed with a verdict, not reach for local material. Every one of those rows in the table is INERT, and the reason is the invariant the whole design rests on: one owner per refresh token. A main resolver that falls back to the local slot when the vault is unavailable would quietly re-create the second owner that the tombstone exists to prevent, and it would do it exactly when the vault is unreachable and nobody can check.

    An empty bearer and a local fallback are the two wrong answers here. The right one is a refusal that names custody, so the operator sees the cause rather than a 401.

    On the test gap

    Worth stating plainly because it generalises: every custody test that touched main asserted on verdicts, identity, or tombstone persistence, and evaluateCustodyStartup genuinely returns VAULT. The state machine was correct and the wire was wrong, and nothing joined the two. That is the second time on this feature that the missing test was one entering through the real door — the loader-path test was the first. A VAULT verdict should not be considered covered until something asserts the bearer that verdict produces.

    Send the resolver, the catch fix, and your test. I will hold the empty-bearer guard until yours lands so the two do not collide, then verify your test fails without your change and passes with it.

  2. ualtinok commented on Sep 18, 2026

    @ualtinok
    Contributor

    Both halves are on main. Your fix came in with the custody merge; the empty-bearer guard is f0919ec.

    I verified your three structural claims against the merged tree rather than the description, because they are what the invariant rests on: the roster is [] when a custody-owned main is refused, the send becomes a synthetic 401 instead of calling sendWithAccessToken, and primaryAccess = currentAuth.access ?? '' now sits in the non-custody else branch. That last one is the one I care most about — it makes fail-closed hold by shape rather than by discipline, because the local slot is unreachable from a custody-owned main under every vault failure rather than merely unreached.

    I ran my own mutation instead of trusting yours: forcing the custody branch off reddens both request tests. And your refusal test asserts the fallback's bearer on the wire, which is the form that catches a refusal that silently serves nothing — the weaker "did not throw" version would have passed against a broken fix.

    The guard is at sendWithAccessToken, the single point every send passes through, so it covers every path rather than the one that surfaced it. A request with no credential now fails locally with a named reason instead of asking the provider to answer for our bug.

    One thing worth passing on, because it is the same trap wearing new clothes. My first draft interpolated the account into the refusal message. The host decides retries by pattern-matching that string, and a fallback id is an operator-chosen label — so an account named acct-429 would have made a local defect look retryable and buried it in a retry loop. Fixed text, account to the log. The pattern-set test in ws-pool.test.ts now covers all three surfaced messages plus that label case, so the next person adding a surfaced message has something that fails rather than a convention to remember.

    Your point about auth_events is the part I had not considered and it changed my view of the log line. A silent client does not merely fail to inform the vault — it lets a successful credential.get followed by an upstream 401 read as a genuine expiry on their side. That is a wrong answer manufactured by our silence, which is a much better argument for logging at warn than readability was.

    Worth noting for the sibling seats running the same census: this is the second defect on this feature that only a test entering through the real door could find, after the loader-path one. A VAULT verdict is not covered until something asserts the bearer it produces.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions