How to install, provision, operate, back up and upgrade the claustrum credential vault. The parts and the rules behind each step are in the architecture; the consumer-facing wire surface is in the wire contract.
There are two programs:
ck-claustrum, the daemon. The CortexKit supervisor (subc) runs it; it serves the read surface (credential.get,get_many,status,report_auth_failureand the scoped operations) over the route channel, and the authenticated admin surface. It is built from thecredentials-modulecrate, and its module id isclaustrum.ck-auth, the operator CLI, invoked asck auth <verb>. It is the only write surface an operator uses. Most verbs commit through the running daemon with no downtime; two are offline-only (see the single-writer rule).ck auth help <verb>prints each verb's flags and semantics.
On a standard install admin commands need no flags: the vault data directory and the running daemon's connection file are both auto-discovered. The flags matter only for a non-default location (a test vault, a second vault on one machine, or a key held outside the OS store).
All admin commands accept --data-dir <dir> (the directory holding store.db),
--subc <file> (the supervisor connection file) and --key-path <file> (an operator key
file instead of the OS store). An explicit --data-dir targets that vault and stays
offline unless a connection file is also given.
--data-dir must be <data_home>/cortexkit/<module_id>, where <module_id> is the
subc.jsonc module key, claustrum. The supervised daemon derives its store path from the
module id verbatim, so a CLI pointed elsewhere opens a different, empty vault under a
different key scope. On a default desktop:
DATA_DIR=~/.local/share/cortexkit/claustrumThe daemon has no --key-path flag; it reads CK_MASTER_KEY_PATH from its
environment, set in subc.jsonc. The two must agree. In practice the recorded
key-source (see section 2) makes overrides unnecessary.
The supervisor supplies the other environment the vault is sensitive to:
XDG_DATA_HOME and XDG_RUNTIME_DIR (which move the data directory and the connection
file) and SUBC_MODULE_ID / SUBC_LAUNCH_NONCE (identity, echoed at registration).
Moving the data directory does not start a fresh vault: the daemon never bootstraps,
so a relocated vault finds no key for its new scope and refuses to serve rather than
coming up empty.
CK_GOOGLE_OAUTH_CLIENT_ID / CK_GOOGLE_OAUTH_CLIENT_SECRET and the CK_ANTIGRAVITY_*
pair override the embedded public OAuth clients. A wrong value fails loudly at the
provider, so no procedure below uses them.
There is exactly one writer at a time, always. What changes is who it is.
When the daemon is running it holds the vault's single-writer lease, so the CLI cannot open the store directly. Instead the CLI sends the operation to the daemon over the route plane, and the daemon performs the write, serialized against any in-flight token refresh. This needs no downtime: re-logging in a provider while agents are reading credentials is safe.
Each operation is authenticated by a challenge-response MAC over the exact operation bytes, keyed from the master key, with a single-use nonce. A caller without the master key cannot mutate anything, and a compromised daemon cannot authorize a mutation it was not given.
When the daemon is not running, the CLI takes the lease itself and writes directly. Same operations, same audit chain, same master-key requirement.
Offline-only verbs. Two commands always require the daemon stopped, because they operate on the store as a whole:
| verb | why it is offline-only |
|---|---|
bootstrap |
creates the vault; there is no daemon yet |
rotate-master-key |
re-wraps every record and the sealed audit key in one transaction |
Every other write verb commits through the running daemon, and falls back to the offline lease path automatically when no daemon is reachable.
Exit codes:
| code | meaning | what to do |
|---|---|---|
| 0 | success | — |
| 3 | the daemon holds the lease and this verb is offline-only | stop the daemon, retry |
| 4 | master key could not be resolved (locked keychain, absent, or wrong) | unlock the keychain, or check --key-path |
| 5 | indeterminate: the op reached the daemon but the reply was lost | see below |
| 1 | usage, IO or other error | read the message |
Exit code 5 needs care. The operation was sent to the running daemon and the
connection dropped before the outcome came back, so it may or may not have committed. Do
not blindly retry: check with ck auth list (did the credential's version change?)
or ck auth audit (is there an entry for it?), then act on what you find.
The other outcomes are unambiguous. A refusal from a live daemon is terminal and safe: nothing was written and the CLI never falls back. No reachable daemon means nothing was dispatched, which is why falling back to the offline path cannot double-execute.
ck setup claustrum installs the vault through the CortexKit dispatcher, installing the
supervisor core first by construction. The binaries live in
~/.local/share/cortexkit/bin/, with ck-auth also symlinked into ~/.local/bin/ for the
ck dispatcher.
Releases from v0.1.3 need subc 0.17.20 or newer. An older supervisor refuses the
module's HELLO with malformed HELLO body: missing field "consumes" and parks it after
three restarts. That message names a protocol field rather than a version mismatch, so it
reads as a broken release when the supervisor predates the field. Upgrade the supervisor
first. This matters only when installing from the release page by hand.
The supervisor runs the daemon from its subc.jsonc, with the vault module marked
reserved: true and a sqlite storage section. reserved: true means only the process
the supervisor spawned, echoing its launch nonce, can register as claustrum.
Verify the install:
ck-auth --version # prints the build revision
ck auth status # after bootstrap: health ladder + inventoryThe master key encrypts every credential at rest. It is created once and never regenerated; a second bootstrap is refused rather than clobbering the existing key.
Native OS store (default for a new vault):
| OS | Master-key storage | Requirements |
|---|---|---|
| macOS | Keychain generic passwords | An accessible login keychain |
| Windows | Credential Manager per-user generic credentials, protected by DPAPI | Run as the vault owner with access to their credential store. The item uses enterprise persistence, so on a domain account with a roaming profile it roams with that profile. Use a key file on such machines if the key must stay on one host |
| Linux | Secret Service default collection (GNOME Keyring or KWallet) over the D-Bus session bus | A reachable service and an already-unlocked collection |
ck auth bootstrap --data-dir "$DATA_DIR"All native stores keep independent current and next slots for crash-safe rotation, scoped by the canonical vault directory. They never request an interactive unlock. If the Linux keyring is locked, unlock it in your desktop keyring manager before starting or retrying the daemon. For an unattended session, arrange an unlocked keyring and its D-Bus session bus before launching the daemon. A missing bus, absent default collection, or refused Credential Manager access is a refusal, not permission to choose another backend.
Operator key file (headless or server): the key file must live outside the data
directory (co-locating the key with the ciphertext defeats at-rest encryption); the CLI
refuses a key path inside --data-dir.
ck auth bootstrap --data-dir "$DATA_DIR" --key-path /etc/cortexkit/master.keyOn headless Linux without an unlocked Secret Service, explicitly choose --key-path <file>
or CK_MASTER_KEY_PATH=<file> when bootstrapping. Use a directory outside the vault tree
that the daemon user owns (for example ~/.config/cortexkit/secrets/master.key), with
the parent directory at mode 0700, created beforehand; bootstrap creates the key file.
There is no automatic fallback from a missing or locked native store.
On Unix every load checks that the key file is a regular file, not a symlink, owned by
the current effective user, with no group or other permission bits. Use chmod 600 <file>
and, if needed, chown <uid> <file>. On Windows every load verifies the opened handle's
owner and DACL: only the current user, SYSTEM and Administrators may own it or hold
grants; null DACLs, unknown grant types and grants to other users or groups are refused.
Bootstrap creates the key with a protected current-user-only DACL from first visibility,
and rotation writes a similarly protected temporary file and renames it. Provision the
key's parent directory with an owner-only DACL too.
A loosened key file is refused as unsafe operator key file: <path> failed owner-only Windows ACL verification: ...; restore a regular key file owned by the current user, SYSTEM or Administrators, remove inherited and explicit grants to all other users/groups, and retry (do not regenerate the key). Use the file's Security / Advanced settings to
disable inheritance and remove broad grants, including Everyone or Users, and preserve the
existing key bytes. Claustrum does not repair an untrusted key file while reading it.
The vault data directory is created or tightened to a protected owner-only DACL on Windows; foreign owners and reparse points are refused. See Windows file custody for every file the vault writes.
Bootstrap also writes $DATA_DIR/key-source, a non-secret, versioned custody record. Its
two lines are version:1 and one of keychain, credential-manager, secret-service,
or file:<absolute path>. Keep it with the vault metadata, not with the secret key. Both
the daemon and ck auth read this choice, so later commands need no key-path override.
Unknown versions or backend names are refused, not guessed.
Both programs honour --key-path and CK_MASTER_KEY_PATH. An override must agree with the
recorded choice; a different file, or a file override for a native-store vault, is refused
with both choices named. If both override names are set they must name the same file.
Remove conflicting overrides rather than editing the record to redirect a live vault.
A vault with store.db but no key-source is migrated on its first successful key
resolution: an explicit file override records that path, otherwise the existing keychain
source is recorded. A failed load or fingerprint mismatch writes nothing, and no key
material is moved.
A credential id is <method>:<provider>[:<label>] (for example oauth:anthropic,
apikey:deepseek, antigravity:google, oauth:anthropic:work). The <method> selects the
credential kind and the refresh adapter stored with the record: oauth → the
provider-named adapter, antigravity → the antigravity adapter, apikey → a static key
with no adapter. --adapter <name> overrides the method default. A legacy
<provider>[:<label>] id (first segment not a known method) defaults to the provider's
OAuth adapter. Ids beginning with category: or containing | are refused.
login mints a NEW, independent credential that the vault solely custodies, so there is
no rotation race with another tool holding the same provider login. Run it with no flags
for an interactive picker over every provider, showing which already have a credential:
ck auth loginThe picker covers OAuth (anthropic, openai, xai, google, antigravity),
device-flow (github-copilot, kimi, meta, and --device for openai and xai), custom
browser flows (cursor, devin, snowflake, digitalocean), and API-key providers
(zai, openrouter, kiro, deepseek, groq, …), which are validated against the
provider before being stored. A few API-key providers have no probe that can tell a bad
key from a good one (for example opencode-go, vercel-ai-gateway, cline-pass); for
those, login prints a NOTE: line saying the key was stored without being checked.
OAuth logins open a browser and complete automatically: a one-shot listener on the
loopback redirect captures the code, so when the browser shows a paste code, ignore it;
the CLI has already finished. If the port is busy, the listen fails, or you pass
--no-listener, the flow falls back to pasting the address-bar URL. Pasted values are read
from stdin only, never argv, and never logged.
ck auth login --provider xai --replace
ck auth login --provider anthropic --id oauth:anthropic:work # a second, labeled accountDefault ids include oauth:anthropic, chatgpt:openai, oauth:xai and oauth:meta. A
bare --provider openai means the ChatGPT subscription login, not apikey:openai. Each
labeled account has its own refresh chain and its own handles. --replace swaps the
credential on an existing id and keeps its handles; it is the usual repair for a
needs_reauth credential. Without it, login is create-only. A native login records a
login audit entry, distinct from import.
Kiro. Create a long-lived ksk_ key at https://app.kiro.dev under Account → Settings →
API Keys, then run ck auth login --provider kiro to store it as apikey:kiro (or a
labeled account). Login checks the key against Amazon Q's model catalog using bearer auth
and TokenType: API_KEY; a successful response verifies it, the 403 auth refusal rejects
it, and other statuses or network errors produce a warning. No profile ARN is stored or
served.
Meta Muse subscription. Run ck auth login --provider meta (or add
--id oauth:meta:work). The device flow prints a verification URL and code, then the
vault mints an inference key. Only that key is served; the Meta identity token and any
optional OAuth refresh token stay sealed in the vault. No live exchange has confirmed a
Meta refresh-token grant, so the vault does not use one. Login prints only whether a
refresh token and expiry fields were present, never their values. Key expiry uses a
response expiry, then the JWT exp, and only otherwise an assumed 24-hour lifetime.
The HTTP fixtures for this adapter are reconstructed from client source rather than
captured from Meta. A mint 401/403 requires re-login. An inactive or payment-required
subscription also latches needs_reauth, but its subscription_required event means
repair the subscription first; re-login alone cannot fix payment. A 429 or server
outage is transient.
Gmail send. Gmail uses your own Google Cloud OAuth client, not the public Gemini or
Antigravity clients. Create an OAuth client of type Desktop app in your Google Cloud
project and enable the Gmail API. Configure the consent screen for exactly these
scopes: https://www.googleapis.com/auth/gmail.send, openid and email, and publish the
app In production. Per
Google's OAuth documentation,
an external-user app in Testing gets refresh tokens that expire after 7 days unless it
requests only basic identity scopes, and gmail.send is not exempt. Keep the client secret
in a 0600 file, never a command argument:
chmod 600 /secure/path/gmail-client-secret
ck auth login --provider gmail --client-id YOUR_DESKTOP_CLIENT_ID --client-secret-file /secure/path/gmail-client-secret
rm /secure/path/gmail-client-secret
ck auth grant --principal reserved:<module> --selector-kind exact --selector oauth:gmail --operation readThe file holds only the secret value (not the downloaded client JSON); trailing whitespace
is stripped. The refresh token and client secret are sealed together. The flow requests
offline access and consent, listens on http://127.0.0.1:8086/oauth2callback, and supports
--no-browser / --no-listener with the full-URL paste fallback. Login refuses a missing
refresh token or a missing userinfo email. Use --id oauth:gmail:<label> for a labeled
account and grant that exact id. A Gmail record carries only the gmail-native
category and no serves vendors: never grant a model category to a mailbox consumer or add
llm-provider to a Gmail record. Reads return the short-lived access token only; the
refresh token and client secret never leave the vault.
import bootstraps from another tool's files. Run it with no flags to pick from detected
accounts, or name the source:
ck auth import --source opencode --provider anthropic --id oauth:anthropic --json /path/to/auth.json--source is one of opencode | pi | gemini-cli | antigravity. The provider's token URL and
client id come from the refresh adapter, not the file.
- API keys: an
apikey:<provider>id imports a{ "type": "api", "key": "..." }entry as a static key;--provider <key>selects the entry from theauth.jsonmap. - OAuth (
auth.json):--provider <key>selects one provider's{ refresh, access, expires }entry. Without it,--jsonmust point at a single provider's object. - Google must be imported from
gemini-cliorantigravity, not opencode. A Google refresh token only refreshes against the OAuth client that minted it.--source gemini-clireads~/.gemini/oauth_creds.json.--source antigravityreads~/.config/opencode/antigravity-accounts.json(--providerselects an account by email or index, default the active one) and is the source for anantigravity:googleid. An opencode-minted Google token cannot be refreshed by either and fails closed toneeds_reauth. - Claustrum tombstones are not importable. After
migrate-opencode, a tombstoned entry is an ownership marker. Re-runck auth migrate-opencodeto converge, or useck auth migrate-opencode --restore <provider>to return the credential to OpenCode (see the OpenCode plugin). --replaceoverwrites an existing id unconditionally (re-seal at version+1, reset toactive) and keeps its handles. Without it,importis create-only.--account-id <id>attaches non-secret account metadata;--emailand--org-namerequire it;--clear-identityis exclusive with all three. A token-onlyimport --replacekeeps the existing identity, which assumes the replacement token belongs to the same account. After a re-login into a different account, runck auth set-identity <id> --account-id <id>.
ck auth set-identity <credential-id> --account-id <id> [--email <email>] [--org-name <name>]
(or --clear) relabels a credential without replacing its token: it re-seals unchanged
material, keeps the lifecycle state, and bumps record_version.
For an API key, DSN, opaque secret or session cookie. Use --payload-file <path> so the
secret never appears in the process list or shell history; trailing whitespace is stripped
(cookie bytes are kept exactly):
ck auth put --id apikey:openai --payload-file ~/.config/openai.key --kind api_keyput is create-only. To rotate a static key in place, pass --replace (unconditional,
keeps handles) or --expected-hash <hex> (compare-and-set). A cookie:<domain>:<label> id
always creates a session-cookie record. A github_app deposit requires --client-id, the
App JWT issuer.
For machine account authentication from the operator's own terminal, generate a dedicated Ed25519 key inside the vault and opt it into the admin signing path:
ck auth mint-signing-key --id signing:account-machine:1 --operator-sign
ck auth sign --id signing:account-machine:1 --payload-file challenge.bin
# Or pipe exact challenge bytes to: ck auth sign --id signing:account-machine:1 --payload-file -The private key never leaves the vault in plaintext. --operator-sign seals the
permission with the key and implies --audit-each-signature. Every operator
signature commits a chained sign audit row with the payload SHA-256 before
signing; an audit failure returns no signature. Only the hash is audited, not the
challenge bytes. The actor is route-admin online and offline-cli offline.
sign uses the same operator master-key authentication as other ck auth admin
commands, or the offline single-writer lease when the daemon is stopped. It needs
no capability handle or enrollment token. Success prints one JSON line with
key_id, public_key_hex and signature_hex; the signature is 128 lowercase hex
characters. Refusal exits 1 and prints no signature. The 64 KiB decoded payload cap and
exact-byte signing apply to both files and stdin.
Keys minted without the flag cannot use this path. Root keys
signing:cortexkit-root:1 and signing:gh-manifest-root:1 must never carry
operator_sign. Use a separate machine key, not a root key. Replacement is a
fresh key and requires --operator-sign again. Older binaries can read the sealed
record but drop operator permission if they rewrite it. Consumer bus signing,
grants and capability handles otherwise retain their existing behavior.
Whether a second holder of the same login is a hazard depends on the provider, not on the arrangement. Read the adapter's refresh-token handling:
ROTATES -> one custodian only; a second holder is a live revocation hazard
OMITS -> a second holder is redundancy, not a race
Anthropic rotates and revokes the whole token family when a rotated-away refresh token is
presented, so keep one custodian. Google's installed-app OAuth omits the refresh token from
the response and the old one stays valid (antigravity.rs keeps the stored token when the
response omits one). xAI rotates but does not revoke the predecessor.
Anthropic subscription logins are also revoked by the provider roughly a month after login,
independent of how often they refresh. The vault discovers this as invalid_grant, latches
needs_reauth, and the repair is ck auth login --provider anthropic --id <id> --replace.
ck auth reactivate does not fix it: that clears a wrong verdict, and this one is right.
A category is a bare name matching ^[a-z][a-z0-9-]{1,31}$. Grants can select on a
category, so a category assignment changes what a grant reaches; assigning one requires the
master key.
The catalog seeds five categories: llm-provider, data-warehouse, cloud-infrastructure,
web-search and browser-session. The set is not closed: an operator can assign any valid
name. The seeds are what a new credential is stamped with at creation:
- API-key entries are
llm-provider, except the search APIs Tavily, Kagi, Exa and Parallel, which areweb-searchand neverllm-provider, so no LLM consumer's grant reaches a search key. - Login entries are
llm-provider, except Snowflake (data-warehouse), DigitalOcean (cloud-infrastructure), Devin (uncategorized) and Gmail (gmail-nativeonly). - A credential that speaks a provider's OAuth protocol is also stamped
<adapter>-native. - Every new
cookie:record is stamped onlybrowser-session.
The assignments are literal per-entry catalog fields in crates/credentials-core/src/catalog.rs;
no fallback derives them from the id. ModelVendor (serves) is a separate, advisory
vocabulary that never authorizes.
ck auth categories # every category, its count, and the grants naming it
ck auth set-category <id> --add llm-provider # or --set <csv> / --remove <csv>
ck auth reclassify --from-registry # refill EMPTY sets from the catalog
ck auth reclassify --from-registry --force # also replace differing non-empty setsreclassify --force refuses if it would remove a category an active grant selects on. A
category set an operator deliberately cleared is refilled by a later reclassify, so keep it
empty by not rerunning reclassify, or clear it again afterward. A transition writes one
set_category audit row; a no-op is silent.
Provider ids are operator-written catalog mappings, not authorization categories. Neither provider ids nor plan tiers are inferred from the credential's name:
ck auth set-providers --id oauth:anthropic --set anthropic claude-code
ck auth set-plan --id oauth:anthropic --tier max_20x
ck auth set-plan --id oauth:anthropic --clear
ck auth listset-plan needs exactly one of --tier or --clear. A tier is a lowercase ASCII letter
followed by lowercase letters, digits or underscores, 1–32 characters. The pricing service
owns the vocabulary; current tiers include Anthropic pro, max_5x, max_20x and OpenAI
plus, pro_100, pro_200, pro_500. --tier always wins over automatic detection;
--clear removes only the override and reveals the detected tier. An absent tier means
unknown, not a default plan. Replacements into a different account and set-identity
clear both the old override and the old detection.
Automatic detection reads ChatGPT's access-token JWT at login and every refresh, and makes one bearer profile request for Claude at login and after each successful refresh; unknown plan strings give no tier rather than a guess. The exact mapping, timing and staleness bounds are in the wire contract. Overrides do not update automatically: reassert or clear one after an out-of-band plan change.
Table layout: when any row has a tier, ck auth list and the inventory in status add a
sixth header word, PLAN_TIER, after PROVIDERS, and every row has a sixth field (the tier
or -). Detected tiers carry a * suffix, e.g. max_20x*, explained in a footnote. An
optional by=<principal> follows as a trailing annotation with no header word. With no
tiers, the five-word STATE VER CREDENTIAL CATEGORIES PROVIDERS header is unchanged. Parsers
must accept both layouts; no field value contains whitespace.
There are three ways a consumer reaches a credential. Pick by what the consumer is.
| consumer | mechanism | operator command |
|---|---|---|
anything that can keep a 0600 file |
capability handle | ck auth mint-handle |
| a module the supervisor spawns | grant to reserved:<module> |
ck auth grant |
| a host-launched process with no supervisor identity | enrollment, then grant to enrolled:<name> |
ck auth enroll approve, ck auth grant |
ck auth mint-handle --id oauth:anthropicThe command prints the raw handle (ckh_...) to stdout exactly once; only its hash is
stored. Write it into the consumer's config as a 0600 file. Mint a separate handle per
consumer: handles are the revocation unit, so cutting off one consumer is one
revoke-handle and nobody else notices. Handles survive login --replace.
ck auth revoke-handle --handle <ckh_...> # or --hash <64 hex> when the raw value is gone
ck auth revoke-all-handles --id <id> # the record stays refreshableTo verify, the consumer's next credential.get succeeds; after a revoke it answers
not_found (indistinguishable from an unknown handle, by design).
ck auth grant --principal reserved:<module> --selector-kind exact --selector <credential-id> --operation read
ck auth grant --principal enrolled:<name> --selector-kind category --selector <category> --operation read
ck auth grants # the complete authority set, read-only
ck auth revoke-grant --principal reserved:<module> --selector-kind exact --selector <credential-id> --operation readexact matches one credential id byte for byte; category matches every credential carrying
that category, and moves as categories are assigned. Each operation is a separate authority
and none implies another:
| operation | authorizes |
|---|---|
read |
credential.get_scoped (the token itself), and scoped status, public_key and report_auth_failure |
sign |
credential.sign with a signing key |
open |
credential.open with a KEM key |
list |
the row in credential.list_scoped, with its identity (account_id, email, org_name) and refresh_adapter, and nothing else |
deposit |
credential.deposit_cookie for cookie: records only; requires a reserved principal and the browser-session category; grants no read access |
list is the account roster without the tokens: grant it to a module that chooses between
accounts but must never fetch one. A list-only principal is refused by every other scoped
operation with the same not_found as a principal with no grant.
grants rows contain the principal kind and id, selector kind, selector, operation and
creation time, sorted by principal, selector, then operation. An empty table prints
no grants. Revocation is exact over all four fields: revoking list leaves a read grant
on the same selector in place.
To verify a grant took effect, the module's next scoped call succeeds, and ck auth events
shows one scoped_first_use row for the (credential, principal, operation) tuple.
A host-launched consumer (for example an editor plugin) has no supervisor launch nonce. It proposes a name over the read plane and waits; only the master key can admit it.
ck auth enroll list # pending requests and live enrollments
ck auth enroll approve --request-id <id> [--name <name>] # admit a NAME; mints no token
ck auth enroll deny --request-id <id>
ck auth enroll revoke --name <name>
ck auth enroll reissue --name <name> > token-file # prints the new token aloneApproval admits a name and mints nothing: the consumer's own poll mints the token,
authenticated by a request secret only it holds, so neither an operator nor a squatter can
collect a token for a name they do not hold. --name overrides the proposed spelling, which
is a stranger's claim rather than a fact. After approval, grant enrolled:<name> what it
needs. In enroll list, awaiting-poll means admitted but never collected: a stalled
handover, not a live consumer.
revoke stops the token from resolving immediately; a revoked enrollment answers like an
unknown token. It keeps the consumer's grants, which record what reach existed and block
re-enrolling the name, so a later consumer cannot inherit that reach. Revoke the grants
before re-enrolling a name. reissue writes a new token; the previous one stops working the
moment the new one is minted. The consumer-side ceremony is in
the wire contract.
The supervisor runs the daemon. Once it is up, a consumer reads a credential:
catalog.list
→ route.open(ManagementSurface, module_id = "claustrum")
→ credential.get { handle: "ckh_..." } // returns the opaque payload
The daemon resolves the handle, refreshes the token if stale, and returns the payload. An
unknown or revoked handle is a uniform not_found.
On Linux, both ck-claustrum and ck-auth make themselves non-dumpable before processing
arguments or loading credentials, and refuse to start if that fails. This also makes
/proc/<pid>/exe, fd/, environ and maps unreadable to same-user processes, so
ck module status reports the running image as not readable (non-dumpable); that is
expected, not a failed deployment. On Windows both binaries replace their process DACL at
the same point. On macOS the hardened-runtime signature does this job.
A consumer that observes a 401/403 should call credential.report_auth_failure { handle, provider_status, record_version } with the version it was served. If the vault has since
refreshed to a newer version, the report is a silent no-op.
ck auth status # health ladder + inventory
ck auth list # one row per credential: <state> v<version> <credential_id> …
ck auth grants # one row per principal-scoped grant
ck auth categories # categories, counts, and the grants naming them
ck auth usable # which credentials can still serve or refreshNone of these prints a secret. They read the running daemon when one is up and fall back to the store directly when it is not.
status reports the same health the supervisor probes:
| status | meaning | what to do |
|---|---|---|
ok |
store readable, lease held, every credential active or retired | — |
degraded |
serving, but ≥1 credential is needs_reauth or corrupt |
repair the named credential (section 8) |
failing |
the store is unreadable, or this daemon lost write authority to a newer instance, or its background health refresher stalled | check disk and lease; a stalled refresher means restart the module |
A degraded vault still serves every healthy credential. Retired credentials are listed separately and do not degrade health.
The supervisor logs every non-ok probe. Strip the colour escapes before searching that log, or field-name patterns match nothing and return a confident zero:
sed -E 's/\x1b\[[0-9;]*m//g' ~/.local/share/cortexkit/run/subc.log \
| grep 'module reported non-ok health' | grep 'module_id=claustrum'Without the sed, status= never appears as literal bytes (on disk it is
status\e[0m\e[2m=\e[0m). The log is shared and interleaved across every supervised module,
so require both terms on one line.
The health probe and ck auth list read metadata only; they never open an envelope. One tool
answers whether every record can still serve, by decrypting each record in memory:
ck auth usable
ck auth usable --data-dir /srv/vault --key-path /etc/cortexkit/master.keyIt is safe against a running vault: a read-only connection, no lease, nothing written. It
scores stranded: a record holding neither a usable access token nor refresh material, so
no get can recover it without a re-login. It deliberately does not score access-token
expiry, which is the routine state of a healthy credential. It also flags an identity that
renders a value while resolving nothing (an email with no account id). stranded: 0 is the
expected reading.
| situation | command | effect |
|---|---|---|
a credential is needs_reauth and the provider really refused it |
ck auth login --provider <name> [--id <id>] --replace |
new material, same id, handles kept |
| a consumer report latched it but the refresh token may be alive | ck auth reactivate --id <id> |
clears needs_reauth; the secret is untouched |
| stop serving, reversibly | ck auth logout --id <id> |
state retired, every handle revoked, record and audit history kept |
| delete permanently | ck auth remove --id <id> |
the record, its refresh intent and its handles are gone; audited, not undoable |
| take out of service (lower-level) | ck auth invalidate --id <id> |
needs_reauth and every handle revoked |
A retired credential does not degrade health. login --replace or reactivate brings it
back; consumers need freshly minted handles because logout revoked theirs.
A consumer report is a claim about a served token; it says nothing about the refresh material, and the two die independently. The vault also cannot interpret a provider status: GitHub returns 403 both for a missing permission and for a rate limit, and other providers use 403 for an entitlement lapse. So a report can kill a working credential. Check whether the vault's own refreshes were healthy:
sqlite3 "file:$DATA_DIR/store.db?mode=ro" \
"SELECT datetime(ts_ms/1000,'unixepoch','localtime'), op
FROM audit_log WHERE credential_id='<id>' ORDER BY seq DESC LIMIT 8;"A refresh_commit shortly before the report means the refresh token was alive then, and a
re-login is the expensive repair for a verdict that may be wrong:
ck auth reactivate --id <id> # clears needs_reauth, does NOT touch the secretThis is self-correcting: the vault re-verifies on next use, so a credential that really is
dead returns to needs_reauth on its own and the wrong guess costs one failed request. It is
refused for corrupt records, where the vault checked its own bytes. If reactivate is
followed within minutes by another report at the new version, the credential is genuinely
dead and login --replace is the repair.
A 401 report at the current record_version against a refreshable credential does
not latch anything: it marks the token stale and the next get refreshes it. Only the
provider's invalid_grant latches needs_reauth. Static credentials have no recovery path,
so a report latches them immediately.
Every durable mutation is recorded in a tamper-evident, HMAC-keyed audit chain.
ck auth verify-audit # safe while the daemon runs
ck auth audit # safe while the daemon runsverify-audit takes no lease, so it runs against a live vault. Four outcomes, deliberately
distinct; the middle two are configuration problems, not tampering:
| outcome | meaning |
|---|---|
audit chain verified: intact |
every MAC chains to its predecessor |
audit chain BROKEN at seq N |
the chain fails at N; inspect from there |
no master key slot holds the key this vault is sealed under |
wrong key or wrong vault, not tampering |
this vault has no audit key |
nothing to verify against; not an empty chain |
The chain is tamper-evident against edits, reorders and inserts, but not against
TRUNCATION: an attacker with write access can delete a suffix and the prefix still verifies.
Detecting that needs an external anchor. The running daemon publishes the global tip in its
health body as auditSeq and auditTipMac; a witness that records the pair over time
detects truncation. The check is not "the sequence never decreases": a truncation followed by
fresh appends returns the sequence to an old value, and only the MAC at that sequence differs.
The MAC observed at a given sequence must be stable forever.
audit lists entries (seq, op, credential, actor, and any alarm), lease-free. A flagged row
prints its reason in brackets. Every admin write is flagged [admin_write] by design, so
scan for the reason, not the flag: [fetch_rate_anomaly] and [overwrite_without_cas] are
the detection signals. An alarm row is a durable record surfaced on demand, not a live
notification.
The chain records mutations, not reasons. ck auth events answers why:
ck auth events # most recent first, 20 by default
ck auth events --limit 100WHEN CREDENTIAL KIND PRINCIPAL DETAIL VER APPLIED
2026-01-01 07:58:09 apikey:openai consumer_report_latch - 401 v3 applied=yes
2026-01-01 07:57:50 chatgpt:openai consumer_report_stale - 401 v5 applied=no
2026-01-01 06:12:03 oauth:xai refresh_failed - 503 status v2 applied=no
Column widths are measured from the rows, so the header is the only stable way to read a
column. PRINCIPAL is - on consumer reports because the read surface does not record one.
- A consumer report means a consumer spent the token and the provider refused it. The status distinguishes a rejected token (401) from a forbidden request (403). The reporter is whoever saw the refusal, not whoever caused it.
refresh_failedmeans the vault's own exchange failed. On a transient failure the record stays active, so without this row nothing would show the attempt happened.appliedsays whether the event changed the credential.noon a report means the version it named had already been replaced, so the report was correctly ignored; a run of those is a consumer working from stale state.
Unlike audit, these rows are not evidence: they are not tamper-evident, and only the newest
64 events per credential are kept, so a flood against one credential cannot evict another's
history. A credential showing exactly 64 events has had at least that many. For what
authoritatively happened, read the chain. Two empty results mean different things:
no authentication events recorded (nothing has failed) versus no authentication-event table yet (the store predates the migration; the daemon creates it on its next start).
The vault has four separate string vocabularies that are easy to confuse. They live in
different tables and columns. Each list below is checked against the code's enum by the
tests in crates/credentials-core/src/audit.rs (vocabulary_documentation_tests).
Table: audit_log
Column: op (TEXT)
These values come from the closed AuditOp enum and name the mutation or chain event:
| Value | Meaning |
|---|---|
put |
Create a new credential without replacing an existing row. |
import |
Import a credential from an external source format. |
login |
Mint a vault-native first-party OAuth credential. |
overwrite |
Replace a credential under an unconditional or compare-and-set write path. |
set_identity |
Re-seal unchanged credential material with updated non-secret account identity. |
set_plan |
Change an operator tier assertion, or clear it atomically when a replacement changes accounts. |
invalidate |
Mark a credential as needing re-authentication. |
rotate_master_key |
Re-wrap the vault under a new master key. |
refresh_commit |
Commit new tokens from a vault-owned refresh. |
report_auth_failure |
Record a consumer report that changes credential state. |
remove |
Permanently remove a credential row while retaining its audit history. |
reactivate |
Clear needs_reauth without changing the stored secret. |
mint_handle |
Mint a capability handle for a credential. |
revoke_handle |
Revoke one or all capability handles for a credential. |
fetch_anomaly |
Record a read-surface fetch-rate or enumeration anomaly. |
grant_create |
Create a principal-scoped exact or category operation grant. |
grant_revoke |
Revoke a principal-scoped exact or category operation grant. |
enroll.approve |
Approve a pending enrolled-consumer incarnation. |
enroll.deny |
Deny a pending enrollment request. |
enroll.revoke |
Revoke one enrolled-consumer incarnation. |
enroll.rotate |
Rotate a live enrollment token at its current generation. |
enroll.reissue |
Operator-reissue a replacement enrollment token. |
category.migrate |
Record migration 10's audited category backfill. |
approval |
Record an approver's approval of the exact artifact bytes identified by a hash. |
The set_category, set_providers, deposit_cookie_create and deposit_cookie_replace ops
and their target formats are specified in the wire contract.
Table: audit_log
Column: alarm_reason (TEXT), paired with the alarm (INTEGER 0/1) flag
op says what the entry records; an alarm reason says why that entry was flagged. The
alarm column is only the presence flag; the strings below are stored in alarm_reason and
come from the closed AlarmReason enum. Do not combine this list with the op list:
| Value | Meaning |
|---|---|
overwrite_without_cas |
An existing credential was overwritten without a compare-and-set guard. |
fetch_rate_anomaly |
A connection's credential-fetch rate or spread crossed the anomaly threshold. |
admin_write |
An administrative write occurred; admin activity is always flagged. |
reconcile_hash_mismatch |
Startup reconciliation found a stored refresh token hash that disagreed with its dangling intent. |
Table: auth_events
Column: kind (TEXT)
These values come from the AuthEventKind enum. This table is a separate, prunable
diagnostics table, not the tamper-evident audit chain:
| Value | Meaning |
|---|---|
refresh_failed |
A provider refresh attempt failed without producing a committed replacement. |
stale_nonrefreshable_latch |
A stale report on a non-refreshable credential was latched as needs_reauth. |
consumer_report_stale |
A consumer report marked a refreshable credential stale for its next read. |
consumer_report_latch |
A consumer report immediately latched a non-refreshable credential. |
scoped_read_refusal |
A principal-scoped read was refused; detail names the internal refusal reason. |
scoped_first_use |
The first time a principal successfully exercised a grant over a credential. Written once per (principal, credential, operation) and never again, so its absence shows a grant that has never been exercised. |
reconcile_needs_reauth |
Startup reconciliation forced a credential to needs_reauth. |
github_app_permissions_changed |
A successful GitHub App mint observed changed installation permissions. |
stale_refresh_deferred |
A read found a consumer's stale mark but served the current token without refreshing, because that version was written less than REPORT_REFRESH_COOLDOWN_MS (60 s) ago. Written at most once per credential version, so one row stands for every held-back read in that window; record_version names the version that kept serving. The mark stays set and the first read after the window refreshes. |
enrollment |
A bounded enrollment proposal, poll, or expiry diagnostic under a fixed auth.enroll_* subject. |
Retired values you may still meet in the data. The table above lists what the code writes; the table on disk also holds what older binaries wrote, and it is never rewritten:
| Retired value | What it was |
|---|---|
consumer_report |
The single kind that preceded the stale/latch split: a consumer report of an auth failure, before the vault distinguished "mark stale so the next read refreshes" from "latch immediately". Superseded by consumer_report_stale and consumer_report_latch. |
Other unknown kinds may appear (a future retirement, a test-harness fixture). Treat them as diagnostics, never as audit operations or alarm reasons.
Before reading a missing row as evidence, ask whether the path could have completed without writing it.
CAN THE PATH THAT WOULD HAVE WRITTEN THIS ROW COMPLETE WITHOUT WRITING IT?
no -> absence is a STATEMENT
yes -> absence is a SILENCE
auth_events is written unconditionally on a consumer report, so a missing row there is a
statement: the report did not arrive. The attributed audit_log row is written only when the
version gate passes, so its absence is a silence: it is missing for every applied=no
report, which are exactly the reports worth tracing. auth_events.principal_kind is always
NULL on consumer reports. A write-on-success path makes absence meaningful; a
write-on-transition path does not, and no row count tells you which you are looking at.
Table: auth_events
Column: reporter_source (TEXT, nullable)
A consumer-asserted, unverified claim about where the reported failure was observed. It is
separate from detail, which records what the vault observed. The vault accepts only this
closed vocabulary; any other wire string becomes unrecognised, and the original string is
never stored:
| Value | Meaning |
|---|---|
direct |
The consumer saw the provider status on a direct response. |
relay_status_field |
The consumer read a structured status field from a relay error event. |
relay_message_parse |
The consumer recovered the status by parsing relay message text. |
unrecognised |
The consumer supplied a source outside the vault's named vocabulary. |
NULL means the report predates this column or the consumer omitted the optional field; it
is normal, not invalid data, and differs from unrecognised.
To inspect a running vault, or to answer "did that write commit?" from outside, read the store read-only. This takes no lease and cannot disturb the daemon:
DB="$DATA_DIR/store.db"
# Recent events. `actor` distinguishes who caused them: `vault` is the refresh engine,
# `route-admin` an operator through the running daemon, `offline-cli` an operator holding
# the lease directly.
sqlite3 "file:$DB?mode=ro" "SELECT seq, op, credential_id, actor,
datetime(ts_ms/1000,'unixepoch','localtime') FROM audit_log ORDER BY seq DESC LIMIT 20;"
# When each credential last completed a real provider token exchange.
sqlite3 "file:$DB?mode=ro" "SELECT credential_id, MAX(datetime(ts_ms/1000,'unixepoch','localtime'))
FROM audit_log WHERE op='refresh_commit' GROUP BY credential_id;"
# Write authority: the store's fence epoch against the daemon's lease.
sqlite3 "file:$DB?mode=ro" "SELECT epoch FROM cortexkit_fence WHERE id = 0;"
cat "$DATA_DIR"/*.leaseconn-<N> in the actor or source columns is a route channel number, reused as bindings
come and go: it is not a consumer identity. For a consumer-reported invalidation the chain
gives you the credential and the instant, never the reporter; correlate the timestamp
against consumer or supervisor logs.
When filtering invalidations, split on actor: actor = vault is the provider refusing a
refresh, while route-admin or offline-cli is an operator stopping the credential on
purpose. They share a row shape.
Rules that keep these reads honest:
- Use
mode=ro, neverimmutable=1. An immutable open skips the write-ahead log, so on a live vault it returns a pre-WAL snapshot.immutable=1is only for a store nobody is writing. mode=rois also what makes the read inert. SQLite checkpoints on close when the closing connection is the last one attached, whatever statements it ran. A plain read-writesqlite3 store.db "SELECT …"against a stopped vault rewrites the main file and deletes the WAL.store.dbwithout itsstore.db-walis incomplete, and usually silently so. The main file holds only what was checkpointed. Depending on the SQLite build it either refuses to open or opens and answers from an arbitrary checkpointed prefix, andPRAGMA integrity_checkstill saysok.-walis the load-bearing companion;-shmis a rebuildable index. Checkls store.db-walbefore opening a copy.- Copy the directory, never the file. And put the copy outside
~/.local/share/cortexkit, for example in~/.local/state/claustrum-snapshots/(mode0700). A directory copy carriesengram-catalog.json, the lease file and the signed-payloads tree. The backup module discovers stores by reading<data_home>/<child>/engram-catalog.jsonone level deep and fails closed fleet-wide on a descriptor it cannot match to an enrolled module, so a copy beside the live directory stops every module's backup until it is moved. - The table is
audit_logand its timestamp column ists_ms. A wrong column in aWHEREclause returns zero rows, indistinguishable from "nothing happened". Check the schema before believing an empty result. - The chain answers "what happened"; the fence epoch answers only "what is true now". An unchanged fence row is consistent with both a rejected write and a healthy writer that had nothing to claim. To ask whether a write committed, read the chain.
- A lane's silence is only anomalous against its own fetch rate.
credential.getis demand-driven, so token lifetime says when a token expires, not when a consumer asks. Compare a credential's quiet period with its own history of mint intervals, not with the token lifetime or with busier credentials. The direct evidence needs no timing: a provider call that was made and refused writes anauth_eventsrow on both failure arms, so zero rows means no call was refused, provided the store was writable in that window. - A refresh rate far above the token lifetime is the signature of a consumer whose
min_ttl_msdemand is close to that lifetime. The vault records refreshes but not reads, so the ratio that names this (refreshes per read) is not computable here. Ask the consumer for its per-credential read count rather than adding read recording to the hot path. A tight demand drives the ratio toward 1.0.
The store is enrolled with engram, the fleet's backup module, and captured whole through an
online backup over a mode=ro connection, so committed WAL frames are included. The daemon
writes the backup descriptor, <data_dir>/engram-catalog.json, at every start from the
constant ENGRAM_CATALOG_JSON in crates/credentials-module/src/main.rs; it covers
store.db, signed-payloads/ and signed-envelopes/. A new directory kept in the data
directory needs an entry there, or it is not backed up and nothing warns.
Restoring needs TWO things, and one is not in the backup:
restored store.db from the backup generation
+ the matching master key from the recorded OS store, or the recorded operator key file
= a vault that opens
The backup holds the audit key sealed under the master key, never the master key itself. That is deliberate: a backup that carried the key would be enough to decrypt every credential. So a machine restoring this vault must carry the key across separately.
Restore one without the other and the failure does not say "missing key". The store opens, the migrations apply, and unsealing the audit key fails, which surfaces as an open error on an intact file. Before concluding anything about the data, compare what the store wants with what this machine can resolve:
sqlite3 "file:$DATA_DIR/store.db?mode=ro" "SELECT key_id FROM vault_secrets LIMIT 1;"
ck_key_verify --data-dir "$DATA_DIR" # feature-gated tool, takes no leaseKeyMismatch names both fingerprints: the store is fine and the key is wrong.
NoSlotMatchesDb means no slot in the recorded backend matches: a native-store item may be
absent or scoped to a different canonical data directory, or the operator file holds the
wrong key.
After a restore, verify:
ck auth verify-audit # intact
ck auth status # every credential serving (N/N), or the expected degraded set
ck auth usable # stranded: 0A whole-store restore brings back the snapshot's state wholesale: every capability handle that was live then (including ones revoked since, because revocation lives in the same table), its open refresh intents, and older key versions. Revoke the handles that should stay dead rather than deleting rows, and re-check keys other machines remember (section 11).
Other backup tools reach this data too. On macOS, Time Machine includes
~/.local/share/cortexkit/claustrum unless excluded, and a Time Machine restore of the data
directory is the same whole-store restore. Handle files and enrollment tokens that consumers
write under ~/.config or ~/.local/state are also copied into every snapshot whatever their
mode, and those copies outlive a revoke (the live vault still refuses them, until a store
restore brings the old handle back). Check with tmutil isexcluded <dir>: the question is not
whether one backup entry covers a path, but whether any backup tool reaches it.
ck auth rotate-master-key # offline: stop the daemon firstThis is a crash-safe two-slot handover: it stages a new key, re-wraps every record and the
sealed audit key in one transaction, then promotes the new key. A crash at any point reopens
cleanly under whichever key matches the database, including when a previous rotation was
itself interrupted (a staged-but-unpromoted rotation is healed before a new one is staged).
Verify with ck auth verify-audit and ck auth status after restarting the daemon.
ck auth mint-signing-key --id signing:<name>[:<generation>] # prints public key hex and key_id
ck auth mint-kem-key --id kem:<name> # prints public_key_hex and key_idBoth generate the key pair in the CLI process and seal the private half directly into the
vault; it is never written to a file, argv, environment variable or output. Both are
create-only by default; normal rotation of a key that verifiers pin by id is a new
<generation> id.
Some signing keys have their public half announced to other machines, which remember it
together with the key's record_version (for example the per-machine message-signing key
signing:msgsig). For these, a rotation must keep the credential id and move the version
forward:
ck auth mint-signing-key --id signing:msgsig --replacerecord_version only increases within one row's life. ck auth remove deletes the row, and
minting the same id again starts at version 1; a whole-store restore also brings back an older
key at its older version. In both cases peers remember a higher version than this machine
serves, keep the key they hold, and raise an alarm naming the version to beat ("rotate above
N"). Run the --replace above until record_version (see ck auth status) is higher than N.
A key rotated to a new generation id instead (for example a bus account key
signing:<name>:<generation>) needs its consumers re-granted on the new id.
The gh routing-manifest root follows its own ceremony: see
manifest signing.
For local daemon/ck auth iteration, the order is fast check → build/sign → place →
live acceptance → full gate and CI. A local fix need not wait for integration CI or landing on master.
Migrations still require the pre-migration snapshot procedure first; the faster build
path does not change the backup or rollback requirements below.
# While iterating (includes committed, staged, unstaged and untracked changes):
./scripts/fast-check.sh
# Commit the final source before building; --fast reruns the check on that clean commit:
PROBE='<command> should print <X> against the live daemon' ./scripts/release-build.sh --fastfast-check.sh compares against origin/master (or an explicit --base <ref>), checks workspace formatting,
workspace, and lints/tests changed crates. It always runs consumer wire-contract tests,
authorization/uniform-refusal/anti-enumeration tests, the no-secret-output and direct-nonce
source scans, and migration safety tests. It names the exact filters and skipped full-gate
arms in its output. Release signing is still checked by the builder, not by the fast check.
release-build.sh --fast permits a clean, committed local revision even before it reaches
origin/master. It runs the fast check itself and refuses if checks change the source or
HEAD. It skips real_daemon_e2e and cli_admin tests against the staged binaries, so live acceptance is mandatory, not an
optional substitute for a source test. build-info.txt records fast_path,
on_origin_master, the observed origin_master SHA, and whether the fetch succeeded. An
unresolved remote is recorded as unknown, not as an ancestry pass. Without --fast, the
existing branch/ancestry guards, catalogue check and staged-artifact suites remain. Published
releases still use the guarded release workflow; the fast lane is not a public release path.
Default and fast builds refuse on a dirty tree, stamp the source revision into both binaries, sign each with its
pinned identifier and the hardened runtime, and prints the revision and sha256 it produced. It
builds with CARGO_TARGET_DIR=target/stamped and copies the artifacts to
target/staged/<rev>/, where a later cargo build cannot overwrite them. A stage is safe from
overwrite but not from cargo clean, which removes everything under target/; do not run it
between staging and placement.
Every build carries a reachability probe: release-build.sh refuses without PROBE. The
probe names the behaviour that proves the change is LIVE rather than merely placed; only the
requester knows it. none: <reason> is a valid answer for a build with no observable change.
Signing uses pinned identifiers:
codesign --force --sign - --options runtime --identifier ck-claustrum target/staged/<rev>/ck-claustrum
codesign --force --sign - --options runtime --identifier ck-auth target/staged/<rev>/ck-authmacOS's default ad-hoc identifier embeds the binary's link-time UUID, so it changes on every
build, and macOS binds privacy grants to that identifier; an unpinned release silently revokes
them. Pinning also makes the published hash equal the placed hash. Do not re-sign at the
destination: one codesign --force --sign - there reverts the identifier.
Place by rename, never by a plain cp over the destination:
cp target/staged/<rev>/ck-claustrum ~/.local/share/cortexkit/bin/.ck-claustrum.incoming
mv -f ~/.local/share/cortexkit/bin/.ck-claustrum.incoming ~/.local/share/cortexkit/bin/ck-claustrumcp rewrites the destination in place, keeping its inode. Overwriting pages under a mapped,
code-signed image invalidates the signature: the placed binary is killed with SIGKILL
(rc=137, no output) the next time it executes, while the running daemon keeps serving from
its old mapping, so a restart in that window takes the vault down. Rename allocates a new inode
and leaves the running process's mapping intact. It also keeps the inode acceptance leg
meaningful: after a cp, "running inode equals disk inode" passes for the wrong reason.
Restart the module, then run the acceptance ladder:
scripts/accept-deploy.sh <rev> target/staged/<rev>Run the script rather than retyping its legs. Each leg is able to fail:
| check | why it discriminates |
|---|---|
| deployed hash equals the new build's hash and differs from the old one | publish both values; comparing the system to itself passes trivially |
<dest> --version reports the revision you built |
the only check that asks the binary what it is |
| the running process's image inode equals the deploy path's inode | proves the process is not still executing an unlinked predecessor |
the open store.db is the one you expect |
every other check answers "is it healthy", not "is it the right vault" |
ck auth status reports every credential serving |
a daemon whose master key was unavailable at boot is alive and serving nothing |
| mint a throwaway handle, then revoke it | exercises the fenced write path and its atomic audit append |
The store check asks the kernel rather than the process:
lsof -p "$(pgrep -a -x ck-claustrum)" | awk '$NF ~ /store\.db$/ {print $NF}'Use pgrep -x, never pgrep -f: -f matches the whole command line, including any shell
whose script text contains the name. And by default pgrep excludes its own ancestors, so from
a shell running under the supervisor pgrep -x ck-subc returns nothing while the supervisor is
alive. pgrep -a includes ancestors; ps -axo pid=,comm= | awk '$2 ~ /ck-subc$/' needs no
flag. An empty pgrep is not evidence a process is down. Do not use pgrep -v as an inverse:
an excluded ancestor lands on the non-matching side.
What the ladder does not cover: whether the behaviour you shipped is reachable. Every leg asks whether the right bytes are in the right place. A CLI change whose logic runs inside the daemon is not live until the daemon half is placed and restarted. Exercise the build's probe end to end, and if it does not show, check which binary owns the behaviour before assuming the deploy failed.
After accept-deploy.sh and the live probe pass, run the full checks and fix forward on any failure:
./scripts/gate.sh
# Land through the train so Linux/Windows CI and the full mutation replay run too.
./scripts/train-push.shscripts/gate.sh is post-deploy feedback for local iteration, not permission to omit CI.
If a hash comparison fails after someone re-signed the binary, re-sign a copy of the known
build with the known identifier and compare that; dwarfdump --uuid (invariant under signing)
and a signature-stripped shasum also settle it. LC_UUID identifies a (commit, path,
toolchain) triple, not a commit: the same commit built in two checkouts gives two UUIDs. To
name the commit a running binary came from, ask --version.
At the behavioural boundary, not on a timer. CLI-only commits accumulate freely. The moment a
commit touches daemon-linked source (credentials-core/src, or anything under
credentials-module/src outside bin/), that is the cut point and the pair ships together.
After deploying a vault that serves credential.open, mint a disposable recipient with
ck auth mint-kem-key --id kem:probe (save the printed public_key_hex and key_id). Grant
open on the exact kem:probe selector to a test module with ck auth grant --principal reserved:<module-id> --selector-kind exact --selector kem:probe --operation open. Seal a
message with cargo run -p credentials-module --example kem_seal_probe -- <public_key_hex> <plaintext> > /tmp/kem-fields.json, then run cargo run -p credentials-module --example vault_read_probe -- --subc <connection-file> --as-module <module-id> --scoped-id kem:probe --open-fields /tmp/kem-fields.json. The probe opens twice on the same bound connection: verify
identical plaintext_b64 values and a key_id matching the minted value. Check for exactly one
scoped_first_use event for (kem:probe, <module-id>, open), then revoke the grant with
ck auth revoke-grant using the same selector and operation, and remove the key with
ck auth remove --id kem:probe. The disposable probe does not exercise a real consumer's
principal, so a real consumer's first open still needs its own verification.
Keep the two most recent rollback copies of each binary
(~/.local/share/cortexkit/bin/ck-auth.pre-<rev>-<ts>) and delete the rest; every one is
reproducible by re-running release-build.sh at its rev. A bin directory full of
similarly-named binaries is where somebody eventually runs the wrong one. Delete on sight any
copy named for an event rather than a rev. release-build.sh prunes staged trees itself,
keeping the three newest and, whatever its age, the stage matching the deployed binary.
A daemon rollback copy stops being a rollback target at the next migration. Rolling the daemon back is safe only while the target's migration chain is at least as new as the store's. A binary refuses to open a store ahead of its chain, naming both versions, but that refusal lives in the binary you roll back TO: a target that predates the refusal migrates and serves a schema it cannot reason about, and fails at the first write of the wrong shape. Check the target against the store:
sqlite3 "file:$DATA_DIR/store.db?mode=ro" \
"SELECT MAX(version) FROM cortexkit_schema_version WHERE namespace='credentials';"
git merge-base --is-ancestor <newest-migration-commit> <target-rev> \
&& echo "target carries that migration" || echo "STALE: not a rollback target"A stale target is not a rollback. Restore the store alongside the binary from a backup taken before the migration (with its master key, section 10), or roll forward.
To roll back a binary: place the rollback copy by rename exactly as above, restart the module,
and run scripts/accept-deploy.sh against the rev you rolled back to.