Stop exposing premium trace content through public IPFS CIDs #154

Overview

Full reasoning traces are OracleDesk's paid product. The backend sells 24-hour daily passes (POST /traces/:id/unlock), and the contracts repo's x402 service sells per-trace access (GET /traces/:onChainTraceId behind paymentMiddleware). Every full trace, though, is pinned in plaintext to public IPFS, and the CID is handed to anonymous users:

• GET /traces (listTraces in src/controllers/trace.controller.ts) selects ipfsCid and traceHash for every trace, with no auth.
• GET /traces/:id without a pass returns the free preview from getTracWithAccessControl (src/services/subscription.service.ts), which still includes ipfsCid next to lockedFields: ['fullSources', 'hedgeConditions', 'betFraction', 'betSizeUsdc'].
• The REASONING_PUBLISHED socket event (src/services/indexer.service.ts) broadcasts ipfsCid to every client.
• reasoning_registry.get_trace(id) returns ipfs_cid to anyone who can simulate a read.

Anyone can fetch https://ipfs.io/ipfs/<cid> and read the "locked" fields for free, so the paywall is cosmetic. Hiding the CID cannot fix this, because the registry publishes it on-chain by design.

This issue makes premium content confidential while keeping the property the product is built on: anyone can verify that a trace's content matches what the agent committed on-chain at decision time. The approach: pin an authenticated encrypted envelope, commit its hash on-chain, keep per-trace keys server-side, and release keys or plaintext only to entitled users.

Background & current behavior

• src/services/trace-publish.service.ts pinAndPublishTrace({ traceId, payload, action, onChainMarketId }): pinJson(payload) → ipfsCid, traceHash = sha256(pinned bytes) → publishTrace({ trace_hash, ipfs_cid }). verifyTraceOnChain fetches the CID and compares sha256(bytes) with the on-chain trace_hash.
• src/services/subscription.service.ts: checkAccess(userId, traceId) finds an ACTIVE DAILY_PASS (any trace) or PER_TRACE (that trace) subscription. getTracWithAccessControl returns the full DB row when access is granted. The DB row holds the plaintext fields (sourcesUsed, hedgeConditions, betFraction, betSizeUsdc), so the backend can serve them without IPFS.
• Contracts repo x402/trace-api.ts loadVerifiedTrace fetches the CID, checks verifyTraceHash, and returns the parsed JSON as content.
• The frontend's lib/stellar/trace.ts fetchAndVerifyTrace(cid, onChainHash) fetches from a public gateway and renders the parsed JSON when the hash matches.
• ReasoningTrace has ipfsCid, traceHash, isPublic (unused), verified and previewSources.

Goals

• New premium traces are pinned only as an encrypted envelope. The plaintext never reaches IPFS.
• Public verifiability: anyone can (a) check that the pinned envelope matches the on-chain trace_hash and (b) once they have the plaintext, check that it matches the plaintextHash committed inside that envelope.
• Per-trace data keys, wrapped by a server-side key-encryption key (KEK) with an id, so the KEK can be rotated.
• Entitled users (daily pass, or later a verified x402 purchase) receive the plaintext from the backend, or the per-trace key for client-side decryption, with the plaintext hash check done client-side.
• A free public subset (the preview) is pinned in the clear inside the envelope header, so the free tier needs no key.

Non-goals

• Deleting plaintext already on IPFS, which cannot be recalled.
• Changing the reasoning-registry contract.
• Implementing the x402 side. This issue writes the change request in docs/contract-requests.md and keeps the backend compatible.
• Encrypting question documents, which are public by design.

Proposed implementation

1. Envelope format (oracledesk.trace.enc v1). Canonical JSON so it hashes like everything else:

   {
     "kind": "oracledesk.trace.enc", "schemaVersion": 1,
     "alg": "XChaCha20-Poly1305", "kid": "kek-2026-10",
     "plaintextHash": "<sha256 hex of canonical plaintext bytes>",
     "public": { "traceId": "…", "agentType": "MARKET_MAKER", "decisionType": "…", "market": {"backendId": "…", "onChainId": "0"},
                 "probabilityEstimate": 0.62, "marketProbability": 0.5, "edge": 0.12, "previewSources": [ … ], "createdAt": "…" },
     "nonce": "<base64 24 bytes>",
     "ciphertext": "<base64>"
   }

   AEAD associated data = canonical bytes of the envelope without nonce and ciphertext (that is, kind, schemaVersion, alg, kid, plaintextHash, public), so the header cannot be swapped without breaking decryption. The on-chain trace_hash stays sha256(pinned bytes) = sha256(canonical envelope), which keeps verifyTraceHash and x402/trace-verification.ts working unchanged on the pinned bytes.
2. src/services/trace-crypto.service.ts (new).

   export interface SealedTrace { envelope: TraceEnvelopeV1; bytes: Buffer; plaintextHash: string; wrappedKey: Buffer; kid: string }
   export function sealTrace(plaintext: Buffer, publicPart: TracePublicPart, kek: Kek): SealedTrace;
   export function openTrace(envelope: TraceEnvelopeV1, dataKey: Buffer): Buffer;   // verifies AEAD + plaintextHash
   export function unwrapKey(wrapped: Buffer, kid: string): Buffer;
   export interface Kek { kid: string; wrap(dk: Buffer): Buffer; unwrap(w: Buffer): Buffer }
   export function loadKeks(cfg: Config): Map<string, Kek>;

   Use @noble/ciphers (xchacha20poly1305, audited) for the payload and for wrapping 32-byte random data keys. KEKs come from a new TRACE_KEKS env var (JSON kid → base64 key, with TRACE_KEK_ACTIVE), validated in src/config/index.ts and redacted in src/lib/logger.ts.
3. Schema (new migration). Add a new table trace_keys(traceId PK/FK → reasoning_traces ON DELETE CASCADE, kid, wrappedKey BYTEA, plaintextHash CHAR(64), createdAt). Add contentMode (PLAINTEXT_LEGACY | ENCRYPTED) and plaintextHash to reasoning_traces. Keeping keys out of the main row stops a select * leaking them.
4. Pin path. In trace-publish.service.ts, pinAndPublishTrace → when TRACE_CONFIDENTIAL=true (default true for new deployments, documented), build the plaintext canonical bytes, then sealTrace → pinJson(envelope) → store trace_keys and contentMode = ENCRYPTED in the same transaction as ipfsCid/traceHash → publishTrace with the envelope hash. Before pinning, decrypt the envelope with the freshly made key and compare bytes, so a broken envelope is never committed.
5. Read paths.
   • getTracWithAccessControl: without access, return the public subset plus ipfsCid, traceHash (now harmless), contentMode and plaintextHash. With access, also return verification: { envelopeHash, plaintextHash } so the client can check both hashes.
   • New GET /traces/:id/key (requireAuth + checkAccess) returns { kid, dataKey (base64), plaintextHash } for client-side decryption. Rate-limit it per user and log each release to AgentLog/an audit table (TRACE_KEY_RELEASED).
   • verifyTraceOnChain: still compares the envelope bytes with the on-chain hash. Additionally parse the envelope, check plaintextHash === trace.plaintextHash, and report contentMode.
6. x402 coordination. Add a section to docs/contract-requests.md: x402 loadVerifiedTrace should detect kind: 'oracledesk.trace.enc' and, after payment, obtain the data key from a new backend endpoint POST /internal/traces/:onChainTraceId/key authenticated with a shared HMAC or mTLS secret (X402_KEY_RELEASE_SECRET), decrypt it, check plaintextHash, and return the plaintext. Implement that backend endpoint in this issue behind a feature flag, with replay protection (a timestamp plus nonce in the HMAC). Document in docs/api.md that, until x402 ships the change, per-trace x402 purchases of encrypted traces return the envelope only.
7. Legacy traces. Rows with ipfsCid and no trace_keys are PLAINTEXT_LEGACY. Ship scripts/unpin-legacy-traces.ts (new, dry-run by default), which lists legacy traces and can call Pinata unpin. State plainly in docs and the script's output that public copies may persist on other nodes and gateways. Do not republish legacy traces encrypted, because the on-chain hash would not match. Mark them in API responses so the frontend can label them "public (legacy)".
8. Docs. Add a "Confidential traces" section to docs/architecture.md (threat model: the backend can read every trace, the operator holds the KEK, verifiability is "commit then reveal to payers"), and update the trace object and new endpoints in docs/api.md.

Edge cases & pitfalls

• Trust model change: a third party can verify that the envelope existed on-chain at publish time and, given the plaintext, that it matches. Without paying, they cannot read the content. Say this honestly in the docs. Consider an optional TRACE_REVEAL_AFTER_RESOLUTION that publishes the data key once the market resolves.
• Never reuse a data key across traces.
• KEK rotation: new traces use TRACE_KEK_ACTIVE, and old kids must stay loadable. Add scripts/rewrap-trace-keys.ts to re-wrap data keys under a new KEK without touching IPFS.
• Losing the KEK loses every encrypted trace for paying users. Tie it into the backup runbook.
• Do not log plaintext, data keys or wrapped keys. Extend the pino redaction paths.
• canonicalJson of the envelope must be computed once. Never rebuild it from the DB, because base64 or field-order drift would change the hash.
• The preview must not leak locked fields. Build public from an allowlist, never by deleting keys from the full object.

Testing plan

• tests/trace-crypto.test.ts (new):
  • sealTrace then openTrace round-trips and plaintext hashes to plaintextHash
  • tampering with public header or ciphertext fails authentication
  • envelope bytes hash equals the trace_hash passed to publishTrace (fake IpfsTransport + stubbed publishTrace)
  • KEK rotation: key wrapped under old kid still unwraps; new traces use active kid
  • preview response contains no locked fields and no key material
• tests/http.test.ts: GET /traces/:id/key requires auth and an active pass, internal key release rejects bad HMAC and replayed nonce.
• verifyTraceOnChain tests for ENCRYPTED and PLAINTEXT_LEGACY modes.
• Commands: npx prisma migrate dev --name confidential_traces, npm run build, npm test, npm run check:contracts.

Acceptance criteria

• With TRACE_CONFIDENTIAL=true, newly pinned traces contain no locked field in plaintext, as shown by a test that scans the pinned bytes.
• The on-chain trace_hash equals sha256 of the pinned envelope, so existing hash verification in the backend and in x402 works unchanged.
• Envelopes bind the public header and plaintextHash through AEAD associated data, and tampering is detected.
• Per-trace data keys are stored wrapped under a KEK identified by kid, KEK rotation is supported, and keys never appear in logs or default selects.
• GET /traces/:id/key releases keys only to entitled users, with audit logging.
• Legacy plaintext traces are labelled, an unpin script exists, and the docs state the limits of unpinning.
• docs/contract-requests.md describes the x402 change, and the backend key-release endpoint exists behind a flag with HMAC and replay protection.
• docs/api.md and docs/architecture.md describe the format, endpoints and trust model.
• npm run build, npm test and npm run check:contracts pass.

Resources

• src/services/trace-publish.service.ts, src/services/ipfs.service.ts, src/services/subscription.service.ts, src/controllers/trace.controller.ts
• src/services/indexer.service.ts (REASONING_PUBLISHED), src/lib/logger.ts, src/config/index.ts, prisma/schema.prisma
• docs/api.md (trace object, x402 section), docs/contract-requests.md, docs/architecture.md
• Contracts repo: x402/trace-api.ts, x402/trace-verification.ts, x402/registry-trace-repository.ts
• @noble/ciphers: https://github.com/paulmillr/noble-ciphers
• XChaCha20-Poly1305 draft: https://datatracker.ietf.org/doc/html/draft-irtf-cfrg-xchacha
• Envelope encryption: https://cloud.google.com/kms/docs/envelope-encryption
• Pinata unpin API: https://docs.pinata.cloud/api-reference/endpoint/ipfs/unpin-file

Generated by Claude Code