Publish a versioned JSON Schema for pinned traces and question documents #153

Overview

The documents OracleDesk pins to IPFS are its public, verifiable artefacts. A market's question document is committed on-chain through question_hash and meta_uri. A reasoning trace is committed through reasoning_registry.publish_trace(trace_hash, ipfs_cid). The frontend's fetchAndVerifyTrace, the contracts repo's x402 trace API (x402/trace-api.ts) and any third-party auditor fetch these bytes, check the hash and then parse the JSON. Their shape is defined nowhere except implicitly in backend code, and it already differs from what the contracts repo's agents produce.

• questionDocument() in src/services/market-chain.service.ts returns { version: 1, question, category, contractCategory, closeTime, resolutionSource, resolutionHash, backendMarketId }, which is versioned but undocumented.
• The trace payload built in generateReasoningTrace() (src/services/reasoning.service.ts) has no version field. It contains optional fields (betFraction, betSizeUsdc, agentWallet) that canonicalJson silently drops when undefined, so whether a key appears depends on the code path. market-maker.agent.ts adds onChainMarketId, which other callers do not. marketId is a backend UUID, not the on-chain id.
• The contracts repo's agents/core/trace.ts defines ReasoningTrace { agent: 'market-maker'|'trader', marketId: number|null, action, reasoning, inputs, timestamp: unix seconds }, which is a completely different shape for the same registry.

Consumers cannot validate content or tell which version they are reading. This issue defines versioned JSON Schemas for both documents, validates every document before it is canonicalised and pinned, serves the schemas at stable URLs, generates the TypeScript types from them, and adds compatibility tests against the contracts repo's trace type. Already-pinned traces must stay verifiable.

Background & current behavior

• src/utils/hash.util.ts: canonicalJson (sorted keys, bigint → string, Date → ISO, drops undefined), canonicalBytes, sha256Hex, verifyTraceHash. The comment says it mirrors canonicalize in agents/core/trace.ts, but the contracts version has no Date case: a Date has no own keys, so it serialises as {} there and as an ISO string here. Both drop undefined object values only because JSON.stringify does.
• src/services/ipfs.service.ts pinJson(document, name) canonicalises and pins whatever it receives, with no validation.
• src/services/trace-publish.service.ts pinAndPublishTrace({ payload: Record<string, unknown>, action, ... }) takes an untyped payload. trade.service.ts passes payload.tracePayload for buy_yes/buy_no.
• No src/schemas/ directory exists. The repo has no JSON Schema validator dependency (zod 4 is used, and zod 4 has z.toJSONSchema()).
• docs/api.md describes the trace API object, not the pinned document.

Goals

• src/schemas/trace.schema.json and src/schemas/question.schema.json (JSON Schema 2020-12), each with $id, a required schemaVersion const, and additionalProperties: false at top level.
• New documents are validated before canonicalisation. Invalid documents are never pinned.
• TypeScript types generated from (or kept provably in sync with) the schemas and used by reasoning.service.ts, market-chain.service.ts and trace-publish.service.ts.
• Schemas served by the API at stable, immutable URLs and documented in docs/trace-format.md.
• Explicit null instead of omitted optional fields, so the canonical form is the same on every path.
• Compatibility tests against the contracts repo's agents/core/trace.ts, plus a mapping that can wrap a contracts-style trace in the backend envelope.
• Legacy (unversioned) traces stay verifiable and are labelled schemaVersion: 0 in read paths.

Non-goals

• Encrypting trace content (separate confidential-traces issue). The schema should allow an envelope type to be added later.
• Changing the registry contract or the hash algorithm.
• Validating LLM output (the existing zod schemas in market.service.ts/reasoning.service.ts stay).

Proposed implementation

1. Author the schemas with zod as the source of truth. Create src/schemas/trace.ts and src/schemas/question.ts (new) exporting zod 4 schemas, and generate the JSON files with a script scripts/gen-schemas.ts (new, npm run gen:schemas) that uses z.toJSONSchema(schema, { target: 'draft-2020-12' }), adds $id/$schema/title, and writes src/schemas/*.schema.json. A --check mode fails CI when the committed JSON is stale, like npm run check:contracts.
2. Trace document v1.

   export const TraceDocV1 = z.object({
     schemaVersion: z.literal(1),
     kind: z.literal('oracledesk.trace'),
     traceId: z.string().uuid(),
     agentType: z.enum(['MARKET_MAKER', 'TRADER']),
     decisionType: z.string().min(1).max(64),
     action: z.string().min(1).max(64),                 // same string as publish_trace action
     market: z.object({ backendId: z.string().uuid(), onChainId: z.string().regex(/^\d+$/).nullable() }),
     probabilityEstimate: z.number().min(0).max(1),
     marketProbability: z.number().min(0).max(1),
     edge: z.number().min(-1).max(1),
     confidenceInterval: z.object({ lower: z.number(), upper: z.number() }).refine(ci => ci.lower <= ci.upper),
     sourcesUsed: z.array(z.object({ source: z.string(), weight: z.number().min(0).max(1), signal: z.string(), rawValue: z.number().nullable() })).max(50),
     betFraction: z.number().nullable(), betSizeUsdc: z.number().nullable(),
     hedgeConditions: z.array(z.string().max(500)).max(20),
     agentWallet: z.string().nullable(),
     createdAt: z.string().datetime(),
   }).strict();

   Numbers must be finite. Document that floats are serialised with JSON.stringify (shortest round-trip) and that consumers must not re-serialise before hashing.
3. Question document v2. Keep version: 1 documents valid as QuestionDocV1 (the exact current shape, so already-committed question_hash values still describe valid documents), and add QuestionDocV2 with schemaVersion: 2, kind: 'oracledesk.question', resolutionSpecHash (64-hex), resolutionSource: string | null, closeTime (int seconds) and category/contractCategory enums that match MarketCategory and ContractCategoryTag (src/lib/categories.ts). questionDocument() switches to v2 behind QUESTION_DOC_VERSION (default 2), with v1 retained for reproducibility tests.
4. Validation before pinning.
   • Add pinDocument<T>(schema: ZodType<T>, doc: T, name: string) to ipfs.service.ts. It runs schema.parse (on failure → AppError(500, 'DOCUMENT_SCHEMA_INVALID', …) with zod issues in logs only), then calls pinJson.
   • pinAndPublishTrace takes a typed TraceDocV1 instead of Record<string, unknown>. Its action argument must equal doc.action (assert).
   • createMarketOnChain pins through pinDocument(QuestionDocV2, …).
5. Build documents explicitly. Replace the ad-hoc tracePayload object in generateReasoningTrace with buildTraceDoc(input, computed): TraceDocV1. Every optional field is set to null, rawValue ?? null, createdAt is ISO, and market.onChainId is filled from the market row (removing the special onChainMarketId spread in market-maker.agent.ts). Update trade.service.ts/trader.agent.ts to pass the typed document.
6. Canonicalisation guard. In hash.util.ts, add canonicalJsonStrict(value) that throws on undefined, functions, symbols, non-finite numbers and Date (callers convert explicitly), and use it inside pinDocument. Keep canonicalJson unchanged for legacy verification. Also add a lightweight check that a document survives JSON.parse(canonical) → canonicalJsonStrict with identical bytes.
7. Legacy handling. verifyTraceOnChain and the read paths parse fetched JSON with TraceDocAny = z.union([TraceDocV1, LegacyTraceDoc]), where LegacyTraceDoc is a permissive passthrough schema without schemaVersion that is mapped to schemaVersion: 0 in API responses. Hash verification never depends on schema validity. A document can be verified: true while also being schemaValid: false, and the API reports both.
8. Contracts-repo compatibility. Add tests/fixtures/contracts-trace.json, built from agents/core/trace.ts buildTrace(...) output (generate it once with a small script run in the contracts/ submodule, contracts/oracledesk-stellar/agents/core/trace.ts, and commit the JSON plus its hash). Then:
   • write fromContractsTrace(t: ContractsReasoningTrace): TraceDocV1 mapping (agent → agentType, inputs → sourcesUsed when shaped, timestamp seconds → createdAt);
   • test that canonicalJson and the submodule's canonicalize produce identical bytes for every v1 fixture;
   • add a request to docs/contract-requests.md asking the contracts repo to adopt the same schemaVersion/kind envelope, or to publish its schema.
9. Serving and docs. Add a new route file src/routes/schemas.routes.ts mounted at /api/v1/schemas that serves GET /schemas/trace/v1.json, /schemas/question/v1.json, /schemas/question/v2.json with Cache-Control: public, max-age=31536000, immutable and Content-Type: application/schema+json. Each $id equals the public URL (from a new PUBLIC_API_BASE_URL env var). Write docs/trace-format.md (new): fields, canonicalisation rules, a worked example with bytes and sha256, versioning policy (any change to required fields or semantics bumps schemaVersion), and manual verification.

Edge cases & pitfalls

• Changing the document shape changes the bytes and therefore the hash of every new trace. That is fine, because old traces keep their pinned bytes. Never "upgrade" a stored document in place, or verification breaks.
• z.toJSONSchema cannot express refinements such as lower <= upper; document them in description.
• market.onChainId is null when a trace is pinned before the market is on-chain (dry-run, or the trace-before-market ordering in market-maker.agent.ts). The schema must allow it, and the on-chain market_id in the registry is authoritative.
• Floats such as 0.1 + 0.2 serialise as 0.30000000000000004. That is deterministic in V8 but may differ in other languages' serialisers. Recommend that third parties verify the hash on raw bytes, never on a re-serialisation.
• Upper limits (max(50) sources, string lengths) must be generous enough that real LLM output never makes the cycle fail. Truncate during building, not during validation.

Testing plan

• tests/trace-schema.test.ts (new):
  • buildTraceDoc output validates against TraceDocV1 for market-maker and trader inputs
  • trace doc never contains undefined and canonicalJsonStrict round-trips
  • pinDocument refuses an invalid trace and never calls the transport (fake IpfsTransport)
  • question v1 documents for existing markets still hash to stored questionHash (fixture from STATUS Phase D / seeded data)
  • question v2 document validates and pins
  • legacy trace without schemaVersion verifies by hash and reports schemaVersion 0
  • backend canonicalJson matches contracts canonicalize on shared fixtures
  • fromContractsTrace maps the contracts fixture to a valid TraceDocV1
  • committed JSON schemas are up to date (runs the generator in check mode)
• tests/http.test.ts: GET /api/v1/schemas/trace/v1.json returns application/schema+json with immutable caching.
• Commands: npm run gen:schemas -- --check, npm run build, npm test, npm run check:contracts.

Acceptance criteria

• src/schemas/trace.schema.json and src/schemas/question.schema.json exist, are generated from zod sources and fail CI when stale.
• Every new trace and question document carries schemaVersion and kind, uses explicit nulls and validates before pinning.
• pinAndPublishTrace and createMarketOnChain take typed documents, and Record<string, unknown> payloads are gone from the pin path.
• Legacy documents still verify by hash, and read paths report their schema version and validity separately from hash verification.
• A compatibility test proves byte-identical canonicalisation with the contracts repo's canonicalize, and a mapping from the contracts ReasoningTrace exists.
• Schemas are served at stable, immutable URLs whose $id matches.
• docs/trace-format.md documents fields, canonicalisation, versioning and manual verification, and docs/contract-requests.md records the alignment request.
• npm run build, npm test and npm run check:contracts pass.

Resources

• src/services/reasoning.service.ts, src/services/market-chain.service.ts, src/services/trace-publish.service.ts, src/services/ipfs.service.ts
• src/utils/hash.util.ts, src/types/index.ts, src/lib/categories.ts, src/agents/market-maker.agent.ts, src/services/trade.service.ts
• docs/api.md, docs/architecture.md, docs/contract-requests.md
• Contracts repo: agents/core/trace.ts, x402/trace-verification.ts, x402/trace-api.ts
• JSON Schema 2020-12: https://json-schema.org/draft/2020-12/release-notes
• zod 4 JSON Schema: https://zod.dev/json-schema
• RFC 8785 (JSON Canonicalization Scheme) for comparison: https://www.rfc-editor.org/rfc/rfc8785

Generated by Claude Code