Spike #828: Passive observer telemetry for harness session analysis

[kmondlane]

Summary

This spike implements a passive-observer telemetry system that extracts observational signals from local harness session logs without transmitting session content. The system reads Claude Code and Codex session files, extracts asset invocation data and token usage, attributes observations to assets, and produces a privacy-preserving telemetry envelope suitable for analyzing asset performance in real user environments.

Key Changes

Schema & Allowlist

• telemetry-field-gate.json: Complete egress allowlist defining 85 leaf paths across 13 disclosure categories (bookkeeping, device/harness identity, run shape, token totals, asset hashes, timing/token stats, etc.)
• telemetry-envelope.schema.json: Closed JSON schema for the wire format with OTLP-like semantics (resource block + typed records), enforcing no wall-clock finer than UTC day, no session duration, no names/paths/costs

Prototype Implementation

• sources/claude_code.py: Parser for Claude Code session transcripts (main + subagent trees), projects all content to hashes/counts before storage
• sources/codex.py: Parser for Codex rollout format with MCP tool namespace handling and token deduplication
• sources/base.py: Harness-neutral data model (SessionFacts, ToolCall, Usage, LoadedSetEvent) with privacy invariant that no free text survives parsing
• extract.py: Derives per-run facts (counts, token totals, invocations, outcomes) from session tree, merging main + child transcripts
• attribute.py: Maps invoked assets to content hashes (in-band > mtime > descriptor > name), tracks binding confidence (exact/mtime/unproven), computes asset attribution tiers
• aggregate.py: Assembles deterministic wire envelope with sorted records/assets, HMAC-based run pseudonyms, and mergeable stats (n, sum, min, max, sumsq)
• check_field_gate.py: Gate checker validating payloads against the allowlist, enforcing closed enums, format rules, and dynamic forbid sets (names, IDs never in string leaves)
• rank.py: Display logic for confidence-tagged asset rankings with Wilson interval ordering, evidence states, and non-causal copy enforcement
• observe.py: Entry point orchestrating the full pipeline (read → extract → attribute → aggregate → gate-check → rank)

Testing & Documentation

• Comprehensive test suite: test_claude_code_source.py, test_codex_source.py, test_extract.py, test_attribute.py, test_aggregate.py, test_gate.py, test_rank.py, test_nonblocking.py, test_lint.py, test_taskcat.py
• CONTRACTS.md: Module-level invariants and contracts (no free text, determinism, fail-open, durations as differences)
• README.md: Comprehensive spike answer with design rationale, privacy model, and scope boundaries
• SCOPE-965.md: Handoff notes for production implementation (#965)
• Fixtures: Invented Claude Code and Codex session files with sentinel markers to verify content never survives
• Worked example: Real scrubbed output demonstrating the ranking display

Supporting Files

• taskcat.py: Task category rule set derived from tool-mix shares (closed enum, published boundaries)
• lint_copy.py: Causal-language lint rejecting phrases that turn observations into claims
• cursor_store.py: Resumable, non-blocking session reader with atomic cursor persistence
• prices.json: Display-time price table for cost derivation (never stored/transmitted)

Notable Implementation Details

• Privacy by projection: Every session line is projected through _project() before any other use; content (text, thinking, tool I/O, attachments) reduced to hashes/lengths; raw line dropped immediately
• Determinism: Same inputs + secret + date → byte-identical envelope; records sorted by (day, run_id), assets by asset_id, JSON canonical (sorted keys, no whitespace)
• Fail-open: Malformed lines counted and

https://claude.ai/code/session_01HzWRjTKapPWoSWrt12re8K

[kmondlane]

@JTP75 For your review

[kmondlane]

Review: verified against vettd#828's acceptance criteria

Checked out the head (6328b63) and ran the prototype rather than reading the PR body. Everything below is a result I reproduced locally, not a restatement of the description.

Verified

Claim	Result
Test suite passes	183 passed, 18 subtests, 0 skipped (~7s)
Gate defines 85 leaf paths	fields = 85 exactly
Pipeline runs end to end	Both claude_code and codex fixtures produce a payload + ranked output
Byte-identical determinism	Two runs, same secret and day → cmp identical
"Refuses to write if the gate fails"	Accurate — observe.py:199-206 gate-checks before the write, not after
No free text survives	Planted SENTINEL, asset names, RULES.md, cwd → 0 occurrences in the payload. Every string leaf is an enum, hash, semver, or UTC day
Gate catches tampering	Injected unknown key → exit 1; a path injected into an enum leaf → exit 1; violation messages do not echo the offending value
Checked-in worked example passes the gate	0 violations; device_id is the zero-UUID placeholder
Copy lint	0 findings over README, SCOPE-965, ranking

The privacy engineering is the strongest part of this. Projection-to-hash happens at parse time, and the value-level forbid sets close the hole a key-path walker structurally cannot see — the README is right that contract/disclosure.rs::validate_payload_coverage is key-only. §7's "cannot show" column is unusually honest about the limits of its own evidence.

Three things worth raising

1. AC 5 is not met. The issue requires a "provenance and score-eligibility assessment written against #795 and #797." Neither issue is cited anywhere in README.md or SCOPE-965.md. The substance is largely present — extractor_version, binding, key_basis, device_id_source and tokens.basis are method provenance, and D5's "a single machine cannot retire the #916/#917 proxies" is effectively the #797 ruling — but there is no field named for execution locus or key provenance the way #795 specifies, and no written eligibility call. This looks like a short section, not a rewrite.

2. Confounding (AC 2) is handled by construction rather than by assessment. The narrow claim in the header plus lint_copy.py failing causal phrasing is arguably a stronger guarantee than prose. But the explicit "which comparative claims this cannot support" analysis lives in the July pre-spike comment on the issue, not in the answer. Strictly read, the AC is covered by proxy.

3. CI ran nothing on this PR. paths-filter in .github/workflows/ci.yml only matches crates/** and friends, so "Lint & test", "Unsafe code audit" and "Supply chain audit" all report skipped. The 183 tests are green only because they were run by hand. SCOPE-965.md correctly requires #965 to rebuild under crates/** so CI gates it — flagging this only so nobody reads the green checkmarks as coverage of the Python.

Related, and worth naming rather than blocking: the issue lists "Implementing the log parser" as out of scope, and this ships ~10k lines of one. Defensible — attribution feasibility is not rulable without proving it, and the prototype is quarantined outside crates/ and labelled throwaway.

One small footgun

observe.py writes a sibling <out>.dynamic.json holding hostname, home directory, cwd, git branch and harness session IDs in the clear. It is documented in --help, gitignored, and never posted — but it sits next to the payload, so shipping the output directory ships that too. Fine for a prototype; #965 should keep those sets in memory.

Verdict

The PR does what it claims, and the decision (own epic, #965 moving wholesale) is recorded on vettd#828 as AC 6 requires. Commenting rather than requesting changes because the one real gap is a missing documentation section, and whether it blocks the merge or lands as a follow-up is the maintainer's call.

---

Generated by Claude Code