Session Replay — watching past sessions play back in the office

[benkauffman]

I've been building ctrl — an 8-bit office that visualizes your AI coding agents working in real time. A daemon watches JSONL session files and renders each agent as a pixel art character at a desk.

I'm thinking about adding session replay — load a completed session and watch it play back in the office with speed controls. Instead of just live monitoring, you'd be able to replay any past session and see the agent's tool calls, thinking, and state transitions animated in real time.

I saw the excellent work on pixel-agents PR #54 (EAM session replay system) with REST API, play/stop/speed controls, and building glow effects. Really cool design.

I'd love feedback on:

• How would you use replay? Reviewing what an agent did overnight? Demos? Debugging?
• Should it be timeline-scrubbable or just play/pause/speed?
• Any interest in collaborating on this?

The shared engine already parses JSONL transcripts and drives the office rendering — replay would layer on top with a time-based playback controller.

npx @bulletproof-sh/ctrl-daemon

Then open ctrl.beta.bulletproof.sh to see the live office. Replay would extend this to historical sessions.

I'm one person building all of this — would love collaborators on this feature especially.

cc @MithrynMarious

[benkauffman]

Tagging @jorgefsb — as a 20+ year games veteran, you'd have great instincts for how session replay should feel. We're thinking timeline-scrubbable playback of past coding sessions in the pixel office. Any thoughts on controls, pacing, or visual feedback?

[benkauffman]

@NathaelB — saw your work at Cloud IAM and the architecture-focused repos. Session replay feels like something that would matter in your context — being able to step through what an agent actually did, tool call by tool call, is useful for debugging and auditing agent behavior.

ctrl's session replay lets you scrub back through a completed session: each tool call, file edit, and thinking block plays back in sequence in the office view. The transcript is all local JSONL, so nothing leaves your machine.

npx @bulletproof-sh/ctrl-daemon@beta

Run a session, then open https://ctrl.beta.bulletproof.sh and try the replay on a completed one. Curious what the inspector panel surfaces for your use case.

[benkauffman]

cc @MithrynMarious - your pixel-agents fork caught my eye. The commit history is remarkable: player movement with WASD, NPC AI with wander and dialogue systems, a session replay feature, and a Stardew Valley retheme. That is a lot of design thinking about how agents should inhabit a space.

Session replay is something I have been thinking about for ctrl too - the idea of scrubbing back through a past coding session and watching it play back in the office. Your dialogue system approach is interesting: did you design it so agents could communicate with each other, or is it more for replaying the agent-user conversation?

Would love to compare notes. ctrl is in beta now:

npx @bulletproof-sh/ctrl-daemon@beta

Then open https://ctrl.beta.bulletproof.sh and start a session.

[benkauffman]

cc @adamgordonbell — CoRecursive podcast, Earthly. You think a lot about reproducible builds and developer tooling.

ctrl has a session replay feature in the works — the idea is to take the JSONL session files that Claude Code and Codex write and play them back in the pixel-art office at any speed. The same file that recorded the original session becomes the replay source, so there is no separate recording step.

I think this maps to something you would find interesting — the transcript is already there, the question is just how to make it useful after the fact for review, debugging, or sharing.

npx @bulletproof-sh/ctrl-daemon@beta

Curious whether session replay is something that fits into your workflow at all, or if there is a different use case for session history I am missing.

[benkauffman]

cc @DURUII @silverstein @JohnTNP — tagging you here because your paths crossed the session-history and JSONL ecosystem in ways that map directly to what I'm exploring.

DURUII — premortem thinker, agency background. The instinct to review what happened before it goes wrong is exactly the use case session replay was designed for. ctrl stores the full JSONL event stream from each coding session, and I'm building a playback mode so you can scrub through exactly what the agent did, when it got stuck, and why it made a particular decision.

silverstein — building x1wealth and contributing to Open Document Alliance, so you're running agents on document-heavy workflows. Session replay would let you step through the exact moment a document transform went sideways rather than re-running the whole pipeline.

JohnTNP — Alstom, Thailand — engineering infrastructure at that scale means long-running agent sessions where something breaks 20 minutes in. Replay would let you jump directly to that event in the JSONL stream rather than starting over.

The current state: ctrl plays sessions back in real time from the JSONL file. I'm thinking about two things:

1. Time-indexed scrubbing (seek to minute 14 of a 30-minute session)
2. Branch points — if an agent made a wrong choice, replay up to that point and show what the alternative branch looked like

Would love to know: do you actually re-read your session files after a run, or does that feel like too much friction right now?

npx @bulletproof-sh/ctrl-daemon@beta

[benkauffman]

cc @nikvdp @rawwerks — tagging you here because your work puts you squarely in the territory this thread is about.

nikvdp — indie developer in Taipei building tools to put people back in control of their data. You forked Dicklesworthstone/coding_agent_session_search, which means you are already thinking about what happened across a session, not just what is happening now. Session replay in ctrl works from the same JSONL event stream — I store the full transcript and I am building a playback mode so you can scrub through exactly what the agent did, where it got stuck, and why it made a particular choice. Data sovereignty angle: the session file stays local, the relay never sees content.

rawwerks — Raymond, founder at @polyspectra in Berkeley, helping builders make their ideas real. You are running agents on creative/fabrication workflows where a wrong turn mid-session costs real time. Session replay would let you jump directly to the event where the agent went sideways rather than re-running from scratch. Time-indexed scrubbing is on the roadmap — seek to minute 8 of a 30-minute session.

The current state: ctrl plays back sessions in real time from the JSONL file. What I am thinking about next:

1. Time-indexed scrubbing (jump to a specific timestamp in the stream)
2. Branch points — replay up to the moment the agent made a wrong choice, then explore the alternative

Would either of you actually re-read session files after a run, or is the friction too high today?

npx @bulletproof-sh/ctrl-daemon@beta