Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,38 @@ results and known failures are indexed there in `docs/quality/README.md`, and
its `AGENTS.md` describes the guidance to read before changing session
attachment, room retention, control, readiness, polling or deployment.

## Temporal workflow tests and the replay determinism gate

`apps/worker` runs every Temporal workflow on the real time-skipping test
server (`@temporalio/testing`, `TestWorkflowEnvironment.createTimeSkipping`)
with fully stubbed activities; see `apps/worker/tests/temporal-sandbox/`.
These tests need no network, database, Docker, Temporal cluster or AI
provider:

```sh
npm test -w @space/worker
```

The same directory contains the replay determinism gate
(`apps/worker/tests/temporal-sandbox/replay-gate.test.ts`). It replays the
synthetic workflow histories recorded under
`apps/worker/tests/fixtures/workflow-histories/` through
`Worker.runReplayHistories`, so a workflow change that is not replay
compatible fails CI with `DeterminismViolationError`. A negative test also
replays one history against a deliberately incompatible workflow variant
to prove the gate detects it.

Re-record histories only when a workflow legitimately changes its command
sequence, for example after shipping a new `patched()` change:

```sh
npm run temporal:record-histories -w @space/worker
```

Never re-record just to make a failing replay pass — that is exactly the
class of non-deterministic workflow change the gate exists to catch. The
fixtures README documents the rules.

Keep changes reviewable and include the exact commands and limitations of your
verification. Never include credentials, private transcripts or local runtime
state in a contribution.
Expand Down
5 changes: 4 additions & 1 deletion apps/worker/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,8 @@
"start": "node dist/worker.js",
"build": "tsc -p tsconfig.json",
"check": "tsc -p tsconfig.json --noEmit",
"test": "vitest run"
"test": "vitest run",
"temporal:record-histories": "RECORD_HISTORIES=1 vitest run tests/temporal-sandbox/record-histories.test.ts"
},
"dependencies": {
"@space/codex-app-server": "0.2.5",
Expand All @@ -26,6 +27,8 @@
"undici": "8.12.3"
},
"devDependencies": {
"@temporalio/proto": "1.24.0",
"@temporalio/testing": "1.24.0",
"typescript": "^5.9.3",
"vitest": "^4.1.12"
},
Expand Down
34 changes: 34 additions & 0 deletions apps/worker/tests/fixtures/workflow-histories/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# Workflow history fixtures (replay determinism gate)

These are synthetic Temporal workflow histories recorded from the real
workflow code on the time-skipping test server with fully stubbed
activities. They contain no real rooms, missions or memory state. The
replay gate (`tests/temporal-sandbox/replay-gate.test.ts`) replays every
history through `Worker.runReplayHistories`, so any change to a workflow
that is not replay compatible fails CI with `DeterminismViolationError`.

There must be at least one history per workflow plus one recorded from
the pre-patch supervisor branch (no `room-background-continuation-v1`
marker), which replays the legacy code path kept for old histories. The
pre-patch fixture is produced by running the production
`roomAgentSupervisorWorkflow` with Temporal's `patchActivationCallback`
forcing that patch off — not by copying workflow code.

## When to re-record

Re-record only when a workflow legitimately changes its command sequence,
for example after shipping a new `patched()` change. Run:

```sh
npm run temporal:record-histories -w @space/worker
```

Never re-record just to make a failing replay pass — that is exactly the
class of non-deterministic workflow change this gate exists to catch. If
a replay fails, fix the workflow code (usually by guarding the change
with `patched()`) or record the pre-change history under a new fixture name
instead of overwriting the old one. The recording command writes only its
fixed fixture names. Before re-recording, copy the affected pre-change JSON
to the new name, then add that name to `EXPECTED_FIXTURES` in
`tests/temporal-sandbox/replay-gate.test.ts` and to the expected list in
`tests/temporal-sandbox/record-histories.test.ts`.
Original file line number Diff line number Diff line change
@@ -0,0 +1,258 @@
{
"workflowId": "fixture-codex-turn",
"history": {
"events": [
{
"eventId": "1",
"eventTime": {
"seconds": "2424028730",
"nanos": 620000000
},
"eventType": "EVENT_TYPE_WORKFLOW_EXECUTION_STARTED",
"workflowExecutionStartedEventAttributes": {
"workflowType": {
"name": "codexAppServerTurnWorkflow"
},
"taskQueue": {
"name": "temporal-sandbox-tests",
"kind": "TASK_QUEUE_KIND_NORMAL"
},
"input": {
"payloads": [
{
"metadata": {
"encoding": "anNvbi9wbGFpbg=="
},
"data": "eyJyb29tSWQiOiJyb29tLWZpeHR1cmUtMDEiLCJwYW5lSWQiOiJwYW5lOnJvb20tZml4dHVyZS0wMS1maXh0dXJlIiwicHJvbXB0IjoiU3ludGhldGljIHR1cm4gcHJvbXB0LiIsImFydGlmYWN0SWRzIjpbXSwicHJvdmlkZXJJZCI6bnVsbCwibW9kZWxJZCI6ImdwdC10ZXN0IiwicmVhc29uaW5nRWZmb3J0IjpudWxsLCJhZ2VudFNlc3Npb25JZCI6InNlc3Npb246cm9vbS1maXh0dXJlLTAxLWZpeHR1cmUiLCJhZ2VudFJ1bklkIjoiYWdlbnRfcnVuOnJvb20tZml4dHVyZS0wMS1maXh0dXJlIiwiYWdlbnRBc3Npc3RhbnRNZXNzYWdlSWQiOiJhZ2VudF9tc2c6YXNzaXN0YW50LXJvb20tZml4dHVyZS0wMS1maXh0dXJlIiwiY29sbGFib3JhdGlvbk1vZGUiOiJkZWZhdWx0IiwidHJhY2VJZCI6InRyYWNlOnJvb20tZml4dHVyZS0wMS1maXh0dXJlIiwicm9vbUFnZW50TWlzc2lvbklkIjoibWlzc2lvbi1maXh0dXJlLTAxIiwicm9vbUFnZW50T3BlcmF0b3JSZXF1ZXN0IjoiU3ludGhldGljIG9wZXJhdG9yIHJlcXVlc3QuIiwic2VsZWN0ZWRUb29sSWRzIjpbInJvb206aW5zcGVjdCJdfQ=="
}
]
},
"workflowExecutionTimeout": {
"seconds": "315360000"
},
"workflowRunTimeout": {
"seconds": "315360000"
},
"workflowTaskTimeout": {
"seconds": "10"
},
"originalExecutionRunId": "2bd10397-54a6-4def-9422-6849ad6af5b4",
"identity": "25805@cursor",
"firstExecutionRunId": "2bd10397-54a6-4def-9422-6849ad6af5b4",
"attempt": 1,
"firstWorkflowTaskBackoff": {},
"header": {}
}
},
{
"eventId": "2",
"eventTime": {
"seconds": "2424028730",
"nanos": 620000000
},
"eventType": "EVENT_TYPE_WORKFLOW_TASK_SCHEDULED",
"workflowTaskScheduledEventAttributes": {
"taskQueue": {
"name": "temporal-sandbox-tests",
"kind": "TASK_QUEUE_KIND_NORMAL"
},
"startToCloseTimeout": {
"seconds": "10"
},
"attempt": 1
}
},
{
"eventId": "3",
"eventTime": {
"seconds": "2424028730",
"nanos": 620000000
},
"eventType": "EVENT_TYPE_WORKFLOW_TASK_STARTED",
"workflowTaskStartedEventAttributes": {
"scheduledEventId": "2",
"identity": "25805@cursor",
"historySizeBytes": "845"
}
},
{
"eventId": "4",
"eventTime": {
"seconds": "2424028730",
"nanos": 845000000
},
"eventType": "EVENT_TYPE_WORKFLOW_TASK_COMPLETED",
"workflowTaskCompletedEventAttributes": {
"scheduledEventId": "2",
"identity": "25805@cursor",
"binaryChecksum": "@temporalio/worker@1.24.0+9a365569848f3b7720675da78257957763d6c9652a5349034772dfe58fee989c",
"sdkMetadata": {
"coreUsedFlags": [
1,
2,
3
],
"sdkName": "temporal-typescript",
"sdkVersion": "1.24.0"
},
"meteringMetadata": {}
}
},
{
"eventId": "5",
"eventTime": {
"seconds": "2424028730",
"nanos": 845000000
},
"eventType": "EVENT_TYPE_ACTIVITY_TASK_SCHEDULED",
"activityTaskScheduledEventAttributes": {
"activityId": "1",
"activityType": {
"name": "runCodexAppServerTurn"
},
"taskQueue": {
"name": "temporal-sandbox-tests",
"kind": "TASK_QUEUE_KIND_NORMAL"
},
"header": {},
"input": {
"payloads": [
{
"metadata": {
"encoding": "anNvbi9wbGFpbg=="
},
"data": "eyJyb29tSWQiOiJyb29tLWZpeHR1cmUtMDEiLCJwYW5lSWQiOiJwYW5lOnJvb20tZml4dHVyZS0wMS1maXh0dXJlIiwicHJvbXB0IjoiU3ludGhldGljIHR1cm4gcHJvbXB0LiIsImFydGlmYWN0SWRzIjpbXSwicHJvdmlkZXJJZCI6bnVsbCwibW9kZWxJZCI6ImdwdC10ZXN0IiwicmVhc29uaW5nRWZmb3J0IjpudWxsLCJhZ2VudFNlc3Npb25JZCI6InNlc3Npb246cm9vbS1maXh0dXJlLTAxLWZpeHR1cmUiLCJhZ2VudFJ1bklkIjoiYWdlbnRfcnVuOnJvb20tZml4dHVyZS0wMS1maXh0dXJlIiwiYWdlbnRBc3Npc3RhbnRNZXNzYWdlSWQiOiJhZ2VudF9tc2c6YXNzaXN0YW50LXJvb20tZml4dHVyZS0wMS1maXh0dXJlIiwiY29sbGFib3JhdGlvbk1vZGUiOiJkZWZhdWx0IiwidHJhY2VJZCI6InRyYWNlOnJvb20tZml4dHVyZS0wMS1maXh0dXJlIiwicm9vbUFnZW50TWlzc2lvbklkIjoibWlzc2lvbi1maXh0dXJlLTAxIiwicm9vbUFnZW50T3BlcmF0b3JSZXF1ZXN0IjoiU3ludGhldGljIG9wZXJhdG9yIHJlcXVlc3QuIiwic2VsZWN0ZWRUb29sSWRzIjpbInJvb206aW5zcGVjdCJdfQ=="
}
]
},
"scheduleToCloseTimeout": {
"seconds": "315360000"
},
"scheduleToStartTimeout": {
"seconds": "315360000"
},
"startToCloseTimeout": {
"seconds": "86400"
},
"heartbeatTimeout": {
"seconds": "15"
},
"workflowTaskCompletedEventId": "3",
"retryPolicy": {
"initialInterval": {
"seconds": "1"
},
"backoffCoefficient": 2,
"maximumInterval": {
"seconds": "100"
},
"maximumAttempts": 5
}
}
},
{
"eventId": "6",
"eventTime": {
"seconds": "2424028730",
"nanos": 845000000
},
"eventType": "EVENT_TYPE_ACTIVITY_TASK_STARTED",
"activityTaskStartedEventAttributes": {
"scheduledEventId": "5",
"identity": "25805@cursor",
"attempt": 1
}
},
{
"eventId": "7",
"eventTime": {
"seconds": "2424028730",
"nanos": 847000000
},
"eventType": "EVENT_TYPE_ACTIVITY_TASK_COMPLETED",
"activityTaskCompletedEventAttributes": {
"result": {
"payloads": [
{
"metadata": {
"encoding": "anNvbi9wbGFpbg=="
},
"data": "eyJ3b3JrZmxvd0lkIjoic3BhY2UtdHVybjp0cmFjZTpyb29tLWZpeHR1cmUtMDEtZml4dHVyZSIsInJvb21JZCI6InJvb20tZml4dHVyZS0wMSIsInBhbmVJZCI6InBhbmU6cm9vbS1maXh0dXJlLTAxLWZpeHR1cmUiLCJ0cmFjZUlkIjoidHJhY2U6cm9vbS1maXh0dXJlLTAxLWZpeHR1cmUiLCJzdGF0dXMiOiJDT01QTEVURUQiLCJtZXNzYWdlIjoiU3ludGhldGljIHR1cm4gY29tcGxldGVkLiIsInJvb21BZ2VudE91dGNvbWUiOnsic3RhdHVzIjoiVkVSSUZJRUQiLCJleGVjdXRlZEFjdGlvbkNvdW50IjoxLCJzdGF0dXNSZWFzb24iOiJTeW50aGV0aWMgcm9vbS1hZ2VudCB0dXJuIHZlcmlmaWVkLiJ9fQ=="
}
]
},
"scheduledEventId": "5",
"startedEventId": "6",
"identity": "25805@cursor"
}
},
{
"eventId": "8",
"eventTime": {
"seconds": "2424028730",
"nanos": 847000000
},
"eventType": "EVENT_TYPE_WORKFLOW_TASK_SCHEDULED",
"workflowTaskScheduledEventAttributes": {
"taskQueue": {
"name": "temporal-sandbox-tests",
"kind": "TASK_QUEUE_KIND_NORMAL"
},
"startToCloseTimeout": {
"seconds": "10"
},
"attempt": 1
}
},
{
"eventId": "9",
"eventTime": {
"seconds": "2424028730",
"nanos": 847000000
},
"eventType": "EVENT_TYPE_WORKFLOW_TASK_STARTED",
"workflowTaskStartedEventAttributes": {
"scheduledEventId": "8",
"identity": "25805@cursor",
"historySizeBytes": "2280"
}
},
{
"eventId": "10",
"eventTime": {
"seconds": "2424028730",
"nanos": 852000000
},
"eventType": "EVENT_TYPE_WORKFLOW_TASK_COMPLETED",
"workflowTaskCompletedEventAttributes": {
"scheduledEventId": "8",
"identity": "25805@cursor",
"binaryChecksum": "@temporalio/worker@1.24.0+9a365569848f3b7720675da78257957763d6c9652a5349034772dfe58fee989c",
"sdkMetadata": {},
"meteringMetadata": {}
}
},
{
"eventId": "11",
"eventTime": {
"seconds": "2424028730",
"nanos": 852000000
},
"eventType": "EVENT_TYPE_WORKFLOW_EXECUTION_COMPLETED",
"workflowExecutionCompletedEventAttributes": {
"result": {
"payloads": [
{
"metadata": {
"encoding": "anNvbi9wbGFpbg=="
},
"data": "eyJ3b3JrZmxvd0lkIjoic3BhY2UtdHVybjp0cmFjZTpyb29tLWZpeHR1cmUtMDEtZml4dHVyZSIsInJvb21JZCI6InJvb20tZml4dHVyZS0wMSIsInBhbmVJZCI6InBhbmU6cm9vbS1maXh0dXJlLTAxLWZpeHR1cmUiLCJ0cmFjZUlkIjoidHJhY2U6cm9vbS1maXh0dXJlLTAxLWZpeHR1cmUiLCJzdGF0dXMiOiJDT01QTEVURUQiLCJtZXNzYWdlIjoiU3ludGhldGljIHR1cm4gY29tcGxldGVkLiIsInJvb21BZ2VudE91dGNvbWUiOnsic3RhdHVzIjoiVkVSSUZJRUQiLCJleGVjdXRlZEFjdGlvbkNvdW50IjoxLCJzdGF0dXNSZWFzb24iOiJTeW50aGV0aWMgcm9vbS1hZ2VudCB0dXJuIHZlcmlmaWVkLiJ9fQ=="
}
]
},
"workflowTaskCompletedEventId": "9"
}
}
]
}
}
Loading