[$600] Export worker and workflow spans over OTLP: one trace from API request to every Temporal activity

Context

The API can already export its request spans over OTLP/HTTP JSON. The exporter has no dependencies, is off by default and is switched on with SPACE_OTEL_EXPORTER_OTLP_ENDPOINT (apps/api/src/otlp-span-exporter.ts). When the API starts a Temporal workflow it passes a W3C traceparent in the workflow input and in the Temporal memo (apps/api/src/turns.ts, workflowCorrelationMemo in apps/api/src/trace-context.ts). The worker reads it for its structured activity logs (apps/worker/src/activity-observability.ts).

But the trace stops at the API. Workers emit no spans, so a slow or failing Room Agent mission, chat turn or memory repair can only be followed by joining log lines by hand. docs/quality/observability.md lists this under "Not covered": "Span export covers API requests only" and "Background continuations start a new trace". In KNOWN_LIMITATIONS.md this is OBS-001 / OBS-007: "worker/host spans" are still open.

Goal

With SPACE_OTEL_EXPORTER_OTLP_ENDPOINT set, one API request that starts work on Temporal produces one connected trace: the API SERVER span, then the workflow execution, then every activity attempt. The trace shows in any OTLP backend (Jaeger, Tempo, Honeycomb, ...) without extra glue.

Acceptance criteria

Every Temporal worker process in apps/worker (worker.ts, memory-maintenance-worker.ts, memory-mutation-worker.ts) exports spans to <endpoint>/v1/traces when SPACE_OTEL_EXPORTER_OTLP_ENDPOINT is set, and does nothing when it is not.
The exporter has the same guarantees as the API exporter: a bounded queue that drops spans and counts the drops, a timeout on each POST, and a slow or failing collector never delays or fails workflow or activity work. The code is shared with the API, not copied. For example, move it into a workspace package and keep the API behavior and its tests (apps/api/tests/otlp-span-exporter.test.ts) unchanged.
Span tree for a workflow started by the API (codexAppServerTurnWorkflow, roomAgentSupervisorWorkflow missions, roomAgentActionWorkflow, dotGoalWorkflow, memory workflows):
- a workflow span whose parent is the span id from the propagated traceparent;
- one span per activity attempt, child of the workflow span, with the attempt number, outcome and the existing reasonCode / category;
- child workflows (memoryMaintenanceWorkflow → memoryConsolidationWorkflow) nest under their parent.
Determinism: workflow code does no I/O for tracing. Spans leave the workflow sandbox only through Temporal interceptors and/or sinks, and replay emits no duplicate spans (for example sinks with callDuringReplay: false). A test proves that replaying a history emits no spans.
A continue-as-new run, and a background continuation (today a new trace:bg-<id>, see apps/worker/src/room-background-continuation.ts), carry an OTLP span link to the run or trace they continue.
Attributes come from an allowlist: ids (trace, workflow, run, room, pane, mission), workflow/activity type, attempt, status, stable error code/category, durations. Prompts, model output, file paths, URLs, error messages and any user text are never exported. A test enforces the allowlist.
Integration test: a fake OTLP HTTP collector (a local http.createServer) plus a Temporal test environment (@temporalio/testing, see [$400] Test every Temporal workflow on a real time-skipping test server and gate replay determinism in CI #60) runs a real workflow with stub activities. The test asserts trace id continuity and the parent/child structure. No Docker or AI provider is needed.
No new tracing dependency (OpenTelemetry SDK, @temporalio/interceptors-opentelemetry) unless it is proposed and agreed in this issue first. The existing exporter shows that the JSON protocol does not need one.
docs/quality/observability.md and docs/configuration-reference.md describe the new coverage. The "Span export covers API requests only" bullet is updated.

How to test

ELECTRON_SKIP_BINARY_DOWNLOAD=1 npm ci
npm run check
npm test -w @space/worker
npm test -w @space/api -- otlp

Manual check: run Jaeger all-in-one (docker run -p 16686:16686 -p 4318:4318 jaegertracing/all-in-one) and point SPACE_OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318 at it for the API and the worker. Start a turn and confirm a single trace from the HTTP request down to the activities. Put a screenshot of the trace in the PR.

Out of scope

Spans from the CLI host (packages/cli-host) and the Browser Host.
Metrics, logs export and sampling configuration (export everything that is recorded, as the API does).
New database columns for trace ids on mission/task rows (a separate open item in observability.md).

Pointers

apps/api/src/otlp-span-exporter.ts, apps/api/src/observability.ts (how the API records SERVER spans)
apps/api/src/trace-context.ts (parseTraceparent, formatTraceparent, workflowCorrelationMemo), apps/api/src/turns.ts, apps/api/src/room-agent.ts
apps/worker/src/worker.ts (where the activity interceptors are registered), apps/worker/src/activity-observability.ts, apps/worker/src/turn-observability.ts
packages/contracts/src/observability.ts
Temporal TypeScript SDK docs: interceptors (WorkflowInboundCallsInterceptor, ActivityInboundCallsInterceptor) and sinks
Before opening a PR, run the checks in CONTRIBUTING.md.

💎 Bounty: $600 USD

This issue carries a cash bounty of $600 USD, paid by the SpaceApp maintainers.

How it works

Comment on this issue that you are working on it (optional, but it helps avoid duplicate work).
Open a pull request that closes this issue and meets every acceptance criterion above, with tests.
The bounty goes to the author of the first pull request that is merged as the solution. If several PRs arrive, we review them in the order they were opened.
We pay within 7 days of the merge, by PayPal, Wise or bank transfer; we agree the method with you in this issue.

Terms

One bounty per issue; the amount is never lowered once posted.
AI-assisted work is welcome when you have reviewed and tested it yourself. Untested or generated-only PRs are closed without review.
Contributions are accepted under the repository's license (see LICENSE).
The "Verify export provenance" CI step fails on outside pull requests by design (this repository is exported from the maintainers' source); reviewers ignore that step. Every other check must pass.
You are responsible for any taxes on the payment. We cannot pay where sanctions forbid it.

If SpaceApp is useful to you, a ⭐ on the repository helps the project grow. It is not required for the bounty.

/attempt

Hi, I’m Jason Ji. With an implementation now marked /attempt, would a separately funded $150 diagnostic of one trace path be useful before merging #61?

I would check one API → workflow → activity path using synthetic inputs, including trace continuity and replay suppression, subject to an agreed setup. Two hours of total work includes setup, investigation, independent review and a written handoff. The outcome may be inconclusive; the report would record exact attempts/results, any verified blockers and next steps.

This is a separate proposed milestone. The full-issue bounty and its acceptance criteria remain as posted. Before work, we would agree the revision/path, written acceptance and correction/refund terms, and confirm funding. I use disclosed AI assistance and review/test the results.

Relevant merged regression work: my Apache Superset fix isolates database sessions for concurrent MCP calls: apache/superset#42629

/attempt #61

Plan: I will start by reproducing the reported issue locally, then implement a focused fix with regression coverage, run the repo's test suite, and submit a PR with /claim #61.

(Automated attempt via bounty radar -- a human reviews before any PR is submitted.)

I’m working on the complete scope of this issue: a shared bounded OTLP exporter, workflow and activity-attempt spans in all three workers, parent/child propagation and continuation links, replay suppression, allowlisted attributes, and real Temporal/fake-collector integration tests. I’ll include the executed validation commands and Jaeger trace evidence.

I have verified that the pinned 1.24.0 time-skipping test environment starts and tears down in my isolated macOS checkout.

@oll4com

hey @oll4com, i see three attempts on #61. is the full $600 USD budget confirmed, and could you reserve the task for another contributor while respecting work already underway?

i'd use Codex agents for implementation, tests and review. would that meet your review requirement, or is separate human technical review needed?

before starting, could we agree a review/merge window and what happens if review stalls? can you also confirm PayPal or bank payment terms for a US contributor, any fees, and whether 'within 7 days' means sent or received?

Thanks for the interest. To answer the questions here: the bounty is $600 USD (raised from $150 on 2026-10-16); the terms in the issue are current. We do not assign or reserve issues: the first pull request that meets every acceptance criterion and is merged receives the bounty, and we review pull requests in the order they were opened. Until #66 is merged, the "Verify export provenance" CI step fails on pull requests from forks; reviewers ignore that step. We do not fund separate diagnostics or partial work; only the merged solution receives the bounty.

I'm working on the full tracing scope: shared exporter, all three worker entrypoints, workflow/activity parentage, continuation links, and replay/privacy tests. I'll include a real API-to-Temporal trace screenshot with synthetic activities and the validation results. PayPal is my preferred payment method: pythonyx135793@gmail.com.

Working on this! Implementing shared bounded OTLP exporter, Temporal workflow/activity interceptors & sinks, traceparent propagation, span links, attribute allowlist, replay suppression, and integration tests with synthetic collector.

I have submitted the complete implementation in PR #82 covering all acceptance criteria:

Shared OTLP/HTTP Exporter: Bounded queue shared via @space/runtime\ with zero new dependencies.
Unified Span Tree: API request (SERVER) -> Workflow (INTERNAL) -> Activity attempts (INTERNAL), with child workflows properly nested.
Determinism Guaranteed: Workflow spans exported exclusively via Temporal sinks with \callDuringReplay: false, verified to emit 0 duplicate spans on replay.
Span Links: Attached on \continue-as-new\ and
oom-background-continuation.ts.
Strict Allowlist: Safe IDs and codes only; zero leaks of prompts, URLs, or file paths.
Full Test Suite & Docs: Hermetic integration suite with fake OTLP HTTP collector passing 100%, and docs updated.

PR: #82

I have opened PR #98 to implement end-to-end OTLP worker and workflow span export across all three Temporal worker processes (space-worker, space-memory-maintenance-worker, and space-memory-mutation-worker), including deterministic workflow sandbox interceptors/sinks (callDuringReplay: false), child workflow nesting, span links for continueAsNew and background continuations ( race:bg-), strict attribute allowlist sanitization, documentation updates, and @temporalio/testing + fake OTLP HTTP collector integration and replay tests.

/attempt #61

fix: address CodeRabbit review comments on all bounty PRs

fix: address CodeRabbit review comments on PR oll4com#304 (oll4com#61 …

fix: address CodeRabbit docstring coverage for PR oll4com#304 (oll4co…

feat(telemetry): export worker and workflow spans over OTLP (fixes ol…

fix(tracing): propagate context to local activities and normalize end…

feat(observability): export worker and workflow spans over OTLP (oll4…

hey opened #352 for this. hooked up the full trace tree from api requests into temporal workflow spans, child workflows, and individual activity retry attempts over otlp json. workflow determinism is preserved via temporal sinks with callDuringReplay false, and added both interceptor replay unit tests plus the time-skipping integration test verifying the connected span hierarchy against an in-memory collector.

feat(telemetry): implement OTLP workflow tracer, replay suppression, …

fix(observability): reject non-positive batchSize at exporter creation (

test(observability): verify workflow header propagation without args …

feat(telemetry): export worker and workflow spans over OTLP (fixes ol…

The SpaceApp bounty programme has ended, so this bounty is withdrawn and we are closing the issue.
Thank you to everyone who took an interest in it and spent time on it.

Navigation Menu

Repository navigation

[$600] Export worker and workflow spans over OTLP: one trace from API request to every Temporal activity #61

Context

Goal

Acceptance criteria

How to test

Out of scope

Pointers

💎 Bounty: $600 USD

75 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

[$600] Export worker and workflow spans over OTLP: one trace from API request to every Temporal activity #61

Description

Context

Goal

Acceptance criteria

How to test

Out of scope

Pointers

💎 Bounty: $600 USD

Activity

t3ratech commented on Oct 16, 2026

goingforstudying-ctrl commented on Oct 16, 2026

hieutran0413 commented on Oct 16, 2026

soundcake commented on Oct 16, 2026

jbrohunt commented on Oct 16, 2026

oll4com commented on Oct 16, 2026

pythonyx135793 commented on Oct 16, 2026

jdjioe5-cpu commented on Oct 16, 2026

jdjioe5-cpu commented on Oct 16, 2026

gelismarvel commented on Oct 16, 2026

75 remaining items

goddrivenbullet commented on Oct 24, 2026

rishiiicreates commented on Oct 28, 2026

oll4com commented on Oct 30, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions