Skip to content

feat (observability): export Temporal worker and workflow spans over OTLP (#61) - #361

Closed
dayton536 wants to merge 1 commit into
oll4com:mainfrom
dayton536:cursor/otlp-workflow-spans-c783
Closed

dayton536 wants to merge 1 commit into
oll4com:mainfrom
dayton536:cursor/otlp-workflow-spans-c783

Conversation

@dayton536

@dayton536 dayton536 commented Oct 29, 2026 •

Copy link
Copy Markdown

Fixes #61

Problem and user impact

When SPACE_OTEL_EXPORTER_OTLP_ENDPOINT is configured, the API exports HTTP request spans over OTLP/HTTP JSON and propagates a W3C traceparent into Temporal workflow inputs and memos, but Temporal workers emit no spans. Operators debugging a slow or failing Room Agent mission, chat turn, or memory workflow had to manually join log lines because traces stopped at the API boundary.

Change

  • Wired the shared bounded OTLP HTTP JSON span exporter in @space/runtime into space-worker, space-memory-maintenance-worker, and space-memory-mutation-worker whenever SPACE_OTEL_EXPORTER_OTLP_ENDPOINT is set (and no-op when unset), without adding any external OpenTelemetry SDK dependency.
  • Added deterministic Temporal workflow and activity tracing via WorkflowInboundCallsInterceptor, WorkflowOutboundCallsInterceptor, ActivityInboundCallsInterceptor, and a replay-safe workflow sink (callDuringReplay: false) so workflow code performs zero tracing I/O and history replay emits no duplicate spans.
  • Connected workflow spans as children of the propagated API traceparent span ID, nested child workflows (memoryMaintenanceWorkflow -> memoryConsolidationWorkflow) under their parent workflow span, emitted per-attempt activity child spans with attempt number, outcome, reasonCode, and category, and attached OTLP span links across continueAsNew and background continuations (trace:bg-<id>).
  • Enforced a strict attribute allowlist so prompts, model outputs, file paths, URLs, raw error messages, and user text are never exported, and updated docs/quality/observability.md, docs/configuration-reference.md, and docs/quality/KNOWN_LIMITATIONS.md.

Verification

  • Relevant automated tests pass (npm run check, npm test -w @space/worker — 244 tests including the end-to-end fake OTLP collector + Temporal time-skipping integration suite and replay silence check, npm test -w @space/api -- otlp).
  • Linux, macOS, and/or Windows coverage is listed (cross-platform Node/Temporal worker & API test suites).
  • UI changes include real browser console/network evidence (N/A — backend/worker OTLP telemetry change only).
  • Every commit has a Signed-off-by: line (Developer Certificate of Origin 1.1), and the contribution is licensed under the MIT License (see CONTRIBUTING.md).
  • No credentials, tokens, personal memory, customer data, private paths, or runtime state are included.
  • Documentation and changelog are updated when behavior changes (docs/quality/observability.md, docs/configuration-reference.md, docs/quality/KNOWN_LIMITATIONS.md).
  • Security, migration, resource, and rollback effects are described (strict attribute allowlist strips sensitive fields; bounded queue with drop counting and POST timeout ensures collector issues never block workflows; disabled by default when SPACE_OTEL_EXPORTER_OTLP_ENDPOINT is unset).

Share the bounded OTLP exporter with every worker so one API request
is one trace through the workflow and each activity attempt. Workflow
spans leave the sandbox only through interceptors and a replay-silent
sink, with span links for continue-as-new and background continuations.

Signed-off-by: Dayton Allen <daytonballen@gmail.com>
Co-authored-by: dayton536 <dayton536@users.noreply.github.com>
@coderabbitai

coderabbitai Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review.

Recent review info
Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: f154ae88-4c44-42b2-b341-4934f31d5ec0
Commits

Reviewing files that changed from the base of the PR and between f20c00a and 7d59c9d.

Files ignored due to path filters: 1
  • package-lock.json is excluded by !**/package-lock.json
Files selected for processing: 24
  • apps/api/src/dot-goals.ts
  • apps/api/src/memory-consolidation-coordinator.ts
  • apps/api/src/memory-mutation-coordinator.ts
  • apps/api/src/room-agent.ts
  • apps/api/src/trace-context.ts
  • apps/api/tests/otlp-span-exporter.test.ts
  • apps/api/tests/trace-correlation.test.ts
  • apps/worker/package.json
  • apps/worker/src/activity-observability.ts
  • apps/worker/src/memory-maintenance-worker.ts
  • apps/worker/src/memory-mutation-worker.ts
  • apps/worker/src/otlp-span-context.ts
  • apps/worker/src/otlp-workflow-interceptors.ts
  • apps/worker/src/worker-otlp.ts
  • apps/worker/src/worker.ts
  • apps/worker/tests/activity-observability.test.ts
  • apps/worker/tests/otlp-fixtures/workflows.ts
  • apps/worker/tests/otlp-span-context.test.ts
  • apps/worker/tests/otlp-workflow-spans.integration.test.ts
  • docs/configuration-reference.md
  • docs/quality/KNOWN_LIMITATIONS.md
  • docs/quality/observability.md
  • docs/quality/review-findings.json
  • packages/runtime/src/otlp-span-exporter.ts

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

Walkthrough

The change propagates API trace context through Temporal workflow memos and headers. Workers export workflow and activity spans, including parent and continuation links. The exporter filters span attributes and links, and workflow span export is suppressed during replay.

Changes: API-to-workflow trace propagation

Layer / File(s) Summary
Request context and workflow memos
apps/api/src/dot-goals.ts, apps/api/src/memory-consolidation-coordinator.ts, apps/api/src/memory-mutation-coordinator.ts, apps/api/src/room-agent.ts, apps/api/src/trace-context.ts, apps/api/tests/trace-correlation.test.ts
API workflow starters pass trace context and correlation fields in workflow memos. traceparentForTraceId returns the registered request traceparent, and tests check the request parent span and Room Agent memo fields.

Workflow and activity span export

Layer / File(s) Summary
Workflow span context and link contracts
apps/worker/src/otlp-span-context.ts, packages/runtime/src/otlp-span-exporter.ts, apps/worker/tests/otlp-span-context.test.ts, apps/worker/tests/otlp-fixtures/workflows.ts
Shared helpers validate identifiers and attributes, encode Temporal headers, and resolve workflow parent and link relationships. The OTLP exporter accepts valid span links. Unit tests cover context resolution, filtering, and span conversion.
Worker workflow and activity export
apps/worker/src/otlp-workflow-interceptors.ts, apps/worker/src/worker-otlp.ts, apps/worker/src/activity-observability.ts, apps/worker/src/worker.ts, apps/worker/src/memory-maintenance-worker.ts, apps/worker/src/memory-mutation-worker.ts, apps/worker/tests/activity-observability.test.ts
Workflow interceptors propagate trace headers and export workflow spans. Worker options configure workflow sinks and activity interceptors. Activity spans use a decoded workflow carrier as parent when available and apply the attribute allowlist.
Integration coverage and tracing documentation
apps/worker/tests/otlp-workflow-spans.integration.test.ts, apps/worker/package.json, apps/api/tests/otlp-span-exporter.test.ts, docs/configuration-reference.md, docs/quality/observability.md, docs/quality/KNOWN_LIMITATIONS.md, docs/quality/review-findings.json
Integration tests check span relationships, retries, continuation links, sensitive-data omission, and replay silence. Documentation describes export configuration, span behavior, and remaining observability gaps.

Priority: Normal

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant API
  participant TemporalWorkflow
  participant TemporalActivity
  participant OTLPExporter
  API->>TemporalWorkflow: Start with trace correlation memo
  TemporalWorkflow->>TemporalActivity: Propagate OTLP context in activity header
  TemporalActivity->>OTLPExporter: Export activity-attempt span
  TemporalWorkflow->>OTLPExporter: Export workflow span through sink
Loading

Possibly related issues

Possibly related PRs

  • oll4com/spaceapp#349: Changes the same API trace propagation, worker span export, activity parenting, and continuation-link paths.
  • oll4com/spaceapp#334: Adds workflow span propagation through Temporal headers, activities, child workflows, and continue-as-new runs.
  • oll4com/spaceapp#319: Changes API-to-Temporal trace propagation and worker workflow and activity spans.
Merge Risk: Minimal · up to 7d59c

No identified tracing defect blocks merging after normal checks.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 62.50% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documenta… Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title says the PR adds OTLP export of Temporal worker and workflow spans. The raw summary supports this. The worker now exports workflow spans and activity-attempt spans over OTLP, with span links…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title says the PR adds OTLP export of Temporal worker and workflow spans. The raw summary supports this. The worker now exports workflow spans and activity-attempt spans over OTLP, with span links for continue-as-new and background continuations, and the title names that main change. The title also includes the issue suffix (#61), which conflicts with the PR objectives that cite PR #361. That mismatch does not change the assessment of the title's content.

Full details: Docstring Coverage

Explanation

Docstring coverage is 62.50% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 40 functions across 19 files. (5 skipped: 5 unsupported.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@dayton536 dayton536 changed the title feat: export Temporal workflow and activity spans over OTLP feat (observability): export Temporal worker and workflow spans over OTLP (#61) Oct 29, 2026
@dayton536

Copy link
Copy Markdown
Author

Fixes #61

Problem and user impact

When SPACE_OTEL_EXPORTER_OTLP_ENDPOINT is configured, the API exports HTTP request spans over OTLP/HTTP JSON and propagates a W3C traceparent into Temporal workflow inputs and memos, but Temporal workers emit no spans. Operators debugging a slow or failing Room Agent mission, chat turn, or memory workflow had to manually join log lines because traces stopped at the API boundary.

Change

  • Wired the shared bounded OTLP HTTP JSON span exporter in @space/runtime into space-worker, space-memory-maintenance-worker, and space-memory-mutation-worker whenever SPACE_OTEL_EXPORTER_OTLP_ENDPOINT is set (and no-op when unset), without adding any external OpenTelemetry SDK dependency.
  • Added deterministic Temporal workflow and activity tracing via WorkflowInboundCallsInterceptor, WorkflowOutboundCallsInterceptor, ActivityInboundCallsInterceptor, and a replay-safe workflow sink (callDuringReplay: false) so workflow code performs zero tracing I/O and history replay emits no duplicate spans.
  • Connected workflow spans as children of the propagated API traceparent span ID, nested child workflows (memoryMaintenanceWorkflow -> memoryConsolidationWorkflow) under their parent workflow span, emitted per-attempt activity child spans with attempt number, outcome, reasonCode, and category, and attached OTLP span links across continueAsNew and background continuations (trace:bg-<id>).
  • Enforced a strict attribute allowlist so prompts, model outputs, file paths, URLs, raw error messages, and user text are never exported, and updated docs/quality/observability.md, docs/configuration-reference.md, and docs/quality/KNOWN_LIMITATIONS.md.

Verification

  • Relevant automated tests pass (npm run check, npm test -w @space/worker — 244 tests including the end-to-end fake OTLP collector + Temporal time-skipping integration suite and replay silence check, npm test -w @space/api -- otlp).
  • Linux, macOS, and/or Windows coverage is listed (cross-platform Node/Temporal worker & API test suites).
  • UI changes include real browser console/network evidence (N/A — backend/worker OTLP telemetry change only).
  • Every commit has a Signed-off-by: line (Developer Certificate of Origin 1.1), and the contribution is licensed under the MIT License (see CONTRIBUTING.md).
  • No credentials, tokens, personal memory, customer data, private paths, or runtime state are included.
  • Documentation and changelog are updated when behavior changes (docs/quality/observability.md, docs/configuration-reference.md, docs/quality/KNOWN_LIMITATIONS.md).
  • Security, migration, resource, and rollback effects are described (strict attribute allowlist strips sensitive fields; bounded queue with drop counting and POST timeout ensures collector issues never block workflows; disabled by default when SPACE_OTEL_EXPORTER_OTLP_ENDPOINT is unset).

@oll4com

oll4com commented Oct 29, 2026

Copy link
Copy Markdown
Owner

Thank you for the time you put into this pull request.
The SpaceApp bounty programme is paused, so pull requests for the bounty issues are not being reviewed or merged for now.
We can't promise any payment or any dates, so please don't put more time into this branch on that basis.
Thanks again for the work.

@oll4com

oll4com commented Oct 30, 2026

Copy link
Copy Markdown
Owner

Thank you for the time and care you put into this pull request.
We have decided to end the SpaceApp bounty programme, so pull requests made for the bounty issues will not be reviewed, merged or paid, and we are closing this one.
We are sorry for the work this affects, and thank you again for your interest in SpaceApp.

@oll4com oll4com closed this Oct 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[$600] Export worker and workflow spans over OTLP: one trace from API request to every Temporal activity

3 participants