Skip to content

Export Temporal workflow and activity spans over OTLP - #3

Open
dayton536 wants to merge 2 commits into
mainfrom
cursor/otlp-workflow-spans-c783
Open

dayton536 wants to merge 2 commits into
mainfrom
cursor/otlp-workflow-spans-c783

Conversation

@dayton536

Copy link
Copy Markdown
Owner

Problem and user impact

An API request that starts Temporal work could export only the API SERVER span. Workers logged activity attempts, but a slow or failing Room Agent mission, chat turn, or memory repair could not be followed as one trace in an OTLP backend. Implements oll4com/spaceapp#61.

This branch is cut from oll4com/spaceapp main (f20c00a). It is not for merging into dayton536:main, which still carries an unrelated Podman merge.

Change

When SPACE_OTEL_EXPORTER_OTLP_ENDPOINT is set, space-worker, space-memory-maintenance-worker, and space-memory-mutation-worker export spans to <endpoint>/v1/traces through the existing bounded exporter in @space/runtime (queue drops, POST timeout, collector failures never fail the work). When the variable is unset, those processes export nothing and do not load the workflow interceptor.

The span tree is the API SERVER span, then an INTERNAL workflow span parented to the propagated traceparent, then one INTERNAL span per activity attempt (attempt, outcome, reasonCode, category). Child workflows nest under the parent workflow span. Continue-as-new and background continuations (trace:bg-<id>) add an OTLP span link to the run or trace they continue.

Workflow code does no tracing I/O. Spans leave the sandbox through Temporal interceptors and a sink with callDuringReplay: false. A patched marker keeps histories from before export was enabled deterministic. Attributes are allowlisted; prompts, model output, paths, URLs, and error messages are dropped. No OpenTelemetry SDK was added. @temporalio/testing is a worker devDependency for the hermetic test only.

Verification

  • Relevant automated tests pass.
  • Linux coverage: npm run check, npm test -w @space/worker (244 tests), and npm test -w @space/api -- otlp (plus trace-correlation and memory coordinator tests) on Linux. macOS and Windows were not run.
  • UI changes include real browser console/network evidence. No UI change.
  • Every commit has a Signed-off-by: line (Developer Certificate of Origin 1.1), and the contribution is licensed under the MIT License (see CONTRIBUTING.md).
  • No credentials, tokens, personal memory, customer data, private paths, or runtime state are included.
  • Documentation and changelog are updated when behavior changes. Observability, configuration reference, and OBS-001 / OBS-007 are updated. No changelog entry (behavior is opt-in and off by default).
  • Security, migration, resource, and rollback effects are described. Export stays off unless the endpoint is set. Turning the endpoint off while a workflow that already recorded the tracing marker is still running can change commands; leave it set until those runs finish. A slow collector cannot block workflow or activity work.

The integration test uses a local http.createServer collector and @temporalio/testing's time-skipping server. It checks trace continuity, parent/child spans, activity attempt attributes, continuation links, and that replaying a history emits zero spans. A Jaeger screenshot was not captured here.

Open in Web Open in Cursor 

oll4com and others added 2 commits October 28, 2026 19:01
…ble 4f01583e71 + d077b0ff0c; Node 22.23.4 base for CVE-2026-111497; acceptance for the tested tree with the Windows first-agent Zen exception, operator decision 54b65a84)
Share the bounded OTLP exporter with every worker so one API request
is one trace through the workflow and each activity attempt. Workflow
spans leave the sandbox only through interceptors and a replay-silent
sink, with span links for continue-as-new and background continuations.

Signed-off-by: Dayton Allen <daytonballen@gmail.com>
Co-authored-by: dayton536 <dayton536@users.noreply.github.com>
}
/** Repeats of one kind of message share a key: digits and dates fold, so "Daily backup missing for 2026-10-25" is one card. */
export function monitoringGuardianKey(message: string): string {
return message.split("\n")[0]!.replace(/\[[^\]]*\]/g, " ").replace(/\d+/g, "N").replace(/[^\p{L}\p{N} _:-]+/gu, " ")
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants