Skip to content

feat(telemetry): keep returned command errors out of failed spans - #82

Merged
yordis merged 1 commit into
mainfrom
yordis/feat-configurable-otel-status
Apr 16, 2026
Merged

yordis merged 1 commit into
mainfrom
yordis/feat-configurable-otel-status

Conversation

@yordis

@yordis yordis commented Apr 15, 2026

Copy link
Copy Markdown
Member
  • Avoids forcing returned command errors into OpenTelemetry error status when applications need different reporting semantics.
  • Lets applications align span status with the OpenTelemetry status API while keeping exception spans as true failures.
  • Preserves the current telemetry behavior by default while opening a narrow override for returned :stop errors.

@cursor

cursor Bot commented Apr 15, 2026 •

Copy link
Copy Markdown

PR Summary

Medium Risk
Changes span status semantics for returned errors and introduces a user-provided callback, which can affect observability and alerting if misconfigured, though defaults preserve current behavior.

Overview
Adds an opt-in error_status callback for application dispatch and aggregate execute tracing so returned :stop errors can map to :unset/:ok/:error (or nil) instead of always forcing error spans.

Wires the validated config through Commanded.OpenTelemetry.setup/1 into the aggregate/application telemetry handlers, centralizes status-setting in Helpers.set_error_status/7 (including a warning telemetry event for unknown status codes), and documents + tests the new behavior (including preserving exception span semantics).

Reviewed by Cursor Bugbot for commit b9e170d. Bugbot is set up for automated code reviews on this repo. Configure here.

@coderabbitai

coderabbitai Bot commented Apr 15, 2026 •

Copy link
Copy Markdown

Warning

Rate limit exceeded

@yordis has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 43 minutes and 34 seconds before requesting another review.

Your organization is not enrolled in usage-based pricing. Contact your admin to enable usage-based pricing to continue reviews beyond the rate limit, or try again in 43 minutes and 34 seconds.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: f1df7618-2d85-4f00-8687-83361ca62c98

📥 Commits

Reviewing files that changed from the base of the PR and between d06926c and b9e170d.

📒 Files selected for processing (7)
  • guides/howtos/setting-up-opentelemetry-tracing.md
  • lib/commanded/opentelemetry.ex
  • lib/commanded/opentelemetry/aggregate.ex
  • lib/commanded/opentelemetry/application.ex
  • lib/commanded/opentelemetry/helpers.ex
  • test/opentelemetry/aggregate_test.exs
  • test/opentelemetry/application_test.exs

Walkthrough

This PR introduces configurable error status handling for OpenTelemetry spans in Commanded. It adds an error_status callback mechanism to the OpenTelemetry setup configuration, allowing users to customize how span status is computed for returned :stop errors in both aggregate and application instrumentation. Changes include new callback type definitions, updated setup functions to accept and propagate configuration, a new helper function to manage error status computation, tests validating callback behavior, and documentation explaining the feature.

Changes

Cohort / File(s) Summary
Documentation
guides/howtos/setting-up-opentelemetry-tracing.md
Added "Override Returned Error Status" section documenting new error_status callback configuration for both application: and aggregate: options, including callback signature, allowed return values, and interaction with exception handling.
Core OpenTelemetry Configuration & Setup
lib/commanded/opentelemetry.ex
Added public error_status_callback/0 type, extended :aggregate and :application option schemas to accept error_status configuration, and updated setup/1 to pass validated config to Aggregate.setup(config) and OTelApplication.setup(config). Includes usage examples.
Telemetry Handler Implementation
lib/commanded/opentelemetry/aggregate.ex, lib/commanded/opentelemetry/application.ex
Updated setup/0 to setup(config \\ []) in both modules, modified telemetry handlers to capture and use config and measurements parameters, and replaced direct span status setting with Helpers.set_error_status/7 for delegated error handling. Introduced event_name constant for stop event telemetry.
Error Status Helper Logic
lib/commanded/opentelemetry/helpers.ex
Added new public set_error_status/7 function that reads config[:error_status], invokes the callback if present, and delegates to private apply_error_status/4 to compute and apply span status. Handles :error, :unset, :ok return values and emits warnings for unknown status codes.
Test Coverage
test/opentelemetry/aggregate_test.exs, test/opentelemetry/application_test.exs
Added test cases for error_status callback integration: aggregate tests verify :unset and :error status computation; application tests verify callback invocation on dispatch :stop events, exception handling paths, and status overrides with returned :stop errors.

Sequence Diagram

sequenceDiagram
    participant Code as Commanded Code
    participant Telemetry as Telemetry System
    participant Handler as Aggregate/Application Handler
    participant Helper as Helpers.set_error_status
    participant Callback as error_status Callback
    participant Span as OpenTelemetry Span

    Code->>Telemetry: Emit :stop event with error metadata
    Telemetry->>Handler: Invoke handler (measurements, meta, config)
    Handler->>Helper: set_error_status(ctx, error, event_name, measurements, meta, config, tracer_id)
    Helper->>Helper: Check config[:error_status]
    alt Callback configured
        Helper->>Callback: fun(event_name, measurements, meta, config)
        Callback-->>Helper: Return status code (:error, :unset, :ok, or nil)
        Helper->>Helper: apply_error_status(status_code)
    else No callback
        Helper->>Helper: Use default :error status
    end
    Helper->>Span: Set span status (with error description or unset)
Loading

Estimated Code Review Effort

🎯 3 (Moderate) | ⏱️ ~30 minutes

Possibly Related PRs

Poem

A rabbit hops through error logs so keen,
With callbacks dancing in between,
Status codes now bend to our command,
Span errors shaped with a deft paw's hand! 🐰✨

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 15.38% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (2 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately describes the main change: adding configurability to prevent returned command errors from being forced into failed OpenTelemetry spans.
Description check ✅ Passed The description clearly relates to the changeset by explaining the motivation for allowing applications to control span error status for returned :stop errors.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch yordis/feat-configurable-otel-status

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit d445522. Configure here.

Comment thread lib/commanded/opentelemetry.ex Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@lib/commanded/opentelemetry/helpers.ex`:
- Around line 72-75: The current case in helpers.ex calls the configured
:error_status callback directly (the branch with fun -> fun.(meta, config)),
which can raise and skip the fallback mapping; change this to invoke the
callback inside a try/rescue (and optionally catch) block so any exception falls
back to :error, emit or dispatch a warning telemetry event when the callback
fails, and keep the existing nil -> :error behavior intact; update the branch
handling for Keyword.get(config, :error_status) to catch errors from fun.(meta,
config), log/telemetry the failure, and return :error on exception.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 9c0984e9-c40d-46e7-95ec-8b027888c983

📥 Commits

Reviewing files that changed from the base of the PR and between 347b033 and beb7158.

📒 Files selected for processing (5)
  • guides/howtos/setting-up-opentelemetry-tracing.md
  • lib/commanded/opentelemetry.ex
  • lib/commanded/opentelemetry/helpers.ex
  • test/opentelemetry/aggregate_test.exs
  • test/opentelemetry/application_test.exs
🚧 Files skipped from review as they are similar to previous changes (4)
  • test/opentelemetry/application_test.exs
  • guides/howtos/setting-up-opentelemetry-tracing.md
  • test/opentelemetry/aggregate_test.exs
  • lib/commanded/opentelemetry.ex

Comment thread lib/commanded/opentelemetry/helpers.ex

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

♻️ Duplicate comments (1)
lib/commanded/opentelemetry/helpers.ex (1)

70-75: ⚠️ Potential issue | 🟠 Major

Guard error_status callback execution to preserve fallback behavior.

The direct call on Line 74 (fun.(...)) can raise/throw and bypass apply_error_status/4, making the error path brittle. Wrap it with try/rescue/catch, emit warning telemetry, and fallback to :error.

Suggested patch
 def set_error_status(ctx, error, event_name, measurements, meta, config, tracer_id) do
   status_code =
     case Keyword.get(config, :error_status) do
       nil -> :error
-      fun when is_function(fun, 4) -> fun.(event_name, measurements, meta, config)
+      fun when is_function(fun, 4) ->
+        try do
+          fun.(event_name, measurements, meta, config)
+        rescue
+          exception ->
+            :telemetry.execute(
+              [:commanded, :opentelemetry, :warning],
+              %{count: 1},
+              %{
+                message: "error_status callback raised, falling back to :error",
+                error: exception,
+                tracer_id: tracer_id
+              }
+            )
+
+            :error
+        catch
+          kind, reason ->
+            :telemetry.execute(
+              [:commanded, :opentelemetry, :warning],
+              %{count: 1},
+              %{
+                message: "error_status callback threw, falling back to :error",
+                error: {kind, reason},
+                tracer_id: tracer_id
+              }
+            )
+
+            :error
+        end
     end

   apply_error_status(ctx, status_code, error, tracer_id)
 end
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@lib/commanded/opentelemetry/helpers.ex` around lines 70 - 75, In
set_error_status, guard the dynamic call to the configured :error_status
callback so it cannot raise/throw and bypass the normal fallback: wrap the call
to the function retrieved from Keyword.get(config, :error_status) in a
try/rescue/catch block (around the branch where fun when is_function(fun, 4) ->
fun.(...)), and on any exception/throw log/emit a warning telemetry event
(including event_name, measurements, meta and the error) and return the fallback
:error; ensure that apply_error_status/4 remains the intended safe alternative
path when the callback is absent or fails so callers still receive :error on
failure.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Duplicate comments:
In `@lib/commanded/opentelemetry/helpers.ex`:
- Around line 70-75: In set_error_status, guard the dynamic call to the
configured :error_status callback so it cannot raise/throw and bypass the normal
fallback: wrap the call to the function retrieved from Keyword.get(config,
:error_status) in a try/rescue/catch block (around the branch where fun when
is_function(fun, 4) -> fun.(...)), and on any exception/throw log/emit a warning
telemetry event (including event_name, measurements, meta and the error) and
return the fallback :error; ensure that apply_error_status/4 remains the
intended safe alternative path when the callback is absent or fails so callers
still receive :error on failure.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: eb288273-a022-47b1-be45-bcf66957f4e1

📥 Commits

Reviewing files that changed from the base of the PR and between beb7158 and d73afe2.

📒 Files selected for processing (7)
  • guides/howtos/setting-up-opentelemetry-tracing.md
  • lib/commanded/opentelemetry.ex
  • lib/commanded/opentelemetry/aggregate.ex
  • lib/commanded/opentelemetry/application.ex
  • lib/commanded/opentelemetry/helpers.ex
  • test/opentelemetry/aggregate_test.exs
  • test/opentelemetry/application_test.exs
✅ Files skipped from review due to trivial changes (1)
  • guides/howtos/setting-up-opentelemetry-tracing.md
🚧 Files skipped from review as they are similar to previous changes (3)
  • test/opentelemetry/aggregate_test.exs
  • test/opentelemetry/application_test.exs
  • lib/commanded/opentelemetry/application.ex

Signed-off-by: Yordis Prieto <yordis.prieto@gmail.com>
@yordis
yordis force-pushed the yordis/feat-configurable-otel-status branch from 2745d15 to b9e170d Compare April 16, 2026 02:26
@yordis
yordis merged commit f586f58 into main Apr 16, 2026
5 checks passed
@yordis
yordis deleted the yordis/feat-configurable-otel-status branch April 16, 2026 02:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant