OpenTelemetry Integration

OpenTelemetry Integration

Since 0.9.0

Optional integration with OpenTelemetry for real-time observability of the AP. The AP exports metrics, traces and logs to any OTLP-compatible backend via an OpenTelemetry Collector.

The integration is shipped as the separate phoss-ap-otel module and is disabled by default. It can be activated by setting a single property; all further configuration is done via the standard OTel environment variables, so vendor choice (Tempo, Jaeger, Datadog, New Relic, Prometheus, …) is decoupled from the application code.

Table of Contents

• Topology
• Bootstrap Style
• Enabling OpenTelemetry
• Configuration via OTel Environment Variables
• SPI Surface
• Emitted Metrics
• Emitted Spans
• Span Kinds
• Attribute Keys
• Trace Abstraction (phoss-ap-api)
• Relationship to phoss-ap-sentry
• Programmatic Access

Topology

+-----------+      OTLP/gRPC       +----------------+      vendor-specific
| phoss AP  | -------------------> | OTel Collector | -------------------> Tempo / Jaeger / Datadog / ...
+-----------+                      +----------------+

The application speaks only OTLP. All routing, transformation, retries, and authentication concerns belong in the Collector configuration — not in the AP. This is the single architectural rule of the integration.

Bootstrap Style

The AP uses the OpenTelemetry SDK autoconfigure module as its primary bootstrap. The SDK is initialized in-process at application startup based on environment variables / system properties. There is no -javaagent: flag and no agent jar in the Docker image.

If you prefer the OpenTelemetry Java Agent (broader auto-instrumentation of third-party libraries), the AP can be run with it as a documented alternative — but the in-tree integration ships the SDK path because it gives reproducible builds and pinned instrumentation versions.

Enabling OpenTelemetry

Add a single property to application.properties:

otel.enabled=true

When the property is set to true and the phoss-ap-otel module is on the classpath (it is included by phoss-ap-webapp by default), OtelBootstrapInitializer will:

1. Build an OpenTelemetrySdk via AutoConfiguredOpenTelemetrySdk.builder().build() and register it with GlobalOpenTelemetry.set(...).
2. Log the installed SDK, or - if a global instance is already present, e.g. from the OpenTelemetry Java Agent - keep that one and shut down the SDK it just built.

The metric and lifecycle handlers themselves live in phoss-ap-core and are registered via META-INF/services independently of this: APMetricsNotificationHandler translates IAPNotificationHandlerSPI (failure) callbacks into counter increments, and APMetricsLifecycleEventHandler translates IAPLifecycleEventSPI (lifecycle / success) callbacks into counter and histogram observations. All manual spans go through com.helger.telemetry.Telemetry, which resolves OtelAPTracerSPI via META-INF/services, so everything ends up in the same OpenTelemetry instance.

Why an ApplicationContextInitializer and not a bean (since v0.13.0)

OtelBootstrapInitializer is registered in META-INF/spring.factories as an ApplicationContextInitializer. Those run after the environment is prepared but before the context is refreshed — that is, before the first bean exists. The ordering is not cosmetic:

The first GlobalOpenTelemetry.get() in a JVM permanently installs the official no-op instance as the global one, and every later GlobalOpenTelemetry.set(...) fails with IllegalStateException: GlobalOpenTelemetry.set has already been called. Up to v0.12.0 the SDK was installed on the ApplicationStartedEvent, which fires after the refresh — and the Flyway migration, which runs while the AS4 servlet bean is being created and is wrapped in a telemetry span by ph-db-flyway, always got there first. The result was a startup failure with otel.enabled=true (issue #102).

Note that resetting the global (GlobalOpenTelemetry.resetForTest()) is not a fix for this. It makes the startup succeed, but the ph-telemetry OTel bindings used to cache the tracer and meter they resolved on first use, so every span would silently stay a no-op for the lifetime of the JVM — and on an agent-instrumented deployment it would throw away the agent's own SDK.

The library side was fixed in ph-telemetry 1.1.1: the OTel bindings resolve via GlobalOpenTelemetry.getOrNoop(), which is the API OpenTelemetry documents for instrumentation and does not claim the global slot, and they only cache the result once an instance is really registered. So an early span can no longer break the bootstrap, and no longer pins the no-op. The two fixes are complementary — the library one makes an early span harmless, this initializer makes it actually recorded.

If a global instance is already present when the initializer runs — realistically only when the OpenTelemetry Java Agent is attached — the AP keeps it, shuts down the SDK it just built and logs a warning. In that setup set otel.enabled=false and let the agent do the instrumentation.

Configuration via OTel Environment Variables

All other behavior is driven by the standard OpenTelemetry SDK autoconfigure variables. The most relevant ones for a phoss AP deployment:

Variable	Purpose	Example
OTEL_SERVICE_NAME	Logical service identifier in the backend	phoss-ap
OTEL_EXPORTER_OTLP_ENDPOINT	OTLP endpoint of the Collector	http://otel-collector:4317
OTEL_EXPORTER_OTLP_PROTOCOL	grpc or http/protobuf	grpc
OTEL_TRACES_EXPORTER	Span exporter; usually otlp	otlp
OTEL_METRICS_EXPORTER	Metric exporter; usually otlp	otlp
OTEL_LOGS_EXPORTER	Log exporter; usually otlp	otlp
OTEL_RESOURCE_ATTRIBUTES	Resource attributes attached to every signal	deployment.environment=test,peppol.seat.id=POP000306
OTEL_TRACES_SAMPLER	Sampler (e.g. parentbased_traceidratio)	parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG	Sampler argument	0.1

System property equivalents (-Dotel.service.name=…) also work. Refer to the upstream documentation for the full list — the AP does not redefine or wrap any of these knobs.

SPI Surface

The AP exposes two complementary SPI interfaces in phoss-ap-api. The OTel module implements both; the Sentry module implements only the failure SPI.

SPI	Purpose	When to implement
IAPNotificationHandlerSPI	Failure callbacks: verification rejections, deferred verifications, inbound duplicate rejections, MLS correlation errors, permanent forwarding/sending failures, reporting failures, unexpected exceptions.	Error trackers (Sentry-like): per-failure grouping, alerting, stack traces.
IAPLifecycleEventSPI	Positive lifecycle events: inbound received/verified/forwarded, MLS correlated (with round-trip duration), outbound accepted/verified/sent (with sending duration + attempts), reporting success, retry/archival/cleanup scheduler cycles (with duration + item count).	Observability backends (OTel-like): throughput counters, SLA histograms, dashboards.

The split is deliberate: positive lifecycle events do not belong in an error tracker (cost and noise). Adding the lifecycle SPI is fully additive — existing notification handlers are unaffected.

Emitted Metrics

All metrics use the phoss.ap.* namespace. Metric names follow OpenTelemetry semantic conventions: lowercase, dot-separated, no units in the name.

Failure counters (driven by IAPNotificationHandlerSPI)

Metric	Instrument	Unit	Description
phoss.ap.inbound.verification.rejections	counter	{transaction}	Inbound transactions rejected by verification
phoss.ap.inbound.verification.deferred	counter	{document}	Inbound documents whose verification was deferred, because a verifier made no verdict (since v0.12.0). Incremented on every deferral, i.e. also on every unsuccessful re-verification — a rising counter means a verifier needs attention
phoss.ap.outbound.verification.rejections	counter	{document}	Outbound documents rejected by verification before sending
phoss.ap.inbound.receiver.not_serviced	counter	{message}	Inbound messages whose receiver is not serviced by this AP
phoss.ap.inbound.mls.correlation_errors	counter	{message}	Inbound MLS messages that could not be correlated to an outbound transaction
phoss.ap.inbound.forwarding.errors	counter	{attempt}	Inbound forwarding attempts that failed (transient or permanent)
phoss.ap.inbound.forwarding.permanent_failures	counter	{transaction}	Inbound transactions that exhausted all forwarding retries
phoss.ap.outbound.sending.permanent_failures	counter	{transaction}	Outbound transactions that exhausted all sending retries
phoss.ap.reporting.failures	counter	{report}	Peppol Reporting (TSR/EUSR) failures
phoss.ap.unexpected_exceptions	counter	{exception}	Unexpected exceptions not covered by a more specific counter
phoss.ap.circuit_breaker.rejections	counter	{call}	Calls rejected by a circuit breaker, i.e. an SMP lookup, an AS4 sending or a forwarding that was not even attempted (since v0.13.0, attributes: phoss.ap.circuit_breaker.key, phoss.ap.circuit_breaker.state)
phoss.ap.circuit_breaker.state_changes	counter	{transition}	Circuit breaker state transitions (since v0.13.0, attributes: phoss.ap.circuit_breaker.key, phoss.ap.circuit_breaker.state = the new state)

Lifecycle counters (driven by IAPLifecycleEventSPI)

Metric	Instrument	Unit	Description
phoss.ap.inbound.received	counter	{message}	Inbound AS4 messages received and persisted (attributes: phoss.ap.is_duplicate_as4, phoss.ap.is_duplicate_sbdh)
phoss.ap.inbound.verification.accepted	counter	{document}	Inbound documents that passed verification
phoss.ap.inbound.mls.correlated	counter	{message}	Inbound MLS messages successfully correlated (attributes: phoss.ap.mls.response_code, phoss.ap.mls.reception_status)
phoss.ap.inbound.forwarded	counter	{transaction}	Inbound documents successfully forwarded (attribute: phoss.ap.is_retry)
phoss.ap.inbound.verification.rejections.forwarded	counter	{transaction}	Inbound documents that were forwarded despite having been rejected by verification (since v0.12.0, attribute: phoss.ap.is_retry). A subset of phoss.ap.inbound.forwarded; only counts the rejection forwarding mode retry, because a best-effort copy never runs through the forwarding state machine
phoss.ap.outbound.accepted	counter	{transaction}	Outbound transactions accepted and queued for sending
phoss.ap.outbound.verification.accepted	counter	{document}	Outbound documents that passed verification
phoss.ap.outbound.sent	counter	{transaction}	Outbound transactions successfully sent via AS4
phoss.ap.reporting.success	counter	{report}	Peppol Reporting (TSR/EUSR) reports successfully sent (attribute: phoss.ap.report.type)

Lifecycle histograms

Metric	Instrument	Unit	Description
phoss.ap.inbound.forwarding.duration	histogram (double)	s	Time from AS4 reception to successful forwarding (attribute: phoss.ap.is_retry)
phoss.ap.outbound.sending.duration	histogram (double)	s	Time from outbound transaction creation to confirmed AS4 receipt
phoss.ap.outbound.sending.attempts	histogram (long)	{attempt}	Number of AS4 sending attempts before confirmed receipt
phoss.ap.mls.roundtrip.duration	histogram (double)	s	Time from outbound send completion to MLS reception — powers MLS-1 / MLS-2 SLA dashboards (attributes: phoss.ap.mls.response_code, phoss.ap.mls.reception_status)
phoss.ap.scheduler.cycle.duration	histogram (double)	s	Wall-clock duration of a scheduler cycle (attributes: phoss.ap.scheduler.name ∈ {retry,archival,cleanup}, phoss.ap.is_outbound)
phoss.ap.scheduler.cycle.items	histogram (long)	{item}	Number of items processed in a scheduler cycle (same attributes as above)

High-cardinality Peppol identifiers (sender, receiver, document type, process, SBDH instance ID, transaction ID) are intentionally not attached as metric attributes — they appear on spans only. The circuit breaker key is an exception: it is bounded by the number of SMPs and remote APs this AP talks to, and it is the dimension that makes the two circuit breaker counters actionable at all.

Emitted Spans

Manual instrumentation is emitted across phoss-ap-core and phoss-ap-forwarding via the trace abstraction. All span names are constants on CPhossAPOtel:

Span	Operation	Typical kind
phoss.ap.inbound.receive	AS4 inbound receipt (top-level inbound span)	CONSUMER
phoss.ap.inbound.duplicate_check	AS4 / SBDH duplicate detection	INTERNAL
phoss.ap.inbound.c4_resolve	Resolution of the C4 country code (sync-forwarding response)	INTERNAL
phoss.ap.inbound.forward	Forwarding to the Receiver Backend (top-level forwarding span)	PRODUCER
phoss.ap.inbound.forward.secondary	Fire-and-forget dispatch to one configured secondary forwarder (one span per forwarder, attribute phoss.ap.forwarder.index)	PRODUCER
phoss.ap.inbound.forward.rejected	Fire-and-forget dispatch of a verification-rejected document with verification.inbound.rejection-forwarding=best-effort (one span per forwarder, attribute phoss.ap.forwarder.index; since v0.12.0)	PRODUCER
phoss.ap.forwarder.dispatch	Inner span emitted by each individual forwarder implementation (HTTP, S3, SFTP, filesystem)	PRODUCER / CLIENT
phoss.ap.outbound.send	AS4 outbound send (top-level outbound span)	PRODUCER
phoss.ap.outbound.sbdh_read	Reading / parsing the SBDH before sending	INTERNAL
phoss.ap.outbound.as4_send	The actual AS4 send attempt (one per attempt)	PRODUCER
phoss.ap.smp.lookup	SMP lookup for the receiver endpoint	CLIENT
phoss.ap.mls.correlate	MLS correlation with an outbound transaction	INTERNAL
phoss.ap.mls.send	Outbound MLS send (as C3)	PRODUCER
phoss.ap.mls.forward.copy	Fire-and-forget dispatch of a copy of a self-generated MLS to the MLS copy sink (since v0.12.0, attributes: phoss.ap.transaction.id, phoss.ap.sbdh.instance_id, phoss.ap.mls.response_code)	PRODUCER
phoss.ap.verification	Optional document verification	INTERNAL
phoss.ap.reporting.tsr	TSR report generation / validation / sending (top-level)	INTERNAL
phoss.ap.reporting.tsr.validate_store	TSR validation + persistence inner span	INTERNAL
phoss.ap.reporting.eusr	EUSR report generation / validation / sending (top-level)	INTERNAL
phoss.ap.reporting.eusr.validate_store	EUSR validation + persistence inner span	INTERNAL
phoss.ap.scheduler.cycle	Wall-clock span around a scheduler cycle (retry, archival, cleanup)	INTERNAL
phoss.ap.archival	Archival of an individual completed transaction	INTERNAL

Each span receives Peppol-specific identifiers as span attributes (not metric tags) to avoid cardinality explosions in the metrics pipeline. Failures during a span are recorded via recordException(...) and the span status is set to ERROR.

Span Kinds

The trace abstraction exposes a small enum (EAPSpanKind) that mirrors the OpenTelemetry SpanKind semantics one-to-one:

Value	Meaning
INTERNAL	Work that does not cross a process boundary (verification, MLS correlation, archival, scheduler cycles, ...)
CLIENT	Synchronous outgoing call where a reply is expected (SMP lookup)
SERVER	Synchronous incoming call handled by the AP
PRODUCER	Asynchronous send to an external system (AS4 send, S3 upload, HTTP forward)
CONSUMER	Asynchronous receive from an external system (AS4 inbound)

Attribute Keys

Key	Type	Where
phoss.ap.transaction.id	string	spans, counters
phoss.ap.sbdh.instance_id	string	spans, counters
phoss.ap.sender.id	string	spans, counters
phoss.ap.receiver.id	string	spans, counters
phoss.ap.doctype.id	string	spans, counters
phoss.ap.process.id	string	spans, counters
phoss.ap.mls.response_code	string	MLS correlation counter + round-trip histogram, spans
phoss.ap.mls.reception_status	string (RECEIVED_AP/RECEIVED_AB/RECEIVED_RE)	MLS correlation counter + round-trip histogram
phoss.ap.report.type	string (TSR/EUSR)	reporting counter, spans
phoss.ap.report.year_month	string (YYYY-MM)	reporting counter, spans
phoss.ap.report.item_count	long	reporting spans
phoss.ap.is_retry	bool	forwarding counters / histograms, spans
phoss.ap.is_outbound	bool	scheduler cycle histograms, spans
phoss.ap.is_duplicate_as4	bool	inbound received counter, spans
phoss.ap.is_duplicate_sbdh	bool	inbound received counter, spans
phoss.ap.scheduler.name	string (retry/archival/cleanup)	scheduler cycle histograms, spans
phoss.ap.scheduler.items	long	scheduler cycle spans
phoss.ap.forwarder.type	string (e.g. http/s3/sftp/filesystem)	forwarder dispatch spans
phoss.ap.smp.url	string	SMP lookup spans
phoss.ap.exception.context	string	exception counter, span events
phoss.ap.exception.class	string	exception counter, span events

Trace Abstraction (phoss-ap-api)

The AP does not depend on the OpenTelemetry API jar from phoss-ap-core, phoss-ap-forwarding or any other in-tree module. Instead, phoss-ap-api ships a small tracing facade in the com.helger.phoss.ap.api.trace package:

Type	Purpose
APTrace	Static facade. APTrace.startSpan(name, kind) and APTrace.withSpan(name, kind, body) are the call sites used throughout the codebase.
IAPSpan	AutoCloseable span with fluent setAttribute, recordException, setStatusOk, setStatusError. All mutators return this. Calling close() ends the span and detaches it from the current thread.
EAPSpanKind	The enum described in Span Kinds.
IAPTracerSPI	The ServiceLoader-discovered SPI implemented by phoss-ap-otel (and pluggable by any other backend).

Resolution: on first use APTrace does a ServiceLoader.load(IAPTracerSPI.class) and caches the first registered implementation. If none is on the classpath, it falls back to an internal no-op tracer that turns every startSpan(...) call into a cheap, side-effect-free stub. This means all instrumented call sites are safe to compile and run without phoss-ap-otel on the classpath — there is no need to guard them with feature flags.

Tests may install a custom tracer via APTrace.install(IAPTracerSPI) (pass null to revert to SPI-discovery).

Relationship to phoss-ap-sentry

The two modules are complementary, not alternative. Both can be enabled simultaneously:

• phoss-ap-otel covers metrics, distributed traces, and structured logs — the basis for dashboards, SLO measurement, and timing analysis (e.g. MLS-1/MLS-2 SLA compliance).
• phoss-ap-sentry is purpose-built for error tracking with stack-trace grouping, release tagging, and alerting on individual exceptions.

If both are active, an unexpected exception increments the phoss.ap.unexpected_exceptions counter, is attached to the current span, and is captured by Sentry as a deduplicated issue. Positive lifecycle events (received, forwarded, sent, scheduler cycles) are only sent to OTel — they do not reach Sentry by design.

Programmatic Access

Spans — preferred: APTrace

In-tree code records spans exclusively through the abstraction in com.helger.phoss.ap.api.trace. Neither phoss-ap-core nor phoss-ap-forwarding (and no module other than phoss-ap-otel itself) takes a compile-time dependency on the OpenTelemetry API jar.

Using try-with-resources:

import com.helger.phoss.ap.api.otel.CPhossAPOtel;
import com.helger.phoss.ap.api.trace.APTrace;
import com.helger.phoss.ap.api.trace.EAPSpanKind;
import com.helger.phoss.ap.api.trace.IAPSpan;

try (final IAPSpan aSpan = APTrace.startSpan (CPhossAPOtel.SPAN_SMP_LOOKUP, EAPSpanKind.CLIENT)
                                  .setAttribute (CPhossAPOtel.ATTR_RECEIVER_ID, sReceiverID))
{
  // ... perform the SMP lookup ...
  aSpan.setStatusOk ();
}
catch (final RuntimeException ex)
{
  // The catch block is optional — APTrace.withSpan (...) below handles exception
  // recording automatically. Use try/catch only when you also need to swallow or
  // rewrap the exception.
  throw ex;
}

Or with the functional helper that wires up exception recording and status for you:

final SMPResult aResult = APTrace.withSpan (CPhossAPOtel.SPAN_SMP_LOOKUP, EAPSpanKind.CLIENT, aSpan -> {
  aSpan.setAttribute (CPhossAPOtel.ATTR_RECEIVER_ID, sReceiverID);
  return performSmpLookup (sReceiverID);
});

If phoss-ap-otel is not on the classpath, or otel.enabled=false, both forms degrade to no-ops without any feature-flag guards.

Metrics — PhossAPTelemetry

The phoss-ap-otel module exposes the cached Meter and counter / histogram instruments via com.helger.phoss.ap.otel.PhossAPTelemetry. Only the OTel-side handlers (APNotificationHandlerOtel, APLifecycleEventHandlerOtel) interact with it directly. New metric call sites should follow the same pattern (add the instrument to PhossAPTelemetry, drive it from a lifecycle / notification handler) rather than emit metrics from business logic in phoss-ap-core.