RFC: Support selectable observability providers for AgentSeek templates

[webup]

RFC: Support selectable observability providers for AgentSeek templates

Summary

AgentSeek should support LangSmith, Phoenix, Langfuse, NeMo Relay, and custom OTLP observability targets at the same time, while letting each generated project pick one preferred default observability provider.

The user-facing goal is simple:

AGENTSEEK_O11Y_ENABLED=true
AGENTSEEK_O11Y_PROVIDER=phoenix

Supported provider values:

• none
• phoenix
• langsmith
• langfuse
• relay
• custom_otlp

Default recommendation:

• Use phoenix as the default local-first option.
• Keep LangSmith and Langfuse as first-class hosted or self-hosted options.
• Treat Relay as a runtime/export layer that can feed OTLP, ATOF, or ATIF sinks.
• Allow multiple sinks only through an explicit advanced mode later.

Why

AgentSeek currently has separate observability paths:

• docs/guides/observability-tracing.md documents LangSmith cloud tracing via LANGSMITH_*.
• templates/langchain/default already has OpenTelemetry export to Phoenix.
• The default local Phoenix stack persists traces through OceanBase seekdb.
• Issue 缺少 LANGSMITH_ENDPOINT 配置导致 APAC 区域 tracing 无法上云,仅能在本地 Studio 中可见 #83 shows real user pain around LangSmith endpoint configuration.

As we add Langfuse and Relay support, we should avoid making users edit multiple unrelated tracing knobs by hand. AgentSeek should provide a single provider selection layer that maps to each backend's native configuration model.

Non-goals

• Do not build a full observability UI inside AgentSeek.
• Do not self-host LangSmith.
• Do not force Langfuse self-hosting into the default stack.
• Do not send traces to multiple external services by default.
• Do not make Relay a required runtime dependency for all templates.
• Do not replace existing vendor environment variables such as LANGSMITH_* or LANGFUSE_*.

Proposed UX

Generated projects should expose a small provider selector in .env.example:

# none | phoenix | langsmith | langfuse | relay | custom_otlp
AGENTSEEK_O11Y_PROVIDER=phoenix
AGENTSEEK_O11Y_ENABLED=true
AGENTSEEK_O11Y_PROJECT_NAME=<project-slug>
AGENTSEEK_O11Y_SERVICE_NAME=<project-slug>

Provider-specific settings remain standard:

# Phoenix / custom OTLP
AGENTSEEK_OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://127.0.0.1:6006/v1/traces

# LangSmith
LANGSMITH_TRACING=true
LANGSMITH_API_KEY=<your-langsmith-api-key>
LANGSMITH_PROJECT=<project-slug>
LANGSMITH_ENDPOINT=https://api.smith.langchain.com

# Langfuse
LANGFUSE_PUBLIC_KEY=<your-langfuse-public-key>
LANGFUSE_SECRET_KEY=<your-langfuse-secret-key>
LANGFUSE_BASE_URL=https://cloud.langfuse.com

# Relay
AGENTSEEK_RELAY_ENABLED=true
AGENTSEEK_RELAY_CONFIG=.nemo-relay/plugins.toml

Provider Mapping

Provider	Default protocol	Configuration path	Notes
phoenix	OTLP HTTP	AGENTSEEK_OTEL_*	Best local default; current langchain/default already uses Phoenix.
langsmith	LangChain/LangGraph native tracing	LANGSMITH_*	Avoid duplicate OTLP instrumentation by default.
langfuse	OTLP HTTP or Langfuse callback	LANGFUSE_* plus derived OTLP headers	Prefer OTLP first for consistency; callback handler can be a richer follow-up.
relay	Relay exporters	Relay plugin config	Relay should wrap/export runtime events and forward to OTLP/ATIF/ATOF sinks.
custom_otlp	OTLP HTTP	explicit endpoint	Escape hatch for Tempo, Honeycomb, Datadog, collector pipelines, etc.

Proposed Architecture

Add an observability provider registry inside generated templates:

.env / shell
  -> ProjectSettings
  -> configure_observability(settings)
      -> provider: none
      -> provider: phoenix
      -> provider: langsmith
      -> provider: langfuse
      -> provider: relay
      -> provider: custom_otlp

The provider registry should preserve the current configure_tracing(settings) behavior for Phoenix, then make other providers explicit rather than hidden side effects.

Suggested Python shape:

def configure_observability(settings: ProjectSettings) -> None:
    provider = settings.o11y_provider
    if not settings.o11y_enabled or provider == "none":
        return
    if provider == "phoenix":
        configure_otlp_tracing(settings.phoenix_endpoint)
    elif provider == "custom_otlp":
        configure_otlp_tracing(settings.otlp_endpoint)
    elif provider == "langsmith":
        configure_langsmith_env(settings)
    elif provider == "langfuse":
        configure_langfuse_otlp(settings)
    elif provider == "relay":
        configure_relay_export(settings)

Compose Profiles

Update generated Docker Compose files to support provider-specific profiles:

• Default: app + frontend + local Phoenix + OceanBase seekdb.
• o11y-phoenix: starts Phoenix and OceanBase seekdb.
• o11y-relay: starts or configures Relay when the template supports it.
• LangSmith: no local service.
• Langfuse: hosted by default; self-host can be a later optional profile.

The default local profile should continue to favor Phoenix backed by OceanBase seekdb.

CLI / Lifecycle Checks

agentseek doctor should understand the selected provider:

• none: no external check.
• phoenix: check local endpoint and compose service health.
• langsmith: check required vars exist, but never print API keys.
• langfuse: check base URL and required keys exist, but never print secrets.
• relay: check Relay binary/config availability.
• custom_otlp: check endpoint is present and valid.

agentseek info should show:

• selected provider
• local UI URL when applicable
• required missing variables as redacted names
• whether traces are expected to be local, hosted, or exported

Documentation Work

Add or update:

• docs/guides/observability-tracing.md
• docs/guides/observability-tracing.zh.md
• templates/langchain/default/README.md
• generated .env.example

The guide should include:

• provider matrix
• recommended defaults
• one setup section per provider
• troubleshooting for missing traces
• warning about duplicate traces when users enable multiple providers manually

Acceptance Criteria

• A generated langchain/default project exposes AGENTSEEK_O11Y_PROVIDER.
• phoenix remains the default local option and still exports traces to Phoenix.
• langsmith can be selected without code changes beyond env vars.
• langfuse has a documented first-class path, preferably OTLP HTTP first.
• relay is documented as an exporter/runtime option, not a mandatory backend.
• agentseek doctor reports provider-specific missing config without leaking secrets.
• Docs clearly say that only one provider is enabled by default.
• Tests cover rendering and docs expectations for the provider matrix.

Open Questions

• Should provider selection live only in generated project .env, or should AgentSeek CLI also accept agentseek create --o11y-provider <provider>?
• Should Langfuse v1 use OTLP only, or add the Langfuse LangChain callback handler immediately?
• Should Relay be a separate template option first, or a provider mode in existing LangChain templates?
• Should custom_otlp support headers in v1, or only endpoint URL?
• Should Phoenix use the current image path in this repo, or switch to the standalone AgentSeek Phoenix image once that migration is fully merged?

Related

• 缺少 LANGSMITH_ENDPOINT 配置导致 APAC 区域 tracing 无法上云,仅能在本地 Studio 中可见 #83
• Existing guide: docs/guides/observability-tracing.md
• Existing template files:
  • templates/langchain/default/{{cookiecutter.project_slug}}/src/{{cookiecutter.project_slug}}/settings.py
  • templates/langchain/default/{{cookiecutter.project_slug}}/src/{{cookiecutter.project_slug}}/observability.py
  • templates/langchain/default/{{cookiecutter.project_slug}}/docker-compose.yml

[kic635]

For the question:

Should Relay be a separate template option first, or a provider mode in existing LangChain templates?

I lean toward keeping Relay as a separate template option initially, rather than introducing it as a regular provider mode.

The existing templates already provide relatively simple observability paths through Phoenix, LangSmith, and Langfuse. For most users, these integrations are sufficient for basic trace collection and analysis, so adding Relay as another ordinary provider could introduce unnecessary configuration and implementation complexity.

Relay is more valuable in scenarios that the existing integrations do not directly cover. In addition to observing Agent, Tool, and LLM execution, Relay can export structured event data in formats such as ATOF and ATIF, and support more complex event processing and export requirements.

Therefore, Relay should initially be presented as a dedicated template for demonstrating these capabilities, rather than as another general observability backend. If concrete use cases emerge later, Relay could be introduced into existing templates as an optional instrumentation or export layer.