Skip to content

feat: wire OTel telemetry spans into evaluation runner #90

Closed

Description

Summary

Integrated the OTel telemetry module (telemetry.py) into services/runner.py to emit structured spans for every evaluation run.

What was done

  • Added init_tracing() call at the start of run_evaluation()
  • Wrapped the entire evaluation body in eval_run_span() context manager
  • Added per-item eval_item_span() for each dataset row
  • Added record_evaluator_span() after threshold evaluation for each evaluator
  • Added set_eval_run_result() with pass_rate and item counts on the root span
  • Added shutdown_tracing() at the end of the run
  • Extracts model and agent_id from run_config.target.endpoint for span attributes

Span hierarchy

eval_run (root)
  eval_item[0]
    evaluator: SimilarityEvaluator
    evaluator: CoherenceEvaluator
  eval_item[1]
    ...

Implemented in

PR #86 (feature/telemetry-doc -> develop)

Activity

  1. placerda commented on Apr 16, 2026

    @placerda
    ContributorAuthor

    Completed in PR #86

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions