Review Evaluation Model Design #50

Objective

Refine and modularize the evaluation model to align with the new AgentOps concepts (run config, bundle, dataset, backend, evaluator), ensuring clarity, extensibility, and consistency across implementations.

Scope

Covers the evaluation architecture, schema, and execution flow, including run configuration, bundle/dataset resolution, backend selection, and evaluator orchestration.

Tasks

• Review current evaluation implementation against new conceptual model

• Align implementation with core concepts:

  • run.yaml (control plane)
  • bundle (evaluators + thresholds)
  • dataset (input/expected)
  • backend abstraction (Foundry, HTTP, local)

• Ensure clear separation of concerns between:

  • runner
  • backend
  • evaluators

• Validate orthogonal target model:

  • target.type, execution_mode, endpoint.kind

• Standardize bundle and dataset resolution:

  • name-based vs path-based

• Review evaluator execution pipeline and scoring consistency

• Validate outputs:

  • results.json schema
  • report.md generation

• Validate exit codes behavior (0, 1, 2)

• Identify gaps for extensibility (new backends, evaluators, scenarios)

• Add support in AgentOps Toolkit for evaluation workflows based on Microsoft Agent Framework.
  References:
  https://github.com/microsoft/agent-framework/blob/main/docs/decisions/0020-foundry-evals-integration.md
  https://github.com/microsoft/agent-framework/tree/main/python/samples/05-end-to-end/workflow_evaluation

Acceptance Criteria

• Evaluation model is consistent with documented concepts
• Implementation is modular and extensible
• Separation of concerns is clear and enforced
• Run config schema is validated and predictable
• Outputs are consistent and structured
• Evaluation flow works across all supported backends
• New scenarios can be added without refactoring core logic