Skip to content

Add AgentOps support for Microsoft Agent Framework evaluations #41

Closed

Description

Objective

Add support in AgentOps Toolkit for evaluation workflows based on Microsoft Agent Framework.

Scope

Enable AgentOps to work with agent and workflow evaluations (including multi-turn and tool-aware scenarios) defined by the framework, while remaining provider-agnostic.

Tasks

  • Map Agent Framework concepts to AgentOps (agents, workflows, conversations, tool calls)
  • Define how run.yaml represents these evaluation scenarios
  • Design input handling for conversations, expected outputs, and tool calls
  • Define normalized output structure for results and reporting
  • Identify minimal integration points (single agent, workflows, no re-run scenarios)

Acceptance Criteria

  • High-level integration design is defined
  • Supports agent and workflow evaluation scenarios
  • Maintains provider-agnostic execution model
  • Output format is defined for CI/reporting

References:

https://github.com/microsoft/agent-framework/blob/main/docs/decisions/0020-foundry-evals-integration.md
https://github.com/microsoft/agent-framework/tree/main/python/samples/05-end-to-end/workflow_evaluation

Activity

  1. changed the title [-]Improve Agent Framework Integration[/-] [+]Add AgentOps support for Microsoft Agent Framework evaluations[/+] on Mar 24, 2026
  2. placerda commented on Mar 27, 2026

    @placerda
    ContributorAuthor

    Merged into #50

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions