Repository navigation
Default avg_latency_seconds threshold (10s) is unrealistic for Foundry cloud eval #116
Description
Source
Discovered during E2E validation of #78.
Problem
bundles/conversational_agent_baseline.yaml ships with:
- evaluator: avg_latency_seconds
criteria: "<="
value: 10.0But Foundry cloud evaluation latency includes the entire judge-evaluator pipeline. Real values measured against FoundryAgent on gpt-5.1:
- Run 1: 22.47s/row
- Run 2: 14.04s/row
Anyone who follows the Foundry tutorial and runs the smoke dataset against a real Foundry agent will hit immediate FAIL on the latency rule, even though scoring is fine across all 4 quality evaluators.
Options
A. Raise the default for Foundry-targeted bundles to e.g. 30s. Same value can be too lax for HTTP/local backends though.
B. Split into two metrics:
agent_latency_seconds— only the agent invocation timeeval_latency_seconds— full pipeline including evaluators
…and apply tighter thresholds only to the agent metric.
C. Document prominently in the Foundry tutorial that avg_latency_seconds measures the full pipeline and users should tune the threshold per backend.
Recommendation: C as a stop-gap, B as the proper fix (also addresses #ISSUE_FAKED_LATENCY).
Severity
Medium (every Foundry tutorial run currently fails on this rule)
Activity
Dongbumlee commentedon Apr 29, 2026
False positive on my part —
results.jsondoes include a top-levelversion: 1field per theRunResultmodel (core/models.py:484). Re-verified just now:The CONTRIBUTING.md requirement is satisfied. Closing.
Dongbumlee commentedon Apr 29, 2026
Reopening — I confused issue numbers when closing. This (latency threshold) is a real issue. Closing #114 (schema_version) instead, which was the actual false-positive.
Merge pull request #118 from Azure/fix/foundry-eval-quick-wins
Dongbumlee commentedon Apr 29, 2026
Fix in flight: PR #118 → develop. Will auto-close when merged.
Dongbumlee commentedon Apr 29, 2026
Reopening — PR #118 is being reverted via #119 because PR #108 (1.0 revamp) supersedes most of those fixes by deleting the systems they patched. Defects still apply; we'll re-scope against the flat
agentops.yamlschema after #108 merges. See #119 for details.