Configuration Schema and Validation #8

Objective

Provide a strict and versioned configuration schema to prevent ambiguous evaluation setups.

Scope

Introduce schema validation for eval.yaml and enforce deterministic configuration behavior.

Tasks

• Define Pydantic or JSON schema model for:

  • bundle
  • dataset
  • evaluators
  • thresholds
  • output options
  • run metadata

• Implement agentops eval validate -c <config>

• Add helpful validation error messages

• Version configuration schema

Acceptance Criteria

• Invalid configs fail with clear message
• Valid configs print resolved evaluation plan
• Schema version is included in config parsing

Out of Scope

• Auto-generation of config files
• Advanced config templating