Skip to content

phase-9-report.md

Latest commit

 

History

History
87 lines (71 loc) · 2.55 KB

phase-9-report.md

File metadata and controls

87 lines (71 loc) · 2.55 KB
Raw

Phase 9 Report: Cross-Model Generalization

Phase 9 adds cross-model generalization evaluation.

Implemented:

  • Model profile schema with provider, role, token cost, and latency overhead.
  • Source vs unseen model matrix execution.
  • Control and treatment runs for every model profile.
  • Transfer effect calculation for pass rate, tokens, latency, tool calls, and cost.
  • Transfer classification:
    • GENERAL
    • MODEL_SPECIFIC
    • NEGATIVE_TRANSFER
    • NO_EFFECT
  • JSON and Markdown reports.
  • CLI:
python scripts\run_generalization.py experiments\candidates\harness-candidate-07c12f3f\manifest.json --local-dev

Verified report:

experiments/results/generalization-harness-candidate-07c12f3f

Model matrix:

[
  {"model_id": "local-seed", "provider": "local", "role": "source"},
  {"model_id": "gpt-4o-mini-profile", "provider": "openai", "role": "unseen"},
  {"model_id": "claude-haiku-profile", "provider": "anthropic", "role": "unseen"},
  {"model_id": "ollama-llama-profile", "provider": "ollama", "role": "unseen"}
]

Observed transfer effects:

[
  {
    "model_id": "local-seed",
    "pass_rate_delta": 0.0,
    "token_relative_delta": -0.6643732169827152,
    "tool_call_relative_delta": -0.19607843137254904,
    "latency_relative_delta": -0.5258669698692439
  },
  {
    "model_id": "gpt-4o-mini-profile",
    "pass_rate_delta": 0.0,
    "token_relative_delta": -0.6643732169827152,
    "tool_call_relative_delta": -0.19607843137254904,
    "latency_relative_delta": -0.29836340746957624
  },
  {
    "model_id": "claude-haiku-profile",
    "pass_rate_delta": 0.0,
    "token_relative_delta": -0.6643732169827152,
    "tool_call_relative_delta": -0.19607843137254904,
    "latency_relative_delta": -0.16516129032258067
  },
  {
    "model_id": "ollama-llama-profile",
    "pass_rate_delta": 0.0,
    "token_relative_delta": -0.6643732169827152,
    "tool_call_relative_delta": -0.19607843137254904,
    "latency_relative_delta": -0.4688249400479616
  }
]

Classification:

GENERAL

Interpretation:

The middleware candidate preserved pass rate and reduced tool calls across the source profile and every unseen profile. Under the current local profile matrix, this is classified as a general harness improvement rather than model-specific behavior or negative transfer.

Caveat:

These are deterministic local model profiles, not live OpenAI, Anthropic, or Ollama API executions. They exercise the generalization infrastructure and cost/latency accounting, while real provider-backed execution remains a later integration step.