Added mcp + dashboard directories - #23
Open
lintyfresh wants to merge 20 commits into
Open
Conversation
…I to orchestrate eval mcp
…latency fix Rework how each agent-eval run is recorded: Prompt registry: prompts.py -> prompts/ package with one JSON per variant (composite_v1.json unchanged in content), each carrying an integer `index`. resolve_prompt() mirrors config.resolve_model (by index/name/default); runs record the derived prompt_key. --prompt selects a variant; the mapping row's judge_prompt is now honored. Central store: mirror each flat run row to a new agentic_runs Mongo collection (tools.save_run_row) alongside the local Parquet sink, so the team can query results (selection_accuracy, save_success, timings, weave_trace_url) without Yen access. --no-mongo-runs opts out; a Mongo hiccup never fails the run. Shared eval_id key: thread the mapping's per-(output x judge) eval_id into both save paths and key both collections on it. save_evaluation is now keyed on eval_id (client-injected in the agent loop, not trusted to the LLM), so the same output graded by different judges no longer collapses last-writer-wins. Migrated the legacy unique index (task/benchmark/model/run) to a partial unique index on eval_id. Latency: scrape GPU metrics outside the wall_time window so wall_time is backend-consistent, and add derived overhead_time (wall - llm) so llm_time (model/service latency) is separable from local tool-execution time. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
playground.json now defines three judge models (gpt-5-mini, Llama-4, claude-sonnet-4-6, all temperature 0). create_eval_mapping enumerates one judge config per playground model, and mapping.sample_paired samples K distinct OUTPUTS (stratified by benchmark) and returns every judge's row for them — an apples-to-apples judge comparison where every judge sees the same outputs. --sample now means K outputs. Adds tests for sample_paired and a config-agnostic reasoning-level test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The eval-run registry is produced by agent_eval.registry.create_eval_mapping and is already listed in .gitignore; it was tracked from before the ignore, so its regeneration churned the diff. Untrack it (the file stays on disk). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pre-flight found the Stanford API key is not authorized for Llama-4 (401 key_model_access_denied). DeepSeek-V3.2 is an open-weight model in the key's allow-list and validated end-to-end (routing + tool-calling + save_evaluation). Update the playground judge and the mapping test fixture accordingly; the generated registry/sample are regenerated separately (untracked). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…flow viz - Group breakdown + summary cards now show latency: avg llm_time (model/service) and overhead_time (local tool+loop), split per the earlier latency work. - Add a "Benchmark" group-by pill alongside Model/Prompt/etc. - Tool-call path/routing DAG renders at true pixel size and scrolls horizontally instead of shrinking to fit (width:100% was scaling the whole diagram down); larger nodes/fonts for legibility. - Glossary: clarify total_tokens (prompt+completion summed across steps, ~95% prompt) vs tokens_per_sec (completion throughput), a common misread. - Annotate + docstring analysis/build_dashboard.py and analysis/queries.py to the push standard (mypy clean on both). - Refresh NEXT_STEPS.md: batch 376856 done, results summary, team review Tue 2026-07-21, queued composite_v2 deep-dive. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Routing trustworthiness:
- composite_v2 prompt: goal-based, drops the numbered procedure / "Call get_task_output" /
"retry once" (the double-fetch triggers); keeps the field_evaluations output contract.
Selectable via --prompt / the mapping judge_prompt column.
- Fix the args-parse bug: on malformed tool-call JSON, return the error to the model instead
of a phantom get_task_output({}) that induced a retry.
- Normalize an absent value ([]) to "" for non-list fields in fetch_evaluable_output (bench-10
read as an empty *list* and mis-routed judges); list fields keep []. Uses the benchmark's
own declared type; not surfaced to the agent.
- routing_path_correct scorer: grades the ACTUAL tool-call path (exactly one get_task_output,
the SUCCESSFUL metric calls == the expected multiset per field, one successful save), beside
the declaration-based selection_accuracy. Wired through runner/sink/agentic_runs/dashboard.
Cost (#1): per-model input_price/output_price (USD/1M) in backends/*.json + config.model_price;
flatten_run computes input/output/total_dollar_cost (local judges $0; unpriced -> null);
dashboard gains a Cost $ metric + group column. DeepSeek-V3.2 has no published price (null).
Versioned stores: code_version (git_commit) is now part of the key of BOTH agentic_runs and
agentic_evaluations (compound unique index eval_id+git_commit), so a re-run under a new commit
coexists with old results instead of overwriting. get_git_commit flags '-dirty'. run_exists +
--skip-existing (default on) skip a (task x judge x prompt) already evaluated at this version.
135 tests green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…Slurm refresh Copy the teammate's tokenviz pattern into an agent_eval-owned dashboard: - analysis/export_runs.py materializes agentic_runs (Mongo) to a Parquet cache. - agent_eval/scripts/refresh_dashboard.slurm is a self-rescheduling job (RCPedia pattern): reschedules the next run ~24h out first (survives a failed build), then export -> rebuild the standalone HTML. Submit once; scancel the pending job to stop. - Generated images/agent_dashboard_live.html is gitignored (data-free tracked template stays the shell). export_runs unit-tested with a fake DB. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two fixes surfaced by the v1 re-run (a transient playground outage errored a whole judge arm): - run_exists now counts only NON-errored runs (stopped_reason != "error"), so a previously errored run is re-runnable instead of being wrongly skipped by skip-existing. - flatten_run persists error_detail (bounded exception text) on errored runs, so a failure is diagnosable from the stored row without reproducing it. Added to the dashboard columns. 136 tests green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…te routing labels Prep for the v1-vs-v2 comparison: - git_commit was being dropped from the payload (not in RUN_COLUMNS) so the version dimension was invisible; add it + a "Version" group-by pill -> compare v1-old / v1-new / v2. - routing_path_correct was only in the raw data; surface it as a summary card, group-table column, runs-table indicator, and selectable metric (aggregated as a rate). - Disambiguate the two routing metrics: "Route (declared)" (selection_accuracy, what the agent saved) vs "Route (path)" (routing_path_correct, what it actually did). Add a Version column to the runs table. 136 tests green; JS validated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The judge's LLM call only retried connection/timeout, so a transient rate-limit (429) or server (5xx) window killed the run instead of backing off — the cause of the gpt-5-mini batch that errored 40/40 mid-run. Now retry the OpenAI SDK's transient error types (RateLimitError, InternalServerError, APITimeoutError, APIConnectionError) plus their message signals, with 5 attempts / exponential backoff to 30s. Permanent errors (BadRequest/Authentication/NotFound) still fail fast so misconfigs surface. AsyncOpenAI is built with max_retries=0 so tenacity is the single retry layer and attempts don't multiply with the SDK's own. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
serve_dashboard.py: render the dashboard from agentic_runs live on each page load (Starlette/uvicorn, ~30s TTL cache), bound to 127.0.0.1 for viewing via VS Code's forwarded port. Read-only; reuses export_runs -> build_snapshot -> render_html. Dashboard edits: - Version (git_commit) filter in the sidebar; layout breakpoint 1180->900px so the filter sidebar sits on the left at normal widths instead of stacking. - Per-version commit date (git show %cs) shown in the version filter + Version group-by labels. - Total $ (sum) column in the group breakdown beside the renamed Avg $. - Renamed routing metrics for clarity: selection_accuracy -> "Metric identified", routing_path_correct -> "Optimal route" (labels + glossary). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
judge_configs now crosses each playground model with each selected prompt, so a prompt sweep is a pure data change. --prompts takes names/indices (default: all registered variants); resolve_prompt_names validates them. --sample-like builds the sample over the EXACT outputs in a reference CSV (via mapping.select_by_outputs) instead of re-sampling — used to reuse the v1 sample's 40 outputs for composite_v2 so v2 pairs directly against the existing v1 baseline. prompts.prompt_names() lists the registry. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wires the agentic eval to a local vLLM-served judge (Qwen3.6-35B-A3B) so it can run as a 4th judge alongside the hosted playground models. Not yet run end-to-end (needs a 2x A40 allocation); validated by unit tests. - backends/qwen.json: framework vllm, base_url via $LOCAL_MODEL_URL, prices 0/0, temperature 0 + parallel_tool_calls false. Keyless (config returns "not-used"). - config._resolve_base_url: expand env vars in a backend base_url (the local server's host:port isn't known until the Slurm job lands); raise if unresolved. build_backend / sync_openai_client return the resolved URL. - config.metrics_url(base_url, framework): vLLM exposes Prometheus at the server ROOT /metrics, not /v1/metrics (NIM). runner passes ctx["framework"]. - create_eval_mapping --judge-backends: cross prompts with any set of judge backends (e.g. "playground,qwen" or "qwen"); --sample-like now also filters by backend so a qwen-only sample doesn't pull in the hosted rows for those outputs. - run_eval_batch.slurm: GPU_METRICS=1 keeps the /metrics scrape on (was hardcoded off) for the local judge. NEXT_STEPS: v2 result recorded (mixed) + the local-judge runbook + the two prompt-independent TBD fixes (history-sanitation, generation-time enforcement). 145 tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A fresh `pip install -r requirements.txt` was broken: unpinned `mcp` resolved to 2.0.0, which removed `mcp.client.streamable_http.streamablehttp_client` that the agent loop imports. Pin `mcp==1.28.1` (the tested version) and add `starlette`/ `uvicorn` (imported by analysis/serve_dashboard.py, previously missing). A venv built from requirements.txt now runs the full suite (145 passing). Also consolidate the two divergent venvs into one repo-local `.venv/` (gitignored) shared by both pipelines: eval SLURM scripts source `$SLURM_SUBMIT_DIR/.venv`, the upstream scripts source `../.venv` (run from scripts/). Add `*.log` to .gitignore. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…tic-eval skill Make the repo (esp. the MCP eval harness) legible + transferable to teammates and their AI agents: - README.md: keep the full upstream-inference walkthrough, add an "Agentic Metric-Eval & Dashboard" section (what it is + quickstart + pointers), a two-pipeline note, and refresh Next Steps to the current work (v2 prompt A/B, local Qwen judge, reliability fixes). - CLAUDE.md (new): terse agent-facing project memory — conventions, the generic-vs-task-specific reuse map, gotchas, and the pipeline name-collisions. - .env.example (new): template of required/optional env vars (no secret values). - .claude/skills/agentic-eval/SKILL.md (new): a run / read-results / extend playbook so an agent auto-discovers how to work with the harness. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.