docs(weave): add "Evaluate your AI agent with Weave" tutorial - #2945
docs(weave): add "Evaluate your AI agent with Weave" tutorial#2945anastasiaguspan wants to merge 3 commits into
Conversation
Add a new Get Started tutorial covering agent evaluation with the Weave Agents/Conversations workflow: tracing an agent as a conversation of turns and tool calls, scoring single-turn and multi-turn task completion with an LLM judge via EvaluationLogger, and comparing agent versions. Includes a companion Colab notebook with the full runnable source. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Resume prompt (for a fresh Claude Code session)
|
|
Preview deployment for your docs. Learn more about Mintlify Previews.
|
📚 Mintlify Preview Links✨ Added (1 total)📄 Pages (1)
📝 Changed (1 total)⚙️ Other (1)
🤖 Generated automatically when Mintlify deployment succeeds |
🔗 Link Checker ResultsPreview: https://wb-21fd5541-aguspan-docs-agent-evals.mintlify.site |
|
Preview deployment for your docs. Learn more about Mintlify Previews.
|
HiveMind Sessions2 sessions · 1h 43m · $39
View all sessions in HiveMind → Run |
Fold per-language prose and code into <Tabs> on the init, conversation-trace, Score, Compare, and Multi-turn sections, keeping shared prose outside the tabs. Demote task_completion to an example LLM judge rather than a Weave concept, trim the scope note and Extend section, and fix the compare_evals and scorers links. Expand the accompanying Python notebook to a runnable 20 cells: %pip install, getpass key entry, run_agent_turn returning (reply, transcript), judge_task_completion, and Evals-tab navigation clues. The Python path is verified end to end. The TypeScript tabs still reference an accompanying TypeScript file that is not yet in the repo, and their claim that conversation spans link to eval rows does not hold on weave JS 0.16.3 — two SDK bugs (complete-mode project writes, EvalLinkSpanProcessor registration) are with engineering. Resolve both before this page ships. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 participant
Summary
New Get Started tutorial, Evaluate your AI agent with Weave (
weave/agent-evals.mdx), with a companion Colab (weave/cookbooks/source/agent_evals.ipynb) and a nav entry under the Weave Get Started group.It teaches agent evaluation with the Agents/Conversations workflow (OTel spans, not Ops/Calls): trace an agent as turns and tool calls, score single-turn and multi-turn task completion with an LLM judge via
EvaluationLogger, and compare agent versions.What's included
weave/agent-evals.mdx— the page. Inline code is intentionally reduced to the Weave surface; the full runnable source lives in the Colab.weave/cookbooks/source/agent_evals.ipynb— companion notebook (agent on Claude Sonnet, judge on Claude Opus).docs.json—weave/agent-evalsadded to the Get Started group (English only; localized nav untouched).Verification
The code patterns were run end-to-end against the real Weave and Anthropic SDKs (agent trace, single-turn eval, multi-turn eval) publishing to a live Weave project. The notebook was validated (JSON parses; every code cell parses).
Recommendations for technical review
Technical accuracy
implicitly_patch_integrations: Falserationale (avoids duplicate legacy Op/Call traces alongside the manual Conversation SDK spans) matches current behavior.claude-sonnet-5(agent) andclaude-opus-4-8(judge) are correct for the docs' target audience.Missing content / confirm
convo_id,new_id(),run_agent_turn,run_tool,response_tool_calls,task_completion,judge_task) defined in a bulleted list and deferred to the Colab — confirm this is enough for a self-contained read.weave/cookbooks/source/agent_evals.ipynbonmain. If the cookbook build renames or generates notebooks, adjust the URLs./weave/guides/evaluation/evaluation_loggerexists onmain; the in-flight branchcodex/link-agent-evals-docsedits that same guide (linked agent eval traces) — coordinate if both land.Related
weave/tutorial-rag.🤖 Generated with Claude Code