diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 1b693ff3..2ec5a9d3 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -45,7 +45,7 @@ or under the root `tests/` directory. ```bash just lint # ruff check + ruff format --check just test-sanity # ~5s -just test-smoke-e2e # live: AppWorld + M3 (needs API keys, AppWorld, M3 containers) +just test-smoke-e2e # live: AppWorld + VAKRA (needs API keys, AppWorld, VAKRA containers) just test-regression # ~7s just security # bandit + pip-audit just ci # lint + test-regression + security (smoke is optional/manual, not included) diff --git a/PROJECT_OVERVIEW.md b/PROJECT_OVERVIEW.md index 620266d6..7cb342f5 100644 --- a/PROJECT_OVERVIEW.md +++ b/PROJECT_OVERVIEW.md @@ -1,7 +1,7 @@ # ๐Ÿงช CUGA Evaluation Framework - Project Overview ## ๐ŸŽฏ Purpose -This is an **evaluation framework** for testing the CUGA Agent (an AI agent system) across three different benchmark scenarios. It measures agent performance on real-world tasks involving API interactions, multi-hop reasoning, and web automation. +This is an **evaluation framework** for testing the CUGA Agent (an AI agent system) across three different benchmark scenarios. It measures agent performance on real-world tasks involving API interactions, multi-hop reasoning, and multi-app personal assistance. ## ๐Ÿ—๏ธ Architecture @@ -31,13 +31,13 @@ This is an **evaluation framework** for testing the CUGA Agent (an AI agent syst - 3 terminals needed: app, registry, eval script - Direct OpenAPI integration -#### **M3** (Multi-hop Q&A) +#### **VAKRA / M3** (Multi-hop Q&A) - Hockey statistics domain - 211 API endpoints for player/team/coach data - Tests multi-hop reasoning (questions requiring multiple API calls) - 2 terminals: registry, eval script -#### **AppWorld** (Web Automation) +#### **AppWorld** (Multi-App Personal Assistance) - 13 service integrations (Gmail, Spotify, Todoist, Venmo, etc.) - Tests complex multi-app workflows - OAuth2 authentication @@ -103,7 +103,7 @@ from cuga.sdk import CugaAgent uv run eval_bench_sdk.py # Terminal 3: Evaluation ``` -**M3/AppWorld** (2 terminals from `benchmarks/{name}/`): +**VAKRA/AppWorld** (2 terminals from `benchmarks/{name}/`): ```bash ./run_registry.sh # Terminal 1: MCP registry uv run eval_{name}.py # Terminal 2: Evaluation @@ -249,7 +249,7 @@ cuga-eval/ ### Evaluation Scripts - `benchmarks/bpo/eval_bench_sdk.py` - BPO evaluation - `benchmarks/oak_health_insurance/eval_bench_sdk.py` - Oak evaluation -- `benchmarks/m3/eval_m3.py` - M3 evaluation +- `benchmarks/m3/eval_m3.py` - VAKRA evaluation - `benchmarks/appworld/appworld_eval.py` - AppWorld evaluation ### Policies @@ -258,7 +258,7 @@ cuga-eval/ ### Data - `benchmarks/bpo/bpo_test_suite_v1.json` - BPO test cases (26 tasks) - `benchmarks/oak_health_insurance/oak_health_test_suite_v1.json` - Oak test cases -- `benchmarks/m3/hockey.json` - M3 test cases (19 samples) +- `benchmarks/m3/hockey.json` - VAKRA test cases (19 samples) - `benchmarks/appworld/eval_config.toml` - AppWorld task groups ## ๐Ÿค Contributing @@ -278,7 +278,7 @@ When adding features or fixing bugs: - [AGENTS.md](AGENTS.md) - Coding rules for AI agents - [BPO README](benchmarks/bpo/README.md) - [Oak Health Insurance README](benchmarks/oak_health_insurance/README.md) -- [M3 README](benchmarks/m3/README.md) +- [VAKRA / M3 README](benchmarks/m3/README.md) - [AppWorld README](benchmarks/appworld/README.md) - [CUGA Agent Repository](../cuga-agent) - The agent being evaluated diff --git a/README.md b/README.md index 8432de42..50826603 100644 --- a/README.md +++ b/README.md @@ -14,8 +14,8 @@ This repository contains: |-----------|-------------|--------| | **[BPO](benchmarks/bpo/README.md)** | BPO recruiting analytics with 32 tool APIs, policies, and error resilience testing | Recruiting Analytics | | **[Oak Health Insurance](benchmarks/oak_health_insurance/README.md)** | Healthcare insurance application with claims, coverage, and benefits tasks | Healthcare | -| **[M3](benchmarks/m3/README.md)** | Multi-hop question answering using hockey domain data | Knowledge Retrieval | -| **[AppWorld](benchmarks/appworld/README.md)** | Complex web application automation and task completion | Web Automation | +| **[VAKRA / M3](benchmarks/m3/README.md)** | Multi-hop, multi-source tool calling over 8,000+ APIs backed by real databases across 62 domains, plus domain-aligned documents | API & Knowledge Retrieval | +| **[AppWorld](benchmarks/appworld/README.md)** | Everyday consumer apps (email, shopping, payments, music) with multi-app personal tasks via APIs | Multi-App Personal Assistance | | **[tau2-bench (ฯ„ยฒ)](benchmarks/tau2/README.md)** | Multi-turn customer-service benchmark (airline/retail/telecom) with user simulator and env-state scoring | Customer Service | --- @@ -84,7 +84,7 @@ LANGFUSE_HOST="https://us.cloud.langfuse.com" ### 5. Per-benchmark setup -Steps 1โ€“4 above (clone, `setup_cuga.sh`, `uv venv && uv sync`, `.env`) are enough to run **BPO** and **Oak Health Insurance** out of the box. **M3**, **AppWorld**, and **tau2 (ฯ„ยฒ)** each need one extra setup step. The five subsections below are independent โ€” run only the ones for benchmarks you actually want to use. +Steps 1โ€“4 above (clone, `setup_cuga.sh`, `uv venv && uv sync`, `.env`) are enough to run **BPO** and **Oak Health Insurance** out of the box. **VAKRA / M3**, **AppWorld**, and **tau2 (ฯ„ยฒ)** each need one extra setup step. The five subsections below are independent โ€” run only the ones for benchmarks you actually want to use. At-a-glance: @@ -92,7 +92,7 @@ At-a-glance: |---|---|---| | BPO | None โ€” base `uv sync` is enough | โ€“ | | Oak Health Insurance | None โ€” base `uv sync` is enough | โ€“ | -| M3 | Yes (one-time) | `./setup_m3.sh` | +| VAKRA / M3 | Yes (one-time) | `./setup_m3.sh` | | AppWorld | Yes (one-time) | `git lfs install && ./setup_appworld.sh` | | tau2-bench (ฯ„ยฒ) | Yes (one-time) | `bash setup_tau2.sh && uv sync --group tau2` | @@ -104,16 +104,16 @@ BPO is ready to run after the base install. Skip ahead to [Quick Start](#-quick- Oak is ready to run after the base install. See [`benchmarks/oak_health_insurance/README.md`](benchmarks/oak_health_insurance/README.md). -#### M3 (Vakra) Setup +#### VAKRA / M3 Setup -If you plan to run the M3 benchmark: +If you plan to run the VAKRA benchmark (folders, scripts and CLI values (e.g. `--benchmark m3`, `--m3-data`) use the name `m3`): ```bash # Hugging Face token for downloading ~30 GB of benchmark data (read access is enough). # Create one at https://huggingface.co/settings/tokens export HF_TOKEN=hf_your_token_here -# Clones vakra, downloads data, builds containers. +# Clones VAKRA, downloads data, builds containers. ./setup_m3.sh ``` @@ -170,7 +170,7 @@ See [`benchmarks/tau2/README.md`](benchmarks/tau2/README.md) for full details. #### Running multiple benchmarks -The setup steps are independent โ€” run each one when you want that benchmark, in any order. The base `uv sync` always succeeds regardless of which benchmark-specific setups have or haven't been run, so a fresh checkout or CI job that only needs BPO/Oak/M3 never has to touch AppWorld, and vice versa. +The setup steps are independent โ€” run each one when you want that benchmark, in any order. The base `uv sync` always succeeds regardless of which benchmark-specific setups have or haven't been run, so a fresh checkout or CI job that only needs BPO/Oak/VAKRA never has to touch AppWorld, and vice versa. --- @@ -206,7 +206,7 @@ Available profiles: `gpt-oss`, `gpt4o`, `gpt4.1`, `opus4.5` The evaluation framework supports the following agent types: - **`cuga`** (default) - The full CUGA agent with planning, reflection, and advanced features -- **`react`** - A lightweight ReAct-style agent. On BPO/M3 it emits JSON tool calls (true ReAct: decisions live between turns). On AppWorld it runs through the Python REPL but the short prompt encourages small one-API-per-step blocks, so behavior is still ReAct-flavored โ€” decisions remain between turns. +- **`react`** - A lightweight ReAct-style agent. On BPO/VAKRA it emits JSON tool calls (true ReAct: decisions live between turns). On AppWorld it runs through the Python REPL but the short prompt encourages small one-API-per-step blocks, so behavior is still ReAct-flavored โ€” decisions remain between turns. - **`codeact`** - **AppWorld only.** Uses an `instructions.txt` few-shot prompt that demonstrates richer code blocks (loops, conditionals, multiple API calls + computation per step). Encodes workflow logic **inside** the generated code instead of deferring every branch to the next LLM turn. Also adds stop sequences, a larger trim budget, and pre-authentication. Other benchmarks reject `--agent codeact` with a clear error. > **Note on naming:** The CodeAct vs ReAct distinction here is about *where decisions live* (between turns vs in-code), not about the action mechanism. AppWorld's `--agent react` and `--agent codeact` share the same Python-REPL harness but use prompts that push the model in different directions. @@ -233,11 +233,11 @@ Use the `--agent` flag to select the agent type: **What is the ReAct Agent?** The ReAct (Reasoning + Acting) agent is a lightweight, prompt-based agent that uses iterative reasoning with tool calls and observations. It provides a simpler alternative to the full CUGA agent for baseline comparisons and research. The ReAct agent: -- Works with BPO, M3 (single-turn only), and AppWorld via the `--agent react` flag +- Works with BPO, VAKRA (single-turn only), and AppWorld via the `--agent react` flag - Uses the same MCP tools and evaluation infrastructure - Enables direct comparison of agent architectures on the same tasks -> **Note:** Oak Health Insurance only supports the CUGA agent โ€” its `eval.sh` rejects any non-`cuga` `--agent` value (both `react` and `codeact`). M3's `--multiturn` mode is also CUGA-only and will error out if combined with `--agent react`. `--agent codeact` is supported only by AppWorld. +> **Note:** Oak Health Insurance only supports the CUGA agent โ€” its `eval.sh` rejects any non-`cuga` `--agent` value (both `react` and `codeact`). VAKRA's `--multiturn` mode is also CUGA-only and will error out if combined with `--agent react`. `--agent codeact` is supported only by AppWorld. --- @@ -303,7 +303,7 @@ cd benchmarks/oak_health_insurance && ./eval.sh --task approved_claims ./benchmarks/oak_health_insurance/compare.sh --models gpt-oss,gpt4o --runs 2 --dry-run ``` -### M3 Benchmark +### VAKRA / M3 Benchmark ```bash # โ”€โ”€ Eval (single-turn) โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ @@ -316,13 +316,13 @@ cd benchmarks/m3 && ./eval.sh --task hockey_395_0 # With ReAct agent (single-turn only โ€” react + --multiturn is rejected) ./benchmarks/m3/eval.sh --agent react --task hockey_395_0 -# M3-specific: multi-turn evaluation (CUGA agent only) +# VAKRA-specific: multi-turn evaluation (CUGA agent only) ./benchmarks/m3/eval.sh --multiturn -# M3-specific: filter by difficulty +# VAKRA-specific: filter by difficulty ./benchmarks/m3/eval.sh --difficulty easy -# M3-specific: run a predefined train/test split from eval_config.toml +# VAKRA-specific: run a predefined train/test split from eval_config.toml ./benchmarks/m3/eval.sh --m3-data benchmarks/m3/data/small_train.zip --eval-key train # โ”€โ”€ Compare by model โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ @@ -410,10 +410,10 @@ Benchmark-specific flags: | Flag | Benchmark | Effect | |---|---|---| | `--eval-key ` | AppWorld | Run a predefined task group from `benchmarks/appworld/eval_config.toml` (e.g. `test_challenge_easy`, `test_normal_all_hard`). | -| `--eval-key train\|test` | M3 | Restrict `--m3-data` to a predefined split from `benchmarks/m3/eval_config.toml`, applied before `--task`/`--domain`/`--capability`. Requires `--m3-data`. | +| `--eval-key train\|test` | VAKRA | Restrict `--m3-data` to a predefined split from `benchmarks/m3/eval_config.toml`, applied before `--task`/`--domain`/`--capability`. Requires `--m3-data`. | | `--specific-task-levels 1 2 3` | AppWorld | Filter by difficulty level. | -| `--multiturn` | M3 | Run multi-turn evaluation (CUGA agent only). | -| `--difficulty easy\|medium\|hard` | M3, Oak | Filter tasks by difficulty. | +| `--multiturn` | VAKRA | Run multi-turn evaluation (CUGA agent only). | +| `--difficulty easy\|medium\|hard` | VAKRA, Oak | Filter tasks by difficulty. | | `--no-policies` | BPO | Disable CUGA policies (for baselining). | ### Reproducibility Bundles @@ -546,9 +546,9 @@ Bundles are stored in `benchmarks/{benchmark}/evaluation_bundles/` and are git-i Named experiments write each task's result to `results/partial/` as it completes, so a kill mid-run preserves everything finished so far. Resuming merges partials and continues with the remaining (and previously-failed) tasks. -All benchmarks' evaluators also converge crash/ interrupt handling onto the on-disk partials (not just the in-memory list), so a task finished moments before a crash is not lost even under concurrent batch execution (M3 config mode). +All benchmarks' evaluators also converge crash/ interrupt handling onto the on-disk partials (not just the in-memory list), so a task finished moments before a crash is not lost even under concurrent batch execution (VAKRA config mode). -M3's legacy (unnamed) `eval.sh`/`compare.sh` path additionally salvages a best-effort bundle when interrupted: +VAKRA's legacy (unnamed) `eval.sh`/`compare.sh` path additionally salvages a best-effort bundle when interrupted: - **Partial result files**: if the evaluator is interrupted or hits an unexpected exception mid-run, it saves whatever task results were already collected to `benchmarks/m3/results/m3_config_partial_*.json` (or `m3_config_no_gt_partial_*.json` with `--no-ground-truth`), distinguishable from complete-run files by the `partial` prefix. - **Bundle on exit**: `create_bundle`/`create_compare_bundle` are idempotent and run from both the success path and the script's `cleanup` trap (`trap cleanup EXIT INT TERM ERR`), so a bundle is produced exactly once whether the run finishes normally, is interrupted, or crashes โ€” picking up the freshest result file written during that run. @@ -626,7 +626,7 @@ When Langfuse is enabled, evaluations collect: - `full_execution_time` - Total execution time - `total_cache_input_tokens` - Cached token usage -For m3 and the SDK appworld path, this is now the fallback: the eval tool +For VAKRA and the SDK AppWorld path, this is now the fallback: the eval tool reads a `RunReceipt` directly from the agent instead (no Langfuse round-trip), which additionally breaks tokens down into `input_tokens`, `output_tokens`, `cache_read_tokens`, `reasoning_tokens`, plus `tool_call_count`, `llm_time_s`, @@ -663,7 +663,7 @@ To add a new benchmark: 1. Create a new directory under `benchmarks/` 2. Add a `README.md` with detailed instructions 3. Create a `config/{benchmark}.env` file -4. Implement an `eval.sh` and Python evaluator. The `templates/` directory contains a few starting points (`eval_loop_template.py`, `calculate_test_score.py`, `simple_example.json`); for the full benchmark layout, copy from an existing one (BPO and M3 are the most complete references). +4. Implement an `eval.sh` and Python evaluator. The `templates/` directory contains a few starting points (`eval_loop_template.py`, `calculate_test_score.py`, `simple_example.json`); for the full benchmark layout, copy from an existing one (BPO and VAKRA are the most complete references). 5. Add MCP servers configuration if needed For examples, see existing benchmark implementations. diff --git a/benchmarks/appworld/README.md b/benchmarks/appworld/README.md index 80d31f59..f458493b 100644 --- a/benchmarks/appworld/README.md +++ b/benchmarks/appworld/README.md @@ -2,8 +2,8 @@ ## ๐Ÿ“Š Overview -The AppWorld benchmark evaluates agent capabilities with complex web application automation and task completion. It tests the agent's ability to: -- Navigate and interact with web applications +The AppWorld benchmark evaluates multi-app personal assistance: personal tasks that span everyday consumer apps (email, shopping, payments, music) and are completed through the apps' APIs. It tests the agent's ability to: +- Call app APIs to read and change app state - Complete multi-step tasks across different apps - Handle realistic application workflows - Reason about application state and context @@ -75,7 +75,7 @@ uv sync # base deps only (AppWorld is removed from venv; # re-add with --group appworld) ``` -Both forms succeed regardless of whether the appworld clone exists. The `appworld` group is opt-in, so running other benchmarks (BPO, M3, Oak) never requires AppWorld to be installed. +Both forms succeed regardless of whether the appworld clone exists. The `appworld` group is opt-in, so running other benchmarks (BPO, VAKRA, Oak) never requires AppWorld to be installed. --- ## ๐Ÿš€ Running the Benchmark diff --git a/benchmarks/m3/README.md b/benchmarks/m3/README.md index b3340540..6e7e61b2 100644 --- a/benchmarks/m3/README.md +++ b/benchmarks/m3/README.md @@ -1,8 +1,8 @@ -# ๐Ÿ’ M3 Benchmark +# ๐Ÿ”ท VAKRA / M3 Benchmark ## ๐Ÿ“Š Overview -The M3 benchmark evaluates multi-hop question answering capabilities across multiple domains. It tests the agent's ability to: +The [VAKRA](https://github.com/IBM/vakra) benchmark (folders, scripts and CLI values (e.g. `--benchmark m3`, `--m3-data`) use the name `m3`) evaluates multi-hop, multi-source tool calling: agents answer questions using 8,000+ locally hosted APIs backed by real databases across 62 domains, plus domain-aligned document collections. It tests the agent's ability to: - Answer single-hop questions requiring one API call - Handle multi-hop questions requiring multiple API calls - Reason across different data sources @@ -29,12 +29,12 @@ The M3 benchmark evaluates multi-hop question answering capabilities across mult 3. **VAKRA Setup** - Download the Vakra data (M3 dataset) and start Docker containers: + Download the VAKRA data and start Docker containers: ```bash export HF_TOKEN=hf_your_token_here bash ./setup_m3.sh ``` - This setup vakra in `vendor/vakra` + This sets up VAKRA in `vendor/vakra` See [VAKRA setup guide](https://github.com/IBM/vakra/blob/main/setup.md) for details on Docker setup, environments, and data downloads. @@ -74,7 +74,7 @@ The M3 benchmark evaluates multi-hop question answering capabilities across mult # OPENAI_BASE_URL=your-endpoint ``` - **M3 benchmark configuration:** (cuga-eval/benchmarks/m3/config/m3.env) + **VAKRA benchmark configuration:** (cuga-eval/benchmarks/m3/config/m3.env) ```bash # Copy example config cp benchmarks/m3/config/m3.env.example benchmarks/m3/config/m3.env @@ -100,7 +100,7 @@ The M3 benchmark evaluates multi-hop question answering capabilities across mult ### Named experiments, resume & background -M3 config-mode runs can span hundreds of domains โ€” use named experiments to survive crashes and long runtimes. Resume skips successfully completed `(task, domain)` pairs; failed domains are re-attempted. +VAKRA config-mode runs can span many domains across capabilities โ€” use named experiments to survive crashes and long runtimes. Resume skips successfully completed `(task, domain)` pairs; failed domains are re-attempted. ```bash # Full small_train run as a named workspace @@ -130,7 +130,7 @@ Partial results land in `evaluation_bundles//results/partial/` as `_ bash benchmarks/m3/eval.sh --task hockey_395_0 ``` -### M3 small_train.zip recipes (Vakra-scored) +### VAKRA small_train.zip recipes (VAKRA-scored) The bundled `data/small_train.zip` is the canonical eval set: 20 domains ร— 10 tasks = **200 test cases** total, split across two capabilities: @@ -144,7 +144,7 @@ The bundled `data/small_train.zip` is the canonical eval set: 20 domains ร— 10 t All commands below use `caffeinate` so macOS doesn't sleep mid-run, and pass `--m3-data benchmarks/m3/data/small_train.zip` so the legacy keyword data files -under `data/` aren't touched. Pass/fail comes from the Vakra LLM judges +under `data/` aren't touched. Pass/fail comes from the VAKRA LLM judges (correctness ร— groundedness ร— exact-match); set `M3_VAKRA_LIVE_MCP=on` if you want byte-identical-to-CLI verdicts and have the capability container running. @@ -219,15 +219,15 @@ corpus) with: uv run python benchmarks/m3/scripts/generate_eval_split.py --ratio 0.5 --seed 42 ``` -### M3 unlabeled / no-ground-truth recipes +### VAKRA unlabeled / no-ground-truth recipes -For datasets that ship only an `input/` side (e.g. the Vakra test set under +For datasets that ship only an `input/` side (e.g. the VAKRA test set under `vendor/vakra/data/test/capability_2_dashboard_apis/`), pass `--no-ground-truth`. The flag tells the loader to skip the missing `output/`, -runs the agent against the test domains, **skips Vakra/judge scoring entirely**, +runs the agent against the test domains, **skips VAKRA/judge scoring entirely**, and dumps each domain's prediction to `benchmarks/m3/results/_vakra/prediction/.json` in the same shape the -Vakra evaluator expects. The summary is a per-sample tool-call count instead +VAKRA evaluator expects. The summary is a per-sample tool-call count instead of pass/fail. The data path can be the capability dir itself or its parent โ€” the loader @@ -242,13 +242,13 @@ caffeinate -i bash benchmarks/m3/eval.sh \ --no-ground-truth \ --capability m3_task_2 --domain california_schools --max-samples 1 -# 2) cuga, full vakra test set for capability 2 (17 domains) +# 2) cuga, full VAKRA test set for capability 2 (17 domains) caffeinate -i bash benchmarks/m3/eval.sh \ --m3-data vendor/vakra/data/test/capability_2_dashboard_apis \ --no-ground-truth \ --capability m3_task_2 --bundle-zip -# 3) cuga, full vakra test set across all capabilities the loader finds +# 3) cuga, full VAKRA test set across all capabilities the loader finds caffeinate -i bash benchmarks/m3/eval.sh \ --m3-data vendor/vakra/data/test \ --no-ground-truth --bundle-zip @@ -266,7 +266,7 @@ holds the raw run metadata. - For zip mode, capability-2 / capability-3 Docker containers must be running (they're what the registry talks to). The first command in the script will surface a clear error if a container is missing. -- Per-task Vakra detail (judge scores + explanations) prints inline; the full +- Per-task VAKRA detail (judge scores + explanations) prints inline; the full judge JSON lands at `benchmarks/m3/results/_vakra/results.json`. @@ -412,7 +412,7 @@ FastAPI REST wrapper MCP server (/app/m3-rest/mcp_server.py) โ†“ FastAPI REST API (localhost:8000 inside container) โ†“ -M3 database tools (.sqlite) +VAKRA database tools (.sqlite) ``` **Registry Mode Features:** @@ -452,7 +452,7 @@ The evaluation script performs: - **Trajectory Data**: `benchmarks/m3/logging/trajectory_data/` - **Activity Logs**: `benchmarks/m3/logging/` -The ground truth format file is compatible with the M3 benchmark evaluation format and includes: +The ground truth format file is compatible with the VAKRA benchmark evaluation format and includes: - Task metadata (uuid, domain, num_turns) - Dialogue structure with queries and responses - Tool call sequences with arguments and results @@ -514,10 +514,10 @@ The `--domain` flag must match available databases in the container: - **Ground Truth Format**: `benchmarks/m3/results//task_1/.json` - **Logs**: `benchmarks/m3/logging/` -The ground truth format output matches the M3 benchmark evaluation format used by `eval_m3.py`, with the same structure: +The ground truth format output matches the VAKRA benchmark evaluation format used by `eval_m3.py`, with the same structure: - Organized by timestamp and task folders - One JSON file per domain -- Compatible with M3 benchmark evaluation tools +- Compatible with VAKRA benchmark evaluation tools ### Architecture