docs: fix AppWorld domain and use VAKRA naming in READMEs by Sergey-Zeltyn · Pull Request #210 · cuga-project/cuga-eval

Sergey-Zeltyn · 2026-10-20T16:19:35Z

Documentation Pull Request

Related Issue

Closes #209

Description

Corrects the AppWorld domain (it is not web automation) and refers to the M3 benchmark by its proper name, VAKRA, across the READMEs. m3 is kept wherever it is a folder, script, file name or CLI flag.

Type of Changes

Changes Made

AppWorld (root README table + benchmarks/appworld/README.md overview): domain is now Multi-App Personal Assistance; description is "Everyday consumer apps (email, shopping, payments, music) with multi-app personal tasks via APIs".
VAKRA naming (root README + benchmarks/m3/README.md): headings and tables say VAKRA / M3, prose says VAKRA; added a note that folders, scripts and the --benchmark value still use m3. Paths, flags and identifiers (benchmarks/m3/, setup_m3.sh, --m3-data, eval_m3.py, vendor/vakra) are unchanged.
VAKRA description: replaced the outdated "multi-hop QA using hockey domain data" with the upstream description (multi-hop, multi-source tool calling over 8,000+ APIs backed by real databases across 62 domains, plus domain-aligned documents); domain column is now API & Knowledge Retrieval.

Verification

I have reviewed the documentation for accuracy
I have checked for spelling and grammar errors
Links and references are working correctly
Code examples (if any) have been tested

Checklist

My documentation follows the project's documentation style
I have performed a self-review of my changes

🤖 Generated with Claude Code

Summary by CodeRabbit

Documentation
- Updated benchmark documentation to consistently use the VAKRA / M3 naming.
- Expanded descriptions of VAKRA’s multi-hop, multi-source API evaluation across 62 domains and 8,000+ APIs.
- Clarified setup, configuration, evaluation, agent support, resilience, metrics, and development guidance.
- Updated AppWorld documentation to describe personal-assistance tasks performed through application APIs.

- AppWorld: replace "web automation" with Multi-App Personal Assistance (everyday consumer apps with multi-app personal tasks via APIs) - Refer to the M3 benchmark as VAKRA / M3 (headings, tables) or VAKRA (prose); keep `m3` in paths, scripts, and CLI flags - Update the VAKRA description to match upstream (multi-hop, multi-source tool calling over 8,000+ APIs across 62 domains) Closes #209 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Sergey Zeltyn <sergeyz@il.ibm.com>

coderabbitai · 2026-10-20T16:19:44Z

Important

🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Repository: cuga-project/cuga-eval/.coderabbit.yaml
Review profile: CHILL
Plan: Advanced
Run ID: 377fd0b9-6959-457f-85e7-55c7c14c4ec2

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: cuga-project/cuga-eval/.coderabbit.yaml
Review profile: CHILL
Plan: Advanced
Run ID: e1f15afa-e44f-4d51-8cf7-b7b510cb34fa

📥 Commits

Reviewing files that changed from the base of the PR and between 059701b and be7c8cf.

📒 Files selected for processing: 3

README.md
benchmarks/appworld/README.md
benchmarks/m3/README.md

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

📝 Walkthrough

Documentation now identifies the benchmark as VAKRA / M3 and describes AppWorld as API-driven multi-app assistance. Setup and evaluation references use VAKRA terminology while retaining the existing m3 paths and command values.

Changes: Benchmark documentation

Layer / File(s)	Summary
Benchmark descriptions and setup `README.md`, `benchmarks/appworld/README.md`, `benchmarks/m3/README.md`	The overview and setup content identifies the benchmark as VAKRA / M3, retains the `m3` paths and command values, and describes AppWorld as API-driven multi-app assistance.
Agent and evaluation guidance `README.md`, `benchmarks/m3/README.md`	Agent support, command references, flags, configuration, recipes, judges, and test-data instructions use VAKRA terminology.
Operational and results documentation `README.md`, `benchmarks/m3/README.md`	Crash handling, Langfuse fallback, architecture references, result formats, and new-benchmark guidance use VAKRA terminology.

Priority: ⬇️ Low

Estimated code review effort: 2 (Simple) | ~8 minutes

Change: Other · Severity of issue fixed: Low

Merge Risk: ⚪ Minimal · up to be7c8

The documentation updates are mergeable; no actionable implementation risk remains.

Pre-merge checks: 5

Check name	Status	Explanation
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title clearly and concisely identifies both main documentation changes: correcting the AppWorld domain and adopting VAKRA naming in the READMEs.
Linked Issues check		Issue `#209` requires correction of the AppWorld description and inclusion of VAKRA terminology for M3. The root README and AppWorld README describe AppWorld as multi-app personal assistance through app…
Out of Scope Changes check		The pull request changes only `README.md`, `benchmarks/appworld/README.md`, and `benchmarks/m3/README.md`. The changes update AppWorld facts, VAKRA naming, VAKRA descriptions, and related benchmark re…
Docstring Coverage		No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…

Full details: Linked Issues check

Explanation

Issue #209 requires correction of the AppWorld description and inclusion of VAKRA terminology for M3. The root README and AppWorld README describe AppWorld as multi-app personal assistance through application APIs, not web automation. The root README and M3 README use VAKRA / M3 naming and retain m3 in paths, scripts, and CLI identifiers. No coding or automated-test requirement applies beyond these documentation changes.

Full details: Out of Scope Changes check

Explanation

The pull request changes only README.md, benchmarks/appworld/README.md, and benchmarks/m3/README.md. The changes update AppWorld facts, VAKRA naming, VAKRA descriptions, and related benchmark references. These changes directly support issue #209 and do not demonstrate unrelated scope.

Full details: Docstring Coverage

Explanation

No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0 files. (3 skipped: 3 unsupported.)

✨ Finishing Touches

🧪 Generate unit tests (beta)

Commit to this branch
Create a new PR

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

haroldship

Review of PR #210

Verdict: Approve. Some small fixes are recommended, but none of them are mandatory.

What is correct

The PR does not change any m3 identifier. Paths, script names, CLI flags, environment variables and --benchmark m3 values all stay the same.
The PR renames three headings. No link in the repository points to these headings, so no link breaks.
The VAKRA data agrees with vendor/vakra/README.md: 8,000+ APIs, real databases, 62 domains and domain-aligned documents.
The domain label "API & Knowledge Retrieval" agrees with the VAKRA name.
The AppWorld text is correct. The harness calls app APIs. It does not control a web browser.
The tables keep 3 columns, so they render correctly.
The PR has no unrelated changes.

Fixes (recommended)

PROJECT_OVERVIEW.md:40: The heading is "AppWorld (Web Automation)". This is the error that #209 tells about. If this PR does not fix it, #209 closes and the error stays in the repository. Change the heading to "AppWorld (Multi-App Personal Assistance)".
benchmarks/appworld/README.md:78: The text says "(BPO, M3, Oak)". The PR changes this file, but it does not change this name. The root README uses "BPO/Oak/VAKRA". Change "M3" to "VAKRA".
benchmarks/m3/README.md:1: The heading has a hockey emoji (🏒). The PR removes the "hockey domain" description everywhere else. The emoji tells readers that the benchmark is about hockey. Remove the emoji, or use a neutral emoji.
benchmarks/m3/README.md:103: The text says "can span hundreds of domains". Line 5 says "62 domains". A reader can think that these two numbers do not agree. Change the text to "can span many domains".
README.md:109 and benchmarks/m3/README.md:5: The two scope notes are different. README.md:109 gives only the folder, the scripts and the --benchmark value. benchmarks/m3/README.md:5 also gives the CLI flags. Use the same text in the two files.

Follow-up (for a new issue)

These documents also have old text. This PR does not need to fix them.

PROJECT_OVERVIEW.md:4 says "web automation".
PROJECT_OVERVIEW.md:35-37 says "Hockey statistics domain" and "211 API endpoints".
benchmarks/m3/CONTAINER_SETUP.md and CONTRIBUTING.md:48 use "M3" in the text. This is not incorrect, but it is not the same as the new READMEs.

haroldship · 2026-10-21T12:30:41Z

@@ -1,8 +1,8 @@
-# 🏒 M3 Benchmark
+# 🏒 VAKRA / M3 Benchmark


The heading has a hockey emoji (🏒). The PR removes the "hockey domain" description everywhere else. The emoji tells readers that the benchmark is about hockey. Remove the emoji, or use a neutral emoji.

Fixed in c49f65d: replaced 🏒 with 🔷, the emoji used by the upstream VAKRA README.

haroldship · 2026-10-21T12:30:41Z

 ## 📊 Overview

-The M3 benchmark evaluates multi-hop question answering capabilities across multiple domains. It tests the agent's ability to:
+The [VAKRA](https://github.com/IBM/vakra) benchmark (named `m3` in this repo's folders, scripts and CLI flags) evaluates multi-hop, multi-source tool calling: agents answer questions using 8,000+ locally hosted APIs backed by real databases across 62 domains, plus domain-aligned document collections. It tests the agent's ability to:


This scope note gives the folders, the scripts and the CLI flags. The note in README.md:109 gives only the folder, the scripts and the --benchmark value. Use the same text in the two files.

Fixed in c49f65d: both files now use the same note — folders, scripts and CLI values (e.g. --benchmark m3, --m3-data) use the name m3.

haroldship · 2026-10-21T12:30:41Z

 ### Named experiments, resume & background

-M3 config-mode runs can span hundreds of domains — use named experiments to survive crashes and long runtimes. Resume skips successfully completed `(task, domain)` pairs; failed domains are re-attempted.
+VAKRA config-mode runs can span hundreds of domains — use named experiments to survive crashes and long runtimes. Resume skips successfully completed `(task, domain)` pairs; failed domains are re-attempted.


The text says "can span hundreds of domains". Line 5 says "62 domains". A reader can think that these two numbers do not agree. Change the text to "can span many domains".

Fixed in c49f65d: now "can span many domains across capabilities". VAKRA has 62 distinct domains; config-mode runs are per (capability, domain) file, which is where the larger count came from.

haroldship · 2026-10-21T12:30:41Z

+#### VAKRA / M3 Setup

-If you plan to run the M3 benchmark:
+If you plan to run the VAKRA benchmark (folder, scripts and `--benchmark` value use the name `m3`):


This scope note gives only the folder, the scripts and the --benchmark value. The note in benchmarks/m3/README.md:5 also gives the CLI flags. Use the same text in the two files.

Fixed in c49f65d: both files now use the same note — folders, scripts and CLI values (e.g. --benchmark m3, --m3-data) use the name m3.

- PROJECT_OVERVIEW: AppWorld is Multi-App Personal Assistance, not web automation; use VAKRA naming for the M3 benchmark - AppWorld README and CONTRIBUTING: M3 -> VAKRA in prose - M3 README: neutral heading emoji; "many domains across capabilities" instead of "hundreds of domains" - Use the same `m3` naming note in README and the M3 README Refs #209 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Sergey Zeltyn <sergeyz@il.ibm.com>

Sergey-Zeltyn · 2026-10-21T13:21:13Z

Thanks for the review. Pushed c49f65d with all five suggestions:

PROJECT_OVERVIEW.md: AppWorld heading is now Multi-App Personal Assistance, and the "web automation" phrase on line 4 is fixed too. I also renamed M3 → VAKRA in that file's headings and prose (the hockey/211-endpoint bullets are unchanged).
benchmarks/appworld/README.md:78: "(BPO, VAKRA, Oak)".
3–5. Replied inline.

Also changed CONTRIBUTING.md:48 to use VAKRA.

Sergey-Zeltyn requested a review from haroldship October 20, 2026 16:19

haroldship approved these changes Oct 21, 2026

View reviewed changes

Sergey-Zeltyn merged commit 3ef04c3 into main Oct 21, 2026
6 checks passed

Sergey-Zeltyn deleted the docs/209-readme-appworld-vakra branch October 21, 2026 13:22

Navigation Menu

Repository navigation

docs: fix AppWorld domain and use VAKRA naming in READMEs - #210

docs: fix AppWorld domain and use VAKRA naming in READMEs#210
Sergey-Zeltyn merged 2 commits into
mainfrom
docs/209-readme-appworld-vakra

Sergey-Zeltyn commented Oct 20, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Oct 20, 2026 •

edited

Loading

Uh oh!

haroldship left a comment

Uh oh!

haroldship Oct 21, 2026

Uh oh!

Sergey-Zeltyn Oct 21, 2026

Uh oh!

haroldship Oct 21, 2026

Uh oh!

Sergey-Zeltyn Oct 21, 2026

Uh oh!

haroldship Oct 21, 2026

Uh oh!

Sergey-Zeltyn Oct 21, 2026

Uh oh!

haroldship Oct 21, 2026

Uh oh!

Sergey-Zeltyn Oct 21, 2026

Uh oh!

Sergey-Zeltyn commented Oct 21, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		@@ -1,8 +1,8 @@
		# 🏒 M3 Benchmark
		# 🏒 VAKRA / M3 Benchmark

Conversation

Sergey-Zeltyn commented Oct 20, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Documentation Pull Request

Related Issue

Description

Type of Changes

Changes Made

Verification

Checklist

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

haroldship left a comment

Choose a reason for hiding this comment

Review of PR #210

What is correct

Fixes (recommended)

Follow-up (for a new issue)

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Sergey-Zeltyn commented Oct 21, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Sergey-Zeltyn commented Oct 20, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Oct 20, 2026 •

edited

Loading