Why
An external field-test review of v0.0.593 (article, Chinese) graded the project B, with the deciding caveat that the full RAG pipeline was unmeasured: no Memgraph/Qdrant running, no model keys — so end-to-end accuracy, query latency, token cost, and indexing time for large monorepos are unknown to an evaluator deciding whether the two databases are "worth feeding."
The deterministic harness in evals/ already answers the graph-layer accuracy question decisively (e.g. retrieval on django/django: graph F1 0.957 vs grep_call 0.789), and its README even names the missing piece — the section "Next step: agentic resolved-rate (out of scope here)" (evals/README.md, ~line 1191) says it "is tracked separately" — but no tracking issue exists. This is that issue.
Deliverables — four benchmarks
1. Agentic resolved-rate (graph tools vs grep, A/B)
The design evals/README.md already specifies (mirroring GitLab GKG's evaluation): hold one agent and one model fixed, vary only the tools (cgr graph/MCP tools vs plain grep), run a SWE-bench-Lite-style issue set (or a curated in-repo question set as a cheaper first cut), report resolved rate per condition. Needs an LLM key, a container harness, and enough runs per condition for a stable number. This is the headline accuracy figure.
2. Query latency on the real stack
Bring up the real stack (cgr daemon — the compose file already binds loopback), index a pinned corpus, replay a fixed query set through the MCP server, record per-query wall-clock (p50/p95). This measures what the in-memory eval harness deliberately bypasses: Memgraph/Qdrant round-trips.
3. Token cost per answer
Same harness as (2)/(1): sum model-API usage per question, report mean tokens per answered question for graph-tools vs grep-tools conditions. This directly answers the reviewer's "is the graph worth the operational overhead" — if graph tools resolve in fewer tool-call round-trips, it shows up here as fewer tokens.
4. Indexing time on large monorepos
Time cgr index on 2–3 pinned large repos (e.g. django/django — already an eval corpus — plus one large polyglot monorepo), report wall-clock, peak RSS, and node/edge counts. Track across releases so indexing-time regressions are visible. No model key needed; CI-friendly.
Deliverable — README front-page comparison table
Results must land on the front page of the README as a benchmark comparison table, in the style AI labs use to present model-vs-model results: cgr (graph tools) as one column, grep-only baseline as the other, one row per metric, with the corpus/model/version pinned in a caption. Sketch:
| Benchmark (corpus, model) |
cgr graph tools |
grep baseline |
| Resolved rate — SWE-bench-Lite subset |
X% |
Y% |
| Retrieval F1 — file-level call localization (django) |
0.957 |
0.789 |
| Tokens per resolved issue |
N |
M |
| Query latency p50 / p95 (ms) |
a / b |
— |
| Indexing time — django, ~2900 files |
T min |
— |
The retrieval row can be populated today from the existing eval; the others fill in as benchmarks 1–4 land. Every number must be reproducible from a committed command (uv run python -m evals.… / a benchmarks/e2e/ runner), with the exact corpus SHA, model id, and cgr version pinned next to the table — the credibility of the table is the reproducibility of its cells.
Suggested phasing
- (4) Indexing benchmark — smallest, no key, immediate README row.
- (2)+(3) Latency + token harness — one runner, two metrics; nightly CI job (token metric skipped when no key is present).
- (1) Agentic A/B — the expensive one; land last, becomes the headline row.
Context
Why
An external field-test review of v0.0.593 (article, Chinese) graded the project B, with the deciding caveat that the full RAG pipeline was unmeasured: no Memgraph/Qdrant running, no model keys — so end-to-end accuracy, query latency, token cost, and indexing time for large monorepos are unknown to an evaluator deciding whether the two databases are "worth feeding."
The deterministic harness in
evals/already answers the graph-layer accuracy question decisively (e.g. retrieval ondjango/django: graph F1 0.957 vs grep_call 0.789), and its README even names the missing piece — the section "Next step: agentic resolved-rate (out of scope here)" (evals/README.md, ~line 1191) says it "is tracked separately" — but no tracking issue exists. This is that issue.Deliverables — four benchmarks
1. Agentic resolved-rate (graph tools vs grep, A/B)
The design
evals/README.mdalready specifies (mirroring GitLab GKG's evaluation): hold one agent and one model fixed, vary only the tools (cgr graph/MCP tools vs plain grep), run a SWE-bench-Lite-style issue set (or a curated in-repo question set as a cheaper first cut), report resolved rate per condition. Needs an LLM key, a container harness, and enough runs per condition for a stable number. This is the headline accuracy figure.2. Query latency on the real stack
Bring up the real stack (
cgr daemon— the compose file already binds loopback), index a pinned corpus, replay a fixed query set through the MCP server, record per-query wall-clock (p50/p95). This measures what the in-memory eval harness deliberately bypasses: Memgraph/Qdrant round-trips.3. Token cost per answer
Same harness as (2)/(1): sum model-API
usageper question, report mean tokens per answered question for graph-tools vs grep-tools conditions. This directly answers the reviewer's "is the graph worth the operational overhead" — if graph tools resolve in fewer tool-call round-trips, it shows up here as fewer tokens.4. Indexing time on large monorepos
Time
cgr indexon 2–3 pinned large repos (e.g.django/django— already an eval corpus — plus one large polyglot monorepo), report wall-clock, peak RSS, and node/edge counts. Track across releases so indexing-time regressions are visible. No model key needed; CI-friendly.Deliverable — README front-page comparison table
Results must land on the front page of the README as a benchmark comparison table, in the style AI labs use to present model-vs-model results: cgr (graph tools) as one column, grep-only baseline as the other, one row per metric, with the corpus/model/version pinned in a caption. Sketch:
The retrieval row can be populated today from the existing eval; the others fill in as benchmarks 1–4 land. Every number must be reproducible from a committed command (
uv run python -m evals.…/ abenchmarks/e2e/runner), with the exact corpus SHA, model id, and cgr version pinned next to the table — the credibility of the table is the reproducibility of its cells.Suggested phasing
Context
evals/README.md"Next step: agentic resolved-rate (out of scope here)" — the design source for (1).