docs(guides): add Results & Scoring guide

[julpayne]

Summary

Adds a dedicated Guides → Results & Scoring page documenting how EvalHub calculates the test values on a completed evaluation job — both the per-benchmark results.benchmarks[].test and the job-level results.test — and the user overrides that affect them.

The existing Collections guide covers the configuration side (weights, primary scores, thresholds) but not the shape or derivation of the results themselves. This new page fills that gap and cross-links back to Collections.

What the page covers

• The two test fields — per-benchmark vs. job-level, with a full results JSON example.
• Per-benchmark test — how the primary metric, threshold, and pass/fail are derived (including lower_is_better), and when the block is omitted.
• Job-level test — the weighted-mean formula, 1 − score inversion for lower_is_better benchmarks, and skip conditions.
• User overrides — precedence tables for primary metric, per-benchmark threshold, weight, and job threshold (job request → collection → provider default → hard-coded fallback), plus a job-request override example.
• Non-numeric metrics — only numeric values are scored (accepted/rejected type table, including the numeric-string caveat), and how a non-numeric primary metric cascades: omitted test block → excluded from the job average → results.test omitted if nothing is numeric.

All content is grounded in the scoring implementation (computeBenchmarkTestResult, computeJobTestResult, getPassCriteriaThreshold, castAnyToFloat32) and the API result structs.

Sidebar

Added under Guides, between Job Lifecycle & States and Collections.

Validation

astro sync regenerates content collections and types with no errors. (An unrelated pre-existing asciinema-player bundling error affects astro build in this checkout and is not touched by this PR.)

🤖 Generated with Claude Code

Summary by CodeRabbit

• Documentation
  • Added a guide explaining how benchmark and job-level test results are scored, including thresholds, weights, and configuration precedence.
  • Added a “Results & Scoring” link to the Guides sidebar.

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

Caution

Review failed

An error occurred during the review process. Please try again later.

📝 Walkthrough

Walkthrough

The pull request adds a Results & Scoring guide and links it from the Guides sidebar. The guide documents per-benchmark scoring, job-level aggregation, configuration precedence, examples, and conditions that omit results.

Changes

Results and Scoring Guide

Layer / File(s)	Summary
Guide entry and per-benchmark scoring astro.config.mjs, src/content/docs/guides/results-and-scoring.mdx	Adds the guide to the sidebar. The guide explains result fields, scoring inputs, numeric value handling, and per-benchmark comparisons.
Job-level aggregation and examples src/content/docs/guides/results-and-scoring.mdx	Documents weighted job-level scoring, threshold resolution, configuration examples, and conditions that omit results.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Other

Suggested reviewers: ruivieira

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely describes the main change: adding a Results & Scoring guide.
Description check	✅ Passed	The description clearly explains the documentation change, its purpose, scope, sidebar placement, and validation result. It does not follow the template headings exactly and omits an issue reference, …
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 1…
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-09-30 07:47 UTC

[nbs-rh]

LGTM. Thanks!