Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
d6f138d
Enforce scoring and executable consistency at release boundaries
lanyue-llk Oct 20, 2026
62259de
Align native title projection controls with body fact acceptance
lanyue-llk Oct 20, 2026
7d21544
Cover declared tracking parameters and page-selecting query differences
lanyue-llk Oct 20, 2026
7c1f130
Use current native title control in offline revision scenarios
lanyue-llk Oct 20, 2026
407fbb8
docs(contributing): define PR and commit title conventions
lanyue-llk Oct 20, 2026
7b89f68
fix(webfetch): allow retained results in partial repair runs
lanyue-llk Oct 20, 2026
f19092f
fix(webfetch): count missing required identity as a failed result
lanyue-llk Oct 20, 2026
76af482
test(webfetch): keep missing native page identity in the denominator
lanyue-llk Oct 20, 2026
36dd2d6
build(webfetch): pin scorer revision with corrected citation metadata
lanyue-llk Oct 20, 2026
52b7035
build(webfetch): pin scorer with resource format preservation
lanyue-llk Oct 20, 2026
695f97c
test(webfetch): cover data-only report instance boundaries
lanyue-llk Oct 20, 2026
95a42ce
fix(reporting): support real-data instances across benchmark tracks
lanyue-llk Oct 21, 2026
5b07190
fix(browser-use): derive method details from the report instance
lanyue-llk Oct 21, 2026
889a1cb
fix(webfetch): derive WCXB presentation from selected methods
lanyue-llk Oct 21, 2026
d6ebe19
fix(webfetch): validate WCXB change summaries against scored rows
lanyue-llk Oct 21, 2026
4f34896
fix(webfetch): align WCXB instance counts across languages
lanyue-llk Oct 21, 2026
df6575b
fix(webfetch): normalize measured version labels
lanyue-llk Oct 21, 2026
be79c1a
fix(reporting): support external instances and disclose acceptance scope
lanyue-llk Oct 21, 2026
45b05a3
test(certification): reject diagnostic evidence during packaging
lanyue-llk Oct 21, 2026
51c9c6a
fix(webfetch): audit saved-output scoring through shared reports
lanyue-llk Oct 21, 2026
4185cca
test(webfetch): load deferred images in snapshot visual checks
lanyue-llk Oct 21, 2026
05f0931
fix(webfetch): retain historical privacy receipts when repacking audits
lanyue-llk Oct 21, 2026
d0761d2
feat(webfetch): publish saved-output audits through the shared entry
lanyue-llk Oct 21, 2026
44e079f
fix(webfetch): verify retained public views of recovered scoring inputs
lanyue-llk Oct 21, 2026
0afd182
fix(report): present saved results as a normal report update
lanyue-llk Oct 21, 2026
dfdc990
fix(report): label version comparisons by measured configuration
lanyue-llk Oct 21, 2026
ab0117b
fix(report): remove snapshot status callouts from result summaries
lanyue-llk Oct 21, 2026
803d1c4
fix(report): bind annotation versions and participant interface metadata
lanyue-llk Oct 21, 2026
6428f56
fix(report): sort comparisons and highlight unique winners
lanyue-llk Oct 22, 2026
cfd5822
fix(reporting): isolate report instance metadata and template discovery
lanyue-llk Oct 22, 2026
e9e585e
style(reporting): order shared asset imports
lanyue-llk Oct 22, 2026
6bc2508
refactor(reporting): remove historical instance assumptions
lanyue-llk Oct 22, 2026
bb442fc
fix(reporting): isolate release history and cross-track labels
lanyue-llk Oct 22, 2026
e74849b
fix(reporting): bind instances to profiles and expose cohort provenance
lanyue-llk Oct 22, 2026
378ed73
chore(deps): pin strict reference error-page validation
lanyue-llk Oct 22, 2026
bcad77b
chore(deps): pin localized navigation regression controls
lanyue-llk Oct 22, 2026
ae63e19
fix(deps): keep benchmark definition bindings aligned with the pinned…
lanyue-llk Oct 22, 2026
3975b07
fix(reporting): bind sample scope and task comparison disclosures
lanyue-llk Oct 22, 2026
596be05
fix(reporting): place sample limits before aggregate results
lanyue-llk Oct 22, 2026
2d83eb5
Revert "fix(reporting): place sample limits before aggregate results"
lanyue-llk Oct 22, 2026
89284b5
Revert "fix(reporting): bind sample scope and task comparison disclos…
lanyue-llk Oct 22, 2026
ac21661
docs: publish the updated project banner
lanyue-llk Oct 22, 2026
6bbafc0
refactor(reporting)!: separate instances and language data
lanyue-llk Oct 22, 2026
aeeb02d
docs(reporting): update public replay profile guidance
lanyue-llk Oct 22, 2026
1595e6d
fix(reporting): reject unresolved message parameters
lanyue-llk Oct 22, 2026
ffb3a8e
fix(reporting): integrate shared method and article contracts
lanyue-llk Oct 22, 2026
da0e089
fix(reporting): localize every registered task family
lanyue-llk Oct 22, 2026
e57c105
test(reporting): verify localized unknown-method access
lanyue-llk Oct 22, 2026
21b961d
ci: retry validation after dependency mirror stall
lanyue-llk Oct 22, 2026
654d90f
chore(license): adopt Apache 2.0 across packages
lanyue-llk Oct 22, 2026
9ff8eb8
fix(reporting): share localized method metadata and labels
lanyue-llk Oct 22, 2026
b138ba0
ci: avoid optional font downloads in reference capture
lanyue-llk Oct 22, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .agents/skills/benchmark-report/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,20 @@ Read the repository's `AGENTS.md` first. Project rules override this general ski
Before writing report HTML, resolve the track through `scripts/report_template.py`. Inspect the maintained implementation and its pinned example. Reuse that implementation and its release checks; a different cohort or diagnostic evidence state calls for a data adapter or explicit mode. If no track fits, create the smallest reusable renderer with a synthetic runnable example and behavioral checks, then register it in `configs/report-templates.json`. Improve the owning template and this skill from demonstrated gaps, without embedding run-specific prose or copying old measurements. A report must carry the selected template identity and input hashes so its release check can detect a bypass.

Derive bilingual counts, budgets and method labels from the selected inputs.
Bind cohort-specific access and interface documentation in the participant registry;
reuse engine-level adapter contracts instead of adding per-version renderer tables.
Keep report-specific prose and its translations in explicitly bound content;
do not choose a historical article from provider names or task-family guesses.
Without a content binding, render the maintained data-derived explanation.
Prove reuse through the public build and check entries with a different external
task/method inventory, not only by searching for old numbers in source code.
Exercise assembly and packaging with that external profile as well. Participant
IDs, labels, interfaces and comparison roles come from the instance registry;
a new cohort must not inherit a historical method's account or timing claims.
Disclose declared literal, fact, body-region, record, resource and identity checks
separately. A task instruction is not a full-text reference. Preserve frozen
verdicts while exposing passes without verified page identity, blocked-page
markers and unexpected resource URLs as diagnostics.

An explicitly requested diagnostic report may be delivered separately from a certified benchmark release. Use the diagnostic build/check entry, label its evidence state and certification gaps on every page, preserve saved scores and outputs, and do not publish rankings or upgrade certification. It does not waive any certified-release reference or review gate. Presentation fixes add no provider calls. Update only the owning code, relevant instructions and demonstrated regression scenarios; do not mechanically modify every skill or copy report-specific conclusions into instructions.

Expand Down
11 changes: 11 additions & 0 deletions .agents/skills/benchmark-report/references/audit-and-revision.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,3 +98,14 @@ message. Reproject retained full responses without new product calls when only
this selection changed; missing historical envelopes require fresh calls across
the full declared correction scope before claiming revised benchmark scores.
Reruns of unchanged compiled acceptance-contract tasks freeze the existing executable dataset. They do not require recovering historical qualification raw files or repeating admission review. Source qualification is part of task authoring and reference/rule correction, not a prerequisite for every provider-version comparison. Preserve historical receipts without applying their evidence-file requirements to a new execution protocol.

A published-snapshot score audit uses the maintained Web Fetch profile adapter,
not a copied renderer. Bind the complete original export, full historical matrix,
initial and corrected scores, per-input original/public/scored hashes, and every
changed decision. Preserve collection facts and timing. Evaluate a suspected
privacy effect with the historical scorer on the same public bytes; a redaction
receipt by itself does not establish causation. Keep source-backed annotation
corrections and executable regression provenance together. Reuse the native
report's routes, interpretation schema, structured history and packer, and run its
explicit snapshot audit browser gate before packaging. The catalog provides the
runnable example and commands. Store instance evidence and prose outside Git.
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@

## Template discovery and diagnostic models

Run `make report-template TRACK=webfetch` (or `trajectory-replay` / `browser-use`) before authoring. The registry returns the maintained owner, current profile, version-pinned example, build and check commands. A reference report supplies presentation conventions, never evidence or the new publication target. Extend the owner instead of copying its HTML into a temporary renderer.
Run `make report-template TRACK=webfetch` (or `trajectory-replay` / `browser-use`) before authoring. The registry returns the maintained owner, version-pinned example, build and check commands. A reference report supplies presentation conventions, never evidence or the new publication target. Extend the owner instead of copying its HTML into a temporary renderer.

For explicitly requested live WebFetch diagnostics, use `make webfetch-diagnostic-report DIAGNOSTIC_PROFILE=evaluation/<instance>/current-report.json OUTPUT=/absolute/new-output`; `EVIDENCE=/absolute/source` optionally supplies an explicit input. The `webfetch_cohort_diagnostic/1` model contains cohorts, per-case Boolean first-call verdicts, declared summary counts, cause/source labels, review counts and hash-bound reference/output/source/log files. `tests/test_report_templates.py:diagnostic_example` is a runnable two-scenario fixture through the public builder; its test command is registered alongside the variant. The release checker recomputes totals and validates rendered verdicts, complete evidence bindings, dialogs and links. A diagnostic remains diagnostic; this path cannot substitute for a certified build.
For explicitly requested live WebFetch diagnostics, use `make webfetch-diagnostic-report DIAGNOSTIC_PROFILE=/absolute/instance-profile.json OUTPUT=/absolute/new-output`; `EVIDENCE=/absolute/source` optionally supplies an explicit input. The `webfetch_cohort_diagnostic/1` model contains cohorts, per-case Boolean first-call verdicts, declared summary counts, cause/source labels, review counts and hash-bound reference/output/source/log files. `tests/test_report_templates.py:diagnostic_example` is a runnable two-scenario fixture through the public builder; its test command is registered alongside the variant. The release checker recomputes totals and validates rendered verdicts, complete evidence bindings, dialogs and links. A diagnostic remains diagnostic; this path cannot substitute for a certified build.

For a new report type, first persist a reusable example and scenario-based positive/negative tests, register its owner and release gate, then generate the requested report. Later corrections update the owner and relevant rules when supported by evidence; do not clone a template per run or embed report-specific conclusions in instructions.

Expand Down
3 changes: 2 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -45,8 +45,9 @@ jobs:
- run: make setup-test
- run: make ci
- name: Exercise rendered reference capture
timeout-minutes: 5
run: |
uv sync --frozen --extra test --extra headless-eval
uv run --frozen --no-sync playwright install --with-deps chromium
uv run --frozen --no-sync playwright install chromium
export BROWSER_EVAL_TEST_BROWSER_BINARY="$(uv run --frozen --no-sync python -c 'from playwright.sync_api import sync_playwright; p = sync_playwright().start(); print(p.chromium.executable_path); p.stop()')"
uv run --frozen --no-sync python -m pytest tests/test_browser_observation_contracts.py -q
Loading
Loading