Repository navigation
docs: lead README benchmarks with LexBench Web Fetch - #1609
lanyue-llk wants to merge 5 commits into
Conversation
286d531 to
8ca9e66
Compare
Spider Bench A/B✅ All benchmark browser service runs completed; results are informational. Public HEAD: 184 / 240 rows; 38 / 48 sites produced rows; 10 unexpected empty sites. ✅ Deterministic fixture contract is clean. Common ancestor Public 48-site run · informational
Public-site content, timing, and memory are noisy. This single A/B run reports evidence only; site outcome counts explain missing rows without treating them as a deterministic regression. CPU and memory timelinesThese are bounded 40-point views of the same complete process-tree samples used by the HTML report. CPU uses 100% per occupied logical core; memory lines are RSS first and PSS second when complete PSS samples are available. Basexychart-beta
title "Base CPU"
x-axis "Elapsed seconds" [1.036, 1.536, 2.537, 4.038, 5.541, 6.543, 8.044, 9.545, 10.547, 12.048, 13.55, 14.551, 16.054, 17.053, 17.554, 18.555, 20.057, 21.557, 22.558, 24.058, 25.56, 26.562, 28.065, 29.568, 30.57, 32.072, 33.575, 34.576, 36.076, 37.579, 38.581, 40.083, 41.584, 42.585, 44.087, 45.588, 46.588, 47.59]
y-axis "CPU percent" 0 --> 200
line [33.92, 125.92, 115.92, 133.83, 131.78, 109.82, 81.86, 127.98, 99.82, 31.96, 3.99, 61.94, 73.87, 0, 2, 15.98, 91.97, 112, 120, 112.03, 123.78, 123.83, 101.85, 123.71, 103.79, 85.96, 121.71, 123.84, 82, 5.99, 37.95, 99.92, 101.9, 99.89, 115.93, 45.97, 123.99, 159.72]
xychart-beta
title "Base memory: RSS then PSS"
x-axis "Elapsed seconds" [0.035, 1.036, 1.536, 2.537, 4.038, 5.541, 6.543, 8.044, 9.545, 10.547, 12.048, 13.55, 14.551, 16.054, 17.053, 17.554, 18.555, 20.057, 21.557, 22.558, 24.058, 25.56, 26.562, 28.065, 29.568, 30.57, 32.072, 33.575, 34.576, 36.076, 37.579, 38.581, 40.083, 41.584, 42.585, 44.087, 45.588, 46.588, 47.59, 47.95]
y-axis "Memory MiB" 0 --> 500
line [13.14, 86.51, 106.44, 116.64, 144.75, 133.98, 129.25, 129.55, 158.95, 120.28, 122.05, 118.43, 124.02, 146.61, 116.41, 114.94, 116.47, 151.21, 146.85, 145.84, 256.96, 195.8, 133.49, 165.89, 142.11, 136.39, 147.19, 181, 232.29, 235.81, 132.61, 144.11, 176.59, 176.73, 176.73, 170.29, 127.28, 161.29, 167.34, 0]
line [10.15, 83.41, 103.18, 113.57, 141.55, 131.21, 126.27, 126.29, 155.79, 117, 118.77, 115.15, 120.74, 144.74, 113.13, 111.66, 113.19, 148.28, 143.57, 141.37, 253.85, 192.5, 130.21, 162.8, 138.8, 133.11, 143.91, 177.03, 228.45, 233.27, 129.32, 140.89, 173.31, 173.43, 173.43, 166.99, 123.97, 156.62, 164.04, 0]
HEADxychart-beta
title "HEAD CPU"
x-axis "Elapsed seconds" [1.548, 2.548, 3.049, 4.051, 5.553, 7.056, 8.557, 10.058, 11.058, 12.559, 14.062, 15.562, 17.062, 17.563, 18.064, 19.564, 21.066, 22.567, 24.068, 25.569, 26.571, 28.072, 29.573, 31.076, 32.577, 33.577, 35.077, 36.58, 38.081, 39.584, 40.584, 42.086, 43.588, 45.09, 46.591, 47.592, 49.093, 50.094]
y-axis "CPU percent" 0 --> 200
line [117.73, 106.05, 117.85, 73.88, 139.7, 89.85, 123.94, 100.04, 138.01, 2, 95.83, 118.02, 2, 0, 59.92, 0, 49.99, 97.96, 99.97, 135.79, 101.8, 105.89, 57.99, 131.91, 107.96, 91.95, 33.99, 149.72, 77.95, 6, 6, 99.7, 99.88, 99.93, 115.88, 41.98, 117.85, 173.81]
xychart-beta
title "HEAD memory: RSS then PSS"
x-axis "Elapsed seconds" [0.046, 1.548, 2.548, 3.049, 4.051, 5.553, 7.056, 8.557, 10.058, 11.058, 12.559, 14.062, 15.562, 17.062, 17.563, 18.064, 19.564, 21.066, 22.567, 24.068, 25.569, 26.571, 28.072, 29.573, 31.076, 32.577, 33.577, 35.077, 36.58, 38.081, 39.584, 40.584, 42.086, 43.588, 45.09, 46.591, 47.592, 49.093, 50.094, 50.436]
y-axis "Memory MiB" 0 --> 500
line [13.04, 98.11, 109.93, 139.21, 137.22, 145.61, 144.94, 135.84, 148.89, 147.59, 129.51, 141.93, 142.86, 115.7, 115.55, 123.81, 119.62, 118.23, 147.08, 263.06, 188.59, 201.66, 135.05, 149.85, 159.54, 144.21, 141.66, 137.88, 227.32, 226.88, 131.64, 131.08, 172.24, 176.3, 176.31, 157.39, 143.69, 162.86, 167.87, 0]
line [10.15, 94.91, 106.76, 136.02, 134.45, 142.43, 141.67, 132.58, 145.62, 145.26, 126.2, 138.59, 139.53, 112.44, 112.28, 121.3, 116.36, 115.05, 143.79, 257.94, 185.32, 198.35, 131.75, 146.54, 155.13, 140.94, 138.41, 134.62, 224.18, 223.65, 128.38, 128.08, 168.98, 173.03, 173.05, 154.12, 140.42, 159.64, 164.78, 0]
Deterministic fixture · required contractThe fixture exercises 15 routes: 8 are expected to emit rows and 7 intentionally exercise empty, loading, or timeout behavior.
Full HTML, JSON, CSV, logs, and page snapshots: workflow run and full The public-web A/B is informational. The exact HEAD fixture contract runs as a separate required CI check; timing and public-site content deltas remain non-blocking. |
CI Regression ReportSource CI run · source state at render:
Release regression — ✅ HEAD/base failures 0/0; raw binary 0.00%Package and image size
Startup latency and PSS
Concurrency matrix
Frontend differential — ✅ 1,020/1,020 cases matched; 0 issues
Agent episodes · Moli vs Chromium — ✅ 8/8 Moli episodes passed; 0 failures
Runtime and CDP session contracts — ✅ 26 contract cases; 0 failures
CDP smoke — ✅ 52/52 groups passed; 621 scenariosWorkers: 4 · cumulative group time: 183.01 s · failed groups: 0
WebMainBench · 545 pages — ✅ 545/545 completed; 0 unexpected failures
Frozen HTML · Linux · Diagnostics: All artifact fields are parsed by the trusted default-branch renderer; missing or invalid inputs remain visible as unavailable. |
Sequential Navigation Soak A/B✅ HEAD completed the 200-navigation resilience and memory observation. One browser process, one target, and one CDP session navigate CSDN → SegmentFault → Huaban → example.com repeatedly. This run issued 200 Common ancestor Session resilience
A public-page failure is reported separately from an unrecoverable session. The soak requires all navigation attempts, zero failed recovery, zero lifecycle/network ordering violations, and complete resource evidence. Process-tree memory
HEAD memory by 50-navigation quarter
Raw reports and per-navigation boundary samples: workflow run and full Public-site timing and memory are observational. Use the A/B deltas and trend shape as evidence, not as a deterministic performance threshold. |
ca510d8 to
cdff888
Compare
cdff888 to
6480a06
Compare
lanyue-llk
left a comment
lanyue-llk
left a comment
There was a problem hiding this comment.
This docs-only PR swaps the old 192-URL crawl sample in all seven READMEs for a five-method LexBench WebFetch table (430 tasks, seven families) and links to the Artifact Site report. The structure is consistent across all seven editions. The arithmetic matches the stated counts: 385, 234, 229, 223 and 192 out of 430 round to the listed percentages. Both GitHub links resolve to public repositories, and git diff --check and a merge with current main are clean. Request changes. The linked report no longer shows these numbers, the link is not pinned, and the headline Moli row comes from a build that is in no release.
Findings not tied to a single diff line
- [blocker] The numbers disagree with what the link serves now, and the link is not pinned.
https://artifact-site.lexmount.cn/v/IqGpMUYHWat-dw4n8usQttTptoPzboAohas no?version=, so it serves the latest site version (ver_abef4867…), not the archivedver_b5584a3d…recorded in browser-eval'sevaluation/publication-archives.json. That latest version includes a 2026-10-21 release, "saved-output-score-audit", which rescored every saved output. Its home table andsummary.jsonnow show Moli 1.1.17 at 384/430 (89.3%), Chromium 233, Lightpanda 227, Obscura 222 and Kitesurf 194. The README still shows 385, 234, 229, 223 and 192. Those come from the earlier 2026-10-15 release ("Fixed Moli scores 385/430 and Lightpanda 229/430"). A reader who follows "the pinned report" today sees different numbers in all five rows. Fix: either update to the current audited numbers, or bind a real version-pinned URL (?version=ver_…) and say it is a historical snapshot. The PR description's claim "pinned report" has the same problem. - [major] The headline "Moli 1.1.17" row is not a release build. The report's
implementation-provenance.jsonlists this row as two binaries:a737f2aefor 374 retained results and0d9a7cebfor 56 corrected results. Both come from the closed integration PR #1501 and sit 46 commits afterv1.1.17. Neither commit is inmain. Several of the split PRs that carry the WebFetch fixes are still open, including #1591 (visible labels and resource URLs) and #1584 (premature completion). A user who installs v1.1.17 or v1.1.18 will not get the measured 89.53%. Meanwhile the released v1.1.15, which the report also measured (366/430, 85.1%), is left out on purpose ("one Moli row"). Fix: lead with a released build, or label the row as an unreleased development build with its commits and add a footnote that it combines two binaries. - [minor] "Task definitions are maintained in lexbench-webfetch" overstates the source. lexbench-webfetch publishes four suites with 450 entries and no 430-task, seven-family population. Its README says the suites "are not a deduplicated 450-site leaderboard". The 430-task set is assembled in browser-eval:
evaluation/webfetch-seven-family-430/dataset-assembly.jsontakes 130 of the 150webfetchentries plus the three 100-task suites. It also applies about 148 KB of profilecase-corrections.json. A reader who follows the link cannot find "430 tasks" there. Say instead that the tasks are drawn from the lexbench-webfetch suites and assembled and corrected in browser-eval. - [minor] English and other non-Chinese readers land on a Chinese-only report. The served report's
report-languages.jsonis{"languages":["zh"],"english_status":"not_generated"}, anden/index.htmlis the "English report has not been generated" notice. The en, de, es, fr, ja and ko READMEs all promise "full report, task contracts and failure evidence" with no language warning. Add a short "(Chinese)" note or generate the English edition. - [minor] Calls were not all made on 2026-10-15. The report's
execution.jsonbatches run from 2026-10-09 (all main Kitesurf batches) through 2026-10-15 UTC, and most local-engine calls ran on 10-14. The report itself warns that the main-matrix calls span dates and batches. Use "October 9–15, 2026" or "calls made October 9–15, scored …" in all seven editions. - Relationship to #1380. #1609 replaces the part of #1380 that also removed the 192-URL sample, and the two now conflict textually:
git merge-treeshows conflicts in the same benchmark section. They do not cover the same ground. #1380 adds a WebMainBench 545-page fidelity table and a benchmark overview table. It is stacked on code PRs (#1334→#1337), has no Korean README, and reports older engine versions (Lightpanda 0.4.1, Obscura 0.2.2, Chrome 154) for a dev build at721bd2e. #1380 also titles WebMainBench "Web Fetch: content extraction across 545 pages", which clashes with this PR's "LexBench WebFetch". I would treat #1609 as superseding #1380's README work. Close #1380, or rebase it on #1609 under a distinct WebMainBench heading with current engine versions and a Korean edition. Do not merge both as they stand. - [nit] Inconsistent labels for the Moli row across languages. en says "repair result", zh "修复版结果" (repaired-build result), de, es and fr say "corrected result" (korrigiertes, corregido, corrigé), ja "修正版" and ko "수정 결과". "Corrected result" suggests a scoring correction, and "repair result" is internal shorthand that readers will not understand. Pick one reader-facing term, such as "development build with WebFetch fixes", and translate it the same way everywhere.
- [nit] Typography. zh and ja mix ASCII parentheses (
Kitesurf (托管服务),Kitesurf (ホスト型)) with the full-width ones used in the Moli row. Their link lines also end in an ASCII "." rather than "。" or nothing. The de, es and fr tables use "89.53%" while the paragraphs just below use "81,88 %".
CI: the only bot comment is the informational Spider Bench A/B, which is unaffected by a docs-only change. No relevant failures.
Verdict: changes requested (posted as a comment because GitHub does not allow requesting changes on your own PR).
Automated review by Claude Code.
lanyue-llk
left a comment
lanyue-llk
left a comment
There was a problem hiding this comment.
Re-review of the updated version (04c718f88)
This docs-only PR replaces the old 192-URL crawl sample in all seven READMEs with a five-method LexBench WebFetch table (430 tasks, seven families) and a version-pinned report link. Approve. Every blocker, major and minor finding from round 1 is fixed. What remains is one minor wording issue and some nits, none of which block the merge.
Verification: I decoded summary.json from the pinned version ver_494ef0fafb1a4529ab3c28762b7c3ca6, which is also the version the unpinned share link serves today. Summed over the seven families, the passes are moli-fixed 384, chromium 233, lightpanda 227, obscura 222 and kitesurf 194. Every method has 430 attempts and zero unmeasured or excluded. All five percentages round correctly. All seven editions carry identical counts and the same URL. git diff --check is clean.
| # | Previous finding | Status | Evidence |
|---|---|---|---|
| 1 | [blocker] Stale numbers, unpinned link | FIXED | The URL now carries ?version=ver_494ef0fa…. That version's summary.json gives 384/233/227/222/194 of 430. |
| 2 | [major] "Moli 1.1.17" is not a release build | FIXED | The row is labelled as an unreleased development build, with both commits and "not the 1.1.17 release binary". a737f2ae and 0d9a7ceb are in no tag (including v1.1.19) and not in origin/main. Leaving out released v1.1.15 (366/430) follows the maintainer's request for a single Moli row, so that part of my suggestion is withdrawn. |
| 3 | [minor] Call dates | FIXED | The execution.json batch timestamps run from 2026-10-09 to 2026-10-15, which matches "October 9 to 15". |
| 4 | [minor] lexbench-webfetch has no 430-task set | FIXED | The README now says "drawn from … suites; browser-eval assembles and corrects". |
| 5 | [minor] Report is Chinese-only | FIXED | report-languages.json is still ["zh"] with not_generated, and all six non-Chinese editions now say "(Chinese)". |
| 6 | [nit] zh/ja typography | FIXED (part) | Full-width parentheses are used and the ASCII period is gone. The de/es/fr tables still use 89.30%, while the paragraphs below use 81,88 %. That is NOT FIXED and stays a nit. |
| 7 | [nit] Inconsistent Moli labels | FIXED | All seven editions now say "development build (WebFetch fixes)" consistently. |
| 8 | Relationship to #1380 | RESOLVED (textually) | git merge-tree is clean in both orders against refs/review/v4-1380. The README in #1380's live head 5e1ea52c7 is identical. See the follow-up below. |
New findings
- [minor] The Moli footnote brings back internal cohort wording ("374 retained tasks", "56 declared correction tasks"). On 2026-10-16 the maintainer asked for this not to be public, and our round-1 request for the commits partly caused its return. "Declared correction tasks" also reads as if the tasks were corrected. In fact, 56 tasks were rerun on a later Moli commit. See the inline comment.
- [nit] The upstream benchmark is titled "LexBench Web Fetch" (with a space), but the README writes "LexBench WebFetch". See the inline comment.
- [nit] The "Chromium 155.0.8059.39" row is Google Chrome for Testing according to the report's binary identity. The neighbouring sections and #1380 call it Chrome / Chrome for Testing. See the inline comment.
- [nit, cross-PR] After #1380 merges, the overview table has no WebFetch row. In the clean merged tree (
e7d79737), "Public test suites" lists WPT, Lexbench automation, CPU/memory and WebMainBench in all seven editions, but not LexBench WebFetch, even though WebFetch is the first results section. Whichever PR lands second should add a row, for exampleLive web fetch | lexbench-webfetch · 430 tasks, 7 families | Task pass rate. The merged tree also places the two extraction sections far apart, "Web content extraction" at the top and "WebMainBench" at the bottom. - [nit, outside this repo] browser-eval has no binding for the pinned version.
evaluation/publication-archives.jsonlists onlyver_b5584a3d…for this share URL. No ref in browser-eval mentionsver_494ef0fa…, so the README's numbers can be traced only on the site itself. Record the 2026-10-21 publication in browser-eval. - [nit] Commit hygiene. The first commit adds superseded numbers and the next two correct them. Squash on merge.
CI: Build release · HEAD failed because its "Build release binary" step was cancelled after about 15 minutes, which points to infrastructure or a timeout. The common-ancestor build and Format, Clippy, Nextest, Python and Node all pass. The Spider and navigation-soak A/B bot comments report infrastructure failure and no comparison. A README-only change cannot cause this. Re-run if a green check is required.
Verdict: approve (posted as a comment; self-approval is not allowed).
Automated review by Claude Code.
lanyue-llk
changed the title
c8ac3ec to
76417a6
Compare
|
The report link and table are now bound to the current public version, The previously pinned This remains a README-only PR targeting |
lanyue-llk
left a comment
lanyue-llk
left a comment
There was a problem hiding this comment.
Final re-review (P0/P1 only) (76417a64b)
Approve. I found no P0/P1 issues at head 76417a64b. This round changes the numbers and the pinned version, and I re-verified both from the report data.
Re-verification: I decoded summary.json from the new pinned version ver_7c1cd4072edf4403a8ec3c64205f878f, which is also what the unversioned share link serves today. Summed over the seven families, the passes are moli-fixed 385, chromium 233, lightpanda 227, obscura 222 and kitesurf 194. Every method has 430 attempts and none are unmeasured or excluded. 385/430 rounds to 89.53%, and the other four percentages also match. The report's newest release, 2026-10-22 "language-equivalence", explains the change: it restores verified English navigation aliases for one task, which adds one pass to each Moli configuration, and leaves calls and timings unchanged. That means the "October 9 to 15" call window still holds. implementation-provenance.json is byte-identical to the previous version: a737f2ae has 374 results and 0d9a7ceb has 56, so "56 of the 430 tasks were rerun on the later commit" is accurate. The binary is recorded as Chrome for Testing. report-languages.json is still ["zh"], so the "(Chinese)" labels are still correct. All seven editions carry identical counts, percentages, commits and the same pinned URL. No stale 384 or ver_494ef…/ver_b5584… remains, and the internal "retained/declared correction" wording is gone. git diff --check is clean.
| # | Previous finding | Status | Evidence |
|---|---|---|---|
| 1 | [blocker] Stale numbers, unpinned link | FIXED (re-verified) | README.md:321-325 and :331. ?version=ver_7c1cd407… serves 385/233/227/222/194 of 430. |
| 2 | [major] "Moli 1.1.17" is not a release build | FIXED | README.md:321 and :327 label it an unreleased development build after v1.1.17, with both commits and "not the 1.1.17 release binary". |
| 3 | [minor, maintainer-requested] Internal cohort wording in the public footnote | FIXED | README.md:327. All seven editions now say "56 of the 430 tasks were rerun on the later commit". git grep finds no "374", "declared" or "修正结果". |
| 4 | Conflict with #1380 | RESOLVED | git merge-tree is clean in both orders against #1380 head 0153c5d6d, and also against current origin/main 14d3c08cc. The merged tree has the new pinned URL once per edition and no stale numbers. |
New P0/P1 findings: none.
CI on 76417a64b: Format, Clippy, Python, Node, Windows storage and Resolve revisions pass. Build release (HEAD and ancestor) and Nextest are still pending. CDP smoke (Chromium) failed with a Puppeteer TargetCloseError: Protocol error (Runtime.callFunctionOn): Target closed. The PR changes only the seven README files, and the same job also fails on unrelated heads (#1380's 0153c5d6d, 9e83f5cb3, 1411b1ffb), so the failure is flaky or infrastructure and not caused by this PR. Re-run it if a green check is required.
Verdict: approve (posted as a comment; self-approval is not allowed).
Automated review by Claude Code.
1 participant
Problem
The README still led with an older 192-URL crawl sample and omitted the reviewed 430-task LexBench Web Fetch result. That made the project page a poor guide to the current Web Fetch coverage.
Change
Replace the old sample in all seven README editions with the audited five-method table:
The text identifies the Moli row as an unreleased build after v1.1.17, gives both measured commits, and says that 56 of the 430 tasks were rerun on the later commit. It also gives the October 9–15 call window, explains that Browser Eval assembles the 430 tasks from LexBench Web Fetch suites, and links to the exact report version. Non-Chinese READMEs state that the linked report is Chinese.
This PR changes README files only. It does not depend on the Browser Eval implementation branch and does not change browser behavior, benchmark evidence, or scores.
Verify
summary.jsonandimplementation-provenance.jsonfrom versionver_7c1cd4072edf4403a8ec3c64205f878f.git diff --check.Pinned report: LexBench Web Fetch