Repository navigation
Web Fetch correctness fixes: split review index - #1501
lanyue-llk wants to merge 24 commits into
Conversation
0b1d2f2 to
e0d6af7
Compare
Spider Bench A/B✅ All benchmark browser service runs completed; results are informational. Public HEAD: 184 / 240 rows; 38 / 48 sites produced rows; 10 unexpected empty sites. ✅ Deterministic fixture contract is clean. Common ancestor Public 48-site run · informational
Public-site content, timing, and memory are noisy. This single A/B run reports evidence only; site outcome counts explain missing rows without treating them as a deterministic regression. CPU and memory timelinesThese are bounded 40-point views of the same complete process-tree samples used by the HTML report. CPU uses 100% per occupied logical core; memory lines are RSS first and PSS second when complete PSS samples are available. Basexychart-beta
title "Base CPU"
x-axis "Elapsed seconds" [1.036, 1.536, 2.538, 4.041, 5.043, 5.543, 6.545, 8.047, 9.551, 10.552, 12.054, 13.555, 14.557, 16.059, 17.56, 18.061, 18.561, 20.062, 21.565, 22.567, 24.068, 25.569, 26.571, 28.072, 29.575, 30.576, 32.078, 33.58, 34.581, 36.084, 37.586, 38.587, 40.09, 41.592, 42.593, 44.095, 45.596, 46.598]
y-axis "CPU percent" 0 --> 500
line [7.98, 131.92, 129.74, 117.72, 209.57, 197.75, 203.67, 123.83, 171.57, 149.89, 177.8, 82.05, 97.79, 131.91, 9.99, 2, 69.97, 149.84, 45.89, 117.87, 115.93, 135.78, 145.7, 119.9, 127.65, 145.94, 179.78, 121.83, 145.82, 77.91, 8, 89.87, 103.78, 13.97, 107.85, 194.02, 125.97, 173.78]
xychart-beta
title "Base memory: RSS then PSS"
x-axis "Elapsed seconds" [0.035, 1.036, 1.536, 2.538, 4.041, 5.043, 5.543, 6.545, 8.047, 9.551, 10.552, 12.054, 13.555, 14.557, 16.059, 17.56, 18.061, 18.561, 20.062, 21.565, 22.567, 24.068, 25.569, 26.571, 28.072, 29.575, 30.576, 32.078, 33.58, 34.581, 36.084, 37.586, 38.587, 40.09, 41.592, 42.593, 44.095, 45.596, 46.598, 47.673]
y-axis "Memory MiB" 0 --> 500
line [12.6, 65.39, 114.2, 127.65, 131.61, 174.65, 179.76, 214.88, 224.44, 167.66, 156.8, 180.08, 154.17, 171.29, 172.5, 144.15, 145.7, 154.37, 179.59, 144.78, 152.48, 181.39, 157.54, 170.06, 282.88, 218.34, 187.89, 173.79, 154.73, 172.36, 166.38, 155.43, 186.64, 189.39, 178.95, 184.25, 185.16, 170.93, 201.2, 0]
line [9.39, 61.88, 110.32, 124.12, 128.08, 171.13, 176.26, 211.44, 220.87, 164.29, 153.21, 176.52, 150.68, 167.68, 168.89, 140.55, 142.1, 150.69, 175.99, 141.16, 148.88, 177.79, 154.01, 166.61, 279.62, 214.75, 184.16, 170.27, 151.2, 168.63, 162.93, 151.83, 183.07, 186.25, 175.35, 180.54, 180.89, 167.44, 197.68, 0]
HEADxychart-beta
title "HEAD CPU"
x-axis "Elapsed seconds" [1.536, 2.537, 3.038, 4.039, 5.04, 5.541, 7.044, 8.546, 9.547, 11.05, 12.552, 14.054, 15.556, 16.558, 18.059, 19.561, 21.062, 22.063, 23.565, 25.067, 26.068, 26.569, 28.072, 29.072, 30.572, 32.075, 33.577, 34.578, 36.08, 37.582, 39.084, 40.586, 41.587, 43.09, 44.591, 46.091, 47.091, 48.593]
y-axis "CPU percent" 0 --> 200
line [133.89, 135.96, 139.74, 107.76, 151.88, 175.66, 163.54, 77.96, 173.77, 57.92, 45.95, 133.75, 1.99, 73.91, 131.85, 0, 75.94, 147.89, 99.95, 147.87, 103.81, 113.81, 147.91, 147.94, 43.93, 103.79, 113.83, 85.89, 95.85, 5.99, 137.66, 83.89, 79.86, 13.97, 115.98, 135.96, 25.95, 140]
xychart-beta
title "HEAD memory: RSS then PSS"
x-axis "Elapsed seconds" [0.035, 1.536, 2.537, 3.038, 4.039, 5.04, 5.541, 7.044, 8.546, 9.547, 11.05, 12.552, 14.054, 15.556, 16.558, 18.059, 19.561, 21.062, 22.063, 23.565, 25.067, 26.068, 26.569, 28.072, 29.072, 30.572, 32.075, 33.577, 34.578, 36.08, 37.582, 39.084, 40.586, 41.587, 43.09, 44.591, 46.091, 47.091, 48.593, 49.916]
y-axis "Memory MiB" 0 --> 500
line [12.46, 116.92, 126.26, 138.18, 134.88, 161.49, 160.5, 187.01, 158.18, 197.11, 169.49, 188.61, 178.09, 152.03, 154.23, 160.27, 144.43, 148.24, 165.33, 159.86, 161.98, 291.2, 232.27, 211.16, 223.85, 164.37, 187.29, 180.58, 163.38, 151.58, 151.73, 224.03, 263.97, 193.38, 166.45, 180.96, 165.11, 153.53, 190.42, 0]
line [9.26, 113.38, 122.59, 134.65, 131.35, 158.01, 156.94, 183.55, 154.61, 193.5, 165.9, 185.02, 174.5, 148.45, 150.65, 156.69, 140.86, 144.64, 162.32, 156.29, 160.05, 288.5, 228.21, 207.62, 220.43, 161.56, 183.71, 176.82, 159.81, 147.98, 148.14, 220.87, 260.5, 189.78, 162.85, 177.38, 161.46, 149.93, 186.85, 0]
Deterministic fixture · required contractThe fixture exercises 15 routes: 8 are expected to emit rows and 7 intentionally exercise empty, loading, or timeout behavior.
Full HTML, JSON, CSV, logs, and page snapshots: workflow run and full The public-web A/B is informational. The exact HEAD fixture contract runs as a separate required CI check; timing and public-site content deltas remain non-blocking. |
CI Regression ReportSource CI run · source state at render:
Release regression — ✅ HEAD/base failures 0/0; raw binary +0.028362%Package and image size
Startup latency and PSS
Concurrency matrix
Frontend differential — ✅ 1,020/1,020 cases matched; 0 issues
Agent episodes · Moli vs Chromium — ✅ 8/8 Moli episodes passed; 0 failures
Runtime and CDP session contracts — ✅ 26 contract cases; 0 failures
CDP smoke — ✅ 52/52 groups passed; 621 scenariosWorkers: 4 · cumulative group time: 163.12 s · failed groups: 0
WebMainBench · 545 pages — ✅ 545/545 completed; 0 unexpected failures
Frozen HTML · Linux · Diagnostics: All artifact fields are parsed by the trusted default-branch renderer; missing or invalid inputs remain visible as unavailable. |
Sequential Navigation Soak A/B✅ HEAD completed the 200-navigation resilience and memory observation. One browser process, one target, and one CDP session navigate CSDN → SegmentFault → Huaban → example.com repeatedly. This run issued 200 Common ancestor Session resilience
A public-page failure is reported separately from an unrecoverable session. The soak requires all navigation attempts, zero failed recovery, zero lifecycle/network ordering violations, and complete resource evidence. Process-tree memory
HEAD memory by 50-navigation quarter
Raw reports and per-navigation boundary samples: workflow run and full Public-site timing and memory are observational. Use the A/B deltas and trend shape as evidence, not as a deterministic performance threshold. |
lanyue-llk
left a comment
There was a problem hiding this comment.
审查提交:0d9a7ceb25f630e7a296e83011e2d11f0279c5a7。结论:建议修复以下两项 P2 后再合并,具体复现和修正方向见行内评论。
本 PR 要解决的是抓取结果缺失或过期:读取当前 iframe 文档、处理 load 后已就绪的任务、等待解析器和子文档的异步启动,以及补齐影响页面脚本执行的浏览器接口。已检查全部 56 个变更文件及相关调用链。
沿用 renderer 的任务调度、保存的原生原型和现有 EventTarget 实现,方向合理;未实现 SourceBuffer 生命周期时停止声称支持 MSE,也比让页面进入不可用的播放路径更合适。两个需要修改的地方分别是状态归属和 URL 属性覆盖:ResizeObserver 的派发状态必须区分文档,iframe 资源地址转换必须覆盖新输出的 poster。这些都应在现有抽象里补齐,无需增加针对特定网站的分支。
本次在独立源码副本中实测:抓取输出 16 项、HTML 转 Markdown 305 项、抓取生命周期 71 项、renderer 相关接口与调度 98 项、Done/Load 边界 4 项,共 494 项现有测试通过。另加 4 个边界场景,复现了行内所述的 2 个问题;有限布局反馈和启动时立即终止 Worker 的场景通过。跨 iframe observer 场景另用 Chrome 对照,Chrome 收到目标宽度 80,当前 PR 的通知列表为空。
这说明现有测试尚未覆盖这两个组合场景。建议补入跨文档回调注册、子文档不同 base 下的媒体封面测试,避免后续回退。
lanyue-llk
left a comment
There was a problem hiding this comment.
复审提交:041584bcd0e0655d4289cccda7c9c9efd144eba5。结论:当前仍有 1 项 P1、4 项 P2,建议修复后再合并。五项均有独立复现,详见行内评论。
本 PR 的目标是修复抓取结果缺失或过期,以及阻碍页面脚本运行的兼容问题。本轮重新核对了全部 56 个变更文件,并按 observer、Markdown/iframe、生命周期与兼容接口三个方向并行审查。上一轮的跨文档 observer 注册和 iframe poster 地址问题已复测通过。
当前剩余问题涉及行为边界:Done 没有保住调用方的绝对超时;ResizeObserver 广播缺少回调间的微任务检查点,深度计算没有跨 ShadowRoot,固定 32 轮上限又会截断合法更新并卡住后续通知;目录链接的处理还会改写真正的源码。这些需要修正现有等待、广播和序列化逻辑。尤其是循环保护应依据正确的树和实际待处理工作,单纯增加次数或再补一个特例不能解决。
本轮实测:现有抓取输出 18 项、Markdown regressions 29 项通过;额外 10 个场景中 6 个失败,对应上述 5 个根因,另外 4 个通过。失败场景已串行重跑,两个 Done 超时场景还分别放进独立进程复测。三个 observer 场景均与安装的 Chrome 对照;深层通知在第二次显式布局更新后仍无法恢复。代码示例污染也对照了基础提交的正确输出。
其他已核查的 iframe base/strip/嵌套展开、原生原型保存、PermissionStatus 继承、MediaSource 能力探测和任务源公平性,未发现足以提交的其他具体缺陷。
lanyue-llk
left a comment
There was a problem hiding this comment.
复审提交:f1ee662f6d2a276b4882a041fb728c8c8e4e1d7a。当前仍有 3 项 P2:两项由本次 observer 修正引入的组合行为错误,以及一项代码块回退尚未修完整。建议处理后再合并。
本轮按三个方向核对了 PR 全部 58 个变更文件及本次增量。上一轮 Done 的返回超时、直接 Shadow DOM 深度、回调之间的微任务顺序,以及单纯 40 层有限布局更新,独立复现均已通过。但预算续任务丢失迭代深度,会重复执行浅层回调并提前误报循环;新检查点 helper 又会把 Promise 拒绝事件提前派发。这两项详见行内评论,都已用 Chrome 对照。
代码块修复仅覆盖 code 子树,Javadoc 的直接 pre 签名和带 language-rust 的 pre 仍被插入链接地址。已在原问题线程补充两类失败用例及官方页面源码,不重复开同根问题。
验证方面,现有抓取输出 18 项、Markdown regressions 31 项通过,上轮 9 个抓取边界场景通过;本轮两个 observer 组合场景和两类直接 pre 场景均失败。observer 对照分别得到:Chrome rootCalls=1 且无错误,当前为 2 且发一次 loop error;Chrome 的 Promise 拒绝事件为空,当前多发 unhandledrejection/rejectionhandled。
另实测到超时返回后忙脚本可能继续阻塞同 Browser 的其他页面,但完全不使用新功能的原有 Load/解析器路径同样复现,相关取消实现与基础提交没有变化。这是既有取消缺陷,未作为本 PR 引入的问题提交。
Review split
This integration PR is superseded by the focused PRs below. Review and merge those units; this PR retains the original source history and review discussion.
mainmainmainmainmainmainmainmainOnly the iframe layer depends on the Markdown layer. The remaining units have no unmerged feature prerequisites. Every PR includes a concrete reproduction, executable regression fixtures and links to the exact test source.
Validation and scope
The final integration source is
1940fb926. Formatting and workspace/all-target/all-feature Clippy pass; the complete workspace nextest suite passes with 20,143 tests and 17 original skips. The split patches reconstruct the complete integration tree exactly on the original base. The extracted PRs retain current main's callback-registry API.Markdown remains a DOM snapshot. These changes add no PDF/DOCX parser, benchmark-specific selector, or extra network extraction step. Permission and media changes preserve the existing capability boundaries.
Published benchmark results remain bound to measured builds
a737f2aecand0d9a7ceb2. This source split does not replace their executables or measurements.