Repository navigation
fix(fetch): align HTML transfer timing and reduce observer overhead - #1702
Merged
Merged
Conversation
Select a fresh ResponseProgress for every HTML attempt and pass that exact writer token to the collector, matching raw streaming. Redirects and retries must not retain observations from a discarded hop. Add a gated loopback 103 -> 302 with body -> 200 without body regression.
Record response end before publishing native success or failure, including cancellation, header-only responses and declared identity body completion. Keep cache availability separate from the end of the cached body reader. Cover delayed consumers, chunked bodies, empty responses, truncated transfers, cancellation, cache hits and encoded Content-Length comparison boundaries. Observe delivered body bytes before publishing declared completion.
Reuse the existing interim status predicate in buffered, HTML and raw collectors so Switching Protocols starts the final response timeline. Cover 101 alone and after 103 across all collectors, plus HTML header completion.
Keep cancellation arbitration and timing in one Arc<ResponseProgressState>. This removes one allocation per progress generation and one reference-count update per clone, reducing the writer token from 16 to 8 bytes on x86_64. Preserve transfer isolation and the existing atomic completion/cancel protocol.
Latch the first body observation in the transfer handler and reset it whenever the easy handle is reused. Record nonempty writes before declared completion, while retaining progress-based detection for compressed empty payloads. Avoid repeated timing locks without skipping cancellation checks.
Redirects and retries retain start/body senders in the reused easy handle. Return those channels to the HTML job before publishing a final cache hit, and use the retained easy when an attempt fails before reconfiguration. Cover network-to-cache redirects with discarded interim/body observations, and preserve the original error instead of reporting a closed start channel.
Keep zero-length completion local to header processing and retain only positive body-length targets in the HTML and raw streaming collectors. Preserve HTML redirect eligibility before committing an empty response. On the same x86_64 dev build, the field shrinks from 16 to 8 bytes, HTML collectors from 2752 to 2744 bytes, and raw collectors from 2728 to 2720 bytes. The enclosing transfer handler also shrinks by 8 bytes. Cover accepted empty responses, unknown lengths and followed empty redirects without changing their completion/cancellation behavior.
Spider Bench A/B✅ All benchmark browser service runs completed; results are informational. Public HEAD: 184 / 240 rows; 38 / 48 sites produced rows; 10 unexpected empty sites. ✅ Deterministic fixture contract is clean. Common ancestor Public 48-site run · informational
Public-site content, timing, and memory are noisy. This single A/B run reports evidence only; site outcome counts explain missing rows without treating them as a deterministic regression. CPU and memory timelinesThese are bounded 40-point views of the same complete process-tree samples used by the HTML report. CPU uses 100% per occupied logical core; memory lines are RSS first and PSS second when complete PSS samples are available. Basexychart-beta
title "Base CPU"
x-axis "Elapsed seconds" [1.036, 1.536, 2.538, 4.039, 5.041, 6.543, 8.046, 9.047, 9.548, 10.549, 12.052, 13.054, 14.555, 15.555, 17.057, 18.559, 19.56, 21.062, 22.062, 23.566, 24.066, 25.067, 26.068, 27.569, 28.571, 30.073, 31.574, 32.576, 34.077, 35.579, 36.58, 38.081, 39.083, 40.585, 42.086, 43.087, 44.589, 45.59]
y-axis "CPU percent" 0 --> 500
line [2, 87.95, 127.73, 141.95, 141.96, 139.84, 173.84, 101.78, 197.8, 143.75, 139.77, 149.61, 127.85, 103.92, 119.85, 2, 85.97, 69.89, 137.94, 99.73, 109.9, 139.99, 162.03, 125.73, 57.93, 155.98, 181.79, 103.79, 113.88, 89.88, 133.85, 2, 3.99, 7.99, 123.88, 132.01, 111.89, 105.89]
xychart-beta
title "Base memory: RSS then PSS"
x-axis "Elapsed seconds" [0.035, 1.036, 1.536, 2.538, 4.039, 5.041, 6.543, 8.046, 9.047, 9.548, 10.549, 12.052, 13.054, 14.555, 15.555, 17.057, 18.559, 19.56, 21.062, 22.062, 23.566, 24.066, 25.067, 26.068, 27.569, 28.571, 30.073, 31.574, 32.576, 34.077, 35.579, 36.58, 38.081, 39.083, 40.585, 42.086, 43.087, 44.589, 45.59, 46.793]
y-axis "Memory MiB" 0 --> 500
line [12.4, 63.15, 97.11, 121.27, 150.4, 158.73, 169.43, 178.12, 184.9, 180.33, 171.02, 189.79, 185.8, 162.07, 185.4, 175.18, 162.37, 165.63, 153.84, 167.74, 221.68, 279.05, 203.39, 209.77, 239.71, 161.09, 158.19, 165.37, 160.47, 164.08, 162.29, 217.76, 167.55, 165.21, 160.68, 184.22, 185.3, 160.47, 178.61, 0]
line [9.43, 59.79, 93.74, 117.89, 146.09, 155.42, 166.1, 174.74, 181.57, 176.95, 167.72, 186.41, 182.33, 158.49, 181.83, 171.58, 158.8, 162.08, 150.27, 164.27, 218.15, 276.66, 199.86, 206.22, 236.44, 157.6, 154.4, 161.79, 156.9, 160.5, 158.69, 214.15, 163.95, 161.61, 157.08, 180.69, 183.3, 156.98, 175.03, 0]
HEADxychart-beta
title "HEAD CPU"
x-axis "Elapsed seconds" [1.537, 2.537, 4.04, 5.543, 6.545, 8.047, 9.55, 11.052, 12.053, 13.554, 15.056, 16.057, 17.558, 19.06, 20.062, 21.564, 23.066, 24.567, 25.568, 26.069, 27.071, 28.572, 29.573, 31.074, 32.575, 33.577, 35.08, 36.582, 37.583, 39.085, 40.588, 42.089, 42.59, 43.091, 44.592, 46.094, 47.095, 48.097]
y-axis "CPU percent" 0 --> 500
line [103.78, 116.03, 155.66, 177.56, 153.77, 115.71, 171.78, 101.94, 55.99, 149.8, 18, 119.94, 47.96, 15.99, 3.99, 87.9, 99.89, 139.92, 99.76, 111.79, 125.78, 99.89, 137.97, 123.83, 124, 119.79, 33.97, 91.74, 29.97, 163.79, 123.81, 16.01, 0, 75.83, 132.04, 63.92, 123.84, 197.64]
xychart-beta
title "HEAD memory: RSS then PSS"
x-axis "Elapsed seconds" [0.035, 1.537, 2.537, 4.04, 5.543, 6.545, 8.047, 9.55, 11.052, 12.053, 13.554, 15.056, 16.057, 17.558, 19.06, 20.062, 21.564, 23.066, 24.567, 25.568, 26.069, 27.071, 28.572, 29.573, 31.074, 32.575, 33.577, 35.08, 36.582, 37.583, 39.085, 40.588, 42.089, 42.59, 43.091, 44.592, 46.094, 47.095, 48.097, 48.491]
y-axis "Memory MiB" 0 --> 500
line [12.37, 101.05, 123.19, 139.6, 155.11, 179.4, 173.68, 179.64, 184.43, 187.14, 159.41, 157.42, 169.46, 176.8, 157.56, 157.09, 167.66, 171.58, 170.86, 274.14, 280.84, 210.47, 230.49, 192.43, 161.57, 192.66, 177.25, 157.79, 158.57, 154.51, 206.82, 190.4, 164.48, 164.48, 176.94, 189.37, 163.23, 182.04, 206.25, 0]
line [9.69, 98.09, 120.22, 136.61, 152.59, 176.44, 170.64, 176.71, 181.73, 184.16, 156.5, 154.43, 166.47, 173.74, 154.58, 154.24, 164.74, 168.87, 167.88, 271.17, 277.51, 207.59, 227.62, 188.55, 158.32, 189.29, 173.97, 154.49, 155.21, 151.15, 202.91, 187.03, 161.18, 161.18, 173.63, 186.12, 159.17, 178.74, 202.64, 0]
Deterministic fixture · required contractThe fixture exercises 15 routes: 8 are expected to emit rows and 7 intentionally exercise empty, loading, or timeout behavior.
Full HTML, JSON, CSV, logs, and page snapshots: workflow run and full The public-web A/B is informational. The exact HEAD fixture contract runs as a separate required CI check; timing and public-site content deltas remain non-blocking. |
CI Regression ReportSource CI run · source state at render:
Release regression — ✅ HEAD/base failures 0/0; raw binary +0.005012%Package and image size
Startup latency and PSS
Concurrency matrix
Frontend differential — ✅ 1,020/1,020 cases matched; 0 issues; 1 Chromium reference recoveries
Recovered Chromium reference infrastructure failures
Agent episodes · Moli vs Chromium — ✅ 8/8 Moli episodes passed; 0 failures
Runtime and CDP session contracts — ✅ 26 contract cases; 0 failures
CDP smoke — ✅ 52/52 groups passed; 621 scenariosWorkers: 4 · cumulative group time: 133.37 s · failed groups: 0
WebMainBench · 545 pages — ✅ 545/545 completed; 0 unexpected failures
Frozen HTML · Linux · Diagnostics: All artifact fields are parsed by the trusted default-branch renderer; missing or invalid inputs remain visible as unavailable. |
Record request start with CURLOPT_PREREQFUNCTION using the handler already owned by Easy2. Reinstall the callback on each attempt and read the collector's writer token across moves, reset and progress reselection. Enable verbose only for requested header observations or proxy CONNECT diagnostics. Cover keepalive GET/HEAD/POST reuse, callback ownership, HTTP/2 multiplexing and gated proxy tunnels.
ldm0
force-pushed
the
fetch-timing-followup
branch
from
October 18, 2026 17:35
dd57dd5 to
9f6055f
Compare
Pin curl and curl-sys to the native callback API from curl-rust#1 and replace the handwritten callback registration with Handler::prereq / Easy2::prereq. Keep each collector's captured timing token and re-enable the callback after reset; the dependency owns userdata lifetime and C panic handling.
Sequential Navigation Soak A/B✅ HEAD completed the 200-navigation resilience and memory observation. One browser process, one target, and one CDP session navigate CSDN → SegmentFault → Huaban → example.com repeatedly. This run issued 200 Common ancestor Session resilience
A public-page failure is reported separately from an unrecoverable session. The soak requires all navigation attempts, zero failed recovery, zero lifecycle/network ordering violations, and complete resource evidence. Process-tree memory
HEAD memory by 50-navigation quarter
Raw reports and per-navigation boundary samples: workflow run and full Public-site timing and memory are observational. Use the A/B deltas and trend shape as evidence, not as a deterministic performance threshold. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
HTML streaming reused one timing token across redirects and retries, and omitted terminal timing on success, failure and cache reads. For example, a 103/302 response with a body followed by an empty 200 could leave the final response reporting the redirect's request start, interim response and body bytes.
Each HTML attempt now captures its own writer token. Native success, failure, cancellation and declared-body boundaries record the terminal timing before publishing completion. Cached availability and cached-reader completion remain separate, without inventing network observations.
The cache audit also reproduced a related failure: a network redirect to an HTML cache hit left response channels inside the reused easy handle, producing
streaming html start channel closed. Cache completion now recovers those channels, and failures before easy reuse preserve their original error.The nine commits separate:
Option<NonZeroUsize>; settle zero-length responses during header processing without committing discarded HTML redirects.dc8c170, and delete the local 54-line FFI adapter.Validation:
cargo fmt --allcargo clippy --workspace --all-targets --all-features -- -D warningscargo nextest run --no-fail-fast: 20,342 passed, 18 skipped.Declared-length storage:
Measured with the same x86_64 dev build and rustc type-layout diagnostics:
The nested handler reflects the same 8B saving. Header-only completion, unknown lengths, compression, chunked transfers and redirect eligibility retain their existing behavior; the new regression checks zero-length acceptance, unknown length and a followed 302.
Navigation and subresource callers that request network metadata still enable verbose to collect actual request headers. The benchmark below isolates the timing-only path.
Request-start semantics and callback ownership:
request_started_atis captured by CURLOPT_PREREQFUNCTION, after a new or reused connection (including TLS setup) is ready and before the origin request is sent. This boundary is earlier thanHeaderOut; it is not presented as an identical timestamp. Pure cache hits do not gain a fabricated request-start timestamp, and a failed CONNECT does not start the origin request.Moli implements
Handler::prereqand enables it witheasy.prereq(true)after each reset. curl-rust owns the native callback registration, stable userdata and C panic boundary, using its existing boxed Inner without another allocation. The collector's captured writer is used instead of the cancel handle's selected generation. The localruntime/request_timing.rsFFI adapter has been removed.curl-rust #1 separates a related existing Multi panic bug from the new API: captured callback panics are rethrown after returning from libcurl, clearing the deferred panic so subsequent callbacks can run. Its Linux/macOS/Windows CI, bundled/system callback tests, 3,151 ABI checks and 15 documentation tests pass; the seven prereq tests also passed 20 stress iterations.
A release A/B probe enabled both callbacks on the same requests, using the pinned libcurl 8.19.0. All 192 requests invoked the pre-request callback once before the origin
HeaderOut; the proxy CONNECT occurred before the origin callback:The committed HTTP/2 regression uses a response barrier and a one-connection limit to prove concurrent streams, and the CONNECT regression explicitly withholds tunnel establishment while checking that no origin request-start has been recorded.
Timing cost evaluation:
A release loopback probe using the newly pinned curl-rust API and libcurl 8.19.0 rotated three modes each round: no request-start observer, verbose + HeaderOut, and
Handler::prereqwith verbose off. The workload was 4,096 HTTP/1.1 keepalive requests, 64 concurrent, with 64B bodies. Nine rounds were measured after one warmup round; all samples were retained.The native pre-request median is about 7.1% below verbose in this timing-only workload. Debug callback invocations fall from about 46,345 to zero per batch, with one pre-request callback per request instead. The control and native-hook ranges overlap, and their small median difference is not evidence that observing timing makes requests faster. These probes isolate transport callback costs; they do not measure browser page-load performance. Navigation and subresource callers requesting actual header metadata still enable verbose.
Body presence takes one timing write per transfer rather than repeatedly locking on download progress. The shared Arc removes one allocation per generation and one reference-count update per clone.
Follow-up to #1666 and #1679. Depends on curl-rust #1, preserving the existing Apple SecTrust integration.