Repository navigation

Don't store resources content in memory - stream to storage (fixes #386) - #643

Open

aivus wants to merge 20 commits into

masterfrom

issue-386-streams

Open

Don't store resources content in memory - stream to storage (fixes #386)#643
aivus wants to merge 20 commits into
masterfrom
issue-386-streams

Commits on Jul 3, 2026
- Add Resource content stream model and stream/fs utils (#386)
  
  aivus
  and
  claude
  committed
  6deb6cf
- Rewrite request layer around got.stream with retry support (#386)
  
  aivus
  and
  claude
  committed
  9585992
- Rework scraper lifecycle to stream resources to storage (#386)
  
  aivus
  and
  claude
  committed
  7395e61
- Add streaming test suites and FS plugin unit tests (#386)
  
  aivus
  and
  claude
  committed
  74bc674
- Add memory benchmark, document v7 streaming changes, bump to 7.0.0 (#386)
  
  aivus
  and
  claude
  committed
  a3c6754
- Merge remote-tracking branch 'origin/master' into issue-386-streams
  aivus
  committed
  ace35a4
Commits on Jul 4, 2026
- Ignore benchmark directory in qlty code coverage
  
  aivus
  and
  claude
  committed
  420a146
Commits on Jul 5, 2026
- Ignore qlty for test directory
  aivus
  committed
  17c58a5
Commits on Oct 19, 2026
- Address review feedback on streaming rework (#386)
  
  aivus
  and
  claude
  committed
  fa91c5f
- Document that afterResponse null body skips the resource (#386)
  
  aivus
  and
  claude
  committed
  1da5e88
- Derive afterResponse string body encoding from content-type (#386)
  
  aivus
  and
  claude
  committed
  8fb52a6
- Use nullish assignment for getBody memoization
  
  aivus
  and
  claude
  committed
  047f83c
- Reduce returns in applyAfterResponse by extracting helpers
  
  aivus
  and
  claude
  committed
  516e7a1
- Merge remote-tracking branch 'origin/master' into issue-386-streams
  
  aivus
  and
  claude
  committed
  a0ec0fb
Commits on Oct 25, 2026
- Don't abort the scrape when a streamed child resource fails mid-body (#386)
  
  aivus
  and
  claude
  committed
  cc1efcd
- Fail the save when the live stream errors, even if the action never settles (#386)
  
  aivus
  and
  claude
  committed
  704720d
- Retry mid-body errors for buffered resources (#386)
  
  aivus
  and
  claude
  committed
  51166ee
- Reject getBody() calls after the afterResponse action has returned (#386)
  
  aivus
  and
  claude
  committed
  18a1160
- Hand out a live content stream only once from getContentStream() (#386)
  
  aivus
  and
  claude
  committed
  3cf7736
- Remove the output directory on error only if the fs plugin wrote to it (#386)
  
  aivus
  and
  claude
  committed
  eff5146

Commits

Commits on Jul 3, 2026

Commits on Jul 4, 2026

Commits on Jul 5, 2026

Commits on Oct 19, 2026

Commits on Oct 25, 2026