feat(lib-storage): S3TransferManager by smilkuri · Pull Request #7731 · aws/aws-sdk-js-v3

smilkuri · 2026-02-11T16:50:24Z

Issue

Internal JS-6610

Description

This PR adds upload functionality to the S3 Transfer Manager, building upon previous PR that wasn't merged. This implementation focuses specifically on single object and multipart upload capabilities commit.

Testing

 yarn test:e2e
  ✓ src/s3-transfer-manager/S3TransferManager.e2e.spec.ts (26 tests) 274836ms

 Test Files  1 passed | 1 skipped (2)
      Tests  26 passed | 92 skipped (118)
   Start at  16:42:31
   Duration  275.51s (transform 665ms, setup 0ms, import 1.13s, tests 274.84s, environment 0ms)

 RUN  v4.0.18 /local/home/smilkuri/aws-sdk-js-v3/lib/lib-storage

Checklist

If the PR is a feature, add integration tests (*.integ.spec.ts).
If you wrote E2E tests, are they resilient to concurrent I/O?
If adding new public functions, did you add the @public tag and enable doc generation on the package?

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

GarrettBeatty · 2026-02-14T03:01:04Z

+   * @internal
+   */
+  private calculatePartSize(contentLength: number): { partSize: number; expectedPartsCount: number } {
+    const partSize = Math.max(this.targetPartSizeBytes, Math.floor(contentLength / 10_000));


pretty sure this needs to be Math.ceil(contentLength / 10_000). check out this issue aws/aws-sdk-net#4332 you might want to add a test case for this

GarrettBeatty

added one comment. i can review rest of pr later if wanted

kuhe · 2026-03-12T18:57:41Z

+                    (response.Body as any).getReader = function () {
+                      return reader;
+                    };
+                  }


what is this override of getReader doing?

uses the same reader instance, when stream is already locked.

that seems dangerous and counter to the point of having a lockable stream

kuhe · 2026-04-08T14:48:49Z

+  private readonly eventListeners: TransferEventListeners;
+  private readonly abortCleanupFunctions = new WeakMap<AbortSignal, () => void>();
+  private readonly maxConcurrentRequests: number;
+  private readonly maxConcurrentUploads: number;


explain the state variables.

what is targetPartSizeBytes? why is checksumAlgorithm a property of this class?

what is the difference between maxConcurrentRequests and maxConcurrentUploads?

targetPartSizeBytes // Size of a part user wants to use for transfer

maxConcurrentRequests // This used only for downloads which determines max no of parts that can be downloaded concurrently or call it like max no of getObject requests that can be made.

maxConcurrentUploads // This is for upload, max no of parts that can be uploaded concurrently

maxConcurrentRequests should be renamed. Uploads are requests too.

renamed it to maxInMemoryParts according to specification wording

maxInMemoryParts is misleading. Are the parts fully buffered in memory? I think not.

And it only applies to downloads?

It should be called maxConcurrentDownloads

+1 there is two separate things. maxConcurrentDownloads (number of http requests) and maxInMemoryParts (number of parts buffered in memory) (if applicable)

+1 to Garrett's comment. Yeah, was a bit confused here.

Removed maxInMemoryParts for downloads because we only initiate the next request after a stream is consumed. So the number of buffered parts can't exceed maxConcurrentDownloads.

feat(lib-storage): added doc comments for interfaces and types feat(lib-storage): address PR review feedback for transfer manager types feat(lib-storage): addressed minor review feedback feat(lib-storage): added example code file feat(lib-storage): changed "handler" to "listener" and changed type file names feat(lib-storage): created TransferManager class and constructor with defaults feat(lib-storage): beginning implementation for download(), created TM index file feat(lib-storage): range multipart download feat(lib-storage): download() improvements post pair-programming feat(lib-storage): transfermanager download() iteration feat(lib-storage): transfermanager interation post pair programming feat(lib-storage): joinstream iteration, web stream not fully functional feat(lib-storage): both cases of range download handled feat(lib-storage): range download working chore: acquire lock on streams feat(lib-storage): bug fixes and test env setup feat(lib-storage): implemented dispatchEvent(), need to add dispatches for complete and fail feat(lib-storage): requests array, eventListeners revision, needs more testing feat(lib-storage): addEventListener & removeEventListener implemented, needs support for options feat(lib-storage): added support for adding event listeners at request level feat(lib-storage): added ETag verification for subsequent GetObjectRequests feat(lib-storage): totalSize changes, and added validateExpectedRanges() feat(lib-storage): addEventListener once parameter, type fixes feat(lib-storage): s3TM SEP test cases and added TODOs feat(lib-storage): validateExpectedRanges fixes and unit tests feat(lib-storage): added check in validateExpectedRanges if final part doesnt download total object feat(lib-storage): s3TM constructor and add event listener tests feat(lib-storage): addEventListener tests and error checking adjustment and dispatchEvent tests feat(lib-storage): added test cases

… node

…testing

kuhe · 2026-06-02T19:11:51Z

  },
  "license": "Apache-2.0",
  "dependencies": {
+    "@aws-sdk/config": "workspace:3.1058.0",


don't use internally, import the implementation pkg

kuhe

will be split into multiple PRs, blocking this

smilkuri requested a review from a team as a code owner February 11, 2026 16:50

smilkuri force-pushed the tm-initial branch from 56d8890 to 8556651 Compare February 11, 2026 18:41

GarrettBeatty reviewed Feb 14, 2026

View reviewed changes

smilkuri force-pushed the tm-initial branch 4 times, most recently from 1bea79a to 8263551 Compare February 18, 2026 22:04

smilkuri force-pushed the tm-initial branch 2 times, most recently from 7ff3777 to 31dd619 Compare February 26, 2026 20:35

smilkuri force-pushed the tm-initial branch 2 times, most recently from 2841950 to 16d443e Compare March 12, 2026 16:59

kuhe reviewed Mar 12, 2026

View reviewed changes

smilkuri force-pushed the tm-initial branch from 16d443e to e413d49 Compare April 8, 2026 03:40

kuhe reviewed Apr 8, 2026

View reviewed changes

Comment thread lib/lib-storage/src/s3-transfer-manager/S3TransferManager.ts Outdated

kuhe reviewed Apr 8, 2026

View reviewed changes

Comment thread lib/lib-storage/src/s3-transfer-manager/S3TransferManager.ts Outdated

kuhe reviewed Apr 8, 2026

View reviewed changes

Comment thread lib/lib-storage/src/s3-transfer-manager/S3TransferManager.ts Outdated

kuhe reviewed Apr 8, 2026

View reviewed changes

smilkuri force-pushed the tm-initial branch 3 times, most recently from eea6c87 to b20066e Compare April 16, 2026 11:43

smilkuri force-pushed the tm-initial branch from b20066e to f006e06 Compare April 23, 2026 15:50

smilkuri force-pushed the tm-initial branch from f006e06 to 60f69f7 Compare June 1, 2026 22:00

lukasdchang and others added 6 commits June 2, 2026 03:06

feat(lib-storage): concurrent requests

947e0d3

feat(lib-storage): spec updates

f9b5329

feat: addressed pr comments

667c29e

feat: addresses change requests for range and part validation

d13ba76

feat: validate part and range test cases

e1819f4

lukasdchang and others added 17 commits June 2, 2026 03:06

feat: promise.all temp fix for promise unhandled rejection issue

830f08f

chore: update join-streams browser to be consistent with join-streams…

5285a5b

… node

chore: cr changes for readme and added download() examples

ec368d6

chore: deleted redundant example code file

04fe586

chore: cr nits

e8a1bcc

chore: deleted unused file

eaa6a14

feat(lib-storage): add multipart upload functionality

8bcf5f2

test(lib-storage): s3tm test examples

678918f

test(lib-storage): update test cases for join streams

1db567e

fix(lib-storage): enable checksum validation for download

6613292

fix(lib-storage): support for blob and tests for event listeners

919c8e5

fix(lib-storage): respect concurrency for download

e88d9eb

feat(lib-storage): update multipart upload functionality

3137c3f

fix(lib-storage): address comments of rev-1 for upload

f63cb9b

fix(lib-storage): remove redundant checks

67f7aa3

fix(lib-storage): address minor comments about wording

2ca1f82

fix(lib-storage): update default maxConcurrentUploads value based on …

a20072b

…testing

smilkuri force-pushed the tm-initial branch 5 times, most recently from 0dedb30 to 3da7b90 Compare June 2, 2026 17:42

smilkuri added 3 commits June 2, 2026 17:50

feat: multipart uploads using worker threads

7bbc985

chore: update inliner script to exclude worker script from inlining

142452f

fix(lib-storage): add dummy browser mapping for worker-http-handler

b2ed762

smilkuri force-pushed the tm-initial branch from 3da7b90 to b2ed762 Compare June 2, 2026 17:52

kuhe reviewed Jun 2, 2026

View reviewed changes

smilkuri mentioned this pull request Jun 4, 2026

feat(lib-transfer-manager): add @aws-sdk/lib-transfer-manager as private package #8074

Merged

8 tasks

samuellagrene90-cpu approved these changes Jun 8, 2026

View reviewed changes

kuhe requested changes Jun 8, 2026

View reviewed changes

Uh oh!

Conversation

smilkuri commented Feb 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Issue

Description

Testing

Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

GarrettBeatty left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kuhe left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

smilkuri commented Feb 11, 2026 •

edited

Loading