Problem
In StreamingRawDataset, the adaptive concurrency budget for remote downloads was previously calculated using hardcoded static assumptions:
_ASSUMED_AGGREGATE_BANDWIDTH_BPS = 100 * 1024 * 1024 # ~100 MB/s
_ASSUMED_REQUEST_RATE = 6000.0 # target req/s
_ASSUMED_REQUEST_LATENCY_S = 0.040 # 40 ms RTT
_DEFAULT_MEDIAN_FILE_BYTES = 256 * 1024
These assumptions do not reflect the large variation in real-world cloud environments.
-
Low-Bandwidth / Shared Nodes: Multi-tenant EKS/Kubernetes nodes may provide only ~10–20 MB/s of effective bandwidth. The static model can overestimate sustainable concurrency, leading to HTTP 429 throttling, socket saturation, and connection timeouts.
-
High-Bandwidth / High-Performance Storage: Dedicated A100/H100 nodes with multi-Gbps networking or low-latency object storage can support significantly higher throughput. The conservative static cap can unnecessarily limit concurrency, resulting in GPU underutilization and reduced data-loading throughput.
-
No Empirical Feedback Loop: Actual download throughput and request latency were not incorporated into subsequent concurrency decisions. As a result, the concurrency budget remained largely independent of the performance characteristics of the current network and remote storage.
in short:
The previous concurrency model was therefore configuration-driven rather than measurement-driven. A more robust approach should use empirical runtime measurements—such as observed download throughput and request latency—to dynamically estimate sustainable concurrency for the current execution environment.
Problem
In
StreamingRawDataset, the adaptive concurrency budget for remote downloads was previously calculated using hardcoded static assumptions:These assumptions do not reflect the large variation in real-world cloud environments.
Low-Bandwidth / Shared Nodes: Multi-tenant EKS/Kubernetes nodes may provide only ~10–20 MB/s of effective bandwidth. The static model can overestimate sustainable concurrency, leading to HTTP 429 throttling, socket saturation, and connection timeouts.
High-Bandwidth / High-Performance Storage: Dedicated A100/H100 nodes with multi-Gbps networking or low-latency object storage can support significantly higher throughput. The conservative static cap can unnecessarily limit concurrency, resulting in GPU underutilization and reduced data-loading throughput.
No Empirical Feedback Loop: Actual download throughput and request latency were not incorporated into subsequent concurrency decisions. As a result, the concurrency budget remained largely independent of the performance characteristics of the current network and remote storage.
in short:
The previous concurrency model was therefore configuration-driven rather than measurement-driven. A more robust approach should use empirical runtime measurements—such as observed download throughput and request latency—to dynamically estimate sustainable concurrency for the current execution environment.