WebClient performance collapse at high concurrency with 300KB octet-stream body

[bigstar119]

Environment

• Spring Boot: 3.5.14

• Reactor Netty: (version from Spring Boot BOM)

• JDK: 17

• OS: Linux (CentOS/RHEL)

• Network: 10GbE, cross-machine deployment

Problem Description

We are using WebClient (Reactor Netty) to send 300KB binary data (application/octet-stream) to a remote HTTP service. Under high concurrency (200+ virtual users), the QPS collapses dramatically while latency spikes. The same scenario with RestTemplate (Apache HttpComponents, blocking I/O) maintains stable throughput with no degradation.

Client Configuration

@Configurationpublic class BenchmarkWebClientConfiguration {​    @Bean    public ReactorClientHttpConnector reactorClientHttpConnector() {        ConnectionProvider connectionProvider = ConnectionProvider.builder("bench-pool")                .maxConnections(800)                .maxIdleTime(Duration.ofSeconds(30))                .maxLifeTime(Duration.ofMinutes(5))                .pendingAcquireMaxCount(2000)                .pendingAcquireTimeout(Duration.ofSeconds(10))                .fifo()                .build();​        LoopResources loopResources = LoopResources.create("bench-io", 8, true);​        HttpResources.set(connectionProvider);        HttpResources.set(loopResources);​        HttpClient httpClient = HttpClient.newConnection()                .option(ChannelOption.WRITE_BUFFER_WATER_MARK,                        new WriteBufferWaterMark(64 * 1024, 256 * 1024));​        return new ReactorClientHttpConnector(httpClient);    }​    @Bean(name = "loadBalanceWebClient")    @ConditionalOnBean(ReactorLoadBalancerExchangeFilterFunction.class)    public WebClient loadBalanceWebClient(ReactorClientHttpConnector connector,                                          ReactorLoadBalancerExchangeFilterFunction loadBalancerFilter) {        return WebClient.builder()                .clientConnector(connector)                .filter(loadBalancerFilter)                .build();    }​    @Bean(name = "benchmarkScheduler")    public Scheduler benchmarkScheduler() {        return Schedulers.newBoundedElastic(                Schedulers.DEFAULT_BOUNDED_ELASTIC_SIZE,                Integer.MAX_VALUE,                "bench-cpu",                60,                true);    }}

Client Code

@Componentpublic class OctetStreamBodyRushTestCase extends RushCase {​    @Resource(name = "loadBalanceWebClient")    private WebClient webClient;​    @Resource(name = "benchmarkScheduler")    private Scheduler benchmarkScheduler;​    private static final byte[] BODY_BYTES = RandomUtil.randomBytes(300 * 1000);​    @Override    public CompletableFuture<?> execute() {        String uri = "http://atomic-service/api/v1/benchmark";        CompletableFuture<InternalMonitorRsp> future = new CompletableFuture<>();​        Mono<String> response = webClient.post().uri(uri)                .contentType(MediaType.APPLICATION_OCTET_STREAM)                .bodyValue(BODY_BYTES)                .retrieve()                .onStatus(HttpStatusCode::isError,                        res -> Mono.error(new RuntimeException(                                res.statusCode().value() + ":" + res.statusCode())))                .bodyToMono(String.class)                .publishOn(benchmarkScheduler)                .timeout(Duration.ofSeconds(30))                .doOnSuccess(s -> {                    SystemResponse<InternalMonitorRsp> systemResponse = JSON.parseObject(                            s, new TypeReference<SystemResponse<InternalMonitorRsp>>() {});                    if (SystemCodeEnum.SYS_200.getCode().equals(systemResponse.getCode())) {                        future.complete(systemResponse.getData());                    } else {                        future.complete(NO_HIT_INTERNAL_MONITOR_RSP);                    }                })                .doOnError(ex -> {                    log.error("http request error, fallback", ex);                    future.complete(NO_HIT_INTERNAL_MONITOR_RSP);                });        response.subscribe();​        CompletableFuture[] completableFutures = {                future.thenAccept(resp -> Assert.notNull(resp, "response is null"))        };        return CompletableFuture.allOf(completableFutures);    }}

WebClient is configured with ReactorClientHttpConnector and ReactorLoadBalancerExchangeFilterFunction for service discovery (Nacos).

Server Code

The server endpoint simply receives the 300KB body and returns a JSON response:

@PostMapping(value = "/api/v1/benchmark", consumes = MediaType.APPLICATION_OCTET_STREAM_VALUE)public SystemResponse<MonitorRsp> benchmark(@RequestBody byte[] body) {    return new SystemResponse<>(processBody(body));}

Server runs on Undertow container. Same-machine testing confirms the server has no performance bottleneck (QPS 4132 at 200 VU, same-machine).

Performance Data

WebClient (Reactor Netty) - Cross-machine, 10GbE

VU	Total Requests	Success Rate	QPS	P50	P90	P95	P99	AVG
100	1,049,421	100.00%	3,497.75	28.88ms	33.72ms	36.37ms	47.15ms	28.51ms
200	588,049	100.00%	1,959.73	73.47ms	191.76ms	219.94ms	410.26ms	101.96ms
300	230,381	100.00%	767.68	341.31ms	665.32ms	687.34ms	715.65ms	390.62ms

QPS drops 44% at 200 VU and 78% at 300 VU. P99 latency increases 15x (47ms -> 715ms).

RestTemplate (Apache HttpComponents, blocking I/O) - Same scenario

VU	Total Requests	Success Rate	QPS	P50	P90	P95	P99	AVG
100	1,107,616	100.00%	3,691.72	27.34ms	28.39ms	29.00ms	31.49ms	27.03ms
200	1,093,366	100.00%	3,643.90	55.35ms	57.90ms	59.57ms	61.73ms	54.82ms
300	1,077,186	100.00%	3,589.64	83.62ms	88.15ms	89.78ms	91.82ms	83.50ms

QPS stays stable at ~3,600 across all VU levels. Latency grows linearly (expected).

webclient-issue-draft.md

[bigstar119]

because issue body can not too big. details please see the webclient-issue-draft.md

[samueldlightfoot]

Hi - I think you have a setup issue here.

Your connection pool is never used. HttpClient.newConnection() hardwires NewConnectionProvider, so the bench-pool you install with HttpResources.set(connectionProvider) is unreachable from that client. Every request opens a fresh TCP connection and closes it.

The fix is one line:

HttpClient httpClient = HttpClient.create(connectionProvider)   // was: HttpClient.newConnection()
        .option(ChannelOption.WRITE_BUFFER_WATER_MARK,
                new WriteBufferWaterMark(64 * 1024, 256 * 1024));

[bigstar119]

Thanks for you advice , I follow your advice fix the code like below , but has the same problem also.
@configuration
public class WebClientConfiguration {

@Bean
public ReactorClientHttpConnector reactorClientHttpConnector(OrchestrationProperties orchestrationProperties) {

    OrchestrationProperties.WebClientConfig webClientConfig = orchestrationProperties.getWebClient();

    ConnectionProvider connectionProvider = ConnectionProvider.builder("bench-pool")
            .maxConnections(webClientConfig.getMaxConnections())
            .maxIdleTime(webClientConfig.getPoolConfig().getMaxIdleTime())
            .maxLifeTime(webClientConfig.getPoolConfig().getMaxLifeTime())
            .pendingAcquireMaxCount(webClientConfig.getPoolConfig().getPendingAcquireMaxCount())
            .pendingAcquireTimeout(webClientConfig.getPoolConfig().getPendingAcquireTimeout())
            .fifo()
            .build();

    int workerCount = Math.max(Runtime.getRuntime().availableProcessors(), orchestrationProperties.getWebClient().getWorkerCount());
    LoopResources loopResources = LoopResources.create("bench-io", workerCount, true);

    HttpResources.set(connectionProvider);
    HttpResources.set(loopResources);

    HttpClient httpClient = HttpClient.create(connectionProvider)
            .option(ChannelOption.SO_KEEPALIVE, true)
            .option(ChannelOption.TCP_NODELAY, true)
            .option(ChannelOption.WRITE_BUFFER_WATER_MARK, new WriteBufferWaterMark(256 * 1024, 1024 * 1024));

    if (webClientConfig.getReceiveBuffer().toBytes() > 0) {
        httpClient = httpClient.option(ChannelOption.SO_RCVBUF, (int) webClientConfig.getReceiveBuffer().toBytes());
    }
    if (webClientConfig.getSendBuffer().toBytes() > 0) {
        httpClient = httpClient.option(ChannelOption.SO_SNDBUF, (int) webClientConfig.getSendBuffer().toBytes());
    }

    return new ReactorClientHttpConnector(httpClient);
}

@Bean(name = "loadBalanceWebClient")
public WebClient loadBalanceWebClient(ReactorClientHttpConnector connector,
                                      ReactorLoadBalancerExchangeFilterFunction loadBalancerFilter) {
    return WebClient.builder()
            .clientConnector(connector)
            .filter(loadBalancerFilter)
            .build();
}


@Bean(name = "benchmarkScheduler")
public Scheduler benchmarkScheduler() {
    return Schedulers.newBoundedElastic(
            Schedulers.DEFAULT_BOUNDED_ELASTIC_SIZE,
            Integer.MAX_VALUE,
            "bench-cpu",
            60,
            true);
}

}

[samueldlightfoot]

I tried to reproduce this and couldn't.

I transcribed your client, bodyValue(byte[]), retrieve, bodyToMono, publishOn(boundedElastic), timeout(30s), the fastjson parse, and the load-balancer filter with a fixed instance list standing in for Nacos and ran it against a RestTemplate arm on pooled Apache HttpClient 5 with the same connection limits. Same 300 KB body, closed-loop driver, reactor-netty 1.2.17. Client and server pinned to separate cores, server behind a link shaped to 8 Gbit.

Same columns you used, 60s per level (hence ~191K requests rather than ~1M).

WebClient (reactor-netty 1.2.17)

VU	Total Requests	Success Rate	QPS	P50	P90	P95	P99	AVG
100	191,208	100.00%	3,185.00	31.40ms	31.44ms	31.46ms	31.49ms	31.39ms
200	191,318	100.00%	3,185.00	62.79ms	62.86ms	62.89ms	62.97ms	62.76ms
300	191,414	100.00%	3,185.00	94.19ms	94.35ms	94.45ms	94.72ms	94.11ms

RestTemplate (Apache HttpComponents, blocking)

VU	Total Requests	Success Rate	QPS	P50	P90	P95	P99	AVG
100	191,105	100.00%	3,183.40	31.41ms	31.45ms	31.46ms	31.48ms	31.40ms
200	191,204	100.00%	3,183.40	62.82ms	62.87ms	62.89ms	62.95ms	62.79ms
300	191,305	100.00%	3,183.40	94.24ms	94.32ms	94.37ms	94.53ms	94.16ms

Your RestTemplate arm reproduces. Your WebClient arm does not: at 300 VU it holds 3,185 QPS against your 768, and P99 stays at 94.72ms against your 715.65ms. My two arms are within 0.05% of each other on every row.

Notes:

• WebClient uses 12–26% more CPU per request than RestTemplate (client-JVM CPU time ÷ requests completed). Within WebClient's own per-request CPU, the load-balancer filter accounts for 7.1%, publishOn + timeout for 8.7%, and the bodyValue(byte[]) copy for 5.3%. All real, all far too small to explain a 4× throughput gap.
• Raw reactor-netty with no Spring uses less CPU per request than blocking Apache, so the extra cost sits above the transport rather than in it.
• Starving the client of CPU does not produce your symptom. Past the CPU limit WebClient degrades better than RestTemplate: 9% more throughput, and P99 91ms against RestTemplate's 650ms.
• Adding 0.15% packet loss reproduces your tail but not your throughput, at 300 VU my P99 rose to 440ms (yours: 715.65ms) while throughput stayed at the ceiling on both arms. Your latency and throughput symptoms may not share a cause.

The largest untested difference between your setup and mine is Nacos: you resolve http://atomic-service/ through live discovery on every request, and I used a static instance list. Could you point WebClient at a literal host:port, change nothing else, and re-run 300 VU?

// was: "http://atomic-service/api/v1/benchmark"
String uri = "http://10.x.x.x:8080/api/v1/benchmark";   // one instance, resolved by hand

I measured the filter's steady-state cost at 7.1% of CPU per request, so if removing it recovers a 4× collapse then the cause is not the filter's CPU, it is something blocking on the event loop behind it, and that would be a genuine bug worth chasing. If it changes nothing, the whole discovery path is ruled out, which is also worth knowing.

Related, and possibly the same question: is your RestTemplate arm load-balanced through Nacos as well, or does it target a fixed host? If it is fixed-host, the two arms are not doing the same work per request and that difference alone could account for the gap.

One last thing if it is cheap: top -H -p <client-pid> during the 300 VU run. Aggregate CPU% hides the case that matters here, a few reactor-http-nio-* threads pegged at 100% while the machine as a whole looks idle.

[bigstar119]

Reply to reactor-netty maintainer

Thank you for the thorough investigation. We followed your suggestion and re-ran the test with a direct host:port URI, bypassing Nacos entirely. The results on our environment still differ significantly from yours, and we have collected deeper diagnostics that we hope you can help us analyze.

Direct-IP Test Results

Both arms now target a literal http://10.x.x.x:7731/api/v1/benchmark with no service discovery involved.

WebClient (reactor-netty)

VU	Total Requests	Success Rate	QPS	P50	P90	P95	P99	AVG
100	1,633,371	100.00%	5,441.84	10.90ms	42.47ms	55.74ms	89.39ms	18.31ms
200	780,027	100.00%	2,598.15	62.72ms	122.42ms	208.93ms	279.18ms	76.86ms
300	884,685	100.00%	2,946.75	84.67ms	156.37ms	246.42ms	347.87ms	101.66ms

RestTemplate (Apache HttpComponents, blocking)

VU	Total Requests	Success Rate	QPS	P50	P90	P95	P99	AVG
100	2,903,428	100.00%	9,677.78	8.28ms	17.19ms	21.32ms	34.37ms	10.23ms
200	881,493~	100.00%	2,936.12	55.35ms	105.71ms	169.35ms	270.27ms	67.99ms
300	698,117	100.00%	2,325.01	95.42ms	268.70ms	326.63ms	436.99ms	128.85ms

Both arms degrade at high VU on our environment. Unlike your result where both held a steady 3,185 QPS with overlapping curves, ours show significant throughput collapse. Notably, RestTemplate is now worse than WebClient at 300 VU (2,325 vs 2,947), which contradicts our earlier observation where RestTemplate was stable.

Our Environment

Both client and server are virtual machines on a virtualized platform, connected through a virtual switch (vSwitch). The virtual NIC has only 4 combined channels:

$ ethtool -l ens18
Channel parameters for ens18:
Pre-set maximums:
  Combined: 4
Current hardware settings:
  Combined: 4

Congestion control is CUBIC on both sides:

$ sysctl net.ipv4.tcp_congestion_control
net.ipv4.tcp_congestion_control = cubic

Evidence: TCP-Level Packet Loss

NIC-level: zero drops on both sides

# Client
$ ip -s link show ens18
    RX: errors=0  dropped=0  overrun=0
    TX: errors=0  dropped=0
​
# Server
$ ip -s link show ens18
    RX: errors=0  dropped=0  overrun=0
    TX: errors=0  dropped=0

TCP-level: continuous loss probe activity

Client:

$ netstat -s | grep TCPLossProbes
    TCPLossProbes: 21,999,854
    TCPLossProbes: 22,001,810   (+1,956 in ~5s)
    TCPLossProbes: 22,003,736   (+1,926)
    TCPLossProbes: 22,005,492   (+1,756)

Server:

$ netstat -s | grep TCPLossProbes
    TCPLossProbes: 2,352,606
    TCPLossProbes: 2,352,866    (+260)
    TCPLossProbes: 2,353,093    (+227)
    TCPLossProbes: 2,353,197    (+104)

Client-side TCP state at 300 VU

cwnd: 5-60 (collapsed from expected 200+)
ssthresh: 2-22
retrans: 15-27 million per connection
dsack_dups: 2,500+ per connection
notsent: 71-260 KB
delivery_rate: 4-98 Mbps per connection (highly variable)

Server-side TCP state at 300 VU

cwnd: 3-10 (collapsed)
ssthresh: 2-7
app_limited: all connections
delivery_rate: ~110-125 Mbps per connection

CPU Profile

Client (top -H)

%Cpu: 3.6 us, 2.9 sy, 88.1% idle
load average: 2.07
​
Per-thread breakdown (bench-io-epoll-*):
  No thread exceeds 6.6% CPU
  All 16 event loop threads evenly loaded

No event loop thread is saturated. The client CPU is overwhelmingly idle.

Server (top -H)

%Cpu: 47.0 us, 26.5 sy, 12.7 si, 13.8% idle
load average: 21.74
15 running / 243 threads
​
XNIO I/O threads: 18.8-25.0%
XNIO task threads: 18.8-25.0%

The server is under heavy load. The high si (12.7%) suggests significant software interrupt processing, possibly related to the 4-channel NIC handling interrupts for hundreds of connections.

Our Hypothesis

We suspect the virtualized network infrastructure is causing TCP-level packet loss at high concurrency:

1. 4 NIC channels must handle interrupts for 300+ concurrent TCP connections

2. Under micro-burst traffic (many connections sending 300KB bodies simultaneously), the vSwitch buffer overflows

3. Packets are dropped inside the vSwitch — invisible to guest OS NIC counters, but detected by TCP (hence TCPLossProbes growing while NIC dropped stays at 0)

4. CUBIC responds to loss by collapsing cwnd, which reduces per-connection throughput

5. Both WebClient and RestTemplate are affected equally because the bottleneck is below the application layer

Your 0.15% packet loss experiment is interesting: it reproduced tail latency (P99 440ms) but not throughput collapse. We wonder whether our loss rate is significantly higher, and whether that would explain the throughput drop on both arms.

Questions for You

1. What is your test infrastructure? Physical machines or VMs? What does the network path look like between client and server (switches, virtual switches, NIC queue count)?

2. Given the evidence above (zero NIC drops but continuous TCP loss probes, cwnd collapse on both arms, both clients degrading simultaneously), do you think this is consistent with an infrastructure-level packet loss issue rather than a reactor-netty issue?

3. Your 0.15% loss experiment reproduced latency but not throughput collapse. At what loss rate would you expect to see throughput degradation as well? We are trying to estimate our effective loss rate from the TCP metrics.

4. Any recommendations for obtaining comparable results in our virtualized environment? For example, should we try to increase the NIC channel count, or is there a reactor-netty configuration that could help smooth out burst traffic?

We appreciate your time and the detailed investigation you have already done. Your suggestion to test with a direct IP was instrumental in narrowing down the problem space.

[samueldlightfoot]

I don't think there's anything you can tune in Reactor Netty to help.

On your loss question, I ran a sweep on my rig (8 Gbit, 300 KB bodies, both clients):

• At 0.15% loss the tail blows up but throughput doesn't move at all.
• Throughput only starts dropping around 2-3%, and by 5% it's down 4×.

So latency and throughput break at very different loss rates, which is why my earlier 0.15% test reproduced your P99 but not your QPS.

Worth checking the network path between the VMs, and your server (your client is 88% idle at 300 VU, so there's nothing to tune there).

What is your Apache client configuration formaxTotal and defaultMaxPerRoute? Reactor-netty's pool defaults to max(cores, 8) * 2 connections, which may be far fewer than Apache's.

[bigstar119]

@samueldlightfoot thank you for your advice. this is my HttpComponentsClientHttp Configs
public OsmiumRestTemplate buildRestTemplate(ConfigProperties configProperties) {
HttpComponentsClientHttpRequestFactoryBuilder requestFactoryBuilder = ClientHttpRequestFactoryBuilder.httpComponents();
if (configProperties.isPool()) {
requestFactoryBuilder = requestFactoryBuilder.withConnectionManagerCustomizer(connectionManager -> {
connectionManager.setMaxConnTotal(configProperties.getPoolSize());
connectionManager.setMaxConnPerRoute(20000);
});
}
requestFactoryBuilder = requestFactoryBuilder.withCustomizer(httpComponentsClientHttpRequestFactory -> {
httpComponentsClientHttpRequestFactory.setConnectTimeout(configProperties.getConnectTimeout());
httpComponentsClientHttpRequestFactory.setReadTimeout(configProperties.getTimeout());
httpComponentsClientHttpRequestFactory.setConnectionRequestTimeout(configProperties.getConnectTimeout());
});
HttpComponentsClientHttpRequestFactory requestFactory = requestFactoryBuilder.build();
//
OsmiumRestTemplate restTemplate = new OsmiumRestTemplate(requestFactory);
InterceptorsUtil.addOsmiumInterceptors(restTemplate);
restTemplate.getMessageConverters().add(4, this.messageConverter());
return restTemplate;
}

osmium:
http-client:
protocol: http
connect-timeout: 3000
timeout: 5000
pool-size: 1000

[samueldlightfoot]

You have different client configurations which may explain the difference.

maxConnPerRoute: 20000 with pool-size: 1000 means Apache is effectively uncapped: it opens one connection per virtual user. Reactor-netty's pool defaults to max(cores, 8) * 2, so on a 16-core box that's 32 connections irrespective of VUs.

Can you lower the Apache pool at 300 VU to 100 & 300 and see if throughput changes.

[samueldlightfoot]

@violetagg I think this can have the bug label removed. At this point they are seeing worse performance with RestTemplate, so it's not something in Reactor Netty, but in their infra, I believe. It would still be interesting to debug this fully @bigstar119.

[github-actions]

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.