[Performance] TP prefill significantly slower than autosplit on 2x RTX 3080 (sm_86)

[sorasoras]

Summary

Tensor parallelism (TP) across 2× RTX 3080 makes prefill (prompt processing) dramatically slower than autosplit (layer split) on the same hardware, while consuming less total power.

Environment

GPU	2× NVIDIA RTX 3080 20 GB (Ampere, sm_86), PCIe — no NVLink
OS	Windows 10/11 x64
Python	3.12
torch	2.9.0+cu128
exllamav3	1.4.2 (prebuilt exllamav3-1.4.2+cu128.torch2.9.0-cp312-cp312-win_amd64.whl)
Model	Qwen3.8-27B EXL3 3.50bpw
Host	tabbyAPI

Observation

Same ~107k-token prompt:

• Autosplit (gpu_split_auto: true, tensor_parallel: false): prefill completed in ~3.5 min at ~513 T/s (107,253 tokens). GPU 0 ~300 W, GPU 1 ~70 W (total ~370 W).
• Tensor parallel (tensor_parallel: true, tensor_parallel_backend: native): prefill was still running after 7+ minutes for the same prompt — I had to abort it. GPU 0 ~150 W, GPU 1 ~150 W (total ~300 W).

Power / utilization

Autosplit drives GPU 0 to ~300 W (GPU 1 ~70 W, ~370 W total) and is faster, while TP spreads the load evenly at ~150 W per GPU (~300 W total) and is much slower. TP uses less total power for less throughput — the signature of inter-GPU communication (PCIe all-reduce, no NVLink on 3080) dominating over compute, so the second GPU isn't adding proportional throughput.

Expected

TP prefill should be comparable to (or faster than) autosplit, not dramatically slower.

Notes

• TP also crashes on the first generation request on this setup (see Tensor Parallel crashes on Windows: illegal memory access in parallel/barrier.cu:47 #244, Illegal memory access in coop_autotune.cu:464 on RTX 3090 (sm_86), Qwen3.8-27B EXL3 #291); this report is about prefill throughput when it does run.

[turboderp]

TP requires the state to synchronize multiple times during the forward pass. Here it happens over PCIe (potentially on an x4 link depending on your hardware) and that can become a serious bottleneck during prefill. Qwen3.8-27B will have 128 synchronizations per prefill chunk. That's something like 4 GB of total traffic for 2048 token chunk, and I can't necessarily make that any faster.

I'm not sure what role Windows is playing in this, since the feature is developed on Linux. Also, your CPU can be a factor if it's very slow. And if the second GPU is on some chipset PCIe port that can complicate things dramatically as well. But are you sure it's actually that much slower and hadn't simply crashed by the time you killed the process? What does performance look like on shorter prompts?

[sorasoras]

TP requires the state to synchronize multiple times during the forward pass. Here it happens over PCIe (potentially on an x4 link depending on your hardware) and that can become a serious bottleneck during prefill. Qwen3.8-27B will have 128 synchronizations per prefill chunk. That's something like 4 GB of total traffic for 2048 token chunk, and I can't necessarily make that any faster.

I'm not sure what role Windows is playing in this, since the feature is developed on Linux. Also, your CPU can be a factor if it's very slow. And if the second GPU is on some chipset PCIe port that can complicate things dramatically as well. But are you sure it's actually that much slower and hadn't simply crashed by the time you killed the process? What does performance look like on shorter prompts?

Does Pcie play roles in prefiling？ CPU is fine but pcie is slow.

[turboderp]

Yes, PCIe is the most significant bottleneck for prefill.

[sorasoras]

Yes, PCIe is the most significant bottleneck for prefill.

Can a PP on the first GPU but decode on both GPU work aka TP but only decode?