Add replicated Qwen2.5-7B tensor-parallel study - #2
Merged
Conversation
There was a problem hiding this comment.
Pull request overview
Adds a replicated, controlled tensor-parallel (TP1 vs TP2) serving study for Qwen/Qwen2.5-7B-Instruct on Modal L4 GPUs, extending the repo beyond prefix-cache experiments with multi-GPU topology comparison artifacts and reporting.
Changes:
- Extend the Modal paired server/
AsyncLLMharness to support--modal-gpu L4:2and record/propagatetensor_parallel_size. - Add comparison + multitrial aggregation scripts that validate protocol parity and generate JSON/CSV/Markdown reports (including hierarchical bootstrap intervals).
- Publish the replicated results + feasibility-boundary artifacts, and document the protocol/limitations in README and a dedicated study doc.
Reviewed changes
Copilot reviewed 35 out of 39 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_tensor_parallel_multitrial.py | Adds unit tests for multitrial aggregation (distinct seeds, counts, markdown content). |
| tests/test_tensor_parallel_comparison.py | Adds unit tests for single-trial TP1 vs TP2 comparison validation and ratios. |
| tests/test_modal_app.py | Extends coverage for new L4:2 remote selection and tensor_parallel_size validation. |
| scripts/build_tensor_parallel_multitrial.py | New multitrial aggregator with hierarchical bootstrap and markdown/CSV/JSON reporting. |
| scripts/build_tensor_parallel_comparison.py | New TP comparison builder with artifact validation, bootstrap intervals, and reporting. |
| modal_app.py | Adds tensor_parallel_size plumbing, new L4:2 remote, and artifact metadata updates. |
| docs/multi-gpu-tensor-parallel.md | Documents the TP1 vs TP2 protocol, acceptance checks, results, and limitations. |
| README.md | Updates top-level framing and summarizes tensor-parallel findings + links to artifacts/docs. |
| results/tensor-parallel-qwen7b-l4-r3-seed4509/tensor-parallel-comparison.md | Published per-trial comparison markdown (seed 4509). |
| results/tensor-parallel-qwen7b-l4-r3-seed4509/tensor-parallel-comparison.json | Published per-trial comparison JSON (seed 4509). |
| results/tensor-parallel-qwen7b-l4-r3-seed4509/tensor-parallel-comparison.csv | Published per-trial comparison CSV (seed 4509). |
| results/tensor-parallel-qwen7b-l4-r3-seed4409/tensor-parallel-comparison.md | Published per-trial comparison markdown (seed 4409). |
| results/tensor-parallel-qwen7b-l4-r3-seed4409/tensor-parallel-comparison.json | Published per-trial comparison JSON (seed 4409). |
| results/tensor-parallel-qwen7b-l4-r3-seed4409/tensor-parallel-comparison.csv | Published per-trial comparison CSV (seed 4409). |
| results/tensor-parallel-qwen7b-l4-multitrial-r6/tensor-parallel-multitrial.md | Published replicated aggregate markdown report. |
| results/tensor-parallel-qwen7b-l4-multitrial-r6/tensor-parallel-multitrial.json | Published replicated aggregate JSON payload. |
| results/tensor-parallel-qwen7b-l4-multitrial-r6/tensor-parallel-multitrial.csv | Published replicated aggregate CSV table. |
| results/modal-vllm-qwen7b-l4x2-tp2-feasibility-r1/paired-server-async.json | Published raw feasibility artifact for TP2 (includes logs and paired runs). |
| results/modal-vllm-qwen7b-l4x2-tp2-feasibility-r1/paired-server-async-summary.csv | Summary CSV for TP2 feasibility artifact. |
| results/modal-vllm-qwen7b-l4x2-tp2-feasibility-r1/paired-server-async-runs.csv | Per-run CSV for TP2 feasibility artifact. |
| results/modal-vllm-qwen7b-l4x2-tp2-batched8192-smoke-r1/paired-server-async-summary.csv | TP2 smoke summary CSV at scheduler budget 8192. |
| results/modal-vllm-qwen7b-l4x2-tp2-batched8192-smoke-r1/paired-server-async-runs.csv | TP2 smoke per-run CSV at scheduler budget 8192. |
| results/modal-vllm-qwen7b-l4x2-tp2-batched8192-r3-seed4509/paired-server-async-summary.csv | TP2 trial summary CSV (seed 4509). |
| results/modal-vllm-qwen7b-l4x2-tp2-batched8192-r3-seed4509/paired-server-async-runs.csv | TP2 trial per-run CSV (seed 4509). |
| results/modal-vllm-qwen7b-l4x2-tp2-batched8192-r3-seed4409/paired-server-async-summary.csv | TP2 trial summary CSV (seed 4409). |
| results/modal-vllm-qwen7b-l4x2-tp2-batched8192-r3-seed4409/paired-server-async-runs.csv | TP2 trial per-run CSV (seed 4409). |
| results/modal-vllm-qwen7b-l4-tp1-batched8192-smoke-r1/paired-server-async.json | Published raw TP1 smoke artifact (includes logs and paired runs). |
| results/modal-vllm-qwen7b-l4-tp1-batched8192-smoke-r1/paired-server-async-summary.csv | Summary CSV for TP1 smoke artifact. |
| results/modal-vllm-qwen7b-l4-tp1-batched8192-smoke-r1/paired-server-async-runs.csv | Per-run CSV for TP1 smoke artifact. |
| results/modal-vllm-qwen7b-l4-tp1-batched8192-r3-seed4509/paired-server-async-summary.csv | TP1 trial summary CSV (seed 4509). |
| results/modal-vllm-qwen7b-l4-tp1-batched8192-r3-seed4509/paired-server-async-runs.csv | TP1 trial per-run CSV (seed 4509). |
| results/modal-vllm-qwen7b-l4-tp1-batched8192-r3-seed4409/paired-server-async-summary.csv | TP1 trial summary CSV (seed 4409). |
| results/modal-vllm-qwen7b-l4-tp1-batched8192-r3-seed4409/paired-server-async-runs.csv | TP1 trial per-run CSV (seed 4409). |
| results/modal-vllm-qwen7b-l4-tp1-batched60640-feasibility-failure-r1/failure.md | Documents the failed high-scheduler-budget feasibility boundary for TP1. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+157
to
+166
| def _paired_run_map(payload: dict[str, Any]) -> dict[tuple[str, int], dict[str, Any]]: | ||
| rows: dict[tuple[str, int], dict[str, Any]] = {} | ||
| for row in payload["paired_runs"]: | ||
| if not isinstance(row, dict): | ||
| raise ValueError("paired run rows must be objects") | ||
| key = (str(row.get("scenario_id")), int(row.get("repeat_index"))) | ||
| if key in rows: | ||
| raise ValueError(f"duplicate paired run key: {key}") | ||
| rows[key] = row | ||
| return rows |
Comment on lines
+75
to
+78
| trial_maps = [ | ||
| (_paired_run_map(single), _paired_run_map(tensor)) | ||
| for single, tensor in artifacts | ||
| ] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Qwen/Qwen2.5-7B-InstructWhy
The inference benchmark previously emphasized prefix-cache behavior. This adds a controlled multi-GPU serving study with replicated measurements and uncertainty analysis, while keeping production-scale and cost-efficiency claims out of scope.
Result
Across two independent Modal run/seed trials and six paired repeats, TP2 reported 5.445x KV-cache capacity, 1.294x median server throughput, and 0.774x p95 server latency. The two-GPU topology used NCCL fallback without GPU P2P/custom all-reduce, and throughput per GPU decreased.
Validation
python3 -m unittest discover -s tests(75 tests)git diff --cached --check