test(kimi-k3): PD-disaggregation simulation bench (pd_sim) - #1216
Draft
dongjiyingdjy wants to merge 2 commits into
Draft
test(kimi-k3): PD-disaggregation simulation bench (pd_sim)#1216dongjiyingdjy wants to merge 2 commits into
dongjiyingdjy wants to merge 2 commits into
Conversation
Rank the agentic serve configs by what a PD-disaggregated deployment cares about, measured separately per role: - P-fresh: prefill-only cold first turns (compute-bound, hit <= 5% guard) - P-cached: primed 50K prefix + turn-2 increment, computed tok/s (bandwidth-bound, hit >= 95% guard) - D-sim: prime-as-transfer, settle, then rolling-admission decode ladder with cache-hit and memory-ledger validity guards All phases replay the frozen agentic dataset through a stdlib phased client (evalscope's plugin has no prime/measure phase control). pd_sim has its own collect script with VOID guards and a P:D provisioning-ratio helper. Validated on-machine (attn_tp8_moe_ep8, B300x8): both sweeps and the merged collect ran end to end; all D rungs and P-cached rungs passed their guards. Known issue recorded for follow-up: the frozen dataset contains 7 duplicate first-turn pairs, which VOIDs two P-fresh rungs (guards fire as designed); dedup selection is planned in pd_client. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: dongjiyingdjy <87510204+dongjiyingdjy@users.noreply.github.com>
Review fixes on the pd_sim bench:
- pd_client dedups conversations by first-turn content before applying
--offset/--number (the frozen artifact has 7 exact duplicate first-turn
pairs, 64 unique of 71; a duplicate reaching a later P-fresh rung scores
as a cache hit and breaks the <=5% guard). Both bench scripts assert the
unique count (>=64: ladder 62 + warmup 2) at dataset check time.
- Summaries record Requested next to Requests; collect VOIDs a mismatch
('short') — a twice-failed p-cached prime silently drops its conversation
from the measured wave with zero Failed Requests.
- Retries keep their full latency (a hiccup is real) but are now counted:
'Retried Requests' in the summary, informational 'Retried' column in
collect.
- Drop attn_dp8_moe_ep8 from the P sweep: DP is a decode-side scaling
choice, not a P-node candidate; it stays in d_bench.sh.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: dongjiyingdjy <87510204+dongjiyingdjy@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #1187 (merged). Adds
test/agentic_benchmark/kimi_k3/tokenspeed/pd_sim/— a PD-disaggregation simulation bench that ranks the serve configs by prefill-node and decode-node throughput separately, which the agentic bench cannot answer (its decode numbers mix prefill interference into every decode figure).What's in pd_sim
p_bench.sh— P-node sweep over the TP configs (DP is a decode-side scaling choice, not a P-node candidate): P-fresh (cold 50K first turns, compute-bound, cache-hit <=5% guard) then P-cached (primed prefix + turn-2 increment, computed tok/s, hit >=95% guard)d_bench.sh— D-node sweep over all three configs: prime-as-transfer at low concurrency, 30s settle, rolling-admission decode ladder with cache-hit guard and a memory ledger (VOIDs on >4GiB climb)pd_client.py— stdlib phased client (evalscope's plugin has no prime/measure phase control); dedups conversations by first-turn content (the frozen artifact has 7 exact duplicate first-turn pairs, 64 unique of 71 — a duplicate reaching a later P-fresh rung scores as a cache hit and breaks the guard)collect_outputs.py— tables + VOID guards (hit,Nfailed,shorton Requested/Requests mismatch,mem-climb) + informationalRetriedcolumn + P:D provisioning-ratio helperOn-machine validation (attn_tp8_moe_ep8, B300x8, 2026-08-23)
Both sweeps + merged collect ran end to end (pre-dedup revision; the dedup fix removes the two P-fresh VOIDs below):
The duplicate-first-turn finding also concerns the parent agentic bench and CI gates, which share the frozen artifact.
🤖 Generated with Claude Code