Skip to content

[staging] Manual import: generate per-turn rows from multi-turn benchmarks #106

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
aas008 wants to merge 6 commits into
base: staging
Choose a base branch
from
Open

[staging] Manual import: generate per-turn rows from multi-turn benchmarks #106

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
6c2b637
feat(manual-import): generate per-turn rows from request-level turn_i…
aas008 Oct 7, 2026
8af00eb
fix: add turn_index to fieldnames so it is written to CSV output
aas008 Oct 6, 2026
57d7e14
refactor: align per-turn stats with Forge PR #215 implementation
aas008 Oct 7, 2026
a424aa0
feat(manual-import): add optional --mlflow-run-id and --mlflow-experi…
aas008 Oct 7, 2026
1be4906
fix: address review findings in per-turn manual import
aas008 Oct 15, 2026
4549b5c
fix: resolve remaining pre-commit hook failures
aas008 Oct 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions llmd_dashboard.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1640,6 +1640,13 @@ def load_rhaiis_data(file_path: str = "consolidated_dashboard.csv") -> pd.DataFr
for col in df.select_dtypes(include=["object"]).columns:
df[col] = df[col].str.strip() # type: ignore[assignment]

# Keep only aggregate rows: per-turn breakdown rows (turn_index 0, 1, 2…)
# from multi-turn benchmarks would double-count runs in comparisons.
# Mirrors the aggregate-row filter in dashboard.py.
if "turn_index" in df.columns:
_ti = df["turn_index"].astype(str).str.strip() # type: ignore[misc]
df = df[df["turn_index"].isna() | (_ti == "") | (_ti == "nan")].copy()

# Assign profile based on prompt/output tokens (same as llmd_data)
if "prompt toks" in df.columns and "output toks" in df.columns:
df["profile"] = df.apply(assign_profile, axis=1)
Expand Down
148 changes: 76 additions & 72 deletions manual_runs/scripts/vllm/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,20 +32,22 @@ python import_manual_runs_json_v2.py <json_file> \

## Arguments

| Argument | Required | Description | Example |
| -------------------- | -------- | ----------------------------------------------- | --------------------------------------- |
| `json_file` | Yes | Path to guidellm JSON output file | `llama-70b-vllm.json` |
| `--model` | Yes | Model name (HuggingFace format) | `meta-llama/Llama-3.3-70B-Instruct` |
| `--version` | Yes | Framework version, optionally suffixed with a run label | `vLLM-0.13.0`, `vLLM-0.28.0-pcon-mnbt` |
| `--tp` | Yes | Tensor parallelism size | `4`, `8` |
| `--accelerator` | Yes | GPU/accelerator type | `H200`, `MI300X`, `TPU` |
| `--runtime-args` | Yes | Server runtime configuration | See examples below |
| `--image-tag` | Yes | Container image tag | `vllm/vllm-openai:v0.13.0` |
| `--guidellm-version` | Yes | guidellm version used | `v0.5.1`, `v0.3.0` |
| `--csv-file` | No | Output CSV path (default: `new_benchmarks.csv`) | `llama-70b-vllm.csv` |
| `--dataset` | No | Real dataset name (for real-dataset runs) | `mlperf-gpt-oss`, `sharegpt` |
| `--spec-decoding` | No | Speculative decoding method used | `eagle3`, `ngram` |
| `--prefix-caching` | No | Whether prefix caching was enabled | `yes`, `no` |
| Argument | Required | Description | Example |
| ------------------------ | -------- | ------------------------------------------------------------- | -------------------------------------- |
| `json_file` | Yes | Path to guidellm JSON output file | `llama-70b-vllm.json` |
| `--model` | Yes | Model name (HuggingFace format) | `meta-llama/Llama-3.3-70B-Instruct` |
| `--version` | Yes | Framework version, optionally suffixed with a run label | `vLLM-0.13.0`, `vLLM-0.28.0-pcon-mnbt` |
| `--tp` | Yes | Tensor parallelism size | `4`, `8` |
| `--accelerator` | Yes | GPU/accelerator type | `H200`, `MI300X`, `TPU` |
| `--runtime-args` | Yes | Server runtime configuration | See examples below |
| `--image-tag` | Yes | Container image tag | `vllm/vllm-openai:v0.13.0` |
| `--guidellm-version` | Yes | guidellm version used | `v0.5.1`, `v0.3.0` |
| `--csv-file` | No | Output CSV path (default: `new_benchmarks.csv`) | `llama-70b-vllm.csv` |
| `--dataset` | No | Real dataset name (for real-dataset runs) | `mlperf-gpt-oss`, `sharegpt` |
| `--spec-decoding` | No | Speculative decoding method used | `eagle3`, `ngram` |
| `--prefix-caching` | No | Whether prefix caching was enabled | `yes`, `no` |
| `--mlflow-run-id` | No | MLflow run UUID — enables clickable MLflow links in dashboard | `c6aa48a0d312448380621e1bf00a8a5d` |
| `--mlflow-experiment-id` | No | MLflow experiment ID (used with `--mlflow-run-id`) | `264` |

For labeled runs, include the label suffix in `--version` (for example,
`--version vLLM-0.28.0-pcon-mnbt`). The dashboard derives the release and label
Expand Down Expand Up @@ -116,7 +118,7 @@ python import_manual_runs_json_v2.py \
--csv-file "gemma-multiturn.csv"
```

> **Note:** `turns`, `prefix_tokens`, `prefix_count`, and `request_type` are auto-detected from the guidellm JSON. `turn_index` is always empty in manual-import output (aggregate rows only) — per-turn rows are generated by the Forge pipeline when `turn_index` data is available in the benchmark JSON.
> **Note:** `turns`, `prefix_tokens`, `prefix_count`, and `request_type` are auto-detected from the guidellm JSON. For multi-turn benchmarks, the script emits one aggregate row per concurrency level (`turn_index` = empty) **plus** one per-turn row per turn per concurrency level (`turn_index` = 0, 1, 2, …) when `requests.successful`/`requests.errored` contain at least two distinct valid `turn_index` values. Other benchmarks emit aggregate rows only.

## Appending to Consolidated Dashboard

Expand All @@ -139,63 +141,65 @@ values remain aligned with the dashboard column order.

## Output CSV Columns

The script outputs 53 columns compatible with the performance dashboard:

| # | Column | Description |
| --- | ------------------------- | ---------------------------------------------------- |
| 1 | `run` | Unique run identifier (`{accelerator}-{model}-{tp}`) |
| 2 | `accelerator` | GPU type (H200, MI300X, etc.) |
| 3 | `model` | Model name |
| 4 | `version` | Framework version, optionally suffixed with a run label |
| 5 | `prompt toks` | Configured prompt token count |
| 6 | `output toks` | Configured output token count |
| 7 | `TP` | Tensor parallelism size |
| 8 | `measured concurrency` | Actual measured concurrency |
| 9 | `intended concurrency` | Requested concurrency (streams) |
| 10 | `measured rps` | Requests per second |
| 11 | `output_tok/sec` | Output tokens per second |
| 12 | `total_tok/sec` | Total tokens per second |
| 13 | `prompt_token_count_mean` | Mean prompt token count |
| 14 | `prompt_token_count_p99` | P99 prompt token count |
| 15 | `output_token_count_mean` | Mean output token count |
| 16 | `output_token_count_p99` | P99 output token count |
| 17 | `ttft_median` | Time to first token - median (ms) |
| 18 | `ttft_p95` | Time to first token - P95 (ms) |
| 19 | `ttft_p1` | Time to first token - P1 (ms) |
| 20 | `ttft_p999` | Time to first token - P99.9 (ms) |
| 21 | `tpot_median` | Time per output token - median (ms) |
| 22 | `tpot_p95` | Time per output token - P95 (ms) |
| 23 | `tpot_p99` | Time per output token - P99 (ms) |
| 24 | `tpot_p999` | Time per output token - P99.9 (ms) |
| 25 | `tpot_p1` | Time per output token - P1 (ms) |
| 26 | `itl_median` | Inter-token latency - median (ms) |
| 27 | `itl_p95` | Inter-token latency - P95 (ms) |
| 28 | `itl_p999` | Inter-token latency - P99.9 (ms) |
| 29 | `itl_p1` | Inter-token latency - P1 (ms) |
| 30 | `request_latency_median` | End-to-end request latency - median (s) |
| 31 | `request_latency_min` | End-to-end request latency - minimum (s) |
| 32 | `request_latency_max` | End-to-end request latency - maximum (s) |
| 33 | `successful_requests` | Number of successful requests |
| 34 | `errored_requests` | Number of errored requests |
| 35 | `uuid` | Unique benchmark run ID |
| 36 | `ttft_mean` | Time to first token - mean (ms) |
| 37 | `ttft_p99` | Time to first token - P99 (ms) |
| 38 | `itl_mean` | Inter-token latency - mean (ms) |
| 39 | `itl_p99` | Inter-token latency - P99 (ms) |
| 40 | `runtime_args` | Server configuration arguments |
| 41 | `guidellm_start_time_ms` | Benchmark start time (epoch ms) |
| 42 | `guidellm_end_time_ms` | Benchmark end time (epoch ms) |
| 43 | `image_tag` | Container image used |
| 44 | `guidellm_version` | guidellm version used |
| 45 | `DP` | Data parallelism size (empty for TP runs) |
| 46 | `dataset` | Real dataset name (empty for synthetic runs) |
| 47 | `spec_decoding` | Speculative decoding method (empty if none) |
| 48 | `prefix_caching` | Prefix caching status (`yes`, `no`, or empty) |
| 49 | `turn_index` | Per-turn breakdown index (empty for aggregate rows) |
| 50 | `turns` | Conversation turns for multiturn benchmarks |
| 51 | `prefix_tokens` | Prefix token count (auto-detected from JSON) |
| 52 | `prefix_count` | Prefix count (auto-detected from JSON) |
| 53 | `request_type` | GuideLLM API endpoint type (auto-detected from JSON) |
The script outputs 55 columns compatible with the performance dashboard:

| # | Column | Description |
| --- | ------------------------- | --------------------------------------------------------------------------- |
| 1 | `run` | Unique run identifier (`{accelerator}-{model}-{tp}`) |
| 2 | `accelerator` | GPU type (H200, MI300X, etc.) |
| 3 | `model` | Model name |
| 4 | `version` | Framework version, optionally suffixed with a run label |
| 5 | `prompt toks` | Configured prompt token count |
| 6 | `output toks` | Configured output token count |
| 7 | `TP` | Tensor parallelism size |
| 8 | `measured concurrency` | Actual measured concurrency |
| 9 | `intended concurrency` | Requested concurrency (streams) |
| 10 | `measured rps` | Requests per second |
| 11 | `output_tok/sec` | Output tokens per second |
| 12 | `total_tok/sec` | Total tokens per second |
| 13 | `prompt_token_count_mean` | Mean prompt token count |
| 14 | `prompt_token_count_p99` | P99 prompt token count |
| 15 | `output_token_count_mean` | Mean output token count |
| 16 | `output_token_count_p99` | P99 output token count |
| 17 | `ttft_median` | Time to first token - median (ms) |
| 18 | `ttft_p95` | Time to first token - P95 (ms) |
| 19 | `ttft_p1` | Time to first token - P1 (ms) |
| 20 | `ttft_p999` | Time to first token - P99.9 (ms) |
| 21 | `tpot_median` | Time per output token - median (ms) |
| 22 | `tpot_p95` | Time per output token - P95 (ms) |
| 23 | `tpot_p99` | Time per output token - P99 (ms) |
| 24 | `tpot_p999` | Time per output token - P99.9 (ms) |
| 25 | `tpot_p1` | Time per output token - P1 (ms) |
| 26 | `itl_median` | Inter-token latency - median (ms) |
| 27 | `itl_p95` | Inter-token latency - P95 (ms) |
| 28 | `itl_p999` | Inter-token latency - P99.9 (ms) |
| 29 | `itl_p1` | Inter-token latency - P1 (ms) |
| 30 | `request_latency_median` | End-to-end request latency - median (s) |
| 31 | `request_latency_min` | End-to-end request latency - minimum (s) |
| 32 | `request_latency_max` | End-to-end request latency - maximum (s) |
| 33 | `successful_requests` | Number of successful requests |
| 34 | `errored_requests` | Number of errored requests |
| 35 | `uuid` | Unique benchmark run ID |
| 36 | `ttft_mean` | Time to first token - mean (ms) |
| 37 | `ttft_p99` | Time to first token - P99 (ms) |
| 38 | `itl_mean` | Inter-token latency - mean (ms) |
| 39 | `itl_p99` | Inter-token latency - P99 (ms) |
| 40 | `runtime_args` | Server configuration arguments |
| 41 | `guidellm_start_time_ms` | Benchmark start time (epoch ms) |
| 42 | `guidellm_end_time_ms` | Benchmark end time (epoch ms) |
| 43 | `image_tag` | Container image used |
| 44 | `guidellm_version` | guidellm version used |
| 45 | `DP` | Data parallelism size (empty for TP runs) |
| 46 | `dataset` | Real dataset name (empty for synthetic runs) |
| 47 | `spec_decoding` | Speculative decoding method (empty if none) |
| 48 | `prefix_caching` | Prefix caching status (`yes`, `no`, or empty) |
| 49 | `turn_index` | Per-turn breakdown index (empty for aggregate rows) |
| 50 | `turns` | Conversation turns for multiturn benchmarks |
| 51 | `prefix_tokens` | Prefix token count (auto-detected from JSON) |
| 52 | `prefix_count` | Prefix count (auto-detected from JSON) |
| 53 | `request_type` | GuideLLM API endpoint type (auto-detected from JSON) |
| 54 | `mlflow_run_id` | MLflow run UUID (from `--mlflow-run-id`, empty if not provided) |
| 55 | `mlflow_experiment_id` | MLflow experiment ID (from `--mlflow-experiment-id`, empty if not provided) |

## Notes

Expand Down
Loading