Skip to content

README.md

Latest commit

 

History

History
58 lines (47 loc) · 2.53 KB

README.md

File metadata and controls

58 lines (47 loc) · 2.53 KB
Raw

Agentic single-node benchmarks

MVP / experimental. Nothing in this directory is an official InferenceX benchmark. Results are not published on https://inferencex.com and are not intended to be cited.

These launchers exist to develop and validate the agentic-coding scenario type before it is promoted to first-class status. The scripts themselves are best-effort and mainly serve as a reference implementation of how the plumbing (env vars, scenario routing, result paths) should work. Specific models and configs may be broken at any given time. Multi-node support in particular is not yet first-class.

DRAM KV offload memory policy

Agentic scenarios use kv-offloading for the resource tier and kv-offload-backend for the backend implementation. kv-offloading is currently either none or dram. When it is dram, the backend must be set:

- dram-utilization: 0.80
  search-space:
  - { tp: 4, kv-offloading: dram, kv-offload-backend: { name: vllm-native }, conc-list: [16, 32] }
  - { tp: 8, kv-offloading: none, conc-list: [16, 32] }

kv-offload-backend.name is required when offloading is enabled. Its version is optional: omit it for framework-native implementations without an independent release, and include it for separately versioned packages such as LMCache or Mooncake.

Agentic matrix generation uses a 3600-second default duration. Reusable workflow callers can still override the duration input explicitly.

Agentic master configs must use an exact cluster:<name> runner label so every search-space point lands on the same hardware fleet. Machine-level host memory is declared once in the configs/runners.yaml hardware entry matching that runner label:

hardware:
  cluster:b300-nv:
    available-cpu-dram-mib: 2964436
    gpus-per-node: 8

The matrix generator combines the master config utilization with runner hardware metadata and emits the aggregate budget as floor(min(available MiB, 2,861,022) * 1,048,576 * utilization * tp / gpus-per-node / 1,000,000,000). The 2,861,022 MiB cap is the 3 TB decimal DRAM limit. For example, TP4 in an eight-GPU B300 search at 80% utilization receives 1,199 GB while TP8 receives 2,399 GB.

Benchmark scripts must consume TOTAL_CPU_DRAM_GB rather than replace it with model-specific constants. Backends with per-rank or per-pool settings must divide this aggregate budget accordingly. DSv4 SGLang is the exception because it exposes only --hicache-ratio. Its empirically measured ratios are capped in the model launchers to remain below the generated byte budget.