diff --git a/CHANGELOG.md b/CHANGELOG.md index 4e2d623d7d..a8247e5fca 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -24,6 +24,8 @@ Each release's page on GitHub has its notes and measurements. See [the 1.0.0 rel streams and copy drafts; there is no drafter or prompt cache yet (#1492, part of #1488). Thanks to @jschmied. - HIP: capture a drafter's requested target-layer taps in each stream's ring, with graph-on/off and planted kernel controls (#2314). +- Docs: `/health`'s `memory` on Metal is the process footprint from the families with a reader, not `{}`, and `/memory`'s `prompt_cache_plan` row names every Mac family that reports it, with Qwen3.5-2B under `--prompt-cache-gib` (#2439). Thanks to @Whq02. + ## 1.2.4 - Docs: four subcommand-flag fixes checked against the code: `slide audit --threshold`'s range, `benchmark --context` and `--kv-cache`, `info` on Flash Next, and `run --device` and `--carveout` on CUDA builds (#2348). diff --git a/docs/api.md b/docs/api.md index 108280b480..d8d22a7559 100644 --- a/docs/api.md +++ b/docs/api.md @@ -963,7 +963,7 @@ configured the body is only `{"status": "ok"}`. An engine that reports unhealthy | `max_batch_size` | integer | The lanes the engine serves at once | | `policy` | string | The engine's policy line; absent when the engine reports none | | `warming` | boolean | Always `false` on the native engines | -| `memory` | object | CUDA: `active`, `cache` and `peak` in bytes (`cache` is 0), plus `limit` when the engine reports one. Metal: `{}` | +| `memory` | object | `active`, `cache` and `peak` in bytes (`cache` is 0), plus `limit` when the engine reports one: CUDA's allocator counts; on Metal the process footprint, from Nemotron, Qwen3.5-2B, Qwen3.8-27B, Qwen3.5-MoE, Flash Next, GLM-5.3-Flash and GLM-5.3 (EXLR8); Flash Next and both GLMs add `limit`, their load limit. `{}` from the other Metal families | | `live` | object | `connections`, `waiting`, `decode_tokens_per_second`, `prefill_tokens_per_second` | | `version` | string | The release this server was built from | | `backend` | string | The backend built in: `metal`, `cuda`, `hip` or `xpu`; absent from a build without one | diff --git a/docs/memory-diagnostics.md b/docs/memory-diagnostics.md index 03aa3d9763..f8c71742e4 100644 --- a/docs/memory-diagnostics.md +++ b/docs/memory-diagnostics.md @@ -17,7 +17,7 @@ zero means a measured or configured zero. The endpoint publishes no paths, UUIDs | `device.active_bytes`, `device.cache_bytes` | Existing backend allocator counters, where implemented. | | `device.peak_bytes` | Backend peak since startup or the last `/health?reset_peak=1`. | | `device.peak_scope` | `since_start_or_health_reset`; distinct from the process lifetime peak. | -| `prompt_cache_plan` | Immutable retained-prefix budget applied at startup, reported by Nemotron, Flash Next and Qwen3.8-27B on Macs. | +| `prompt_cache_plan` | Immutable retained-prefix budget applied at startup, reported by Nemotron, Flash Next, Qwen3.8-27B, Qwen3.5-MoE, Gemma 4, LFM2.5-24B-A2B and LFM3-1.7B on Macs, and by Qwen3.5-2B with `--prompt-cache-gib`. | The cache plan contains `source`, `budget_bytes`, and `explicit_budget`. With OS readings, `source` is `physical_footprint` and the plan also includes the sizing inputs: `ram_bytes`, @@ -30,11 +30,11 @@ rather than merely the presence of an override flag. Without both OS readings, the existing fallback uses an explicit budget (`source: explicit`) or the spare Metal working set up to 16 GiB (`source: metal_working_set`). Its OS inputs and unknown `over_cap` remain omitted. -Nemotron and Flash Next report a disabled retained-prefix cache as a known `budget_bytes: 0`; Qwen3.8-27B omits -`prompt_cache_plan` when its budget is 0. GLM-5.3-Flash and the CUDA server omit it too: GLM keeps -`--prompt-cache-gib` or 16 GiB inside its load limit, and the CUDA server keeps `--prompt-cache-gib` or 16 GiB inside -what its streams leave and names that size in its startup line. The 2B keeps prompt states only with -`--prompt-cache-gib`, and then reports the plan. +Nemotron, Flash Next, LFM2.5-24B-A2B and LFM3-1.7B report a disabled retained-prefix cache as a known `budget_bytes: 0`; +Qwen3.8-27B, Qwen3.5-MoE and Gemma 4 omit `prompt_cache_plan` when their budget is 0. GLM-5.3-Flash and the CUDA server +omit it too: GLM keeps `--prompt-cache-gib` or 16 GiB inside its load limit, and the CUDA server keeps `--prompt-cache-gib` +or 16 GiB inside what its streams leave and names that size in its startup line. The 2B keeps prompt states only with +`--prompt-cache-gib`, and then reports the plan, `budget_bytes: 0` included. This plan is a cache allowance, not a process limit, current cache occupancy, or an admission guarantee. `/metrics` keeps `tensorfold:process_footprint_bytes` and adds `tensorfold:process_footprint_peak_bytes`