From 544142e88e9ca7e4a97df1f81c7b86b37d0a4de4 Mon Sep 17 00:00:00 2001 From: Whq02 <178854749+Whq02@users.noreply.github.com> Date: Sat, 7 Nov 2026 10:20:36 +0000 Subject: [PATCH 1/2] docs: /health's memory on Metal, and which families report prompt_cache_plan docs/api.md said /health's `memory` is `{}` on Metal. The Metal families with a memory reader (Nemotron, Qwen3.5-2B, Qwen3.8-27B and Qwen3.5-MoE through qwen27_served.zig, Flash Next, GLM-5.3-Flash and GLM-5.3 (EXLR8)) answer `active`, `cache` (0) and `peak` from the process footprint, as status_routes writes whatever the engine reads, and Flash Next and both GLMs add `limit`, their load limit; only the families without a reader answer `{}`. docs/memory-diagnostics.md listed Nemotron, Flash Next and Qwen3.8-27B as the families that report `prompt_cache_plan`: Qwen3.5-MoE (the 27B's host), Gemma 4 (the plan when its budget is positive), LFM2.5-24B-A2B and LFM3-1.7B (the plan at any budget) and the 2B with an explicit `--prompt-cache-gib` (the plan at any budget, `budget_bytes: 0` included) report it too. The row and the zero-budget sentences now say what the hosts do. --- docs/api.md | 2 +- docs/memory-diagnostics.md | 12 ++++++------ 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/api.md b/docs/api.md index 108280b480..d8d22a7559 100644 --- a/docs/api.md +++ b/docs/api.md @@ -963,7 +963,7 @@ configured the body is only `{"status": "ok"}`. An engine that reports unhealthy | `max_batch_size` | integer | The lanes the engine serves at once | | `policy` | string | The engine's policy line; absent when the engine reports none | | `warming` | boolean | Always `false` on the native engines | -| `memory` | object | CUDA: `active`, `cache` and `peak` in bytes (`cache` is 0), plus `limit` when the engine reports one. Metal: `{}` | +| `memory` | object | `active`, `cache` and `peak` in bytes (`cache` is 0), plus `limit` when the engine reports one: CUDA's allocator counts; on Metal the process footprint, from Nemotron, Qwen3.5-2B, Qwen3.8-27B, Qwen3.5-MoE, Flash Next, GLM-5.3-Flash and GLM-5.3 (EXLR8); Flash Next and both GLMs add `limit`, their load limit. `{}` from the other Metal families | | `live` | object | `connections`, `waiting`, `decode_tokens_per_second`, `prefill_tokens_per_second` | | `version` | string | The release this server was built from | | `backend` | string | The backend built in: `metal`, `cuda`, `hip` or `xpu`; absent from a build without one | diff --git a/docs/memory-diagnostics.md b/docs/memory-diagnostics.md index 03aa3d9763..f8c71742e4 100644 --- a/docs/memory-diagnostics.md +++ b/docs/memory-diagnostics.md @@ -17,7 +17,7 @@ zero means a measured or configured zero. The endpoint publishes no paths, UUIDs | `device.active_bytes`, `device.cache_bytes` | Existing backend allocator counters, where implemented. | | `device.peak_bytes` | Backend peak since startup or the last `/health?reset_peak=1`. | | `device.peak_scope` | `since_start_or_health_reset`; distinct from the process lifetime peak. | -| `prompt_cache_plan` | Immutable retained-prefix budget applied at startup, reported by Nemotron, Flash Next and Qwen3.8-27B on Macs. | +| `prompt_cache_plan` | Immutable retained-prefix budget applied at startup, reported by Nemotron, Flash Next, Qwen3.8-27B, Qwen3.5-MoE, Gemma 4, LFM2.5-24B-A2B and LFM3-1.7B on Macs, and by Qwen3.5-2B with `--prompt-cache-gib`. | The cache plan contains `source`, `budget_bytes`, and `explicit_budget`. With OS readings, `source` is `physical_footprint` and the plan also includes the sizing inputs: `ram_bytes`, @@ -30,11 +30,11 @@ rather than merely the presence of an override flag. Without both OS readings, the existing fallback uses an explicit budget (`source: explicit`) or the spare Metal working set up to 16 GiB (`source: metal_working_set`). Its OS inputs and unknown `over_cap` remain omitted. -Nemotron and Flash Next report a disabled retained-prefix cache as a known `budget_bytes: 0`; Qwen3.8-27B omits -`prompt_cache_plan` when its budget is 0. GLM-5.3-Flash and the CUDA server omit it too: GLM keeps -`--prompt-cache-gib` or 16 GiB inside its load limit, and the CUDA server keeps `--prompt-cache-gib` or 16 GiB inside -what its streams leave and names that size in its startup line. The 2B keeps prompt states only with -`--prompt-cache-gib`, and then reports the plan. +Nemotron, Flash Next, LFM2.5-24B-A2B and LFM3-1.7B report a disabled retained-prefix cache as a known `budget_bytes: 0`; +Qwen3.8-27B, Qwen3.5-MoE and Gemma 4 omit `prompt_cache_plan` when their budget is 0. GLM-5.3-Flash and the CUDA server +omit it too: GLM keeps `--prompt-cache-gib` or 16 GiB inside its load limit, and the CUDA server keeps `--prompt-cache-gib` +or 16 GiB inside what its streams leave and names that size in its startup line. The 2B keeps prompt states only with +`--prompt-cache-gib`, and then reports the plan, `budget_bytes: 0` included. This plan is a cache allowance, not a process limit, current cache occupancy, or an admission guarantee. `/metrics` keeps `tensorfold:process_footprint_bytes` and adds `tensorfold:process_footprint_peak_bytes` From 16d489cbf46a468edb32d37b3456ed453c46272d Mon Sep 17 00:00:00 2001 From: Whq02 <178854749+Whq02@users.noreply.github.com> Date: Sat, 7 Nov 2026 12:29:05 +0000 Subject: [PATCH 2/2] changelog: the two memory sentences under Unreleased (#2439) --- CHANGELOG.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4e2d623d7d..a8247e5fca 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -24,6 +24,8 @@ Each release's page on GitHub has its notes and measurements. See [the 1.0.0 rel streams and copy drafts; there is no drafter or prompt cache yet (#1492, part of #1488). Thanks to @jschmied. - HIP: capture a drafter's requested target-layer taps in each stream's ring, with graph-on/off and planted kernel controls (#2314). +- Docs: `/health`'s `memory` on Metal is the process footprint from the families with a reader, not `{}`, and `/memory`'s `prompt_cache_plan` row names every Mac family that reports it, with Qwen3.5-2B under `--prompt-cache-gib` (#2439). Thanks to @Whq02. + ## 1.2.4 - Docs: four subcommand-flag fixes checked against the code: `slide audit --threshold`'s range, `benchmark --context` and `--kv-cache`, `info` on Flash Next, and `run --device` and `--carveout` on CUDA builds (#2348).