Skip to content

docs: /health's memory on Metal, and which families report prompt_cache_plan #2439

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Whq02 wants to merge 2 commits into
base: main
Choose a base branch
from
+9 −7
Open

docs: /health's memory on Metal, and which families report prompt_cache_plan #2439

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ab95948
docs: /health's memory on Metal, and which families report prompt_cac…
Whq02 Nov 7, 2026
443e1a0
changelog: the two memory sentences under Unreleased (#2439)
Whq02 Nov 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@ Each release's page on GitHub has its notes and measurements. See [the 1.0.0 rel

- Internal: `zig/src/native/metal.zig` goes from 593 to 546 lines; six helpers the Metal open paths share (`chip`, `deviceName`, `prefillStep`, `modelContext`, `gibBytes` and the chip classes) live in `metal_shared.zig`, unchanged (#2425).

- Docs: `/health`'s `memory` on Metal is the process footprint from the families with a reader, not `{}`, and `/memory`'s `prompt_cache_plan` row names every Mac family that reports it, with Qwen3.5-2B under `--prompt-cache-gib` (#2439). Thanks to @Whq02.

## 1.2.4

- Docs: four subcommand-flag fixes checked against the code: `slide audit --threshold`'s range, `benchmark --context` and `--kv-cache`, `info` on Flash Next, and `run --device` and `--carveout` on CUDA builds (#2348).
Expand Down
2 changes: 1 addition & 1 deletion docs/api.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -963,7 +963,7 @@ configured the body is only `{"status": "ok"}`. An engine that reports unhealthy
| `max_batch_size` | integer | The lanes the engine serves at once |
| `policy` | string | The engine's policy line; absent when the engine reports none |
| `warming` | boolean | Always `false` on the native engines |
| `memory` | object | CUDA: `active`, `cache` and `peak` in bytes (`cache` is 0), plus `limit` when the engine reports one. Metal: `{}` |
| `memory` | object | `active`, `cache` and `peak` in bytes (`cache` is 0), plus `limit` when the engine reports one: CUDA's allocator counts; on Metal the process footprint, from Nemotron, Qwen3.5-2B, Qwen3.8-27B, Qwen3.5-MoE, Flash Next, GLM-5.3-Flash and GLM-5.3 (EXLR8); Flash Next and both GLMs add `limit`, their load limit. `{}` from the other Metal families |
| `live` | object | `connections`, `waiting`, `decode_tokens_per_second`, `prefill_tokens_per_second` |
| `version` | string | The release this server was built from |
| `backend` | string | The backend built in: `metal`, `cuda`, `hip` or `xpu`; absent from a build without one |
Expand Down
12 changes: 6 additions & 6 deletions docs/memory-diagnostics.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ zero means a measured or configured zero. The endpoint publishes no paths, UUIDs
| `device.active_bytes`, `device.cache_bytes` | Existing backend allocator counters, where implemented. |
| `device.peak_bytes` | Backend peak since startup or the last `/health?reset_peak=1`. |
| `device.peak_scope` | `since_start_or_health_reset`; distinct from the process lifetime peak. |
| `prompt_cache_plan` | Immutable retained-prefix budget applied at startup, reported by Nemotron, Flash Next and Qwen3.8-27B on Macs. |
| `prompt_cache_plan` | Immutable retained-prefix budget applied at startup, reported by Nemotron, Flash Next, Qwen3.8-27B, Qwen3.5-MoE, Gemma 4, LFM2.5-24B-A2B and LFM3-1.7B on Macs, and by Qwen3.5-2B with `--prompt-cache-gib`. |

The cache plan contains `source`, `budget_bytes`, and `explicit_budget`. With OS readings,
`source` is `physical_footprint` and the plan also includes the sizing inputs: `ram_bytes`,
Expand All @@ -30,11 +30,11 @@ rather than merely the presence of an override flag.

Without both OS readings, the existing fallback uses an explicit budget (`source: explicit`) or the spare
Metal working set up to 16 GiB (`source: metal_working_set`). Its OS inputs and unknown `over_cap` remain omitted.
Nemotron and Flash Next report a disabled retained-prefix cache as a known `budget_bytes: 0`; Qwen3.8-27B omits
`prompt_cache_plan` when its budget is 0. GLM-5.3-Flash and the CUDA server omit it too: GLM keeps
`--prompt-cache-gib` or 16 GiB inside its load limit, and the CUDA server keeps `--prompt-cache-gib` or 16 GiB inside
what its streams leave and names that size in its startup line. The 2B keeps prompt states only with
`--prompt-cache-gib`, and then reports the plan.
Nemotron, Flash Next, LFM2.5-24B-A2B and LFM3-1.7B report a disabled retained-prefix cache as a known `budget_bytes: 0`;
Qwen3.8-27B, Qwen3.5-MoE and Gemma 4 omit `prompt_cache_plan` when their budget is 0. GLM-5.3-Flash and the CUDA server
omit it too: GLM keeps `--prompt-cache-gib` or 16 GiB inside its load limit, and the CUDA server keeps `--prompt-cache-gib`
or 16 GiB inside what its streams leave and names that size in its startup line. The 2B keeps prompt states only with
`--prompt-cache-gib`, and then reports the plan, `budget_bytes: 0` included.
This plan is a cache allowance, not a process limit, current cache occupancy, or an admission guarantee.

`/metrics` keeps `tensorfold:process_footprint_bytes` and adds `tensorfold:process_footprint_peak_bytes`
Expand Down
Loading