Skip to content

docs: /health's memory on Metal, and which families report prompt_cache_plan - #2439

Open
Whq02 wants to merge 2 commits into
ashhart:mainfrom
Whq02:mercury/docs-memory-lag
Open

Whq02 wants to merge 2 commits into
ashhart:mainfrom
Whq02:mercury/docs-memory-lag

Conversation

@Whq02

@Whq02 Whq02 commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

What this changes

Two places in the docs say something the server does not do. Both were read beside the server's own JSON (1.2.2 on a
Mac with Qwen3.5-2B) and the code on main at 342b237.

  1. docs/api.md's health table says memory is {} on Metal. A Metal family with a memory reader answers the process
    footprint (the 2B, served with --prompt-cache-gib 0):

    $ curl -s http://127.0.0.1:8080/health
    {"status": "ok", "model": "qwen3.5-2b", ..., "memory": {"active": 2837990136, "cache": 0, "peak": 2837990136}, ..., "backend": "metal", ...}

status_routes.zig writes whatever engine.memory() reads and {} when it reads nothing. cache_fit.hostMemory is
the reader of Nemotron (metal.zig), the 2B (qwen35.zig), the 27B and Qwen3.5-MoE (qwen27_served.zig, the host
metal.zig opens both through), Flash Next (flashnext_batch_host.zig, flashnext_host.zig), GLM-5.3-Flash
(glm_host.zig) and GLM-5.3 (EXLR8) (glm_dsa_host.zig); Flash Next and the two GLMs pass their load limit, which
the route writes as limit. The families without a reader (Gemma 4, LFM2.5, LFM3, Llama, MiniMax, Spark, Step-5,
Kimi, Qwen4, and Drama 3, whose drama_host.zig reads memory on CUDA only) are the ones that answer {}. The row now
says that, keeping the limit clause the table already had.

  1. docs/memory-diagnostics.md lists Nemotron, Flash Next and Qwen3.8-27B as the families that report
    prompt_cache_plan. On a Mac the hosts that publish it are those three and Qwen3.5-MoE (qwen27_served.zig, the
    27B's host: the plan when the budget is positive, omitted at 0, as the page already said of the 27B), Gemma 4
    (gemma4.zig: the same rule), LFM2.5-24B-A2B (lfm2_moe.zig) and LFM3-1.7B (lfm3.zig), both at any budget, 0
    included, and the 2B with an explicit --prompt-cache-gib (qwen35.zig: the plan at any budget, the store only when
    it is positive). With --prompt-cache-gib 0:

    $ curl -s http://127.0.0.1:8080/memory
    {..., "prompt_cache_plan": {"source": "physical_footprint", "budget_bytes": 0, "explicit_budget": true, "over_cap": false, "ram_bytes": 51539607552, "ready_footprint_bytes": 3356707864, "cap_bytes": 36077725250, "room_bytes": 30332138026, "margin_bytes": 2147483648}}

The table row and the zero-budget sentences now name them; the sentence #2367 added about the 2B keeping prompt states
only with --prompt-cache-gib stays, with the budget_bytes: 0 case beside it.

Nothing else on either page changes.

Mercury CLI integrates TensorFold and reads /health and /memory to show what a served model holds; it was written
to the sentences and then read the JSON.

Checks on this head

  • Head commit: 443e1a0
  • Lean checker, python3 tools/zig/lean_check.py, on the base and on this head: python3 tools/pr_check.py --base origin/main --pr 2439 on this head: merge base 342b237, 3 changed files, lean 116 on the base and 116 on this head, 0 new; 0 errors, 0 warnings. The pages are outside the checker's trees (zig, tools/zig); no Zig file changes.
  • Tests run, with the command and pass, fail and skip counts: git diff --check origin/main..HEAD clean; zig build test-golden --summary all on the docs commit's tree (the two pages unchanged since): Build Summary: 5/5 steps succeeded, 322/335 and 55/55 equal to the server goldens, 0 unexpected differences (nothing in the tree reads these pages; the 13 are its recorded known differences).
  • CI: read on this head after the push; the first run on 281721b (the same docs change, before the CHANGELOG line and the identity fix) had benchmark-clients (ubuntu-latest) and zig-macos-dev-tools green when the head moved.
  • Not run, and why: zig build test; no source changes.

Receipt

n/a: documentation; the /health and /memory bodies this corrects are quoted above.

Checklist

  • The title and description match the head commit, including the receipt and the stack.
  • New tests fail before the change, pass after it, and skip cleanly without their dependency. (n/a: no test; the receipt is the server's JSON beside the sentence.)
  • Comments and docstrings are one line. No measurements or history in the source.
  • No personal data, machine names, internal hosts or local paths in the diff or its commits. No AI attribution lines.

Whq02 added a commit to Whq02/TensorFold that referenced this pull request Nov 7, 2026
@Whq02
Whq02 force-pushed the mercury/docs-memory-lag branch from 281721b to 9ba6a08 Compare November 7, 2026 12:16
Whq02 added 2 commits November 7, 2026 12:28
…he_plan

docs/api.md said /health's `memory` is `{}` on Metal. The Metal families with a memory reader (Nemotron, Qwen3.5-2B,
Qwen3.8-27B and Qwen3.5-MoE through qwen27_served.zig, Flash Next, GLM-5.3-Flash and GLM-5.3 (EXLR8)) answer `active`,
`cache` (0) and `peak` from the process footprint, as status_routes writes whatever the engine reads, and Flash Next and
both GLMs add `limit`, their load limit; only the families without a reader answer `{}`. docs/memory-diagnostics.md
listed Nemotron, Flash Next and Qwen3.8-27B as the families that report `prompt_cache_plan`: Qwen3.5-MoE (the 27B's
host), Gemma 4 (the plan when its budget is positive), LFM2.5-24B-A2B and LFM3-1.7B (the plan at any budget) and the 2B
with an explicit `--prompt-cache-gib` (the plan at any budget, `budget_bytes: 0` included) report it too. The row and
the zero-budget sentences now say what the hosts do.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant