GLM Metal 1.1.8: high idle memory use and repeated prefills, restart releases 44.5 GiB

Problem

With four long coding-agent sessions, TensorFold eventually reports memory pressure and repeatedly prefills most
of each prompt.

Two sessions had ended. With no running or queued requests, the process still used approximately 246 GiB.

A restart reduced memory use to 201.5 GiB with the same version and arguments.

Is this expected buffer retention, a missing release, or a leak?

Environment

M5 Ultra, 256 GiB unified memory, macOS 27.0.1.
TensorFold 1.1.8, Metal backend.
Model: TensorFold/GLM-5.3-Flash-MLX-oQ4-MTP.
No other substantial workloads.

Relevant settings:

GLM_LOAD_LIMIT_GB=240
GLM_MTP=1

--parallel 4 --context 1048576 --prompt-cache-gib 16
--learn-conversations --learn-gib 200 --spill-gib 200
--learn-daily-gib 1024 --learn-min-free-gib 20
--vision --kv-cache bf16 --prompt-precision bf16

Observations

Measurements from October 29, 2026:

                                      Before restart       After restart
Time (CET)                            ~00:02               00:31:37
Running / queued requests             0 / 0                0 / 0
Process physical footprint            246.02 GiB           201.52 GiB
Prompt cache                          5.36 GiB, 39 states   Empty
macOS memory pressure                 Warning              Normal
Disk swap                             Zero                 Zero

The new process had completed startup but had not processed any inference requests.

The restart released 44.51 GiB. The reported prompt cache accounts for only 5.36 GiB of that difference.

A separate 48-second idle sample showed approximately 246 GiB with normal memory pressure. High memory use did not
always trigger a warning.

Affected requests reused only 16,384 tokens from prompts of approximately 102,000 to 170,000 tokens. Other
requests still achieved deep cache reuse.

TensorFold logged:

warning: memory pressure: warning: one more stream only when none runs, no new kept prompt states, the prompt
cache sheds toward 8192 MiB

Relevant checks

Startup reported approximately 200.7 GiB of retained allocations.
Startup confirmed that all four streams share one MLA context pool, rather than four full context pools.
The daily SSD write ledger showed approximately 75 GiB of the 1024 GiB allowance.
The disk cache was near its 200 GiB limit. TensorFold normally makes room through eviction.
The current process logged no DailyBudgetSpent or OverCap errors.
The current process also logged 101 StagingFull errors. These concern a separate write-staging limit and do
not establish the memory cause.

The 240 GB load setting is not a hard process-memory limit.

Expected behavior / questions

I understand that the pressure guard intentionally prevents new prompt states. However, this also prevents those
new states from reaching the SSD cache.

What owns the additional memory after requests finish? Can TensorFold release or reuse it before sustained
pressure prevents new prompt states?

/atc claim

@wSzki

Thanks @wSzki for the before and after numbers. They show 44.51 GiB that the prompt cache (5.36 GiB) doesn't explain.

What the log line means: at warning pressure the server lets a new stream start only when none runs, keeps no new
prompt states, and sheds the prompt cache toward half its budget (8,192 MiB of your 16 GiB). That is why later turns
resumed at only 16,384 tokens: their newer states were never kept. So the repeated prefills follow from the pressure,
and the question is what holds the extra memory once requests end. We don't know that yet.

To narrow it down, could you send:

the footprint after each session ends, not only at the end, to see whether it grows by turn or by session;
the same run without --learn-conversations and --spill-gib, to tell their buffers apart from the engine's;
vmmap --summary <pid> before a restart, if you can.

On the StagingFull errors: on main, in 1.2.0, a save that fits the staging permit waits for a write in flight instead
of being skipped (#1548). That won't free memory, but more states should reach the disk.

Fresh data from a new process: the growth replicated, plus the evidence you asked for. Sections 1-3 are v1.1.8 (Metal, M5 Ultra 256 GiB, flags unchanged from the original report, process launched 2026-10-31 02:02Z; all timestamps UTC). The owner has since upgraded to 1.2.0; a follow-up on that version is at the end.

Growth, fresh process (1.1.8)

Time	Footprint	Notes
02:04 (2 min after load, warm-idle)	202.05 GiB	4 prompt states, 564 MiB
08:41 (6 h 39 m later)	230.63 GiB	41 states, 5.64 GiB; tuple captured
08:43	233.5 GiB	+3.0 GB observed within 90 s, 2 concurrent streams
09:00	237.3 GiB	still climbing

Average ≈ 4.6 GB/h under normal use (one ~130K-token session plus shorter ones).
Startup residency 200.7 GiB wired. The prompt-cache delta explains ≈ 5.1 GiB of the +28.6 GiB; ≈ 23 GiB remains unattributed, the same shape as the 44.51 GiB gap, on a new process.
Kernel pressure level 1 at every sampled time (02:04 baseline, 08:37+ probes, 15 s sampling from 08:42Z), and zero memory-pressure log lines in the process's whole segment: no observed governor engagement during the growth. (Not claiming continuous monitoring before 08:42.)
32 … would take its residency set past the 223.5 GiB load limit: it stays out refusals; the admission guard held throughout. Swap flat at 118 MiB; preemptions 0.

1) Footprint after each session ends

15 s series (footprint, connections, completed-request count, kernel pressure). Inter-request idle windows (done-count plateau ≥45 s, conn=0, dtps=0, same PID):

UTC	done	Footprint
08:48	4058	233.63 GiB
08:52	4069	234.58 GiB
08:53	4075	234.59 GiB

No material drop within any window (flat to slightly higher), +32.5 GiB above the 02:04 idle baseline. Caveat: these are ~45–60 s gaps, not captured session terminations, so per-turn vs per-session stays open with this data. (The 1.2.0 follow-up below adds hour-long windows.)

3) vmmap --summary before restart

Captured at the grown state (08:41:15Z, launch 02:02:26Z, footprint 230.7G, peak 231.4G); raw output inline below. Key numbers from it:

dirty 230.5G ≈ physical footprint 230.7G, with 229.5G of the dirty in IOAccelerator (graphics): 232.5G resident in that family across 326 regions.
The separate 164.7G plain-IOAccelerator family (90 regions) carries only 112K dirty.
Host heap is small: malloc zones ~13 MiB total, untagged ~820 MiB, swapped-out 0K.
One snapshot: it says where the footprint sits at that moment, not which allocations produced the growth; the summed RESIDENT (398.4G) may double-count aliases between the two IOAccelerator families.

vmmap --summary 25834: 2026-10-31 09:41:15 CET (launch 03:02:26, footprint 230.7G, peak 231.4G)

Process:         tensorfold-native [25834]
Path:            /opt/homebrew/*/tensorfold-native
Load Address:    0x10471c000
Identifier:      tensorfold-native
Version:         0
Code Type:       ARM64
Platform:        macOS
Parent Process:  launchd [1]
Target Type:     live task

Date/Time:       2026-10-31 09:41:15.375 +0100
Launch Time:     2026-10-31 03:02:26.608 +0100
OS Version:      macOS 27.0.1 (26A434)
Report Version:  7
Analysis Tool:   /usr/bin/vmmap

Physical footprint:         230.7G
Physical footprint (peak):  231.4G
Idle exit:                  untracked
----

ReadOnly portion of Libraries: Total=1.9G resident=107.8M(6%) swapped_out_or_unallocated=1.8G(94%)
Writable regions: Total=398.6G written=961.9M(0%) resident=398.2G(100%) swapped_out=0K(0%) unallocated=427.0M(0%)

                                VIRTUAL RESIDENT    DIRTY  SWAPPED VOLATILE   NONVOL    EMPTY   REGION 
REGION TYPE                        SIZE     SIZE     SIZE     SIZE     SIZE     SIZE     SIZE    COUNT (non-coalesced) 
===========                     ======= ========    =====  ======= ========   ======    =====  ======= 
Activity Tracing                   256K      32K      32K       0K       0K      32K       0K        1 
Deferred Reclaim                  64.0M      16K      16K       0K       0K       0K       0K        1 
Guard                            385.0G       0K       0K       0K       0K       0K       0K        1 
IOAccelerator                    164.7G   164.7G     112K       0K       0K   164.7G       0K       90 
IOAccelerator (graphics)         232.5G   232.5G   229.5G       0K       0K   232.5G       0K      326 
IOKit                               64K      64K      64K       0K       0K      16K       0K        4 
IOSurface                           16K       0K       0K       0K       0K       0K       0K        1 
Malloc Guard Page                 3600K       0K       0K       0K       0K       0K       0K        4         see MALLOC ZONE table below
Malloc Metadata                    400K     400K     400K       0K       0K       0K       0K        3         see MALLOC ZONE table below
Malloc Metadata metadata          1440K     768K     768K       0K       0K       0K       0K        2         see MALLOC ZONE table below
Malloc Nano metadata                32K      32K      32K       0K       0K       0K       0K        1         see MALLOC ZONE table below
Malloc Small                      56.0M    11.8M    11.8M       0K       0K       0K       0K       16         see MALLOC ZONE table below
Malloc Small (empty)              24.0M     160K     160K       0K       0K       0K       0K        6         see MALLOC ZONE table below
Malloc Tiny                       4096K     240K     240K       0K       0K       0K       0K        1         see MALLOC ZONE table below
OS Alloc Once                       32K      16K      16K       0K       0K       0K       0K        1 
Stack                            177.5M     912K     912K       0K       0K       0K       0K       17 
Stack (reserved)                   544K       0K       0K       0K       0K       0K       0K        1         reserved VM address space (unallocated)
Stack Guard                        288K       0K       0K       0K       0K       0K       0K       18 
Untagged                         931.3M   820.3M   820.3M       0K       0K       0K       0K     2354 
__AUTH                            1502K     454K      70K       0K       0K       0K       0K      170 
__AUTH_CONST                      18.8M    4694K       0K       0K       0K       0K       0K      329 
__CTF                               824      824       0K       0K       0K       0K       0K        1 
__DATA                            5447K    2075K     658K       0K       0K       0K       0K      341 
__DATA_CONST                      17.6M    6302K      40K       0K       0K       0K       0K      330 
__DATA_DIRTY                      1299K     718K     468K       0K       0K       0K       0K      287 
__FONT_DATA                        2352     2352       0K       0K       0K       0K       0K        1 
__LINKEDIT                         1.6G    17.7M       0K       0K       0K       0K       0K        5 
__OBJC_RO                        118.0M    54.6M       0K       0K       0K       0K       0K        1 
__OBJC_RW                         2735K     959K      47K       0K       0K       0K       0K        1 
__TEXT                           329.8M    90.1M       0K       0K       0K       0K       0K      395 
__TPRO_CONST                       132K      84K      48K       0K       0K       0K       0K        4 
mapped file                       79.3M    9576K      24K       0K       0K       0K       0K       11 
owned unmapped (network)           192K     192K       0K       0K       0K       0K       0K        2 
page table in kernel             127.2M   127.2M   127.2M       0K       0K       0K       0K        1 
shared memory                      720K     192K     192K       0K       0K       0K       0K       12 
unused but dirty shlib __DATA      169K     169K     169K       0K       0K       0K       0K       81 
===========                     ======= ========    =====  ======= ========   ======    =====  ======= 
TOTAL                            785.7G   398.4G   230.5G       0K       0K   397.2G       0K     4820 
TOTAL, minus reserved VM space   785.7G   398.4G   230.5G       0K       0K   397.2G       0K     4820 

                                 VIRTUAL   RESIDENT      DIRTY    SWAPPED ALLOCATION      BYTES DIRTY+SWAP          REGION
MALLOC ZONE                         SIZE       SIZE       SIZE       SIZE      COUNT  ALLOCATED  FRAG SIZE  % FRAG   COUNT
===========                      =======  =========  =========  =========  =========  =========  =========  ======  ======
DefaultMallocZone_0x105810000      85.4M      13.0M      13.0M         0K      14906       9.8M      3225K     25%      24
objc-class_rw_t_0x109f8c000          32K        32K        32K         0K        676        16K        16K     51%       2
===========                      =======  =========  =========  =========  =========  =========  =========  ======  ======
TOTAL                              85.4M      13.0M      13.0M         0K      15582       9.9M      3242K     25%      26

2) Run without --learn-conversations / --spill-gib

Not run yet.

Follow-up: same shape on 1.2.0

The owner restarted the process for the 1.2.0 upgrade at 12:44Z (not forced for this report; the 1.1.8 process above was still live and being sampled until then). Same flags and model.

Restart recovery: 1.1.8 final sample 244.51 GiB (12:44:30Z) → 1.2.0 warm idle 202.22 GiB at the first hour-long idle window (13:00Z, 16 min after launch). ≈ 42.3 GiB recovered, same order as the 44.51 GiB in the original report.
Growth replicated on the new build: 202.22 GiB (13:00Z) → 239.41 GiB (01:00Z, last valid sample), +37.2 GiB over 12.0 h and 730 completed requests. The owner then stopped the process at 01:07Z (SIGTERM, clean drain, 0 requests in flight) and switched the box back to oMLX; a same-build restart floor will be added here if the TensorFold lane runs again.
Kernel pressure level 1 at all 2,896 15 s samples through 00:57Z, and zero memory-pressure log lines in the process's whole segment (20,459 lines, launch to exit).
Idle windows up to an hour. 23 qualifying windows (done-count plateau ≥ 60 s, dtps=0 throughout, conn=0 in ≥ 98% of samples, same PID, startup ramp excluded); the three longest, each 59 min:

UTC window	Length	conn=0 samples	Footprint start → end
14:00–15:00	59 min	237/237	202.50 → 202.30 GiB
15:00–16:00	59 min	236/236	202.91 → 202.71 GiB
20:00–21:00	59 min	237/237	216.91 → 216.71 GiB

Largest footprint decline inside any of the 23 windows: 0.20 GiB. All net increase appeared across the active periods between windows (202 → 208 → 217 → 239.4 GiB), and the footprint never returned toward the 202 GiB floor in any sampled window. So no material drop in any of these sampled idle windows, three of them 59 minutes long. What stays open with this data is per-turn vs per-session accumulation, since the steps line up with sessions.

StagingFull: 164 skips still logged in the 1.2.0 segment, e.g. warning: prompt cache: imprint: saving 82861 tokens skipped (StagingFull). Not claiming whether these fit the staging permit; reporting that skips still occur on 1.2.0.

Environment: 1.1.8 (sections 1-3) and 1.2.0 (follow-up), Metal, M5 Ultra 256 GiB, --parallel 4 --context 1048576 --prompt-cache-gib 16 --learn-conversations --learn-gib 200 --spill-gib 200 --learn-daily-gib 1024 --max-tokens 65536 --kv-cache bf16 --prompt-precision bf16 --vision --vision-max-images 16, GLM_LOAD_LIMIT_GB=240.

@wSzki

Thanks @wSzki, the full vmmap and timings helped. We found one definite accounting bug that fits your numbers: with --learn-conversations, a GLM prompt state restored from disk was charged as a small resident save (its KDA states) while the restore allocates its whole attention image, so the prompt cache could hold about 2.5 GB more per restored state at 130,000 tokens than its budget allowed. That matches growth during active periods that stays through idle, as you saw. The fix is #1818. We'll run GLM on our M3 Ultra with your flags before it lands, and say here whether it accounts for all of the growth or only part.

Navigation Menu

Repository navigation

GLM Metal 1.1.8: high idle memory use and repeated prefills, restart releases 44.5 GiB #1572

Problem

Environment

Observations

Relevant checks

Expected behavior / questions

Growth, fresh process (1.1.8)

1) Footprint after each session ends

3) vmmap --summary before restart

2) Run without --learn-conversations / --spill-gib

Follow-up: same shape on 1.2.0

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

GLM Metal 1.1.8: high idle memory use and repeated prefills, restart releases 44.5 GiB #1572

Description

Problem

Environment

Observations

Relevant checks

Expected behavior / questions

Activity

ashhart commented on Oct 29, 2026

ashhart commented on Oct 31, 2026

wSzki commented on Nov 1, 2026

Growth, fresh process (1.1.8)

1) Footprint after each session ends

3) vmmap --summary before restart

2) Run without --learn-conversations / --spill-gib

Follow-up: same shape on 1.2.0

ashhart commented on Nov 1, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions