mem: HG_MALLOC - a hugepage slab allocator with elastic, auto-scaling arenas

[Lt-Flash]

Summary

HG_MALLOC is a core memory allocator: size-class slabs carved from a huge-page-backed reservation the allocator makes itself, with per-process lock-free free caches — and, new in this submission, elastic arenas that grow and shrink at runtime between a starting size and a reserved cap.

This supersedes #4126. It combines all features of the v1 and v2 allocator generations and extends them, making it a production-ready allocator:

• v1 — the fast path (mem: add HG_MALLOC, a hugepage-backed slab allocator, with CPU pinning #4126): 21 size classes, bump-carved chunks, per-process LIFO free stacks (lock-free fast path, no shared cache lines); a boundary-tag large tier for >64K; a 4-tier hugepage ladder (MAP_HUGETLB → MADV_HUGEPAGE → MADV_COLLAPSE → plain 4K), every tier verified via /proc, never inferred; and the CPU-affinity subsystem (pin_workers, per-process-type CPU lists, per-UDP-listener pin_cpus) that keeps the per-process caches warm.
• v2 — correctness and reclamation: an in-page buddy grid; class-dedicated blocks with in-block headers; ownership checks on every header read — a foreign pointer is refused and counted, never dereferenced; event-driven GC — a drained block is un-typed, returns to the buddy and is re-cut for any class, which retires v1's "size 2× your F_MALLOC needs" rule; per-process cache-line-isolated stats (no racy RMW on the fast path); corruption counters (class_mismatch, double_free, nfree_underflow, bad_class, foreign_ptr).
• v3 — the elastic arena: -m INIT:CAP / -M INIT:CAP reserve the cap as address space pre-fork (free) and commit physical pages only as demand arrives.

Why elastic

-m and -M are fixed at boot, and both are a bet. Bet low and shared memory exhausts in the best traffic hour — allocations fail and the fix is a restart, during the incident. Bet high and pinned memory idles: a production load balancer we measured peaks at 11.9 MB of shm over 15 h of full traffic inside a 128 MB arena, and -M multiplies by the worker count — with a fixed -M every worker is sized for the busiest worker's worst minute.

v3 replaces both bets with a range. Growth commits whole 16 MB granules by mlock() (populates exactly, clean errno, no fork-window SIGBUS), bounded by three independent limits enforced where each is real: the profile's scale-up target, the backing tier (a hugetlb pool reserves the mapping at map time), and the host's actual free RAM — pkg deltas charged × the worker count. Shrink releases whole-free top granules back to the kernel — MADV_REMOVE on the shm object returns hugetlb pages to the pool for all mappers; private pkg uses MADV_DONTNEED — and never relocates live data. Policy is the existing auto_scaling_profile grammar:

auto_scaling_profile = MEM_SHM
    scale up to 1024 on 80% for 3 cycles within 10
    scale down to 256 on 30% for 120 cycles

auto_scaling_profile = MEM_PKG
    scale up to 64 on 80% for 2 cycles within 5
    scale down to 8 on 20% for 60 cycles

shm_auto_scaling_profile = MEM_SHM
pkg_auto_scaling_profile = MEM_PKG

The pkg profile governs every worker's private arena individually, so its numbers are per-worker scale — an order of magnitude below the shared arena's.

A resource refusal (not an admin ceiling doing its job) arms a latched GROW-BLOCKED state: one WARN, a gauge statistic, and an E_CORE_SHM_GROW_BLOCKED event re-raised while held — one alertable signal instead of a log flood. A dry-run mode (hg_autoscale_dry_run = 1) narrates every decision without acting. With no cap configured, nothing changes: the arena is fixed at -m/-M, byte-for-byte the fixed-size behaviour. The allocator remains runtime-selected via -a HG_MALLOC; the default allocator is untouched, and without -DHG_MALLOC the tree builds exactly as before.

Measured

• Five-allocator A/B (one binary, topology-hiding + cache workload, 12k CPS): HG 5.29 cores vs F_MALLOC 6.42 (−18%); F_PARALLEL −8%, HP −6%, Q +13%.
• Real dispatcher load balancer, 20,000 concurrently-held calls: 20–45% less CPU than F_MALLOC, gap widening with load, memory flat while F_MALLOC grew 751→1126 MB into the run.
• Production billing gateways: HG peaks 14.8 MB shm against Q_MALLOC's 16.1 MB in the same role while carrying 4× the traffic — measurement-driven sizing (hg_advise MI) replaces multiplier folklore.
• First elastic soak, two production-adjacent nodes, every point from the arena's own log lines:

xychart-beta
    title "LB: committed MB (top line shm, flat line pkg)"
    x-axis ["23:49 boot", "23:59", "00:09", "08:19", "10:09", "14:00"]
    y-axis "committed MB" 0 --> 70
    line [64, 48, 32, 48, 34, 34]
    line [16, 16, 16, 16, 16, 16]

Loading

xychart-beta
    title "GW re-cut to -m 8:512 -M 2:32 mid-soak: shm; pkg typical; pkg 3 grown workers"
    x-axis ["23:50 boot", "00:10", "00:30", "00:57 restart", "01:17 re-cut", "01:19", "14:00"]
    y-axis "committed MB" 0 --> 70
    line [64, 48, 32, 64, 8, 24, 24]
    line [16, 16, 16, 16, 2, 2, 2]
    line [16, 16, 16, 16, 2, 18, 18]

Loading

The LB shrinks an idle 64 to its 32 floor in 20 minutes, grows to 48 under morning traffic, and the after-peak shrink releases only what is genuinely empty (committed lands on 34: 2 MB of the growth still holds a live allocation — by design, nothing is relocated). The gateway was deliberately re-cut mid-soak to a near-empty -m 8:512 -M 2:32: demand pulled shm from 8 to 24 within two minutes, and exactly 3 of 54 workers grew their private arenas — the other 51 idle at 2 MB. Whole window, both nodes: 0 grow refusals, 0 GROW-BLOCKED, all corruption counters 0.

Testing

test	result
fail-first rigs (growth, three-limit ceiling, latch+event, shrink, profiles, hugetlb pool accounting)	pass — each rig proven to fail against the bug before the fix
modules/hgstress multi-process stamp/verify soak (pid-stamped words, re-verified)	0 torn words; validated by failing when the fork-reset fix is removed
5-allocator benchmark + real-LB benchmark	above
fixed-size v2 in production (3 billing gateways + 2 LBs)	corruption counters all zero
elastic v3 soak (2 nodes, 14 h+)	charts above
CI on this tree (fork): gcc-9…14 + clang-9…19 × Ubuntu 20.04/22.04/24.04, -Werror	green, 0 warnings
multi-arch: arm32/arm64/i386/mips64 cross + qemu; full-tree arm32/arm64 native-userland container builds	green; 0 warnings, 131/131 modules link
upstream unit-test suite	2564/2564 pass

Key kernel behaviours the design rests on were measured, not assumed (and two earlier candidate designs were proven memory-corrupting on rigs before any allocator code was written): post-fork MAP_FIXED remapping corrupts pre-forked siblings — hence the pre-fork whole-cap mapping; mlock commit vs fault-in; MADV_REMOVE returning hugetlb pages under foreign mlock; map-time hugetlb pool reservation; the fork-window COW margin a hugetlb pool needs.

Documentation

Full operator + design documentation ships in-tree: mem/README.hg_malloc_v3.md (quickstart, concepts, the three-limit ceiling, GROW-BLOCKED, shrink safety argument, profile reference, deployment cookbook with the soak charts, monitoring/alerting, sizing rules, troubleshooting, measured kernel facts, internals map), plus mem/README.hg_malloc and mem/README.hg_arena_v2 for the v1/v2 layers.