fix(system_info): resolve process cgroup path for accurate memory/cpu detection in containers#972
Open
Jrojas-Baubap wants to merge 1 commit into
Open
Conversation
… detection in containers
On Fargate/Docker/Kubernetes, a process rarely lives at the cgroup fs
root — it sits in a nested sub-cgroup like /ecs/<task>/<container>
(v2) or /docker/<id> (v1). Reading only /sys/fs/cgroup/{memory.max,
cpu.max,...} silently reports the host's totals, so cbm sizes its
memory budget against the underlying node instead of the container
cap. In practice this causes OOM-kills inside Fargate tasks
(budget_mb=7845 total_ram_mb=15691 despite an 8 GB task limit).
Fix: parse /proc/self/cgroup once, resolve the process's own
sub-cgroup path against the cgroup fs root, and read the limits from
there. If resolution fails for any reason (missing /proc, unusual
formats, buffer overflows) we fall back to the previous root-reading
behaviour, so no environment regresses. cgroup version is auto-
detected by probing <root>/cgroup.controllers.
New internal helpers (declared in system_info_internal.h so the tests
can drive them against a fake filesystem):
- cbm_detect_cgroup_mem_file(path)
- cbm_detect_cgroup_cpus_file(path, is_v2)
- cbm_resolve_process_cgroup_mem_path(...)
- cbm_resolve_process_cgroup_cpu_path(...)
Tests: nine new Linux-only cases in tests/test_platform.c covering
v2 sub-cgroup + root + unlimited paths, v1 sub-cgroup, v1 combined
"cpu,cpuacct" controller matching, and the missing-/proc fallback.
Reuses the existing cgroup_test_setup / cgroup_test_write / recursive
teardown harness.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On Fargate/Docker/Kubernetes,
cbmreports the host's RAM instead of the container's cgroup limit, and sizes its memory budget accordingly. The subprocess then OOM-kills mid-pipeline.Real production log (baubap-code-graph running
cbm cli index_repositoryon a Fargate task with an 8 GB container limit on a 16 GB node):budget_mb=7845is derived fromtotal_ram_mb=15691, i.e. the node's RAM — not the 8 GB task limit.Root cause
src/foundation/system_info.creads cgroup limits from a hardcoded root:cbm_detect_cgroup_mem("/sys/fs/cgroup")→ reads/sys/fs/cgroup/memory.max(v2) or/sys/fs/cgroup/memory/memory.limit_in_bytes(v1)cbm_detect_cgroup_cpus("/sys/fs/cgroup")→ same shape for CPUBut under Docker / Kubernetes / ECS Fargate, the process does not live at the cgroup filesystem root — it sits in a nested sub-cgroup like
/ecs/<task-id>/<container-id>(v2) or/docker/<container-id>(v1). The container's effective limits live at that nested path. Reading only the root files reports the host's totals, socbm_detect_cgroup_memreturns 0 (root file missing or "max") anddetect_system_linuxfalls back tosysinfo()— the host RAM.Fix
Resolve the process's own sub-cgroup path via
/proc/self/cgroupand read the limit from there. Falls back to the original root-reading behaviour if resolution fails, so no existing environment regresses.New internal helpers (declared in
system_info_internal.hfor testability):cbm_detect_cgroup_mem_file(path)— readmemory.max/memory.limit_in_bytesfrom a direct file pathcbm_detect_cgroup_cpus_file(path, is_v2)— readcpu.max(v2 file) or v1 dir containingcpu.cfs_{quota,period}_uscbm_resolve_process_cgroup_mem_path(...)— parse/proc/self/cgroup, auto-detect v2/v1 via<root>/cgroup.controllers, compose the concrete memory file pathcbm_resolve_process_cgroup_cpu_path(...)— same for CPUdetect_system_linux()now tries the resolver first; on any failure it falls back tocbm_detect_cgroup_{mem,cpus}("/sys/fs/cgroup")(the pre-fix path).min(cgroup, host)clamping is preserved.The v1 controller matching handles the
cpu,cpuacctcombined-controller line by scanning comma-separated tokens for an exact match, so9:cpu,cpuacct:/docker/abcresolves cleanly for both thecpuandcpuacctnames.Testing
Nine new Linux-only tests in
tests/test_platform.cusing the existingcgroup_test_setup/cgroup_test_write/ recursive teardown harness (mocks a fake cgroup fs and a fake/proc/self/cgroupunder/tmp):resolve_v2_mem_sub_cgroup— Fargate-shaped/ecs/task-abc/cont-defpath; asserts 8 GB limit readresolve_v2_mem_root— process at/, verifies no double-slash in composed pathresolve_v2_mem_unlimited—memory.max = "max"returns 0resolve_v1_mem_sub_cgroup— multi-line/proc/self/cgroup,12:memory:/docker/abc123, 2 GBresolve_v1_combined_controller_no_memory_match— line only hascpu,cpuacct; memory resolver returns -1 (no silent controller confusion)resolve_missing_proc_file_returns_error— non-existent/proc/self/cgroupmock → -1; verifies root fallback still worksresolve_v2_cpu_sub_cgroup— Fargate CPU path,cpu.max = "200000 100000"→ 2 CPUsresolve_v1_cpu_combined_controller—9:cpu,cpuacct:/docker/abcresolves; 4 CPUsRun locally:
The new tests are
#ifdef __linux__-gated to match the existing cgroup suite. On non-Linux hosts the file compiles and the suite skips them, so macOS/Windows CI is unaffected.