Skip to content

fix(system_info): resolve process cgroup path for accurate memory/cpu detection in containers#972

Open
Jrojas-Baubap wants to merge 1 commit into
DeusData:mainfrom
Jrojas-Baubap:fix/resolve-process-cgroup-path
Open

fix(system_info): resolve process cgroup path for accurate memory/cpu detection in containers#972
Jrojas-Baubap wants to merge 1 commit into
DeusData:mainfrom
Jrojas-Baubap:fix/resolve-process-cgroup-path

Conversation

@Jrojas-Baubap

Copy link
Copy Markdown

Problem

On Fargate/Docker/Kubernetes, cbm reports the host's RAM instead of the container's cgroup limit, and sizes its memory budget accordingly. The subprocess then OOM-kills mid-pipeline.

Real production log (baubap-code-graph running cbm cli index_repository on a Fargate task with an 8 GB container limit on a 16 GB node):

level=info msg=mem.init budget_mb=7845 total_ram_mb=15691
level=info msg=pipeline.discover files=24267 elapsed_ms=406
... (parallel extract starts on 25 MB files) ...
[process dies with SIGKILL rc=-9]

budget_mb=7845 is derived from total_ram_mb=15691, i.e. the node's RAM — not the 8 GB task limit.

Root cause

src/foundation/system_info.c reads cgroup limits from a hardcoded root:

  • cbm_detect_cgroup_mem("/sys/fs/cgroup") → reads /sys/fs/cgroup/memory.max (v2) or /sys/fs/cgroup/memory/memory.limit_in_bytes (v1)
  • cbm_detect_cgroup_cpus("/sys/fs/cgroup") → same shape for CPU

But under Docker / Kubernetes / ECS Fargate, the process does not live at the cgroup filesystem root — it sits in a nested sub-cgroup like /ecs/<task-id>/<container-id> (v2) or /docker/<container-id> (v1). The container's effective limits live at that nested path. Reading only the root files reports the host's totals, so cbm_detect_cgroup_mem returns 0 (root file missing or "max") and detect_system_linux falls back to sysinfo() — the host RAM.

Fix

Resolve the process's own sub-cgroup path via /proc/self/cgroup and read the limit from there. Falls back to the original root-reading behaviour if resolution fails, so no existing environment regresses.

New internal helpers (declared in system_info_internal.h for testability):

  • cbm_detect_cgroup_mem_file(path) — read memory.max / memory.limit_in_bytes from a direct file path
  • cbm_detect_cgroup_cpus_file(path, is_v2) — read cpu.max (v2 file) or v1 dir containing cpu.cfs_{quota,period}_us
  • cbm_resolve_process_cgroup_mem_path(...) — parse /proc/self/cgroup, auto-detect v2/v1 via <root>/cgroup.controllers, compose the concrete memory file path
  • cbm_resolve_process_cgroup_cpu_path(...) — same for CPU

detect_system_linux() now tries the resolver first; on any failure it falls back to cbm_detect_cgroup_{mem,cpus}("/sys/fs/cgroup") (the pre-fix path). min(cgroup, host) clamping is preserved.

The v1 controller matching handles the cpu,cpuacct combined-controller line by scanning comma-separated tokens for an exact match, so 9:cpu,cpuacct:/docker/abc resolves cleanly for both the cpu and cpuacct names.

Testing

Nine new Linux-only tests in tests/test_platform.c using the existing cgroup_test_setup / cgroup_test_write / recursive teardown harness (mocks a fake cgroup fs and a fake /proc/self/cgroup under /tmp):

  • resolve_v2_mem_sub_cgroup — Fargate-shaped /ecs/task-abc/cont-def path; asserts 8 GB limit read
  • resolve_v2_mem_root — process at /, verifies no double-slash in composed path
  • resolve_v2_mem_unlimitedmemory.max = "max" returns 0
  • resolve_v1_mem_sub_cgroup — multi-line /proc/self/cgroup, 12:memory:/docker/abc123, 2 GB
  • resolve_v1_combined_controller_no_memory_match — line only has cpu,cpuacct; memory resolver returns -1 (no silent controller confusion)
  • resolve_missing_proc_file_returns_error — non-existent /proc/self/cgroup mock → -1; verifies root fallback still works
  • resolve_v2_cpu_sub_cgroup — Fargate CPU path, cpu.max = "200000 100000" → 2 CPUs
  • resolve_v1_cpu_combined_controller9:cpu,cpuacct:/docker/abc resolves; 4 CPUs

Run locally:

scripts/test.sh                     # full C tests, ASan + UBSan
make -f Makefile.cbm test-foundation  # foundation only (fast)

The new tests are #ifdef __linux__-gated to match the existing cgroup suite. On non-Linux hosts the file compiles and the suite skips them, so macOS/Windows CI is unaffected.

… detection in containers

On Fargate/Docker/Kubernetes, a process rarely lives at the cgroup fs
root — it sits in a nested sub-cgroup like /ecs/<task>/<container>
(v2) or /docker/<id> (v1). Reading only /sys/fs/cgroup/{memory.max,
cpu.max,...} silently reports the host's totals, so cbm sizes its
memory budget against the underlying node instead of the container
cap. In practice this causes OOM-kills inside Fargate tasks
(budget_mb=7845 total_ram_mb=15691 despite an 8 GB task limit).

Fix: parse /proc/self/cgroup once, resolve the process's own
sub-cgroup path against the cgroup fs root, and read the limits from
there. If resolution fails for any reason (missing /proc, unusual
formats, buffer overflows) we fall back to the previous root-reading
behaviour, so no environment regresses. cgroup version is auto-
detected by probing <root>/cgroup.controllers.

New internal helpers (declared in system_info_internal.h so the tests
can drive them against a fake filesystem):

  - cbm_detect_cgroup_mem_file(path)
  - cbm_detect_cgroup_cpus_file(path, is_v2)
  - cbm_resolve_process_cgroup_mem_path(...)
  - cbm_resolve_process_cgroup_cpu_path(...)

Tests: nine new Linux-only cases in tests/test_platform.c covering
v2 sub-cgroup + root + unlimited paths, v1 sub-cgroup, v1 combined
"cpu,cpuacct" controller matching, and the missing-/proc fallback.
Reuses the existing cgroup_test_setup / cgroup_test_write / recursive
teardown harness.
@Jrojas-Baubap Jrojas-Baubap requested a review from DeusData as a code owner July 8, 2026 23:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant