Skip to content

Managed store: the v8 hardware-sizing block never re-derives work_mem (stuck at the pre-resize value), and it re-appends itself on every hypertable-count change #4207

Description

@erikdarlingdata

Problem

The v8 block (#2845) exists so that a resized host re-derives its memory and worker sizing. The v1–v7 blocks are marker-guarded and are written only once. v8 re-states effective_cache_size, maintenance_work_mem, timescaledb.max_background_workers and max_worker_processes, but not work_mem. So after a RAM resize, work_mem stays at whatever the v3 block computed for the old hardware, indefinitely.

A second, smaller defect: the v8 block is appended again whenever its fingerprint changes, and the fingerprint includes the worker derivation, which moves with the hypertable count. Each managed store carries three v8 blocks (max_background_workers 72 → 73 → 74). postgresql.conf grows by one block every time the aggregate or hypertable count changes. The last one wins, so the effect is only clutter today, but it compounds with every schema change.

Measured

All three production managed stores, build 3.8.0-nightly.20260924.467, 2026-09-25. The hosts are 31.5 GiB (33,788,809,216 bytes), 8 vCPU, resized from 16 GiB.

  • pg_settings.work_mem = 31744 kB, sourceline = the v3 block (work_mem = 31MB, computed at 16 GiB). DeriveMemorySettings at 31.5 GiB gives clamp(RAM/512, 16 MB, 64 MB) = 63 MB.
  • effective_cache_size = 23808MB and maintenance_work_mem = 1587MB both come from the latest v8 block, so the heal works for the settings it names.
  • postgresql.conf shows the v8 marker three times on each store (lines ~925/941/948), with worker counts 72/73/74.

Cost of the stale value: pg_stat_database.temp_bytes since cluster creation is 33 TB on one store and 7 TB on another. The busiest statements spill at 31 MB (see #3953 and the file-I/O read). Not all of that would be avoided at 64 MB, but the host is sized for it.

Where

  • Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs:
    • ConfMarkerV8 doc ~:188–205: "v8 re-states the SAME formulas (it calls DeriveMemorySettings …)". The block builder emits only three of MemorySettings' four values.
    • DeriveMemorySettings ~:805–890: the work_mem formula that is never re-applied.
    • The v8 append logic keys on ConfHardwareFingerprintPrefix. It appends a new block instead of replacing the prior v8 block.

Fix shape

  • Emit work_mem in the v8 block, from the same DeriveMemorySettings call.
  • Replace instead of append. On a fingerprint change, rewrite the existing v8 block in place: find the marker, replace through the block end. Alternatively, keep v8's derived values in postgresql.auto.conf-style single-owner storage. Either way, one v8 block per store.
  • Consider separating the worker fingerprint from the hardware fingerprint, so a hypertable-count change doesn't re-trigger the memory lines.
  • Pin: a unit test that the v8 block contains all four MemorySettings values, and a conf-rewrite test that two successive fingerprint changes leave exactly one v8 block.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingclient-siteOwned by the client-site agents (other laptop). Local sessions never pick these up.

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions