DO NOT MERGE (managed store sizing): re-derive work_mem after a resize and keep one v8 block (#4207)

[erikdarlingdata]

Closes #4207.

Why

The v8 hardware-sizing block (#2845) re-derives sizing after a host resize. It re-states effective_cache_size, maintenance_work_mem, timescaledb.max_background_workers and max_worker_processes. It never re-stated work_mem. So after a RAM resize, work_mem stayed at whatever the v3 block computed for the old hardware, forever. On the field stores that is 31 MB, computed at 16 GiB.

DeriveMemorySettings at the reported 33,788,809,216 bytes (nominally "31.5 GiB", actually 31.47 GiB) now gives 62 MB. One store spilled 33 TB to temp files since creation. Another spilled 7 TB.

work_mem was excluded from v8 on purpose in #2845. That exclusion rested on a measurement that does not apply here. The comparison found a regression at 512 MB, which is 8x this formula's own 64 MB ceiling. Nothing in this codebase writes that value. Nobody had measured the value the formula actually derives. #4207 measures the cost of leaving it out instead: 33 TB and 7 TB of pg_stat_database.temp_bytes, on hosts sized for the RAM they now have.

Second, smaller defect: the v8 block was appended fresh every time its fingerprint changed. The fingerprint includes the worker count, and the worker count moves with the hypertable count. Every field store now carries three v8 blocks (worker counts 72, 73, 74), and postgresql.conf grows by one block on every schema change from here.

What changes

Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs:

• BuildHardwareSizingConfAppend now emits work_mem, from the same DeriveMemorySettings call that already derives the other three settings. shared_buffers stays excluded. That exclusion is structural, not measurement-based. See the doc comment.
• New ReplaceOrAppendHardwareSizingBlock rewrites the v8 block in place instead of appending. It finds every existing v8 block (FindHardwareSizingBlockSpans), rewrites the first one with the fresh content, and drops every later copy outright. No existing block still falls back to a plain append, unchanged from today. A block has no end marker of its own today. FindHardwareSizingBlockEnd defines the end marker from how every block in this file is written. Each Build*ConfAppend opens with its own leading blank line and has no blank line inside itself. So the first blank line after the marker is always the start of whatever comes next. The doc comment states that rule and its one known edge case: an operator line spliced in with no blank line.
• EnsureConfAppended's v8 step re-reads postgresql.conf from disk immediately before rewriting, rather than reusing the snapshot taken at the top of the method. v1-v7 above it can append their own healing blocks straight to disk in the same call. Reusing the stale snapshot loses them silently.
• Chose not to split the worker fingerprint from the hardware fingerprint. Managed-store memory sizing never re-derives after a host resize (effective_cache_size stale at 75% of the OLD RAM) #2845 considered that split to prevent a hypertable-count change from re-triggering the memory lines. With an in-place rewrite a worker-only change costs exactly what a memory-only change costs: one block rewritten, not a new one appended. There is nothing left for the split to buy.
• Updated the doc comments on ConfMarkerV8/BuildHardwareSizingConfAppend to record the reversal on work_mem. Updated the log line to report the derived work_mem value and how many stale copies, if any, were collapsed.

Manual overrides keep winning, unchanged. postgresql.auto.conf (ALTER SYSTEM) is read after postgresql.conf in its entirety, and always wins regardless of anything here. A hand edit inside postgresql.conf wins by last-occurrence.

Rewriting at the position of the first v8 block preserves that. Collapsing can only move the v8 lines earlier in the file, never later. Anything that used to lose to the last block was already losing. Anything that used to beat it still does.

Test plan

• Darling.Tests builds with 0 warnings.
• New/changed tests in DarlingManagedPostgresTests.cs, proven red against the pre-fix source once. Temporarily reverted only the source file and rebuilt against the new tests:
  • HardwareSizingConfAppend_EmitsAllFourMemorySettings (pin: all four MemorySettings values present)
  • HardwareSizingConfAppend_MeasuredFieldRam_EmitsWorkMem62Mb (the exact field byte count. The issue's rough "63 MB" is 31.5 GiB's raw RAM/512 without the QuantizeRam step. The reported bytes quantize down to 31 GB and derive to 62 MB. This is noted in both the test and the source doc comment.)
  • HardwareSizingConfAppend_EmitsHostDerivedSettings (extended with the work_mem assertion)
  • ReplaceOrAppendHardwareSizingBlock_TwoSuccessiveFingerprintChanges_LeaveExactlyOneBlock (pin: two successive changes leave exactly one block)
  • ReplaceOrAppendHardwareSizingBlock_ThreeExistingBlocks_CollapseToOneAtTheFirstPosition (pin: three existing blocks collapse to one, at the first block's position)
  • ReplaceOrAppendHardwareSizingBlock_LinesOutsideBlocks_AreByteIdenticalAfterRewrite (pin: bytes outside any v8 span, including an operator line sitting between two blocks, survive untouched)
  • ReplaceOrAppendHardwareSizingBlock_NoExistingBlock_AppendsOne (fallback path unchanged)
  • EnsureConfAppended_ThreeExistingV8Blocks_CollapseToOneOnTheNextHeal (the field shape end to end, through the real heal path, not just the pure function)
• DarlingManagedPostgresTests, StartupCommandTimeoutTests, DocCommentHygiene: 254 total, 0 failed.
• Full Darling.Tests suite, once: 13767 total, 0 errors, 0 failed. 690 skipped (gated on DARLING_TEST_PGRUNTIME/_OLD, not set in this run). 1 not run (an unrelated gated theory in DarlingStoreUpgradeTests). The known flake (ServerListAndSummaryPlanShapeTests.TheShippedReads_TouchFarFewerChunks...) did not fail.
• The DARLING_TEST_PGRUNTIME-gated E2E tests were not run: Bootstrap_EndToEnd_..., the maintenance_work_mem formula lands too low for compression throughput: +70% measured from raising it, plateaus by 1.5 GB #1777 and job_history is silently empty on clusters predating ~2026-08-17, because its GUC lives in the one postgresql.conf block EnsureConfAppended cannot heal #3175 conf-propagation tests, the PostgreSQL 17 cap tests. No rig was started for this lane. They exercise the real pg_ctl start path rather than the v8 block itself. The new pins cover the v8 behavior directly. One goes through the real EnsureConfAppended path against a fabricated three-block conf.

This never touches a live store. The change reaches the field stores only through a build and an install. That is Erik's call, not this PR's.

CHANGELOG entry

SECTION: Fixed
ENTRY:

• Managed PostgreSQL stores now re-derive work_mem after a hardware resize, and stop piling up duplicate sizing blocks ([DO NOT MERGE (managed store sizing): re-derive work_mem after a resize and keep one v8 block (#4207) #4225]). After a RAM resize, a managed store's work_mem stayed at the value computed for the old hardware indefinitely. The block that re-derives sizing after a resize never re-stated work_mem. It now does. The same block was also appended fresh every time its derivation inputs changed, instead of replacing the block it superseded. A store accumulated several copies over its lifetime. It now rewrites the existing block in place.
  REF:
  [DO NOT MERGE (managed store sizing): re-derive work_mem after a resize and keep one v8 block (#4207) #4225]: DO NOT MERGE (managed store sizing): re-derive work_mem after a resize and keep one v8 block (#4207) #4225

---

Generated with Claude Code

https://claude.ai/code/session_01FVjn4PBJN71NQXdFo6ZxNQ