Managed store sizing: make the derivation platform-aware (Windows 487 cap only on Windows, Linux cgroup limits, commit headroom) and write it to one service-owned config file instead of appended blocks

[erikdarlingdata]

Problem

The derivation is one formula for every platform. DeriveMemorySettings caps shared_buffers at min(25% RAM, 1 GB). The cap has two stated reasons: co-location/double caching, and the Windows error-487 shared-memory reattach failures (#1559/#1581; upstream BUG #14050/#18954). The 487 mechanism is Windows-only. PostgreSQL on Linux maps shared memory with mmap/SysV and has no reattach step, so a Linux store is capped for a Windows reason. In the other direction, a Linux store in a container must size from the cgroup limit, not host RAM, or shared_buffers + work_mem headroom gets OOM-killed. On Windows the shared segment counts against the commit limit (RAM + pagefile), which nothing checks today.

The config is appended, not owned. Every sizing change is a new marker block appended to postgresql.conf (v1–v14; last occurrence wins). On the three production stores:

• the v8 hardware block appears four times on one store and three on the others (it re-appends on every fingerprint change, Managed store: the v8 hardware-sizing block never re-derives work_mem (stuck at the pre-resize value), and it re-appends itself on every hypertable-count change #4207);
• shared_buffers is stated three times (default 128MB, v3 4048MB/1024MB, v5 1024MB);
• work_mem survives only in the v3 block, because no later block restates it.

Reading the effective value means reading the whole file bottom-up, and an operator can't tell "the service set this" from "someone edited this".

Proposal

1. Platform-aware derivation (pure, testable, same inputs as the host profile in the companion issue):

• Windows: keep the 487-motivated cap, and make it a named, documented Windows term. Add a commit-headroom check: shared_buffers + a safety margin must fit under (commit limit − committed). Below that, clamp and say so.
• Linux: shared_buffers at the standard 25% of the effective memory limit (cgroup-aware), huge_pages = try. Keep the co-location factor as its own term if the service and viewer share the host.
• All platforms: work_mem is in the same re-derivation as everything else (Managed store: the v8 hardware-sizing block never re-derives work_mem (stuck at the pre-resize value), and it re-appends itself on every hypertable-count change #4207). Consider scaling hash_mem_multiplier once work_mem is ≥ 40 MB, per the PostgreSQL docs' guidance.

2. One service-owned file. Put include_dir 'darling.conf.d' (or include_if_exists 'darling-managed.conf') in postgresql.conf once, and have the service rewrite that file wholesale on every start from the current derivation. It's atomic (write to temp, then rename) and has a single header recording the derivation inputs: platform, RAM/limit, CPUs, hypertable count, formula version.

• The v1–v14 append blocks are migrated once into the file and removed. The migration is a single, idempotent step with a backup of the original conf.
• Operator overrides keep working: ALTER SYSTEM (postgresql.auto.conf) is read after postgresql.conf and wins, and the host-profile check reports it as operator-override.
• Restart-only settings (shared_buffers, worker counts) set pending_restart. The check reports it, and the service logs it once instead of re-appending.

Pins:

• derivation unit tests per platform (Windows cap, Linux cgroup, the commit clamp);
• a conf-ownership test: two starts with a changed fingerprint leave exactly one managed file and zero appended blocks;
• a migration test from a conf carrying today's v1–v14 blocks.

Context: measured 2026-09-25 on the three production managed stores (m7i/r7i, 8 vCPU, 31.5 → 63 GiB). A test on one store raised shared_buffers to 8 GB and work_mem to 64 MB. 487 retries stayed at the pre-change rate (~1/h, all LOG-level retries, 0 FATAL) in the first minutes. The 24 h verdict will be added here.

[erikdarlingdata]

Design: platform-aware sizing, written to one service-owned file

Problem

DeriveMemorySettings uses one formula everywhere: a Linux store gets the Windows-only 1 GiB cap for error 487, and nothing checks the Windows commit limit or a Linux container limit. The results go into postgresql.conf as appended v1-v14 blocks, where the last line wins and nobody can tell the service's lines from an operator's.

Everything below was checked on a local PostgreSQL 18.6 + TimescaleDB rig (the bundled runtime) and on a 128 GB Windows workstation. No production store was touched. The scripts are prototypes, not product code.

1. The derivation, per platform

The file header (section 2) records every input. They are the platform, the memory limit and its source, the commit limit and charge (Windows), CPUs, hypertable count, PostgreSQL major, data-volume free and total space (for WAL), and the formula version.

Setting	Windows	Linux
memory limit M	physical RAM, GlobalMemoryStatusEx, quantized as today	min(MemTotal, cgroup limit)
shared_buffers	min(25% M, WindowsReattachCap), then the commit clamp	25% (M - ColocatedServiceReserve)
effective_cache_size	75% M	75% M
maintenance_work_mem	today's formula (#1777, #3909)	same
work_mem	clamp(M/512, 16, 64 MB), re-derived every time (#4207)	same
hash_mem_multiplier	PostgreSQL default 2.0, not written (see below)	same
huge_pages	not written (needs the "Lock pages in memory" right)	try, written only to state intent: try is already the default

Windows reattach cap

The 1 GiB term becomes a named constant, WindowsReattachCapMb = 1024, applied only when the platform is Windows. Its doc comment carries the 487 history (#1559, #1581, pgsql-bugs BUG #14050 / #18954). Raising it is then a one-line, reviewed formula change with its own test. The 24-hour verdict on the 8 GB canary is the evidence for or against raising it.

Windows commit headroom

Windows charges the whole shared memory segment against the commit limit (RAM + pagefile) when the postmaster starts, before any page is used. Measured on the workstation:

Reading	Value
RAM	127.7 GiB
commit limit (RAM + a 23.8 GiB system-managed pagefile)	151.5 GiB
committed, PostgreSQL stopped	139.9 GiB (94% in use, with other work running)
committed bytes added by starting PostgreSQL at shared_buffers = 1024MB	+1,186 MiB
committed bytes added by starting PostgreSQL at shared_buffers = 1280MB	+1,386 MiB (shared_memory_size = 1374MB)

The limit and the charge come from the same GlobalMemoryStatusEx call the service already makes: ullTotalPageFile is the commit limit, and ullTotalPageFile - ullAvailPageFile is the charge. On this machine they matched the \Memory\Commit Limit counter to the byte (162,623,135,744) and Win32_OperatingSystem.TotalVirtualMemorySize. The rule:

• segment(sb) + margin <= commit limit - committed, measured before pg_ctl start, so PostgreSQL's own segment is not counted. The --check-settings verb from Store host visibility: a cross-platform host profile, a --check-settings verb, and a get_store_host read, so "is this store sized right" has an answer without shell access #4214 runs while the server is up, so it adds the live segment back.
• segment(sb) is exact from postgres -D <data> -C shared_memory_size, which works only while the server is stopped. That is exactly when the service derives. On the rig it refused with "lock file already exists" while the server was running. If it fails, fall back to 1.1 x sb + 64 MB.
• margin = 2 GiB + maintenance_work_mem: room for one maintenance or compression job at full size, plus backends and the service.
• If the rule fails, step shared_buffers down in 128 MB steps to a 128 MB floor. Log one warning with the numbers. The Store host visibility: a cross-platform host profile, a --check-settings verb, and a get_store_host read, so "is this store sized right" has an answer without shell access #4214 check reports clamped-commit-headroom.

This workstation shows why the clamp matters: a plain 25% asks for a 32 GiB segment against 11.6 GiB of headroom. I did not provoke that failure on a shared machine. From the source, CreateFileMapping fails and the postmaster does not start.

Linux: effective memory limit

The managed runtime is Windows-only today. Its class and constructor carry [SupportedOSPlatform("windows")], and the runtime zip is a Windows x64 build. So the Linux term ships as a pure function with unit tests and no caller until a Linux runtime exists. There is no Linux box here, so these points come from the docs:

• cgroup v2: find the cgroup from the 0::/path line of /proc/self/cgroup. Read memory.max there and in each parent up to the root, and take the smallest number. max means no limit. The kernel's cgroup-v2 doc calls memory.max the hard limit.
  In its words: "If a cgroup's memory usage reaches this limit and can't be reduced, the OOM killer is invoked in the cgroup." The same doc counts shmem ("shm segments, shared anonymous mmap()s") in the cgroup's memory, so shared_buffers is charged to the container as its pages are touched.
• cgroup v1: memory.limit_in_bytes under the memory controller path. A value near 2^63 means no limit.
• M = min(MemTotal, that limit). This holds because PostgreSQL runs as the service's child, so it shares the service's cgroup. memory.high is a throttle, not a kill line: report it, but do not size from it.
• 25% is the docs' figure for a dedicated server: "a reasonable starting value for shared_buffers is 25% of the memory in your system ... it is unlikely that an allocation of more than 40% of RAM to shared_buffers will work better". The managed store is not dedicated, because the service runs beside it. So ColocatedServiceReserve (proposed: 2 GiB) comes off before the 25%, as its own named term.
• huge_pages = try: the docs say "try (the default) ... will try to request huge pages, but fall back to the default if that fails". Real huge pages need vm.nr_hugepages sized from postgres -C shared_memory_size_in_huge_pages. That is host setup, not the service's job, so the check reports huge_pages_status. On the Windows rig it reads off.

hash_mem_multiplier

The docs: "Consider increasing hash_mem_multiplier in environments where spilling by query operations is a regular occurrence ... Higher settings in the range of 2.0 - 8.0 or more may be effective in environments where work_mem has already been increased to 40MB or more." That guidance has two conditions: spills are regular, and work_mem is at 40 MB or more. The derivation reaches 62-64 MB on the field hosts, so the second holds. For the first, #4207's temp-byte totals were measured at 31 MB, before #4225, and nobody has split them into hash spills and sort spills. I propose a named term that stays at 2.0 and is not written. If the #4214 check shows hash-node spills at the new work_mem, a formula-version bump raises it to 4.0. Worst case at 4.0 is 256 MB per hash node per backend.

WAL settings (#4246)

These are constant or disk-derived terms in the same file, not memory terms:

Setting	Term	Applies on
wal_compression	a constant. The bundled 18.6 build lists {pglz,lz4,zstd,on,off}, and a session accepted both lz4 and zstd on the rig. lz4 matches the store's default_toast_compression = lz4 and costs the least CPU. zstd compresses more.	reload (context superuser)
checkpoint_timeout	a constant, if #4246 goes ahead (#3892 declined 900 s on fsync grounds)	reload (sighup)
max_wal_size, min_wal_size	stay derived from free disk as v12 does today	reload (sighup)

WAL volume is workload, not hardware, so the derivation does not guess it. To keep checkpoints time-driven, the check reads pg_stat_checkpointer and reports wal-size-too-small when requested checkpoints outnumber timed ones. An ALTER SYSTEM value for any of the three behaves exactly like the shared_buffers case below: postgresql.auto.conf wins, and the check says so. Whether to change them on production is decided on #4246.

2. One service-owned file

Layout

postgresql.conf gets a single include 'darling-managed.conf' line at its end, and the file sits next to it in the data directory. The file holds every setting the v1-v14 blocks set: 23 keys in the rehearsal. That covers sizing, port, listen_addresses, the preload list, logging, timezone, lc_messages, job logging, WAL, and statement statistics.

It opens with a header of inputs, with no timestamp, so an unchanged derivation renders byte-identical text and the service skips the write:

# darling-managed.conf -- owned by the PerformanceMonitor Darling service; rewritten whole on every start.
# Do not edit. To pin a value, use ALTER SYSTEM (postgresql.auto.conf wins over this file). See #4215.
# formula=2 platform=windows ram-bytes=33788809216 ram-gb=31 commit-limit-bytes=... committed-bytes=...
# cpus=8 hypertables=72 pg-major=18 data-volume-free-gb=400 total-gb=500
# body-sha256=<hash of the lines below>
shared_buffers = 1024MB   # min(25% of 31 GiB, WindowsReattachCap 1024MB); commit headroom ok
...

Which include form

Checked with postgres -C against each form with a missing target:

Form	Target missing	Other finding
include 'darling-managed.conf'	FATAL: configuration file ... contains errors; the server does not start	reads exactly one named file
include_if_exists 'darling-managed.conf'	LOG: skipping missing configuration file, and it starts on defaults (work_mem fell back to 4MB)	same
include_dir 'darling.conf.d'	FATAL (missing directory)	a stray Copy of 50-darling-managed.conf sorts after 50-darling-managed.conf in C order and won (31MB over 62MB). A .conf.tmp file was ignored

Recommendation: plain include. The file carries shared_preload_libraries, port and listen_addresses. If it goes missing, a silent start on port 5432 without TimescaleDB is worse than a refusal that names the file. The service writes the file before every start it owns, so it never hits the missing case itself. include_dir buys drop-in files, which is exactly the hazard the stray-copy test shows.

Where the line goes: the end

initdb writes live lines near the top for shared_buffers, max_connections, max_wal_size, min_wal_size, timezone and lc_messages. An include at the end beats them. Anything an operator adds after the include line beats the service. That becomes the second override path (section 4).

Precedence, proved on the rig

Setup: initdb's shared_buffers = 128MB at line 132, the include at the end with shared_buffers = 256MB / work_mem = 62MB, then ALTER SYSTEM SET shared_buffers = '384MB' and a restart:

pg_file_settings row	setting	applied
postgresql.conf : 132	shared_buffers 128MB	f
darling-managed.conf : 1	shared_buffers 256MB	f
postgresql.auto.conf : 3	shared_buffers 384MB	t
darling-managed.conf : 2	work_mem 62MB	t

pg_settings then showed shared_buffers from postgresql.auto.conf line 3, and work_mem from darling-managed.conf line 2. A hand line work_mem = 70MB appended after the include (line 898, include at 897) won after a reload: sourcefile = postgresql.conf, sourceline = 898. Raising the service file to 512MB while the ALTER SYSTEM value stood changed nothing and set no pending restart. The docs agree: "Settings in postgresql.auto.conf override those in postgresql.conf", and "all but the last one are ignored".

Atomic rewrite on Windows

Write darling-managed.conf.tmp in the same directory, flush it to disk, then replace (File.Move(tmp, dest, overwrite: true), which is MoveFileEx with MOVEFILE_REPLACE_EXISTING). I raced 4,000 re-parses of pg_file_settings (the same parser a reload runs) against rewrites of a 72-line file:

Write method	Reads that saw a torn file (no work_mem line)	Writer errors
in-place File.WriteAllText	477 of 4,000 (11.9%)	0
temp file + replace	0 of 4,000	170 of 1,964 replaces: UnauthorizedAccessException (0x80070005) while PostgreSQL had the file open

With a retry every 5 ms, every replace succeeded: 120 on the first try, 84 on the second, 3 on the third. A torn read during a reload silently puts every reloadable setting back to its postgresql.conf value. So:

• The writer retries the replace up to 40 times, 50 ms apart.
• On failure it keeps the old file, which is always complete, and logs.
• It never writes in place.

The service writes before pg_ctl start, so contention there is only from antivirus or backup agents. It matters for any later rewrite made while the server is running.

Each start

The service:

1. derives;
2. renders the file;
3. stops here if the file is already byte-identical;
4. if the body no longer matches its own body-sha256, saves the hand-edited file as darling-managed.conf.edited-<utc>.bak and logs a warning. It never silently discards an edit.
5. replaces the file atomically and logs each key's old and new value, and whether it needs a restart;
6. puts the include line back at the end, with a warning, if someone removed it.

If the service ever rewrites while PostgreSQL is running, it calls pg_reload_conf() afterwards and checks for pending restarts (section 5).

3. Migrating the v1-v14 blocks

What moves

A block is its marker line plus the non-blank lines after it, which is the rule FindHardwareSizingBlockEnd already uses. Each block version has a known key list (v8: effective_cache_size, maintenance_work_mem, work_mem and the two worker counts). v12 also has its stamp and comment. Inside a block, a line is the service's only if its key is on that version's list and is new to that block. Anything else stays: a duplicate key, an unknown key, or a line spliced in without a blank line (the edge case the #4207 code documents).

Rules

1. Keep a backup of the original: postgresql.conf.pre-4215.<utc>.bak. Never overwrite an existing backup.
2. Build the file from the effective values, not a fresh derivation. For each key, use the value on the last service line. Then the migration changes no value, and any derivation change comes later as its own logged step.
3. Keep operator lines winning. Take an operator line that is currently effective for a key the file will own. It moves, verbatim, below the include line under a comment, so it keeps winning. A line that was already overridden stays where it is and stays overridden.
4. Write in two atomic steps. First the managed file, which is harmless while nothing includes it. Then the new postgresql.conf in one replace. A crash between them leaves the old conf working, and a re-run converges on the same result.
5. Verify against PostgreSQL itself, while the server runs. pg_file_settings re-parses the files on disk every time it is read. The migration snapshots its applied values before and after, and requires them identical, with zero parse errors. On any difference it restores the backup. The values are identical, so no reload or restart is needed. Reading pg_file_settings takes superuser, which the managed store's bootstrap role is.
6. Run it once: when no marker is left and the include line is present, it does nothing.

Rehearsal on a synthetic copy shaped like the field

It held initdb's conf plus v1-v14. The v8 block appeared four times, with fingerprints for 16 then 31 GB and 66-72 hypertables. shared_buffers was stated three times (initdb 128MB, v3 4048MB, v5 1024MB). There were 17 marker lines in all, plus three kinds of hand edit:

• max_connections = 300 between v4 and v5, which was effective;
• work_mem = 70MB spliced into the second v8 copy with no blank line, which was not effective;
• log_timezone = 'UTC' at the end.

Then ALTER SYSTEM SET shared_buffers = '1280MB' and a restart, to stand in for tonight's field change.

Check	Result
lines removed / keys in the file / operator lines moved	48 / 23 / 1 (max_connections = 300, now below the include)
pg_file_settings applied values, before vs after, no reload	identical, 0 parse errors
pg_settings values from config files, after a restart	identical
second run	no-op; both files byte-identical
fault injected after the managed file was written	postgresql.conf byte-identical to before; applied values unchanged; the re-run produced the same bytes as the clean run; one backup
shared_buffers after	1280MB from postgresql.auto.conf line 3: the ALTER SYSTEM value survived untouched
maintenance_work_mem after	2047MB (the v14 value) from darling-managed.conf: the effective value moved, not re-derived
max_connections after	300 from postgresql.conf line 900, below the include line

Not rehearsed: a value edited inside a block, for example work_mem = 128MB typed over the v8 line. By position it counts as the service's line, and the next derivation would replace it. I propose this rule. Take a block line whose value that block's builder cannot produce from the block's recorded inputs. Treat it as an operator line and move it below the include. v8 records its fingerprint, and v1, v4, v6, v9-v11, v13 and v14 are constants, so each of those can be rebuilt and compared. v2, v3, v5 and v7 were derived from inputs they did not record, so for them the test is only "within the formula's possible range".

4. Operator overrides

How to pin a value

ALTER SYSTEM SET ... is the supported way, and it is what the field used tonight. postgresql.auto.conf is read after postgresql.conf and its includes, so it wins, as the proof above shows. The service never edits postgresql.auto.conf and never runs ALTER SYSTEM RESET. That stays the operator's move. The v12 WAL heal already follows this rule.

A line placed after the include in postgresql.conf also wins, and gets the same report. That is the path if allow_alter_system = off, a PostgreSQL 17+ setting the rig lists. I recommend no extra operator include file: ALTER SYSTEM already beats it, and it would add a third layer to explain.

How the service shows it

After each start, for every key it owns, the service reads pg_settings.sourcefile / sourceline and compares the value in effect with the derived value. Anything not sourced from darling-managed.conf becomes operator-override, with the file and line, for example "shared_buffers = 8GB from postgresql.auto.conf line 3; the service derives 1024MB". This is the per-setting verdict #4214 already specifies. It feeds --check-settings, get_store_host and the Settings UI from one implementation, and the service logs one warning per overridden key per start. The service still writes its own value into its file, so ALTER SYSTEM RESET falls back to the derived value, not to a stale block.

5. Rollout, restart versus reload

Restart-only (postmaster context): shared_buffers, max_connections, max_worker_processes, timescaledb.max_background_workers, shared_preload_libraries, huge_pages, port, listen_addresses, logging_collector.

Reload (sighup, superuser or user): work_mem, maintenance_work_mem, effective_cache_size, hash_mem_multiplier, max_wal_size, min_wal_size, checkpoint_timeout, checkpoint_completion_target, wal_compression, timezone, lc_messages, the other log settings, and pg_stat_statements.track_utility.

Detecting a pending restart on Windows

Do not trust pg_settings.pending_restart from a new connection. On the rig, after the file changed shared_buffers and pg_reload_conf() ran, the postmaster logged parameter "shared_buffers" cannot be changed without restarting the server. pending_restart was t in the session that was open across the reload, and f in every new session. Windows backends are started fresh, not forked, so they do not inherit that flag.

The reliable signal, from any session, is a pg_file_settings row whose error is setting could not be applied. The service and the check use that, and the service logs it once, with no re-append. The PG-target collectors (DarlingPgServerConfigReader, PgTargetFactCollector.Config) read pending_restart from new connections, so they likely miss it on Windows-hosted targets. That deserves its own issue.

Two releases

• Release A, file ownership with no value changes. On the first start with A, PostgreSQL starts on the old conf, and the migration runs once online, checked by pg_file_settings. It needs no reload or restart. From the next start on, the service writes the file before pg_ctl start, so restart-only values apply at that start. A store with an ALTER SYSTEM value keeps it and shows operator-override.
• Release B, formula version 2. This brings the platform terms: the named Windows cap, the commit clamp, and the Linux function. On today's Windows fleet it changes a value only where the clamp binds, and it logs each change with its reason. Restart-only changes take effect at the next service-owned start. Reloadable ones take effect at the same start, or via pg_reload_conf() if the file is rewritten while the server runs.

Pins (the issue's three, plus two):

• derivation unit tests per platform (the Windows cap, the commit clamp including the floor, the Linux cgroup v1 and v2 parsers with max and nested limits);
• conf ownership: two starts with a changed fingerprint leave one managed file and zero blocks;
• migration from a generated conf with today's blocks, the four v8 copies and the hand edits above;
• a live test that pending_restart detection reads pg_file_settings;
• a test that the writer never writes the file in place.

Lite: Lite has no PostgreSQL (it stores in DuckDB), so none of this applies to it.

What Erik decides

1. The include form: plain include (recommended), include_if_exists, or include_dir.
2. The Windows cap after the 24 h canary verdict: keep 1 GiB, or raise it (for example min(25% M, 8 GiB)) in Release B.
3. The commit margin (proposed 2 GiB + maintenance_work_mem), and whether a binding clamp lowers shared_buffers or only warns.
4. The override paths: ALTER SYSTEM plus lines after the include (recommended), or also an operator file or service-config pins.
5. hash_mem_multiplier: stay at 2.0 until hash spills are shown (recommended), or 4.0 whenever work_mem >= 40MB.
6. The Linux ColocatedServiceReserve value (proposed 2 GiB).
7. Whether to ship as two releases (A, then B; recommended) or one.
8. For Managed store: 66–90% of WAL is full-page images, with wal_compression off and checkpoint_timeout at the 5-minute default; together they are worth an estimated half of WAL #4246: which wal_compression method, and whether checkpoint_timeout changes. This design only gives them a home.

Open risks

• Downgrade after migration. An older build finds no markers, so it appends v1-v14 below the include, and those blocks win again. The store still works, but the file grows back. One fix: the migration leaves one comment line per legacy marker just above the include. An old build then appends nothing, and v8's in-place rewrite lands above the include, where the file still overrides it. Not rehearsed.
• Edits inside a block, as described in section 3: the rule is designed but not rehearsed.
• The commit reading is a snapshot. A system-managed pagefile can grow later. ullTotalPageFile is the smaller of the system and process limits, so a job object on the service would lower it. The field stores' commit limits are unmeasured, so Store host visibility: a cross-platform host profile, a --check-settings verb, and a get_store_host read, so "is this store sized right" has an answer without shell access #4214's host profile should record them before Release B.
• Replace contention. Antivirus or backup agents can hold the file open longer than PostgreSQL does. The retry bound decides when the service gives up, and giving up keeps the old file.
• The 487 evidence is still open. Raising the Windows cap rests on the canary's 24 h verdict, not on this design.

[erikdarlingdata]

Adversarial review of the design (R2)

I reviewed the posted design against dev at 73c1219e, the PostgreSQL 18 source (guc.c, main.c, win32_shmem.c on REL_18_STABLE), and the PostgreSQL and Microsoft docs. I ran nothing against a server. Line numbers without a file name are in Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs. Other file paths are relative to Darling/PerformanceMonitor.Darling.Service/ unless they start with Darling/.

Parts of the design hold up:

• PostgreSQL on Windows commits the whole shared memory segment when it starts. win32_shmem.c:220 and :286 call CreateFileMapping on the page file with plain PAGE_READWRITE. The CreateFileMapping doc says SEC_COMMIT is then assumed, and "the system must have enough committable pages to hold the entire mapping".
• The pending_restart result from the rig matches the source. write_one_nondefault_variable (guc.c:5592-5660) gives a new Windows backend the value, source file, line and source. It does not give it the pending-restart flag.
• One file, plain include, and temp-file-then-replace are the right choices.

The problems are in the failure paths, the upgrade paths and the reporting.

High

H1. A value that PostgreSQL rejects stops the store, and each start writes the same value again

• Scenario. A formula or a constant produces a value that one runtime refuses. This happened before: Managed store writes maintenance_work_mem = 2048MB, one kB over PostgreSQL 17's Windows maximum: a 17.x store on a 40 GB+ host cannot restart #3909's maintenance_work_mem = 2048MB was FATAL on PostgreSQL 17 on Windows. A store can still be on 17 after a reverted upgrade. Managed store: 66–90% of WAL is full-page images, with wal_compression off and checkpoint_timeout at the 5-minute default; together they are worth an estimated half of WAL #4246 is a likely next case, because the design checked zstd only on the 18.6 build. The service writes the file and runs pg_ctl start, and the start fails. The service restarts, derives the same value, and the start fails again.
• Recovery is worse than today. The operator reads the server log, sees darling-managed.conf line N, and corrects that line. On the next start, step 4 finds that the hash does not match. It saves the corrected file as .edited-<utc>.bak and writes the rejected value again. ALTER SYSTEM, the pin path in the file header, needs a running store. The two ways out are a line after the include in postgresql.conf or a hand edit of postgresql.auto.conf. The design names neither of them in the failure path.
• The restart signal cannot tell the two cases apart. In the source, a rejected value returns 0 from set_config_option (guc.c:3713-3716). A restart-only change also returns 0 (guc.c:3734-3747). ProcessConfigFileInternal maps every 0 to the same text, "setting could not be applied" (guc.c:574-579). The pg_file_settings doc agrees: the error field covers an "invalid value, or the setting cannot be changed after server start". So section 5 logs "needs a restart" for a value that a restart turns into a FATAL.
• Change.
  1. Before pg_ctl start, validate the new file with postgres -C <any key>. That call parses all the configuration files and fails on a bad value. The design already runs postgres -C for the segment size, so this costs nothing extra.
  2. Keep a darling-managed.conf.last-good copy after each successful start. When validation fails, start on the last-good copy and raise an alert.
  3. Do not write over a hand edit of the managed file. Keep the edit in force, report it as an override, and raise an alert.
  4. Split the error rows. A row whose key has pg_settings.context = 'postmaster' and whose value passes validation needs a restart. Report every other error row as a rejected value.
  5. Put the recovery path (a line after the include) in the start failure message.

H2. The downgrade fix in "Open risks" can delete the include line, and a downgrade without it undoes operator lines

• With the fix as written ("one comment line per legacy marker just above the include"). Today's build runs the v8 check on every start. The check fires when the last fingerprint line is missing (ShouldAppendHardwareSizing, line 1183), and the migrated file has no fingerprint line. The v8 rewrite then replaces the span that starts at the v8 marker. That span runs through every non-blank line that follows, up to the first blank line (FindHardwareSizingBlockEnd, lines 1286-1313). The replace is in ReplaceOrAppendHardwareSizingBlock, lines 1376-1405. If the marker comments, the include line and the moved operator lines have no blank line between them, the rewrite deletes all of them. That start then runs without the managed file. It has no timescaledb in shared_preload_libraries, and it gets initdb's max_connections, timezone and logging_collector.
• Without the fix. The old build finds no markers. It appends v1 to v14 after the include and after the operator lines that the migration moved below the include. Take the design's own rehearsal fixture. The v4 block's max_connections = 200 then beats the operator's max_connections = 300 on line 900, with no warning. v1's literal shared_preload_libraries = 'timescaledb' also replaces an operator's longer preload list. v13 then adds back only pg_stat_statements.
• Change. Choose one approach. The simple one is to not support a downgrade in place. Document "restore postgresql.conf.pre-4215.<utc>.bak before you install an older build". If the marker comments stay, put a blank line after each marker comment and before the include. Also add a pinned test that runs today's EnsureConfAppended on a migrated conf. The test asserts that the include line and the moved lines survive.

H3. The major upgrade path is not covered, and it drops ALTER SYSTEM pins, tonight's 8 GB included

• Scenario 1: the store does not upgrade. A major upgrade runs initdb on a new cluster. It writes the new cluster's managed conf before pg_upgrade starts it, because shared_preload_libraries must be set by then (DarlingStoreUpgrade.cs:2674-2697, context.AppendManagedConf(newDataDirectory)). That delegate must also write darling-managed.conf in the new directory. If it adds only the include line, the new cluster fails to start, because plain include of a missing file is FATAL. pg_upgrade then fails, and the upgrade reverts.
• Scenario 2: intent is lost. pg_upgrade does not carry postgresql.auto.conf or files included by postgresql.conf. The pg_upgrade doc tells you to adjust "postgresql.conf (and any files included by it), postgresql.auto.conf" in the new cluster by hand. DarlingStoreUpgrade.cs never mentions auto.conf. So at the next major bump, the production store loses its ALTER SYSTEM SET shared_buffers = '8GB' with no warning. Every operator line below the include is lost too. This happens today as well. But the design makes ALTER SYSTEM "the supported way" to pin a value, so the design must keep that promise across an upgrade.
• Scenario 3: order. The v14 heal runs before the upgrade's old-cluster start, because EnsureConfAppended runs too late for it (lines 489-491 and 2447-2451). The design does not say where the new writer runs. Suppose the v14 append stays. On a 17 store it writes a marker block after the include. The migration's "run once" test (no marker left) then fails, and the migration runs again.
• Change. Specify the upgrade path:
  1. Render the file for the old major before EnsureDataDirectoryMajorAsync. Move the v14 logic into a derivation that knows the major, and remove the v14 append.
  2. In AppendManagedConf, render the file for the new major in the new directory.
  3. After the upgrade succeeds, copy postgresql.auto.conf and the lines below the include into the new cluster, and log each one.
  4. Pin all of this with tests.

H4. The commit clamp does not check the value in force, which is the case of tonight's store

• Scenario. shared_buffers = 8GB comes from postgresql.auto.conf. The service's file says 1024MB. The design does not say which size the clamp measures, and both readings are wrong:
  • Without -c shared_buffers=<candidate>, postgres -C shared_memory_size reads auto.conf. It returns the 8 GB segment for every candidate, so the step-down loop never passes. The loop writes the 128 MB floor into the file and logs clamped-commit-headroom, and nothing real changes. Later, the operator runs ALTER SYSTEM RESET, and the store comes back at 128 MB, not 1 GB.
  • With -c, the loop checks the 1 GB candidate and passes. The start then commits 8 GB, and the clamp gave no warning.
• Why it matters. With the Windows cap at 1 GiB, the clamp binds only on a nearly full host. The one store where commit headroom is a live question is the store with the override.
• Change. Measure headroom for the value in force, not the file's value. When an override is in force, never step down the file's value. If the override does not fit, log a warning and raise an alert before the start. Record the measured shared_memory_size in the header, so the decision can be reproduced.

Medium

M1. The command line is a fourth layer above postgresql.auto.conf, and the design leaves it out

The service starts the store with -p <port> -c listen_addresses=.... An exposed store also gets the SSL settings. The options come from BuildServerRuntimeOptions (lines 4109-4123), called at line 3497 and passed to pg_ctl at line 3518. The code comment at lines 4102-4103 says -c outranks postgresql.auto.conf. The docs agree: command-line settings "override those set via postgresql.conf or ALTER SYSTEM". The file will own port and listen_addresses, so:

• Section 4's rule ("anything not sourced from darling-managed.conf becomes operator-override") flags both keys on every start of every store. Their source is command line, and sourcefile is NULL.
• pg_file_settings compares a postmaster-context file value with the running value, even when a higher-priority source set that value. The source calls this "plow ahead anyway" (guc.c:3689-3699, then 3483-3495). On an exposed store, the file's '127.0.0.1' never equals the running '127.0.0.1,<ip>'. So the restart signal reports listen_addresses as pending a restart, forever. I read this in the source and did not run it. It is in the unverified list.

Change. Add the command-line layer to the precedence model. Keep the two keys in the file, so that a manual pg_ctl start still gets them. Leave out every key whose source is command line from the override report and from the restart check.

M2. The MCP and viewer roles cannot compute the verdicts

Section 4 says one implementation feeds --check-settings, get_store_host and the Settings UI. pg_settings.sourcefile is NULL for a role that is not a superuser and does not have pg_read_all_settings (doc). The MCP host's mcp role gets NULL, and the code says not to give it more rights (Mcp/DarlingStoreMetricsReader.cs:223-230). pg_file_settings is superuser-only by default (doc). Suppose the implementation runs live under those roles. It then reports every key as an override, and it never sees a pending restart.

Change. Compute the verdicts at start as the owner and store them. The readers read the stored rows.

M3. The header records raw readings that change on every start

The header includes committed-bytes, data-volume-free-gb and the raw ram-bytes. So the file is never byte-identical, and step 3 never skips the write. The code learned this lesson twice already:

• The v12 stamp records outputs, because "the raw free-disk figure moves on every start" (lines 516-518).
• QuantizeRam exists because ullTotalPhys "is not guaranteed bit-identical across reboots" on these cloud VMs (lines 1117-1124).

The clamp adds a worse effect. It makes shared_buffers, a restart-only value, depend on the commit charge at the moment of start. A restart during a backup can then get a smaller buffer pool than a restart at boot.

Change. Log the raw readings, and keep them out of the header. Put only the derived values and the quantized inputs in the header. Lower shared_buffers only when the value in force fails the check. Do not raise it again on one passing reading.

M4. A key that the file starts to own later beats an older hand edit, with no report

Rule 3 moves operator lines below the include only for the keys that the file owns at migration time. #4246 adds wal_compression and checkpoint_timeout later. Suppose an operator put a checkpoint_timeout line in postgresql.conf above the include. That line then loses to the file, as the design's own precedence proof shows. The value in force then comes from darling-managed.conf, so section 4 reports nothing.

Change. Run rule 3 for each new key whenever the owned set grows. Make it part of every formula-version bump, with a pin.

M5. The preload list must stay a merge after the migration

v13 re-states the list in force plus pg_stat_statements. It never writes a literal, because a literal drops the operator's libraries (lines 457-461 and 3080-3094). Rule 2 carries the merged list into the file once. The design does not say how later starts derive the list. Suppose a later start derives the constant timescaledb,pg_stat_statements. A library that v13 merged in, for example auto_explain, is then dropped at the second start after Release A. Nothing reports it, because no operator line is left to beat.

Change. Render the list as the merge of the file's current list and the required libraries. Pin that the list never shrinks.

M6. The migration's failure paths and upgrade inputs are not defined

• The verification fails. The migration restores the backup. What do later starts do? If Release A removes the v1 to v14 appenders, that store stops getting the v8 and v12 re-derivations. It only logs a line on each start. Keep the old appenders as the fallback until the migration succeeds, and raise an alert, not only a log line.
• A crash between the replace and the verification. Suppose the service stops after it replaces postgresql.conf and before the verification finishes. The next start sees an include and no markers, so it never runs the verification. Write a verified stamp only after the verification passes, and run the verification again while the stamp is missing.
• An older store. A store from a build older than v9 to v14 has no service line for some keys. For those keys, rule 2 ("the value on the last service line") gives no value. Say that those keys take the fresh derivation, and that each one is a logged value change.
• Release A changes values. Release A's second start re-derives every key. So a store whose v3 or v5 shared_buffers came from a smaller host changes value. That contradicts "no value changes".
• "Zero parse errors" needs a definition. A store can have a real pending restart. One example is an adopted postmaster, which EnsureRunningAsync never restarts (lines 2509-2516). That store has error rows before and after the migration. Do not block the migration on those rows.

M7. The Linux term targets a deployment that does not exist, and it has no floor

On Linux the store is a separate container from the official TimescaleDB image. The compose file says "the bundled zero-admin store never runs here" (Darling/compose/docker-compose.yml:1-3, :20, :31). So "PostgreSQL runs as the service's child, so it shares the service's cgroup" is false for the only Linux deployment. A caller reads the service container's limit, not the store's. The formula 25% (M - 2 GiB) gives 0 at M = 2 GiB and a negative number below that. It has no cap on large hosts.

Change. Remove the Linux function from this design, or point it at the compose store (the command: flags in the compose file). If you keep it, give it a 128 MB floor and a cap. Either way, state that this design does not meet #4215's Linux request.

Low

• L1. Put -C first. postgres skips its "not as administrator" check only when -C is the first argument (main.c:175-187). In any other position, it refuses to run as an administrator (main.c:470-472). So postgres -D <data> -C shared_memory_size fails under LocalSystem or an admin domain account, and the clamp uses its fallback with no warning. Use postgres -C shared_memory_size -D <data>.
• L2. Commit readings. The MEMORYSTATUSEX doc says ullAvailPageFile is what the current process can commit. ullTotalPageFile is the smaller of the system and process limits. For system-wide values, the doc names GetPerformanceInfo (CommitLimit - CommitTotal). Use it, so that a job object on the service cannot change the result. Also, the margin allows for one maintenance_work_mem job. But autovacuum's workers and the TimescaleDB compression jobs can each use up to that much at the same time. Say that the margin is an estimate, and size it from Store host visibility: a cross-platform host profile, a --check-settings verb, and a get_store_host read, so "is this store sized right" has an answer without shell access #4214's host profile.
• L3. Edits inside a block. Each builder writes its lines in one fixed form. A block line in a form that its builder never writes is a hand edit, whatever its value. An example is 2GB where the builder writes 2048MB. This test also covers v2, v3, v5 and v7, where the "within range" test lets an edit inside the range through.
• L4. Design: size raw chunk interval, compress_after and the hourly CAGG refresh window from ingest rate; a large store keeps 1–2 days of raw uncompressed and it outgrows RAM #4211 must use the same M. Design: size raw chunk interval, compress_after and the hourly CAGG refresh window from ingest rate; a large store keeps 1–2 days of raw uncompressed and it outgrows RAM #4211's budget is "25% of host RAM on a managed store, where the sizing path knows the host". Design: size raw chunk interval, compress_after and the hourly CAGG refresh window from ingest rate; a large store keeps 1–2 days of raw uncompressed and it outgrows RAM #4211 must read the M that this design derives, and not call GlobalMemoryStatusEx itself. If it does, the two disagree once M is less than physical RAM (a container limit).
• L5. No way to opt out. Step 6 puts the include line back when someone removes it. So an operator who removes it on purpose, to set everything by hand, cannot opt out. Decide this once and document it.

Where the rig and the sources disagree

• I believe the rig about pending_restart, because the source agrees. I also believe it about the whole-segment commit, because the docs and win32_shmem.c agree.
• I disagree that "setting could not be applied" is a reliable restart signal. The source and the pg_file_settings doc show that the same text covers rejected values (H1). The source also shows it for postmaster-context keys set on the command line (M1). The rig most likely tested neither case.

Unverified

• The permanent listen_addresses pending restart on an exposed store (M1). It comes from reading guc.c, not from a rig run. To check it, start the rig with -c listen_addresses=0.0.0.0 and read pg_file_settings.
• The Windows account of each field store. The README's sc create line uses NT SERVICE\PerformanceMonitor Darling, which is not an admin. So L1 matters only on hosts that use LocalSystem or an admin account.
• Whether the bundled PostgreSQL 17 runtime has zstd. It has lz4, because v1 writes default_toast_compression = lz4 to every store (line 683).
• The commit limits and page file setup of the field stores. The design says the same.
• Whether the timescale/timescaledb image runs timescaledb-tune when it creates the compose store. That decides whether the Linux store has any memory sizing today.
• When PostgreSQL 19 reaches the bundle. H3's scenario 2 happens at that bump.

Verdict

Build with the listed changes. The core idea is sound: one file, plain include, temp-file-then-replace, and ALTER SYSTEM as the pin. Before Release A, the migration and the restart check need H1, H2, H3, M1, M2 and M6. Before Release B, the clamp needs H4 and M3. M4 must be in place before #4246 adds its keys. Remove the Linux term or point it at the compose store (M7). No section needs a redesign.

🤖 Generated with Claude Code

https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3

[erikdarlingdata]

Ruled. The design (issuecomment-5827624802) is built with every change the adversarial review (issuecomment-5827863184) lists. Below are the answers to the design's "What Erik decides" list, and the choices the review left open.

The eight decisions

1. Include form: plain include.
2. Windows cap: stays at 1 GiB in Release A. Whether to raise it waits for the canary's 24-hour verdict, and a raise ships in Release B.
3. Commit margin: 2 GiB + maintenance_work_mem, measured system-wide with GetPerformanceInfo (CommitLimit - CommitTotal, review L2). Release A only warns. Release B follows H4 and M3:
   • measure the value in force;
   • lower the file's shared_buffers only when the value in force fails the check;
   • never step down an override (warn and raise an alert instead);
   • never raise it again on one passing reading.
4. Override paths: ALTER SYSTEM, plus lines after the include. There is no operator file and no pins in the service config.
5. hash_mem_multiplier: stays at 2.0 until hash spills are shown.
6. Linux: remove the Linux function from this design (review M7). The only Linux deployment is the compose store, which is a separate container that this service doesn't manage. The PR says this design doesn't meet the issue's Linux request. Linux compose store: check and set its memory sizing (the #4215 design leaves it out) #4322 covers the compose store's sizing.
7. Two releases: yes. Release A is the file, the migration and the restart check. Release B is the commit clamp and any change to the Windows cap.
8. Managed store: 66–90% of WAL is full-page images, with wal_compression off and checkpoint_timeout at the 5-minute default; together they are worth an estimated half of WAL #4246: the file only gives its keys a home. wal_compression is lz4, as ruled on Managed store: 66–90% of WAL is full-page images, with wal_compression off and checkpoint_timeout at the 5-minute default; together they are worth an estimated half of WAL #4246. checkpoint_timeout stays unchanged until the use1 experiment reports. When those keys join the file, M4's rule runs for them.

The review's open choices

• H2 (downgrade): an in-place downgrade is not supported. The docs and the release notes say to restore postgresql.conf.pre-4215.<utc>.bak before installing an older build. No marker comments are left in the file.
• L5 (opting out): there is none. A removed include line comes back on the next start, and the log says so. An operator who wants a value pinned uses ALTER SYSTEM.
• H1, H3, H4, M1-M6, L1-L4: each change as the review wrote it. Release A needs H1, H2, H3, M1, M2, M4 (as a pinned rule for every formula bump), M5 and M6. Release B needs H4 and M3.
• PostgreSQL on Windows: pending_restart reads false on connections opened after the reload, so we can miss a pending restart #4251: Release A's restart check closes it. Error rows are split by pg_settings.context after postgres -C validation (H1 item 4), and keys whose source is command line are left out (M1).

How it gets built

This is too large for one lane. Release A goes in two lanes, one after the other, because both edit DarlingManagedPostgres.cs:

• A1: the file writer. That covers rendering, the atomic replace, postgres -C validation with -C first, the last-good fallback, never overwriting a hand edit, the stored verdicts, the restart check, the preload merge and the command-line layer.
• A2: the v1-v14 migration (with the verified stamp and the fallback appenders), the major-upgrade path (H3), and the downgrade docs.

Each PR gets a review round before it leaves draft, because it rewrites postgresql.conf on every store. Release B follows after the canary verdict. The work is ranked as tier 4, below the field-reported items in the queue.

🤖 Generated with Claude Code

https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3