Darling: a runtime update never clears the only runtime that opens the store, and an unfinished update says how to recover

[erikdarlingdata]

Follow-up to #4934.

What was wrong

A runtime update moves the live runtime aside into pg-runtime-prev, extracts the new one, and (on a major change) runs pg_upgrade. Until the new runtime is known to open the store, the copy in pg-runtime-prev is the only one that does. Nothing recorded that. A failed extract whose revert met a lock could leave a partial pgsql that already held bin\pg_ctl.exe. The next start then treated it as a normal runtime: if that partial pg_ctl answered with the store's major, the update cleared pg-runtime-prev (deleting the only good runtime) and re-extracted, and a second failed extract left no runtime that opens the store.

Separately, the restore's TimescaleDB check abstained when the store had no TimescaleDB record. Every 3.x store without a record is on TimescaleDB 2.28.1, so a rescued runtime without 2.28.1 could be put back in front of a store it cannot open.

The fix

A rescue marker. pg-runtime-prev\rescue-in-progress exists exactly while pg-runtime-prev holds the only runtime known to open the store. It holds the hash of the package being installed.

• Written after the last update's rescued copy is cleared and BEFORE the live runtime is moved aside, so no start can find a rescued runtime without it. If it cannot be written, nothing has moved and the update is deferred. A crash between the write and the move leaves a marker over an empty folder, which the next start removes.
• Removed wherever pgsql is known to open the store again: after a same-major swap's stamp once the new runtime answers with the store's major; when a major swap's in-place upgrade commits; when RevertRuntime succeeds; when the failed-extract revert has put the old runtime back; after the start-time restore has put the rescued runtime back; and at the "stamp names the package" early return when the live runtime opens the store.
• While the marker exists, nothing clears pg-runtime-prev unless that folder provably cannot open the store, or the update that wrote the marker provably finished. The marker is stale only when: the folder has no pg_ctl.exe; there is no store (PG_VERSION absent); the rescued runtime and the store both report a PostgreSQL major and they differ; the rescued runtime lacks the store's TimescaleDB; or the marker's package equals the runtime stamp (the stamp is written only after a good extract, so that update finished). A stale marker is removed, with the reason logged, and the update proceeds. Anything else defers, including a version probe that timed out or gave no answer, so a good rescued runtime behind a failed probe is never deleted. The deferral repeats on every start and names the marker file to delete; it is logged at Error when the live runtime cannot answer with the store's major, and at Warning otherwise.
• If the store then fails to start while the marker exists (DarlingManagedPostgres.InterruptedRuntimeUpdateHint), the start failure is logged at Error and its message adds: "An earlier runtime update did not finish; the runtime that last opened the store is at \pgsql. Delete to let the next start clear it and re-extract.", with the full paths. It reads the marker's existence only and changes no state.
• Under the marker, the start-time restore fires even when pgsql\bin\pg_ctl.exe is present (that runtime is no proof of a finished update). A store with no stamp file, or a stamp cut short to zero length, still restores; the swap then re-runs in the same start and writes the stamp. The running-server, same-major and TimescaleDB checks still apply; a stamp that exists but cannot be read still refuses; and a stamp that names the package (a major swap waiting for its data upgrade) still refuses so that upgrade resumes. A marker that outlived a finished update counts as no marker here too.

TimescaleDB with no record. Outside the marker, the restore's check takes a store with no TimescaleDB record to be on 2.28.1, the version every 3.2-3.8 release shipped; a store with no extension at all therefore keeps the first-run extract. Under the marker it abstains, because the marker proves the rescued runtime was this store's live runtime moments before. A store whose record lists no versions still does not block.

What it costs

• Under the marker, the restore and the update each probe a runtime's version, so a start can spend up to two probes of up to about five minutes each on a hung binary.
• One rare chain still needs an operator: a crash mid-extract after pg_ctl.exe landed, then a rescued runtime that stops answering its version probe for good. Every start then defers and the store fails to start; the start-failure Error above names the file to delete.

Pins

In DarlingStoreUpgradeTests, using the existing seams, so they run on every OS:

• the marker exists when the rescue move runs; a marker that cannot be written moves nothing;
• under the marker the update never clears the rescued runtime; a rescued runtime whose probe gives no answer is kept and the deferral names the marker; a deferral whose live runtime cannot open the store is an Error;
• a stale marker is removed and the swap proceeds: over an empty folder, when the rescued runtime lacks the store's TimescaleDB, and when the update that wrote it finished; a stale marker does not count as a marker in the restore;
• a failed extract keeps the marker while the runtime is aside and removes it after the restore; a restore that stays locked keeps it; a re-extract that fails again leaves the good runtime;
• a major swap keeps the marker until the upgrade commits (with a source pin on the delete); a same-major swap leaves none; a successful RevertRuntime removes it;
• under the marker: a partial pgsql with pg_ctl.exe gets the rescued runtime back; no stamp, or an empty main stamp, still restores, and an empty stamp is restored and then swapped in the same start; an unreadable stamp, a running server and a stamp naming the package each move nothing; no TimescaleDB record with a rescued runtime lacking 2.28.1 still restores;
• without the marker: a pgsql with pg_ctl.exe moves nothing; no record and a rescued runtime without 2.28.1 moves nothing, with 2.28.1 it restores; the 3.8.0-to-3.9.0 legacy-stamp case still restores;
• in DarlingManagedPostgresTests: the start-failure hint names the marker and the rescued runtime when the marker exists and is null without it; a source pin checks the start-failure path adds it and logs at Error before it throws.

The new pins that change behaviour fail on the code before this change and pass with it.

CHANGELOG

None: this extends #4934's Fixed entry.