Add a node failure policy: retries, and what happens when retrying doesn't help

[Todd J. Green (tjgreen42)]

Implements the node failure policy from docs/spec-failure-policy.md (issue #155).

The problem

Nothing is retried today, and one failed node fails the whole instance. Any workflow that runs long enough hits a deadlock, a lock timeout, or a dropped connection. A background compactor meant to run indefinitely is marked failed on the first bad night, and nothing runs again until a human notices.

What this adds

Three new df.start() arguments:

Argument	Default	Meaning
max_attempts	1	Total attempts for a failing node, including the first
max_backoff	'16 seconds'	Upper bound on the wait between attempts
on_failure	'fail'	What to do once the attempts are spent

The defaults are the pre-0.2.7 behaviour, so the feature is entirely opt-in and upgrading changes nothing. Silently converting every existing workflow's fail-fast semantics into retry-and-continue is not a change a caller should discover in production. This is also why no existing E2E test needed editing: the five that assert a node failure still observe one.

A failing df.sql(), df.http(), or df.http_multipart() node is retried with exponential backoff (1s, doubling, capped at max_backoff). The wait is a durable timer, so it holds no connection and survives a restart.

Once the attempts are spent, on_failure => 'continue' abandons the rest of the current loop iteration and starts the next one; 'fail' fails the instance. Outside a loop there is no next iteration, so both settings fail the instance — 'continue' is a statement about recurring work.

-- A compactor that survives a bad batch and picks up the next tick.
SELECT df.start(
    df.loop('CALL compact_next_partition()' ~> df.wait_for_schedule('*/5 * * * *')),
    'compactor',
    max_attempts => 5,
    on_failure   => 'continue'
);

-- A one-shot migration that must not be retried (the default).
SELECT df.start('CALL migrate_tenant(42)', 'migrate-42');

What is not retried

An error that is a property of the statement rather than of the moment is not retried at all, however high max_attempts is, because a retry reproduces it byte for byte: SQLSTATE classes 42 (syntax or access rule violation), 23 (integrity constraint), 28 (invalid authorization), 3D and 3F. Everything else is retried, including every df.http() error, which carries no SQLSTATE. Class 22 (data exception, e.g. division by zero) is retried deliberately — it describes the data, which another node can change between attempts.

duroxide's schedule_activity_with_retry retries every error with no predicate hook, so the retry loop is hand-rolled. It emits the identical sequence of durable operations, which is what preserves replay compatibility.

Graph-level errors (malformed graph, unknown node type, failure to start a sub-orchestration) are not transient and still fail immediately.

df.instance_activity()

Also adds a way to tell a working workflow from a wedged one:

SELECT * FROM df.instance_activity('10 minutes') ORDER BY idle_for_seconds DESC;

status alone reads 'running' for a healthy eternal loop, one blocked on a signal that never arrives, and one retrying a broken node — and 'continue' makes that last case reachable. This reports every non-terminal instance with its last node transition, how long it has been quiet, its running and failed node counts, and its most recent error.

It is a LANGUAGE SQL function rather than a view because view-level RLS pass-through needs security_invoker, which is PostgreSQL 15 and this extension supports 13; as an ordinary SECURITY INVOKER function the existing policies on df.instances and df.nodes filter it to the caller's own rows.

This matters because the PR also removes the 100,000-iteration df.loop() cap. That cap was never a meaningful storage bound — a large carried result exhausts storage long before the count trips — and it gave a workflow meant to run indefinitely an arbitrary expiry date. Removing it means the way to spot a loop that is running without making progress has to be something better than waiting 27 hours for it to die.

Behaviour change

None by default. The one caveat applies only when you opt in: under 'continue' the loop's while condition is deliberately skipped along with the rest of the iteration, because it usually reads named results the abandoned iteration never produced. The consequence, documented in the user guide: a while loop whose body always fails never terminates. df.instance_activity() is how you see that, and df.cancel() is how you stop it.

Replay and upgrade

• FunctionInput.retry and SubtreeInput.retry are not serialized when they hold the legacy policy. A serde(default) alone is not sufficient and the first draft of this PR got that wrong: a default governs deserialization only, the field is still written, and duroxide matches a StartSubOrchestration against history on name and input equality. A 0.2.7 parent emitting a retry key would have produced an envelope unequal to the one a 0.2.6 parent recorded, failing every in-flight JOIN branch, RACE branch, and non-root loop child with a nondeterminism error. Same break class as v0.2.4 → v0.2.5. docs/upgrade-testing.md said the opposite of the truth here and is corrected.
• The retry loop schedules the first attempt with the same durable operation the previous binary recorded, so in-flight histories replay unchanged.
• sql/pg_durable--0.2.6--0.2.7.sql drops the four-argument df.start and creates the seven-argument one against a new symbol (start_v3_wrapper). Both cannot coexist — a four-argument call would match both. The four-argument Rust start_v2() stays in the binary as #[pg_extern(sql = false)], so un-upgraded schemas keep resolving to it (Scenario B1).
• transaction_mode => 'new' forwards the policy over the loopback session only when one was supplied; otherwise it issues the original three-positional-argument df.start(), which resolves on every shipped schema.

Testing

Built test-first, one commit per task.

• Unit: 313 passing — argument validation, backoff sequence derivation against duroxide's delay_for_attempt, SQLSTATE classification (permanent classes, transient classes, no marker, malformed marker), legacy deserialization for both input types, non-serialization of a legacy policy in both directions, and policy round-trip through continue_as_new.
• E2E: 56/56. tests/e2e/sql/68_failure_policy.sql — transient recovery, 'continue' in a loop, no-enclosing-loop, 'fail' with one attempt, defaults are legacy, and argument validation. Attempts are counted with a sequence, since nextval() survives the rollback of the failed attempt that produced it. tests/e2e/sql/69_instance_activity.sql — live instance, idle threshold, a loop wedged under 'continue' surfacing its error while still 'running', terminal exclusion, and RLS isolation.
• Upgrade: 86/86 (Scenario A schema comparison, B1 against 0.2.2–0.2.6 schemas, B2 data compatibility).

Docs: USER_GUIDE.md, docs/api-reference.md, docs/upgrade-testing.md, docs/spec-failure-policy.md, CHANGELOG.md.