Can a node take event-subscription agents but not durability agents? (warming a new projection version before it serves traffic)

[erdtsieck]

Context

We run Wolverine-managed event subscription distribution
(UseWolverineManagedEventSubscriptionDistribution = true) over a Marten store with 512 shard
databases, ~854 tenants and Events.UseTenantPartitionedEvents = true. That works out to roughly:

• ~5 100 projection agents (tenants × 6 async projections), and
• 512 durability agents (one per shard database — each shard has its own
  wolverine_incoming_envelopes),

for about 5 650 distributed agents. Every API pod is a Wolverine node and carries a share of them,
assigned with database affinity. Background from our side that you have already helped with:
#3698,
#3701,
#3733.

What we are trying to do

Marten writes each projection version to its own tables. When we bump projection.Version, the new
version's tables start empty and the daemon builds them from event 0, per tenant, per shard. At our
size that takes hours, and the code that queries them has no knowledge of the previous version's
tables — so the moment a pod running the new version serves traffic, it serves empty read models.

So we want to run a separate set of pods on the new image, off the load balancer, let the daemon
build the new projection version there, and only roll the serving fleet once those read models have
caught up. The old pods keep serving the old version's tables throughout, which works naturally
because the projection version is part of the agent identity
(provided-cares:V23:All:{tenant}) — old and new pods advertise disjoint projection agents.

Those warming pods must be full Wolverine nodes; that is the only way the leader assigns them the new
version's projection agents. But being a node also brings message handling, which is exactly what they
must not do while their read models are still cold:

	Can we switch it off?
External listeners (AWS SQS here)	Yes — IEndpointCollection.StartListenerAsync / StopListenerAsync
Local durable queue listeners	Yes, same API
Projection side effects	Yes — Marten's GateSideEffectsBehindPriorVersion
Durability agents	This is our question

Durability agents are assigned per shard database and are version-independent, so the leader hands
them to warming pods too. They recover and execute persisted incoming envelopes, which means a warming
pod runs real handlers against half-built read models. A durability agent is not a listener, so
StopListenerAsync does nothing about it.

The actual question

DurabilitySettings.DurabilityAgentEnabled looks like it might be exactly the lever, but its doc
comment only says "Should the message durability agent be enabled during execution. The default is
true." We cannot tell from that whether, under Wolverine-managed distribution, it also:

• keeps the node from advertising durability agents, so the leader never assigns it any; or
• only disables the local durability loop while the node still gets assigned durability agents that
  then sit inert — which would be worse than not setting it, because those shard databases would have
  no working durability agent at all.

If it does the first, that is our answer and this issue is just a documentation request. If it
does the second, we would like a supported way to express "this node takes event:// agents but not
durability ones" — a node-level role, or a capability predicate over agent URI schemes evaluated per
node. We note IAgentFamily.SupportedAgentsAsync() is already documented as "All supported agent
uris by this node instance", so the mechanism seems to exist; we just cannot find the knob.

What we ruled out

• NodeAgentController.DisableAgentsAsync / WolverineRuntime.DisableAgentsAsync — documented
  "STRICTLY FOR TESTING".
• DisableAllWolverineMessagePersistence — disables persistence bootstrapping entirely, built for
  OpenAPI generation; we still need the message store for the distribution itself.
• DurabilityMode.Solo / Serverless / MediatorOnly — Balanced is what enables the leader
  election and control queue the distribution needs, and the others disable persistence or start no
  agents at all.
• Pointing the warming pods at a separate Wolverine main database. This is attractive for a
  different reason — it gives the warming pods their own node registry, leader and control queue, so
  warming pods joining and leaving no longer trigger a full rebalance of the serving cluster's ~5 650
  agents. But it does not answer this question: both clusters are built over the same sharded Marten
  store, so both would assign durability agents against the same 512 per-shard
  wolverine_incoming_envelopes tables. That looks like it turns one problem into two coordinators
  competing over the same durability targets. If that reading is wrong we would be glad to hear it.

Why this might be generally useful

Any system with a large read-model rebuild has this shape: you want nodes that build projections
without taking on message handling, then hand the caught-up state to the serving fleet. It is also a
different scaling axis — sometimes you want more projection capacity without more message-handling
capacity.

Environment

• Wolverine 6.24.2, Marten 9.22.0, JasperFx 2.36.3, .NET 10
• UseWolverineManagedEventSubscriptionDistribution = true, DurabilityMode.Balanced
• Marten sharded tenancy: 512 databases, Events.TenancyStyle = Conjoined,
  Events.UseTenantPartitionedEvents = true
• Control queue on the database (AWS SQS provides no per-node control queue)

[erdtsieck]

Sharpening the ask, because the DurabilityAgentEnabled question above is really just the first thing we tripped over. What we actually want is one switch:

a node that does event-subscription (projection) distribution and no messaging at all.

Concretely, such a node would:

• start no listeners — external transports and local queues alike;
• advertise no durability agents, so the leader never assigns it any;
• run no scheduled/recurring message work — no outbox recovery, no scheduled-message dispatch, no DLQ sweeps;
• still register as a node, take part in leader election, and use the control queue;
• still advertise and run event-subscription agents normally.

So: full cluster member for projections, inert for messages. Something like

opts.Durability.NodeRole = NodeRole.ProjectionDistributionOnly;

Why a supported switch beats composing it ourselves

We can get part of the way today, and that is exactly the problem — the parts do not add up to a guarantee:

• we skip our ListenToSqsQueue(...) registrations when a config flag says "warm-up node", which works but is our code, not a supported mode;
• we guess at DurabilityAgentEnabled, per the question above;
• we hand-disable our own BackgroundServices one by one.

Three places that can drift, none of which we can prove complete — and every new listener or hosted service someone adds is a chance to forget one. Worse, the failure is silent: a warm-up node quietly handling one message against a half-built read model looks exactly like a healthy node until the data is wrong. A single supported switch is also something you can test, which we cannot do for our hand-rolled version.

Why it is not just our problem

Any system with a read model large enough that rebuilding it takes hours has this shape: you want nodes that build projections without taking on message handling, then hand the caught-up state over to the serving fleet. It is also a genuinely different scaling axis — "I need more projection throughput" and "I need more message throughput" are separate questions, and today the only way to answer the first is to also answer the second.

We are happy to test any shape of this against our 512-database / ~5 650-agent deployment, which seems to be a reasonable stress case for it.

[erdtsieck]

Took a first cut at this in #3747, to make the discussion concrete rather than abstract.

Also answers my own opening question, from reading NodeAgentController:

if (runtime.Options.Durability.DurabilityAgentEnabled)
{
    _agentFamilies[_runtime.Stores.Scheme] = _runtime.Stores;
}

So DurabilityAgentEnabled = false does do the first thing I hoped — the family is never registered, so the node never advertises those agents and the leader cannot assign it any. Good.

The catch is the other side of the same line: because EvaluateAssignmentsAsync iterates the registered families, a node with it off also cannot assign durability agents while it is the leader. That is pre-existing and not something the PR introduces, but it is why I left leader eligibility alone there and flagged it for you instead of guessing.

[erdtsieck]

Closing this — we do not need the node-level flag any more, and the reason is worth recording because it is not what either of us expected.

The whole premise was that the pods building a new projection version have to be full Wolverine nodes to be assigned that version's event-subscription agents. They do not have to be anything: we now run Marten's own daemon directly from a CLI command in an indexed Kubernetes Job, against the shard databases each pod owns, with the host built but never started. No cluster membership, so no leader election, no durability agents over half-built read models, no listeners, and no doubling of an assignment set that is already ~5 650 agents. It exits 0 per pod when every async projection version it owns is within its lag threshold, and the ordinary deploy then rolls onto warm tables.

So the flag would have solved a problem we created by insisting the warmer be a node.

Two things from this thread outlived it. Your analysis in #3747 — that capabilities only ever come from static families, that DistributeEvenly and RunOnLeader never consult them, and that a leader which does not register a family stops assigning it cluster-wide — is what convinced us not to go near DurabilityAgentEnabled, and we deleted the config knob that was waiting on this issue. And it is what sent me reading DistributeByGroupAffinity, which is where #3792 came from.

That fix does unblock the serving variant, if we ever want green to take traffic instead of just warming: a second fleet now gets the bumped version's agents where before the leader handed a shard database's whole group to the incumbent pod. We have a regression test on our side pinned to it. But for the plain case — get the replay out of the deploy path — the Job is cheaper in every dimension, so that is what we shipped.