Database affinity is per agent family, not per database: 73% of shard databases have durability and projections on different nodes

[erdtsieck]

Rewritten 2026-08-03 after measuring this on production for a full hour. The original report claimed that
nodes joining a running cluster get only the durability family and never take projection work. That is
wrong and I am retracting it — see "Retracted: the imbalance does converge" below. What survives, and what
this issue is now about, is that database affinity is applied per agent family rather than per database, so
two nodes end up connecting to the same shard database. That part reproduced on production and held steady
over an hour.

Agent assignment honours database affinity within the wolverinedb:// durability family and within the
event-subscriptions:// family, but not between them. The result is that for most shard databases one node
owns the durability agent while a different node owns that database's projection agents.

What we measured

Production, 5 pods, one Marten store over 512 tenant shard databases with Events.UseTenantPartitionedEvents,
~8,700 agents (6 async projections × ~1,385 tenants, plus ~460 durability agents). Measured on a converged
cluster, then again every five minutes for an hour:

	databases
durability agent on the same node as that database's projections	71
on a different node	375
only one family assigned	65

That is 73% split, and it did not move over the hour (it was 397/81 an hour earlier, so it drifts a little with
rebalancing but the ratio holds). The pool count derived from it stayed flat between 902 and 945.

What it costs, measured rather than estimated

Npgsql keeps one connection pool per (connection string, database) per process, so a split database means two
pools against one shard instead of one. Joining pg_stat_activity on the shard server to the assignment table:

connections held by	count
the projection owner (the node doing the work)	1,483
the durability owner	434
nodes owning neither family (ordinary API traffic for that tenant)	1,025

The split costs 425 connections — the connections a durability owner holds against databases it would
otherwise never open. That is 14% of the total, not a doubling: a durability pool holds ~1.1 connections on
average, against 3 for a busy projection pool. I want to be precise about that because "two pools instead of
one" sounds like twice the connections and it is not.

14% still mattered. Earlier the same day this server sat at 2,393 of its 2,400 max_connections for half an
hour during a projection version bump, throwing 7,500+ 53300 remaining connection slots are reserved… in
fifteen minutes, briefly taking two of five pods out of the load balancer, and dropping assigned agents from
8,008 to 5,554 because an agent that lost its slot could not get a connection to restart. 425 connections was
almost exactly the margin we did not have. (We have since raised the limit to 3,600, so this is no longer
urgent for us — but the waste is structural and the next 512-shard cluster will meet the same wall.)

Worth adding for calibration: only 140 of ~3,000 connections are active at any instant. This is not a
concurrency problem, it is a slot-scarcity problem — which is why halving the pool count is the useful lever
and lowering the per-pool cap is not (that would throttle exactly the 660 pools that are doing the replay).

What we would expect

That a shard database's durability agent is assigned to whichever node owns that database's projection agents,
so one node holds one pool against it. Concretely: make affinity a property of the database across families
rather than a property of each family independently.

Retracted: the imbalance does converge

The original version of this issue said the new nodes stayed at ~107 agents thirty-five minutes after a
scale-out and that the split "had not moved". Measured over a longer window on production, that is a transient
of the catch-up, not a steady state. The last pod to join a rolling restart:

07:37       6 agents
08:12     167
08:39     936
08:40   1,230
08:55   1,893

Eighty minutes to go from 6 to above its fair share. Over the following hour the lowest node stayed between 88%
and 96% of an even share, and the final distribution over 8,716 agents was:

node	agents	% of even share	databases
507	1,665	96%	179
508	1,732	99%	181
509	1,787	103%	154
510	1,749	100%	188
511 (last to join)	1,783	102%	200

The node that started with six agents ended up holding the most databases of the five. So placement is not
starving new nodes; it is slow to confirm them.

The slowness has a known cause and it is not in Wolverine: while a bumped projection's shards are warming up,
the replay runs inside the agent start path, so an agent only counts as assigned once its replay finishes
(JasperFx/jasperfx#610). During that window the assignment table advances in bursts and looks exactly like a
stuck cluster. That is what I mistook for permanent starvation.

The one observation from the original report I still find odd, offered without a conclusion: over a six-minute
window during the ramp we counted ~45,000 ReassignAgent decisions in the AssignmentChanged descriptions,
essentially all aimed at the nodes that already held everything and about twenty at each new node. Given that
the outcome converges, this may simply be the leader re-deciding against a fleet whose starts have not
confirmed yet — in which case it is a symptom of #610 too, and only worth looking at if it is cheap to make
the evaluation aware of pending starts.

Environment

Wolverine 6.24.4 / Marten 9.22.2 / JasperFx 2.37.2, .NET 10, 5 pods, DurabilityMode.Balanced with
UseWolverineManagedEventSubscriptionDistribution, database control queue, one Marten store over 512 tenant
databases with Events.UseTenantPartitionedEvents. Production, and a canary on a restored copy of it, so we
can reproduce at full scale — happy to run whatever measurement would be useful.

Related: JasperFx/jasperfx#610 (warm-up inside the start path), #3753 (the outcome-level
report).

[erdtsieck]

It is not scale-out specific — a plain rolling deploy does it too

I framed this as "scaling out mid-run". That was too narrow. We rolled 3.2.6 to production this morning — five replicas throughout, no manual scaling, an ordinary maxSurge: 0 rolling replacement — and got the identical split. Five minutes after the rollout finished:

node	joined	event-subscription agents	durability agents
501	07:27:08	1,095	0
502	07:28:02	930	0
503	07:28:41	1,030	0
504	07:29:22	949	0
505	07:30:18	0	103

The four nodes that came up first share all the projection work and hold no durability agents; the one that came up last holds every durability agent and no projection work at all. Four minutes later the first four had climbed past 1,000 each while 505 was still at exactly 103.

So the trigger is not "someone scaled the cluster" — it is the last node to join, and on a rolling deploy that is simply whichever pod Kubernetes replaces last. Every deploy produces one.

This also answers the question I asked at the bottom of the original report. Coming up at five replicas from the start does not avoid it, because a rolling replacement is still a sequence of joins, not a simultaneous start.

What that changes

• It is not an edge case an operator can avoid by never scaling out. It happens on every deploy of a cluster this shape.
• The affected node is idle for projection purposes, so on a five-node cluster you effectively deploy onto four.
• On the earlier canary run the same node stayed at ~103 for over half an hour, so it does not resolve on its own within any window we have watched.

Everything else in the original report stands, including the correction that this is currently not costing us wall-clock — the shard database is the ceiling, not the node count.

For reference, the production deploy itself went well: 1037 migrations in 12 minutes, all five pods rolled without a restart, 17 minutes end to end, and zero AgentStopped throughout — the churn from #3750 is genuinely gone in 6.24.4.

[erdtsieck]

Correcting my previous comment — on a rolling deploy it does resolve

I posted that comment fifteen minutes into the production rollout and read a snapshot as a steady state. It was not one. The last node kept climbing:

time	node 505 projection agents
07:31	0
07:46	88
07:47	314
07:48	433

while the other four sat still at 1,150–1,350. So on production the last node to join is served — it just starts roughly fifteen minutes after the others and then catches up on its own. Calling that "no projection work" was wrong, and I should not have generalised from a single reading.

What the two runs actually show

The distinction I collapsed is worth keeping:

• Canary, scaled 3 → 5 mid-run: the two new nodes sat at exactly 103 (durability only, zero projection agents) for over half an hour with no movement, while the incumbents grew past an even share. That is the behaviour in the original report and it did not self-correct in any window we watched.
• Production, ordinary rolling deploy at 5 replicas: the last node was similarly empty for ~15 minutes, then filled in steadily toward a fair share.

So the original framing was closer to right than my correction: scaling out into a cluster that has already placed its agents is the case that does not recover; a rolling deploy has a delay but resolves. I have reverted the title accordingly.

What still seems worth looking at, in descending order of confidence:

1. The durability/projection split itself. In both runs the newest node holds every durability agent and the incumbents hold none — 103/0 versus 0/1,300. That is a strange partition regardless of whether the projection side later balances, and it is the same 97%-of-databases split-affinity finding from the original report.
2. The delay. Fifteen minutes before a healthy node receives any projection work is a long time on a deploy you are watching, and it is indistinguishable from the stuck case until it resolves.
3. The scale-out case, which is the one that appears not to recover at all.

Apologies for the churn on this issue — the underlying measurements are solid, my reading of one of them was not.