Summary
On a 2-node brigade cluster the Erlang mesh forms correctly (/status reports partition.size: 2, in_quorum: true), but each node's Mnesia store is an isolated island. lib/brigade/store/mnesia.ex creates both tables with ram_copies: [node()] and there is no cross-node join anywhere in the tree (no add_table_copy, extra_db_nodes, change_config, or schema merge on nodeup). The moduledoc says the tables are "replicated across the mesh", but the code makes them node-local.
# lib/brigade/store/mnesia.ex — setup!/0
create_table(@vms, attributes: [...], index: [...], ram_copies: [node()])
create_table(@hosts, attributes: [...], ram_copies: [node()])
Brigade.HostRegistry.SelfRegister registers only the local node's host into the local table, and Brigade.Scheduler's handle_info({:nodeup, up}, ...) is a documented no-op. (Cross-node host aggregation appears to be intended for M3 per the moduledocs in host_registry.ex / self_register.ex.)
Impact — two observable failures on 2 nodes
1. Host aggregation never happens. /status reports schedulable_hosts: 1 and a hosts array containing only the scheduler node's own host, despite partition.size: 2 / in_quorum: true. The scheduler packs every VM onto the one visible host and refuses the (N)th with RESOURCE_EXHAUSTED: "no host has capacity"; the other host runs zero VMs.
"hosts":[{"id":"brigade@10.106.32.2","labels":{"node":"host0"}, ...}],
"partition":{"size":2,"in_quorum":true}, "schedulable_hosts":1(GRPC.RPCError) no host has capacity for 1 vcpu / 1024 MB # Nth VM; host0 full, host1 invisible
2. VM records are invisible from the non-scheduler node. The scheduler singleton writes VM records into its node's Mnesia. A client dialing the other node's north edge gets NOT_FOUND from GetMicroVM for a VM that was just created successfully:
(GRPC.RPCError) microvm 01KXQQBFE02W6VYWBZP4R8NB7S not found # GetMicroVM from the non-scheduler node
Because which node wins the scheduler-singleton election is nondeterministic, basic CRUD passes or fails depending on whether the client happens to dial the scheduler's node.
Suggested direction
Replicate the tables across the mesh — coordinated schema merge (:mnesia.change_config(:extra_db_nodes, Node.list())) plus table copies added on join, or land the M3 shared-registry provider so every node's scheduler sees every host and every VM record.
Environment
Summary
On a 2-node brigade cluster the Erlang mesh forms correctly (
/statusreportspartition.size: 2,in_quorum: true), but each node's Mnesia store is an isolated island.lib/brigade/store/mnesia.excreates both tables withram_copies: [node()]and there is no cross-node join anywhere in the tree (noadd_table_copy,extra_db_nodes,change_config, or schema merge onnodeup). The moduledoc says the tables are "replicated across the mesh", but the code makes them node-local.Brigade.HostRegistry.SelfRegisterregisters only the local node's host into the local table, andBrigade.Scheduler'shandle_info({:nodeup, up}, ...)is a documented no-op. (Cross-node host aggregation appears to be intended for M3 per the moduledocs inhost_registry.ex/self_register.ex.)Impact — two observable failures on 2 nodes
1. Host aggregation never happens.
/statusreportsschedulable_hosts: 1and ahostsarray containing only the scheduler node's own host, despitepartition.size: 2/in_quorum: true. The scheduler packs every VM onto the one visible host and refuses the (N)th withRESOURCE_EXHAUSTED: "no host has capacity"; the other host runs zero VMs.2. VM records are invisible from the non-scheduler node. The scheduler singleton writes VM records into its node's Mnesia. A client dialing the other node's north edge gets
NOT_FOUNDfromGetMicroVMfor a VM that was just created successfully:Because which node wins the scheduler-singleton election is nondeterministic, basic CRUD passes or fails depending on whether the client happens to dial the scheduler's node.
Suggested direction
Replicate the tables across the mesh — coordinated schema merge (
:mnesia.change_config(:extra_db_nodes, Node.list())) plus table copies added on join, or land the M3 shared-registry provider so every node's scheduler sees every host and every VM record.Environment
main@5a08280(2026-07-16, "Merge pull request feat(release): publish OTP tarball + GHCR image on semver tag #12 from liquidmetal-dev/feat/release-workflow")v0.9.0(a7c62f3), Erlang/OTP 27, Elixir 1.18.4