ansible-devnet: 8-subnet homogeneous layout and full leanpoint upstreams

README.md

@@ -86,14 +86,14 @@ Grafana is started with the two pre-provisioned dashboards from [leanMetrics](ht
 ### Aggregator Selection
 
 ```sh
-# Let the system randomly select an aggregator (default behavior)
+# Default: keep isAggregator flags from validator-config.yaml
 NETWORK_DIR=local-devnet ./spin-node.sh --node all --generateGenesis
 
-# Manually specify which node should be the aggregator
-NETWORK_DIR=local-devnet ./spin-node.sh --node all --generateGenesis --aggregator zeam_0
+# Randomly select one aggregator per subnet (unique by client type when possible)
+NETWORK_DIR=local-devnet ./spin-node.sh --node all --generateGenesis --randomize
 
-# The aggregator selection is applied automatically and the isAggregator flag
-# is updated in validator-config.yaml before nodes are started
+# Set one subnet's aggregator without changing other subnets
+NETWORK_DIR=local-devnet ./spin-node.sh --node all --generateGenesis --aggregator zeam_0
 ```
 
 ### Leanpoint deployment
@@ -107,7 +107,7 @@ After validator nodes are spun up, leanpoint is deployed so it can monitor them.
 1. `convert-validator-config.py` reads `validator-config.yaml` and generates `upstreams.json` (validator URLs for health checks).
 2. `sync-leanpoint-upstreams.sh` either deploys leanpoint locally (local devnet) or syncs to the tooling server and recreates the remote container (Ansible).
 
-**Remote defaults:** Tooling server `46.225.10.32`, user `root`, remote path `/etc/leanpoint/upstreams.json`, container name `leanpoint`, published port **5555→5555** (`LEANPOINT_HOST_PORT`, default **5555**). Override with env vars (see script header in `sync-leanpoint-upstreams.sh`).
+**Remote defaults:** Tooling server `46.225.10.32`, user `root`, remote path `/etc/leanpoint/upstreams.json`, container name `leanpoint`, published port **5555→5555** (`LEANPOINT_HOST_PORT`, default **5555**). **`LEANPOINT_IMAGE`** defaults to **`0xpartha/leanpoint:latest`** (do not use **`blockblaz/leanpoint:latest`** on the shared tooling VM unless that image is verified on the host CPU). Override with env vars (see script header in `sync-leanpoint-upstreams.sh`).
 
 **SSH key for remote sync:** When using Ansible deployment, the tooling server may require a specific SSH key. Pass `--sshKey ~/.ssh/id_ed25519_github` (or `--private-key`) so the sync can succeed.
 
@@ -217,12 +217,11 @@ Every Ansible deployment automatically deploys an observability stack alongside
     - On Ctrl+C cleanup, the metrics stack is stopped automatically
 
     Note: Client metrics endpoints are always enabled regardless of this flag.
-12. `--aggregator` specifies which node should act as the aggregator (1 aggregator per subnet).
-   - If not provided, one node will be randomly selected as the aggregator
-   - If provided, the specified node will be set as the aggregator
-   - The aggregator selection updates the `isAggregator` flag in `validator-config.yaml`
-   - Example: `--aggregator zeam_0` to make zeam_0 the aggregator
-   - Example: Without flag, a random node will be selected automatically
+12. Aggregator flags (1 aggregator per subnet; `isAggregator` in `validator-config.yaml`):
+   - **Default:** existing `isAggregator` assignments are kept (not reshuffled on each run)
+   - `--randomize` resets and randomly selects one aggregator per subnet (unique by client type when possible)
+   - `--aggregator zeam_0` sets that node as aggregator for its subnet only; other subnets unchanged
+   - With `--randomize --aggregator zeam_0`, that subnet uses `zeam_0` and remaining subnets are filled at random
 13. `--checkpoint-sync-url` specifies the URL to fetch finalized checkpoint state from for checkpoint sync. Default: `https://leanpoint.leanroadmap.org/lean/v0/states/finalized`. Only used when `--restart-client` is specified.
 14. `--restart-client` comma-separated list of client node names (e.g., `zeam_0,ream_0`). When specified, those clients are stopped, their data cleared, and restarted using checkpoint sync. Genesis is skipped. Use with `--checkpoint-sync-url` to override the default URL.
 15. `--prepare` verify and install the software required to run lean nodes on every remote server, and open + persist the necessary firewall ports.
@@ -252,11 +251,12 @@ Every Ansible deployment automatically deploys an observability stack alongside
       - `tmp/ansible-run-DD-MM-YYYY-HH-MM.log` for Ansible deployments
     - Example: `NETWORK_DIR=local-devnet ./spin-node.sh --node all --logs`
 19. `--network` sets the network name label attached to every metric and log stream scraped by the observability stack.
-   - **Required for Ansible deployments** — the script exits with an error if omitted when `deployment_mode: ansible`
-   - For local deployments, falls back to the default network name if not specified
-   - Propagated to Ansible as the `network_name` variable, used in `prometheus.yml.j2` and `promtail.yml.j2` templates
-   - Appears as the `network` label on all Prometheus scrape targets and Promtail log streams, so you can filter by network in Grafana
-   - Example: `--network <your-network-name>`
+20. `--stop-all-containers` runs `docker ps -a` on each unique `enrFields.ip` in the active `validator-config.yaml` and removes **every** container except the per-host observability stack (`prometheus`, `promtail`, `cadvisor`, `node_exporter`). Stale containers from old deploys, manual runs, or other clients are removed even when their names are not listed in `validator-config.yaml`.
+   - **Ansible mode only** — fails if `deployment_mode` is not `ansible`
+   - Does not require `--node`; runs one SSH session per unique host IP (same deduplicated inventory as `--prepare`)
+   - Unlike `--stop --node all`, this is not limited to validator row names — it clears the whole Docker namespace on each host except observability
+   - **Allowed with `--stop-all-containers`:** `--validatorConfig`, `--subnets N`, `--sshKey` / `--private-key`, `--useRoot`, `--deploymentMode ansible`, `--network`, `--dry-run`, `--logs`, and `NETWORK_DIR`
+   - Example: `NETWORK_DIR=ansible-devnet ./spin-node.sh --stop-all-containers --network devnet-4 --sshKey ~/.ssh/id_ed25519 --useRoot`
 
 ### Preparing remote servers
 
@@ -281,6 +281,13 @@ NETWORK_DIR=ansible-devnet ./spin-node.sh --prepare --sshKey ~/.ssh/id_ed25519 -
 - Passing deploy-only flags (e.g. `--node`, `--generateGenesis`, `--stop`, `--metrics`) alongside `--prepare` produces a prominent error. Use `--validatorConfig`, `--subnets N`, `--sshKey`, `--useRoot`, `--deploymentMode ansible`, `--network`, `--dry-run`, or `--logs` when needed so inventory and firewall match your deploy.
 - `--node` is not required; the prepare playbook runs on **one play per unique IP** (deduplicated inventory) so parallel prepares do not fight over the same host
 
+To stop every non-observability container on each validator host (including stale containers not in `validator-config.yaml`) while keeping observability running:
+
+```bash
+NETWORK_DIR=ansible-devnet ./spin-node.sh --stop-all-containers \
+  --network devnet-4 --sshKey ~/.ssh/id_ed25519 --useRoot
+```
+
 Once preparation succeeds, proceed with the normal deploy command:
 
 ```sh

ansible-devnet/genesis/test-validator-config-subnet2.yaml

@@ -1,14 +1,14 @@
 shuffle: roundrobin
-# Test layout: 2 subnets × 7 nodes each (1 zeam + 1 grandine + 1 gean + 2 ethlambda + 1 qlean + 1 lantern per subnet
+# Test layout: 2 subnets × 7 nodes each (2 zeam + 1 grandine + 1 gean + 3 ethlambda per subnet)
 #              except subnet 1 which has 1 ream instead of ethlambda_3).
 # Each server hosts one node per subnet per client type on distinct ports.
 #
 #   Server layout (IP → nodes):
-#     37.27.89.135     grandine_0 (s0,9001)   zeam_3      (s1,9002)    qlean_0   (s0,9003)
-#     157.180.20.55    zeam_1    (s0,9001)    zeam_4      (s1,9002)    qlean_1   (s1,9003)
+#     37.27.89.135     grandine_0 (s0,9001)   zeam_3      (s1,9002)    zeam_8    (s0,9003)
+#     157.180.20.55    zeam_1    (s0,9001)    zeam_4      (s1,9002)    zeam_9    (s1,9003)
 #     178.104.133.162  gean_0    (s0,9001)    ethlambda_2 (s1,9002)
-#     178.104.151.50   ethlambda_0 (s0,9001) ream_0      (s1,9002)    lantern_0 (s0,9003)
-#     178.104.149.91   ethlambda_1 (s0,9001) gean_1      (s1,9002)    lantern_1 (s1,9003)
+#     178.104.151.50   ethlambda_0 (s0,9001) ream_0      (s1,9002)    ethlambda_8 (s0,9003)
+#     178.104.149.91   ethlambda_1 (s0,9001) gean_1      (s1,9002)    ethlambda_9 (s1,9003)
 #
 #   ./spin-node.sh --validatorConfig ansible-devnet/genesis/test-validator-config-subnet2.yaml
 deployment_mode: ansible
@@ -68,7 +68,7 @@ validators:
     subnet: 0
     isAggregator: false
     count: 1
-  - name: "qlean_0"
+  - name: "zeam_8"
     privkey: "8e9f81c9caa9e29d26a7327311ca63a38254efdfccf3ce1362bae47eae0b18b3"
     enrFields:
       ip: "37.27.89.135"
@@ -78,7 +78,7 @@ validators:
     subnet: 0
     isAggregator: false
     count: 1
-  - name: "lantern_0"
+  - name: "ethlambda_8"
     privkey: "d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2c3d4e5"
     enrFields:
       ip: "178.104.151.50"
@@ -139,7 +139,7 @@ validators:
     subnet: 1
     isAggregator: false
     count: 1
-  - name: "qlean_1"
+  - name: "zeam_9"
     privkey: "1991db904e455114bc836a0860689e77c1d2fa313a15567c6c7aa81e180925ac"
     enrFields:
       ip: "157.180.20.55"
@@ -149,7 +149,7 @@ validators:
     subnet: 1
     isAggregator: false
     count: 1
-  - name: "lantern_1"
+  - name: "ethlambda_9"
     privkey: "582644c8e66b61df3dafb33fa496cee97060566542fb8594fca661be3332710f"
     enrFields:
       ip: "178.104.149.91"

ansible-devnet/genesis/validator-config-expanded.yaml

@@ -155,7 +155,7 @@ validators:
   isAggregator: false
   count: 1
   subnet: 4
-- name: qlean_0
+- name: zeam_8
   privkey: c2bbdac5e876b3e9d4b8b6b8c2bbdac5e876b3e9d4b8b6b8c2bbdac5e876b3e9
   enrFields:
     ip: 178.104.133.162
@@ -165,7 +165,7 @@ validators:
   isAggregator: false
   count: 1
   subnet: 0
-- name: qlean_1
+- name: zeam_9
   privkey: 927eed3828e0af660ab7525d0971f7f29a3055f8832b947b03eab0bc5683bd97
   enrFields:
     ip: 178.104.133.162
@@ -175,7 +175,7 @@ validators:
   isAggregator: false
   count: 1
   subnet: 1
-- name: qlean_2
+- name: zeam_10
   privkey: 7fdceb8166b9d7ab4169d80345ab85fd042b3684ba307c4c02e851c1520975b5
   enrFields:
     ip: 178.104.133.162
@@ -185,7 +185,7 @@ validators:
   isAggregator: false
   count: 1
   subnet: 2
-- name: qlean_3
+- name: zeam_11
   privkey: 1027ce52661ef76b5ada68bc8cd45c13ebefb09c99dcad251b9bd4893c71815f
   enrFields:
     ip: 178.104.133.162
@@ -195,7 +195,7 @@ validators:
   isAggregator: false
   count: 1
   subnet: 3
-- name: qlean_4
+- name: zeam_12
   privkey: 0763e1a9f1909df7eae31ef4e93d5aaf45a506624ef81df893fc02e08a4ab94f
   enrFields:
     ip: 178.104.133.162
@@ -205,7 +205,7 @@ validators:
   isAggregator: false
   count: 1
   subnet: 4
-- name: lantern_0
+- name: ethlambda_8
   privkey: d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2c3d4e5
   enrFields:
     ip: 178.104.149.91
@@ -215,7 +215,7 @@ validators:
   isAggregator: false
   count: 1
   subnet: 0
-- name: lantern_1
+- name: ethlambda_9
   privkey: 582644c8e66b61df3dafb33fa496cee97060566542fb8594fca661be3332710f
   enrFields:
     ip: 178.104.149.91
@@ -225,7 +225,7 @@ validators:
   isAggregator: false
   count: 1
   subnet: 1
-- name: lantern_2
+- name: ethlambda_10
   privkey: 943151f2327ea91357632dd8a3a498242b07eeeebad973ea82d6bead80627d3e
   enrFields:
     ip: 178.104.149.91
@@ -235,7 +235,7 @@ validators:
   isAggregator: false
   count: 1
   subnet: 2
-- name: lantern_3
+- name: ethlambda_11
   privkey: 31258c1b84e4a64954fb18dbb04ec9de7079ad4d0273b40380aa1f4ca2f20804
   enrFields:
     ip: 178.104.149.91
@@ -245,7 +245,7 @@ validators:
   isAggregator: false
   count: 1
   subnet: 3
-- name: lantern_4
+- name: ethlambda_12
   privkey: a5dac14f317084d255c57e5080985b1dbd79cf8fc625a9c7fba781abf07001bd
   enrFields:
     ip: 178.104.149.91