Coordinate measurement batches with a readiness barrier

.github/workflows/ci.yml

@@ -44,6 +44,7 @@ jobs:
           name: evidence-${{ matrix.lane }}-${{ github.sha }}-${{ github.run_attempt }}
           path: |
             .cache/ci/gofmt.txt
+            .cache/ci/gofmt.diff
             .cache/ci/tests.json
             .cache/ci/race-tests.json
             .cache/ci/kernel-tests.json

docs/README.md

@@ -6,6 +6,7 @@ This directory is the versioned source for the [Debuglet documentation site](htt
 
 - [CLI reference](cli.md) — commands, local demo, and scriptable workflows.
 - [Go client library](client.md) — submit debuglets and read results from an application.
+- [Reusable and coordinated measurements](measurements.md) — saved profiles, retries and experiments across executors.
 - [Portable results](results.md) — export retained output and admission facts for offline analysis.
 - [Controlled latency experiment](research/latency-evaluation.md) — compare native and WASM probes under known local network faults.
 - [Write a debuglet](debuglets.md) — execution model and Go authoring interface.

docs/debuglets.md

@@ -68,6 +68,11 @@ ABI-v1 guests remain supported without recompilation. The baseline ABI label
 alone does not advertise this extension; use a release that includes it. The
 [extension contract](development/guest-io.md) records its wire signatures.
 
+Coordinated guests using `Ready` require the optional
+`debuglet_experiment_v1.ready` extension on the executor and matching dispatcher
+support. See [coordinated measurements](measurements.md#coordinate-a-batch-of-debuglets)
+and its five-executor example. Existing guests do not acquire this requirement.
+
 ### Other languages
 
 Any WebAssembly module that targets `wasm32-wasip1` and imports only the

docs/measurements.md

@@ -107,3 +107,23 @@ never replays an ambiguous submission. On success, failure, deadline or caller
 cancellation it requests cancellation of every known run, then inspects cleanup
 under an independent ten-second bound. `confirmed_absent` records a current
 executor observation; `unconfirmed` requires follow-up using the returned IDs.
+
+## Coordinate a batch of debuglets
+
+An experiment submits a fixed set of ordinary runs together. Choose each run's
+executor, WASM, arguments and policy independently. The existing transaction ID
+groups their results. `Client.SubmitExperimentTEST` accepts a saved definition
+and returns a receipt with run IDs and program hashes; `ExportExperiment` reads
+the group's results and `CancelExperiment` requests cancellation of its known runs.
+
+Each guest finishes its setup, then calls `debuglet.Ready(ctx, metadata)`. Once
+all batch members have declared readiness, it receives their bounded metadata
+and a common future start time. Call `debuglet.WaitStart(ctx, experiment)` before
+starting the measurement. Hosts need synchronized clocks for close timing;
+this is not a guarantee of simultaneous execution. Experiment traffic uses the
+normal permitted sockets, while readiness travels through the dispatcher.
+
+The [five-executor example](../examples/experiments/README.md) includes a manifest,
+runner and UDP guest. It requires the new dispatcher and executor readiness
+extension; it does not add a console flow. Missing members, retries and partial
+results remain decisions for the experiment's author, within the run deadlines.

docs/operations/configuration.md

@@ -203,7 +203,7 @@ OAuth, external TLS and SCION state need the deployment's complete backup plan;
 a database snapshot alone does not include every required credential or config.
 Never start original and restored copies with the same identity simultaneously.
 
-Dispatcher schema 27 and executor schema 8 are the current schema boundaries.
+Dispatcher schema 28 and executor schema 8 are the current schema boundaries.
 Recognized older databases require the explicit upgrade below. Dispatcher
 schemas below 3 and executor schemas below 2 lose recorded `debuglets` and
 `debuglet_logs` on upgrade and require explicit acceptance. Preserved paid rows

examples/experiments/README.md

@@ -0,0 +1,74 @@
+# Five-executor experiment
+
+This example submits five ordinary TEST runs as one batch. The batch transaction
+is the experiment ID; the saved receipt records each order ID, run ID, executor,
+WASM path and SHA256, arguments and requested policy. Each participant may use a
+different WASM file and arguments. There is no role or topology configuration.
+
+The included `peer` guest obtains its UDP listener before announcing readiness.
+The dispatcher exchanges the participants' opaque endpoint metadata and assigns
+one future start time after all five are ready. Each guest waits for that time,
+sends a datagram directly to each of the other four executors, and reports all
+four arrivals. The output contains the requested start, actual local start,
+send/receive timestamps and final peer count. No experiment traffic passes
+through the dispatcher.
+
+Use a dispatcher and five distinct connected executors built from this revision.
+Configure UDP listeners with reachable public hosts and available port ranges on
+each executor. Both operator destination policy and the per-run `addresses` must
+permit the peer hosts; metadata does not grant network access. `dbl up` has only
+one executor and is not sufficient for this example.
+
+```sh
+GOOS=wasip1 GOARCH=wasm go build -o examples/experiments/peer/debuglet.wasm ./examples/experiments/peer
+# Use your existing saved connection and login:
+dbl nodes
+cp examples/experiments/five.json experiment-definition.json
+# Replace executor-1 through executor-5 with actual IDs and the example
+# addresses with peer IPs/CIDRs allowed by your operator policy.
+go run ./examples/experiments/run -manifest experiment-definition.json \
+  -receipt experiment-receipt.json > experiment-results.json
+```
+
+The WASM paths in the definition are relative to your working directory. Set
+`sha256` to a known lower-case digest to reject changed files before submission;
+leave it empty to record the submitted digest. The runner uses the same saved
+connection and credentials as `dbl`; `-config` and `-dispatcher` select another
+profile. `-endpoint http://127.0.0.1:9000` explicitly selects a local development
+dispatcher without loading a saved credential. Remote TEST use also needs
+`-allow-remote-test` and server authorization. No payment activation is performed.
+
+The receipt file must not already exist. It is written even when submission
+returns an error, preserving known identities for inspection. An uncertain
+submission must not be blindly resubmitted. Existing output and cancellation
+routes remain usable independently:
+
+```sh
+go run ./examples/experiments/run -action results -receipt experiment-receipt.json \
+  > experiment-results.json
+go run ./examples/experiments/run -action cancel -receipt experiment-receipt.json
+# Decode the ordinary result exports' base64 guest output:
+jq -r '.results[].output.entries[].output | @base64d' experiment-results.json
+```
+
+`results` captures a snapshot without waiting. `submit` waits up to `-timeout`
+(default 90s), then requests cancellation of known runs. Cancellation
+acknowledgement is not proof of termination. A missing participant is bounded by
+the guest/dispatcher deadlines; a lost UDP datagram is bounded by each run's
+`timeout_ms`. This example fails rather than retrying packets or replacing members.
+Ready waits for at most 30 seconds and is also bounded by the run lifetime and
+caller deadline. Keep the policy budget longer (the sample uses 60s). Metadata
+is limited to 4KiB per participant. The guest requires the optional
+`debuglet_experiment_v1.ready` host extension. Declaring readiness is persistent;
+cancelling the local wait does not withdraw it. Cancel the run to withdraw.
+
+The shared start is a requested time, not atomic distributed execution. Host
+clocks may differ, and these guest-reported timestamps are not a clock
+synchronization or authenticated packet-evidence claim. UDP can lose packets;
+network reachability, NAT traversal and clock setup remain operator concerns.
+
+Native applications can call `client.SubmitExperimentTEST`,
+`client.ExportExperiment` and `client.CancelExperiment`. Guest authors call
+`debuglet.Ready(ctx, metadata)` after their setup and
+`debuglet.WaitStart(ctx, experiment)` before beginning their own algorithm.
+All participants must call Ready; membership is fixed by the submitted batch.

examples/experiments/five.json

@@ -0,0 +1,84 @@
+{
+  "participants": [
+    {
+      "order_id": 0,
+      "executor_id": "executor-1",
+      "wasm_path": "examples/experiments/peer/debuglet.wasm",
+      "sha256": "",
+      "args": [],
+      "policy": {
+        "floor_bw": 1000000,
+        "ceil_bw": 1000000,
+        "timeout_ms": 60000,
+        "addresses": [
+          "192.0.2.0/24"
+        ],
+        "listen_udp": true
+      }
+    },
+    {
+      "order_id": 1,
+      "executor_id": "executor-2",
+      "wasm_path": "examples/experiments/peer/debuglet.wasm",
+      "sha256": "",
+      "args": [],
+      "policy": {
+        "floor_bw": 1000000,
+        "ceil_bw": 1000000,
+        "timeout_ms": 60000,
+        "addresses": [
+          "192.0.2.0/24"
+        ],
+        "listen_udp": true
+      }
+    },
+    {
+      "order_id": 2,
+      "executor_id": "executor-3",
+      "wasm_path": "examples/experiments/peer/debuglet.wasm",
+      "sha256": "",
+      "args": [],
+      "policy": {
+        "floor_bw": 1000000,
+        "ceil_bw": 1000000,
+        "timeout_ms": 60000,
+        "addresses": [
+          "192.0.2.0/24"
+        ],
+        "listen_udp": true
+      }
+    },
+    {
+      "order_id": 3,
+      "executor_id": "executor-4",
+      "wasm_path": "examples/experiments/peer/debuglet.wasm",
+      "sha256": "",
+      "args": [],
+      "policy": {
+        "floor_bw": 1000000,
+        "ceil_bw": 1000000,
+        "timeout_ms": 60000,
+        "addresses": [
+          "192.0.2.0/24"
+        ],
+        "listen_udp": true
+      }
+    },
+    {
+      "order_id": 4,
+      "executor_id": "executor-5",
+      "wasm_path": "examples/experiments/peer/debuglet.wasm",
+      "sha256": "",
+      "args": [],
+      "policy": {
+        "floor_bw": 1000000,
+        "ceil_bw": 1000000,
+        "timeout_ms": 60000,
+        "addresses": [
+          "192.0.2.0/24"
+        ],
+        "listen_udp": true
+      }
+    }
+  ]
+}

examples/experiments/peer/main.go

@@ -0,0 +1,119 @@
+// SPDX-License-Identifier: Apache-2.0
+// Copyright 2026 ETH Zurich
+
+// peer exchanges one UDP datagram with every other experiment participant.
+package main
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"os"
+	"time"
+
+	"github.com/netsec-ethz/debuglet/pkg/debuglet"
+)
+
+type endpoint struct {
+	Address string `json:"endpoint"`
+}
+type packet struct {
+	ExperimentID string `json:"experiment_id"`
+	RunID        string `json:"run_id"`
+	SentAtNS     int64  `json:"sent_at_ns"`
+}
+
+func main() {
+	if err := run(); err != nil {
+		fmt.Fprintln(os.Stderr, err)
+		os.Exit(1)
+	}
+}
+
+func run() error {
+	address, err := debuglet.ListenUDPAddr()
+	if err != nil {
+		return err
+	}
+	metadata, err := json.Marshal(endpoint{Address: address})
+	if err != nil {
+		return err
+	}
+	ctx, cancel := context.WithTimeout(context.Background(), 45*time.Second)
+	defer cancel()
+	readyAt := time.Now().UnixNano()
+	experiment, err := debuglet.Ready(ctx, metadata)
+	if err != nil {
+		return err
+	}
+	if len(experiment.Participants) != 5 {
+		return errors.New("peer example requires five participants")
+	}
+	seen := map[string]bool{}
+	peers := map[string]string{}
+	self := ""
+	for _, member := range experiment.Participants {
+		if seen[member.ExecutorID] {
+			return errors.New("peer example requires five distinct executors")
+		}
+		seen[member.ExecutorID] = true
+		var listener endpoint
+		if err := json.Unmarshal(member.Metadata, &listener); err != nil {
+			return err
+		}
+		if listener.Address == "" {
+			return errors.New("participant has no endpoint")
+		}
+		if listener.Address == address {
+			self = member.ID
+		} else {
+			peers[member.ID] = listener.Address
+		}
+	}
+	if self == "" || len(peers) != 4 {
+		return errors.New("listener endpoints must be distinct")
+	}
+	if err := debuglet.WaitStart(ctx, experiment); err != nil {
+		return err
+	}
+	startedAt := time.Now().UnixNano()
+	packets := []map[string]any{}
+	for id, destination := range peers {
+		conn, err := debuglet.ConnectUDP(destination)
+		if err != nil {
+			return err
+		}
+		sentAt := time.Now().UnixNano()
+		data, err := json.Marshal(packet{ExperimentID: experiment.ID, RunID: self, SentAtNS: sentAt})
+		if err != nil {
+			conn.Close()
+			return err
+		}
+		err = conn.Write(data)
+		conn.Close()
+		if err != nil {
+			return err
+		}
+		packets = append(packets, map[string]any{"direction": "sent", "peer_run_id": id, "endpoint": destination, "sent_at_ns": sentAt})
+	}
+	buffer := make([]byte, 1024)
+	received := map[string]bool{}
+	for len(received) < len(peers) {
+		n, from, err := debuglet.ReadFromUDP(buffer)
+		if err != nil {
+			return err
+		}
+		receivedAt := time.Now().UnixNano()
+		var message packet
+		if err := json.Unmarshal(buffer[:n], &message); err != nil {
+			return err
+		}
+		if message.ExperimentID != experiment.ID || peers[message.RunID] == "" || received[message.RunID] {
+			return errors.New("unexpected peer datagram")
+		}
+		received[message.RunID] = true
+		packets = append(packets, map[string]any{"direction": "received", "peer_run_id": message.RunID, "endpoint": from, "sent_at_ns": message.SentAtNS, "received_at_ns": receivedAt})
+	}
+	return json.NewEncoder(os.Stdout).Encode(map[string]any{"experiment_id": experiment.ID, "run_id": self, "endpoint": address, "ready_at_ns": readyAt, "start_time_ns": experiment.StartTimeNS, "actual_start_ns": startedAt, "lateness_ns": startedAt - experiment.StartTimeNS, "sent": len(peers), "received": len(received), "finished_at_ns": time.Now().UnixNano(), "packets": packets})
+}