Skip to content

Update systemd readme deployment #167

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
anhthii merged 1 commit into from
Oct 7, 2026
Merged

Update systemd readme deployment #167

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
284 changes: 184 additions & 100 deletions deployments/systemd/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,206 +21,282 @@ Mpcium is a distributed threshold cryptographic system that requires multiple no

### Software Dependencies

- **Go 1.25+** on all nodes
- **Go 1.27.1** on all nodes
- **Git** for source code management
- **NATS server** with credentials
- **Consul** for service discovery
- **mkcert** for TLS certificate generation
- **jq** (used by `setup-config.sh verify` to check identity files; without it those checks are silently skipped)
- **systemd v250+** (required for `systemd-creds`)
- **NATS server** with TLS (mTLS client certificates)
- **Consul** for service discovery (HTTPS + ACL token)

### Deployment

Follow these steps for manual deployment across your cluster:
All commands assume you are operating as `root`. If you are SSH'd in as a sudo user, use full paths (e.g. `/root/...`) rather than `~`.

#### Step 1: Install Go
Steps marked **(designated node)** are run **once**, on a single node (e.g. `node0`). Steps marked **(all nodes)** are run on every node.

Before installing Mpcium, you need Go 1.25+ installed on all nodes.
#### Step 1: Install Go (all nodes)

**Download Go Binaries with wget:**

For amd64 (x86_64):
Download Go 1.27.1 (auto-detects `amd64` / `arm64`):

```bash
wget https://go.dev/dl/go1.25.0.linux-amd64.tar.gz
GO_VERSION=go1.27.1
ARCH=$(dpkg --print-architecture) # amd64 or arm64
wget "https://go.dev/dl/${GO_VERSION}.linux-${ARCH}.tar.gz"

sudo rm -rf /usr/local/go
sudo tar -C /usr/local -xzf "${GO_VERSION}.linux-${ARCH}.tar.gz"
```

For arm64 (AArch64, e.g. Graviton or Apple M1/M2 Linux VMs):
Make Go available system-wide — for root, `sudo`, and every login shell:

```bash
wget https://go.dev/dl/go1.25.0.linux-arm64.tar.gz
```
# GOROOT + PATH for all users (including root) in login shells
sudo tee /etc/profile.d/go.sh >/dev/null <<'EOF'
export GOROOT=/usr/local/go
export PATH=$PATH:$GOROOT/bin
EOF
source /etc/profile.d/go.sh

**Extract to /usr/local:**
# sudo resets PATH to secure_path, which includes /usr/local/bin
sudo ln -sf -t /usr/local/bin /usr/local/go/bin/go /usr/local/go/bin/gofmt

Remove any old Go installation first:

```bash
sudo rm -rf /usr/local/go
go version && sudo go version && go env GOROOT # GOROOT should print /usr/local/go
```

Extract the archive:
#### Step 2: Install Mpcium and Prepare the Host (all nodes)

```bash
# For amd64
sudo tar -C /usr/local -xzf go1.25.0.linux-amd64.tar.gz
# System packages
sudo apt-get install -y git jq
systemctl --version | head -1 # must be 250 or newer

# Clone and install binaries into /usr/local/bin
cd /root
git clone https://github.com/fystack/mpcium.git
cd /root/mpcium
sudo -E make install

# For arm64
sudo tar -C /usr/local -xzf go1.25.0.linux-arm64.tar.gz
# Create system user and directories
sudo useradd -r -s /bin/false -d /opt/mpcium -c "Mpcium MPC Node" mpcium
sudo mkdir -p /opt/mpcium/identity /etc/mpcium/certs
```

**Update PATH:**
#### Step 3: Generate Peer Configuration (designated node)

Add Go binary to your environment. Edit `~/.bashrc` or `~/.zshrc`:
`mpcium-cli` reads and writes `peers.json` and `identity/` relative to the **current directory**, so always run it from `/opt/mpcium`:

```bash
export PATH=$PATH:/usr/local/go/bin
cd /opt/mpcium
mpcium-cli generate-peers -n 3
```

Reload shell:
Copy `peers.json` to the other nodes **now** — `generate-identity` (Step 9) fails with `peers file peers.json does not exist` if it is missing:

```bash
source ~/.bashrc
scp /opt/mpcium/peers.json root@<node1-ip>:/opt/mpcium/
scp /opt/mpcium/peers.json root@<node2-ip>:/opt/mpcium/
```

**Verify Installation:**
#### Step 4: Create config.yaml (all nodes)

```bash
go version
cp /root/mpcium/config.prod.yaml.template /etc/mpcium/config.yaml
sudo chown root:mpcium /etc/mpcium/config.yaml
sudo chmod 640 /etc/mpcium/config.yaml
```

You should see:

```
go version go1.25.0 linux/amd64
Edit `/etc/mpcium/config.yaml` and fill in the fields below. `event_initiator_pubkey` and `chain_code` are filled in Steps 5 and 8.

```yaml
nats:
url: tls://<nats-host>:4222 # must be tls:// in production
username: "" # leave empty when NATS authenticates via mTLS
password: ""
tls:
client_cert: "/etc/mpcium/certs/client-cert.pem"
client_key: "/etc/mpcium/certs/client-key.pem"
ca_cert: "/etc/mpcium/certs/rootCA.pem"

consul:
address: https://<consul-host>:8500
token: "<consul ACL token>"
ca_cert: "/etc/mpcium/certs/consul-rootCA.pem" # only if Consul is set up with TLS (https://); not in the template

mpc_threshold: 1
environment: production
```

or
| Field | Notes |
| ------------------------ | --------------------------------------------------------------------------------------- |
| `environment` | Must be `production`. Remove the template's trailing `# ...` comment on this line — `setup-config.sh` does not strip it and will otherwise treat the node as development and skip the TLS checks |
| `nats.url` | `tls://...` in production |
| `nats.tls.*` | Paths to the certificates from Step 6 |
| `consul.address` | `https://...` in production |
| `consul.token` | Consul ACL token (only sent when `environment: production`) |
| `consul.ca_cert` | **Only required if Consul is set up with TLS** (`https://` address signed by a private CA). Omit for plain `http://` Consul. Only used when `environment: production` |
| `mpc_threshold` | `t` in the t-of-n threshold |
| `chain_code` | 64 hex chars, **identical on all nodes** (Step 5) |
| `event_initiator_pubkey` | Public key from Step 8, **identical on all nodes** |
| `badger_password` | Do **not** set — supplied via systemd credentials (Step 11) |

```
go version go1.25.0 linux/arm64
```
#### Step 5: Set chain_code (designated node, then copy)

#### Step 2: Prepare Environment
`chain_code` is a 32-byte value used for HD wallet key derivation. Generate it once and use the **exact same value** on every node — a mismatch prevents the cluster from working correctly.

```bash
# Install Mpcium binaries (preserve environment to access Go)
sudo -E make install
CC=$(openssl rand -hex 32)
sed -i -E "s|^([[:space:]]*chain_code:).*|\1 \"$CC\"|" /etc/mpcium/config.yaml
echo "$CC"
```

# Create system user and directories
sudo useradd -r -s /bin/false -d /opt/mpcium -c "Mpcium MPC Node" mpcium
sudo mkdir -p /opt/mpcium /etc/mpcium
Set the same `chain_code` value in `/etc/mpcium/config.yaml` on the other nodes, and back it up in your password manager.

#### Step 6: Install TLS Certificates (all nodes)

Copy the NATS client certificates from your infrastructure/PKI host into `/etc/mpcium/certs/`. The Consul root CA is **only required if Consul is set up with TLS** — skip the last line otherwise:

```bash
scp <pki-dir>/nats/client-cert.pem root@<node-ip>:/etc/mpcium/certs/
scp <pki-dir>/nats/client-key.pem root@<node-ip>:/etc/mpcium/certs/
scp <pki-dir>/nats/rootCA.pem root@<node-ip>:/etc/mpcium/certs/
# Only if Consul is set up with TLS
scp <pki-dir>/consul/rootCA.pem root@<node-ip>:/etc/mpcium/certs/consul-rootCA.pem
```

#### Step 3: Configure Permissions
The service runs as `mpcium`, so these files must be group-readable. `setup-config.sh` does **not** fix permissions on the `certs/` directory:

```bash
# Application data directories (service-owned)
sudo chown -R mpcium:mpcium /opt/mpcium
sudo chmod g+s /opt/mpcium
sudo chmod 750 /opt/mpcium

# Configuration directory (root-controlled, service-readable)
sudo chown root:mpcium /etc/mpcium
sudo chmod 750 /etc/mpcium
sudo chown -R root:mpcium /etc/mpcium/certs
sudo chmod 750 /etc/mpcium/certs
sudo chmod 640 /etc/mpcium/certs/*.pem

# Confirm the service user can read them
for f in /etc/mpcium/certs/*.pem; do sudo -u mpcium test -r "$f" && echo "ok $f" || echo "FAIL $f"; done
```

#### Step 4: Generate Peer Configuration
#### Step 7: Register Peers to Consul (designated node)

On **one designated node** only:
Run from the directory containing `peers.json`. `--environment production` is **required** — without it the Consul token and `consul.ca_cert` are ignored:

```bash
cd /opt/mpcium
mpcium-cli generate-peers -n 3
mpcium-cli register-peers --config /etc/mpcium/config.yaml --environment production
```

#### Step 5: Copy Config and Update Configuration
#### Step 8: Generate Event Initiator Key (designated node)

```bash
# Copy configuration template
cp ~/mpcium/config.prod.yaml.template /etc/mpcium/config.yaml
# Set proper configuration permissions
sudo chown root:mpcium /etc/mpcium/config.yaml
sudo chmod 640 /etc/mpcium/config.yaml
cd /root
mpcium-cli generate-initiator --encrypt
```

Edit `/etc/mpcium/config.yaml` to include:
This writes `event_initiator.identity.json` (public) and `event_initiator.key.age` (encrypted private key) to the current directory.

- NATS server connection details and credentials
- Consul service discovery configuration
- MPC threshold settings (`mpc_threshold`)
- Event initiator public key (will be updated in Step 5)
- Copy the `public_key` value from `event_initiator.identity.json` into `event_initiator_pubkey` in `/etc/mpcium/config.yaml` on **all nodes** — the value must be identical everywhere.
- The private key is used by the application that initiates MPC operations, not by the MPC nodes. Move `event_initiator.key.age` off the node to secure storage and keep its passphrase in your password manager.

#### Step 6: Generate Event Initiator Key
#### Step 9: Generate Node Identity (each node, on its own server)

On **one designated node** only:
Each node generates its own identity locally so the private key never leaves the server. Run from `/opt/mpcium` (output goes to `./identity/`, and `./peers.json` must exist):

```bash
mpcium-cli generate-initiator --encrypt
cd /opt/mpcium
mpcium-cli generate-identity --node node0 --encrypt # node1 on node1, node2 on node2
```

⚠️ **Important**:
This creates `identity/node0_identity.json` (public) and `identity/node0_private.key.age` (encrypted private key). Save the passphrase in your password manager (e.g. `mpcium-node0-identity-passphrase`) — it is needed in Step 11.

- This creates an encrypted private key file with `.key.age` extension that you'll need to securely distribute to application nodes that initiate MPC operations
- Copy the public key from `initiator_identity.json` and update the `event_initiator_pubkey` field in `/etc/mpcium/config.yaml` on **all nodes**
#### Step 10: Distribute Public Identity Files (all nodes)

#### Step 7: Configure Each Node
Each node needs the `*_identity.json` of **every** node, but only **its own** private key:

```bash
# Register peers
mpcium-cli register-peers --config /etc/mpcium/config.yaml --environment production

# Generate node identity (with encryption) - replace node0 with actual node name
mpcium-cli generate-identity --node node0 --encrypt
# e.g. from node0
scp /opt/mpcium/identity/node0_identity.json root@<node1-ip>:/opt/mpcium/identity/
scp /opt/mpcium/identity/node0_identity.json root@<node2-ip>:/opt/mpcium/identity/
```

#### Step 8: Generate TLS Certificates
⚠️ Never copy `*_private.key` / `*_private.key.age` to other nodes.

Before continuing, double-check on every node that:

#### Step 9: Configure Database Encryption
- `/opt/mpcium/peers.json` exists
- `/opt/mpcium/identity/` has all `nodeX_identity.json` files plus this node's own private key
- `chain_code` and `event_initiator_pubkey` in `/etc/mpcium/config.yaml` match the other nodes

#### Step 11: Configure Credentials (all nodes)

```bash
cd ~/mpcium/deployments
cd /root/mpcium/deployments/systemd
./setup-mpcium-cred.sh
# Enter BadgerDB password when prompted
# ⚠️ IMPORTANT: Backup password to secure storage (e.g., Bitwarden)
```

#### Step 10: Deploy Service
The script prompts, in order, for:

1. **BadgerDB password** — must be exactly 16, 24 or 32 characters; use a different one per node. Generate it in your password manager first (e.g. `mpcium-node0-database-password`) — if lost, the database cannot be recovered.
2. **"Did you use `generate-identity --encrypt`?"** — answer **`y`** if you followed Step 9. Answering `n` skips the identity credential and the node will fail to decrypt its private key.
3. **Identity passphrase** — the passphrase from Step 9.

It writes `/etc/mpcium/mpcium-db-password.cred` and `/etc/mpcium/mpcium-identity-password.cred`, encrypted with the systemd host secret.

> Note: this script resets `/etc/mpcium` to `root:root 700`, which the `mpcium` user cannot read. Step 12 restores it to `root:mpcium 750` — always run Step 12 afterwards.

#### Step 12: Deploy Service (all nodes)

```bash
sudo ./setup-config.sh
```

#### Step 11: Verify Deployment
Enter this node's name when prompted (e.g. `node0`). It must match a key in `peers.json` exactly — the script does not validate it.

The script fixes ownership of `/opt/mpcium` and `/etc/mpcium`, installs `/etc/systemd/system/mpcium.service` with the encrypted credentials, writes `/opt/mpcium/.env`, and **enables** the service. It does **not** start it.

Check the installed unit has no leftover placeholders (if it does, the matching `.cred` file was missing — re-run Step 11, then `sudo ./setup-config.sh update-creds`):

```bash
# Check service status
sudo systemctl status mpcium
grep -c BASE64_BLOB_DATA /etc/systemd/system/mpcium.service # must print 0
```

Start the service:

```bash
sudo systemctl start mpcium
```

# Monitor logs
#### Step 13: Verify Deployment (all nodes)

```bash
sudo systemctl status mpcium
journalctl -f -u mpcium
```

Look for: service `active (running)`, the node connecting to NATS and Consul, and discovering its peers — without repeated TLS or identity/signature errors.

## Directory Structure

After deployment, the following directory structure is created:

```
/opt/mpcium/ # Application home (mpcium:mpcium, 750)
├── certs/ # TLS certificates
│ ├── client-cert.pem
│ ├── client-key.pem
│ └── rootCA.pem
├── db/ # BadgerDB storage (auto-created)
├── backups/ # Encrypted backups (auto-created)
├── identity/ # Node identity files (auto-created)
├── identity/ # Node identity files
│ ├── node0_identity.json
│ ├── node1_identity.json
│ ├── node2_identity.json
│ └── node{X}_private.key # Current node's private key
│ └── node{X}_private.key.age # Current node's encrypted private key only
├── peers.json # Peer registry
└── .env # Environment variables

/etc/mpcium/ # Configuration (root:mpcium, 750)
└── config.yaml # Main configuration (root:mpcium, 640)
└── .env # MPCIUM_NODE_NAME (root:mpcium, 640)

/etc/mpcium/ # Configuration (root:mpcium, 750)
├── config.yaml # Main configuration (root:mpcium, 640)
├── mpcium-db-password.cred # Encrypted BadgerDB password (root:root, 600)
├── mpcium-identity-password.cred # Encrypted identity passphrase (root:root, 600)
└── certs/ # TLS certificates (root:mpcium, 750; files 640)
├── client-cert.pem
├── client-key.pem
├── rootCA.pem
└── consul-rootCA.pem # only if Consul is set up with TLS
```

### Identity Directory Examples
Expand Down Expand Up @@ -319,6 +395,14 @@ sudo ./setup-config.sh verify
- Check firewall rules between nodes
- Validate TLS certificates

**Known `setup-config.sh` quirks:**

- `verify` / `validate-only` stop at the **first** failed check without printing a summary — fix that item and re-run until it passes.
- "Development environment - TLS validation skipped" on a production node — remove the trailing `# ...` comment from the `environment: production` line in `config.yaml`.
- "CA certificate file not found" with two paths printed — harmless when `consul.ca_cert` is set; the NATS `ca_cert` lookup also matches the Consul one. Confirm both files exist manually.
- "MPCIUM_ENVIRONMENT: unbound variable" — `/etc/mpcium/config.yaml` does not exist yet (Step 4).
- `TLS handshake` / `permission denied` on certs at runtime — the `mpcium` user cannot read files in `/etc/mpcium/certs` (see Step 6).

### Log Analysis

Service logs are available via systemd journal:
Expand Down
Loading