Skip to content

Repository files navigation

License: MIT Python Docker Prometheus Grafana CI

typing

Why NodeForge exists

Running a blockchain node is easy for the first week. Then a disk fills up, or a peer table stalls, or the process OOMs during a network upgrade - and you find out the hard way that docker run isn't infrastructure.

NodeForge was built after the third missed-block penalty from a node that silently stopped syncing and nobody noticed for six hours. It wraps whatever client you already run (Geth, Reth, Cosmos SDK binaries, Solana's agave-validator) with the layer that turns a node into infrastructure.

What's inside

🚀 Deploy Docker Compose stacks or hardened systemd units, per-chain templates nodeforge/deploy
👀 Watch Prometheus exporters + Grafana dashboards: sync lag, peers, disk, missed blocks nodeforge/monitoring
🔃 React Failover daemon promotes standby + repoints DNS when health checks fail nodeforge/failover
💾 Recover Scheduled snapshots to S3-compatible storage, one-command restore nodeforge/snapshot

Quick start

git clone https://github.com/sqauproba/nodeforge.git
cd nodeforge
cp config.example.yaml config.yaml     # chain, network, alert webhook: that's the minimum
# containerized (most operators)
nodeforge deploy --chain ethereum --mode validator

# bare metal, hardened systemd unit instead of Docker
sudo nodeforge install --chain ethereum --mode rpc --systemd

Day-to-day commands

nodeforge status                          # is it healthy right now?
nodeforge logs --tail 200                 # client logs, formatted
nodeforge snapshot restore --source s3://your-bucket/eth-mainnet/latest
nodeforge failover setup --primary node1.internal --standby node2.internal
📊 nodeforge status output
ethereum-mainnet-01

  Status:        HEALTHY
  Sync:          100%  (block 20,481,203, 0 blocks behind head)
  Peers:         48
  Disk:          62%  (1.2TB / 2TB)
  Uptime:        47d 6h
  Last failover: never
  Last snapshot: 4h ago (s3://nodeforge-snapshots/eth-mainnet-01/)

How failover works

flowchart LR
    P[primary node] -- "health check / 15s" --> D{failover daemon}
    D -- "3 consecutive failures" --> F
    subgraph F[automatic response]
        direction TB
        S1["promote standby"]
        S2["repoint DNS (Cloudflare / Route53)"]
        S3["alert Telegram / Discord"]
        S4["mark primary degraded"]
        S1 --> S2 --> S3 --> S4
    end
Loading

Health checks aren't just "is the process running" - they measure sync lag, peer count and RPC latency against thresholds you set. A node that's up but stuck at block N-500 fails just as hard as a crashed process.

⚠️ Validator note - failover is disabled by default for consensus nodes. Two active signers on one key = slashed. Read docs/validator-failover.md before enabling it.

Snapshots & fast resync

Full sync takes days. Snapshots take ~20-40 min to restore:

backup:
  snapshot_schedule: "0 3 * * *"
  storage: s3://your-bucket/snapshots/
  retention_days: 14
nodeforge snapshot restore --source s3://your-bucket/snapshots/eth-mainnet/latest

Monitoring stack

nodeforge monitoring up      # Prometheus + Grafana via Compose
                             # Grafana :3000, dashboards auto-imported
🔔 Alert routing is config-driven, not code-driven
alerts:
  telegram:
    bot_token: "${TELEGRAM_BOT_TOKEN}"
    chat_id: "${TELEGRAM_CHAT_ID}"
  triggers:
    - missed_block
    - sync_stall_10min
    - disk_above_85pct
    - peer_count_below_5

Repo layout

.
├── nodeforge/
│   ├── deploy/          Compose + systemd unit generators
│   ├── failover/        health-check daemon, DNS provider adapters
│   ├── monitoring/      Prometheus exporter, Grafana dashboards
│   ├── snapshot/        backup/restore, S3-compatible clients
│   └── cli.py
├── chains/              ethereum · cosmos-sdk · solana definitions
├── systemd/             hardened unit templates
├── docker/              per-chain Dockerfiles + Compose stacks
├── docs/                validator-failover · adding-a-chain · runbooks/
├── tests/
├── config.example.yaml
└── pyproject.toml

Cost reality check

From running an Ethereum mainnet RPC pair (primary + standby) on DigitalOcean:

Item Spec Monthly
Primary node 8 vCPU / 32GB / 2TB NVMe ~$190
Standby node same spec ~$190
Object storage ~200GB retained ~$5
Bandwidth varies with RPC traffic $0–100+

Failover roughly doubles compute cost. Whether that's worth it depends on what an outage costs you.

Contributing

PRs for new chain templates are the most useful contribution - see docs/adding-a-chain.md and copy-paste chains/ethereum/ as your starting point.


:test your failover on testnet before trusting it on mainnet. a failover system nobody has tested is just a second way for things to break at 3 AM.

About

Deployment, monitoring, and failover toolkit for blockchain validator and RPC nodes

Resources

Stars

90 stars

Watchers

1 watching

Forks

Releases

Packages

Contributors

Languages