OPERATIONS.md

Bitcoin Machine Code — Operations Guide

This guide covers installing, configuring, running, upgrading and troubleshooting the bmcbitcoind daemon of Bitcoin Machine Code on a Linux x86-64 host under systemd. It describes the system as it is; what the node does and does not implement is in README.md and docs/FEATURE_GAPS.md.

Overview

One binary, asm/daemon/bmcbitcoind (named bitcoind until 2026-09-07; the sources are still bitcoind.asm), runs the node as a fork-model daemon:

• a parent accepts inbound peers (one forked serve child per connection) and runs the embedded JSON-RPC server on its own thread;
• a forked download worker owns the outbound peer legs, block download and keep-up, UTXO application and reorg handling;
• a background UTXO compaction forks a short-lived child per merge; it appears as a second bmcbitcoind process with the same command line.

Everything the node persists lives in one datadir, and every chain gets its own subdirectory of it: <datadir>/main, <datadir>/signet, <datadir>/testnet4, <datadir>/regtest. bitcoin.conf is shared at the datadir root (or ../config/bitcoin.conf); all other state is per chain, so two chains never share block, UTXO or wallet state.

The companion client is asm/daemon/bmc_cli. With -datadir=<dir> it reads the chain, RPC port and credentials from that datadir's bitcoin.conf and cookie, so no other flags are usually needed.

Installation

Prerequisites

• Linux x86-64 (NASM ELF64, System V ABI).
• nasm, gcc, make, python3 (build, test suite, build audits).
• Disk: the mainnet block archive is about 0.7 TB at the current height (713 GB across 5,733 blk*.dat files) and grows with the chain; the UTXO store is about 13–15 GB; the optional indexes add about 27 GB (the txid index), 91 GB (the txo-spender index), ~200 GB (the address history) and 13 GB (the block filters), each as sorted run files the daemon builds during the sync (2026-09-16, docs/devlog/INDEX_RUNS.md). The archive re-layout tool needs archive-sized scratch space (about 1.1 TB). Pruned mode fits in a few GB.
• RAM: several GB; the initial-sync memtable is sized from dbcache (default 450 MiB, Core's; the benchmarks use 8192). As in Core, dbcache is the total: with the async flush the live memtable and the flush writer's frozen copy get half each (2026-10-08, plan M2).
• Optional: tor (SOCKS 9050, control 9051), i2pd (SAM 7656), cjdroute.

Build

cd /path/to/repo/asm
make runtime                  # the daemon, bmc_cli, and every helper the daemon execs
make daemon/bmcbitcoind              # the daemon alone (about 31 MB)
make daemon/bmc_cli           # the RPC client
make tests/tool_archive_relayout  # archive maintenance tool (optional)

The daemon runs its index builders (bmc_build_tx_index, bmc_build_txospender_index, bmc_build_addr_hist, bmc_build_coinstats_hist, bmc_merge_index_runs) from its own directory. Without them it keeps running and logs builder ... not executable, and the configured indexes are never folded into runs; make runtime builds the whole set (make print-runtime-helpers lists it).

asm/build.sh is an equivalent wrapper. The build uses gcc -no-pie -O2 and nasm -f elf64.

User and directories

The reference deployment runs the service as an unprivileged user with:

path	purpose
/path/to/repo/asm	service working directory; binaries under asm/daemon/
/path/to/repo/data	the datadir (root of all chains)
/path/to/repo/config/bitcoin.conf	configuration (git-ignored)
/path/to/repo/logs/<chain>/bitcoin.<chain>.log	console (stdout/stderr) log per chain
/etc/bmc/wallet.pass	wallet passphrase file, outside the datadir (optional)

Create the datadir and logs/<chain> before the first start. The datadir may be a symlink or mount point; the daemon resolves it with realpath and changes directory into <datadir>/<chain>.

A first start on an empty datadir bootstraps itself: DNS-seed discovery, headers-first sync, parallel block download, then a from-genesis UTXO build with full script verification. Expect days, CPU-bound on signature verification. The pulse is [dl] heartbeat: tip=<h> peers=<live>/<wanted> txouts=<utxo count> uptime=<d:hh:mm:ss>.

Host tuning

One kernel setting, applied on the reference host 2026-09-10 and kept in /etc/sysctl.d/90-bmc-tcp.conf:

net.ipv4.tcp_slow_start_after_idle = 0

By default Linux resets a TCP connection's send window to its initial 10 segments after one round trip of quiet. A block connection is quiet for ten minutes between blocks, so every block this node serves to an inbound peer (and every blocktxn reply) started from the smallest window: a 40 KB compact block took three round trips and an 800 KB blocktxn seven on a 0.8 s peer, measured on a passive capture. With the setting off, a connection keeps the window it earned. It governs what THIS host sends; a remote peer's kernel decides how fast it sends to us. It applies to every socket on the box, the Core oracle and a benchmark run included.

Configuration

Location

bmcbitcoind serve <datadir> resolves its configuration in this order: a -conf=<path> argument (an unreadable file is fatal, never a fallback); $BITCOIN_CONF; <datadir>/bitcoin.conf if readable; <datadir>/../config/bitcoin.conf. The path used is echoed at boot as [config] loaded <path>: N setting(s) applied, followed by a [config] block with every resolved value (conns, utxo, pool, mpol, net, chain). Check it after any change; values are read once at start-up. Unknown keys are ignored with a message; a Core option the node does not implement is named at boot as having no effect.

config/bitcoin.sample.conf lists every key with its default. Keys are Bitcoin Core's names and defaults unless the comment says otherwise.

Chain and ports

chain=main                 # main | testnet4 | signet | regtest (testnet4=1 / signet=1 / regtest=1 also accepted)
# signetchallenge=<hex>    # CUSTOM signet; also determines the network magic
port=8333                  # P2P; chain default if omitted (see Reference)
rpcport=8332               # RPC; chain default if omitted
listen=1                   # IPv4 and IPv6 listeners on `port`

chain=test/testnet (testnet3) is refused. A custom signet gets no chain-work floor and no DNS seeds. rpcport must differ from port: the P2P listener binds first and the RPC server then loses its bind, leaving the node without RPC. A fresh non-main chain directory self-seeds its genesis.

RPC authentication and access

• Cookie (default). <datadir>/<chain>/.cookie is written mode 0600 at start (__cookie__:<64 hex>) and deleted on shutdown. rpccookiefile= relocates it.
• rpcauth=user:salt$hash (repeatable, up to 8) for a fixed credential; a malformed entry is ignored with a log line. rpcuser=/rpcpassword= work too but keep a plaintext secret on disk.
• The server starts if any credential source is available and refuses to start when none is.
• Bind and ACL. The listener binds 127.0.0.1. rpcbind= is honoured only with at least one rpcallowip= (up to 16 subnets; 127.0.0.0/8 and ::1 are always allowed); otherwise the log says Option -rpcbind was ignored because -rpcallowip was not specified. A malformed rpcallowip is fatal.
• Peer permissions (2026-09-01). whitelist=[perms@]<subnet> and whitebind=[perms@]addr:port grant Core's noban, relay, forcerelay, mempool, download, addr (and in); a bare entry gets Core's implicit set (noban, mempool, relay per whitelistrelay, forcerelay per whitelistforcerelay). bloomfilter and out are refused at start-up with the reason. blocksonly=1 refuses transactions from every peer without relay (disconnect, not scored, Core's log words), sends fRelay=0 and no feefilter, and reports localrelay:false; inboundrelaypercent caps the inbound peers that negotiate tx relay (only relaying peers count). getpeerinfo now itemizes inbound peers (slots 64..127 of the shared table) with permissions and a truthful relaytxes.

Wallet passphrase source

An encrypted wallet (bmcwallet.enc) is adopted locked at boot. To unlock it automatically, provide a passphrase source, in order of precedence:

1. $BMC_WALLET_PASS (visible in systemctl show and /proc/<pid>/environ);
2. walletpassfile=<absolute path> in bitcoin.conf.

The file is refused, with a [wallet] walletpassfile ... not usable: <why> line, if it is not absolute, is inside the datadir, is world-accessible (any 007 bit), is group-writable, is not a regular file, or is empty. The first line is the passphrase. The intended shape:

sudo install -d -m 0755 /etc/bmc
sudo install -o root -g <service-group> -m 0640 secret /etc/bmc/wallet.pass

A legacy <store>.pass beside the wallet is not read and is warned about.

Anonymity networks

proxy=127.0.0.1:9050       # SOCKS5 for every network
onion=127.0.0.1:9050       # SOCKS5 for .onion only (defaults to proxy; "0" disables)
onlynet=onion              # restrict networks (repeatable): ipv4 ipv6 onion i2p cjdns
discover=0                 # do not learn/announce our own address
torcontrol=127.0.0.1:9051  # tor control port: creates OUR onion service
torpassword=               # only if tor uses HashedControlPassword
i2psam=127.0.0.1:7656      # SAM bridge; empty = i2p off
cjdnsreachable=1           # the cjdns tun (fc00::/8) is present
v2transport=1              # BIP324 encrypted transport, default on

• proxy= (unlike onion= alone) also stops resolver lookups for peer names, skips the DNS seeds, randomises SOCKS5 credentials per connection (one Tor circuit each), and never advertises a clearnet address to onion or I2P peers. Seed the address book or use addnode=/connect=.
• Onion service. With torcontrol= (and listen=1, listenonion=1) the node authenticates on the control port (cookie file named by PROTOCOLINFO, or torpassword), issues ADD_ONION with a key persisted in <chaindir>/onion_v3_private_key, and listens on a loopback socket at the chain default P2P port + 1 (127.0.0.1:8334 on mainnet). The service is ephemeral and exists only while the daemon holds the control connection. It takes a few minutes after start; the confirming line is [tor] onion service <addr>.onion:<port> -> 127.0.0.1:<port+1> (key onion_v3_private_key). The service account needs read access to tor's control cookie: under systemd, SupplementaryGroups=debian-tor (group membership in the user database is not applied to a service). Outbound onion needs only onion=/proxy=.
• I2P. i2psam= opens a SAM session with a destination persisted in <chaindir>/i2p_private_key; boot line [dial] i2p session up via SAM <ip:port>, our address <b32>.b32.i2p. With listen=1 an acceptor thread sits in SAM STREAM ACCEPT on that session and inbound I2P streams reach the serve loop like any other inbound peer; boot line [i2p] accepting inbound streams on <b32>.b32.i2p, per-peer line [serve] inbound over i2p from <b32>. i2pacceptincoming=0 turns that off. A router that reports itself firewalled cannot publish a leaseset, so no inbound will arrive until it can.
• CJDNS. Needs a running cjdroute (a systemd unit in the reference deployment; the node's unit is ordered after it) and IPv6 on the host; boot line [dial] cjdns reachable (fc00::/8 over IPv6). Inbound cjdns peers arrive on the IPv6 listener. Without IPv6: [dial] no IPv6 on this host: ipv6 and cjdns peers are unreachable.
• BIP324. Inbound v2 accepted, v2 dialled to peers advertising NODE_P2P_V2, v1 detected in band. Boot line [net] BIP324 v2 transport enabled (services=0x...); N of M known peers advertise v2.

getnetworkinfo reports per-network reachability, the onion hostname and the I2P destination.

ZMQ and other keys

• zmqpubhashblock, zmqpubhashtx, zmqpubrawblock, zmqpubrawtx, zmqpubsequence take tcp://<interface>:<port>; tcp://*:PORT is refused, name an interface (127.0.0.1 for local subscribers). A publisher has no authentication. zmqpubsequence (Core's A/R/C/D stream with the mempool sequence number) was refused until 2026-09-19.

• Address history (addrindex=1, 2026-09-16): the daemon builds it DURING the sync. The live journal (addrindex.tail) records every funding and spend from genesis; every bmc.indexrunblocks heights (default 20,000, kept 144 below the applied height) the trailing builder folds the next range into a sorted run addr_hist.r<from>-<to>.dat — its spends read from undo, so a run is self-contained — rotates the journal to drop what the run now covers, and merges runs at six. The Esplora /address routes, getaddressbalance and getaddresstxids read the runs plus the journal. addrindex=1 must still be set before the sync (or within the undo window of the tip): a journal that starts late cannot reconstruct historic spends. A node with the old single addr_hist.dat keeps it: it is simply the run that starts at 0. The whole-chain builder (daemon/bmc_build_addr_hist <chaindir> [to], ~700 GB of temp, hours) still exists for a one-off rebuild; nothing on a fresh sync needs it. /address/:addr/utxo needs the txospender index too.

  getaddresstxids takes an optional inclusive height window, following the addrindex patch set's convention (Core has no address index, so that lineage is the only convention there is):

  getaddresstxids {"addresses": ["bc1..."], "start": 800000, "end": 810000}
  

  Both bounds are optional and the bare getaddresstxids "<addr>" form is unchanged. The window is applied before a run event's txid is resolved, and resolving one costs a block read, so a windowed query reads only that range's blocks. Without a window the reply is capped at 100,000 txids — page with start/end rather than relying on that cap, which truncates silently. getaddressbalance takes no window: a balance is a property of the whole address, and it is summed over every event regardless.

• coinstatsindex=1 keeps the coin statistics index live (coinstats.dat, the running MuHash and totals at the applied tip) and, since 2026-09-08, one row per committed height in coinstats_hist.dat (1 KB each: the set's counters, both MuHash accumulators, the cumulative per-block amounts), so gettxoutsetinfo <hash_type> <hash_or_height> answers at any height the rows cover, in Core's shape, block_info included. A boot with no valid persisted state seeds the index from a walk of the set (minutes); rows exist from that height on in the TAIL (coinstats_hist.dat), and a query below it is refused by name until the BASE exists: coinstats_hist_base.dat, rows 0..to_height, written by daemon/bmc_build_coinstats_hist <chaindir> [to_height] [workers] from the archive alone (about two hours and ~500 GB of scratch under csh_tmp/ on mainnet with eight workers; the daemon may keep running). The reader prefers the base for every height it covers. Its numbers are Core's coinstatsindex's: verified byte for byte at height 963,967 and at the tip. mempool.space reads txouts and block_info.prevout_spent per indexed block once getindexinfo reports the index synced at that height. Off (0) the index is not maintained and the RPC refuses. The base repairs itself (2026-09-08, after a build died in an OOM and its half-written rows and 209 GB of leftovers went unnoticed): once a heartbeat the fold worker checks the base (header every time; every row, the header's hash and the seam with the tail the first time) and, when it is absent, spawns the builder beside its own executable, niced, never during initial block download, at most three attempts per boot with six hours between them; bmc.coinstatshistrepair=0 turns that off. A node synced from block 0 skips it (2026-10-04): its tail starts at genesis, so after the download the fold worker checks every tail row once (hash and one generation, no re-seed in between) and answers from the tail; a torn row or a re-seed falls back to the build. A base that fails the check is renamed *.broken-<epoch> (delete those when you have looked) and rebuilt. The builder resumes at the pass after its last csh_tmp/passN.done marker, discards scratch without a marker (and the old layout's csh_*.tmp), sizes pass 3 from free RAM, holds csh_tmp/lock so two builders never share a scratch, and writes the base to a temp name that is renamed into place with the complete flag set last. gettxoutsetinfo below the base names the state ("being rebuilt", "waits for initial block download", ...); coinstats_hist.status in the chain directory carries the same line.

• bmc.esploraport=<port> (2026-09-08) opens the Esplora facade for mempool.space's BACKEND: "esplora": a second, unauthenticated listener (bmc.esplorabind, default 127.0.0.1) answering Esplora's REST routes from the node's own RPC handlers in process. Keep it on loopback or behind a proxy. Address routes need the address history index (above). See releases/2026-09-08-esplora-facade.md.

• txindex=1 (Core's meaning since 2026-09-16): the daemon builds the txid index during the sync as sorted runs (txindex.dat if a base exists, then txindex.r<from>-<to>.dat) trailing the applied height by 144, with txindex.tail for the heights above the highest run; the tail is rotated after every run and the runs merged at six (bmc_merge_index_runs). getrawtransaction <txid> asks every run, then the tail. Nothing waits for initial block download, and the same builders keep trailing after it.

• txospenderindex=1 (2026-09-16): the same shape for Core's txo-spender index (txospender.dat / txospender.r*.dat / txospender.tail); gettxspendingprevout answers confirmed spends from the first run on. The whole-chain builders (daemon/bmc_build_tx_index, daemon/bmc_build_txospender_index <datadir> [from] [to] [out]) remain for a one-off rebuild; a run is the same file under a run name.

• bmc.indexrunblocks=<n> (default 20000, 1000..200000): heights per run for the three trailing builders above. The unsorted tail a lookup scans linearly is never longer than this; a run costs one archive walk over its range, at nice 10, one builder per index at a time.

• blockfilterindex=1: the daemon creates bfilters.dat/bfilters.idx itself and builds every filter from genesis at the new-block choke point (prevouts from undo), closing any gap in slices of 256 heights per block so a catch-up never stalls the apply path. The offline backfill (daemon/bmc_build_block_filters) is no longer part of the path. coinstatsindex=1 is opt-in as in Core (default 0 since 2026-09-06) and, since 2026-09-16, survives the two pre-BIP34 duplicate coinbases on a fresh sync (the overwrite is a remove + add, as Core's coinstatsindex does; it used to invalidate the index at 91,842 and re-seed only at the next boot).

• assumevalid=<hash> skips script evaluation for blocks at and below that block (PoW, merkle, structure and every UTXO check still run); the height is resolved from the archive at boot ([utxo_live] assumevalid: block found at height N). Unset = the chain's built-in block (Core v31: mainnet 938343, testnet4 123613, signet 293175; regtest none). assumevalid=0 evaluates every script of every block.

• pid=<file> writes the pid once the RPC port is bound. blocknotify, alertnotify, startupnotify, shutdownnotify run a shell command with %s sanitised to [A-Za-z0-9._:/-].

• bmc.utxocompactthreshold (default 12) is the run count that triggers a background compaction; bmc.bootcatchup=0 skips the boot-time parallel download.

• bmc.dlcchunk=<n> (default 16, 4..64; 2026-09-30, default 40 until 2026-10-01) is the blocks per request of the parallel download: each worker asks one peer for a chunk of this many blocks in one getdata. Core keeps 16 in flight per peer, refilled as each lands. The [dlc] Core's shape: line prints the value in use and the staged-chunk count at which the committer asks a worker to fetch the chunk the apply is waiting on.

• debuglogfile=<path> relocates the daemon's own leveled log (relative to the chain directory or absolute; 0 disables it); disablewallet=1 loads no wallet; bytespersigop (default 20) sets the sigop-adjusted size used for fee-rate checks.

Running as a service

Reference unit, /etc/systemd/system/bmcbitcoind.service:

[Unit]
Description=Bitcoin Machine Code Daemon (experimental AI-authored asm node)
After=network-online.target
Wants=network-online.target
StartLimitIntervalSec=600
StartLimitBurst=5

[Service]
Type=simple
User=<service-user>
Group=<service-group>
WorkingDirectory=/path/to/repo/asm
ExecStart=/path/to/repo/asm/daemon/bmcbitcoind.live serve /path/to/repo/data
StandardOutput=append:/path/to/repo/logs/main/bitcoin.main.log
StandardError=append:/path/to/repo/logs/main/bitcoin.main.log
Restart=on-failure
RestartSec=10
TimeoutStopSec=90
KillSignal=SIGTERM
# Hardening (audit 2026-09-02 N5). This process auto-unlocks the wallet at
# boot and holds the decrypted seed for its lifetime: no core dumps (a dump is
# the debugger-backtrace leak of 2026-08-30 in file form), no privilege gain,
# no /home, a private /tmp. ProtectSystem=full leaves /etc read-only, which is
# all the daemon needs from it.
LimitCORE=0
NoNewPrivileges=yes
ProtectSystem=full
ProtectHome=read-only
PrivateTmp=yes

[Install]
WantedBy=multi-user.target

Drop-ins in bmcbitcoind.service.d/ in the reference deployment raise TimeoutStopSec=900 (headroom for a compaction in progress at stop time), add SupplementaryGroups=debian-tor (tor control cookie access), and carry the hardening block above as 50-hardening.conf. The block is reproduced inline here so that a unit built from this page is hardened without the drop-in; the reference deployment's unit itself is a local artifact and is deliberately not vendored into this repository (operator decision, 2026-09-05 — see releases/2026-09-05-audits-closed.md).

• The reference deployment's data/bitcoin.conf (2026-09-08) has five keys: port=8332 and rpcport=8331 (the operator's choice: the router forwards 8332 to this node's P2P side, and the ports the 09-07 build happened to use are the ones peers and tooling know); bind=<host address> because a Core oracle on the same host listens on 127.0.0.1-16:8333 for the benchmark's loopback replay and Linux refuses a wildcard bind against a listening specific address on the same port; debuglogfile=0 and printtoconsole=1 so the log keeps flowing through the unit's capture below rather than <chaindir>/debug.log.
• ExecStart runs bmcbitcoind.live, a symlink to a dated snapshot bmcbitcoind.deploy-<YYYYMMDD><letter>, not the tree binary. A rebuild changes nothing until the symlink moves; systemctl restart boots whatever it points at.
• The datadir argument is the root; chain= selects <datadir>/<chain>.
• stdout/stderr append to logs/<chain>/bitcoin.<chain>.log.
• SIGTERM is honoured (the parent forwards it to the download worker); TimeoutStopSec is the wait before systemd SIGKILLs.
• Restart=on-failure, RestartSec=10, at most 5 starts per 600 s.
• No LimitNOFILE is set, so the process gets the default soft limit of 1,024 descriptors (Max open files in /proc/<pid>/limits). A healthy production process holds about 66. The Core oracle's unit sets 65,536. On 2026-10-07 a leak used all 1,024 and RPC stopped accepting for 2 h 17 m (devlog/INCIDENT_2026-10-07_reader_lane_fd_leak.md). Raising the limit is an open decision. It would only buy time against a leak, so keep the descriptor check below either way.

sudo systemctl daemon-reload
sudo systemctl enable --now bmcbitcoind
systemctl is-active bmcbitcoind

Log rotation

config/logrotate-bmc.conf rotates logs/*/*.log when a file exceeds size 2M (the knob), keeps 60 compressed rotations, and uses copytruncate so systemd's append descriptor stays valid. It is driven by bmc-logrotate.timer (5 min after boot, then every 15 min).

The timer reads /etc/bmc/logrotate-bmc.conf, not the repo copy. logrotate 3.21 refuses a config that is group-writable or not root-owned, so a file inside the service-user-owned checkout can never be the live config -- pointing the timer at the repo copy is why rotation silently did nothing for a week (audit 2026-09-02, N6).

The repo copy is the SOURCE. Its su line carries a __SVCUSER__ placeholder rather than a hard-coded account name, because the account name is host configuration and committing it publishes who runs the node. Substitute and install in one deliberate step, run AS the service user so id -un resolves to the right account:

sudo sed "s/__SVCUSER__/$(id -un)/g" config/logrotate-bmc.conf \
  | sudo install -m 0644 -o root -g root /dev/stdin /etc/bmc/logrotate-bmc.conf

A leftover __SVCUSER__ makes logrotate fail loudly on the next tick, which is the safe direction -- a wrong-but-plausible username would quietly rotate as the wrong account.

Upgrading and rollback

An upgrade is build, audit, gate, snapshot, relink, restart. Snapshots are host-local build products, not in git; keep the last few.

cd /path/to/repo/asm
git pull
make daemon/bmcbitcoind          # safe while the old binary runs: ld unlinks first

# the three build audits (each is also a prerequisite of `make test`)
make prereq-check             # every file a recipe uses is a declared prerequisite
make link-check               # every rule links the files defining the symbols it needs
make header-check             # every rule that compiles daemon/main.c lists the headers it includes
make runlist-check            # every test is gated or declared manual with a reason
make abi-check                # SysV stack-alignment audit of asm->C call sites

# the gate: record make's OWN exit status, then audit the finished log
make -k test > gate.log 2>&1; echo MAKE_EXIT=$? >> gate.log
make gate-log-check LOG=gate.log
echo GATE_CHECK_EXIT=$?       # act on THIS status only

gate-log-check (scripts/gate_log_audit.py) exits 0 only if make exited 0, every test named in the test: recipe actually ran, and no failure or crash marker appears; it prints GATE LOG AUDIT OK or GATE LOG AUDIT FAILED with the tests that never ran. Read its exit status directly, never through a pipe (a pipeline reports the last command's status), and do not grep the log for failure words: a gate whose test: recipe never ran because a prerequisite failed to link contains no FAIL lines.

Deploy:

cp -a daemon/bmcbitcoind daemon/bmcbitcoind.deploy-$(date +%Y%m%d)a   # next free letter
ln -sfn bmcbitcoind.deploy-$(date +%Y%m%d)a daemon/bmcbitcoind.live
sudo systemctl restart bmcbitcoind

Letters advance per deploy on the same day (a..z, aa, ...). Never cp onto the path of a running binary (ETXTBSY); write a new name and relink. Copy the wallet files (see Backup) before a deploy that changes wallet or index formats.

Rollback:

ln -sfn bmcbitcoind.deploy-<previous> daemon/bmcbitcoind.live
sudo systemctl restart bmcbitcoind

Not every earlier snapshot is a safe rollback target (as of 2026-10-07):

• Nothing before deploy-20261006d. B3's async flush (#396, first live in 06d) retires the UTXO WAL's flushed generations: it records the first live byte in utxo.idx and punches a hole over the bytes before it. Earlier builds replay the WAL from byte 0 and would drop tombstones from a datadir that 06d or later has written. That includes deploy-20261006c, the snapshot immediately before. Do not roll back past 06d without working out the datadir first.
• deploy-20261006d leaks three descriptors per Esplora facade request (the getblock reader lane, fixed by #402, as does every build from deploy-20261005a). It is the only rollback target for deploy-20261007a. It runs, but RPC stops accepting within hours to a day under mempool.space, so use it only as a bridge and watch the descriptor count.

The current production snapshot and its predecessors, with what each one carried, are in devlog/DEPLOYMENT_HISTORY.md.

A deploy is verified only when all four hold:

1. a NEW block arrives and is applied: its hash equals the oracle's, and the heartbeat shows tip == stored;
2. at least 10 minutes in, systemctl show -p NRestarts,ActiveState bmcbitcoind still reads 0 / active;
3. the RPC clients (mempool.space, the facade) have reconnected meanwhile;
4. the descriptor count stays flat across that client traffic: ls /proc/$(systemctl show -p MainPID --value bmcbitcoind)/fd | wc -l (about 66 on deploy-20261007a), well under Max open files.

The second, third and fourth are not optional. On 2026-10-02 a build applied its first block correctly, then crash-looped ~40 s after every start once mempool.space re-synced through the facade, until systemd gave up. One good block proves only the apply path. On 2026-10-07 a build that had passed the first three for two days ran out of descriptors: every facade request leaked three, and only a count taken over time shows that.

The Core oracle

One Bitcoin Core node is the reference everything here is measured against: the IBD harness's capstone, the RPC and REST differentials, the mempool differential, the next-block check after a deploy. Since 2026-10-01 it is the v31.1 release, the version this node targets. A v31.99 development build used to sit on these paths; it was retired because diffing against it invented work (docs/PARITY_RPC_FIELDS.md).

unit	bitcoin-oracle (config/core-oracle/bitcoin-oracle.service, __SVCUSER__ substituted at install as for logrotate): SIGTERM only, TimeoutStopSec=900, never SIGKILL -- Core flushes a large chainstate on the way down
binaries	/storage/bitcoin-core-v31.1/bin (bitcoind, bitcoin-cli); source tree /storage/bitcoin-core-v31.1/source
datadir	/storage/core-oracle, config config/core-oracle/bitcoin.conf: txindex, coinstatsindex, blockfilterindex, dbcache=8192
network	P2P 8333 on 127.0.0.1-16 (the loopback replay benchmarks dial them), RPC 8335, ZMQ 28432
query	/storage/bitcoin-core-v31.1/bin/bitcoin-cli -conf=/storage/core-oracle/bitcoin.conf -datadir=/storage/core-oracle getblockcount

It lives on /storage, not on the benchmark NVMe, so it does not compete with a timed run for I/O. Manage it with sudo systemctl {status,restart,stop} bitcoin-oracle; never start a second copy by hand.

Verifying a restart

The RPC server comes up about 40 s after start on the reference host; allow about a minute. Check, in order:

systemctl is-active bmcbitcoind
ss -ltnp | grep bmcbitcoind            # P2P, RPC, onion target, ZMQ endpoints
daemon/bmc_cli -datadir=/path/to/repo/data getblockcount

Expected lines in logs/<chain>/bitcoin.<chain>.log, in order:

===== bmcbitcoind  LOG START: ...   pid N  vX.Y.Z  built ...  mode=serve
[config] loaded <path>: N setting(s) applied
[net] BIP324 v2 transport enabled (services=0x...); N of M known peers advertise v2
[boot] chain=<chain> datadir=<datadir>/<chain> port=<p> dnsseed=<0|1>
[boot] tx-validation snapshot ready (NN.NNs) -- inbound peers inherit it
[boot] chain archive loaded: tip=<h> (N.NNs)
[boot] archive check clean (N.NNs)         # or "found N problem(s) -- see [check] lines above"
[boot] catch-up check done: N block(s) written
[boot] boot phase complete (NN.NNs total)
[rpc] no rpcuser/rpcpassword -- using cookie authentication
[rpc] encrypted wallet adopted and unlocked from the configured passphrase source
                                           # or "adopted (locked -- use walletpassphrase)"
[rpc] JSON-RPC server on 127.0.0.1:<rpcport> (live-node + chain, user=...)
[rpc] cookie authentication enabled (.cookie, mode 0600)
[utxo_live] init dir=... reload applied_height=<h> manifest_n=N live=<count>
[mempool] loaded mempool.dat: N accepted, M rejected of T (...)
[tor] onion service <addr>.onion:<port> -> 127.0.0.1:<port+1> (key onion_v3_private_key)
serving on port <p> (N outbound peer(s))...

applied_height equals the archive tip once caught up, and the heartbeat's tip tracks the network within a block. The mempool reload runs after the RPC server is listening, so getrawmempool is briefly partial. The onion service line appears a few minutes after start.

Reading the parallel catch-up ([dlc]) since 2026-09-07

Two lines every 10 s (since 2026-09-08; before that the tick was eight lines):

[dlc] == elapsed 2:32:32 | eta 00:10:31:52 | overall: 442051/965954 stored (45.76% of real tip) | in flight 232 of window 4096 through 442282 (oldest gap 0s at 441270, 99.95% landed) | applied=440754 lag=515 ==
[dlc] -- recv 11.0MB/s (avg 9.8MB/s) | write 10.9MB/s (avg 10.1MB/s) | floor 32.0 KB/s (median 812.3) | banned 1/126 | events 0 rot 0 wait 0 help 0 fail 0 abandon (run 780/545/64/68/0) --

• eta is DD:HH:MM:SS at the last ten minutes' block rate; blocks grow toward the tip, so on the tail it reads optimistic. --:--:--:-- until a rate exists.
• in flight N of window W is the claimed-not-yet-landed set. With the defaults since 2026-10-01 (10 workers, bmc.catchupworkers, on 16-block chunks, bmc.dlcchunk) it is at most a few hundred. It is bounded by the window: Core's 1,024 blocks above the connected tip (since 2026-09-29), or workers × chunk if that is larger. The archive consolidates behind it.
• oldest gap Ns at h is how long height h has been the lowest unfilled height. A gap that stays oldest for 60 s prints as STRANDED: height h has been the oldest gap for Ns — the one line on this subject that deserves a look.
• applied is the UTXO connect's height; lag is how far behind the contiguous prefix it is.
• The dashed line carries this tick's network receive and disk write rates with their averages since the start, the dead-weight floor with the pool median it derives from, how many peers are banned right now, and the event counts since the last tick with the run totals in brackets: rotations (a worker left a peer under half the pool median at a chunk boundary, nothing discarded), window waits (a worker paused rather than claim past the window), helps (an idle worker fetched the blocking chunk), failed attempts (a chunk fetch that failed on a peer; the worker backs off and the peer's standing halves), abandons (a chunk that went to the retry ring after 400 attempts). Before the event counts, staged N commit M (2026-09-08): chunks waiting for the in-order committer right now, and chunks it has appended. staged above a handful for more than a tick means the chunk at the first hole has not arrived. While the window is full, the stall rule below judges whoever holds it.
• The NODE_WITNESS drop count prints only when it changes.
• The 16-line peer table prints every five minutes.
• [dl] new block and [dl] announced tip are silent while the tip is older than maxtipage (Core's definition of initial block download, during which Core relays no blocks); one line says so, and at the tip both return, one per block. A compaction is one line, at completion.

Lines that still print one per event, because each one is worth reading: chunk [lo,hi] ABANDONED -> retry ring, dead weight (...) (the parent's pool-relative verdict, with the measured rate and the bar; since 2026-10-08, #409, the rate covers only the part of the 10 s tick the worker was free to fetch, and a tick it spent mostly waiting at the full window is not judged at all), stalled: no block for 120s, attempt N failed after M ms: <reason> (only the third in a row on one chunk and every hundredth), and REJECT h=.

The download's other one-per-event lines (2026-10-05 to 10-08):

line	meaning
[dlc] header probe: the first page from N candidate(s) in Xs: <peer> R KB/s (...), ... -- <peer> leads, the rest are fallbacks	B5: the first header page was asked of four candidates. The fastest leads the header download.
[dlc] headers from X fell to N KB/s over its last 4 page(s), under half of Y's probed M KB/s -- switching at height H (+S stored, P page(s) held and carried); Y continues from there	B12 (#404): the header leader slowed. The next candidate continues from H, with nothing re-downloaded. Never printed under bmc.dlshape=core.
[dlc] header ranges: N range(s) from height A to B (anchors every S) across P peer(s), the probe's fastest	B12 ranges (#410): the probe's answering peers (up to 4) fetch the chain in parallel, one range between two anchor hashes at a time. The anchors are mainnet block hashes every 50,000 in chainparams (see CORE_DIVERGENCES.md). Not printed under bmc.dlshape=core, with fewer than two answering peers, or on a chain without anchors.
[dlc] header ranges: K of N range(s) in Xs (MB, rate): <peer> r range(s)/p page(s), ...; heights a..b are contiguous[ -- the sequential fetch continues from there]	the end of the parallel fetch. Each range was checked for linkage and proof of work and ends on its anchor's hash. Only the contiguous prefix from our tip is used.
[dlc] header ranges: range j (heights a..b) was given up | was not finished after N attempt(s)	the first gap. A range that failed 3 times is given up; the sequential fetch takes it and everything after.
[dlc] header ranges: range j held by P for Xs against a median of Ys -- taken back for a waiting peer | and given up	a straggler: held over 3x the finished ranges' median (at least 5 s) with nothing left to hand out. Its peer is stopped and the range handed back.
[dlc] header ranges: +N stored through the checks (total T)[, the rest held below -minimumchainwork through height H]	the prefix stored by the same code as a peer's pages, every check and the -minimumchainwork hold included. A hold left over is carried into the sequential fetch. the replay refused them -- nothing from them stored means a check failed, and the sequential fetch starts from the tip as before.
[dlc] headers from X stopped answering after +N (chain at H) -- the next candidate takes over	the leader went silent. What it served is rolled back and the next candidate starts over.
[dlc] wN <peer> is stalling the window: chunk [lo,hi] is the oldest missing and the window (W above A) is full -- dropped after N s (next timeout T s; peer <verdict>) | holder <phase> for N s, ...	Core's stall rule. The verdict is disconnected, a first stall (banned on a second) or BANNED for the run (its second stall) (B13), disconnected, not banned (Core) under bmc.dlshape=core, or one of the guards: manual, kept selectable, at the usable floor, kept selectable, already banned.
[dlc] chunk [lo,hi] goes to the retry ring for an idle worker as well (Core: ...)	the stalled chunk was handed to an idle worker at once, as Core re-requests a disconnected staller's blocks.
[dlc] connected N block(s) during the download; connected tip T (...); idle waits cut short N (B9)	end of the download: blocks the applier connected while downloading, and how often the idle tick was cut short by a new tip (B9 part 2).
[dl] peers announce N but the pool's median claim was M Ns ago -- not believed (...); using K	B10: an announced height the chain could not have reached is clamped.
[ready] ...	the finish line: every index at the tip (bmc.benchlog).

Operating

Logs

• The log: <datadir>/<chain>/debug.log, Core's file in Core's place. Since 2026-09-08 everything the daemon prints goes there by default, every [boot], [config], [rpc], [dl], [dlc], [utxo_live], [tx_accept], [tor], [zmq] line, from every forked child. Before that stderr was the log and only the assembly logger's few lines reached debug.log. printtoconsole=1 sends the same lines to the console as well (Core's semantics); debuglogfile=0 turns the file off. shrinkdebugfile=1 (default) keeps the last 200 KB of a file over 10 MB at start-up; between restarts the file grows, as Core's does, so rotate it (config/logrotate-bmc.conf covers data/*/debug.log with copytruncate).
• What still reaches the launcher's stderr: the lines printed before the chain directory is entered, i.e. the config echo, chain selection and a datadir-lock failure, and a [boot] logging to ... line naming the file. Under systemd those land in logs/<chain>/bitcoin.<chain>.log. A deployment that wants the old arrangement, everything on stderr and nothing in the chain directory, sets printtoconsole=1 and debuglogfile=0.
• journalctl -u bmcbitcoind holds only systemd's lines (start, stop, Killing on a stop timeout).

Lines to monitor

line	meaning
[dl] heartbeat: tip=.. peers=a/b txouts=.. uptime=..	the pulse; tip should track the network
[tx_accept] last 30s: +N accepted (mempool M) | rejected: N missing-inputs, N invalid, N policy | N already confirmed	relay summary; missing-inputs rejects are normal churn
[txrelay] last 60s ..., [txrelay] orphans: ...	relay and orphan-pool counters
[pool] N peer(s) sampled from the book: ipv4 a, ipv6 b, onion c, i2p d, cjdns e (...)	the dial pool's per-network composition at boot; anonymity networks appear only when their transport is configured
[mempool] block <h>: removed N pool tx (confirmed/conflicted)	a block connected
compaction of N run(s) ... started in background pid P / background compaction done in Xs	UTXO maintenance
[utxo_live] merge of N run(s) deferred: the apply is X blocks behind the archive (waits under M runs)	during a sync, a merge waits while the apply is 256 or more blocks behind, until twice the count threshold (48 → 96 runs). Since #408 it starts at once when the run files pass the run budget (35% of RAM), at any count. A deferral that lasts more than a few minutes with the apply falling further behind is run 42's 1 h 35 m stall. Look for the compaction done line.
RECOVERY: rolled back N ghost block(s), rolled back ghost application, init: swept N orphan file(s)	healthy crash recovery at boot
[utxo_live] flush: async (...), [bench] freeze ...	B3: the memtable is frozen and a forked writer flushes it. The writer is a short-lived child that is adopted between blocks.
[rpc] accept: out of file descriptors (...) -- backing off; the RPC listener is degraded	printed once per process. RPC and the facade are no longer accepting. See Troubleshooting.
REJECT, FATAL, DEGRADED, ghost-rollback FAILED, INCONSISTENT	alert on these

RPC access

daemon/bmc_cli -datadir=/path/to/repo/data getblockchaininfo
daemon/bmc_cli -datadir=/path/to/repo/data -chain=signet getblockcount
daemon/bmc_cli -rpcport=<n> -rpcuser=<u> -rpcpassword=<p> <method> [params...]

gettxoutsetinfo and scantxoutset refuse while the UTXO set is being written; call again between blocks or stop the daemon for an authoritative value. gettxout retries briefly while the worker is busy.

Wallet

Wallet files live in the chain directory by default; walletdir=<dir> (absolute, or relative to the chain directory) moves them -- the default wallet's store and journal, the key and scan records, and every named wallet under <walletdir>/wallets/<name>/. The directory is created 0700 if absent. Keeping it outside the datadir keeps wallet keys out of chain backups; the passphrase file must still live elsewhere again.

With bmcwallet.enc present the wallet boots locked unless a passphrase source is configured. walletpassphrase "<pass>" <seconds> unlocks it for that long, walletlock re-locks, walletpassphrasechange re-wraps in place. encryptwallet on a plaintext bmcwallet.dat seals it and writes bmcwallet.enc. There is no recovery without the passphrase.

Shutdown

systemctl stop bmcbitcoind sends SIGTERM. The parent logs [serve] shutting down (signal 15): tip=..., saves the mempool ([mempool] saved N transaction(s) to mempool.dat), forwards SIGTERM to the download worker ([dl] shutting down (signal 15): ...), runs shutdownnotify if set, removes .cookie and the pidfile, and exits. A normal stop takes seconds. A signal during the UTXO reload exits immediately, leaving the state the previous clean shutdown left.

If a stop stalls: after TimeoutStopSec systemd sends SIGKILL (journalctl shows Killing) and boot recovery repairs the interrupted write. To intervene by hand, identify the process by PID and readlink /proc/<pid>/exe (it points at the deployed snapshot) and signal that PID. Do not use a broad pkill -f bitcoind: other Bitcoin daemons on the host match it. A compaction child shares the daemon's command line; killing it is harmless (the merge is redone).

Backup and recovery

item	files in <datadir>/<chain>/	notes
wallet	bmcwallet.enc (or bmcwallet.dat + bmcwallet.dat.txlog), walletkeys.dat, walletscan.dat	copy with the daemon stopped
wallet passphrase	/etc/bmc/wallet.pass (outside the datadir)	store separately from the wallet backup
network identity (optional)	onion_v3_private_key, i2p_private_key	private keys; keep to preserve the .onion/.b32.i2p address
address book (optional)	peers2.dat	faster re-bootstrap
configuration	config/bitcoin.conf, unit file and drop-ins	holds any rpcauth/rpcuser credential

The block archive, headers, UTXO store, indexes and mempool.dat are re-downloadable or rebuildable. Never keep the passphrase file with the wallet files: a backup carrying both is the failure the walletpassfile rules prevent, which is why the daemon refuses a passphrase file inside the datadir. Do not copy UTXO files or point tools at a datadir while the daemon writes it; stop it first. daemon/bmc_utxo_setinfo <datadir> --muhash on a stopped datadir is the instrument for comparing the set with a trusted node's gettxoutsetinfo muhash. Live, ask both nodes for the same height (gettxoutsetinfo muhash <height>): the no-height answer was printed byte-reversed on an indexed node from 2026-09-25 to 2026-09-27 (#321), and Core's no-height call forces a UTXO flush on the oracle.

A coin the node calls spent that a trusted node calls unspent (a block rejected for "input references a missing/already-spent UTXO" that Core accepted): daemon/bmc_utxo_probe_one <datadir> <txid> <vout> resolves the outpoint through both read paths, the point lookup and the full walk. Both missing: the set lacks the coin. Walk found, lookup missed: a run file is at fault -- symlink the run files into a scratch directory with a hand-written one-entry manifest (UMN2, n, total_live, then (gen, run_no) pairs) and probe each run alone. Bench run 18 (2026-09-09) found a run holding a key twice, a PUSH then a DEL, from two defects fixed that day (docs/releases/2026-09-09-one-record-per-key.md); a run written before the fix is repaired by its next compaction.

Recovery:

• Unclean stop or crash. Start the service. Boot rolls back any ghost block application from undo_<h>.dat before anything reads the set, sweeps orphan run files and re-applies. No operator action.
• Missing archive blocks. The node fills gaps and the tail itself ([boot] checking for archive gaps / missing blocks...).
• Corrupt or lost block index. Set reindex=1 in bitcoin.conf and start. Before opening the archive the node rebuilds index.dat, headers.dat and chainwork.dat from the blk*.dat frames alone -- every block on disk is re-linked from genesis by prev-hash, the best chain by cumulative work wins, and duplicates, orphans, stale forks and frames that fail their own proof of work are left out -- then drops the UTXO set and the height-positional indexes so they rebuild. The previous index files stay as *.pre-reindex. The key is one-shot (reindex.done marks it done); remove it afterwards. The rebuild itself takes seconds (an archive of 150,000 blocks in 93 files rebuilds in about five); the UTXO rebuild that follows is the long part. The log line to expect is [reindex] rebuilt: tip=<h> from <n> frame(s) in <f> file(s); ....
• Reorg rehearsal. asm/tests/reorg_drill <datadir-COPY> --depth N disconnects and reconnects the last N blocks on a copy and requires the UTXO walk and tip hash to return to their prior values. It refuses the production datadir by path.

Maintenance

Archive re-layout

Archives filled by the parallel downloader BEFORE 2026-09-08 are not in height order, and every boot reports [check] block data is NOT laid out monotonically (first break at height H) -- truncation and pruning will refuse to run. This is informational; only in-place pruning and physical truncation refuse (the whole-file pruner and the index-only truncation run instead). Since 2026-09-08 the download's in-order committer appends the archive in height order from a single process (docs/releases/2026-09-08-in-order-committer.md), so a fresh sync passes the check; the tool below is for archives built earlier.

tests/tool_archive_relayout <archive-dir> <out-dir> reads index.dat, copies every block frame into new blk*.dat files in height order (128 MiB rotation), writes a matching index.dat, verifies every block hash and leaves the result in <out-dir>. It reads only index-committed frames, so it can run against a live daemon; blocks appended meanwhile are not captured and are re-fetched at the next boot. The swap is done stopped:

tests/tool_archive_relayout /path/to/data/main /path/to/scratch/main-relayout
sudo systemctl stop bmcbitcoind
mkdir /path/to/backup && mv /path/to/data/main/blk*.dat /path/to/data/main/index.dat /path/to/backup/
mv /path/to/scratch/main-relayout/* /path/to/data/main/
sudo systemctl start bmcbitcoind

Keep the old files until the new boot's archive check is clean; they are the rollback. The scratch copy needs as much space as the archive.

Troubleshooting

symptom	cause and action
bmc_cli connection refused right after start	RPC is not up yet. Boot runs config, the tx-validation snapshot (tens of seconds), archive load and check, catch-up check and hash index before starting RPC (about a minute). Wait for [rpc] JSON-RPC server on ....
active service, but no RPC ever	rpcport equals port (the RPC bind is lost), or no credential source exists. Read the [rpc] lines.
bind failure at start / node exits	The port is in use. ss -ltnp | grep :<port> shows the PID holding it: a stale daemon from a previous run or another node. Stop that PID.
systemctl stop stalls	See Shutdown. After TimeoutStopSec systemd SIGKILLs; boot recovery repairs the interrupted write.
[zmq] notification ring overrun: N transaction(s) not published (total T)	Producers lapped the 64-slot ring (typical during the mempool reload). Harmless unless a subscriber needs every event; a subscriber must resync via RPC on a per-topic sequence gap, as with Core. Logged at most once a minute; the total is cumulative.
missing-inputs rejects in the [tx_accept] last 30s summary	Normal relay churn (parents not yet seen). The summary is the record; there is no per-event line.
[tor] no onion service: ...	Control-port authentication failed or the port is unreachable. Check torcontrol=, cookie readability (SupplementaryGroups=debian-tor) or set torpassword=. Outbound onion is unaffected.
[dial] no IPv6 on this host: ipv6 and cjdns peers are unreachable	Enable host IPv6 (and run cjdroute) for cjdnsreachable=1.
[wallet] walletpassfile "..." not usable: <why> / is inside the datadir -- refusing	Fix path and mode (absolute, outside the datadir, 0640 or stricter, not group-writable). The wallet stays locked until then.
[boot] archive check found N problem(s)	Read the [check] lines above it, and do NOT assume the non-monotonic layout notice is benign. It was expected before the in-order committer (2026-09-08) and is a DEFECT after it: a fresh sync on this build must pass the check. On run 26 it meant six layout breaks, one per restart during the download, each scattering ~33 blocks into the tail gaps of older blk files — which disables truncation and pruning. Fixed by the append-frontier guard (2026-09-17); an archive built before that keeps its breaks until tests/tool_archive_relayout rewrites it. Other findings name the height.
systemctl status: failed (Result: core-dump), "Start request repeated too quickly"	A crash loop: systemd stopped restarting. Roll back first (above), then sudo systemctl reset-failed bmcbitcoind and start. The cause is in the log's [crash] lines (next row).
[crash] SIGSEGV (11) at address ...	A fatal signal (2026-10-03, daemon/crash_trace.c). The lines give the fault address, RIP/RSP/RBP, a backtrace, and the stack's return addresses; resolve each with addr2line -fe asm/daemon/bmcbitcoind.deploy-<x> <addr> (the binary is non-PIE). There is no core file, by design: LimitCORE=0 keeps the decrypted wallet seed off disk, and the report prints addresses only.
[rpc] getmininginfo took N ms: refresh ..., networkhashps ..., ...	One call held the exclusive RPC lock over 1 s; every other RPC waited behind it. The split names the slow step. Seen once after a restart into catch-up (149.5 s on 2026-10-03), cause still open.
a second bmcbitcoind with the same command line	A child of the running daemon: the serve worker, a compaction, the B3 flush writer (seconds per freeze), the B4 index worker or the coinstats builder. Check /proc/<pid>/exe and the parent PID before assuming a duplicate daemon; never run two daemons on one chain directory. A <defunct> flush writer that lasts longer than a block interval was the pre-#400 shape (fixed 2026-10-07).
RPC and the facade time out, but is-active is fine and the tip advances; mempool.space stalls	Check descriptors first: ls /proc/<pid>/fd | wc -l against Max open files in /proc/<pid>/limits, and grep the log for accept: out of file descriptors. At the limit, P2P keeps working because its sockets are already open, while accept() fails. Roll forward to a build without the leak, or restart as a stopgap. 2026-10-07: devlog/INCIDENT_2026-10-07_reader_lane_fd_leak.md.

Running more than one chain

One bitcoin.conf selects one chain, so a second chain needs its own configuration:

1. Separate datadir. Give the second chain its own datadir root with its own bitcoin.conf (chain=signet, distinct port=/rpcport=) and start a second daemon on it, by hand or as a copy of the unit with a different name, ExecStart datadir and log path (logs/signet/bitcoin.signet.log, already covered by the logrotate glob).
2. Wrapper datadir with a symlinked chain directory. The wrapper <wrapper>/bitcoin.conf selects the chain and <wrapper>/<chain>/ holds the state, while <maindatadir>/<chain> is a symlink to <wrapper>/<chain>. The main datadir then shows every chain under one root, and each daemon reads its own configuration.

Ports must not collide between daemons. Each daemon writes its own .cookie in its chain directory, so bmc_cli -datadir=<that datadir> resolves the right one. Regtest and testnet4 configurations usually pin peers with connect=<ip:port>, which disables discovery and DNS seeding.

Reference

Default ports per chain (asm/daemon/chainparams.c)

chain	P2P	RPC	magic (wire)
main	8333	8332	f9 be b4 d9
testnet4	48333	48332	1c 16 3f 28
signet	38333	38332	0a 03 cf 40 (derived from the challenge)
regtest	18444	18443	fa bf b5 da

Extra listeners: onion service target at chain default P2P port + 1 (loopback); ZMQ endpoints as configured (commonly 28332/28333).

File inventory (<datadir>/<chain>/)

file	contents
blk*.dat, index.dat	block archive (framed blocks, 128 MiB rotation) and positional height index (48-byte records)
headers.dat, chainwork.dat	header chain and cumulative work
utxo.dat, utxo.idx, utxo_run_*.dat, utxo_manifest.dat, utxo_applied_height.dat, utxo_lsm_*.map	LSM UTXO store: the WAL; its checkpoint (utxo.idx, the first byte of the generations not yet retired; B3, 2026-10-06; retired bytes are hole-punched); sorted runs; manifest; applied height; and the memtable files. Since B11 (2026-10-07) the memtable lives in anonymous memory, and the .map files are only created at their full size for the offline tools, which read their sizes. Only the table's first page (the live count) is shared through the file. bmc.memtableanon=0 restores the file mappings.
undo_<h>.dat	per-block undo data for the recent tail (reorg and crash rollback)
append.lock	archive append lock
mempool.dat	mempool saved at shutdown, reloaded after RPC start
peers2.dat (peers.dat legacy)	address book
.cookie	RPC cookie, mode 0600, removed at shutdown
bmcwallet.enc / bmcwallet.dat (+ .txlog), walletkeys.dat, walletscan.dat	wallet container / plaintext store and journal, HD keys, rescan records
onion_v3_private_key, i2p_private_key	persisted onion service key and I2P destination
rev%05u.dat + undo.idx	undo data for every block (2026-09-08; Core's rev files). Pruned only with the block store. A node upgraded from the per-height undo_<h>.dat files folds them in at start; older history needs -reindex-chainstate
txindex.dat / txindex.r*-*.dat + txindex.tail, txospender.dat / txospender.r*-*.dat + txospender.tail, addr_hist.dat / addr_hist.r*-*.dat + addrindex.tail, bfilters.dat + bfilters.idx, coinstats.dat + coinstats_hist.dat	optional indexes: sorted runs the daemon builds during the sync plus a bounded tail each (2026-09-16); addr_index.dat is the retired UTXO-by-address snapshot
debug.log	the daemon's log, everything, as Core (since 2026-09-08; before that only the leveled logger's lines; logs/bitcoind.log before 2026-09-06)

Outside the chain directory: <datadir>/bitcoin.conf or <repo>/config/bitcoin.conf; <repo>/logs/<chain>/bitcoin.<chain>.log; <repo>/asm/daemon/bmcbitcoind.live and bmcbitcoind.deploy-*; <repo>/config/logrotate-bmc.conf.

Config options added 2026-09-01 (Core names, Core defaults)

config/bitcoin.sample.conf documents every key. Of note for operators:

• uacomment=<text> (repeatable) appears in the user agent on the wire and in getnetworkinfo; ( ) / \ ; : and control characters are refused.
• peerblockfilters=1 is now required to serve BIP157 filters and to advertise NODE_COMPACT_FILTERS — Core's default is 0 and this node follows it. blockfilterindex= still controls whether the index is maintained.
• rpcwhitelist=<user>:<m1>,<m2> locks an RPC user to a method list (HTTP 403 otherwise); once any whitelist exists, users without one may call nothing unless rpcwhitelistdefault=0. rpcthreads/rpcworkqueue size the HTTP server (503 "Work queue depth exceeded" past the queue); rpcservertimeout bounds a slow client. rpccookieperms=group|all widens the cookie file's mode (0640 / 0644).
• Wallet: addresstype/changetype pick what getnewaddress and change use (a type must be activated with createwalletdescriptor first); fallbackfee is 0 by default, so with no fee estimate a send fails with Core's "Fee estimation failed. Fallbackfee is disabled..." — set it (BTC/kvB) on a fresh node. walletbroadcast=0 returns txids without broadcasting. walletnotify=<cmd> runs with %s = txid on mempool arrival and on confirmation. wallet=<name> loads a named wallet at start-up (one active wallet at a time; further names are logged as skipped).
• Mining: blockmaxweight, blockreservedweight, blockmintxfee (BTC/kvB), blockversion, printpriority shape getblocktemplate.
• Logging: logtimestamps=0, logtimemicros=1, logthreadnames=1, logsourcelocations=1 change the stderr prefix from the point the config is read; shrinkdebugfile (default 1) truncates a debuglogfile over 10 MB to its last 200 KB at start-up. [rpc] exec lock: lines (2026-09-29): every RPC but the lock-free few runs under one execution lock, and four times in three days the whole surface stalled for over 90 s right after a block with nothing to say who held it. A wait or a hold of BMC_RPC_EXEC_LOG_MS or longer (an environment variable on the unit; default 2000 ms, 0 switches it off) now logs one line naming the JSON-RPC method, or the facade / REST route and the method it dispatched: ... getmininginfo (excl) waited 90123 ms; the last exclusive holder was esplora GET /internal/block/<hash>/txs -> getrawtransaction (held 90101 ms); 3 still waiting, and the holder's own ... held 90101 ms (waited 0 ms); 3 waiting behind it. A waiter names the last exclusive holder because that is what the writer-preferring lock made it wait for; a hold shorter than the wait means the wait was behind readers. [mempool] pool lock: lines (2026-09-30): the mempool's own lock is shared by the download worker, every inbound serve child and the RPC process, and the seconds the lines above attributed to getrawmempool, getmempoolinfo and getrawtransaction were spent waiting for it while the worker held it at a new block. A wait or a hold of BMC_MEMPOOL_LOCK_LOG_MS or longer (environment; default 1000 ms, 0 switches it off) logs one line from the process that waited or held, naming the site (the worker's or child's function, or the RPC method) and the pid: ... getrawtransaction (pid 12602) waited 3467 ms; the holder was tx_accept_block_connect_h/rm/reindex (pid 12606, held 3450 ms); 2 still waiting, and the holder's own ... tx_accept_block_connect_h (pid 12606) held 3450 ms (waited 0 ms): fest_begin 0 ms, mark 118 ms, remove_marked 0 ms, rm/seq 12 ms, ..., rm/reindex 2900 ms, seq_C 1 ms; 2 waiting behind it -- the steps are block connect's passes, so the line says which one. Each line is written by the process that waited or held, on its stderr, which every process of the node shares (the worker's, a serve child's and the RPC process's lines land in the same log, interleaved).
• includeconf=<file> (relative to the main file's directory) is read after it; an included file may not include another.
• Core options that are accepted WITHOUT effect are each named at start-up with the reason ([config] <key>= is a Bitcoin Core option that has NO EFFECT here: ...); the full classification of all 181 Core options is in docs/FEATURE_GAPS.md.

Decrypting a wallet by hand (no passphrase on disk)

Nothing needs a .pass file or BMC_WALLET_PASS any more; both still work and take precedence when present.

• wallet_cli getaddress|getprivkey|load [...] on an encrypted wallet with no passphrase supplied asks for it on the terminal with echo off (via /dev/tty, so redirecting stdout is fine). With stdin not a terminal it reads one line from stdin instead: printf '%s\n' "$PASS" | wallet_cli getaddress.
• wallet_cli init with no passphrase argument asks twice; a passphrase entered this way is not written to a .pass file. Enter nothing for a plaintext wallet, as before.
• The RPC client follows Core: bmc_cli -stdinwalletpassphrase walletpassphrase 60 reads the passphrase as the first line of stdin (echo off on a terminal) and sends it as the first parameter, so it never appears in ps or the shell history; -stdin reads the remaining parameters one per line.