NBD device-release race + partscan noise: isDeviceFree predicate too weak, causes dmesg I/O errors

[AdaAibaby]

Symptom

On nodes with rapid sandbox create/destroy churn (dev-sandbox-2, kernel 6.14.0-37), dmesg fills with NBD I/O errors clustered exactly at the moment one sandbox is being torn down while its NBD device is still releasing and a new sandbox's allocation request arrives:

block nbd76: Receive control failed (result -32)
nbd78: detected capacity change from 0 to 47847424
block nbd76: NBD_DISCONNECT
block nbd76: Send disconnect failed -32
I/O error, dev nbd76, sector 0 op 0x0:(READ) ...
Buffer I/O error on dev nbd76, logical block 0, async page read
ldm_validate_partition_table(): Disk read failed.
nbd76: unable to read partition table

Root cause

1. isDeviceFree predicate is too weak (pool.go)

DevicePool.isDeviceFree decides a slot is free from only two signals:

1. /sys/block/nbdX/pid absent (no connected server), and
2. /sys/block/nbdX/size == 0.

But NBD disconnect completes in two stages:

• Synchronous stage: Status.Connected flips false, the pid file disappears, and nbd_bdev_reset calls set_capacity(disk, 0) so size reads 0. These finish inside the disconnect call path.
• Asynchronous stage: completion/cancellation of in-flight blk_mq requests, page-cache invalidation (__invalidate_device), reclaim of readahead / partition-scan reads, and exit of the recv work on nbd->recv_workq. These run on a workqueue and are not guaranteed to be done at the instant size hits 0.

So size==0 && no pid only proves the capacity was reset — it does not prove the device is quiescent. The dmesg Receive control failed (result -32 / -EPIPE) and I/O error sector 0 lines are exactly the async-stage probe reads / in-flight requests hitting a socket that is already closed.

For in-process slot reuse, the usedSlots bit is only cleared in ReleaseDevice, so the same process will not hand the slot out early. But ReclaimLeaked (which, after an orchestrator restart, decides who is still connected purely from the pid file) and any bare device-probe path cannot rely on this predicate to prove the device has come to rest.

2. partscan noise (deployment layer)

The nbd module is loaded with max_part=16, so every /sys/block/nbdX/partscan == 1. As a result, both nbdnl.Connect's set_capacity and the os.Open(devicePath) in Open() trigger the kernel's automatic partition scan, issuing probe reads against sector 0 / RDB / LDM. Those reads land on connections that are either mid-teardown (socket already closed) or freshly established and not yet stable, producing the I/O error sector 0 and unable to read partition table lines.

The sandbox rootfs is a bare ext4 image with no partition table, so these scans are pure waste: they spam dmesg, burn backend I/O on the create hot path, and can drown out genuine EIO.

Confirmed runtime facts (dev-sandbox-2)

• /sys/block/nbdX/inflight exists; idle value is 0 0 (two columns: in-flight reads / writes).
• /sys/block/nbdX/holders/ exists; empty when nothing holds the device.
• /sys/block/nbdX/partscan == 1, read-only (governed by module param max_part=16).
• nbds_max == 4096.

Proposed fix

A. Strengthen isDeviceFree: on top of no pid + size==0, additionally require

• inflight == "0 0" (no in-flight blk_mq requests), and
• holders/ empty (no udev / partition-probe / mount reference).

Only when all hold is the device declared free. This actually proves quiescence and closes the window in which the async-stage probe reads occur. Both new signals are best-effort: if the sysfs files are absent (older kernels), the check falls back to the size-only decision so the pool never wedges on a signal that will never appear.

B. Suppress partscan (deployment): set options nbd nbds_max=4096 max_part=0 via /etc/modprobe.d/nbd.conf so nbd creates no partitions and triggers no scan.

The accompanying PR lands A (Go layer, with tests). B is a deployment-side recommendation given in the PR description.