feat(broker): add [broker] max_open_files and explain descriptor exha…

CHANGES.md

@@ -185,6 +185,49 @@ exactly as before until you opt in. Details and examples are in
   load time. The same file therefore scales identically under the Director GUI
   and under `mads up` / `mads doctor --plan`.
 
+- **`[broker] max_open_files`, and a broker that says when it runs out of
+  descriptors.** Fleet size is bounded by the broker's open-file limit, not by
+  anything in libzmq: every connected agent holds two of the broker's
+  descriptors for as long as it stays connected -- its publisher on the XSUB
+  frontend, its subscriber on the XPUB backend -- plus a third while it fetches
+  its settings. With the usual soft `RLIMIT_NOFILE` of 1024 that caps a fleet
+  at roughly **495 agents**.
+
+  Past that ceiling the broker used to fail in the least helpful way possible.
+  libzmq's TCP listener counts `EMFILE`/`ENFILE` among the errnos `accept()`
+  may fail with harmlessly, so it raised `ZMQ_EVENT_ACCEPT_FAILED` -- which
+  nothing listened for -- and refused every new agent in complete silence. The
+  only thing that printed was the service-discovery thread, because its
+  once-a-second `getifaddrs()` and broadcast sockets are the broker's only
+  *timed* descriptor allocations and so were the first to fail. The result was
+  a broker that looked healthy while turning agents away, reporting
+  `ServiceDiscovery advertising failed: getifaddrs failed: Too many open
+  files`, which points at discovery rather than at the limit actually
+  responsible.
+
+  Three changes, all backwards-compatible:
+  - `max_open_files` under `[broker]` raises the soft limit at startup, before
+    any socket is bound. Unset (the default) leaves it untouched and merely
+    reports it; `0` asks for as much as the process is permitted. It cannot
+    exceed the hard limit -- a larger value is clamped with a warning, since
+    only `LimitNOFILE=` or a privileged `ulimit -Hn` can lift that. Portable:
+    the raise is capped by `kern.maxfilesperproc` on macOS and `fs.nr_open` on
+    Linux, and reported as not applicable on Windows, which has no per-process
+    descriptor limit for sockets.
+  - The broker now watches its bound sockets for `ZMQ_EVENT_ACCEPT_FAILED` and
+    reports a refused agent explicitly, naming the endpoint, the limit in
+    force, the agent count it allows and how to raise it. Rate-limited,
+    because a full descriptor table leaves the listening socket permanently
+    readable and libzmq retries the failing accept as fast as it can poll.
+  - Any discovery socket error caused by a full descriptor table now says so,
+    and the advertising loop reports an unchanged failure at most once every
+    30 seconds instead of every second.
+
+  The shipped systemd template (`mads service`) now sets `LimitNOFILE=65536`;
+  without it a unit inherits systemd's `DefaultLimitNOFILE`, which is that same
+  1024 on most distributions. `mads doctor` gained a check reporting the limit
+  in force and whether it leaves room for the fleet.
+
 ## Fixes
 
 - **Broker `p`/`r` keys now actually pause and resume.** The interactive

mads.ini

@@ -73,6 +73,13 @@ prefer_loopback_for_local_services = true
 # endpoint. One agent fetching a large plugin attachment no longer blocks
 # every other agent's settings request behind it; see `man mads-broker`.
 # settings_workers = 2
+# Open file descriptors the broker may use, which is what bounds fleet size:
+# every connected agent costs two of them (its publisher and its subscriber),
+# so the usual soft limit of 1024 stops at roughly 495 agents. Unset leaves the
+# limit alone and just reports it at startup; 0 asks for as many as the OS
+# allows. Cannot exceed the hard limit (LimitNOFILE= in the systemd unit, or
+# `ulimit -Hn`), and has no effect on Windows. See `man mads-broker`.
+# max_open_files = 65536
 
 
 [logger]

share/man/mads-broker.md

@@ -110,6 +110,32 @@ unedited settings file behaves exactly as before they existed.
    agent's plain REQ settings request works unchanged. A value below 1 is
    clamped to 1 with a warning rather than refused.
 
+**max_open_files** (`[broker]`, integer, unset by default)
+:  Open file descriptors the broker process may use, which is what bounds
+   fleet size. Every connected agent holds **two** of the broker's descriptors
+   for as long as it stays connected -- its publisher on the XSUB frontend and
+   its subscriber on the XPUB backend -- plus a third while it fetches its
+   settings. With the usual soft limit of 1024 and the broker's own ~30
+   descriptors of overhead, that caps a fleet at roughly **495 agents**, and
+   libzmq refuses everything past it *almost silently*: its listener treats
+   `EMFILE` as a non-fatal `accept()` error. Set this and the broker raises its
+   own soft limit at startup, before binding anything, and reports the
+   resulting ceiling and the agent count it implies.
+
+   Unset leaves the limit exactly as inherited and only reports it, warning
+   when it is low enough to be worth raising. `0` asks for as many descriptors
+   as the process is permitted. **Cannot exceed the hard limit** -- a larger
+   value is clamped with a warning, because raising the hard limit needs
+   `LimitNOFILE=` in the systemd unit or a privileged `ulimit -Hn`, not a
+   settings key. A negative value is ignored with a warning rather than
+   refused, like `io_threads`.
+
+   Has no effect on Windows, which has no per-process descriptor limit for
+   sockets, and is reported as not applicable there. On macOS the raise is
+   additionally capped by `kern.maxfilesperproc`, and on Linux by
+   `fs.nr_open`. `mads doctor` reports the limit in force and whether it
+   leaves room for the fleet.
+
 # BUGS
 
 The upstream bug tracker can be found at https://github.com/pbosetti/MADS/issues.

share/templates/service.tpl

@@ -22,6 +22,13 @@ Type=simple
 Restart=always
 RestartSec=1
 User=root
+# systemd hands a unit that does not say otherwise its DefaultLimitNOFILE soft
+# value, which is 1024 on most distributions. The broker holds two descriptors
+# per connected agent (its publisher and its subscriber), so that default caps
+# a fleet at roughly 495 agents -- and libzmq refuses everything past it almost
+# silently. Can also be set from the settings file with [broker] max_open_files,
+# but only up to the hard limit this line establishes.
+LimitNOFILE=65536
 ExecStart={{command}}
 
 [Install]