Queue subscription dies permanently and silently after a broker-initiated channel close (e.g. consumer ack-timeout precondition_failed)

[arobson]

Summary

When RabbitMQ force-closes a channel (not the whole connection) — e.g. via a precondition_failed channel exception such as a consumer ack-timeout — a queue's subscription dies permanently and silently. The queue FSM transitions to its closed state and just sits there forever. There is no automatic recovery, no automatic re-subscription, and no documented signal telling the application it needs to do anything. The application's connection stays healthy throughout (rabbot.on('unreachable', ...) never fires, since the connection itself never drops), so nothing in a typical integration notices anything is wrong.

In our case this caused complete, silent, unrecovered loss of message consumption for ~16 hours across every replica of a service, discovered only because a human happened to notice downstream data wasn't updating — not because of any error, log line, or alert.

Environment

• rabbot 2.1.0
• Node.js 24.x
• RabbitMQ 3.13 (management image), single node
• Consumer declared via rabbot.handle('#', handler, queueName) + rabbot.configure({ exchanges, queues, bindings }), subscribe: true on the queue definition. No limit (prefetch) set on the queue definition.

What happened (production incident)

1. A long-running consumer's handler took long enough processing one message (compounded by a slow/restarting downstream dependency at the time) to exceed RabbitMQ's default 30-minute consumer ack-timeout (consumer_timeout, 1,800,000 ms).
2. RabbitMQ logged and force-closed the channel:

   Consumer <N> on channel 5 has timed out waiting for delivery acknowledgement. Timeout used: 1800000 ms.
   Channel error on connection ..., channel 5:
   operation none caused a channel exception precondition_failed: delivery acknowledgement on channel 5 timed out.
   

   This happened on 4 separate process replicas simultaneously (each independently exceeded the timeout at nearly the same moment).
3. None of the 4 processes crashed or restarted — their HTTP servers, other unrelated functionality, etc. all continued running normally. The TCP connection to RabbitMQ stayed up (confirmed via rabbitmqctl list_connections — connections present throughout).
4. The RabbitMQ queue for each of the 4 consumers silently disappeared from rabbitmqctl list_queues shortly after (its x-expires idle-timeout kicked in once nothing was consuming from it).
5. For the next ~16 hours, every message published to the exchange these queues were bound to was accepted by the broker (the exchange itself was fine) but had no bound queue to route to, and was silently dropped — no error, no dead-letter, no publisher-confirm failure (the publisher only confirms broker acceptance into the exchange, which succeeded every time).
6. The only fix was manually restarting all 4 processes (kubectl rollout restart), which re-ran the full rabbot.configure()/rabbot.handle() sequence from scratch and recreated working consumers.

Root cause, traced through rabbot's own source

src/queueFsm.js binds to the channel's closed event and transitions the FSM into its own closed state:

handlers.push(queue.channel.on('closed', function () {
  this.handle('closed', queue);
}.bind(this))
);

The closed state itself does nothing to recover:

closed: {
  _onEnter: function () {
    this.subscribed = false;
    this._release(true);
    this.emit('closed');
  },
  check: function () {
    this.deferUntilTransition('ready');
    this.transition('initializing');
  },
  ...
  subscribe: function () {
    this.deferUntilTransition('ready');
  }
},

Note that check() would kick the FSM back into initializing (which re-runs _define/_listen/subscribe) — but nothing calls it automatically. subscribe() in the closed state just defers forever, waiting for a ready transition that never arrives on its own.

src/amqp/iomonad.js (the underlying channel abstraction) confirms a broker-initiated channel close is logged and transitions to closed:

channel.on('close', function () {
  ...
  log.warn(...'closed by the broker with reason...');
  machine.transition('closed');
});

Its own closed state's operate() only re-acquires on the next discrete RPC call (a publish/ack) — which never happens for an idle, long-lived consumer subscription that isn't actively being invoked.

The only application-facing recovery path is calling rabbot.getQueue(queueName).check() (or presumably .reconnect()), but:

• This is never called automatically by rabbot itself in response to the queue's own closed event.
• It isn't clearly documented as something a consumer of the library must wire up for correctness.
• The only broker-level event rabbot surfaces at the top level (rabbot.on('unreachable', ...)) is connection-scoped and does not fire for this failure mode, since the connection never actually drops — only the channel does.

Expected behavior

One of:

1. (Preferred) rabbot's queue FSM automatically attempts recovery (re-declare, re-bind, re-subscribe) when its channel is closed by the broker, the same way a fresh configure() would establish it — mirroring what already happens for a full connection loss/reconnect.
2. At minimum, rabbot should emit a clearly-documented, prominent, queue-scoped event (e.g. rabbot.on('queue.closed', ...), analogous to the existing connection-level unreachable) that unambiguously tells the application "this queue's subscription is dead and will stay dead until you call .check()/.reconnect() yourself" — today this is only discoverable by reading queueFsm.js's source directly.

Actual behavior

The consumer dies permanently and silently. No error is thrown, no rejection surfaces anywhere in application code, no built-in event fires that a typical integration would already be listening to (the existing unreachable handler many apps already wire up does not cover this case). The only way to notice is external monitoring of the symptom (e.g. "downstream data stopped updating"), not the cause.

Suggested fixes (would be happy to help validate/PR)

1. Add an automatic recovery path from closed — e.g. closed._onEnter schedules a this.handle('check') (possibly with backoff), instead of requiring the application to notice and call it manually.
2. Regardless of (1), emit a clearly-named, documented event distinct from unreachable specifically for a queue-level channel closure, so applications that want to build their own recovery/alerting have something reliable to listen for.
3. Document this failure mode explicitly in the README's messaging-patterns/production-usage section — right now nothing in the docs suggests a channel (as opposed to the connection) can silently die and require manual recovery.
4. Related/possibly relevant prior art: Add support for clean teardown + reconstruction #35 ("Add support for clean teardown + reconstruction") looks adjacent but is about connection-level retry-limit exhaustion, not a channel-level broker-initiated close while the connection stays healthy — flagging in case it's useful context, but this looks like a distinct failure mode.

Happy to provide a minimal reproduction (a consumer that sleeps past consumer_timeout before acking) if useful.