Skip to content

eve dev intermittently answers 503 {"error":"read ECONNRESET"} to channel requests under CPU load #5001

Open

Description

@dretanacr

Summary

Under heavy local CPU load, eve dev sometimes answers an incoming channel request (a Teams activity POST in our case) with HTTP 503 and the body {"error":"read ECONNRESET"}. The request never reaches the agent's channel handler, so the message is lost unless the sender retries.

Where it happens

In DrainedNitroDevServer, the public dev server forwards every request to the dev worker with runner.fetch(...). When that call throws, the catch hands the error to writeRequestError (dist/src/internal/nitro/host/dev-server-http.js), which writes status 503 with {"error": <message>}.

runner.fetch goes through the env-runner proxy (dist/src/compiled/env-runner/index.js). It uses a shared http.Agent({ keepAlive: true, maxSockets: 256, maxFreeSockets: 64 }) and does not retry on a reset socket.

This code is byte-identical in 0.79.1 and 0.80.1 (dev-server-http.js, drained-nitro-dev-server.js, dev-runner.js, compiled/env-runner/index.js, compiled/env-runner/node-worker.js).

What we observed

  • Two identical requests sent at once: the second one got the 503.
  • A burst of 12 requests sent at once: 1 of them got the 503.
  • Frequency: about 1 run in 22, with load average around 50 on an 18-core Mac (macOS 26).
  • Node 24, eve 0.79.1.
  • The production server (eve start / the built output) has no dev forwarding hop, and we have not seen this there.

Likely cause (hypothesis, not proven)

This looks like the known Node keep-alive race. The agent picks an idle pooled socket for the next request at the moment the worker's HTTP server closes that socket on its keep-alive timeout, and the write fails with ECONNRESET. Heavy CPU load stretches the timing and makes the window more likely to hit. We have not captured socket-level evidence to confirm it.

Possible fixes (suggestions)

  1. Retry the forward once when ECONNRESET happens on a reused socket before any response bytes arrive. The incoming body is streamed (Readable.toWeb(req)), so a retry would have to buffer the body or retry only when none of it has been sent yet.
  2. Turn off keep-alive for the dev forward (or set the worker server's keepAliveTimeout higher than the agent's idle socket lifetime). The extra connection cost does not matter on localhost.
  3. At minimum, document that eve dev can return this 503 under load and that channel senders should be expected to retry.

Current vs. expected behavior

Current: an occasional channel request gets 503 {"error":"read ECONNRESET"} from eve dev and never reaches the handler.

Expected: the dev forwarding hop does not drop requests that the worker is able to serve.

Activity

  1. roboeve commented on Oct 17, 2026

    @roboeve
    Contributor

    Confirmed on origin/main: when the eve dev parent's event loop is busy past the worker's keep-alive timeout, its next forward goes out on a pooled socket the worker already closed, and the dev server answers 503 (socket hang up on Linux, read ECONNRESET on macOS). The forwarding agent has no idle timeout of its own, so it only drops a dead socket after it reads the worker's FIN.

    Repro: #5003
    Fix: #5004
    Regressed in: not a recent regression (same failure on eve@0.79.0 and eve@0.50.0)

  2. added 2 commits that reference this issue on Oct 17, 2026
    55b902d
    f291ee8
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingbuildp2

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions