A stack with no broker dials the shared estate Kafka instead of failing

[catinspace-au]

A dfe-docker stack with no in-stack broker silently dials the SHARED ESTATE broker instead of failing.

Found standing up a second stack on desktop-derek. The loader was briefly on a kafka-transport config with no broker in its own compose project. It did not fail fast. It connected to 10.66.0.200 -- the devex estate broker.

The mechanism is DNS, not config:

• The container's /etc/resolv.conf carries search devex.hyperi.io with ndots:0.
• ndots:0 means a bare single-label name gets the search suffix applied.
• So kafka resolves to kafka.devex.hyperi.io, which is the estate broker.

Nothing in the stack is wrong. The default kafka hostname is correct for a compose network that HAS a broker. On a devex host without one, it leaves the box.

In our case the broker closed the connection on metadata fetch, so nothing was written or consumed. That was luck, not design. A stack that reached a broker it could actually talk to would have produced into shared infrastructure, from a developer laptop, with no indication anything was unusual.

This bites any kafka-transport dfe-docker stack started on a host inside devex.hyperi.io where the broker did not come up -- a profile mistake, a crash-looping broker, a second stack that forgot the kafka service.

Worth considering:

• Fail fast on a broker outside the compose network, rather than connecting. The loader knows what project it is in.
• Or make the default broker name fully qualified to the compose network so the search domain cannot apply.
• ndots:0 plus a single-label service name is the general shape here, not just kafka. Every bare service name in these configs has the same exposure on a devex host.

Done when a dev stack cannot reach estate infrastructure by accident.

Found while building a second docker stack for the Playwright acceptance suite.

[catinspace-au]

This is worse than I first filed it. The containers carry the search domain too, not just the host.

dfe-e2e_default containers have the same search devex.hyperi.io with ndots:0. Docker's embedded resolver answers a bare kafka from the compose network alias, so a stack WITH a broker is safe -- but only because the alias exists, not because anything prevents the leak.

Proof that the search domain does resolve from inside a container, using a name with no alias:

docker exec e2e-dfe-receiver getent hosts dfe-docker
  -> 10.66.0.141  dfe-docker.devex.hyperi.io

So any DFE app in a compose stack whose config names kafka:9092 while no broker container is running connects to 10.66.0.200, the shared estate broker, from inside the container, with no error anywhere.

What contains it today is scripts/resolve_profile.py:476-480, which only emits DFE_KAFKA_BOOTSTRAP_SERVERS on a bus tier. That is a good guard and it is the only one. A hand-driven docker compose run, or one edited config file, is enough to get past it -- and both are normal things to do when debugging a stack, which is exactly when a broker is most likely to be missing.