Repository navigation
[wanda] Reach the agent by name instead of sharing its network - #501
Merged
Merged
Conversation
Replaces the RAYCI_BUILD_NETWORK option added in #500. Same goal -- let an image build reach a service on the agent it runs on, since a wanda step runs there directly rather than in a container -- with less exposure. --network=host works, but it hands the build the agent's whole network namespace, including anything listening on loopback. --add-host gives it one hosts entry and leaves the namespace intact. Measured with docker 28.1.1 against a service on the host: both reach it, on the BuildKit and the legacy builder alike. It is also a name rather than an address, which matters because inferring the address is exactly what broke every ray wheel build (ray postmerge 19281): the bridge gateway was read from `docker network inspect bridge`, the build could not reach it, and an index that cannot be reached fails a build outright rather than falling back. docker resolves host-gateway itself, so there is nothing to infer. rayci.localhost is the name ci/ray_ci/linux_container.py already passes to `docker run`, so a service is now addressed identically from a test container and from an image build. No configuration: unset variables were an easy way to end up half-wired. Signed-off-by: Ray CI Test <rayci@ray.io>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request replaces the use of host networking with a specific host entry (--add-host rayci.localhost:host-gateway) to allow builds to securely reach services running on the agent host. Feedback was provided regarding the use of the .localhost TLD, which can cause resolution issues in Go-based tools because Go's DNS resolver hardcodes .localhost to loopback addresses, bypassing /etc/hosts. It is recommended to use rayci.internal instead.
| // Deliberately not --network=host, which would also work: that shares the agent's | ||
| // network namespace with the build, exposing whatever else listens there, including on | ||
| // loopback. This adds one hosts entry and leaves the namespace intact. | ||
| args = append(args, "--add-host", "rayci.localhost:host-gateway") |
Contributor
There was a problem hiding this comment.
Using a .localhost TLD (like rayci.localhost) can cause unexpected resolution issues. According to RFC 6761, many resolvers and runtimes treat .localhost specially. Specifically, Go's built-in DNS resolver (used by Go binaries, especially those compiled with CGO_ENABLED=0) hardcodes the resolution of any hostname ending in .localhost to loopback addresses (127.0.0.1 and ::1), completely bypassing /etc/hosts. If any Go-based tool or build step inside the container attempts to resolve rayci.localhost, it will resolve to the container's own loopback instead of the host gateway, defeating the purpose of --add-host. Consider using a different domain suffix that is not reserved for loopback, such as rayci.internal or rayci.build.
Suggested change
| args = append(args, "--add-host", "rayci.localhost:host-gateway") | |
| args = append(args, "--add-host", "rayci.internal:host-gateway") |
elliot-barn
added a commit
to ray-project/ray
that referenced
this pull request
Aug 19, 2026
wanda passes --add-host rayci.localhost:host-gateway from 0.47.0 (ray-project/rayci#501), which is the address ci/pypi_proxy_agent.sh exports as RAYCI_IMAGE_PIP_INDEX_URL. The export is gated on this version, so until the pin moves image builds stay on public PyPI. Signed-off-by: Elliot Barnwell <elliot.barnwell@anyscale.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
elliot-barn
added a commit
to ray-project/ray
that referenced
this pull request
Aug 20, 2026
## What Passes `--add-host rayci.localhost:host-gateway` to the `docker build` that builds custom BYOD images, which is what wanda already passes to every image build it starts (ray-project/rayci#501). ## Why Release build 104848, on the commit that became #65578, failed `hello_world_custom_byod.aws` and `hello_world_custom_byod.gce`: ``` #11 6.043 error: Request failed after 3 retries in 5.7s #11 6.043 Caused by: failed to lookup address information: No address associated with hostname #11 ERROR: process "/bin/bash -c bash /tmp/install_python_deps.sh python_depset.lock" did not complete successfully: exit code: 2 ``` Custom BYOD images are built here by a plain `docker build`, not by wanda, so they inherit none of wanda's networking flags. `RAYCI_IMAGE_PIP_INDEX_URL` reaches the build as `http://rayci.localhost:35999/simple` — a name the build has no way to resolve — and `install_python_deps.sh` resolves the depsets with **uv**, which fails hard where pip would have fallen back to PyPI. So the image build dies rather than degrading. Everything else on that build was green (78 passed at the time, including `wanda: forge`, all five wheel builds, the `base-extra-testdeps` images, and the `hello_world*` release tests on AWS, GCE and released images) — this is the one path that reaches the agent proxy without going through wanda. Unconditional, matching wanda: docker resolves `host-gateway` itself, and one extra hosts entry is inert for a build that never uses the name. ## Testing `python -m pytest release/ray_release/tests/test_byod_build.py` — 4 passed. The flag goes in after `-t <image>`, so the argument prefix those tests match on is unchanged. AI assistance was used for this change. Signed-off-by: Elliot Barnwell <elliot.barnwell@anyscale.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
elliot-barn
added a commit
that referenced
this pull request
Aug 20, 2026
## Description Adds `add-host: rayci.localhost:host-gateway` to the docker buildkite plugin config that `makeRayDockerPlugin` emits for job containers. Wanda's `docker build` (#501) and ray's `ci/ray_ci/linux_container.py` both already pass this mapping, so a service listening on the agent — the package index proxy — is reachable by one stable name from an image build and from a test container. Containers rayci starts through the docker buildkite plugin were the one launcher missing it. The gap became visible when ray's CI images started baking the name into `PIP_INDEX_URL` (ray-project/ray#65578, armed by ray-project/ray#65605): any `job_env` step that runs pip at runtime resolves the baked index and fails DNS with `[Errno -5] No address associated with hostname`. Observed on ray's `doc: check API annotations` / `check API doc consistency` (via setuptools `fetch_build_eggs` for cython), `pyrefly type check` and `build: pip-compile dependencies` (direct runtime pip installs). The mapping is emitted unconditionally, matching the other two launchers. The docker buildkite plugin v5.8.0 supports `add-host` natively (verified in its plugin.yml). Agents where nothing listens on the name are unaffected — the entry only matters to a container that dials it. ## Testing - New `TestMakeRayDockerPlugin_addHost` asserts the emitted mapping. - `go test ./raycicmd` passes; `go test ./...` green except pre-existing local-env failures in `raycilint` (host git too old for `--initial-branch`), identical on clean `main`. Rollout note: after this merges and a release is cut, ray consumes it via `.rayciversion`; the `pyrefly type check` step is a live canary since it still pip-installs at runtime. AI assistance (Claude Code) was used to prepare this change. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Ray CI Test <rayci@ray.io> Co-authored-by: Ray CI Test <rayci@ray.io>
nadongjun
pushed a commit
to nadongjun/ray
that referenced
this pull request
Oct 6, 2026
## What Passes `--add-host rayci.localhost:host-gateway` to the `docker build` that builds custom BYOD images, which is what wanda already passes to every image build it starts (ray-project/rayci#501). ## Why Release build 104848, on the commit that became ray-project#65578, failed `hello_world_custom_byod.aws` and `hello_world_custom_byod.gce`: ``` ray-project#11 6.043 error: Request failed after 3 retries in 5.7s ray-project#11 6.043 Caused by: failed to lookup address information: No address associated with hostname ray-project#11 ERROR: process "/bin/bash -c bash /tmp/install_python_deps.sh python_depset.lock" did not complete successfully: exit code: 2 ``` Custom BYOD images are built here by a plain `docker build`, not by wanda, so they inherit none of wanda's networking flags. `RAYCI_IMAGE_PIP_INDEX_URL` reaches the build as `http://rayci.localhost:35999/simple` — a name the build has no way to resolve — and `install_python_deps.sh` resolves the depsets with **uv**, which fails hard where pip would have fallen back to PyPI. So the image build dies rather than degrading. Everything else on that build was green (78 passed at the time, including `wanda: forge`, all five wheel builds, the `base-extra-testdeps` images, and the `hello_world*` release tests on AWS, GCE and released images) — this is the one path that reaches the agent proxy without going through wanda. Unconditional, matching wanda: docker resolves `host-gateway` itself, and one extra hosts entry is inert for a build that never uses the name. ## Testing `python -m pytest release/ray_release/tests/test_byod_build.py` — 4 passed. The flag goes in after `-t <image>`, so the argument prefix those tests match on is unchanged. AI assistance was used for this change. Signed-off-by: Elliot Barnwell <elliot.barnwell@anyscale.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces the
RAYCI_BUILD_NETWORKoption from #500 — same goal, less exposure.A wanda step runs directly on the agent rather than in a container, so a service listening there (a package index, say) is outside the build's own network namespace.
--network=hostachieves that but hands the build the agent's entire network namespace, including anything bound to loopback.--add-hostadds one hosts entry and leaves the namespace intact. Both were measured with docker 28.1.1 against a service on the host, on BuildKit and the legacy builder:It is also a name rather than an address, which matters: inferring the address is what broke every ray wheel build (ray postmerge 19281 vs 19280). The bridge gateway was read from
docker network inspect bridge, the build could not reach it, and an unreachable index fails a build outright rather than falling back to PyPI.host-gatewayis resolved by docker, so there is nothing to infer.rayci.localhostis the nameci/ray_ci/linux_container.pyalready passes todocker run, so a service is addressed identically from a test container and from an image build.No configuration knob: an unset variable was an easy way to end up half-wired.
Tested:
TestDockerCmdBuild_addHostdrives a real build;go test ./wanda/passes.