feat(stacks): resolve GPU intents to an NVIDIA reservation too (ADR-0032) - #73
Merged
Conversation
…032) ADR-0031 skipped NVIDIA, and its reason was right: mapping /dev/nvidia* is not enough, because the container also needs the toolkit-injected user-space driver. What it did not follow through on is that the supported NVIDIA route is itself an intent - a Compose reservation carries no paths and no GIDs, so it fits the ADR's host-neutral model better than render nodes do, not worse. Two things made the exclusion actively misleading rather than merely incomplete. NVIDIA is usually invisible to the probe, because it only appears under /sys/class/drm when nvidia-drm is loaded - common on desktops, not guaranteed on the headless hosts that hold the cards. So the "needs the container toolkit" note fired where it was least needed, and a genuine NVIDIA server was told "No mappable host GPU was detected", with no hint that a working route exists. And the toolkit's presence is knowable: it registers itself as a Docker runtime, which GET /info reports. - HostGpuProbe reports /dev/nvidiactl, which exists whenever the kernel driver is loaded, and reads the daemon's runtimes; NvidiaUsable needs both, because a reservation on a host without the toolkit does not degrade - Compose fails the whole deploy - DeviceMappingPlan resolves the existing intent per host: render nodes plus group_add where they exist, a reservation where NVIDIA does. No new row, no new control - a stack moving between an Intel host and an NVIDIA one changes nothing - ComposeOverrideFile emits deploy.resources.reservations.devices with capabilities [gpu, video]. "video" is not optional here: the toolkit only injects the encoder libraries when asked, so the usual [gpu] spelling leaves CUDA working and NVENC failing to open - The GPU-less note no longer fires on an NVIDIA host, and the Settings read-out names the toolkit state The emitter writes its lines with explicit \n rather than a raw string literal, which would carry the source file's CRLF into the generated document on a Windows checkout - caught by the render test.
swimmesberger
marked this pull request as ready for review
August 29, 2026 13:47
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft — this amends ADR-0031's decision 3, so it's a design call for you before it's a code review.
ADR-0031 skipped NVIDIA, and the reason it gave is right: mapping
/dev/nvidia*isn't enough, because the container also needs the toolkit-injected user-space driver. What it didn't follow through on is that the supported NVIDIA route is itself an intent:No paths, no GIDs, nothing host-specific. That fits ADR-0031's host-neutral model better than render nodes do — so NVIDIA was excluded from the transcoding feature on the hosts most likely to be transcoding boxes.
Two things made the exclusion misleading, not just incomplete
NVIDIA is usually invisible to the probe. It only appears under
/sys/class/drmwhennvidia-drmis loaded — common on desktops, not guaranteed on the headless hosts that hold the cards. So the "needs the container toolkit" note fired where it was least needed, and a real NVIDIA server got:True, unhelpful, and with no hint that a working route exists.
The toolkit's presence is knowable. It registers itself as a Docker runtime, which
GET /inforeports. That matters because emitting a reservation without it doesn't degrade — Compose fails the entire deploy withcould not select device driver "nvidia" with capabilities: [[gpu]]. SoNvidiaUsablerequires both the card and the runtime; a card without the toolkit reserves nothing and says why.What changed
stack_gpu_mappingsalready means "give this service the host's GPUs"; resolution now picks the mechanism per host — render nodes plusgroup_addwhere they exist, a reservation where NVIDIA does. A stack moving from an Intel host to an NVIDIA one changes nothing.HostGpuProbereports/dev/nvidiactl(exists whenever the kernel driver is loaded) and reads the daemon's runtimes.ComposeOverrideFileemits the reservation.capabilitiesincludesvideo— the toolkit only injects the encoder libraries when asked, so the usual[gpu]orcompute,utilityspelling leaves CUDA working and NVENC failing to open. Video transcoding is the motivating workload, so that one isn't optional.Reviewer notes
ServiceDeviceMappings.NvidiaGpussays the service wants the vendor's GPUs; onlyComposeOverrideFileknows Compose spells that as a device reservation. Kubernetes saysnvidia.com/gpu.deploy.resources.reservations.devicesoutside Swarm, unlike most ofdeploy:. Genuinely surprising, so it's commented at the emitter rather than left looking like a mistake.count: allmatches ADR-0031's choice for render nodes; single-card selection would bedevice_idsand is deferred for the same reason.'\n'. The emitter now writes\nexplicitly.Verification
dotnet build Watchtower.slnxclean (0 warnings). Application tests: 1900 passed, 23 failed — the same 23 fail on the base commit with these changes stashed (ACME/certificate X509 chain building on this Windows box), so no regressions; +13 new tests. Frontendgenerate:rpc+typecheckclean,buildsucceeds.rpc-schema.jsonregenerated — the only diff is the newnvidiaobject onstacks.hostGpus.Not verified against real NVIDIA hardware, same as the rest of the GPU feature. The emitted block is the documented Compose form; the
videocapability comes from observed toolkit behaviour rather than assumption, but a real Docker host with the toolkit would be the honest end-to-end check before this leaves draft.