fix(nemo-gym): survive a real multi-server NeMo-RL GRPO run - #24
Open
Meirtz wants to merge 2 commits into
Open
Conversation
…NeMo-RL GRPO run Bringing up a real NeMo-RL GRPO run (not a bespoke trainer) over the NeMo Gym resources server backed by ShinkenComputerEngine fork-reset surfaced three integration breaks; each is fixed with a failing-first test: - **Starlette lifespan compat**: newer Starlette dropped `add_event_handler`, so the engine-shutdown binding crashed the server at startup with AttributeError. Fall back to wrapping the router's lifespan context (same lifetime contract). - **generation fence over a lossy transport**: the per-rollout generation is fenced via the Starlette session cookie, but each server namespaces its own cookie and the multi-server agent chains `resources_server_cookies` from one tool response to the next. A tool/verify route that only READ the session never re-emitted our Set-Cookie, so the chain broke and the next call arrived session-less (or carrying a sibling server's cookie), failing the fence. Two-part fix: (a) recover the generation from the engine keyed by the reliably-threaded `session_id` when the cookie drops it; (b) re-assert the generation into the session on every tool/verify response so Starlette re-emits the cookie and the agent's chain keeps carrying it end to end. - **SHINKEN_NG_DEBUG** request tracer (off by default): logs inbound cookies/session_id/generation per request — the diagnostic that pinned the cross-server cookie drop; kept for adapting to other agents. Verified end-to-end 2026-07-15: a real NeMo-RL GRPO step closed on one GPU node (Qwen3-4B, vLLM gen + DTensor train) — validation + the num_prompts×num_generations training group collected as forks of the golden desktop checkpoint, then rewards→advantages→logprobs. README §4 and status.md reconciled from 'config sketch, never run' to the verified run. Tests: tests/test_nemo_gym_integration.py (+lifespan shim, +session_id fallback, +cookie re-assert, +current_generation recovery) — 51 pass; full SDK suite green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A real CUA-Gym scale run surfaces tasks whose reward.py crashes on the unsolved state (a common corpus quirk: reward scripts with `assert reward == 1.0` self-tests that only pass against a golden solution). By default the engine treats a reward.py fault as a typed CuaGymError (the strict eval contract), which over a 10k-task RL collection aborts the whole batch on one bad task. Add `ShinkenComputerEngine(scorer_error_reward=...)`: when set (the resources server reads `SHINKEN_SCORER_ERROR_REWARD`), a scorer fault is logged and scored as that value (0.0 = 'task not satisfied') so collection continues. Default stays None = strict raise, preserving the single-task eval contract. Tests: strict-default raises; tolerant mode scores the configured reward and still tears the replica down so the batch proceeds. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
Bringing up a real NeMo-RL GRPO run (not a bespoke trainer) over the NeMo Gym resources server backed by
ShinkenComputerEnginefork-reset surfaced three integration breaks between ourshinken.integrations.nemo_gymserver and the current NeMo Gym multi-server agent. Each is fixed here with a failing-first test.Fixes
Starlette lifespan compat — newer Starlette dropped
add_event_handler, so the engine-shutdown binding crashed the server at startup withAttributeError. Fall back to wrapping the router'slifespan_context(same start-maintenance / close-engine lifetime contract).Generation fence over a lossy multi-server transport — the per-rollout generation is fenced via the Starlette session cookie, but each server namespaces its own cookie (
ShinkenComputerResourcesServer___shinken_cuavsSimpleAgent___...) and the agent chainsresources_server_cookiesfrom one tool response to the next. A tool/verify route that only read the session never re-emitted ourSet-Cookie, so the chain broke and the next call arrived session-less (or carrying a sibling server's cookie) → the fence rejected it with "no rollout generation". Two-part fix:session_idwhen the cookie drops it (ShinkenComputerEngine.current_generation);SHINKEN_NG_DEBUGrequest tracer (off by default) — logs inbound cookies / session_id / generation per request; the diagnostic that pinned the cross-server cookie drop. Kept for adapting to other agents.Verification
Closed a real NeMo-RL GRPO step on one GPU node (Qwen3-4B, vLLM generation + DTensor training, container-matched
run_grpo_nemo_gym.py): the validation set and thenum_prompts_per_step × num_generations_per_prompttraining group were collected as forks of the golden desktop checkpoint, thenProcessing rewards → Computing advantages → Computing logprobs, clean exit. README §4 and status.md are reconciled from "config sketch, never run" to the verified run.tests/test_nemo_gym_integration.pygains coverage for the lifespan shim, thesession_idfallback, the cookie re-assert, andcurrent_generationrecovery — 51 pass; full SDK suite green.Not in scope (follow-ups)
The ≥500-rollout scale run over the real CUA-Gym corpus (in flight), and generalizing the 16-way concurrency soak into an env-gated live test.
🤖 Generated with Claude Code