Skip to content

fix(nemo-gym): survive a real multi-server NeMo-RL GRPO run - #24

Open
Meirtz wants to merge 2 commits into
mainfrom
fix/nemo-gym-multiserver-session
Open

fix(nemo-gym): survive a real multi-server NeMo-RL GRPO run#24
Meirtz wants to merge 2 commits into
mainfrom
fix/nemo-gym-multiserver-session

Conversation

@Meirtz

@Meirtz Meirtz commented Jul 15, 2026

Copy link
Copy Markdown
Owner

Context

Bringing up a real NeMo-RL GRPO run (not a bespoke trainer) over the NeMo Gym resources server backed by ShinkenComputerEngine fork-reset surfaced three integration breaks between our shinken.integrations.nemo_gym server and the current NeMo Gym multi-server agent. Each is fixed here with a failing-first test.

Fixes

  1. Starlette lifespan compat — newer Starlette dropped add_event_handler, so the engine-shutdown binding crashed the server at startup with AttributeError. Fall back to wrapping the router's lifespan_context (same start-maintenance / close-engine lifetime contract).

  2. Generation fence over a lossy multi-server transport — the per-rollout generation is fenced via the Starlette session cookie, but each server namespaces its own cookie (ShinkenComputerResourcesServer___shinken_cua vs SimpleAgent___...) and the agent chains resources_server_cookies from one tool response to the next. A tool/verify route that only read the session never re-emitted our Set-Cookie, so the chain broke and the next call arrived session-less (or carrying a sibling server's cookie) → the fence rejected it with "no rollout generation". Two-part fix:

    • recover the generation from the engine keyed by the reliably-threaded session_id when the cookie drops it (ShinkenComputerEngine.current_generation);
    • re-assert the generation into the session on every tool/verify response, so Starlette re-emits the cookie and the agent's chain keeps carrying it end to end.
  3. SHINKEN_NG_DEBUG request tracer (off by default) — logs inbound cookies / session_id / generation per request; the diagnostic that pinned the cross-server cookie drop. Kept for adapting to other agents.

Verification

Closed a real NeMo-RL GRPO step on one GPU node (Qwen3-4B, vLLM generation + DTensor training, container-matched run_grpo_nemo_gym.py): the validation set and the num_prompts_per_step × num_generations_per_prompt training group were collected as forks of the golden desktop checkpoint, then Processing rewards → Computing advantages → Computing logprobs, clean exit. README §4 and status.md are reconciled from "config sketch, never run" to the verified run.

tests/test_nemo_gym_integration.py gains coverage for the lifespan shim, the session_id fallback, the cookie re-assert, and current_generation recovery — 51 pass; full SDK suite green.

Not in scope (follow-ups)

The ≥500-rollout scale run over the real CUA-Gym corpus (in flight), and generalizing the 16-way concurrency soak into an env-gated live test.

🤖 Generated with Claude Code

Meirtz and others added 2 commits July 16, 2026 05:44
…NeMo-RL GRPO run

Bringing up a real NeMo-RL GRPO run (not a bespoke trainer) over the NeMo Gym
resources server backed by ShinkenComputerEngine fork-reset surfaced three
integration breaks; each is fixed with a failing-first test:

- **Starlette lifespan compat**: newer Starlette dropped `add_event_handler`, so
  the engine-shutdown binding crashed the server at startup with AttributeError.
  Fall back to wrapping the router's lifespan context (same lifetime contract).

- **generation fence over a lossy transport**: the per-rollout generation is
  fenced via the Starlette session cookie, but each server namespaces its own
  cookie and the multi-server agent chains `resources_server_cookies` from one
  tool response to the next. A tool/verify route that only READ the session never
  re-emitted our Set-Cookie, so the chain broke and the next call arrived
  session-less (or carrying a sibling server's cookie), failing the fence.
  Two-part fix: (a) recover the generation from the engine keyed by the
  reliably-threaded `session_id` when the cookie drops it; (b) re-assert the
  generation into the session on every tool/verify response so Starlette re-emits
  the cookie and the agent's chain keeps carrying it end to end.

- **SHINKEN_NG_DEBUG** request tracer (off by default): logs inbound
  cookies/session_id/generation per request — the diagnostic that pinned the
  cross-server cookie drop; kept for adapting to other agents.

Verified end-to-end 2026-07-15: a real NeMo-RL GRPO step closed on one GPU node
(Qwen3-4B, vLLM gen + DTensor train) — validation + the
num_prompts×num_generations training group collected as forks of the golden
desktop checkpoint, then rewards→advantages→logprobs. README §4 and status.md
reconciled from 'config sketch, never run' to the verified run.

Tests: tests/test_nemo_gym_integration.py (+lifespan shim, +session_id fallback,
+cookie re-assert, +current_generation recovery) — 51 pass; full SDK suite green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A real CUA-Gym scale run surfaces tasks whose reward.py crashes on the unsolved
state (a common corpus quirk: reward scripts with `assert reward == 1.0`
self-tests that only pass against a golden solution). By default the engine
treats a reward.py fault as a typed CuaGymError (the strict eval contract), which
over a 10k-task RL collection aborts the whole batch on one bad task.

Add `ShinkenComputerEngine(scorer_error_reward=...)`: when set (the resources
server reads `SHINKEN_SCORER_ERROR_REWARD`), a scorer fault is logged and scored
as that value (0.0 = 'task not satisfied') so collection continues. Default stays
None = strict raise, preserving the single-task eval contract.

Tests: strict-default raises; tolerant mode scores the configured reward and still
tears the replica down so the batch proceeds.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant