Activity

1. ashhart commented on Oct 31, 2026

   

   ashhart

   on Oct 31, 2026

   Owner

   More actions

   Thanks @DaviddevIDM. A trigger that fails every time is very useful.

   On your questions:

   1. We can't say yet. Before relu^2 keeps a NaN: grouped experts, ck experts and Nemotron's pocket #1628 (now on main), squared ReLU in those kernels turned a NaN into 0. That hides a NaN
      rather than spreading it, so it's not an obvious cause of a failed forward. With relu^2 keeps a NaN: grouped experts, ck experts and Nemotron's pocket #1628 a NaN now reaches the output,
      so a run on main would show one if it's there.
   2. We don't have a NaN audit switch for the Bolt forward that we can point you to yet.

   What would help most:

   • the engine's stderr from start to the first 500, including the BoltForward lines and the warnings before them;
   • the same workload on current main, built from source, rather than the v1.1.6 recipe image;
   • the KV dtype and the exact serve flags the recipe passes;
   • whether NC=2 also gives 500s before the process exits, and its last lines.

   This was run through a third-party recipe image. If you can, please confirm the failure with our own build, so we know
   the image isn't adding anything.
2. DaviddevIDM commented on Nov 1, 2026

   

   DaviddevIDM

   on Nov 1, 2026

   Author

   More actions

   Thanks for the fast triage @ashhart — answering your four asks in order, and being explicit about what we can't recover and what we owe you.

   Environment, pinned: DGX Spark (GB10, sm_121), driver 580.178.04, kernel 7.0.0-1019-nvidia. Recipe MiaAI-Lab/Nemotron-3.5-Bolt-TensorFold-DGX-Spark @ 2cea422, engine = TensorFold v1.1.6 at the recipe's pinned revision eea9dc9c9c8b. Weights: nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB snapshot a326822af111290f2af5ac59ef4b5040347dabc4 (IQ2_XXS+NVFP4+Q8_0 + LiLiCorr draft).

   1) Engine stderr start → first 500 — unrecoverable from the October runs; closest artifact attached.
   Honest answer first: the failing runs' engine stderr is gone. The recipe's start.sh runs its container --rm and our launcher only teed the harness log, so engine stderr died with the container on all three deaths. What I'm attaching instead: a persistent-stderr relaunch of the identical recipe on a second Spark (docker logs -f wired from launch; 118,609 lines across the full repro series). It contains the launch banner and the steady-state warning class — e.g. [tensorfold] warning: a reply reached max_tokens while still thinking, so its content is empty... and warning: non-loopback server has no API key; requests are open — but zero BoltForward lines, because the relaunch never produced a single 500 (repro detail under 4). The client-side verbatim transcript (first 500 + the full 516-line ramp) is in the body of the original report and in item 4 below.

   2) Same workload on current main, built from source — not done yet; scheduled, will report here.
   This is the one we owe you. Reading your note the other way: with #1628 on main, squared ReLU no longer turns NaN→0, so a NaN should now reach the output instead of hiding — meaning a main run is our NaN test as much as a crash test. Plan: build TensorFold from source at main on the same hardware class, replay the identical escalating profile (NC=2 suite → concurrency ladder 1/2/4/8/16 → 30-min NC=16 soak → the exact death config, NC=2 gen2-lite with 100% live traffic), persistent stderr from launch, results in this thread either way.

   3) KV dtype + exact serve flags — bf16, recipe defaults untouched.
   Container env: BOLT_KV=bf16 BOLT_KV_PIN=0 BOLT_SSM_LAZY=1 BOLT_DRAFT_Q8=1 BOLT_NV4_FP4=1 (scripts/config.sh defaults). Engine banner, verbatim:
   bolt: weights resident, 40 mamba 40 moe 8 attention, 16 sequences of context 262144 (bf16...
   Serve line, verified byte-identical between the October failing runs and the November relaunches (docker-inspect vs the original start log):
   tensorfold-native serve <snapshot-path> --name Nemotron-3.5-Bolt --host 0.0.0.0 --port 8888 --context 262144 --parallel 16 --max-tokens 32768 --temperature 1.0 --top-p 0.95 --prompt-cache-gib 8 --thinking --vision --vision-max-images 64 --vision-max-videos 4 --vision-image-tokens 262144 --vision-video-tokens 131072

   4) NC=2 — yes, the 500s strictly precede the exit. First 500 and last lines, verbatim:
   NC=2 suite, 2026-10-29, first task at item 24/70:

   Requesting API: 34%| 24/70 [06:12<10:49, 14.12s/it]
   2026-10-29:08:54:00 WARNING  [models.api_models:521] API request failed! Status code: 500, Response text: {"error": {"message": "BoltForward"}}. Retrying...
   2026-10-29:08:54:00 ERROR    Exception:ClientResponseError(... POST /v1/chat/completions ...)
   

   From that instant every request and every retry returns {"error": {"message": "BoltForward"}} — the engine never recovers — until the harness dies. Last lines of the run (client side):

     File "/home/daviddv/evals/lm-evaluation-harness/lm_eval/models/api_models.py", line 514, in amodel_call
       async with session.post(
     ...
     File "/home/daviddv/evals/.venv/lib/python3.12/site-packages/aiohttp/client.py", line 581, in _request
       raise RuntimeError("Session is closed")
   RuntimeError: Session is closed
   

   The engine process then exits rc=1. Its own last lines are not recoverable (same lost-stderr problem as item 1); the Nov relaunch's persistent stderr shows no exit because nothing died. The NC=4 run shows the same signature as a slower ramp: first 500 at item 140/3500 (16:39:25Z), then 516 status-500 lines over 5 h 38 m — all {"error": {"message": "BoltForward"}}, zero other status codes, zero KV-full/OOM/timeout lines.

   Repro status on pinned v1.1.6 (2026-10-31) — no fault found under full exposure.
   We rebuilt the recipe locally from identical pins (TF eea9dc9c9c8b, Zig 0.17.0, patches ad4a80c8bf48) — side note: the recipe's ghcr image publishes only :latest; the pinned-revision tag 404s, and a pinned tag would make community pinning byte-exact. Escalating load mirroring the death profile: NC=2 suite → ladder 1/2/4/8/16 → 30-min NC=16 soak (the Oct NC=4 engine died right after exactly this) → the death config itself: NC=2 gen2-lite suite, 100% live traffic (NO_CACHE). Ran to our 4 h cap, 2,012/3,500 live items: 0 HTTP 500, 0 BoltForward, /health 200 throughout and at teardown. The NC=4 ladder rung streamed clean (172.8 tok/s aggregate) where October had been spilling BoltForward lines. Total exposure 7 h 33 m — past the failing run's 5 h 38 m time-to-death. Control: same weights + same suite completed cleanly on the official vLLM stack on the same hardware.
   Caveats we own: different physical Spark unit; locally built binary (identical inputs, native binary sha differs — presumed Zig build non-determinism); earlier ladder phases mixed cached traffic (the final uncached phase compensated).

   So, directly to your "please confirm with our own build": we cannot make it fail — on the recipe image or on a from-pinned-inputs local build. Whether main changes that is exactly what item 2 will answer, and we'll post it here either way.

   One question while that runs: is engine-restart-on-first-BoltForward (the approach in drowzeys/Imagination-Studio#90) the best known mitigation for sustained-concurrency serving in the meantime?

   ---

   Bolt-74B TensorFold — reply package for #1654

   collected 2026-10-31 from run evidence (spark-820c control head / spark-7186 GPU node, GB10)

   A. Exact serve configuration (recipe-resolved, verbatim from launch log + config.sh defaults)

   tensorfold-native serve /root/.cache/huggingface/hub/models--nvidia--NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB/snapshots/a326822af111290f2af5ac59ef4b5040347dabc4 --name Nemotron-3.5-Bolt --host 0.0.0.0 --port 8888 --context 262144 --parallel 16 --max-tokens 32768 --temperature 1.0 --top-p 0.95 --prompt-cache-gib 8 --thinking --vision --vision-max-images 64 --vision-max-videos 4 --vision-image-tokens 262144 --vision-video-tokens 131072
   resolved defaults (scripts/config.sh): CONTEXT=262144 PARALLEL=16 MAX_TOKENS=32768 KV_DTYPE=bf16 KV_PIN=0 SSM_LAZY=1 DRAFT_Q8=1 NV4_FP4=1 DRAFTS=1 THINKING=1 VISION=1
   container env trio: BOLT_KV=bf16 BOLT_KV_PIN=0 BOLT_SSM_LAZY=1 (+BOLT_DRAFT_Q8=1 BOLT_NV4_FP4=1)
   engine: TensorFold v1.1.6 via recipe image zig-eea9dc9 (recipe repo @ 2cea422); weights = official model repo snapshot a326822af111...

   B. NC=2: YES — 500s precede the exit (verbatim client log, 2026-10-29 run, NC=2)

   Requesting API: 34%| 24/70 [06:12<10:49, 14.12s/it]
   2026-10-29:08:54:00 WARNING [models.api_models:521] API request failed! Status code: 500, Response text: {"error": {"message": "BoltForward"}}. Retrying...
   2026-10-29:08:54:00 ERROR [models.api_models:557] Exception:ClientResponseError(... POST /v1/chat/completions ...)
   ... (repeats per in-flight request; every retry after this point also fails)
   last lines: aiohttp RuntimeError: "Session is closed" - client gives up after retries exhausted against the poisoned engine.

   C. NC=4 run (2026-10-28): warning ramp quantified

   first 500 at item 140/3500 of first task: 2026-10-28:16:39:25 - Status code: 500, Response text: {"error": {"message": "BoltForward"}}
   from that moment to death (22:17:25, 5h38m): 516 x Status-500 lines, ALL BoltForward, zero other status codes.
   No KV-full, OOM, or timeout lines in client log - the engine itself answers 500 {"BoltForward"} for every request thereafter.

   D. Engine stderr: NOT persisted by our harness at run time (honest gap)

   start.sh output went to a launcher log that kept the startup banner but engine stderr rotated away with container teardown.
   Planned: relaunch of the SAME recipe image (still cached on our Spark) with persistent stderr capture; engine slot frees tonight
   after our other stack verdict files - capture posted here as follow-up.
   Build-from-main: queued as the next experiment after the stderr capture (needs ~1 GPU free window); we will report whether
   the deterministic trigger survives #1628.

   4. REPRO ATTEMPT on current recipe — NOT REPRODUCED (2026-10-31, node spark-8f32)

   Recipe @2cea422, image tensorfold-nemotron-bolt:zig-eea9dc9 (ghcr revision tag 404s - only :latest
   is published - so we BUILT from the pinned inputs: TF eea9dc9, Zig 0.17.0, patches
   ad4a80c8bf48; binary therefore not byte-identical to our Oct build, all inputs identical).
   Weights sha-identical snapshot a326822af111. Serve line verbatim-identical to the failing Oct run
   (docker-inspect vs start log): --context 262144 --parallel 16 --max-tokens 32768 --temperature 1.0
   --top-p 0.95 --prompt-cache-gib 8.
   Load profile applied (mirrors the deaths, escalating):
   P1 NC=2 full gen2-lite suite (cached mix) 10:28Z -> no fault
   P2 concurrency ladder 1/2/4/8/16 + 30min NC=16 11:48Z -> no fault (nc16 agg 286.68 tok/s)
   soak (D's engine died right after this profile)
   P4 NC=2 suite with ALL traffic uncached (the 13:34Z -> no fault: 2012/3500 live items,
   exact Oct death config), 4h cap 0x 5xx, 0 BoltForward; rc=124 = our cap
   Engine: 7h33m uptime, 7h30m under load (> Oct NC=4 time-to-death 5h38m), health 200 at end.
   Persistent raw engine stderr captured THROUGHOUT this time: 45,542 lines (the artifact missing
   from the Oct deaths - available as attachment; notable recurring line class:
   "[tensorfold] warning: a reply reached max_tokens while still thinking, so its content is empty
   and its text is truncated" x N - correlates with our early content=null observation; no crash-
   class lines: 0 OOM/panic/abort/BoltForward).
   Oct death record (for contrast): NC=4 suite -> 516x HTTP 500 {"error":{"message":"BoltForward"}}
   over 5h38m then rc=1; NC=2 -> 500s at item 24/70, rc=1. Engine stderr then: LOST (not persisted).
   Open question for you: does #1628 plausibly cover this failure mode? We can rebuild with TF_REF=
   main and re-run the identical profile - say the word. Caveats we own: different physical Spark
   unit; locally-built image (inputs pinned); Oct runs had 100%-live suite traffic vs our mix.
3. ashhart commented on Nov 2, 2026

   

   ashhart

   on Nov 2, 2026

   Owner

   More actions

   /atc claim
4. ashhart commented on Nov 2, 2026

   

   ashhart

   on Nov 2, 2026

   Owner

   More actions

   Thanks @DaviddevIDM, this is a very complete follow-up, and keeping stderr on the new runs is exactly what we needed.

   We traced BoltForward. It comes from the older libbolt adapter, which our tree replaced before v1.1.6: neither eea9dc9c9c8b nor current main returns that error anywhere. So the failing binary was built from the recipe's patch set on top of v1.1.6, not from our v1.1.6 alone. That also explains why your clean rebuild from pinned inputs couldn't make it fail.

   For the main run you're planning, it would help to include:

   • the tensorfold-native binary's sha256, the recipe's patch list, and the hash of any libbolt it loads, so we know which code produced each result;
   • for NC=2, engine stderr from startup through the first failed request, with requests still live.

   On your question: until we know the cause, restarting the whole process on the first failure is a reasonable stopgap, since one failure leaves every later request failing. We haven't validated restart-on-first-error as a fix, though, and #1628 isn't a confirmed fix either. If main built from our tree serves your workload cleanly, that points at the adapter; if it fails, the stderr will show us where.
5. DaviddevIDM commented on Nov 3, 2026

   

   DaviddevIDM

   on Nov 3, 2026

   Author

   More actions

   Your trace checks out on our artifacts — we can now show why our rebuild and the recipe image behaved differently, and what each binary is.

   Artifact chain, hashed (this is the "which code produced which result" table):

   artifact	sha256	note
   recipe patches/0002-register-bolt.patch	1d2b3e6181dcb6f76c60b4da41b48e8fc6848f662788fa715e609ea5387d562a	adds zig/build/bolt.zig: builds libbolt.so from -Dbolt-src .cu sources via nvcc and installs to native/lib/libbolt.so; recipe comments say "which a Bolt checkpoint opens"
   tensorfold-native (/usr/local/bin/)	5072f89187a2a3ba3c6e52ec59aa656cf41b3f1140a60fba1bfcfb24ac24b255	in the recipe image (zig-eea9dc9); contains the string BoltForward ×3 — matching your finding that neither eea9dc9c9c8b nor main returns it, so it entered via the recipe's patch set, not your tree
   /opt/tensorfold/native/lib/libbolt.so	3d84c0f368d30c14f93b2e41bfd0b53e5604e3d0edae1aa4173e912f93da1fe1	present in the image; not a dynamic dependency of the binary (ldd shows no bolt entry — consistent with the runtime "opens" path in the patch comment); contains 0 BoltForward strings — the error class is raised by the patched binary around the adapter call
   build pins (scripts/config.sh)	TF_REF=eea9dc9c9c8b459c0a03de7f6fe0c6f4835d69ab, ZIG_VERSION=0.17.0, BASE_IMAGE=nvcr.io/nvidia/pytorch:26.07-py3	recipe @ 2cea422

   One honest gap before the next line: our three October failing runs' exact binary can't be cryptographically re-identified. start.sh rebuilds the image when BASE_IMAGE mismatches and the ghcr publishes only :latest (repeated from last time, now with a concrete cost). Serve line and container env were byte-identical across all of it (verified via docker-inspect), but binary identity we can now only pin from 2026-11-01 forward: every sustained window since ran 5072f89187a2… with the adapter bundled — and none of them produced a single 500 or BoltForward, including the exact NC=2 death config at full live traffic. That silence after ~24 h of exposure on the patched build is itself consistent with your adapter theory being necessary-but-not-sufficient: the patched build needs whatever the October image had (older base-toolchain build of libbolt.so? different bolt-src?) that we can no longer reconstruct.

   The main run, with your hash protocol — planned, and one design question decided by your tree note:

   • Run A — main, your tree, recipe patches applied (so 0002-register-bolt re-registers the adapter): if the death class appears, BoltForward is reproducible on your current toolchain and stderr will catch it.
   • Run B — main, your tree alone (no recipe patches): by your trace this build cannot even produce that error class; a clean run here + dirty Run A closes the loop that the adapter is the trigger.
   • Both runs: tensorfold-native sha256 + libbolt.so sha256 captured from the build before first traffic; NC=2 suite traffic live; persistent engine stderr from startup through the first failed request (or through 8 h if none); posted here either way.

   One question: is 0002-register-bolt still the intended way to run Bolt-on-CUDA against current TF, or is that patch stale relative to the pre-v1.1.6 libbolt replacement — i.e., should a current Bolt checkpoint be served without the adapter at all? (If the latter, the recipe — MiaAI-Lab's, not yours — is the bug surface, and we'll file the same chain there.)