Native 1.0.12 grammar vocabulary lookup misses llm_config for Nemotron Super VL #951

TensorFold native 1.0.12 (v1.0.12, commit 833bbfcba9109bb38c7af7303f80da0ca12b213c) serves ordinary chat and automatic tools for NVIDIA Nemotron 3.5 Super VL, but rejects a named function, tool_choice: "required", and a strict JSON-schema response with HTTP 400:

structured output needs the checkpoint's config.json vocab_size

The error is prefixed with tool_choice "required" or a named function: for forced tools and structured output: for the schema response.

An existing fix is proposed in PR #897, by BobClawblaw, specifically commit 5f094543. It adds the missing llm_config lookup within the broader Bolt contribution. This issue records the released Super VL failure and links the existing proposed fix.

Environment

• Native Linux aarch64 1.0.12 release archive, CUDA backend on one GB10.
• Checkpoint: nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4, revision 82d93699acd3be9e4617c53360fe5b3f5dbeb1b3.
• Checkpoint precision, context argument --context 114688, four lanes.
• The grammar library is available; duplicating only the vocabulary field at the root makes the same requests succeed with the same release binary and weights.

Reproduction

Serve the unmodified checkpoint with native 1.0.12 and its grammar library installed:

tensorfold-native serve /path/to/checkpoint \
  --backend cuda --name nemotron-super-vl \
  --host 127.0.0.1 --port 8080 --context 114688 --parallel 4 \
  --precision checkpoint --thinking --vision \
  --vision-max-images 4 --vision-image-tokens 13312 --no-update-check

Send a strict schema request:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "nemotron-super-vl",
    "messages": [{"role": "user", "content": "Return an object with ok set to true."}],
    "temperature": 0,
    "max_tokens": 128,
    "chat_template_kwargs": {"enable_thinking": false},
    "response_format": {
      "type": "json_schema",
      "json_schema": {
        "name": "result",
        "strict": true,
        "schema": {
          "type": "object",
          "properties": {"ok": {"type": "boolean"}},
          "required": ["ok"],
          "additionalProperties": false
        }
      }
    }
  }'

The following alternative request reproduces the forced-tool failure. Use either "required" or the named-function object as tool_choice:

{
  "model": "nemotron-super-vl",
  "messages": [{"role": "user", "content": "Look up WidgetFactory with lookup_symbol."}],
  "temperature": 0,
  "max_tokens": 128,
  "chat_template_kwargs": {"enable_thinking": false},
  "tools": [{
    "type": "function",
    "function": {
      "name": "lookup_symbol",
      "parameters": {
        "type": "object",
        "properties": {"symbol": {"type": "string"}},
        "required": ["symbol"],
        "additionalProperties": false
      }
    }
  }],
  "tool_choice": {"type": "function", "function": {"name": "lookup_symbol"}}
}

Cause and expected behavior

This is the relevant shape of the official config (a reduced excerpt, not a complete loadable checkpoint):

{
  "model_type": "nemotron_h_omni",
  "eos_token_id": 11,
  "llm_config": {
    "model_type": "nemotron_h",
    "vocab_size": 131072,
    "eos_token_id": 11
  }
}

zig/src/server/server.zig at 1.0.12 checks only text_config.vocab_size, then root vocab_size. Neither exists in this checkpoint, so it refuses the request before opening the grammar compiler. The native Nemotron parser already reads the text model from llm_config for nemotron_h_omni.

Expected: the grammar compiler reads llm_config.vocab_size for this supported checkpoint, preserving existing text_config and root behavior, and enforces the named/required-tool or schema request without editing the checkpoint.

Evidence and proposed fix

Observed before the compatibility workaround: named tool, required tool and schema each returned HTTP 400 in two request repetitions. A separate derived config that duplicates llm_config.vocab_size as root vocab_size made all six requests return HTTP 200. The official checkpoint, release binary and weights were unchanged. This demonstrates the metadata lookup failure; it is not a native patched-binary GPU qualification.

The lookup is still missing llm_config on upstream main at 46a08c18608153bfc566703a88acc883bc6140f4 (checked 2026-10-16 UTC). PR #897 already proposes text_config, then llm_config, then root. A focused extraction of that lookup adds reduced Omni and existing-layout precedence regressions, with a positive u32 check before the cast. It is credited to the existing proposal. EOS handling is unchanged; both locations in the official config name token 11.

The proposed upstream fix is already available in PR #897, specifically commit 5f094543.

Local regression checks

The focused patch uses the repository's existing grammar module, JSON parser, arena-backed Zig tests and server test harness. It adds no dependency. With exact Zig 0.17.0 on macOS arm64:

ZIG=zig bash zig/tests/server/zig_test.sh --test-filter 'grammar vocabulary'
ZIG=zig bash zig/tests/server/zig_test.sh
zig build test test-golden -Dcpu=apple_m1 -j4 --summary all
zig fmt --check zig/src/server/grammar.zig zig/src/server/root.zig zig/src/server/server.zig
python3 tools/zig/lean_check.py

• Red: both lookup tests fail with the old text_config/root lookup extracted into the helper, before the final test layout and three additional boundary/fallback cases. The precedence table stops at its first mismatch. The reduced Omni config returns no vocabulary; a conflicting root vocabulary overrides llm_config.
• Green: focused server checks 5/5, JSON checks 5/5, engine-seam checks 168/168. The full server suite passes 370/370.
• Full host/golden run: 208/208 build steps succeed, 2,993/3,003 tests pass and 10 skip. After adding three final boundary/fallback cases, the final run also succeeds in all 208 build steps; it executes 763/764 tests with one skip, with the other test steps cached from the successful run. Frozen server goldens report 309/319 equal, 10 known differences and 0 unexpected differences.
• Formatting and whitespace checks pass. The lean checker retains its inherited 517 findings and adds zero.

The workaround only establishes the cause and a temporary compatibility path. The candidate source patch still needs normal review and native GPU qualification before claiming a fixed release.