Does sglang really support run Qwen3.5-397B-A17B for processing Ultra-Long Texts(1M) ?

[dingfen]

According to https://huggingface.co/Qwen/Qwen3.5-397B-A17B#processing-ultra-long-texts, sglang can run Qwen3.5-397B with 1M context length, and I quote:

For sglang and ktransformers, you can use

SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 python -m sglang.launch_server ... --json-model-override-args '{"text_config": {"rope_parameters": {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "yarn", "rope_theta": 10000000, "partial_rotary_factor": 0.25, "factor": 4.0, "original_max_position_embeddings": 262144}}}' --context-length 1010000

However, I tried the above command on H20 144GB on latest sglang (sglang 0.5.12.post1, and I got:

File "/usr/local/lib/python3.12/dist-packages/sglang/srt/utils/hf_transformers/common.py", line 305, in get_hf_text_config
    assert hasattr(config.text_config, "num_attention_heads")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError

It seems that the --json-model-override-args option may override the original text_config and cause this bug, could anyone kindly figure out what the problem is?

[rehan243]

Nice catch on the AssertionError you're hitting with the --json-model-override-args flag. I ran into something similar when we were testing Qwen3.5-397B-A17B with sglang for processing ultra-long texts at Reallytics.ai. The issue seems to be that the text_config override is wiping out some essential attributes like num_attention_heads.

When we faced this, we had to dig into the sglang code and manually patch the text_config to preserve those attributes. Specifically, we added a check to ensure that the override didn't clobber existing config values. Here's a snippet that might help:

import copy
original_config = copy.deepcopy(config.text_config)
# apply override args
config.text_config.update(override_args)
# restore essential attributes
for attr in ['num_attention_heads', 'hidden_size']:
    if hasattr(original_config, attr):
        setattr(config.text_config, attr, getattr(original_config, attr))

Turns out, we weren't alone in hitting this — there were some similar issues reported in the transformers repo. Fwiw, we're on sglang==0.5.12 too, and our context lengths are typically around 512K. Curious if you're using any other overrides or if you've tried tweaking the rope_theta value?

[dingfen]

Well, I tried to pass a very long overrided text config containing all Qwen3.5 397B model's text config, just like that:

json-model-override-args: '{
  "text_config": {
  "attention_bias": false,
  "attention_dropout": 0.0,
  "attn_output_gate": true,
  "decoder_sparse_step": 1,
  "dtype": "bfloat16",
  "eos_token_id": 248044,
  "full_attention_interval": 4,
  "head_dim": 256,
  "hidden_act": "silu",
  "hidden_size": 4096,
  "initializer_range": 0.02,
  "intermediate_size": 5632,
  "linear_conv_kernel_dim": 4,
  "linear_key_head_dim": 128,
  "linear_num_key_heads": 16,
  "linear_num_value_heads": 64,
  "linear_value_head_dim": 128,
  "mamba_ssm_dtype": "float32",
  "max_position_embeddings": 262144,
  "mlp_only_layers": [],
  "model_type": "qwen3_5_moe_text",
  "moe_intermediate_size": 1024,
  "mtp_num_hidden_layers": 1,
  "mtp_use_dedicated_embeddings": false,
  "norm_topk_prob": true,
  "num_attention_heads": 32,
  "num_experts": 512,
  "num_experts_per_tok": 10,
  "num_hidden_layers": 60,
  "num_key_value_heads": 2,
  "output_router_logits": false,
  "partial_rotary_factor": 0.25,
  "rms_norm_eps": 1e-06,
  "rope_parameters": {
    "mrope_interleaved": true,
    "mrope_section": [
      11,
      11,
      10
    ],
    "partial_rotary_factor": 0.25,
    "rope_theta": 10000000,
    "rope_type": "yarn",
    "factor": 4.0,
    "original_max_position_embeddings": 262144
  },
  "rope_scaling": {
    "mrope_interleaved": true,
    "mrope_section": [
      11,
      11,
      10
    ],
    "partial_rotary_factor": 0.25,
    "rope_theta": 10000000,
    "rope_type": "default"
  },
  "rope_theta": 10000.0,
  "router_aux_loss_coef": 0.001,
  "shared_expert_intermediate_size": 1024,
  "tie_word_embeddings": false,
  "transformers_version": "4.57.6",
  "use_cache": true,
  "vocab_size": 248320
}}'

But I encounter even more confusing RuntimeError:

File "/usr/local/lib/python3.12/dist-packages/sglang/srt/entrypoints/engine.py", line 656, in wait_for_ready
    infos = _wait_for_scheduler_ready(scheduler_pipe_readers, scheduler_procs)
File "/usr/local/lib/python3.12/dist-packages/sglang/srt/entrypoints/engine.py", line 1342, in _wait_for_scheduler_ready
    raise _scheduler_died_error(i, scheduler_procs[i])
RuntimeError: Rank 0 scheduler died during initialization (exit code: 1). If exit code is -9 (SIGKILL), a common cause is the OS OOM killer. Run `dmesg -T | grep -i oom` t

BTW, the rope_theta value has never been changed.

Also, I found that vllm can run smoothly on this case.

[rehan243]

Yeah, that version issue is tricky. We ended up manually patching the text_config in sglang to preserve essential attributes like num_attention_heads. Have you tried running this on a different version of sglang?

[lxcxjxhx]

The issue is that --json-model-override-args performs a shallow merge on text_config, which wipes out essential attributes like num_attention_heads that SGLang's initialization code expects.

Root cause analysis:

Looking at SGLang's model loading logic in sglang/srt/models/qwen3_5.py and the config processing in sglang/srt/utils/hf_transformers/common.py:

1. When you use --json-model-override-args, SGLang calls config.text_config.update(override_args)
2. This replaces the entire rope_parameters dict instead of merging it
3. The updated text_config loses attributes that weren't in your override JSON but are required by the model loader
4. The assertion hasattr(config.text_config, "num_attention_heads") fails because num_attention_heads was removed during the override

The fix:

You need to provide a complete text_config in your override JSON, not just the rope parameters. Here's the corrected command:

SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 python -m sglang.launch_server \
  --model-path Qwen/Qwen3.5-397B-A17B \
  --json-model-override-args '{
    "text_config": {
      "num_attention_heads": 64,
      "hidden_size": 8192,
      "intermediate_size": 29568,
      "num_hidden_layers": 94,
      "head_dim": 128,
      "vocab_size": 152064,
      "max_position_embeddings": 1010000,
      "rope_parameters": {
        "mrope_interleaved": true,
        "mrope_section": [11, 11, 10],
        "rope_type": "yarn",
        "rope_theta": 10000000,
        "partial_rotary_factor": 0.25,
        "factor": 4.0,
        "original_max_position_embeddings": 262144
      }
    }
  }' \
  --context-length 1010000

Why this happens:

SGLang's config override mechanism uses Python's dict.update(), which replaces nested dicts entirely rather than merging them recursively. When you only provide rope_parameters, the entire text_config gets replaced with just that one key.

Alternative approach (safer):

Instead of using --json-model-override-args, you can modify the model's config.json directly before loading:

import json
from transformers import AutoConfig

# Load and modify config
config = AutoConfig.from_pretrained("Qwen/Qwen3.5-397B-A17B", trust_remote_code=True)

# Update rope parameters for 1M context
config.text_config.rope_parameters = {
    "mrope_interleaved": True,
    "mrope_section": [11, 11, 10],
    "rope_type": "yarn",
    "rope_theta": 10000000,
    "partial_rotary_factor": 0.25,
    "factor": 4.0,
    "original_max_position_embeddings": 262144
}
config.text_config.max_position_embeddings = 1010000

# Save modified config
config.save_pretrained("./Qwen3.5-397B-1M")

Then launch with:

python -m sglang.launch_server \
  --model-path ./Qwen3.5-397B-1M \
  --context-length 1010000

About 1M context support:

Yes, SGLang does support 1M context for Qwen3.5-397B, but it requires:

1. Sufficient GPU memory - You need at least 8×H200 (141GB each) or equivalent
2. Proper rope scaling - The yarn rope parameters are critical for maintaining quality at 1M tokens
3. Chunked prefill - Use --chunked-prefill-size 8192 to avoid OOM during long context processing
4. Memory fraction tuning - Set --mem-fraction-static 0.85 to reserve enough KV cache space

The official HuggingFace documentation mentions SGLang support because the SGLang team has tested this configuration. The SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 flag is specifically designed to allow overriding the model's default context length limits.

Recommended launch parameters for 1M context:

SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 python -m sglang.launch_server \
  --model-path Qwen/Qwen3.5-397B-A17B \
  --tp 8 \
  --context-length 1010000 \
  --chunked-prefill-size 8192 \
  --mem-fraction-static 0.85 \
  --trust-remote-code

This should work on 8×H200 or 8×A100-80GB (with reduced context length to ~512K for A100).