Extreme Low-Bit Inference in Reasoning Models

Code and experiments for:

Extreme Low-Bit Inference in Reasoning Models: Failure Modes and Targeted Recovery arXiv: 2606.02011

TL;DR

Low-bit quantization can make reasoning models cheaper per token, but that does not automatically make reasoning cheaper end to end.

In long reasoning tasks, aggressive low-bit inference can change the generation process itself: the model may produce longer traces, repeat itself, delay the final answer, exhaust the token budget, or fail to close the reasoning segment.

This repository studies those process-level failure modes and lightweight ways to recover reasoning quality while preserving low-bit speed.

The core idea is simple:

Reasoning traces do not have to be generated by one model at one precision.

A high-precision model can plan or rescue difficult parts, while a low-bit actor does most of the long reasoning work.

Keywords

low-bit reasoning, quantized reasoning models, extreme low-bit inference, 2-bit quantization, GPTQ, NVFP4, KV-cache quantization, reasoning acceleration, chain-of-thought, long reasoning traces, token inflation, loop detection, loop rescue, FP16 planning, hybrid inference, staged inference, vLLM, Qwen3, MATH-500, AIME, GSM8K.

What problem does this repository study?

Reasoning models are expensive because they generate long traces, not just final answers. A quantized model may be faster per generated token, but still lose end-to-end speed if it generates too many tokens or gets stuck in unstable reasoning behavior.

We therefore evaluate quantization at the level of the full reasoning trace:

• final-answer accuracy;
• parse success;
• generated token count;
• delayed commitment;
• repetitive loops;
• unclosed reasoning segments;
• budget exhaustion;
• per-stage latency and throughput.

The goal is not just to ask:

Does low-bit inference preserve accuracy?

but also:

Does low-bit inference preserve the reasoning process well enough to deliver real wall-clock speedup?

Main idea

This repository implements multi-stage reasoning execution.

A single solution is represented as a Trace. Different actors append different TraceSegments to the same reasoning history:

• a full-precision planner can produce a short outline;
• a quantized actor can generate the main reasoning trace;
• a full-precision model can verify, finalize, or rescue unstable generations.

The result is one coherent solution trace, not a set of independent generations.

Main findings covered by the code

1. Extreme low-bit inference can fail through process instability. The model may generate longer traces, repeat reasoning steps, miss the final answer format, or run out of budget.

2. Accuracy alone hides important failures. Parse rate, reasoning length, loop behavior, and commit timing are needed to understand quantized reasoning.

3. FP16 planning can stabilize low-bit reasoning. A short high-precision plan can guide the low-bit actor through the main reasoning phase.

4. Loop rescue can recover failed traces. Repetitive or unstable traces can be detected and either committed early or routed back to a stronger model.

5. Hybrid execution can preserve low-bit speedups. Instead of running the entire trace in full precision, the expensive model is used only where the reasoning process is most sensitive.

What is in here

Component	Purpose
Trace representation	Stores one coherent reasoning history with multiple model-produced segments.
Hybrid reasoning pipelines	Run FP16 → low-bit → FP16-style execution schemes.
Quantized actor configs	GPTQ 2-bit, NVFP4, NVFP4 KV-cache, FP16 baselines, and hybrid variants.
Quality runs	Evaluate accuracy, parse rate, reasoning length, loops, commit gap, and related metrics.
Throughput benchmarks	Measure per-token and end-to-end serving behavior under vLLM.
Staged execution	Run one stage for all examples, unload the model, then run the next stage.
Timing replay	Re-run saved trace segments to get cleaner latency measurements.
Artifacts	Save traces, judgements, metrics, timing, configs, summaries, and errors.

Supported benchmarks

The repository includes adapters and configs for reasoning and QA benchmarks such as:

• MATH-500;
• GSM8K;
• AIME;
• ARC;
• PIQA;
• WinoGrande;
• StrategyQA.

Some datasets are committed under data/; others are loaded from HuggingFace at runtime.

Models and precision modes

The paper runs focus on Qwen reasoning models and variants such as:

• Qwen3-32B FP16;
• Qwen3-32B GPTQ 2-bit;
• Qwen3-32B NVFP4;
• Qwen3-32B NVFP4 with NVFP4 KV cache;
• Qwen3-8B FP16;
• Qwen3-8B NVFP4;
• Qwen3.6-35B MoE NVFP4.

The code is organized so new actors, backends, benchmarks, and metrics can be added.

Install

python -m venv .venv
source .venv/bin/activate

pip install -r requirements-vllm-cu130.txt
pip install -e .

source scripts/vllm_env.sh

The reported runs used a single 8×B200 machine with Python 3.11, CUDA 12.9, vLLM 0.21.0, torch 2.11+cu130, and transformers 5.8.

Before editing YAMLs, set:

actors:
  - backend_kwargs:
      cuda_visible_devices: "<your GPU ids>"

Quick start

Run one quality experiment:

python scripts/run_experiment.py configs/math500/qwen32b_fp16/hybrid_fp16_fp16.yaml -v

Run a hybrid low-bit experiment:

python scripts/run_experiment.py configs/math500/qwen32b_gptq2bit/hybrid_fp16_gptq2bit.yaml -v

Artifacts are written to:

results/<category>/<run_id>/

Typical files:

config.json
traces.jsonl
judgements.jsonl
metrics.jsonl
timing.jsonl
summary.json
errors.jsonl

Throughput / KV-cache benchmarks

Use:

python scripts/bench_qwen_throughput.py \
    --model "Qwen/Qwen3-32B" \
    --batch-mode max \
    --max-num-seqs-max 8 \
    --n-prompts 16 \
    --max-new-tokens 32768 \
    --max-model-len 36864 \
    --kv-cache-dtype auto \
    --gpu 0 \
    --gpu-memory-utilization 0.85 \
    --no-enforce-eager \
    --warmup 2 \
    --no-isolated \
    --output-dir results/perf_batch_kv/bs8_kvauto_mnt32768

The main number is:

throughput_tokens_per_sec

Useful launch scripts:

scripts/launch_perf_h1h5.sh
scripts/launch_8b_bs1_sweep.sh
scripts/launch_batch_kv_sweep.sh
scripts/launch_capacity_sweep.sh
scripts/launch_moe_kv_sweep.sh
scripts/launch_perf_nvfp4_kv.sh
scripts/launch_perf_queue.sh

Reproduce configs

Canonical configs live under:

configs/<dataset>/

Common groups:

qwen32b_fp16/
qwen32b_gptq2bit/
qwen32b_nvfp4/
qwen32b_nvfp4_kv4/
qwen8b_fp16/
qwen8b_nvfp4/
qwen8b_moe35b_nvfp4/

Regenerate final YAMLs from templates:

python scripts/generate_final_configs.py

Metrics

The runner evaluates both final quality and reasoning behavior.

Common metrics include:

Metric	Meaning
accuracy	Whether the extracted final answer is correct.
parse_rate	Whether the final answer was extracted reliably.
reasoning_length	Total generated tokens in the trace.
loop_detected	Whether repetitive reasoning behavior was detected.
loop_onset_tokens	Approximate token index where a loop begins.
commit_gap	Tokens generated after the first answer-like candidate.
tokens_to_first_correct	Approximate tokens until the correct answer first appears.
finish_commit	Whether the answer is visibly committed before final boxed output.
verification_spiral	Count of re-check / correction phrases.
actor_token_split	Token count by actor.
total_generation_ms	Total generation time over timed segments.
segment_timing_ms	Generation time by actor.
tokens_per_second	Tokens per second by actor.

Per-example metrics are saved to:

metrics.jsonl

Aggregate metrics are saved to:

summary.json

Timing replay

For fairer latency comparisons, replay saved trace segments one request at a time:

python scripts/replay_timing.py <run_id> --results-dir results -v

Or pin one model for all segments:

python scripts/replay_timing.py <run_id> \
    --model-id Qwen/Qwen3-32B \
    --precision bf16

Outputs are written to:

results/<run_id>/timing_replay/

Project structure

configs/      experiment YAMLs
data/         benchmark data and vendored files
quantlab/     trace, config, actors, runner, metrics, evaluation, artifacts
scripts/      experiment launchers, throughput benchmarks, replay, data prep
results/      saved run artifacts
tests/        tests

Citation

If you use this code or data, please cite:

@article{alimaskina2026extreme,
  title   = {Extreme Low-Bit Inference in Reasoning Models: Failure Modes and Targeted Recovery},
  author  = {Alimaskina, Ekaterina and Rudas, Darya and Shveykin, Denis and Molodtsov, Gleb and Vasiliev, Pavel and Beznosikov, Aleksandr},
  journal = {arXiv preprint arXiv:2606.02011},
  year    = {2026},
  doi     = {10.48550/arXiv.2606.02011},
  url     = {https://arxiv.org/abs/2606.02011}
}