Instructions for Reproducing Table 1 of the HIGGS Paper

[badeok0716]

Dear Han Guo,

Hello, I have read the paper "Pushing the Limits of Large Language Model Quantization via the Linearity Theorem", which utilizes FLUTE kernel as a backbone kernel for the efficient inference of non-uniformly quantized LLMs.

I would like to reproduce the “FLUTE” results reported in Table 1 for Llama‑3.1‑8B on an RTX 4090 (shown below for convenience):

	2 bits	3 bits	4 bits
bs = 1	173	150	139
bs = 4	687	592	548
bs = 16	2432	2122	1979

Could you please share the exact steps needed to reproduce these numbers? Any scripts or configuration files you can provide would be greatly appreciated.

Thank you for your time and for the excellent work.

[HanGuo97]

Thank you for your interest!

@BlackSamorez may have more information.

Just a brief update: the library has recently changed, especially with the introduction of a new autotune feature. This change didn’t seem to go well with some other features (such as 2/3-bit). It might be a good idea to revert to a prior version (before v0.4.0) for this purpose.

[BlackSamorez]

Hi @badeok0716 !

We used the FLUTE vLLM integration.

Specifically, I think we used the 0.2.6 version. That version didn't support integrated Hadamard Transform at the time, so, if my memory servers me well, we hard-coded the hadamard_transform from fast_hadamard_transform like this x = hadamard_transform(x.view(-1, 1024), scale=1024**(-1/2)).view_as(x) around here.
To test the generation speed at different batch sizes, we used those command:

• 1: CUDA_VISIBLE_DEVICES=1 python -m flute.integrations.benchmark_vllm --model <MODEL> --input_len 1 --output_len 256 --max-model-len 8192 --batch-size 1 --num_iters 32 --num-iters-warmup 4
• 4: CUDA_VISIBLE_DEVICES=1 python -m flute.integrations.benchmark_vllm --model <MODEL> --input_len 1 --output_len 256 --max-model-len 8192 --batch-size 4 --num_iters 10 --num-iters-warmup 4
• 16: CUDA_VISIBLE_DEVICES=1 python -m flute.integrations.benchmark_vllm --model <MODEL> --input_len 1 --output_len 256 --max-model-len 8192 --batch-size 16 --num_iters 10 --num-iters-warmup 4
• 64: CUDA_VISIBLE_DEVICES=1 python -m flute.integrations.benchmark_vllm --model <MODEL> --input_len 64 --output_len 1 --max-model-len 8192 --batch-size 1 --num_iters 256 --num-iters-warmup 4
• 256: CUDA_VISIBLE_DEVICES=1 python -m flute.integrations.benchmark_vllm --model <MODEL> --input_len 256 --output_len 1 --max-model-len 8192 --batch-size 1 --num_iters 64 --num-iters-warmup 4
• 1024: CUDA_VISIBLE_DEVICES=1 python -m flute.integrations.benchmark_vllm --model <MODEL> --input_len 1024 --output_len 1 --max-model-len 8192 --batch-size 1 --num_iters 64 --num-iters-warmup 4

Llama-3.1 models from the HF collection should probably work there. Although we changed the interface once or twice. We'll have to check in those checkpoints are still/yet compatible.

[badeok0716]

Hi @BlackSamorez,

Thanks for the quick follow‑up!

I tried the commands with v0.2.6, but it looks as though benchmark_vllm.py is missing from the flute/integrations package in that tag (see here: https://github.com/HanGuo97/flute/tree/v0.2.6/flute/integrations).
Because of that, the run ends with

No module named 'flute.integrations.benchmark_vllm'

I may simply be looking in the wrong place—could you point me to the correct script or commit that contains benchmark_vllm.py?

If the script has been replaced by a newer benchmarking workflow, I’m equally happy to use that. Any pointer (scripts, configs, or a simpler benchmarking recipe) would be greatly appreciated.

Many thanks!