Questions about the papaer "Fast Matrix Multiplications for Lookup Table-Quantized LLMs"

[sunzj]

Figure 2:
Quantized Weights:int4 -->unpacking(shift + mask):int16 -->vectorized table lookup.
So the indice of the lookup table is "int16"? As i understand, we can use a pair of int4 (int8) as indices.

2. The dequantized weights are two weight: float16x2, but matmual need float16. Convert float16x2->float16 is inefficient by OpenCL. Did you encounter such issue? Since i ecounter that issue when try to implement the "LUT" with OpenCL, you may not have such issue when use CUDA.

[HanGuo97]

Thanks for the question!

• Does CUDA allow indexing with int8?
• Not sure if I correctly understood your question, but the (vectorized) dequantized weights will be treated as separate weights. They will then be sent to TensorCore.

[sunzj]

Hi Han,

Thanks for your reply.

• The question is: whether the unpacking(shit + mask) is necessary. If we treat two int4 as int8 (0~256) and build a LUT: [int8]-->half2. Then we don't need to do unpacking.

[HanGuo97]

Hmm, it is not immediately clear to me if your suggestion is better (maybe it is!).