Since experts are 4 bit, we can do this:
-
Compress experts (call it Q) into two matrices AB of medium rank, say r=64 or r=128.
-
As experts are integers only, this LoRA has a margin of error of ±0.5 to work with before we change any one weight (if we round)
-
Normally, LoRA optimizes the L² error (euclidean distance between two vectors). But here we instead optimixe the L∞ (maximum error). Since our goal is to ensure no value in AB exceeds an error of 0.5, we minimize the maximum error (via Alternating Linear Programming or L-BFGS)
-
Since we may not be able to achieve full minimax, keep a sparse matrix S that stores any non zero error between each element and round(AB).
Now we have compressed experts by a factor of multiple times making it far easier to lazily load and avoid long loading times if the predictor was wrong.
Additional potential optimization:
instead of x(round(AB)+S), do (xA)B + xS. this reduces computation (by a lot!) and memory (we do not need to store full matrix in RAM) even further with minimal accuracy loss (with a caveat that we must make the rounding error over columns zero-sum. basically: y_fast_diff_j = mean_x × sum_d_in(E_ij) + sum_d_in(fluctuation_i×error_ij), if we make the rounding error over columns zero sum, the first term is completely eliminated, and all that is left is noise from the fluctuations which does not hurt too much)
Since experts are 4 bit, we can do this:
Compress experts (call it Q) into two matrices AB of medium rank, say r=64 or r=128.
As experts are integers only, this LoRA has a margin of error of ±0.5 to work with before we change any one weight (if we round)
Normally, LoRA optimizes the L² error (euclidean distance between two vectors). But here we instead optimixe the L∞ (maximum error). Since our goal is to ensure no value in AB exceeds an error of 0.5, we minimize the maximum error (via Alternating Linear Programming or L-BFGS)
Since we may not be able to achieve full minimax, keep a sparse matrix S that stores any non zero error between each element and round(AB).
Now we have compressed experts by a factor of multiple times making it far easier to lazily load and avoid long loading times if the predictor was wrong.
Additional potential optimization:
instead of x(round(AB)+S), do (xA)B + xS. this reduces computation (by a lot!) and memory (we do not need to store full matrix in RAM) even further with minimal accuracy loss (with a caveat that we must make the rounding error over columns zero-sum. basically: y_fast_diff_j = mean_x × sum_d_in(E_ij) + sum_d_in(fluctuation_i×error_ij), if we make the rounding error over columns zero sum, the first term is completely eliminated, and all that is left is noise from the fluctuations which does not hurt too much)