LLM Quantization: INT8, GPTQ, and AWQ Explained
The weights don't fit on the GPU you have. How LLM quantization shrinks them to INT8 or INT4, why GPTQ and AWQ beat naive rounding, and where it breaks.
The weights don't fit on the GPU you have. How LLM quantization shrinks them to INT8 or INT4, why GPTQ and AWQ beat naive rounding, and where it breaks.