Illustration by tuput
English
Four papers from 2022 and 2023 cut the memory a large language model needs, from 8-bit LLM.int8 to 4-bit GPTQ, AWQ and QLoRA. Here is what each paper measured, in the authors' own numbers, and which formats are in use now.
In August 2022 Tim Dettmers, Mike Lewis, Younes Belkada and Luke Zettlemoyer reported a way to store a language model’s weights in 8 bits instead of 16, which cut the memory needed to run it by half with no loss of accuracy in their tests, up to 175 billion parameters. In October 2022 a team at ETH Zurich and IST Austria published GPTQ, which pushed the weights down to 3 or 4 bits and ran the 175 billion parameter OPT model on a single 80 GB graphics card. In May 2023 the QLoRA paper fine-tuned a 65 billion parameter model on one 48 GB card, a job its authors say needs more than 780 GB in ordinary 16-bit form.
All of this is quantisation: storing a model’s numbers with fewer bits. The Hugging Face documentation describes it as lowering the memory needed to load and use a model by keeping the weights at lower precision while trying to preserve as much accuracy as possible. This article goes through the papers in order, says what each one measured, and marks which claims come only from the authors or the company selling the hardware.
A weight is a number, and the bits decide how finely it is written
A language model’s parameters, also called weights, are the numbers it learned in training. GPT-3 has 175 billion of them. At two bytes (16 bits) each that is 350 GB, the figure the AWQ paper gives; the GPTQ paper counts 326 GB because it uses binary gigabytes. The GPTQ authors say that exceeds the capacity of even the highest-end single graphics card, and the AWQ paper notes that Nvidia’s newer B200 has 192 GB.
A 16-bit number can take 65,536 distinct patterns, an 8-bit number 256 and a 4-bit number 16. That is plain arithmetic, and it is the whole trade. NVIDIA’s NVFP4 post lists the magnitudes a 4-bit floating-point value can hold: 0, 0.5, 1, 1.5, 2, 3, 4 and 6, with a sign. Every weight in a matrix has to be rounded to one of those few points, after the matrix is scaled so that its largest value lands at the top of the range.
Here is a made-up example, not taken from any paper. If a row of weights holds values near 0.02 and 0.4 but also one at 12, scaling to 8 bits puts steps of about 0.09 across the range, and 0.02 rounds to zero. One large value coarsens the grid for all the others. The methods below are different ways of dealing with that.
Real formats also carry scale constants, so “4-bit” is a little more than four. The llama.cpp README tabulates Llama 3.1 8B: 16.0005 bits per weight and 14.96 GiB at 16 bits, 8.5008 bits and 7.95 GiB in its 8-bit format, Q8_0, and 4.8944 bits and 4.58 GiB in Q4_K_M, a 4-bit format its examples use. NVIDIA’s NVFP4 stores one 8-bit scale per 16 values, which it counts as 4.5 bits per value.
LLM.int8(): the 0.1 percent of values that broke 8-bit
The LLM.int8() paper, presented at NeurIPS 2022, starts from a failure. Earlier 8-bit methods had only been studied on models below 350 million parameters, and the authors found that plain 8-bit rounding failed as models grew. From about 6.7 billion parameters, a handful of hidden dimensions carry values up to 20 times larger than the rest. At 6.7 billion, they report about 150,000 such outliers per sequence, concentrated in only 6 feature dimensions. Setting those dimensions to zero degraded perplexity, a score where lower is better, by 600 to 1,000 percent, though they are about 0.1 percent of the features.
The fix is a split. Dimensions holding a value of magnitude 6 or more are multiplied in 16-bit, and the other 99.9 percent or so in 8-bit, with a separate scale for each inner product. On the paper’s C4 test, a 13 billion parameter model scored a perplexity of 12.45 in 32-bit and 19.08 with plain 8-bit rounding, and 12.45 with LLM.int8(). The authors report no degradation on OPT models up to 175 billion parameters and say it makes OPT-175B and BLOOM usable on a single server with consumer GPUs. Their software was integrated into Hugging Face Transformers.
They also state a limit. The 8-bit method can run slower than 16-bit on models below 6.7 billion parameters, though those fit on most GPUs anyway, and it is about twice as fast on the large matrix multiplications of a 175 billion parameter model.
GPTQ: compressing after training in about four GPU hours
LLM.int8() keeps 8 bits. The GPTQ paper, from Elias Frantar, Saleh Ashkboos, Torsten Hoefler and Dan Alistarh, aimed lower and with no retraining. It is a one-shot method, meaning the finished model is rounded in one pass using approximate second-order information, and the authors report quantising a 175 billion parameter model in about four GPU hours on one A100.
Its headline table is on OPT-175B, measured as WikiText2 perplexity. The 16-bit model scores 8.34. GPTQ at 4 bits scores 8.37, while simple round-to-nearest scores 10.54. At 3 bits round-to-nearest collapses to about 7,300 and GPTQ scores 8.68. The authors call the accuracy loss negligible and note that larger models generally looked easier to quantise, with OPT-66B an exception.
The memory arithmetic is in the paper. At 3 bits OPT-175B takes about 63 GB, plus about 9 GB for the cache of past keys and values at 2,048 tokens, so it fits in one 80 GB card. The authors say 16-bit execution needs five such cards and LLM.int8() needs three. They also report generation 3.25 times faster than 16-bit on A100s and 4.5 times on cheaper A6000s, and state that the gain comes from loading less memory, since mainstream hardware could not yet multiply 16-bit by 4-bit numbers directly.
AWQ: protect 1 percent of the weights, chosen by looking at activations
The AWQ paper, with Ji Lin as first author and Song Han of MIT as corresponding author, starts from an observation: not all weights matter equally. Keeping only about 1 percent of the weights in 16-bit greatly reduces the damage from rounding the rest to low precision.
Which 1 percent is the finding. Its Table 1 uses OPT-6.7B at 3 bits with groups of 128 weights, scored in WikiText perplexity, where the 16-bit model gets 10.86 and round-to-nearest gets 23.54. Keeping 1 percent of the weights in 16-bit, chosen by how large the activations feeding them are, gives 11.39. Choosing the same share by the weights’ own size gives 22.37, and choosing at random gives 24.23. The salient weights are found from the data flowing through the model, not from the weights.
Mixed 16-bit and 4-bit storage is awkward for hardware, so AWQ scales the salient channels up before rounding instead, which reaches similar scores without the split. The authors pair it with TinyChat, an inference system, and report an average speedup of 3.2 to 3.3 times over Hugging Face’s 16-bit code on desktop, laptop and mobile GPUs. They report Llama-2-70B on a single Jetson Orin with 64 GB and a 13 billion parameter model at 30 tokens a second on a laptop RTX 4070 with 8 GB. These are the authors’ measurements. The paper’s 2026 revision lists an MLSys 2024 Best Paper Award.
LoRA and QLoRA: fine-tuning a frozen 4-bit model
Fine-tuning retrains a model on new data, and it costs far more memory than running it. LoRA, from Edward Hu and colleagues at Microsoft in June 2021, froze the pretrained weights and trained small added matrices instead. For GPT-3 175B the authors report 10,000 times fewer trainable parameters and 3 times less GPU memory than fine-tuning with the Adam optimiser, with quality on par or better on the models they tested and no added delay when the model runs.
QLoRA joined the two ideas. It keeps the pretrained model frozen in 4 bits and passes gradients back through it into LoRA adapters. The paper adds three pieces. NormalFloat 4, or NF4, spaces the 16 available values to match the bell-curve shape of trained weights, so each value covers about the same number of weights. Double quantisation also rounds the scale constants, saving about 0.37 bits per parameter, or about 3 GB for a 65 billion parameter model. Paged optimisers move training state to CPU memory when the GPU runs short.
The authors report this brings 65 billion parameter fine-tuning from more than 780 GB to under 48 GB, with no loss against 16-bit fine-tuning on the tasks they tested. They released models called Guanaco. The paper gives Guanaco 65B as 41 GB and Guanaco 7B as 5 GB, which it says fits modern phones.
The formats in use today
The Hugging Face overview page, read on 12 November 2026, lists 22 quantisation methods in its compatibility table. The table includes AWQ at 4 bits, bitsandbytes (the library the QLoRA paper links to) at 4 and 8, NVFP4 at 4 and two FP8 methods at 8, along with GGUF for llama.cpp and GPT-QModel, a toolkit whose own README lists GPTQ among its formats.
GGUF is the file format used by llama.cpp, a program for running models on ordinary computers. Its specification describes a single file that holds the weights and everything needed to load them. The llama.cpp README says quantisation shrinks a model and can speed it up, may cost accuracy that is usually measured in perplexity or KL divergence, and that the loss can be reduced with an importance matrix, a file of data used to decide which weights matter more.
FP8 keeps eight bits but spends them as a small floating-point number. A 2022 paper by Paulius Micikevicius and 14 co-authors proposed two layouts, E4M3 and E5M2, and reported matching 16-bit quality in training runs of up to 175 billion parameters. NVIDIA’s NVFP4 post of 24 June 2025 describes a 4-bit format that cuts memory by about 3.5 times against 16-bit and reports 1 percent or less accuracy loss against FP8 on DeepSeek-R1-0528 in seven evaluations. That figure is NVIDIA’s own test of its own format.
What each claim rests on
Every result above is for a named model and a named test. GPTQ’s numbers are perplexity on OPT and BLOOM, AWQ’s tables use OPT and Llama-2, and the NVFP4 figure is one model. Perplexity measures how well the model predicts text, and it is not the same as how useful the model is to a user. The llama.cpp README itself says a quantised model may lose accuracy.
QLoRA’s most quoted number is 99.3 percent. It is the score of Guanaco 65B relative to ChatGPT on the Vicuna benchmark, an 80-prompt test in which GPT-4 acts as the judge and scores both answers, with a 95 percent confidence interval of plus or minus 4.4 points in the paper’s table. The authors say the intervals are very wide across the table and many models overlap, and they conclude that current chatbot benchmarks are not trustworthy for measuring chatbot performance. The problem is covered in the piece on AI benchmarks. How the models being squeezed were made is in the explainer on training a large language model.
Small models for Indian devices
Sarvam AI’s post of 14 February 2026 announcing Sarvam Edge says its models are designed to run on the devices people already have. It lists a translation model of about 150 million parameters and about 334 MB on the device, covering 10 Indian languages and English, and does not say which quantisation format, if any, it uses. The phone-side picture is in the look at small models on phones, and the wider push is in the piece on India’s own-language AI.
Sources & further reading
- Dettmers, Lewis, Belkada and Zettlemoyer: LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (arXiv:2208.07339, NeurIPS 2022)
- Frantar, Ashkboos, Hoefler and Alistarh: GPTQ, Accurate Post-Training Quantization for Generative Pre-trained Transformers (arXiv:2210.17323, ICLR 2023)
- Lin et al.: AWQ, Activation-aware Weight Quantization for LLM Compression and Acceleration (arXiv:2306.00978, MLSys 2024)
- Dettmers, Pagnoni, Holtzman and Zettlemoyer: QLoRA, Efficient Finetuning of Quantized LLMs (arXiv:2305.14314, 23 May 2023)
- Hu et al.: LoRA, Low-Rank Adaptation of Large Language Models (arXiv:2106.09685, 17 June 2021)
- Micikevicius et al.: FP8 Formats for Deep Learning (arXiv:2209.05433, 12 September 2022)
- Hugging Face: Transformers documentation, Quantization overview
- ggml-org: llama.cpp quantize tool README
- ggml-org: GGUF file format specification
- NVIDIA Technical Blog: Introducing NVFP4 for Efficient and Accurate Low-Precision Inference (24 June 2025)
- Sarvam AI: Announcing Sarvam Edge (14 February 2026)
Researched and written with the help of AI tools and edited for accuracy. Provided for general information and discussion only, not professional advice. See our editorial standards and disclaimer. Spotted an error? Tell us.
Enjoyed this? Get the next one.
One good read at a time, straight to your inbox. No spam, unsubscribe anytime.