LLM quantization, explained: GGUF, AWQ, GPTQ and what Q4 really costs

Quantization is why a 70-billion-parameter model can run on one server instead of a rack. Understanding it well is the difference between an affordable deployment and an over-provisioned one.

What quantization does

Model weights are normally stored as 16-bit numbers. Quantization stores them at lower precision — 8-bit, 4-bit, sometimes lower — shrinking memory needs and speeding up inference. A 70B model needs roughly 140 GB of VRAM at 16-bit but around 40 GB at 4-bit, moving it from multi-node territory onto a single multi-GPU server.

The formats you will meet

• GGUF — the format used by llama.cpp and Ollama. Runs on CPUs and consumer GPUs; the workhorse for smaller deployments and edge devices.
• GPTQ — GPU-oriented post-training quantization, widely supported by serving stacks; a solid default for 4-bit GPU inference.
• AWQ — activation-aware quantization that protects the weights that matter most; often preserves quality slightly better than naive 4-bit at similar size.
• FP8 / INT8 — the conservative choice on modern data-centre GPUs: near-lossless quality with a meaningful memory saving.

What you lose at 4-bit

On general language tasks, well-made Q4 models typically stay within a few percent of full-precision quality — often imperceptible in practice. Degradation shows up first in the hardest cases: long multi-step reasoning, edge-case code generation, and low-resource languages. The honest answer is workload-specific, which is why we benchmark quantized candidates on your tasks rather than quoting leaderboard numbers.

Practical pairings

• Pilot or small team: 7–14B model, Q4 GGUF, one workstation-class GPU.
• Department workhorse: 30–70B model, AWQ or GPTQ 4-bit, single server with 2–4 data-centre GPUs, served by vLLM.
• Quality-critical enterprise: 70B+ at FP8/INT8 on a multi-GPU node — spend memory to protect the hard cases.

Three rules from our deployments

1. Quantize the biggest model that fits before choosing a smaller full-precision one — a Q4 70B usually beats an FP16 13B.
2. Never accept a quantized model without running your own evaluation suite against the full-precision baseline.
3. Leave VRAM headroom for context: long-context workloads consume memory beyond the weights, and undersizing here is the most common sizing mistake we see.