How to Choose the Right LLM Quant for Your GPU

Every open-weight model on Hugging Face comes in a dozen versions: BF16, FP8, AWQ, GPTQ, Q8_0, Q6_K, Q4_K_M, IQ3_XXS, NVFP4, MLX 4-bit. Most guides say "use Q4_K_M" and stop there. That's a reasonable default, but it isn't always the right answer. On my four RTX 3090s, two builds both labelled 4-bit ran 13% apart, and a dtype setting that suited the draft model mattered more than the bit count.

This post is how I work out which quant to use on a given card now.

TL;DR

1. Know your card: how much memory, how much bandwidth, and which number formats its tensor cores support.
2. Budget memory for the KV cache first, then fit the weights into what's left.
3. Use a format your engine runs fast on your card: GGUF for llama.cpp, Ollama and LM Studio; AWQ, GPTQ, INT8 or FP8 for vLLM and SGLang; MLX on a Mac.
4. 4-bit is the default for a reason. Single-request decode speed is roughly memory bandwidth divided by model size, and research finds 4-bit gives about the best accuracy per bit.
5. Benchmark on your own prompts with the features you'll actually use, especially speculative decoding.

The shortcut: tools that do most of this for you

Before any math, these four tools answer most of the question:

1. Tell Hugging Face what hardware you have. Add your GPUs (or Mac, or CPU) at huggingface.co/settings/hardware. After that, model pages with GGUF or MLX files show a Hardware compatibility panel that estimates which quants will fit. It's an estimate of whether the weights fit, not how fast they'll run, and your hardware list is public by default.

2. Let llama.cpp size it for you. Recent llama.cpp builds have --fit turned on by default: it picks the context size and GPU offload that fit your free memory. To see what it would choose without starting a server:

   llama-fit-params --model model-Q4_K_M.gguf

   It prints the -c, -ngl and offload settings it would use. If a quant only fits with a tiny context or with layers on the CPU, step down a size.

3. Check vLLM's hardware table. The "Supported Hardware" table on vLLM's quantization page shows which methods run on which GPU generation. For example, FP8 W8A8 is ❌ on Ampere and ✅ on Ada and Hopper. It currently has no column for RTX 50 cards and no row for NVFP4, so check the docs for those separately.

4. Look inside the quant before you download it. For vLLM-style models, the quantization_config block in config.json shows the real bit width, group size and which layers were left alone. For GGUF, open the file's info panel on Hugging Face (or add ?show_tensors=<file>.gguf to the URL) to see each tensor's type. Two quants with the same label can be quite different, as I found out below.

Then shortlist the two biggest quants that fit, and test both with the speed and quality checks at the end of this post. The rest of the post explains what's going on underneath, so you can tell when the tools' answer is wrong for you.

What a quant changes

A quant stores the model's weights in fewer bits. The BF16 version of the Qwen3.8 27B I run is 55.6 GB on Hugging Face (that includes a vision encoder). The same model as GGUF Q8_0 is 29.1 GB, Q6_K is 22.4 GB and Q4_K is 16.8 GB.

That matters for speed because of how generation works. To produce each token, the GPU reads every active weight from memory once. On a single request the math is quick and the memory reads are the bottleneck, so a good first estimate is:

max decode tok/s ≈ memory bandwidth (GB/s) ÷ model size (GB)

Real numbers come in below that ceiling, but it gets the ratios right. For a 27B at Q4 (about 17 GB), that's about 56 tok/s on an RTX 3090 (936 GB/s) and about 15 tok/s on a Strix Halo (about 256 GB/s). My single 3090 measured 38.7 tok/s on Qwen3.6 27B Q4_K_M at 450 W with llama.cpp, below the ceiling as expected.

Halving the bits roughly doubles the ceiling. On the Strix Halo, the same model at UD-Q8_K_XL (31.5 GB) and UD-Q4_K_XL (17.4 GB), both with the same DFlash2 draft model, ran at 18 and 27 tok/s on prose, and 43 and 56 tok/s on code. Those are above the ceiling because the draft lets each pass through the weights produce several tokens.

Prompt processing (prefill) is different. It does lots of math per weight, so it depends on compute: the bit count matters less there, and whether your tensor cores support the format matters more.

Step 1: Know your card

nvidia-smi --query-gpu=name,compute_cap,memory.total --format=csv

The compute capability tells you which formats get hardware acceleration:

A format your card can't compute natively may still run, but without the speedup. vLLM runs FP8 checkpoints on Turing and Ampere as weight-only W8A16 through its FP8 Marlin kernel: 8-bit weights, 16-bit math. You get the memory savings of 8 bits, but each token still reads twice the bytes of a 4-bit build. That's why I skipped FP8 on my 3090s.

Step 2: Budget for the KV cache first

The weights aren't the only thing in memory. Each token of context stores keys and values for every attention layer. For a standard transformer:

KV bytes per token = 2 × attention layers × KV heads × head dim × bytes per value

All four numbers are in the model's config.json. Llama 3 8B, for example, has 32 layers and 8 KV heads with a head dim of 128 (4,096 ÷ 32 heads). In FP16 that's 128 KiB per token, so a 32k-token context needs 4 GiB on top of the weights. A coding agent that wants 128k context needs 16 GiB, three times the size of the Q4 weights.

So work backwards:

1. Decide how much context you actually need.
2. Work out the KV cache for that context.
3. Leave 1-2 GB for activations and CUDA graphs. llama.cpp's --fit keeps a 1 GiB margin per GPU by default.
4. What's left is your weight budget. Pick the highest-quality quant that fits.

Hybrid models change this math a lot. Qwen3.5 and later repeat a block of three Gated DeltaNet layers, which keep a fixed-size state, and one full-attention layer. In Qwen3.8 27B only 16 of the 64 layers have a KV cache, with 4 KV heads of dim 256: 64 KiB per token in BF16. That's 16 GiB for the full 262k context, against 64 GiB if every layer were attention. It's why I can serve the full context on four 3090s using 68% of their memory.

The KV cache can be quantized too: --cache-type-k q8_0 --cache-type-v q8_0 in llama.cpp (a quantized V cache needs flash attention, which llama.cpp turns on automatically), or --kv-cache-dtype fp8 in vLLM. 8-bit KV is generally safe. Research methods hold up well at 4 bits and need special tricks at 2 (KIVI, KVQuant), but those aren't the same as llama.cpp's q4_0 cache type. If you go below 8 bits, test long-context retrieval and multi-step reasoning, which is where those papers saw the damage.

Step 3: Use a format your engine runs fast

The same bit count can run at very different speeds depending on whether the engine has a fast kernel for it on your card:

On a multi-GPU NVIDIA box, vLLM is usually much faster. It uses tensor parallelism, where every card works on every layer at once. llama.cpp's default --split-mode layer gives each card a slice of the layers and runs them as a pipeline, so on a single request only one card is busy at a time. (llama.cpp's row mode, and a newer experimental tensor mode, split each layer instead.) On one card, a Mac or a Strix Halo, llama.cpp is usually the simpler and faster choice.

Not every build with the same label is the same. My two "4-bit" builds of Qwen3.8 27B were 25.1 GB and 19.6 GB. The bigger one's quantization_config showed why: its attention layers and last eight MLP layers were 8-bit, and the rest used 4-bit with small groups of 32. The smaller one was plain 4-bit with groups of 128. The smaller one was 13% faster (77 vs 87 tok/s); the bigger one may be the more accurate. Read the config, not just the name.

Step 4: Pick a bit width

Here's what the research supports:

• 8-bit is close to lossless. Kurtić et al. ran over 500,000 evaluations on Llama 3.1 8B, 70B and 405B and found FP8 weights and activations "effectively lossless" at every size, and well-tuned INT8 within 1-3% (arXiv:2411.02355). Their 8-bit formats quantize the activations too, which is different from llama.cpp's weight-only Q8_0.
• 4-bit weight-only holds up well. The same study found carefully tuned 4-bit weight-only (W4A16) models "rivaling 8-bit quantization", recovering over 99% of baseline accuracy on average. The smallest models dropped a bit more.
• 4-bit for one user, 8-bit for throughput. The same paper found W4A16 the most cost-efficient for single-request, latency-bound serving, and W8A8 better for heavily batched serving. That matches my rig: 4-bit is fastest for one or two requests at a time.
• Calibrated methods do better than rounding. AWQ finds the roughly 1% of weight channels that see the largest activations and scales them up before quantizing so they lose less precision (arXiv:2306.00978). GPTQ quantizes the weights column by column and uses second-order information from calibration data to adjust the remaining weights and cancel out each step's error (arXiv:2210.17323).
• 4-bit is about the best accuracy per bit. Dettmers and Zettlemoyer ran over 35,000 zero-shot experiments on models from 19M to 176B parameters and found 4-bit "almost universally optimal" for accuracy against total model bits; at 3 bits the trend reversed (arXiv:2212.09720). Their models were 2022-era, and their quantization used no calibration data, so methods like GPTQ and today's IQ and dynamic quants may move that line lower.

That answers a common question: should you run a bigger model at lower bits, or a smaller one at higher bits? Down to about 4 bits, the bigger model usually wins at the same memory; Dettmers and Zettlemoyer's advice is to keep 4 bits and choose the model size that fits. Below 4 bits, losses grow quickly. There's also evidence that heavily trained models lose more from quantization than lightly trained ones (Ouyang et al., Kumar et al.), and current small open models are trained on enormous amounts of data. So be most careful going below 4 bits on small models.

My defaults:

MoE models change the speed math. Speed depends on the active parameters (about 3B for a 30B-A3B model), but memory still depends on the total. They're very fast for their size, and gain less from a higher power cap, as I found in my power-limit post.

Step 5: Test with the features you'll actually use

Speculative decoding can change the ranking, because the draft model has to work well with your target model, and that's not guaranteed. On my 3090s:

• AWQ + DFlash2 ran at 143 tok/s on prose. BF16 + DFlash2 crashed vLLM.
• The same AWQ build with --dtype half instead of bfloat16 ran at 47 tok/s with a 0% draft acceptance rate, and logged no error.
• MTP gave a smaller gain (132 tok/s) and halved total throughput with 8 requests at once.

The full story is in My Four RTX 3090s Were Slower Than a Strix Halo. The short version: the fastest quant on its own may not be fastest once you add a drafter, so test the whole setup.

To measure speed:

# llama.cpp: prompt processing and generation speed
llama-bench -m model-Q4_K_M.gguf -p 2048 -n 256
 
# vLLM: start the server, then benchmark it
vllm bench serve --model <model> --dataset-name random \
  --random-input-len 2048 --random-output-len 256 --num-prompts 50

Test the kinds of output you actually generate. On my rig, code and JSON ran at about twice the speed of prose with a draft model. If you use speculative decoding, check the acceptance rate in the log, not just the final tok/s.

To measure quality, compare each quant against the full-precision model on the same text. llama.cpp can report the KL divergence: how far the quantized model's next-token probabilities drift from the original's.

# Save the reference probabilities once, ideally from the BF16 model
llama-perplexity -m model-BF16.gguf -f test.txt --kl-divergence-base base.kld
# Then compare each quant against it
llama-perplexity -m model-Q4_K_M.gguf -f test.txt --kl-divergence-base base.kld --kl-divergence

Use a modest test file: the saved probabilities are large, about 37 GiB for Llama 3 on the full Wikitext-2 test set, according to llama.cpp's docs. Lower KL divergence is better.

KL divergence is more sensitive than perplexity. Dutta et al. found that perplexity can hide changes, because errors in opposite directions cancel out (arXiv:2407.09141). In their Llama 2 13B example, a 4-bit GPTQ model's perplexity barely moved (4.57 to 4.68), while its KL divergence was about 250× higher than the 8-bit version's.

Then try your 10 or 20 most common real prompts, including tool calls if you use agents. Broken tool calls are often the first sign that something's wrong. When I tried vLLM's variable draft size, the tool calls broke even though most of the text looked fine.

A worked example: my 4× RTX 3090 rig

1. Card: Ampere, compute cap 8.6. No FP8, so the choices are W4A16 (AWQ/GPTQ), INT8 W8A8 or BF16, in vLLM.
2. Memory: 96 GB across four cards, but vLLM only gets 68% so other jobs fit too. Hybrid attention keeps 262k context to 16 GiB.
3. Engine: vLLM with tensor parallelism across all four cards.
4. Bits: BF16 fits, but ran at 45 tok/s. The two 4-bit builds ran at 77 and 87.
5. Full setup: the smaller 4-bit build with DFlash2 and --dtype bfloat16 ran at 143 tok/s on prose and about 280 on code, and passed a 1,092-request soak with no errors.

The Strix Halo is the same exercise with a different answer. It has up to 128 GB of shared memory but only about 256 GB/s of bandwidth, so the smaller quant matters even more. It runs a GGUF Q4 build in gufo, because Q8 cost a third of its speed.

Caveats

• The quality claims come from published research, not my own evals. I checked my quants for correctness, tool calls and refusals, but didn't run a formal accuracy comparison between them.
• The papers mostly test Llama-family and older models. Newer architectures, like the hybrid Qwen models, may behave differently under quantization.
• Speeds vary by engine version. vLLM 0.28 ran my model at 10 tok/s; 0.32 ran it at 72. Retest after upgrades.
• Kernel support changes fast, especially for FP4 on Blackwell and on AMD. Check your engine's docs for what it currently supports on your card.