GGUF vs GPTQ vs AWQ vs EXL2: LLM Model Formats Explained (2026)

First, separate 2 ideas: containers vs.quantization methods Most confusion comes from mixing 2 layers.A container defines how tensors are stored on disk.A quantization method defines how weights are squeezed into fewer bits.Containers: safetensors, GGUF, PyTorch pickle (.bin / .pt).

Methods: GPTQ, AWQ, bitsandbytes NF4, llama.cpp K-quants and I-quants.Both at once: EXL2 and EXL3 are a method plus a storage layout tied to one inference library.A quick memory rule of thumb Weight memory ≈ parameters × bits-per-weight ÷ 8.Model16-bit~4.5 bits per weight8B~16 GB~4.

5 GB70B~140 GB~39 GB This is arithmetic, not a vendor benchmark.It covers weights only.The KV cache and runtime overhead add more on top.1.Full precision: safetensors and PyTorch .bin Unquantized models usually ship as 16-bit weights, in either pytorchmodel.bin or model.safetensors.The older .

bin / .pt files use Python pickle.Loading a pickle file can execute arbitrary code, which makes untrusted checkpoints a security risk.Safetensors, created at Hugging Face, removes that risk.A file is a small JSON header plus raw tensor buffers, with nothing executable inside.

Tensors can be memory-mapped and loaded one at a time without reading the whole file.Safetensors is now listed as a PyTorch Foundation project.Important nuance: most GPTQ, AWQ, EXL2, EXL3, and MLX models are also stored in .safetensors files.

The quantization lives in the tensor contents and a config file, not in a new container.2.GGUF (llama.cpp) What it is GGUF is a binary format for running models with GGML and GGML-based executors such as llama.cpp.It was created by Georgi Gerganov, who also leads llama.cpp (Hugging Face docs).

It was introduced on August 21, 2023 as the replacement for the older GGML format.Why it replaced GGML The older GGML, GGMF, and GGJT files could not say which architecture a model belonged to.Adding a new hyperparameter broke every existing file.

GGUF switched to typed key-value metadata, so new fields can be added without breaking old files.Design goals The spec lists 5 goals: single-file deployment, extensibility, mmap compatibility, easy loading, and complete information inside the file.

Unlike tensor-only formats, GGUF can carry the tokenizer, special tokens, and a Jinja chat template alongside the weights.Reading GGUF quant names The suffix in a name like Q4KM.gguf tells you the scheme.Figures below come from the Hugging Face GGUF docs.

TypeHow it worksBits per weightQ40 / Q41 (legacy)4-bit round-to-nearest in 32-weight blocks; Q41 adds a block minimum4.5 / 5.0Q80 (legacy label)8-bit round-to-nearest in 32-weight blocks8.5Q2K16 blocks × 16 weights per super-block, 4-bit scales and mins2.625Q3K16 blocks × 16 weights, 6-bit scales3.

4375Q4K8 blocks × 32 weights, 6-bit scales and mins4.5Q5K8 blocks × 32 weights, 6-bit scales and mins5.5Q6K16 blocks × 16 weights, 8-bit scales6.5625IQ4XS256-weight super-blocks, uses an importance matrix4.25IQ3XXSSame I-quant family3.06IQ2XXSSame I-quant family2.06IQ1SSame I-quant family1.

56 *Derived by hand, not listed in the HF table: 32 weights plus a 16-bit scale (and a 16-bit minimum for Q41).Checking the Q4K math: A super-block holds 256 weights.256 × 4 bits = 1,024 bits.Add 8 blocks × 12 bits of scales and minimums (96 bits).

Add a 16-bit super-scale and 16-bit super-minimum (32 bits).Total: 1,152 ÷ 256 = 4.5 bits per weight.What S, M, L mean: These are mixes, not new types.For example, llama.cpp describes Q4KM as using Q6K for half of the attention.wv and feedforward.w2 tensors and Q4K elsewhere (Unsloth docs).

That is why a Q4KM file averages above 4.5 bits per weight.Newer types: The HF table also lists TQ10 and TQ20 for ternary weights, plus MXFP4, a 4-bit microscaling floating-point type.A labeling quirk: Hugging Face files Q80 under “legacy” types.

In practice, Q80 remains the standard near-lossless GGUF choice.Quality vs.size Hugging Face’s reference table for a Llama-2-7B-class model shows the trade-off: QuantPerplexityChange vs FP16SizeFP165.9565baseline13.0 GBQ805.9584+0.03%7.0 GBQ6K5.9642+0.13%5.5 GBQ5KM5.9796+0.39%4.8 GBQ4KM6.0565+1.

68%4.1 GB Illustrative only.These numbers come from a 2023-era 7B model; newer models can react differently.Importance matrix (imatrix) GGUF quantization can use calibration data.llama.cpp’s llama-imatrix computes an importance matrix from a text file.

llama-quantize --imatrix then uses it to improve quality.For 1-bit and 2-bit mixes, llama-quantize warns if no imatrix is supplied.Naming convention The spec defines filenames as base name, size label, fine-tune, version, encoding, type, and shard.Shards use a 5-digit counter such as 00003-of-00009.

Optional mmproj- and mtp- prefixes mark vision projectors and multi-token-prediction draft modules.Where GGUF runs GGUF is native to llama.cpp and its ecosystem.Hugging Face documents use with llama.cpp, LM Studio, GPT4All, and Ollama.vLLM support exists but is limited.

vLLM calls it highly experimental and under-optimized, and GGUF now needs the out-of-tree vllm-gguf-plugin.3.GPTQ GPTQ was written by Elias Frantar (IST Austria), Saleh Ashkboos and Torsten Hoefler (ETH Zurich), and Dan Alistarh (IST Austria & Neural Magic).

It first appeared on arXiv on October 31, 2022.It was published at ICLR 2023.How it works GPTQ is a one-shot, post-training weight quantization method.It uses approximate second-order (Hessian) information to decide how to round weights.

Rounding error in one column is compensated by adjusting weights not yet quantized.It needs a small calibration dataset but no retraining.Main results Quantized 175B-parameter models in about 4 GPU hours, down to 3 or 4 bits per weight (arXiv).Reported negligible accuracy loss at those bit widths.

End-to-end speedups over FP16 of about 3.25x on NVIDIA A100 and 4.5x on A6000 (HF paper page).Reading GPTQ names GPTQ repos often include GPTQ or tags like 4bit-128g in the name.Group size (128g): one scale per 128 weights.Smaller groups improve accuracy but add a little size.

Act-order (desc_act): quantizes columns in order of importance, usually improving accuracy.Tooling status in 2026 The original AutoGPTQ library is no longer maintained.GPTQModel states it has fully supplanted AutoGPTQ and AutoAWQ for Transformers, Optimum, and PEFT.

Its output runs in Transformers, vLLM, and SGLang.llm-compressor also implements GPTQ, but saves results in the compressed-tensors format.Hugging Face estimates GPTQ calibration for an 8B model at about 20 minutes on 1 A100.4.

AWQ AWQ (Activation-aware Weight Quantization) comes from Song Han’s group at MIT.It first appeared on arXiv on June 1, 2023.It won the MLSys 2024 Best Paper Award.Core idea Not all weights matter equally.Protecting roughly 1% of ‘salient’ weights sharply reduces quantization error.

The twist: AWQ finds those salient channels by looking at activation magnitudes, not the weights themselves.It does not store those channels at higher precision.Instead, it scales them up through a mathematically equivalent transformation, keeping a uniform, hardware-friendly format.

AWQ uses no backpropagation or reconstruction, so it is less likely to overfit its calibration set.Speed and cost The paper’s TinyChat runtime ran more than 3x faster than the Hugging Face FP16 implementation on desktop and mobile GPUs.

Hugging Face estimates AWQ calibration for an 8B model at about 10 minutes on 1 A100, roughly half of GPTQ’s estimate.Tooling status in 2026 AutoAWQ is officially deprecated.Its last tested setup was Torch 2.6.0 and Transformers 4.51.3.

vLLM adopted the functionality into llm-compressor, now the recommended AWQ workflow.MLX-LM also supports AWQ on Apple Silicon.5.EXL2 (ExLlamaV2) What it is EXL2 is the native format of ExLlamaV2, an inference library by turboderp for consumer GPUs.

It uses the same optimization method as GPTQ and supports 2, 3, 4, 5, 6, and 8-bit quantization.Wh