Retrieved article excerpt
Open article · Retrieved 2026-09-23T12:21:57.520418+00:00
[Back to Articles](https://huggingface.co/blog)
# Transformers now runs llama.cpp quants
Published
September 22, 2026
[Update on GitHub](https://github.com/huggingface/blog/blob/main/transformers-llama-cpp-quants.md)
[Upvote
43](https://huggingface.co/login?next=%2Fblog%2Ftransformers-llama-cpp-quants)
- +37
[Marc Sun's avatar](https://huggingface.co/marcsun13)
[Marc Sun
marcsun13
Follow](https://huggingface.co/marcsun13)
[Arthur Zucker's avatar](https://huggingface.co/ArthurZ)
[Arthur Zucker
ArthurZ
Follow](https://huggingface.co/ArthurZ)
[Lysandre's avatar](https://huggingface.co/lysandre)
[Lysandre
lysandre
Follow](https://huggingface.co/lysandre)
**We're adding support for running GGUF models efficiently in transformers**, so you can use checkpoints sized for your laptop's memory through the familiar transformers APIs. Pick a GGUF from the Hub, load it with `from_pretrained`, and start generating on your own machine.
[
](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/transformers-llama-cpp-quants/transformers-gguf.mp4)
Running AI models on your laptop has become much easier, and [llama.cpp](https://github.com/ggml-org/llama.cpp) has been a big part of that. Its inference engine powers local AI tools such as Ollama, LM Studio, and Jan. Alongside projects like [MLX](https://github.com/ml-explore/mlx), it has helped make local inference a practical option for everyday use.
*A recent example of what local AI can feel like:*
> This is where we are right now. And i’m not gonna lie it feels pretty magical 🧙♀️
>
> Qwen3.6 27B running inside of Pi coding agent via Llama.cpp on the MacBook Pro
>
> For non-trivial tasks on the [@huggingface](https://x.com/huggingface?ref_src=twsrc%5Etfw) codebases, this feels very, very close to hitting the latest Opus in Claude… [pic.twitter.com/lsIxLoUneU](https://t.co/lsIxLoUneU)
>
> — Julien Chaumond (@julien\_c) [April 24, 2026](https://x.com/julien_c/status/2047647522173104145?ref_src=twsrc%5Etfw)
**GGUF**, developed by the llama.cpp team, is a widely used format for local inference. The team also shares quantized checkpoints under [ggml-org on the Hub](https://huggingface.co/ggml-org). Publishers such as [Unsloth](https://huggingface.co/unsloth), [LM Studio Community](https://huggingface.co/lmstudio-community), and [bartowski](https://huggingface.co/bartowski) also provide ready-to-use GGUF checkpoints in a range of quantizations, so users can pick the version that fits their machine. GGUF models have been downloaded millions of times.
We want to make it easier to run these models locally with transformers, too. Compatibility is only useful if the model is pleasant to run. To bring performance close to llama.cpp, we're reusing its underlying ggml kernels through the [`kernels`](https://huggingface.co/docs/kernels/index) library, and reducing overhead in `generate`. Our initial focus is local inference on Apple Silicon, starting with the Qwen3.5 architecture.
## What is the GGUF file format?
[GGUF](https://github.com/ggml-org/ggml/blob/master/docs/gguf.md) packages model weights and metadata, including tokenizer information and an optional chat template, in one file. It supports different quantization levels, letting you trade some precision for a smaller memory footprint. Variants such as `Q4_K_M` mix tensor precisions, using mostly 4-bit weights while keeping sensitive tensors at higher precision.
Here's how quantization changes the file size of [Unsloth's Qwen3.5-4B](https://huggingface.co/unsloth/Qwen3.5-4B-GGUF/tree/main):
| GGUF variant | File size | Tradeoff |
| --- | --- | --- |
| `BF16` | 8.42 GB | Unquantized reference |
| `Q6_K` | 3.53 GB | More precision than the smaller variants |
| `Q5_K_M` | 3.14 GB | A middle ground between size and precision |
| `Q4_K_M` | 2.74 GB | A practical starting point for local inference |
We suggest starting with `Q4_K_M`, then trying `Q5_K_M` or `Q6_K` if you have more memory available. More aggressive quantization can help larger models fit, but the quality tradeoff depends on the model and the task. Evaluate it on the work you actually want the model to do. The [Hub's GGUF documentation](https://huggingface.co/docs/hub/gguf#quantization-types) describes the available quantization types.
## Load GGUF with transformers
To get started, you need:
- **An Apple Silicon Mac**.
- **A PyTorch version supported by the published [ggml-quantization kernel builds](https://huggingface.co/kernels/ggml-org/ggml-quantization)**, usually the two latest PyTorch releases.
- **The latest version of transformers (main for now, until the next release) and a compatible version of `kernels`**.
```
pip install -U "git+https://github.com/huggingface/transformers.git" kernels
```
To load a GGUF model, pass its Hub `model_id` and filename as `gguf_file` to `from_pretrained`.
No extra configuration is needed: when the weights stay packed on Metal, transformers automatically loads the compatible ggml/Metal layer kernels and uses `ggml-org/ggml-attn` as the attention implementation. If that kernel cannot be fetched, the model falls back to `"sdpa"` with a warning, and you can always force `"sdpa"` by passing `attn_implementation="sdpa"` explicitly. See the [GGUF documentation](https://huggingface.co/docs/transformers/main/en/quantization/gguf) for more loading options.
```
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "unsloth/Qwen3.5-4B-GGUF"
filename = "Qwen3.5-4B-Q4_K_M.gguf"
tokenizer = AutoTokenizer.from_pretrained(model_id, gguf_file=filename)
model = AutoModelForCausalLM.from_pretrained(
model_id,
gguf_file=filename
)
```
That is the only GGUF-specific step. Everything after it is the standard transformers API:
```
messages = [{"role": "user", "content": "Explain why the sky is blue in a few sentences."}]
inputs = tokenizer.apply_chat_template(
messages,
tokenize=True,
add_generation_prompt=True,
return_dict=True,
return_tensors="pt",
).to(model.device)
with torch.inference_mode():
outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```
> Without a compatible quantization kernel, the loader falls back to dequantizing the model and uses more memory.
## Serve GGUF with your preferred interface
You can also use the same checkpoint with [`transformers serve`](https://huggingface.co/docs/transformers/main/en/serve-cli/serving), which exposes an OpenAI-compatible API:
```
pip install -U "transformers[serving] @ git+https://github.com/huggingface/transformers.git" kernels
transformers serve "unsloth/Qwen3.5-4B-GGUF:Qwen3.5-4B-Q4_K_M.gguf"
```
The model argument uses `<model_id>:<filename>.gguf`: before the colon is the Hub repository (`unsloth/Qwen3.5-4B-GGUF`), and after it is the file to load (`Qwen3.5-4B-Q4_K_M.gguf`). This selects a specific quantization from a repository that may contain several.
For models whose chat template supports thinking, add `--reasoning off` to skip it or `--reasoning on` to enable it. The default, `--reasoning auto`, follows the chat template’s default. See the [reasoning options](https://huggingface.co/docs/transformers/main/en/serve-cli/serving#enable-reasoning-on-the-server) for details.
You can connect a client such as [Jan](https://www.jan.ai/docs/desktop/remote-models/custom-endpoint) or [Pi](https://pi.dev) by adding a custom OpenAI-compatible provider with these settings:
| Setting | Value |
| --- | --- |
| Base URL | `http://localhost:8000/v1` |
| Model ID | `unsloth/Qwen3.5-4B-GGUF:Qwen3.5-4B-Q4_K_M.gguf` |
transformers runs the model on your Mac, while the client provides the conversation interface. The same endpoint can be used by other clients that support this API.
## Benchmarking against llama.cpp
Our reference for local inference performance is llama.cpp. The comparison below focuses on three GGUF checkpoints: a small dense model, a larger dense model, and a mixture-of-experts model.
The llama.cpp column comes from the [`llama-bench`](https://github.com/ggml-org/llama.cpp/tree/master/tools/llama-bench) tool (build `5f55650a7`, release b10200, Metal backend from ggml 0.18.0), run as `llama-bench -m <file> -p 0 -n 128 -r 3`, which reports `tg128`: the token-generation rate over 128 decoded tokens, averaged across three repetitions, with prompt processing excluded. The transformers column is `generate` producing the same 128 tokens from a 12-token prompt, best of three warmed runs, and it includes prefill.
Measured on a MacBook Pro M2 Max, 32 GB unified memory, macOS 26.6, PyTorch 2.12.1, kernels 0.17.0,
plugged in.
The benchmark script
```
import time
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id, filename = "unsloth/Qwen3.5-4B-GGUF", "Qwen3.5-4B-Q4_K_M.gguf"
model = AutoModelForCausalLM.from_pretrained(model_id, gguf_file=filename)
tokenizer = AutoTokenizer.from_pretrained(model_id, gguf_file=filename)
inputs = tokenizer("The capital of France is Paris. The capital of Germany is", return_tensors="pt")
inputs = inputs.to(model.device)
with torch.inference_mode():
model.generate(**inputs, max_new_tokens=8, min_new_tokens=8, do_sample=False) # warm up
torch.mps.synchronize()
for _ in range(3):
time.sleep(90) # let the machine cool: back-to-back runs decay by 10% or more
start = time.perf_counter()
model.generate(**inputs, max_new_tokens=128, min_new_tokens=128, do_sample=False)
torch.mps.synchronize()
print(f"{128 / (time.perf_counter() - start):.1f} tok/s")
```
For the other column:
```
llama-bench -hf unsloth/Qwen3.5-4B-GGUF:Q4_K_M -p 0 -n 128 -r 3
```
[GGUF generation throughput compared with llama.cpp](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/transformers-llama-cpp-quants/benchmark-comparison.svg)
Transformers is close to llama.cpp across all three checkpoints. The chart uses the same measurements described above; it does not imply identical benchmark conditions, since the Transformers measurement includes prefill while `llama-bench` reports decode-only throughput.
## transformers and llama.cpp
When [GGML and llama.cpp joined Hugging Face](https://huggingface.co/blog/ggml-joins-hf), we described their complementary roles: llama.cpp provides a foundation for local inference, while transformers provides a foundation for model definition. GGUF support brings those two closer together.
**llama.cpp remains our recommended engine when your priority is efficient local inference.** Its dedicated runtime, memory management, and broad hardware support are built around that goal. This integration gives developers a convenient way to work with the same GGUF checkpoints inside transformers:
- **Experiment with GGUF in Python and PyTorch.** Inspect intermediate activations with hooks, modify a model's forward pass, or prototype custom layers using familiar PyTorch tools.
- **Evaluate GGUF models.** Use your existing transformers evaluation workflows to measure the quality of quantized checkpoints.
- **Validate GGUF conversions.** For us as developers, loading the original checkpoint and its GGUF conversion in transformers makes it easier to check that the weights were converted correctly, accounting for quantization error.
- **Try new decoding ideas.** Use custom logits processors and stopping criteria with `generate`, or write your own generation loop in Python.
- **Fine-tune from a GGUF checkpoint.** Dequantize the weights and continue with a standard transformers training workflow.
For that last case, use `GgufConfig(dequantize=True)`:
```
import torch
from transformers import AutoModelForCausalLM, GgufConfig
model = AutoModelForCausalLM.from_pretrained(
"unsloth/Qwen3.5-4B-GGUF",
gguf_file="Qwen3.5-4B-Q4_K_M.gguf",
quantization_config=GgufConfig(dequantize=True),
dtype=