Hugging Face shipped native GGUF quants support in Transformers on September 22, 2026, and the number that made me stop scrolling was 40 tokens per second — Qwen3.5-4B at Q4_K_M on a MacBook Pro M2 Max, generated from a plain from_pretrained call. I run local models weekly through Ollama for client material that should not leave my machine, so a second runtime learning to read the same quantized files is not an abstract library note. It is a question about whether a one-person local stack still needs two separate tools. After reading the release and the docs, my answer is narrow: not this week, but the gap closed further than I expected.
In this article
- What Hugging Face actually shipped with GGUF quants on September 22
- Why GGUF quants now run packed instead of unpacked
- The three speed numbers worth copying into your notes
- Why my Ollama setup is not moving yet
- Where GGUF quants change the work for a solo operator
- The read I am keeping until the next release
What Hugging Face Actually Shipped on September 22
The release is narrower than the headline sounds, and that is the most useful thing about it. Transformers can now load a GGUF checkpoint straight from the Hub — AutoModelForCausalLM.from_pretrained("unsloth/Qwen3.5-4B-GGUF", gguf_file="Qwen3.5-4B-Q4_K_M.gguf") — and generate from it through the same API you would use for any unquantized model. GGUF is the single-file format built by the llama.cpp team, and it has been the default currency of local inference for two years.
Three constraints shape everything downstream. The fast path for GGUF quants is Metal only, which means Apple Silicon. The only architecture with full support today is Qwen3.5, dense and mixture-of-experts, with compatible Qwen3.8 checkpoints. Everything else drops to the legacy loader, which still works but unpacks the weights.
That legacy loader is not a small list — Llama, Mistral, Qwen2, Phi3, Bloom, Falcon, StableLM, GPT2 and Starcoder2 all load. They just load dequantized, which is the old behavior wearing a new coat. If you are reading this to find out whether the GGUF quants already sitting on your drive suddenly got faster, the answer is almost certainly no.
GGUF Quants Now Run Packed, Not Unpacked
The real change is that the weights stay compressed while the model is running. On Metal devices, ggml kernels fetched from the Hub run the matrix multiplications directly on the packed GGUF blocks, so a Q4_K_M file occupies roughly its on-disk size in memory rather than expanding back to a dense tensor. Attention gets the same treatment through a separate ggml-attn kernel — the same flash attention llama.cpp uses, for both decode and prefill.
Everywhere else, the library dequantizes at load time and hands back a regular dense model. That fallback is explicit in the Transformers GGUF documentation, and you can also request it deliberately with GgufConfig(dequantize=True) when you want a GGUF checkpoint converted into a normal dtype.
Understanding which of those two paths you are on is the whole skill here. Same file, same one-line call, two completely different memory profiles. A 27B model that fits comfortably as packed GGUF quants will not fit at all once it unpacks, and nothing in the calling code tells you which happened.
The Three Speed Numbers Worth Copying Into Your Notes
The GGUF quants benchmarks came from a MacBook Pro M2 Max with 32GB of unified memory, which is a fair proxy for the machine a solo consultant actually owns rather than a rented H100. Three checkpoints, all Q4_K_M:
- Qwen3.5-4B — around 40 tokens per second
- Qwen3.5-27B — around 15 tokens per second
- Qwen3.8-MoE-A6.2B — around 35 tokens per second
Hugging Face describes this as close to llama.cpp across all three. That phrasing is doing honest work: the claim is parity, not victory, and parity is the correct goal for a library that was never going to out-optimize the project whose kernels it borrowed.
Parity with llama.cpp is the entire point. A Python-native path to GGUF quants only matters if it costs you nothing in tokens per second — and for the first time, roughly, it does not.
Fifteen tokens per second on 27B GGUF quants is readable-speed drafting, not interactive chat. I have argued before that 512GB of unified memory changes which models a one-person shop can host, and these numbers sit in the same frame: the constraint on local work has been memory and patience, in that order, for a while now.
Why My Ollama Setup Is Not Moving Yet
My local runs still go through Ollama, and nothing in this release gives me a reason to change that this month. Ollama already reads the same files — you point a Modelfile at FROM /path/to/file.gguf and run ollama create. It does not quantize during import, so the GGUF quants you feed it are the ones you get — the same contract Transformers now offers.
What Ollama gives me is a model registry, a resident server, and the habit I described after my first month of running models on my own laptop — drafting on material I will not send to a hosted API. What Transformers would give me is a Python object. Those are different jobs, and the second one only becomes valuable when I am writing code around the model rather than talking to it.
There is also the architecture ceiling. The GGUF quants I keep locally are not Qwen3.5 checkpoints, so the packed path would not even engage — I would get the dequantized fallback, which is strictly worse than what I have. Swapping a working runtime for a slower one to gain an import statement is not a trade I can justify against billable hours.
Where GGUF Quants Change the Work for a Solo Operator
The serving story is where GGUF quants get interesting for anyone who writes glue code. transformers serve exposes an OpenAI-compatible endpoint and names a GGUF model as <repo>:<file>.gguf, because one repository holds several quantizations and the request has to say which one it wants. That naming detail is small and tells you the design intent for GGUF quants: switching between Q4_K_M and Q6_K is meant to be a routing decision, not a reinstall.
Three situations where I think this actually earns a slot in a solo stack:
- You already write Python around your models — evaluation harnesses, batch classification, retrieval pipelines — and maintaining a second runtime protocol is overhead you would rather delete.
- You need one API surface across hosted and local models, so a client project can fail over to a local checkpoint without rewriting the call site.
- You are testing several quantization levels of the same model and want the comparison in a notebook rather than across two tools.
For everything else, the honest read is that GGUF quants in Transformers replaces a small annoyance rather than a large one. The official announcement is candid about the rough edges too: padding and batching still have performance gaps, and architecture coverage is limited. Those are the words of a team shipping a foundation, not a finished product.
The Read I Am Keeping Until the Next Release
Two conditions would move this from interesting to installed, and both are visible from here. The first is architecture coverage — the moment the GGUF quants I actually run join Qwen3.5 on the packed path, the calculation flips. The second is CUDA, because MPS-only means the release currently serves the half of solo operators who bought Apple hardware and nobody else.
Until then, the checks I would run before swapping a local runtime are the same three I have used since the first time I pointed a local model at a folder full of client documents: do the GGUF quants actually engage the packed path on my hardware, does the quantization level hold up on my real prompts rather than a benchmark, and can I get back to the old setup in under ten minutes if it does not.
For me, this lands as a release worth bookmarking rather than acting on. GGUF quants running natively in Transformers removes a genuine seam in the local AI stack, and the speed numbers say the removal cost nothing. But a one-person shop upgrades tooling on evidence of a bottleneck, not on architectural elegance, and my bottleneck this month is still memory rather than runtime choice. I will check back when the architecture list grows.
Sources
- Transformers now runs llama.cpp quants — Hugging Face Blog, September 22, 2026
- GGUF — Transformers documentation — Hugging Face
- Import a model — Ollama documentation — Ollama
AI-assisted research and drafting. Reviewed and published by ToolMint.