🌐 · English · 中文 · Español · 한국어 · · English Hub

A follow-up to yesterday’s generation-speed chart. This time it’s quantized versions of the same models. I didn’t run a single one locally. Download sizes are all measured today from Hugging Face live file listings, and quality figures are cited with explicit sources from each model card and public measurement blogs. File sizes across 104 builds for all models combined.

1. Format terminology — what’s inside each label

2. Quantization ladder per model (15 models · 104 builds)

Qwen3.8-27B

Dense 27.4B · multimodal · 256K context · sourceunsloth/Qwen3.8-27B-GGUF

Qwen3.8-Flash-Next (125B)

MoE 125B-A95B class · 262K · measured across summed shard files · sourceunsloth/Qwen3.8-Flash-Next-GGUF

Qwen3.6-35B-A3B

MoE 35B-A3B · 3B active · sourceunsloth/Qwen3.6-35B-A3B-GGUF

Gemma 4 26B-A4B

MoE 26B-A4B · 128 experts · QAT co-training · sourceunsloth/gemma-4-26B-A4B-it-GGUF

gpt-oss-20b

MoE 21B-A3.6B · weights natively MXFP4 · sourceunsloth/gpt-oss-20b-GGUF

Qwen3-Coder-30B-A3B

MoE 30B-A3B · coding-specialized · sourceunsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF

Mistral-Small-3.2-24B

Dense 24B · sourceunsloth/Mistral-Small-3.2-24B-Instruct-2506-GGUF

DeepSeek-R1-Distill-8B

Dense 8B distillation · sourceunsloth/DeepSeek-R1-Distill-Llama-8B-GGUF

Phi-4

Dense 14.6B · Sourceunsloth/phi-4-GGUF

Llama-3.1-8B

Dense 8B · sourcebartowski/Meta-Llama-3.1-8B-Instruct-GGUF

Ornith-1.5-35B-A3B

MoE 35B-A3B · new (26-08) · sourceornith-ai/Ornith-1.5-35B-A3B-GGUF

Nemotron-3.5-Lightning-30B-A3B

MoE 30B-A3B · built-in MTP head · sourceggml-org/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF

Muse-Glimmer-30B

Dense 30B · new from Meta (26-08) · sourcemeta-models/Muse-Glimmer-30B-GGUF

Laguna-XS-2.1

MoE 34B-class · Poolside new (26-06) · Sourceggml-org/Laguna-XS-2.1-GGUF

3. Where quality actually breaks

Quantization quality — where and how much it breaks: measured case of Qwen3.8-27B

One measurement published the BF16 comparison using 300 fixed tasks (pass@1) and a 240-task agent ladder (720 points max):

The most important line here isn’t the last one — it’s Q4_K_XL. The 17.6GB build, compared to BF1613 points higher on agent tasks. Statistically a tie, yet first on the agent ladder — quantization noise appears to actually diversify reasoning paths, though the benchmarkers summarized it as ‘no measurable degradation’. Back to the reasoning-budget point: the same model in BF16, reasoning off 61.3% → max 80.3%. That’s 7 times the entire quantization band (±3 points). Unless you’re on Q4 because GPU VRAM is too small, spending the saved VRAM on longer context and thinking wins.

MoE dislikes quantization even more — KL measurements across 80 variants on Gemma 4 26B-A4B

There are measurements of KL divergence against BF16 across 80 builds from 6 uploaders, using about 250K tokens (coding, chat, tool calling, science, non-Latin scripts, long context). Summary:

• A 4B-active MoE is already at KL 0.544 on Q8_0 — 3.3x that of a dense 31B peer (0.163). MoE is simply weak to quantization.
• Same Q4_K_M, yet ggml-org 2.126, lmstudio 2.116, bartowski 1.093 —The uploader matters more than the quantization name.
• unsloth UD monopolizes the Pareto front (one exception: bartowski Q6_K_L).
• The MoE-only format MXFP4_MOE (16.6GB, KL 0.963) loses to the same family’s UD-Q4_K_XL (17.0GB, KL 0.747).

And there’s a measurement showing calibration data decides real task performance: in the same recipe, switching only the calibration data from code/math to mixed improved KL but dropped GSM8K by another 9%P. It’s a public proof that ‘KL improvement = task quality improvement’ does not hold.

4. Recommended matrix for my machine

Size vs memory — a formula you memorize by measurement

The real RAM you need on top of the download size looks roughly like this: weights + (for a 16K context) 1~4GB KV cache (halved with a q8_0 cache) + 1~2GB overhead. Example: Qwen3.8-27B Q4_K_XL 17.6GB + 16K context ≈ 20GB → fits on a 24GB card. With M5 Max unified memory this term grows the longer you push the context, so even a 128GB machine can’t handle a 262K context on builds over 110GB (Flash-Next Q5 and up). The reason Flash-Next MLX held 59 tok/s in yesterday’s chart is the 4bit + SSD offload combo; anything above that bit depth is a luxury on this machine.

5. Sources and Cautions

Source

• All file sizes:unsloth/Qwen3.8-27B-GGUFList of 14 additional repo files (measured via API 2026-09-27, shards summed)
• Qwen3.8-27B pass@1 / agent ladder: vramcalculator.com quantization report (@superalesha’s fixed 300-task measurement is the original source)
• Qwen3.8-27B KL/top-1 table:AtomicChat/Qwen3.8-27B-GGUF(vs BF16 reference, 87 held-out chunks)
• 80 Gemma 4 KL variants: localbench.substack.com GGUF Quality Benchmark
• Gemma 4 QAT top-1: SC117/gemma-4-26B-A4B-it-qat-heretic-GGUFModel card
• HumanEval+/EvalPlus table:tooltd/Qwen3.8-27B-IQ4-XS-16GB-VRAM-GGUF
• Unleashed Q3 vs Q4 PPL/token speed:outsourc-e/Qwen3.8-27B-Unleashed-GGUF (4090, llama.cpp)

Caution: PPL/KL across different harnesses cannot be compared in absolute terms — only comparisons within the same table are valid (the measurers’ common disclaimer).

This pageLocal LLM Hub — What Runs on My Computeris part of. The per-hardware supported-model table is in the hub.