Models That Fit Your Hardware, Honestly: Decoding the Filename

We loaded a 9B, a 12B, and a 27B onto one 16 GB GPU overnight. All three weight dumps landed inside a 1.3 GB band. Parameter count turned out to be the least interesting word in the filename.

The overnight run

The claim we decided to test is one you'll have seen in some form: that in the 8–24 GB VRAM band there is effectively one model that gives you reliability. The framing is usually "which model should I run" — singular, definitive, a verdict.

We can't check that on our GPU by arguing. So we measured. On a 4070 Ti SUPER (16,376 MiB total) under LocalAI on the llama.cpp backend, overnight bench, three models that between them cover the entire "which model should I run" debate:

Decode is sustained short-prompt output (three ~350-chunk runs, no meaningful prefill). 8K prefill is the time to swallow a ~9,165-token prompt before the first token comes out. Raw receipts (the JSON, with timestamps and chunk counts) sit next to this post's working files, and we're publishing them with it — inspectable or it didn't happen.

Three models, three sizes spanning a 3× parameter range, three different quantization families. All three loaded and served reliably. All three weight dumps landed between 10,301 and 11,567 MiB. That is the finding: on a 16 GB GPU, parameter count is negotiable; the envelope isn't. The runtime will happily fill roughly the same 11 GB whether you started from 9B or 27B, because you pick the quant to fit the GPU, and the GPU picks the size of the bill.

What actually differs is everything else in the table. The 9B Q4 decodes at 115 tok/s. The 27B that "fits the same GPU" decodes at 50 — a 2.3× tax you pay for parameters you compressed into existence, plus a 5.9-second wait for every 8K prompt versus under 2. "Fits" was never the interesting question. Which point on this trade curve will you sit with daily is.

Sources: Our own benchmark run, 2026-10-20 20:00–20:02 NZDT (quant-receipts-20261020.json, RTX 4070 Ti SUPER, LocalAI/llama.cpp). Derived prefill rates (9,165 tokens ÷ TTFT): ≈4,600 tok/s Qwen, ≈3,850 Gemma, ≈1,550 Swift.

The filename is a spec sheet — if you read it

Every number in that table was stamped into a filename before we downloaded anything. The jargon isn't decoration; it's a compressed engineering decision. Field guide, four terms that cover most of what lands on your disk:

Q4_K_M vs IQ2_XS (quant families). Both are GGUF quantizations — weights stored at fewer bits per parameter. K-quants (Q4_K_M) chop weights into blocks, each with its own scale, and let the most important layers go slightly wider (that's the M). They're honest arithmetic: lossy, but the loss is spread deliberately. I-quants (IQ2_XS) go further — a codebook approach tuned per-tensor with importance matrices, buying bits so aggressively that a 27B model lands in the same 11 GB a 9B Q4 occupies. IQ2_XS doesn't make the 27B lossless; it makes the distribution of loss more survivable at a size where ordinary 2-bit quantization falls apart. Our table is the trade in one line: more parameters, approximately half the tokens/second, and the worst 8K prefill of the three.

QAT (quantization-aware training). Gemma 4 12B's filename says it-qat-q4_0 — instruction-tuned, quantization-aware-trained, plain old q4_0. QAT means the model was trained with the quantization error in the loop, so it's good at being small on purpose rather than good at being full-size and then squashed. The q4_0 format is one of the blander, oldest k-quants; the QAT is why the model card is interesting anyway. The same week's sleeper result from an unrelated repo: the humanizer 12B rewrite model's sensitivity-aware bit allocation plus QAT and distillation pushes a 3.9 GB 2-bit GGUF to 87% top-1 agreement with its bf16 original, versus 70% for a standard quant of the same size — seventeen points recovered at the same bytes. Quant quality is a recipe, not a size class. That's why "Q4" alone is no longer a fair comparison between two files.

A×B (active parameters). Qwen3.6-35B-A3B — 35B parameters on disk, 3B active per token. Mixture-of-experts: the weights are a library, and each token only reads a few shelves. Total size sets the download and VRAM bill; the active number sets decode speed and per-token compute. Which is why a well-quantized MoE can be simultaneously the biggest file on the GPU and one of the cheapest to run. It's also why "816B open-weight model" headlines and "runs on your workstation" threads can both be technically true and about different things.

E×B (effective parameters). Same trick, different denominator — and Google's Gemma 4 family stamps it right in the filenames: E2B is 2.3B effective, 5.1B total; E4B is 4.5B effective, 8B total. Here nothing is dormant and no router is choosing sides: the gap is embedding tables. Gemma 4's per-layer embeddings (PLE) spread each layer's input-embedding lookups into large tables that are read, not compute-multiplied — so the transformer body doing the mathematical work is 2.3B, while 5.1B of bytes still occupy your disk and your VRAM. The "effective" count describes compute per token, not memory. Which is exactly the distinction this whole post keeps hammering: parameter marketing describes what runs, your envelope only ever pays for what's stored. Three letters on a filename, and you need to know which three letters to know what they're not telling you.

One rule for all of them. Whatever the letters — A, E, total — every one of them describes parameters. None of them describes bytes. A quantized model's real size is on the file listing, not in the name.

MTP (multi-token prediction). A second output head on the model that drafts the next couple of tokens for the main model to verify — speculative decoding, built in, no separate draft model. llama.cpp turns it on with --spec-type draft-mtp --spec-draft-n-max 2 and reports roughly double tok/s for common cases; the community's honest caveat is a slight precision cost at some quants. Worth knowing before you decide your GPU "can't go faster." One more wrinkle, caught by the runtime's own diary while this post was in review. Our boot log says it plainly: [mtp] embedded MTP head detected; enabling draft-mtp speculative decoding name=qwen3.8-9b-q4 — LocalAI reads the filename for you and switches the drafter on at load. A re-measurement today with the identical method returns 119.2 tok/s against the overnight 115.3: same build, same model, same numbers within noise. So the 9B row in our table is an MTP number, not a number without it. The 27B and the 12B have no detectable heads; theirs are honest single-token baselines. Which adds a fifth thing to decode: the runtime's spec sheet is also a filename, and you should read it.

Sources:

• Julien Chaumond (Hugging Face), X post on llama.cpp MTP flags and ~2× decode (Qwen3.6-27B-MTP / 35B-A3B examples);
• unsloth MTP docs (llama.cpp commands, Gemma 4 / Qwen3.6 recipes);
• ggml-org/llama.cpp issue #27469 (MTP spec-decode internals, draft-mtp, EAGLE/n-gram unification);
• google/gemma-4-E2B HF model card ("E" = effective; E2B 2.3B effective / 5.1B total, per-layer embeddings);
• jialinyyzz/humanizer HF model card (87% vs 70% top-1 agreement at 3.9 GB 2-bit, QAT + sensitivity-aware allocation; fetched via HF API by our scout, 2026-10-20).

The quiet one: context is a reservation

Here's the mechanism that bit us before we thought to measure it. Weights are a one-time cost. The KV cache — the memory holding what your model remembers about the prompt it's processing — is charged per token, every token, and it scales linearly. Set a 262K context window and the runtime reserves room for 262K tokens whether or not you ever send one.

Our receipts show the cheap end of this: an 8K prompt costs between 6 and 24 MiB of VRAM on top of weights (the 27B reading moved by −277 MiB, which tells you anything under ~300 MiB is allocator noise, a useful calibration). At 8K, context is a time tax — 0.1 s first token on a short prompt versus 2–6 s after chewing 9K — not a memory crisis.

At full window it is both. When we pinned Qwen3.8-9B to its advertised 262K context in LM Studio, the allocation OOMed the GPU. The production config (LocalAI) runs 80K, and that's a deliberate number: 80K fits inside 16,376 MiB with weights, compute buffers, and our 4.2–5.2 GB of measured headroom accounted for. And nothing we run locally needs heavy token work anyway — 80K is the sweet spot for the job, not for the model. The model didn't change; the reservation did, and we sized it to the work, not to the spec sheet. If your GPU "runs" a model but dies the moment someone pastes a long log file, this is the mechanism — your context setting isn't a capability, it's a bill.

What this says about "the one model"

Back to the claim. On our GPU — 16 GB, the modal self-hoster VRAM, the boundary case — "only one reliable model" is false as written. We ran three reliably, 4–5 GB apart from the ceiling each, and none of them is the one from the threads.

But the claim is pointing at something real with the wrong shape. The honest version: there's one budget — roughly 11 GB of weights and one context reservation you have to choose consciously — and the filename tells you what you're spending it on. Quantization converts "how many parameters" from a hardware question into a quality-per-byte question. Which is also exactly the post Google's new originality criteria reward and nobody can remix: these numbers are a property of this GPU, this backend, this night. Your 4070 Ti SUPER will agree. Everyone else's rig will disagree, usefully.

What's in your control

• The actual bytes on your GPU: load it, weigh it, keep the JSON. One evening of benchmarks beats a year of benchmark threads.
• The context reservation: set it to what your work needs, not to what the config template advertised.
• Which trade-curve point you sit with: 115 tok/s of 9B or 50 tok/s of 27B is a workflow decision, not a leaderboard one.

What isn't

• Model architecture, release cadence, and whether "A3B" or "E2B" means what last year's post said it meant. The spec sheets outrun the vocab; re-check model cards before trusting old rules of thumb.
• Whether the frontier tier ever fits your envelope. It won't. That's the point of the envelope.

What we're testing next

• The MTP comparison we planned has inverted: the 9B was already drafting. Now we need the control run — drafter explicitly off, to find the true single-token floor, then on at --spec-draft-n-max 2 and 6, with a quality check at each step.
• A KV-curve sweep: 8K → 32K → 64K → 80K on the 9B, actual MiB per step, so "context eats VRAM" stops being a slogan and becomes a line chart.
• The humanizer 12B as a drop-in editorial pass on our own drafts (same base model as the Gemma 4 12B in the table, Apache-2.0, llama.cpp GGUF). Swap weights, keep the pipeline — and score the output. Update: The humanizer pass ran, and it surprised us twice. Every one of our 58 numbers, dates and names survived the rewrite — and the prose rhythm didn't: mean sentence length fell from 19 words to 6, and every em-dash vanished. The tool is a slop-remover, and it found the slop-free text in front of it suspicious.

Disclosure: the benchmark run, the receipts JSON, and the model-card fetches behind this post were executed overnight by CloudHerder's agent on our own hardware; the numbers were re-checked against the raw JSON before publication; the post was edited and signed by a human.

Full source list

1. CloudHerder benchmark receipts — quant-receipts-20261020.json + summary table quantization-numbers-oct26.md (run 2026-10-20 20:00–20:02 NZDT, RTX 4070 Ti SUPER 16,376 MiB, LocalAI/llama.cpp; first-party).
2. Julien Chaumond (@julien_c), X — llama.cpp MTP one-liners (--spec-type draft-mtp --spec-draft-n-max), ~2× decode claim, Qwen3.6-27B-MTP and Qwen3.6-35B-A3B examples; comment thread notes precision loss at Q6_K_XL 27B.
3. unsloth docs — "How to Run MTP Models" (llama-cli/llama-server commands for Gemma 4 and Qwen3.6 MTP GGUFs, incl. gemma-4-12B-it-qat-GGUF).
4. ggml-org/llama.cpp GitHub issue #27469 — MTP/draft speculative-decode internals, dual-context architecture, draft-mtp unification with EAGLE/n-gram.
5. jialinyyzz/humanizer — Hugging Face model card (87% vs 70% bf16 top-1 agreement at 3.9 GB 2-bit; QAT + sensitivity-aware bit allocation; Apache-2.0; gemma-4-12B base). Verified via HF API by CloudHerder scout 2026-10-20; re-verify at publish.
6. Hugging Face model card — google/gemma-4-E2B (+ per-layer-embeddings explainer, Maarten Grootendorst's "A Visual Guide to Gemma 4"): E2B = 2.3B effective / 5.1B total, E4B = 4.5B / 8B; MoE variant 26B-A4B: 8 active experts + 1 shared of 128.
7. CloudHerder — "Open Weights, New Bouncers" (Oct 2026), for the A×B example (meta-models/Muse-Spark-Open-816B-A42B: 816B total / 42B active per token, HF API receipt).
8. CloudHerder config history — Qwen3.8-9B Q4_K_M with the full 262K window OOMs the 16 GB card (observed by Dallum in LM Studio, confirmed 2026-10-21); the production LocalAI config pins context_size: 80000 (first-party receipt: qwen3.8-9b-q4.yaml).