Artificial Intelligence 9 min read

llama.cpp Learned FP8. Your GPU Probably Can't Accelerate It

FP8 landed in llama.cpp's CUDA backend as a full kernel path. The fast path is gated to Hopper and Blackwell, a 27B won't fit 16GB at that size, and the KV-cache numbers are the useful part.

A dark bar chart. The top bar, labelled "27B · FP8 weights", runs to 19.56 GiB and overruns a red line marked "card capacity" by 3.57 GiB. The lower bar, labelled "RTX 4070 Ti SUPER", ends at 15.99 GiB. Below, three WikiText-2 perplexity figures: BF16 5.4527, Q8_0 5.4491, FP8 E4M3 5.4550.
Photo: Dallum Brown

Build b12284 landed on 11 November with a commit title that undersells it: "CUDA: Add OCP FP8 support." Two days later v1.0.0 shipped and made it official, naming FP8 KV caches among the headline features. Open Compute Project FP8 is the 8-bit float format the datacentre has converged on, and llama.cpp's CUDA backend now has a real implementation of it — not a conversion path, a kernel path.

Before anyone with a consumer GPU gets excited: this is a datacentre feature, and the numbers in the pull request are a single Blackwell workstation. The interesting part is not the speed. It is what the patch reveals about how quantisation choices work now.

What actually shipped

The merge is broader than "support FP8." From the commit list and the pull request:

  • Weight-only FP8 in the decode path — W8A16, meaning 8-bit weights against 16-bit activations, for the matrix-vector kernels that run when generating one token at a time.
  • FP8 KV cache — the attention cache stored in FP8, wired through the row-copy path rather than a block dequant. This one is a separate commit: the cache support, the flags and the tests landed in #28392 a day earlier, and v1.0.0 lists it as a headline feature. #28413 supplies the CUDA kernels that make it fast.
  • Fused weight_scale multiplication, so the per-tensor scaling does not cost a separate pass.
  • FP8 FlashAttention preparation in fattn-vec.cuh.
  • Batched FP8 GEMMs through cuBLASLt, with explicit workspace handling and a fallback path when the library cannot serve the shape.
  • Strided copies to and from FP8, which is the unglamorous work that decides whether the format composes with everything else.

Three things were deliberately left out, and the reasons are the most informative part of the pull request. Mixture-of-experts kernels are deferred because, in the author's words, "folks tend to quantize those layers to 4/2-bit" — there is no audience for FP8 experts yet. FP8-native tile paths during prefill are deferred, so prefill still goes through the slower route. And honouring accumulator and src1 precision specifications is an open TODO.

The pull request also discloses its own provenance: "AI usage disclosure: YES - paired with codex on this." Worth noting without commentary, given the week the industry has had.

What the numbers say, and what they don't

The benchmark table is one machine: an RTX PRO 4500 Blackwell with 32,165 MiB, a 27B model at 19.56 GiB.

variant size prefill (pp8196, ub 512) decode (tg128)
FP8 weights 19.56 GiB 3,582 t/s 40.23 t/s
fp8_q8 19.98 GiB 3,063 t/s 40.01 t/s

Two things to be careful about here, because the coverage will not be.

The 3,582 → 3,063 drop is not a KV-cache result. Those are two different files, not one file with the KV cache switched on and off. The digest that flagged this to us read it as "an FP8 KV cache cost 15% of prefill" — that inference is not in the data. The only clean KV comparison in the materials is the quality table, and there the result is flat:

WikiText-2 perplexity BF16 KV Q8_0 KV FP8 E4M3 KV
FP8 weights 5.4527 5.4491 5.4550
fp8_q8 5.4727 5.4641 5.4812

FP8 KV lands within 0.002 of BF16 on the first model and 0.009 on the second. That is the case for quantising the cache, and it says nothing about throughput either way. Anyone quoting a prefill penalty for FP8 KV is quoting a file comparison.

The depth number is a depth number. A separate run in the same commit message, at 32,000 tokens of context, shows prefill falling from 2,781 to 1,616 t/s. That is the cost of depth, not of the format, and it is the figure that will actually govern whether long-context work is practical on a given GPU.

One more inconsistency in the contributor's own materials: the pull request benchmarks a path named Qwen3.8-27B-NVFP4_fp8.gguf, while the commit-message table names Qwen3.6-27B-NVFP4_fp8.gguf, and both report the model as "qwen35 27B" with 27.32B parameters. A GGUF named NVFP4 that contains FP8 weights is a naming pile-up, not a scandal — but it is exactly the sort of thing to check before quoting a number across runs.

Where it runs, which is not where you are

FP8 arithmetic needs instruction support, and the commit is explicit about the gates:

  • The FP8 conversion intrinsic mmvf_f8x2_e4m3_to_bf162 requires CUDA toolkit 11.8 or newer.
  • A newer intrinsic was guarded repeatedly at 13.2 and 13.4, with the commit noting that "only cvt.rn.bf16x2.e4m3x2 is exposed from 13.2 onwards" — an earlier fix had assumed more availability than the toolkit provides.
  • The commit enables "Hopper and other BW SKUs" — datacentre Blackwell.

Consumer Ada, which is what an RTX 4070 Ti SUPER is, is not named anywhere in the change, the pull request or the discussion. That does not prove Ada cannot run the path; it means the code was gated for Hopper and Blackwell SKUs and nobody has published an Ada measurement. If you own a 40-series GPU and you assume "FP8 landed, therefore I have FP8," you are assuming.

The silence is not the same as incapacity, and this is the detail worth carrying away. Ada's fourth-generation tensor cores include an FP8 engine — NVIDIA's own Ada whitepaper and the 40-series launch materials both say so — and NVIDIA has stated that Hopper and Ada require different FP8 kernels. That is why a patch can target one generation and not the other: the arithmetic exists on both, the instruction set does not match. What a 40-series card is missing is the kernel path, not the silicon.

Which is the economic version of the same fact. You are not buying compatibility with a format; you are buying the generation whose kernels somebody has written. For anyone budgeting hardware against a model rather than the other way round, that is the line item that matters — and it moves every time a backend merges a path.

And the size question settles it independently. A 27B model at FP8 is 19.56 GiB of weights. On a 16 GB GPU there is no context window left — there is no model. The practical ceiling on that hardware stays 4-bit, where a 27B Q4_K_M is already brushing the limit once a real context is in play.

The actual shift: formats, not file sizes

llama.cpp's quantisation surface has fragmented. Q4_K_M, IQ4_XS, MXFP4, NVFP4 and now OCP FP8 all coexist, and each has a different support matrix per backend. CUDA gets the FP8 kernels today; the Vulkan, SYCL and ROCm backends each move on their own schedule, and the MoE and prefill paths are explicitly still missing.

That changes the selection rule. "Smallest file that fits" was a reasonable heuristic when quantisation only traded quality against size on one code path. Now the thing that decides your tokens per second is which formats your backend actually accelerates, and two files of the same size on the same GPU can differ by multiples depending on whether the fast path exists for that type.

It also means the answer is per-machine. Our own measurement discipline has been to distrust format-matrix blog posts that do not name the card, the build and the load mode — and this is the release that makes that discipline mandatory rather than pedantic.

How to get your own number

If you want to know whether FP8 buys you anything, the only credible route is to measure, because nobody has published a consumer Ada result:

# CUDA build of llama.cpp b12284 or newer
llama-bench -m <model>.gguf -p 8196 -ub 512 -d 0,32000 -n 128 --load-mode dio -fa on

# and the same run with the FP8 cache (v1.0.0 or newer)
llama-bench -m <model>.gguf -p 8196 -ub 512 -d 0,32000 -n 128 --load-mode dio -fa on \
  -ctk f8_e4m3 -ctv f8_e4m3

Compare like with like: same model family, same quantisation level, same context depth, same micro-batch. Then compare the FP8 KV cache against BF16 at the same weights — on v1.0.0 that is -ctk f8_e4m3 -ctv f8_e4m3 against the default, which is the comparison the release makes possible for the first time — that is the one variable the upstream table isolates cleanly, and the one most likely to matter on a memory-constrained GPU.

What we have not done is run this on our own 16 GB workstation, and we are not going to quote a Blackwell figure as if it were ours. When we do run it, the numbers will go in a post with the build hash and the load mode attached.

What this release is actually for

FP8 on CUDA matters most to the people serving models they did not quantise themselves — datacentre weights increasingly ship in FP8, and until this week llama.cpp could not use them natively on the fastest path. That is a real gap closing, for someone else's hardware.

For a self-hoster on consumer silicon, the honest summary is narrower and still useful: the format landscape got bigger, the fast paths are now unevenly distributed, and the only number that matters is the one you can reproduce. The release notes tell you what landed. They cannot tell you what your GPU will do with it.

If you do run the benchmark, the number worth posting is the KV-cache comparison at matched weights. That is where the upstream evidence is thinnest.

Researched and drafted by Claw. Verified, edited and published by Dallum.


Sources

D

Dallum Brown

Writer and curator exploring the impact of technology on everyday life.

View All Articles

Subscribe to
The Brief

Our curated selection of tech news and other discoveries, delivered every month.

No spam. Unsubscribe anytime.

Comments (0)

Please sign in to leave a comment.

No comments yet. Be the first to share your thoughts!

Further Reading