Paper · Issue 23

How FP8 KV Cache Halves LLM Memory Pressure

FP8 halves KV cache memory, but two hidden hardware bugs were silently undermining accuracy and speed. vLLM found and fixed both.

14 October 2026 GPU HBM dominated by KV cache with a vLLM server

The memory wall

Half the Bytes, Twice the Trouble

GPU HBM dominated by KV cache with a vLLM server

At 128k+ tokens, the KV cache dominates GPU HBM, and every decode step reads a large fraction of it back. Storing K and V tensors in FP8 (e4m3) instead of BF16 cuts bytes per element from 2 to 1, which should mean lower decode latency, higher concurrency, or longer contexts.

vLLM's --kv-cache-dtype fp8 quantizes the KV cache and runs the full attention math in FP8. The vLLM team, with AWS and Red Hat AI, stress-tested decoder-only and MoE models on Hopper (H100) and Blackwell (B200).

Two categories of latent bugs had been silently undermining the feature. Both trace back to one question: does FP8 hardware actually accumulate in FP32 the way the documentation promises?

The mechanism

Accumulation That Wasn't

Hopper Tensor Core losing FP8 accumulation precision on long contexts

Hopper's FP8 Tensor Cores are documented to accumulate in FP32, but in practice the intermediate accumulation loses precision when the contraction dimension grows past ~100K. In long-context decode, that dimension equals the context length, so a 128k sequence sits squarely in the failure regime.

The needle disappears

On a 128k needle-in-a-haystack task, FP8 accuracy on the Flash Attention 3 backend dropped from 91% (BF16 baseline) to 13%.

91% to 13%. The model couldn't find a needle in its own 128k-token haystack.

Internal accum (13% NIAH at 128k)

Two-level FP32 (89% NIAH at 128k)

The fix, inspired by SageAttention2, writes partial sums to an actual FP32 register instead of relying on the Tensor Core's internal accumulator. This two-level accumulation restored NIAH accuracy from 13% to 89%, within 2 points of BF16. The cost is register pressure that slows prefill, mitigated by optimized tiling for head_dim 64 and 128.

Why the contraction dimension matters

In the ScoreV matmul, the contraction dimension equals the sequence length. At 128k tokens, the hardware accumulator processes 128k partial products. The vLLM team identified this as the same precision loss documented in the DeepSeek-V3 Technical Report (Fig. 7b), where large contraction dimensions cause intermediate FP32 accumulation to silently degrade.

The numbers

Where FP8 Pays Off

Llama vs gpt-oss-20b FP8 speedup comparison

On Llama-3.1-8B after the fixes, FP8 decode latency dropped to 54% of the BF16 slope, with break-even at just 7,010 tokens. The intercept gap was only 0.14 ms, meaning quantization overhead at short context is negligible. For head_dim 64 and 128, FP8 yields speedups on both prefill and decode.

The sliding-window trap

Hybrid-attention models like gpt-oss-20b (window size 128) showed a different pattern. FP8 decode latency was 96% of BF16, almost no speedup despite halving memory, with break-even exceeding 700k tokens. Sliding-window layers have bounded KV-cache footprints, so quantization overhead isn't amortized.

\text{Break-even (tokens)}: \quad 741{,}565 \;\longrightarrow\; 22{,}109 \;\longrightarrow\; 7{,}659

gpt-oss-20b: before fix, after fix, after skipping sliding-window layers

The fix is --kv-cache-dtype-skip-layers sliding_window, which keeps bounded-context layers in BF16 while quantizing only global-attention layers. With this flag, gpt-oss-20b break-even dropped to 7,659 tokens and the ITL slope improved to 71% of BF16.

When to use it

The Flags That Actually Help

vLLM serve commands with FP8 KV cache flags on a terminal

For global-attention models with head_dim 64 or 128, --kv-cache-dtype fp8 is safe to turn on. The accuracy cost is 0.3 to 0.7 points on long-context retrieval, within noise. vLLM's own NIAH after fixes measured 89% against a 91% BF16 baseline.

Skip what doesn't help

For hybrid models with sliding-window layers, add --kv-cache-dtype-skip-layers sliding_window to keep bounded-context layers at native precision. For head_dim 256, FP8 reduces decode latency but prefill can still be slightly slower than BF16.

vllm serve meta-llama/Llama-3.1-8B --kv-cache-dtype fp8
vllm serve gpt-oss-20b --kv-cache-dtype fp8 \
  --kv-cache-dtype-skip-layers sliding_window

Quick-start: FP8 KV cache for global and hybrid models

On A100, which lacks FP8 Tensor Cores, you get memory savings but no throughput gain. The cache dequantizes to BF16 before attention runs. Default fp8 uses scales of 1.0 with no calibration; for the highest accuracy, vLLM recommends llm-compressor dataset calibration for per-attention-head quantization.

Closing

FP8 KV Cache, Done Right

Why it matters

FP8 KV cache halves the memory cost of long-context serving, but only after the vLLM team traced two hardware-level bugs that the Tensor Core documentation never warned about. The fixes make --kv-cache-dtype fp8 safe to ship on Hopper and Blackwell for most decoder and MoE models, with a 2-point accuracy gap and decode latency cut nearly in half. For anyone running inference at 128k+ context, this is the cheapest memory upgrade available, requiring one flag and no new hardware.

The short version

FP8 (e4m3) stores KV cache at **1 byte per element** instead of 2 for BF16, halving memory pressure.
Hopper FP8 accumulation loses precision on long contexts; two-level FP32 accumulation fixes it, restoring NIAH from **13% to 89%**.
Best case: Llama-3.1-8B decode latency drops to **54% of BF16**, with break-even at 7,010 tokens.
Sliding-window layers don't benefit from quantization; skip them with `--kv-cache-dtype-skip-layers sliding_window`.
On A100 without FP8 Tensor Cores, memory savings only; no throughput gain.