System Design · Issue 22

How NVMe Offload Breaks the 100K Token Memory Wall

When a single user's KV cache eats half an H100, spilling cold blocks to NVMe is how production systems keep serving long context.

10 October 2026 An H100 GPU's HBM filled by one user's KV cache and model weights

The constraint

One user eats the whole GPU

An H100 GPU's HBM filled by one user's KV cache and model weights

At 128K context length, a single user's KV cache on Llama 3.1 70B in BF16 consumes ~40 GB. An H100 has 80 GB of HBM. After model weights, there is room for maybe one or two concurrent users before you hit out-of-memory. The bottleneck is the KV cache. Model weights are fixed; the per-request attention state grows with every token of context.

\text{KV bytes} = 2 \times n_\text{layers} \times n_\text{kv\_heads} \times d_\text{head} \times \text{seq} \times b

Per-user KV cache size. At 128K tokens, BF16 (b=2), a 70B model with GQA lands near 40 GB.

The cache exists because attention recomputes from scratch without it. Every generated token needs to attend to every prior token's K and V vectors, so those vectors stay resident for the whole request. Double the context, double the cache. A million-token window would need hundreds of GB per user, far beyond any single GPU.

The mechanism

Tier the cache, keep the hot blocks on GPU

Three-tier KV cache hierarchy: HBM, DRAM, NVMe

The fix is a storage hierarchy. Hot blocks stay in GPU HBM at ~3.35 TB/s, warm blocks spill to CPU DRAM at ~63 GB/s, and cold blocks land on NVMe SSD at ~7 GB/s. A block moves back to HBM only when attention needs it.

Blocks move up the tiers on demand. When HBM fills, cold blocks evict back down to DRAM and NVMe.

LMCache plugs into vLLM's prefix cache and adds persistent NVMe and Redis backends, so cold blocks route straight to disk. Up to 99% of context in production is repeated system prompts, documents, or conversation history. Compute the prefix KV state once, store it on NVMe, and subsequent requests load from disk instead of recomputing.

How prefix caching skips the prefill compute

When a request arrives, vLLM checks its in-process prefix cache for a matching KV state. With LMCache, that check extends to NVMe and Redis backends. If the prefix KV state exists on disk, vLLM skips the prefill attention passes for those tokens entirely and copies the cached K and V tensors directly into HBM. A 128K-token system prompt that took 11 seconds to process on first load takes ~1.5 seconds on a cache hit, because the work becomes a sequential disk read instead of a matrix multiply cascade.

The numbers

From 6 users to 15, and 11 seconds to 1.5

Concurrency scaling from 6 to 15 users with NVMe offload

Combine NVMe offload with FP8 KV quantization, which halves per-user cache from ~40 GB to ~20 GB, and a single B200 with 192 GB of HBM scales from roughly 6 concurrent users to an estimated 12-15 at 128K context. The GPU that could barely serve one person now handles a small queue.

Without offload (11s TTFT for 128K prompt)

~6 users on B200

With NVMe offload (1.5s TTFT on cache hit)

~12-15 users on B200

The latency win is just as dramatic. VAST Data benchmarks using NVIDIA's ICMSP with all-NVMe storage saw Time-To-First-Token drop from ~11 seconds to ~1.5 seconds for a 128K-token system prompt, because prefill was replaced entirely by a cache hit loaded from disk. Standard vLLM --swap-space keeps the cache in CPU RAM and loses it on every server restart. NVMe-backed caches survive restarts and can be shared across replicas via Redis.

The tradeoff

It only works when prefixes repeat

Tradeoff between NVMe I/O latency and HBM access speed

NVMe I/O adds ~100 µs to 1 ms of latency per block, compared to <1 µs for HBM. For short-context, single-user workloads where GPU memory is not the bottleneck, offloading is pure overhead.

The gains are strictly tied to cache hit rate. A RAG pipeline reusing the same document chunks benefits; a one-shot summarization of a fresh document eats the I/O penalty with no compute savings.

Hardware-accelerated paths like NVIDIA's ICMSP use BlueField-4 DPUs to move cache traffic off the GPU's PCIe path, so streaming multiprocessors do not stall on I/O. Without that hardware, CPU staging of block transfers becomes the bottleneck.

Closing

The cache is the new database

Why it matters

KV cache management is becoming the defining constraint of LLM inference, and the GPU is no longer where the answer fits. Moving cold cache to NVMe treats the KV state like a persistent data store: computed once, reused across requests, shared across replicas, survived across restarts. The teams that win at long context will be the ones who engineer the cache hierarchy as carefully as the model itself.

The short version

A single user's KV cache at 128K context takes **~40 GB**, leaving no room for concurrency on an 80 GB H100.
Tiering cold KV blocks to NVMe via LMCache + vLLM lets a B200 scale from **~6 to 12-15 concurrent users**.
Prefix caching on NVMe cut TTFT from **11s to 1.5s** for a 128K-token prompt by replacing prefill with a disk read.
NVMe offload only pays off when prefixes repeat; unique-prompt workloads eat the I/O penalty with no compute savings.
Hardware like BlueField-4 DPUs keep the I/O off the GPU's path; without them, CPU staging becomes the bottleneck.