System Design · Issue 23

How NVIDIA ICMS Pools KV Cache Across Nodes

A new flash tier sits between local SSD and durable storage, turning KV cache into a pod-scale shared resource.

14 October 2026 Four memory tiers stacked vertically with a gap between local SSD and enterprise storage

The constraint

A gap in the memory hierarchy

Four memory tiers stacked vertically with a gap between local SSD and enterprise storage

KV cache grows linearly with context length and persists for the lifetime of a session. As inference shifts from one-shot prompts to long-running agentic workflows, the stored attention state that lets a model skip recomputing prior context outgrows every fast tier of memory.

KV cache that overflows G3 lands in G4, where millisecond latency stalls decode.

NVIDIA defines four memory tiers. G1 is GPU HBM: nanosecond access, but capacity-limited. G2 is host DRAM: a staging buffer, node-local. G3 is local SSD: warm KV, but tied to one node. G4 is shared enterprise storage: durable and scalable, but with millisecond latency that stalls the decode pipeline. The gap between G3 and G4 is where active KV cache that no longer fits in HBM or DRAM gets pushed to storage optimized for durability, not for serving tokens.

G3.5 tier — NVIDIA's name for a pod-level, Ethernet-attached, flash-backed KV cache pool that sits between local node SSD (G3) and durable enterprise storage (G4).

The design

A pod-level flash tier for shared context

GPU nodes connected to BlueField-4 flash arrays via Spectrum-X Ethernet

ICMS, also called CMX (Context Memory eXtension), inserts a new G3.5 tier between local SSD and durable storage. It is a pod-level, Ethernet-attached, flash-backed KV cache pool that holds petabytes of shared context per GPU pod, accessible via RDMA over Spectrum-X.

Dynamo + NIXL prestage KV blocks from CMX into G2/G1 ahead of decode.

NVIDIA Dynamo's KV Block Manager works with the NVIDIA Inference Transfer Library (NIXL) to move KV blocks from CMX into host DRAM or GPU HBM ahead of the decode phase, so the GPU never waits for storage. The BlueField-4 DPU runs the KV I/O plane underneath: it terminates NVMe-oF and RDMA protocols, handles cache placement, and offloads the storage data path from the host CPU.

The GPU never waits for storage.

Inside Dynamo's KV Block Manager

The KVBM has three layers. A model integration layer connects inference engines (TensorRT-LLM, vLLM, SGLang) without engine-specific glue. A memory management layer tracks where data lives and handles allocation, organization, and reuse with customizable offload strategies. A storage/transfer layer built on NIXL connects to CPU, SSD, file systems, and cloud platforms, and supports fast cross-machine transfers via a plugin system for third-party storage providers.

The numbers

What prestaging buys

GPU servers with high throughput fed by KV cache from storage tiers below

NVIDIA claims up to 5x higher tokens-per-second and 5x better power efficiency for the CMX platform versus traditional storage. The company has not published the baseline workload, model, context length, or concurrency level behind those numbers.

Speedups from prestaging and shared KV cache across four benchmarks.

Independent benchmarks are more specific. An IBM and Supermicro reference architecture measured 56x faster time-to-first-token at 130K tokens and 22x higher concurrent throughput (0.19 to 4.26 RPS), holding 18x over recompute even under 200 GB/s of noisy-neighbor I/O. On the storage side, Vast delivered 35 GB/s to a single H100 and WEKA 270 GB/s across 8, saturating GPUs rather than storage.

The tradeoff

Ephemeral by design

Ephemeral flash cache on the left, durable storage vault on the right, DPU in center

KV cache is derived data. It can always be recomputed with a forward pass. ICMS treats it as a distinct AI-native data class optimized for speed, scale, and power efficiency, not as an enterprise record that needs durability guarantees. If a CMX node fails, the cache is rebuilt from the source tokens. G4 durable storage stays for inactive state, query history, and logs.

Stage 1: POSIX/NFS offload (immediate HBM relief)
Stage 2: Optimized fabrics (fleet-scale pooling)
Stage 3: Full CMX (platform infrastructure)

BlueField-4-based platforms ship in H2 2026. Until then, WEKA outlines a staged path: start with backward-compatible KV offload over POSIX/NFS for immediate HBM pressure relief, move to optimized high-bandwidth fabrics for fleet-scale pooling on existing servers, then adopt full CMX as platform infrastructure. The Dynamo software stack already works with existing storage from Vast, WEKA, Pure, and IBM to deliver partial benefits today.

Wrap-up

Context as a pod-scale resource

Why it matters

ICMS reframes KV cache from a per-GPU memory constraint into a shared, pod-scale resource. The architectural bet is that inference context behaves like a cache, not like a database record: it is derived, recomputable, and valuable only while it is warm. If that bet holds, the bottleneck for large-scale agentic inference moves from GPU memory capacity to network fabric and flash throughput, which are easier to scale horizontally.

The short version

**G3.5** is a new flash tier between local SSD and durable storage, holding petabytes of shared KV cache per GPU pod.
BlueField-4 DPUs run the KV I/O plane; Dynamo + NIXL prestage blocks into HBM ahead of decode so GPUs never stall.
NVIDIA claims **5x TPS** and **5x power efficiency**; IBM measured **56x TTFT** and **22x throughput** on a related architecture.
The tier is **ephemeral by design**: KV cache is recomputable, so durability is traded for speed and scale.
BlueField-4 hardware ships in **H2 2026**; the Dynamo software stack delivers partial benefits on existing storage today.