Deep Dive · Issue 24

Where Continuous Batching Hits a Throughput Cliff

GPU utilization hits 100% and throughput goes backward. The ceiling is memory.

17 October 2026 GPU serving a static batch with idle padded slots

The setup

Why GPUs Sat Idle on Padding

GPU serving a static batch with idle padded slots

Traditional LLM serving uses static batching: you group requests and process them together until the longest sequence finishes generating. Short requests that complete early still hold their GPU slot, padded with wasted compute.

Continuous batching, introduced in the Orca system, shifts scheduling from the request level to the token level. After every decoding iteration, the scheduler frees memory from finished sequences and admits waiting requests into the vacated slots.

Static batching wastes slots on finished sequences. Continuous batching refills them immediately.

The gains were substantial. Anyscale measured 8x throughput over naive HuggingFace serving, and vLLM reported 24x over HuggingFace Transformers and 3.5x over Text Generation Inference. The Orca system reached up to 36.9x over FasterTransformer.

The mechanism

Schedule After Every Token

Iteration-level scheduler freeing and admitting sequences after each token

The core loop runs one decoding step for the whole batch, then pauses. It checks for finished sequences, frees their KV cache, and admits waiting requests before the next step. Because this happens after every token, a finished request is replaced within milliseconds.

The loop repeats after every token. Finished sequences are evicted and new ones admitted within a single iteration.

PagedAttention makes this work by mapping logical KV cache blocks to non-contiguous physical memory through a block table, with a default block size of 16 tokens. Memory waste drops from 60-80% under pre-allocation to under 4%. Copy-on-write sharing for parallel sampling cuts memory use by up to 55% and boosts throughput 2.2x.

How selective batching handles variable-length sequences

Matrix multiplications need uniform shapes, but batched requests have different token lengths. Orca's selective batching splits operations by type. Non-attention layers like Linear and GeLU are batched normally across all tokens, since they operate per-token without cross-sequence interaction. Attention layers, which need to know sequence boundaries, run through fused CUDA kernels like PagedAttention and FlashAttention that handle variable-length sequences natively.

Neither piece works alone: scheduling without paged memory would hit the cliff sooner, and paged memory without scheduling would leave freed slots sitting idle.

The cliff

When VRAM Fills Up, Throughput Goes Backward

KV cache exhaustion causing eviction cascade on GPU

The practical limit on batch size is VRAM, not compute. Every active sequence holds a growing KV cache in GPU memory, and once that memory is exhausted, the scheduler must preempt running sequences to make room. Preemption means either swapping KV blocks to CPU over PCIe or discarding them entirely to recompute later.

n_{\text{seq}} = \frac{V_{\text{GPU}} - V_{\text{weights}}}{\text{KV per seq}} = \frac{40 - 26}{\approx 2} \approx 7

40GB A100, 13B model, 2048-token average sequence length. Roughly 7 concurrent sequences before preemptions begin.

Under sustained long-context load, this becomes an eviction cascade. Sequences are evicted, re-admitted, and evicted again in a loop. The signature is unmistakable: GPU utilization reads 100%, but useful throughput flatlines or declines, preemption counters climb continuously, and P99 latency spikes from ~200ms to several seconds.

The GPU reads 100% utilization while generating fewer useful tokens than it did at half the load.

Beyond the cliff

Three More Ways Saturation Bites

Prefill-decode interference on a shared GPU

Prefill is compute-bound; decode is memory-bandwidth-bound. When a massive prefill runs, it saturates the GPU's matrix multiply units, stalling decode for all active sequences and causing Time-Between-Tokens spikes. Chunked prefill mitigates this by splitting long prompts into 512-2048 token chunks interleaved with decode steps.

Prefill-decode interference. A single long prompt blocks every active decode.

A subtler head-of-line blocking persists at saturation. If GPU memory is exhausted and the FIFO queue cannot admit the front request, the entire queue stalls, even if shorter requests further back would fit. Documented HOL blocking times reach tens of seconds under sustained saturation.

Why bigger blocks waste more memory

vLLM's default block size is 16 tokens. Increasing it to 128 for long-context workloads causes internal fragmentation: each sequence rounds up to the nearest block boundary, wasting the unused slots. With 256 concurrent sequences and 128-token blocks, roughly 16,000 token slots go unused, space that could have fit additional sequences.

Framework overhead adds another ceiling. Before vLLM v0.6.0, CPU and HTTP overhead limited GPU compute to only 38% of wall time. Post-v0.6.0 optimizations yielded 2.7x higher throughput and 5x faster time-per-output-token.

Wrap-up

The Cliff Is Made of Memory

Why it matters

Continuous batching transformed LLM serving by making the GPU work on the right thing at every token step. Its ceiling is memory capacity. When KV cache runs out, the same scheduling agility that enabled 36x throughput turns into an eviction cascade, keeping the GPU at 100% utilization while generating fewer useful tokens. The tell is not in GPU utilization but in preemption counters and P99 latency: those are the signals that throughput has started going backward.

The short version

Continuous batching schedules after every token, yielding up to **36.9x** throughput over prior systems.
The throughput cliff is driven by **KV cache exhaustion**, not compute limits.
At saturation, GPU utilization reads **100%** but throughput declines and P99 latency spikes from **~200ms to seconds**.
Prefill-decode interference, head-of-line blocking, and framework overhead compound the problem.
**Chunked prefill** mitigates prefill stalls but does not eliminate the memory ceiling.