Deep Dive · Issue 22

How Chunked Prefill Stops Decode Starvation

Long prompts freeze token generation for everyone else. Chunked prefill slices the problem into fixed-size pieces and keeps decodes flowing.

10 October 2026 A GPU blocked on a long prefill while decode tokens wait

The stall

One Long Prompt Freezes Every Other User

A GPU blocked on a long prefill while decode tokens wait

LLM inference runs in two phases with opposite profiles. Prefill processes the whole prompt at once, saturating GPU tensor cores. Decode generates one token at a time, reading the full KV cache from HBM each step and leaving compute cores idle. GPU utilization falls to 20-40% during decode-only iterations.

Prefill-prioritizing scheduling: a new long prompt stalls every ongoing decode

Continuous batching systems like Orca and vLLM traditionally use prefill-prioritizing scheduling: when a new request arrives, the scheduler pauses all ongoing decodes to run the full prefill in one shot. A prompt of 8K, 32K tokens can take hundreds of milliseconds to seconds. Every other user's token generation freezes for the entire duration, what Sarathi-Serve calls a generation stall.

p50 looks fine. p95 inter-token latency spikes 5-10x the moment a long-context request joins the batch.

The fix

Slice the Prefill, Interleave the Decodes

Prefill split into chunks interleaved with decode tokens through a GPU

Chunked prefill splits a long prefill into fixed-size chunks (typically 512 tokens). Each iteration processes one chunk while decodes from active sequences run alongside. Causal masking holds because each chunk attends to all previous chunks of the same prompt.

Each iteration pairs one prefill chunk with ongoing decodes. No decode waits for a full prompt.

The scheduler builds a hybrid batch: one prefill chunk plus as many decodes as fit under a token budget tuned so each iteration finishes within the time-between-tokens SLO. Decodes piggyback on the prefill's compute at up to an order of magnitude less cost than a decode-only batch.

Token budget — The cap on total tokens (prefill chunks + decode tokens) processed in a single forward pass. Chosen so each iteration finishes within the time-between-tokens SLO.

In vLLM's V1 engine, the policy prioritizes decodes: it batches all pending decodes first, then schedules prefills with whatever budget remains. If a prefill doesn't fit, it gets chunked.

Why 512 is the sweet spot

SARATHI aligns chunk size and decode count to GPU matmul tile boundaries. On A100 FP16, the knee where a kernel transitions from memory-bound to compute-bound sits around 128-512 tokens. The optimal prefill-to-decode ratio is approximately C/(B−1), placing the batch at the knee to maximize both FLOPs and bandwidth utilization.

The payoff

2.6× to 5.6× More Requests at the Same Latency

Throughput comparison showing chunked prefill outperforming baseline

Sarathi-Serve's gains scale with model size. On Mistral-7B with a single A100, serving capacity rises 2.6× over vLLM at the same tail-latency target. On Yi-34B with 2×A100, it hits 3.7×, and on Falcon-180B across 8 GPUs, 5.6×. The system needs only two parameters: chunk size and token budget. NVIDIA TensorRT-LLM ships the same technique.

\text{one-shot: } \Theta(L_p^2) \qquad \text{chunked: } \Theta\!\left(\sum_{i=1}^{\lceil L_p/C \rceil} C \cdot iC\right) = \Theta(L_p^2)

Chunking doesn't change the asymptotic attention cost. At C=512 the measured overhead is <3%.

The chunking overhead is small. Total FFN cost is unchanged from one-shot prefill, and total attention cost stays Θ(Lp²) regardless of chunk count. At C=512, SARATHI measures <3% extra attention compute, and prefill loss stays under 10%. On vLLM v0.18.0 with an H100, chunked prefill cuts TTFT p95 for a 32K-token prompt from 2,800 ms to 890 ms, a 68% reduction.

The knob

One Budget, Two Directions

Token budget dial controlling the throughput-latency tradeoff

The token budget is the single knob controlling the throughput-latency tradeoff. A smaller budget improves inter-token latency but can hurt time-to-first-token. A larger budget packs more prefill work per iteration, improving TTFT but letting decodes wait longer.

Small budget (512)

better ITL, worse TTFT

Large budget (2048+)

better TTFT, worse ITL

vLLM's default started at 512 in v0.4.2, tuned for ITL on A100, then rose to 2048 in v0.8.5. The docs recommend >2048 for throughput-oriented workloads. In V1, the budget is dynamic, typically 8192-32768.

The catch

The gains come with constraints. In vLLM v0.18.0, chunked prefill is incompatible with draft-model-based speculative decoding, and CUDA graphs aren't supported for chunked iterations. Chunk size is also tuned offline with no adaptive policy when load or hardware changes.

Wrap-up

The Worst Case Is Now Bounded

Why it matters

The shift from prefill-prioritizing to decode-prioritizing scheduling changes what tail latency means in LLM serving. Before chunked prefill, a single long prompt could freeze every user sharing a GPU; after, the worst case is bounded by chunk size, not prompt length. For teams running vLLM or TensorRT-LLM in production, p95 inter-token latency becomes a tunable parameter instead of a gamble on workload mix.

The short version

Chunked prefill splits long prefills into fixed-size chunks (typically **512 tokens**) and interleaves decode steps between chunks, eliminating head-of-line blocking.
A **token budget** caps total tokens per iteration, keeping each step within the time-between-tokens SLO.
Sarathi-Serve achieves **2.6× to 5.6×** higher serving capacity at the same tail latency, with only two tunable parameters.
On vLLM with an H100, a 32K-token prompt's TTFT p95 drops from 2,800 ms to **890 ms**.
The token budget is the single knob: smaller for inter-token latency, larger for time-to-first-token.