Published · Phase 1
Prefill vs Decode in LLM Inference
Prefill is the work of processing the prompt before generation; decode is the iterative work of producing output tokens afterward. They use different execution patterns, so a model can have fast prompt ingestion and modest generation—or the reverse—without contradiction.
Editorial review: complete · Updated 2026-08-30
Plain-English answer
Prefill reads the input in parallel and builds the starting cache; decode then generates one step at a time while reusing that cache. The two phases should be reported separately because their bottlenecks and user impact differ.
Why it matters
A long-document workflow can spend much of its wait before the first token, while a short chat may be dominated by generation cadence. Optimizing the wrong phase can leave the actual user experience unchanged.
Real StackBench examples
| Configuration | Evidence state | Metric | Source |
|---|---|---|---|
| Qwen3.8-27B · RTX 5090 owner-collected public lab · llama.cpp | Measured · Grade A | 3515.02 tokens/second | StackBench controlled lab approved source qwen38-rtx5090-owner-matrix-20260816 |
| Qwen3.8-27B · RTX 5090 owner-collected public lab · llama.cpp | Measured · Grade A | 66.8177 tokens/second | StackBench controlled lab approved source qwen38-rtx5090-owner-matrix-20260816 |
Common misconception
Time to first token is not automatically prefill time. Queueing, tokenization, network handling, scheduling, cache hits, and the first decode step can all be included in TTFT.
Measured / example table
| Configuration | Evidence state | Metric | Source |
|---|---|---|---|
| Qwen3.8-27B · RTX 5090 owner-collected public lab · llama.cpp | Measured · Grade A | 3515.02 tokens/second | StackBench controlled lab approved source qwen38-rtx5090-owner-matrix-20260816 |
| Qwen3.8-27B · RTX 5090 owner-collected public lab · llama.cpp | Measured · Grade A | 66.8177 tokens/second | StackBench controlled lab approved source qwen38-rtx5090-owner-matrix-20260816 |
Decision rule
For long inputs, inspect strict prefill or a clearly labeled proxy and TTFT; for long outputs, inspect per-request decode; for services, add aggregate throughput without replacing the per-request metrics.
Related benchmarks
The controlled RTX 5090 context matrix provides one paired prefill/decode example. Keep its model, runtime, prompt, output, context, and concurrency attached before extending the phase distinction elsewhere.
Planner / Explore CTA
Carry this workload into the ComputeSage Planner. It keeps missing evidence visible rather than inventing a recommendation.
Sources / methodology
Review the ComputeSage evidence methodology. Bound selector: protocol-v2 methodology + controlled rows. Eligible source: StackBench controlled lab approved source qwen38-rtx5090-owner-matrix-20260816; StackBench controlled lab approved source qwen38-rtx5090-owner-matrix-20260816.