Prefill vs Decode in LLM Inference

Plain-English answer

Plain-English answer guidance: Prefill measures how quickly a runtime processes the input prompt; decode measures how quickly it emits new tokens. Use TTFT/prefill for interactive responsiveness and decode for long-answer speed—never collapse them into one performance number.

Why it matters

Why it matters guidance: Chat products need low TTFT plus adequate decode; batch summarization values total throughput; RAG/agents benefit strongly from prefix caching and prompt discipline. Context length affects prefill nearly linearly or worse depending on attention implementation, while cache memory grows with stored tokens and sequences.

Real StackBench examples

Common misconception

Common misconception guidance: Many published results omit prompt length, batch size, or whether tok/s is aggregate, limiting comparison. TTFT includes scheduling and queueing in a service, not only model prefill.

Measured / example table

Decision rule

Decision rule guidance: For chat, prioritize p95 TTFT and single-stream decode. For batch/API service, prioritize aggregate prefill/decode under the target concurrency.

Related benchmarks

Related benchmarks guidance: A representative result can show thousands of prompt tok/s but only tens of decode tok/s. That is normal: they exercise different phases and should guide different product choices. Shorten prompts, cache prefixes, retrieve fewer chunks, or use a smaller model when TTFT is the problem; use higher bandwidth or speculative decoding when decode is the problem.

Planner / Explore CTA

Compare this recommendation against your model, context, concurrency, latency, and budget in the ComputeSage Planner.

Sources / methodology

Recommendations combine official specifications, public model/runtime documentation, adjacent benchmark observations, and clearly labeled engineering estimates. Review the ComputeSage evidence methodology.

• vLLM installation and hardware documentation
• llama.cpp project and backend documentation
• StackBench controlled lab approved source qwen38-rtx5090-owner-matrix-20260816