TTFT vs Prefill Throughput

Plain-English answer

Plain-English answer guidance: Time to first token is the better responsiveness metric; prefill tok/s explains part of it but not queueing, startup, cache hits, or network overhead. Optimize p95 TTFT for interactive products, not the maximum prompt-processing rate.

Why it matters

Why it matters guidance: Interactive chat, coding completion, voice, and agent tools should prioritize TTFT; offline summaries can accept slower first output for higher total throughput. Long context directly increases prefill work, so context limits are a latency policy as well as a memory policy.

Real StackBench examples

Common misconception

Common misconception guidance: It varies with load and transport, so a single unloaded localhost measurement is insufficient for a service SLA. Report median and tail latency with concurrency, not only the best run.

Measured / example table

Decision rule

Decision rule guidance: Set a target such as sub-2-second p95 for interactive use, then tune prompt size, cache, queue depth, and model before buying hardware solely on prefill tok/s.

Related benchmarks

Related benchmarks guidance: A reported TTFT around several seconds can coexist with thousands of prefill tok/s if the prompt is large or initialization/queueing dominates. Record prompt tokens beside the latency. Warm the model, cache shared prefixes, reduce retrieved chunks, stream immediately, or route simple queries to a smaller model.

Planner / Explore CTA

Compare this recommendation against your model, context, concurrency, latency, and budget in the ComputeSage Planner.

Sources / methodology

Recommendations combine official specifications, public model/runtime documentation, adjacent benchmark observations, and clearly labeled engineering estimates. Review the ComputeSage evidence methodology.

• vLLM installation and hardware documentation
• llama.cpp project and backend documentation
• StackBench controlled lab approved source ninfer-qwen38-27b-nvfp4-rtx5090-owner-20260821