How Much VRAM Does a Local LLM Need?

Direct answer

Direct answer guidance: Buy the smallest tier that leaves 15% headroom for the real context and concurrency. Move up a tier instead of planning production around CPU offload.

Model / quant variants

Model / quant variants guidance: llama.cpp is flexible near memory limits; vLLM works best with resident supported checkpoints and explicit memory-utilization/concurrency settings.

Approximate weight footprint

Approximate weight footprint guidance: Calculate parameters × bits ÷ 8, add 10–20% runtime overhead, then add KV cache, vision encoders, adapters, draft models, and concurrent sequences. Keep 10–20% free after all of that.

Hardware tiers

Hardware tiers guidance: Calculate parameters × bits ÷ 8, add 10–20% runtime overhead, then add KV cache, vision encoders, adapters, draft models, and concurrent sequences. Keep 10–20% free after all of that. Cloud GPUs cover occasional high-memory work; model distillation/quantization can reduce the tier; multiple GPUs add capacity but require topology and sharding validation.

Observed configurations

Observed configurations guidance: More memory does not automatically mean more speed. Bandwidth and kernels decide tok/s once the workload fits; insufficient memory causes offload, failure, or severe latency.

Context / concurrency effects

Context / concurrency effects guidance: Context and concurrency are first-class memory inputs. A card sized from weights alone is often one tier too small.

Performance expectations

Performance expectations guidance: More memory does not automatically mean more speed. Bandwidth and kernels decide tok/s once the workload fits; insufficient memory causes offload, failure, or severe latency.

Best-value configurations

Best-value configurations guidance: Memory premiums are justified only when they prevent offload, enable required precision, or support revenue-producing concurrency. Buy the smallest tier that leaves 15% headroom for the real context and concurrency. Move up a tier instead of planning production around CPU offload.

What will not fit / weak evidence

• MoE active parameters do not determine resident weight memory.
• Peak runtime memory should be measured on the exact model revision before final procurement.

Planner CTA

Compare this recommendation against your model, context, concurrency, latency, and budget in the ComputeSage Planner.

Sources / methodology

Recommendations combine official specifications, public model/runtime documentation, adjacent benchmark observations, and clearly labeled engineering estimates. Review the ComputeSage evidence methodology.

• NVIDIA GeForce RTX 5090 specifications
• NVIDIA RTX PRO 6000 Blackwell specifications
• NVIDIA DGX Spark specifications
• Qwen3.8-27B model repository
• OpenAI gpt-oss introduction and deployment guidance
• llama.cpp project and backend documentation
• vLLM installation and hardware documentation
• StackBench controlled lab approved source qwen38-rtx5090-owner-matrix-20260816

Plain-English answer

Plain-English answer guidance: Choose GPU memory by the complete workload: 16 GB for small models, 24–32 GB for 20B–35B at 4 bit, 48 GB for comfortable 35B/limited 70B, 80–96 GB for 70B high precision or 120B low bit, and 128 GB for large local experimentation.

Why it matters

Why it matters guidance: This guide is for buyers who know the model family and precision but need a defensible VRAM tier. Context and concurrency are first-class memory inputs. A card sized from weights alone is often one tier too small.

Real StackBench examples

Common misconception

Common misconception guidance: Vendor/model packaging and dynamic buffers make every estimate a range rather than an exact byte guarantee. MoE active parameters do not determine resident weight memory.

Measured / example table

Decision rule

Decision rule guidance: Buy the smallest tier that leaves 15% headroom for the real context and concurrency. Move up a tier instead of planning production around CPU offload.

Related benchmarks

Related benchmarks guidance: More memory does not automatically mean more speed. Bandwidth and kernels decide tok/s once the workload fits; insufficient memory causes offload, failure, or severe latency. Cloud GPUs cover occasional high-memory work; model distillation/quantization can reduce the tier; multiple GPUs add capacity but require topology and sharding validation.