What Models Fit in 32GB VRAM?

Direct answer

Direct answer guidance: Choose 32 GB if a 27B–35B 4-bit model satisfies your evaluation. Move to 48–96 GB when higher precision, 70B, long context, or concurrency is required.

Model / quant variants

Model / quant variants guidance: Use llama.cpp for GGUF and vLLM for a supported API service. Full GPU residency is more important than forcing a larger checkpoint through CPU offload.

Approximate weight footprint

Approximate weight footprint guidance: Target no more than 24–28 GB for weights plus runtime before cache. Qwen3.8-27B and Qwen3.5/3.6-35B are strong 4-bit examples; 70B 4-bit is generally too tight.

Hardware tiers

Hardware tiers guidance: Target no more than 24–28 GB for weights plus runtime before cache. Qwen3.8-27B and Qwen3.5/3.6-35B are strong 4-bit examples; 70B 4-bit is generally too tight. 24 GB works for smaller 4-bit models with tighter cache; 48 GB provides a safer bridge; 128 GB unified memory prioritizes capacity.

Observed configurations

Observed configurations guidance: Modern 32 GB high-bandwidth cards can deliver roughly 70–160 tok/s for optimized 27B–35B 4-bit models, and much more for small models.

Context / concurrency effects

Context / concurrency effects guidance: 8K–16K is the default for 27B–35B; use retrieval instead of assuming maximum context will fit comfortably.

Performance expectations

Performance expectations guidance: Modern 32 GB high-bandwidth cards can deliver roughly 70–160 tok/s for optimized 27B–35B 4-bit models, and much more for small models.

Best-value configurations

Best-value configurations guidance: This tier often gives the strongest throughput per dollar because it avoids professional-memory pricing while holding capable models. Choose 32 GB if a 27B–35B 4-bit model satisfies your evaluation. Move to 48–96 GB when higher precision, 70B, long context, or concurrency is required.

What will not fit / weak evidence

• A 28 GB checkpoint is not a 28 GB service after cache and runtime buffers.
• Check power, chassis, and host-system compatibility as part of the purchase.

Planner CTA

Compare this recommendation against your model, context, concurrency, latency, and budget in the ComputeSage Planner.

Sources / methodology

Recommendations combine official specifications, public model/runtime documentation, adjacent benchmark observations, and clearly labeled engineering estimates. Review the ComputeSage evidence methodology.

• NVIDIA GeForce RTX 5090 specifications
• Qwen3.8-27B model repository
• Qwen3.6-35B-A3B model card
• Qwen3.5-35B-A3B model repository
• vLLM installation and hardware documentation
• llama.cpp project and backend documentation
• StackBench controlled lab approved source qwen38-rtx5090-owner-matrix-20260816