What Models Fit in 48GB VRAM?

Direct answer

Direct answer guidance: Choose 48 GB for 35B 8-bit or carefully sized 70B 4-bit. Choose 80–96 GB for reliable 70B service with long context or concurrency.

Model / quant variants

Model / quant variants guidance: Use vLLM for resident serving and llama.cpp for quantized flexibility. Measure peak allocation rather than trusting file size.

Approximate weight footprint

Approximate weight footprint guidance: Treat roughly 40–42 GB as the safe model-plus-runtime budget. A 70B 4-bit checkpoint can consume 38–45 GB after metadata, leaving limited cache; 35B 8-bit is a more comfortable target.

Hardware tiers

Hardware tiers guidance: Treat roughly 40–42 GB as the safe model-plus-runtime budget. A 70B 4-bit checkpoint can consume 38–45 GB after metadata, leaving limited cache; 35B 8-bit is a more comfortable target. 32 GB is cheaper/faster value for 35B 4-bit; 80–96 GB is safer for 70B; DGX Spark fits much larger checkpoints at lower bandwidth.

Observed configurations

Observed configurations guidance: A high-bandwidth 48 GB card can retain desktop-class latency while avoiding some 32 GB fit constraints. CPU offload should be a test fallback, not the production plan.

Context / concurrency effects

Context / concurrency effects guidance: For 70B 4-bit, start short and keep concurrency one. For 35B 8-bit, 16K–32K may be practical with measured headroom.

Performance expectations

Performance expectations guidance: A high-bandwidth 48 GB card can retain desktop-class latency while avoiding some 32 GB fit constraints. CPU offload should be a test fallback, not the production plan.

Best-value configurations

Best-value configurations guidance: The tier is valuable only when a 32 GB deployment fails a defined precision/context requirement. Otherwise the premium may buy unused memory. Choose 48 GB for 35B 8-bit or carefully sized 70B 4-bit. Choose 80–96 GB for reliable 70B service with long context or concurrency.

What will not fit / weak evidence

• Some 48 GB cards are much slower than current 32 GB flagships.
• Quantization metadata and runtime workspaces can push a nominal 35 GB weight estimate above 40 GB.

Planner CTA

Compare this recommendation against your model, context, concurrency, latency, and budget in the ComputeSage Planner.

Sources / methodology

Recommendations combine official specifications, public model/runtime documentation, adjacent benchmark observations, and clearly labeled engineering estimates. Review the ComputeSage evidence methodology.

• Qwen3.5-35B-A3B model repository
• Qwen3.6-35B-A3B model card
• vLLM installation and hardware documentation
• llama.cpp project and backend documentation
• NVIDIA Developer Forums Qwen3.6 NVFP4 cross-platform report