Published · Phase 2

What Models Fit on an RTX 5090?

The useful RTX 5090 model range is broader than a simple parameter cutoff: 7B–14B runs at high precision, 20B–35B runs well at 4 bit, and 70B usually requires compromises that erase the card's speed advantage.

Editorial review: complete · Updated 2026-08-30

Direct answer

Direct answer guidance: Choose 7B–14B for maximum responsiveness, 27B–35B for the best capability/speed balance, and higher-memory hardware for 70B unless occasional slow experimentation is acceptable.

Model / quant variants

Model / quant variants guidance: llama.cpp makes it easy to test Q4/Q5 GGUF files and partial offload. vLLM is better when the selected Transformers checkpoint fits fully and the goal is concurrent serving.

Approximate weight footprint

Approximate weight footprint guidance: Use weight math first: parameters × bits ÷ 8, then add 10–20% overhead and KV cache. A reliable service target is below 28 GB total, leaving 4 GB for CUDA, cache growth, and request bursts.

Hardware tiers

Hardware tiers guidance: Use weight math first: parameters × bits ÷ 8, then add 10–20% overhead and KV cache. A reliable service target is below 28 GB total, leaving 4 GB for CUDA, cache growth, and request bursts. 48 GB cards widen 35B precision/context options; 80–96 GB cards handle 70B at useful precision; DGX Spark trades bandwidth for 128 GB capacity.

Observed configurations

Observed configurations guidance: 7B–14B models can reach hundreds of tok/s, while 27B–35B models commonly land in the 70–160 tok/s range under good 4-bit kernels. CPU-offloaded 70B models can be far slower despite technically loading.

Context / concurrency effects

Context / concurrency effects guidance: A 4-bit 35B model may fit at short context but fail under long prompts. Include the intended context and concurrency in every fit calculation.

Performance expectations

Performance expectations guidance: 7B–14B models can reach hundreds of tok/s, while 27B–35B models commonly land in the 70–160 tok/s range under good 4-bit kernels. CPU-offloaded 70B models can be far slower despite technically loading.

Best-value configurations

Best-value configurations guidance: The best-value model is the smallest one that passes your workload evaluation. Do not pay the latency and power cost of a larger model simply because it can be made to load. Choose 7B–14B for maximum responsiveness, 27B–35B for the best capability/speed balance, and higher-memory hardware for 70B unless occasional slow experimentation is acceptable.

What will not fit / weak evidence

  • Vision encoders, adapters, speculative draft models, and runtime graphs consume memory beyond the base checkpoint.
  • Parameter count alone is insufficient for MoE models: resident weights follow total parameters while compute follows active parameters.

Planner CTA

Compare this recommendation against your model, context, concurrency, latency, and budget in the ComputeSage Planner.

Sources / methodology

Recommendations combine official specifications, public model/runtime documentation, adjacent benchmark observations, and clearly labeled engineering estimates. Review the ComputeSage evidence methodology.