Published · Phase 3

What Can You Run With 128GB Unified Memory?

A 128 GB AI system can run 70B at high precision, 120B around 4–8 bit, and selected larger MoE checkpoints. The best use is capacity-heavy local work—not assuming every model will be fast simply because it fits.

Editorial review: complete · Updated 2026-08-30

Direct answer

Direct answer guidance: Choose 128 GB when a validated workload exceeds 80–96 GB or when local capacity/low power matters more than maximum token speed.

Model / quant variants

Model / quant variants guidance: llama.cpp offers the widest low-bit model access; vLLM/NIM/SGLang provides stronger serving when the model/architecture is supported.

Approximate weight footprint

Approximate weight footprint guidance: Use approximately 100–105 GB as a comfortable complete-workload ceiling. That leaves room for cache, runtime, OS, multimodal components, and modest concurrency.

Hardware tiers

Hardware tiers guidance: Use approximately 100–105 GB as a comfortable complete-workload ceiling. That leaves room for cache, runtime, OS, multimodal components, and modest concurrency. RTX PRO 6000 is faster below 85 GB; multi-GPU servers supply more bandwidth/concurrency; cloud GPUs handle intermittent giant-model work.

Observed configurations

Observed configurations guidance: Expect a broad range: 25–60 tok/s for many 27B–120B optimized workloads on Spark, lower for giant dense/MoE deployments, and much higher aggregate rates under batching.

Context / concurrency effects

Context / concurrency effects guidance: Use surplus memory for reliability first, then longer context or additional sequences. Advertised maximum context can still have impractical TTFT.

Performance expectations

Performance expectations guidance: Expect a broad range: 25–60 tok/s for many 27B–120B optimized workloads on Spark, lower for giant dense/MoE deployments, and much higher aggregate rates under batching.

Best-value configurations

Best-value configurations guidance: Capacity can avoid a server, but buying 128 GB for a 20 GB model wastes the platform's main advantage. Choose 128 GB when a validated workload exceeds 80–96 GB or when local capacity/low power matters more than maximum token speed.

What will not fit / weak evidence

  • 128 GB unified memory is not equivalent to a 128 GB high-bandwidth discrete GPU.
  • Reserve at least 15% rather than designing to the last gigabyte.

Planner CTA

Compare this recommendation against your model, context, concurrency, latency, and budget in the ComputeSage Planner.

Sources / methodology

Recommendations combine official specifications, public model/runtime documentation, adjacent benchmark observations, and clearly labeled engineering estimates. Review the ComputeSage evidence methodology.