What LLMs Fit on a DGX Spark?

Direct answer

Direct answer guidance: Estimate weight memory as parameters × bits ÷ 8, then add 10–20% runtime overhead and a separate KV-cache budget. If the result exceeds about 105 GB, reduce quantization/context or move to two nodes/a larger GPU server.

Model / quant variants

Model / quant variants guidance: Use llama.cpp for the broadest GGUF quant choices and partial offload. Use vLLM, SGLang, or NIM for supported Transformers/NVFP4 models when serving throughput matters.

Approximate weight footprint

Approximate weight footprint guidance: Comfortable tiers are: 7B–32B at 4–8 bit, 70B around 4–6 bit, and 120B-class MoE models around 4 bit. A 200B-class claim is plausible only with a highly compact checkpoint, modest context, and careful runtime memory control.

Hardware tiers

Hardware tiers guidance: Comfortable tiers are: 7B–32B at 4–8 bit, 70B around 4–6 bit, and 120B-class MoE models around 4 bit. A 200B-class claim is plausible only with a highly compact checkpoint, modest context, and careful runtime memory control. Use RTX 5090 for models whose complete budget stays below 28 GB, RTX PRO 6000 below roughly 80–85 GB, and dual Spark only when a single node cannot hold the weights plus cache.

Observed configurations

Observed configurations guidance: Small and medium models can be interactive; dense 27B public results sit around 30 tok/s on vLLM. Very large memory-resident models may deliver roughly 10–60 tok/s depending on sparsity, format, and runtime.

Context / concurrency effects

Context / concurrency effects guidance: Long context is the first feature to trim when a model barely fits. Start at 8K–32K, then test KV-cache use and TTFT before opening 64K+ or multiple concurrent sessions.

Performance expectations

Performance expectations guidance: Small and medium models can be interactive; dense 27B public results sit around 30 tok/s on vLLM. Very large memory-resident models may deliver roughly 10–60 tok/s depending on sparsity, format, and runtime.

Best-value configurations

Best-value configurations guidance: The best-value setup is the smallest representation that meets your quality target while leaving at least 15 GB free. Paying for a second node just to avoid a modest quantization step is usually poor value. Estimate weight memory as parameters × bits ÷ 8, then add 10–20% runtime overhead and a separate KV-cache budget. If the result exceeds about 105 GB, reduce quantization/context or move to two nodes/a larger GPU server.

What will not fit / weak evidence

• Vision encoders, speculative draft models, adapters, and large KV caches all consume memory beyond the base weight estimate.
• MoE active-parameter counts reduce compute per token but do not eliminate the need to store the full expert weights unless the runtime offloads them.

Planner CTA

Compare this recommendation against your model, context, concurrency, latency, and budget in the ComputeSage Planner.

Sources / methodology

Recommendations combine official specifications, public model/runtime documentation, adjacent benchmark observations, and clearly labeled engineering estimates. Review the ComputeSage evidence methodology.

• NVIDIA DGX Spark specifications
• llama.cpp project and backend documentation
• vLLM installation and hardware documentation
• OpenAI gpt-oss introduction and deployment guidance
• Mistral Small 4 model card
• DeepSeek-V4-Flash model card
• DeepSeek V4 Flash DGX Spark final benchmarks