Qwen3.8 27B on RTX 5090

Measured / sourced results

What the numbers mean

What the numbers mean guidance: Public configurations place ordinary llama.cpp-style decode around the high tens of tok/s and specialized kernels around or above 100 tok/s. Budget 70–130 tok/s after warm-up for a well-tuned 4-bit deployment. Start with 4 bit, 16K context, and full GPU offload. Move to 5/6/8 bit only after a task-specific evaluation shows a meaningful quality improvement.

What stands out

Recommended configuration: 4-bit weights, 16K context, full GPU offload.

Expected decode: roughly 70–130 tok/s with a current runtime.

Avoid 8 bit unless short context and measured quality gains justify the tight memory budget.

Evidence quality / source

Evidence quality / source guidance: Official repository size anchors the precision estimates; the throughput range is based on multiple public and local configuration-specific observations. NInfer, vLLM, and llama.cpp numbers are not interchangeable unless runtime, quantization, prompt, and batching match.

What is unknown

• NInfer, vLLM, and llama.cpp numbers are not interchangeable unless runtime, quantization, prompt, and batching match.
• Multimodal inputs add encoder and cache memory beyond the text-only weight estimate.

Workload fit

Workload fit guidance: Recommended for coding, document analysis, agents, and local multimodal work where a 27B model offers better quality than small models without leaving consumer-GPU territory. A practical 4-bit build is approximately 16–19 GB and leaves room for long prompts or a second small model. An 8-bit build can approach 29–33 GB and is operationally tight on a 32 GB card.

Alternatives

Alternatives guidance: Qwen3.6-35B-A3B can be faster per active token for agent work; a 14B model reduces latency; DGX Spark enables higher precision and much longer cache headroom but is slower on this size.

Planner CTA

Compare this recommendation against your model, context, concurrency, latency, and budget in the ComputeSage Planner.

Sources / methodology

Recommendations combine official specifications, public model/runtime documentation, adjacent benchmark observations, and clearly labeled engineering estimates. Review the ComputeSage evidence methodology.

• Qwen3.8-27B model repository
• NVIDIA GeForce RTX 5090 specifications
• llama.cpp project and backend documentation
• vLLM installation and hardware documentation
• StackBench controlled lab approved source ninfer-qwen38-27b-nvfp4-rtx5090-owner-20260821