Two Rigs, Five Runtimes: My Qwen3.8-27B Benchmark Across NVIDIA and AMD
Qwen3.8-27B across two rigs and five runtimes: NVFP4 on Blackwell hits 150 tok/s, the AMD box peaks at 57. The 4-bit path on sm_120 isn't what it seems.
TL;DR: NVFP4 vLLM on the Blackwell rig: 150 tok/s raw, 63 ms TTFT. The AMD rig peaks at 57 tok/s with llama.cpp + MTP. The 4-bit path on sm_120 isn't what it seems.
Why I Ran This
I run several agentic platforms and keep multiple AI projects alive. After building HYDRA10 and retooling HYDRA09, I couldn't tell which rig I could actually rely on as a default serving environment. So I gave both the same exam: Qwen3.8-27B, five runtimes, same prompts.
The Rigs
| Rig | GPU | Stack | Notes |
|---|---|---|---|
| vargos-110 | RTX PRO 6000 Blackwell Max-Q, 96 GB | CUDA 13.0 | 300 W power cap, local XFS (loads in ~5 s) |
| vargos-109 | 2× AMD Radeon AI PRO R9700, 2×32 GB | ROCm 7.x, gfx1201, no CUDA | P2P broken → every run forced to one GPU, gigabit-NFS loads (~11 min) |
The Hunt
In the order I tried them:
- llama.cpp Vulkan (109, Q4_K_M) - 57 tok/s raw, 29 answer-only. MTP draft acceptance 0.86–0.96.
- Ollama (109, Q4_K_M) - 30 tok/s raw, 13 answer-only. No MTP support, and
HSA_VISIBLE_DEVICESis a no-op. - vLLM radiance (109, AWQ INT4) - 22 tok/s single, but the only runtime that batched cleanly: 55 tok/s at 4-way (2.5× scaling).
- vLLM 0.20.1 (110, bf16) - 85 tok/s raw, 43 answer-only, 2.4–3.5 k tok/s prefill. 4-way crashed the engine (CUDA illegal memory access, reproduced 3×).
- vLLM 0.27.1 (110, NVFP4) - 150 tok/s raw, 84 answer-only, 3.8 k tok/s prefill, 63 ms TTFT.
Wait, Is NVFP4 Really FP4?
sm_120 (compute cap 12.0) is not detected as FP4-capable in vLLM 0.27.1. So "NVFP4" here is weight-only W4A16 through the Marlin kernel, dequanted to bf16 on the fly. The 1.7× speedup comes from halving weight bandwidth (51.9 → 19.6 GiB) plus MTP, not from FP4 tensor cores; those live on datacenter Blackwell, not this workstation SKU.
What I'm Relying On
- Default serving environment: vargos-110 + vLLM 0.27.1 + NVFP4 + MTP. That's where my agentic platforms will run.
- AMD path stays: vargos-109 with llama.cpp Vulkan + MTP is the best it can serve; the box remains my ROCm training box.
- Honest hedge: NVFP4 on this SKU is a workaround, not a feature. Worth revisiting when workstation Blackwell gets real FP4 support.
The headline number is 150 tok/s, but the number I keep coming back to is the 63 ms TTFT. That's the one that decides whether an agent feels instant or feels laggy.
Continue in AI