Two Rigs, Five Runtimes: My Qwen3.8-27B Benchmark Across NVIDIA and AMD

TL;DR: The AMD rig peaks at 57 tok/s with llama.cpp + MTP. And, NVFP4 vLLM on the NVIDIA rig: 150 tok/s raw, 63 ms TTFT. Using 4bit precision is more than enough than 16bit one for most of my use case.

Why I ran this

I run some agentic platforms across different nodes in my proxmox, to keep multiple projects maintained. After building HYDRA10 and HYDRA09, I am not sure which rig I could rely on as default and fallback environment since I switch context a lot. So I built another proxmox node with mutliple AMD GPUs on top of consumer motherboard (non-bifurcated), which I will focus to make it alive as local inference box, and reserves the other one for my playground.

So I gave both the same exam: Qwen3.8-27B, five runtimes, same prompts.

The Rigs

The Hunt

Notes:

• vLLM radiance was the only AMD runtime that batched cleanly (22 → 55 tok/s at 4-way).
• vLLM 0.20.1 on the NVIDIA rig scaled to ~126 tok/s at 2-way (~1.4× the ~88 tok/s single-stream rate), then crashed at 4-way (CUDA illegal memory access, reproduced 3×).

How much did MTP actually earn? On the NVIDIA vLLM runs, per-position acceptance decayed 0.78 / 0.58 / 0.43 / 0.33 / 0.20 across the 5 draft positions - an average draft acceptance of 44-67%. llama.cpp was higher: 0.59–0.96, mean draft length ~3.3–3.9 tokens. MTP plus the Vulkan Q4_K_M kernel against vLLM's eager path is why the AMD single-stream number beats vLLM radiance.

This model likes to think!

Qwen3.8-27B reasons before it answers. Roughly half of the output tokens are hidden chain-of-thought, which is why the raw 150 tok/s drops to 84 tok/s of visible text. I've been running it with reasoning effort set to low or medium to keep it sane. The answer-only rate is the number my platforms actually feel.

On llama.cpp I cap thinking with --reasoning-budget 4096 (the model is forced to answer once it hits the cap) and keep it with --reasoning-preserve, which returns the thinking in a separate reasoning_content field.

vLLM runs set preserve_thinking: false, so thinking is stripped from content but still - returned in the reasoning field. Either way, the split lands at ~50/50 - in one math benchmark run it was 173 thinking tokens vs 173 answer tokens.

Configs

Exact launch args are recorded in my benchmark notes.

Sampling was held constant across the llama.cpp and vLLM runs: temp 1.0, top_p 0.95, top_k 20, min_p 0, presence penalty 0, repeat penalty 1.0.

Context window + KV dtype: vLLM 0.20.1 (bf16) and 0.27.1 (NVFP4) ran --max-model-len 262144 (256K) with fp8 KV - 729,444 KV cache tokens for bf16, 1,588,535 for NVFP4; vLLM radiance ran --max-model-len 196608 (192K) with fp8 KV (256K didn't fit); llama.cpp ran -c 524288 (512K) with q4_0 KV.

Full Launch Commands

The bench pinned every AMD run to a single GPU with Vulkan build (P2P didn't work for me). However, these are the commands I'm running now, with the 2-GPU layer split back online with HIP build.

llama.cpp (AMD rig)

# 2-GPU layer split; q4_0 KV halves the cache size - what lets -np 4 fit
# Spec decode (MTP): min 3 / max 5 draft tokens
llama-server \
  -m /path/to/Qwen3.8-27B-Uncensored-Q4_K_M.gguf \
  --host 0.0.0.0 --port 11446 \
  --alias qwen3.8-27b \
  \
  -ngl 99 --split-mode layer --tensor-split 1,1 \
  \
  -c 524288 -np 4 -b 2048 -ub 512 \
  \
  --flash-attn on \
  --cache-type-k q4_0 --cache-type-v q4_0 \
  \
  --spec-type draft-mtp --spec-draft-n-min 3 --spec-draft-n-max 5 \
  \
  --mmproj /path/to/mmproj-F16.gguf \
  \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 \
  --presence-penalty 0.0 --repeat-penalty 1.0 \
  \
  --reasoning on --reasoning-effort low --reasoning-budget 4096 \
  --reasoning-preserve \
  --chat-template-kwargs '{"reasoning_effort":"low", "preserve_thinking": false}' \
  --reasoning-budget-message "Wait, I'm overthinking this. Let's either clarify to user or answer now." \
  --jinja

vLLM (NVIDIA rig)

# Env: VLLM_ATTENTION_BACKEND=FLASHINFER, FLASHINFER_DISABLE_VERSION_CHECK=1,
# VLLM_HTTP_TIMEOUT_KEEP_ALIVE=600
# NVFP4 W4A16 checkpoint (19.15 GiB), 256K ctx, fp8 KV
vllm serve /path/to/Qwen3.8-27B-Uncensored-NVFP4A16 \
  --host 0.0.0.0 --port 11401 \
  --served-model-name qwen3.8-27b-uncensored \
  --quantization compressed-tensors \
  --gpu-memory-utilization 0.90 \
  --dtype auto \
  --kv-cache-dtype fp8 \
  --max-model-len 262144 \
  --max-num-batched-tokens 32768 \
  --max-num-seqs 8 \
  --attention-backend flashinfer \
  --enable-prefix-caching \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --default-chat-template-kwargs '{"reasoning_effort": "medium", "preserve_thinking": false}' \
  --speculative-config '{"method":"mtp","num_speculative_tokens":5}' \
  --trust-remote-code \
  --enforce-eager

Will it fit your card?

What I'm Relying On

• Default serving environment: the NVIDIA rig + vLLM 0.27.1 + NVFP4 + MTP. That's where my agentic platforms will run.
• AMD path stays: the AMD rig with llama.cpp Vulkan + MTP is the best it can serve; that box remains my ROCm training box.

The headline number is 150 tok/s, but the one I keep coming back to is the 63 ms TTFT - the number that decides whether an agent feels instant or laggy.