Qwen3.8 Flash-Next on a 32 GB R9700: Two Users at 250K Context
Flash-Next Gyro-S on one R9700 with 32 GB of VM RAM: two users passed a 249K-token retrieval test, while four hit a runtime watchdog.
I wanted to replace the Qwen3.8-35B-A3B serving my agents with Flash-Next after seeing it run on an RTX 4090. My AMD box managed two users at nearly 249K input tokens each. Four users stalled.
That was with just 31.3 GiB of RAM inside the VM. The host has 128 GB, but I hadn't increased the VM allocation yet.
What ran on the card
I downloaded AgentionAI's Gyro-S Abliterated GGUF, a 58.5 GB file named Qwen3.8-Flash-Next-Gyro-S-abliterated-TQ1_0.gguf.
The file is bigger than the card. Gyro keeps its n-gram lookup table on disk, and this format needed Strata; my existing llama.cpp build couldn't load it. These results apply to this particular quantization and runtime.
| Setting | Tested configuration |
|---|---|
| GPU | One Radeon AI PRO R9700, 32 GB, on ROCm 0 |
| VM RAM | 31.3 GiB |
| Runtime | Strata HIP 0.1.48 |
| Context | 250,000 tokens per slot |
| KV cache | INT8; 32,768 tokens on GPU, remainder streamed from RAM |
| Storage | NFS; cold starts took several minutes |
Whisper and Chatterbox stayed on ROCm 1. All request timings below exclude model startup.
The wait at 249K tokens
Each request contained 248,894 input tokens: a synthetic archive with three target values near the beginning, middle, and end. I asked the model to return them exactly, allowing 768 output tokens. Successful requests reported no cached prompt tokens.
| Concurrent users | Complete answers | Retrieval |
|---|---|---|
| 1 | 109.6 seconds | All three targets correct |
| 2 | 179.4 and 241.5 seconds | Both returned all three targets |
| 4 | No output; test ended after about 485 seconds | Watchdog terminated the run |
Single-user generation reached about 74 tokens/s after first output. With two users, one request's prompt processing paused the other's generation. The slower user waited almost four minutes for a finished answer.
My previous 35B model took about 563 seconds to reach first output on the same archive text, compared with 102 seconds here. Different chat templates changed the token count slightly, and the old run's lower output cap cut off its answer. That comparison only tells me about first-output latency.
Short prompts were much easier
| Concurrent users | Generation per user | Aggregate throughput* |
|---|---|---|
| 1 | 68.1 tokens/s | 60.0 tokens/s |
| 2 | 39.6 to 43.1 tokens/s | 73.1 tokens/s |
| 4 | 19.9 to 22.5 tokens/s | 76.1 tokens/s |
Aggregate throughput includes prompt processing and waiting. All token rates include reasoning.
These requests had a 256-token output cap. Several stopped before a final answer, so this measures throughput, not answer quality. Four users added little total throughput over two while roughly halving each user's generation rate.
Why I kept two slots
The four-user long-context run stopped advancing at input token 248,889. After 60 seconds without progress, the engine watchdog killed it. The VM had around 2.4 GiB of RAM available, and four slots had reduced the GPU expert cache from about 24 GiB to 22.3 GiB.
Memory pressure was present; the logs don't establish what caused the hang. More RAM still needs testing.
I left the service on port 11449 with two 250K slots, INT8 KV cache, and MTP draft and vision disabled. The old service is stopped, and my six VARGOS model catalogs now point to Flash-Next.
The benchmark data and test conditions include the failed run and the model checksum. Next I'll give the VM around 96 to 100 GB of RAM and repeat the concurrency test, then try a real agent workflow.
Continue in AI