My Four RTX 3090s Were Slower Than a Strix Halo. Now They Do 280 tok/s.

My homelab has two inference boxes. A Strix Halo Framework Desktop handles everyday requests, and a rack server with four RTX 3090s handles training and runs inference when it isn't training. Both serve the same model, Qwen3.8 27B, behind a LiteLLM proxy that sends requests to the 3090s first.

Then I noticed that the Strix Halo was faster. It was generating 27 tok/s; the four 3090s were doing 10 tok/s. That's 96 GB of GDDR6X at 936 GB/s per card losing to one APU with about 256 GB/s of shared memory.

After two days of testing, the same four cards now generate 143 tok/s on prose and 273-295 tok/s on code and JSON, about 5× the Strix Halo.

TL;DR

• The hardware was never the problem. The server was running vLLM in eager mode because CUDA graphs crashed on this model in vLLM 0.28. Upgrading fixed that, for a 7× gain.
• Speculative decoding was the biggest single win once the basics were right. A small draft model proposes tokens and the 27B model checks them in one pass. Code and JSON are predictable, so they gain the most.
• One flag cost me 3× without any error message. With --dtype half the target model accepted none of the draft's tokens. --dtype bfloat16 fixed it.
• Prefix caching cut the wait for a repeated 11k-token prompt from 7 s to 0.86 s, which matters a lot for coding agents.
• 250 W per card is the sweet spot for decode. Going higher mostly adds heat.

Step 1: It wasn't PCIe, it was eager mode

My first theory was the interconnect. The cards have no NVLink, they talk over PCIe, and one of them sits on a long riser that logs corrected PCIe errors. Tensor parallelism across four cards has to sync after every layer, so a slow link should hurt.

It wasn't that. The same model on two cards in eager mode was just as slow as on four. The riser card measured 26.0 GB/s against 26.3 GB/s for a healthy one. The slow part was eager mode itself.

--enforce-eager makes vLLM run the model as plain PyTorch, with both torch.compile and CUDA graphs turned off. Every token becomes a long chain of small GPU kernels, each launched from the CPU, and on four cards that need to sync after every layer the GPUs spend most of their time waiting for the next launch. CUDA graphs record those launches once and replay them, which removes most of that CPU overhead. I'd turned them off weeks earlier because they crashed vLLM 0.28 with this model, a hybrid with Gated DeltaNet layers that was new at the time. I had wrongly blamed the crash on Ampere.

vLLM 0.32 fixed whatever was crashing. Same model, same cards, same 200 W cap, graphs on:

Lesson: if a multi-GPU vLLM setup is mysteriously slow, check for --enforce-eager in the service file before you blame the hardware. A bug workaround can easily outlive the bug.

Step 2: A better 4-bit quant

The old model was a community build of an abliterated (refusal-removed) Qwen3.8 27B, labelled INT4. Its config.json shows it's actually a mix: the attention layers and the last eight layers' MLPs are 8-bit, and the other MLPs are 4-bit with small groups of 32. I tried a different abliteration quantized as plain W4A16: every linear layer at 4 bits with groups of 128, and 16-bit math, which vLLM runs with its Marlin kernels on Ampere. It's 19.6 GB on Hugging Face against 25.1 GB for the old build.

On the 3090s, decode speed mostly depends on how many bytes each token has to read from VRAM. A smaller model is a faster model:

BF16 (55.6 GB) is nearly 3× larger but only half as fast. Bandwidth isn't the only cost: at this size, the sync between cards after every layer takes up a bigger share of each token. The old mixed 8/4-bit build may well be the more accurate of the two, since it kept attention at 8 bits; I didn't measure that. I'll cover how to pick a quant for your own card in a follow-up post.

Step 3: Speculative decoding

Single-request decode on a 27B model leaves the GPUs' compute mostly idle: each token is one pass that reads every weight once, and the reading is the slow part. Speculative decoding uses that idle compute. A cheap draft proposes several tokens, the big model checks all of them in one pass, and it keeps the ones it agrees with. vLLM's docs describe this as lossless up to floating-point precision: you get the big model's output, faster.

I tried two drafters:

• MTP: Qwen3.8 ships with its own multi-token-prediction head. No extra model to download.
• DFlash2: a separate 3.9 GB draft model (z-lab/Qwen3.8-27B-DFlash2) trained for Qwen3.8 27B. It's a block-diffusion drafter: it predicts a whole block of tokens in a single pass instead of one at a time.

DFlash2 won clearly. Code and JSON gain the most because they're predictable: once a function signature or a JSON key has started, the draft guesses the rest well. Over a 15-minute soak, vLLM's median acceptance length was 4.5 tokens per verification pass, peaking at about 6.

There's a cost, and it's in the last column. Every draft token gets verified, including the rejected ones, so under heavy concurrent load the GPUs do wasted work. MTP drops total throughput at 8 requests from 395 to 201. DFlash2 only drops it to 351, which is why it's worth keeping on. My server mostly handles one or two requests at a time from agents and chat, so single-request speed matters more to me.

I also tried vLLM's option to shrink the draft size as the batch grows, which should give you the best of both. It produced corrupted text (less! about "tricking! mechanism!) and broke tool calls, so I dropped it.

The one-flag trap: --dtype half

My first production deploy of DFlash2 ran at 47 tok/s, slower than no draft at all. The benchmark harness had used --dtype bfloat16. The production service file still had --dtype half from the old model's config. The vLLM log showed:

Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 259.68 tokens/s

The draft model was proposing 260 tokens a second, and every one was rejected. In fp16 the draft and the target disagreed on every token, and nothing logged an error. I didn't dig into why; the model and the drafter are both published in bf16, and running them in their native type fixed it. Switching to bfloat16 brought it back to 130-143 tok/s. If speculative decoding doesn't speed anything up, check the acceptance numbers in the log before blaming the drafter.

Step 4: Prefix caching

Coding agents send the same long system prompt and file context with every request. Without prefix caching vLLM processes it from scratch each time: about 7 seconds for 11k tokens on this rig. With --enable-prefix-caching, the repeated prefix comes from cache:

I measured no cost to decode speed. It had been turned off in the old config; on vLLM 0.32 it works with this hybrid model.

What didn't work

• BF16 + DFlash2 crashed vLLM with an illegal-instruction error, and the kernel logged 64 Xid 13 "Misaligned Register" exceptions on one card. It looks like a software fault: all four cards passed a correctness check afterwards, and the same cards run AWQ + DFlash2 without errors. Since BF16 was half as fast anyway, I didn't chase it.
• FP8. The 3090 has no FP8 tensor cores. vLLM runs FP8 checkpoints on it as weight-only W8A16 through its FP8 Marlin kernel, so there's no faster math, and each token reads twice the bytes of a 4-bit build.
• Moving to two cards. AWQ + MTP on two cards did 98 tok/s on prose against 132 on four. More cards still helps a 27B dense model, even over PCIe.

Power: 250 W is the sweet spot

These cards have been capped at 200 W since four of them at full power tripped a breaker. With the power situation sorted, I swept the caps again, all four cards under sustained BF16 load with fans at 100%:

Decode speed barely moves above 250 W: +7% from 200 to 250 W, then about +1% for each extra 50 W. That fits what I found in my power-limit post: on a 4-bit model split across four cards, each card isn't working hard enough to use more power. Training is different; matrix math keeps scaling. I settled on 250 W and plan to raise the cap only for long training runs.

The TFLOPS column is with gpu-unlock. Its README explains that GeForce cards run FP16, BF16 and TF32 tensor math with FP32 accumulation at half speed; gpu-unlock is a DKMS kernel module, loaded from the initramfs before the NVIDIA driver binds the cards, that lifts that cap. At 200 W it took dense BF16 matrix math on one of my cards from 56 to 74 TFLOPS. The README is clear that it doesn't help decode or 4-bit/8-bit integer math, and my numbers agree: it speeds up training and prompt processing, not generation. RTX 30 support is marked stable, RTX 20 and 40 experimental, so read the README before you install it.

The final config

vllm serve twolven/Qwen3.8-27B-abliterated-AWQ-MTP \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.68 \
  --max-model-len 262144 \
  --dtype bfloat16 \
  --enable-prefix-caching \
  --speculative-config '{"method":"dflash","model":"z-lab/Qwen3.8-27B-DFlash2","num_speculative_tokens":7}' \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_xml \
  --trust-remote-code

--gpu-memory-utilization 0.68 leaves room on each card for other jobs, and the full 262k context still fits because most of Qwen3.8's layers are Gated DeltaNet, which keep a fixed-size state instead of a growing KV cache.

After the switch, I ran a 15-minute soak against the production service: 1,092 mixed requests (concurrent generation, tool calls, image input, thinking and long prompts) with 0 failures, 0 restarts and 0 new GPU errors.

Where this leaves the two boxes

The 3090s are about 5× faster, using about 7-8× the power. The Strix Halo stays always-on for everyday use, and LiteLLM sends requests to the 3090s when they're not training. While a training job holds the GPUs, a small arbiter service stops vLLM, and LiteLLM fails over to the Strix Halo within a few seconds.

Caveats

• Single-request numbers use enable_thinking: false and a short prompt, and are medians of several runs. Your mix of prose and code will land somewhere between the two columns.
• I didn't measure answer quality between the two 4-bit builds beyond a set of correctness checks, tool calls, image input and 8 borderline prompts. Speculative decoding doesn't change outputs, but switching quants can.
• Restarts are slower. Loading the draft model and capturing CUDA graphs takes about 5.5 minutes, against 2.6 before.
• My Strix Halo numbers are lower than gufo's own. Its README quotes up to 71.55 tok/s for Qwen3.8 27B Q4_K_XL with DFlash2, on a short-prompt peak and Unsloth's stock quant. On my prompts, with thinking off and an abliterated build, I measured 27-60 tok/s. Treat them as my workload, not the ceiling.
• This is one model on one rig. The general lessons should carry over: check for eager mode, measure the draft's acceptance rate, and turn on prefix caching for agent workloads. The exact numbers won't.