Benchmarks
Local LLM speed results across models, backends, hardware, and power profiles. Decode tok/s is the headline metric; latency, raw engine runs, and workload context stay visible in their own views.
content/benchmarks/runs/Advanced filters
Hardware tested
Rig metadata and microbenchmarks are shown here so memory bandwidth and tensor math do not get mixed into model-serving rankings.
| cap | theory | copy | fp16 | bf16 |
|---|---|---|---|---|
| 200 W | 672 GB/s | 271 GB/s | 67.9 TF | 68.4 TF |
| 250 W | 672 GB/s | 271 GB/s | 69.5 TF | 68.2 TF |
| 300 W | 672 GB/s | 270 GB/s | 69.6 TF | 68.4 TF |
A daily-driver gaming PC on CachyOS with an RTX 5070, pressed into service as a benchmark host between gaming sessions. The card sits at a 250 W of 300 W stock power cap (83%) by default on this rig; that limit is captured in the YAML and surfaced on each run.
Inference uses the prebuilt llama.cpp Vulkan binary (no CUDA toolkit or sudo on this host), so all RTX 5070 numbers here are Vulkan-backed rather than CUDA. That makes them directly comparable to the Strix Halo Vulkan numbers (same backend, different silicon) but understates what the card can do with CUDA. A CUDA pass will land later.
- GPU: NVIDIA GeForce RTX 5070, 12 GiB GDDR7, 250 W cap (300 W max)
- CPU: AMD Ryzen 9 7900 (12-core); the integrated Radeon iGPU is also visible to Vulkan as a second device but explicitly excluded from every bench via
--device Vulkan0 --split-mode none --main-gpu 0 - Driver: 595.58.03
- OS: CachyOS rolling, Linux 7.0
- VRAM-fit verification: every run snapshots GPU memory before and after the benchmark process starts and aborts if the delta is smaller than the model file size, which guards against silent CPU spill
Best workload row per rig
| Rig | Best workload row | Decode tok/s | Backend / mode |
|---|---|---|---|
| GeForce RTX 5070 · 12 GiB | LFM2.5-350M · rag | 905.0 | llama.cpp baseline |
Use these rows for GPU-to-GPU comparisons when the model, quant, backend, driver family, power policy, and benchmark shape match closely.
Use these rows to compare a similar software stack. They are useful, but backend, server path, driver, cache, or power settings may still influence the number.
Treat these as real workload measurements, not pure hardware rankings. They include prompt mix, API/server overhead, cache behavior, and local software details.