Benchmarks
Local LLM speed results across models, backends, hardware, and power profiles. Decode tok/s is the headline metric; latency, raw engine runs, and workload context stay visible in their own views.
1181 source rows568 matching source rowslatest run May 21, 2026schemas v1-v4source
content/benchmarks/runs/Filters
Advanced filters
One row per model, hardware, backend, quant, and normalized mode. Decode tok/s is shown as a range across workload shapes at the selected concurrency.
Showing top 50 of 81 grouped leaderboard rows from 568 matching source rows.
Notes:hardware comparablematched GPU comparisonstack comparablesimilar stackstack realisticreal workload rowlegacyolder harness350 W caprecorded power limitdrv 590driver branch
| Model | Quant | Hardware | Backend / Mode | Decode tok/s | TTFT | Notes |
|---|---|---|---|---|---|---|
| LFM2.5 350M 4 workload shapes | Q4_K_M | GeForce RTX 3090 · 24 GiB | llama.cppbaseline | 821.0-940.7 | 14ms | legacystack comparable350 W capdrv 590 |
| LFM2.5-350M 4 workload shapes | Q4_K_M | GeForce RTX 5070 · 12 GiB | llama.cppbaseline | 843.9-905.0 | 7ms | legacystack comparable250 W capdrv 595 |
| LFM2 1.2B 4 workload shapes | Q4_K_M | GeForce RTX 3090 · 24 GiB | llama.cppbaseline | 468.4-617.7 | 21ms | legacystack comparabledrv 590 |
| LFM2 1.2B-Tool 4 workload shapes | Q4_K_M | GeForce RTX 3090 · 24 GiB | llama.cppbaseline | 461.0-617.3 | 20ms | legacystack comparabledrv 590 |
| LFM2 1.2B 4 workload shapes | Q4_K_M | GeForce RTX 5070 · 12 GiB | llama.cppbaseline | 517.9-539.1 | 12ms | legacystack comparable250 W capdrv 595 |
| LFM2 1.2B-Tool 4 workload shapes | Q4_K_M | GeForce RTX 5070 · 12 GiB | llama.cppbaseline | 517.6-534.8 | 11ms | legacystack comparable250 W capdrv 595 |
| LFM2 8B-A1B 4 workload shapes | Q4_K_M | GeForce RTX 3090 · 24 GiB | llama.cppbaseline | 333.6-439.9 | 34ms | legacystack comparabledrv 590 |
| LFM2 8B-A1B 4 workload shapes | Q4_K_M | GeForce RTX 5070 · 12 GiB | llama.cppbaseline | 362.1-372.3 | 30ms | legacystack comparable250 W capdrv 595 |
| LFM2 2.6B 4 workload shapes | Q4_K_M | GeForce RTX 3090 · 24 GiB | llama.cppbaseline | 234.4-332.6 | 34ms | legacystack comparabledrv 590 |
| LFM2 2.6B 4 workload shapes | Q4_K_M | GeForce RTX 5070 · 12 GiB | llama.cppbaseline | 266.8-273.7 | 21ms | legacystack comparable250 W capdrv 595 |
| Gemma-4 E2B-it 4 workload shapes | Q4_K_M | GeForce RTX 3090 · 24 GiB | llama.cppbaseline | 195.1-226.4 | 55ms | legacystack comparable350 W capdrv 590 |
| Gemma-4 E2B-it 4 workload shapes | Q4_K_M | GeForce RTX 5070 · 12 GiB | llama.cppbaseline | 212.6-220.3 | 36ms | legacystack comparable250 W capdrv 595 |
| LFM2 1.2B 4 workload shapes | Q4_K_M | Strix Halo · Radeon 8060S · 128 GiB unified (96 GiB VRAM) | llama.cppbaseline | 199.2-211.5 | 21ms | legacystack comparable |
| Qwen3-Coder 30B-A3B-Instruct 4 workload shapes | Q4_K_XL | GeForce RTX 3090 · 24 GiB | llama.cppbaseline | 154.9-203.0 | 159ms | legacystack comparabledrv 590reasoning |
| Gemma-3 4b-it 4 workload shapes | Q4_K_M | GeForce RTX 3090 · 24 GiB | llama.cppbaseline | 127.9-184.7 | 177ms | legacystack comparabledrv 590 |
| NVIDIA-Nemotron-3-Nano-Omni 30B-A3B-Reasoning 4 workload shapes | Q4_K_M | GeForce RTX 3090 · 24 GiB | llama.cppbaseline | 142.3-183.9 | 206ms | legacystack comparabledrv 590reasoning |
| Gemma-3 4b-it 4 workload shapes | Q4_K_M | GeForce RTX 5070 · 12 GiB | llama.cppbaseline | 168.4-172.6 | 48ms | legacystack comparable250 W capdrv 595 |
| Qwen3.6 35B-A3B-MTP 4 workload shapes | Q4_K_M | GeForce RTX 3090 · 24 GiB | llama.cppMTP n=2 | 122.1-169.0 | 201ms | legacystack comparabledrv 590reasoning |
| Gemma-3 4b-it 4 workload shapes | Q4_K_M | GeForce RTX 5070 · 12 GiB | llama.cppbaseline | 156.0-168.8 | 57ms | legacystack comparable250 W capdrv 595 |
| Qwen3.6 35B-A3B-MTP 4 workload shapes | Q4_K_M | GeForce RTX 3090 · 24 GiB | llama.cppMTP n=3 | 136.6-161.4 | 208ms | legacystack comparabledrv 590reasoning |
| LFM2 8B-A1B 4 workload shapes | Q4_K_M | Strix Halo · Radeon 8060S · 128 GiB unified (96 GiB VRAM) | llama.cppbaseline | 149.2-154.8 | 37ms | legacystack comparable |
| Qwen2.5-Coder 7B-Instruct 4 workload shapes | Q4_K_M | GeForce RTX 3090 · 24 GiB | llama.cppbaseline | 81.8-151.9 | 62ms | legacystack comparabledrv 590 |
| Qwen2.5 7B-Instruct 4 workload shapes | AWQ | GeForce RTX 3090 · 24 GiB | vLLMbaseline | 77.2-151.3 | 27ms | legacystack comparable420 W capdrv 590 |
| Qwen2.5-Coder 7B-Instruct 4 workload shapes | AWQ | GeForce RTX 3090 · 24 GiB | vLLMbaseline | 77.2-150.9 | 27ms | legacystack comparable420 W capdrv 590 |
| Qwen3.6 35B-A3B-MTP 4 workload shapes | Q4_K_M | GeForce RTX 3090 · 24 GiB | llama.cppbaseline | 148.1-148.6 | 188ms | legacystack comparabledrv 590reasoning |
| Qwen3.5 35B-A3B 4 workload shapes | Q4_K_M | GeForce RTX 3090 · 24 GiB | llama.cppbaseline | 146.0-148.5 | 198ms | legacystack comparabledrv 590reasoning |
| Gemma-4 E4B-it 4 workload shapes | Q4_K_M | GeForce RTX 3090 · 24 GiB | llama.cppbaseline | 116.2-148.2 | 119ms | legacystack comparabledrv 590 |
| Gemma-4 E4B-it 4 workload shapes | Q4_K_M | GeForce RTX 5070 · 12 GiB | llama.cppbaseline | 124.8-129.8 | 63ms | legacystack comparable250 W capdrv 595 |
| granite-4.1 8b 4 workload shapes | Q4_K_M | GeForce RTX 3090 · 24 GiB | llama.cppbaseline | 71.5-127.0 | 147ms | legacystack comparabledrv 590 |
| Qwen3.5 35B-A3B 4 workload shapes | Q4_K_XL | GeForce RTX 3090 · 24 GiB | llama.cppbaseline | 122.7-126.6 | 325ms | legacystack comparable200 W capdrv 590reasoning |
| GLM-4.7-Flash 4 workload shapes | Q4_K_XL | GeForce RTX 3090 · 24 GiB | llama.cppbaseline | 114.5-123.7 | 161ms | legacystack comparable200 W capdrv 590 |
| Qwen2.5-Coder 7B-Instruct 4 workload shapes | Q4_K_M | GeForce RTX 5070 · 12 GiB | llama.cppbaseline | 117.6-120.7 | 59ms | legacystack comparable250 W capdrv 595 |
| Gemma-4 26B-A4B-it 4 workload shapes | Q4_K_M | GeForce RTX 3090 · 24 GiB | llama.cppbaseline | 116.3-119.5 | 330ms | legacystack comparable300 W capdrv 590 |
| granite-4.1 8b 4 workload shapes | Q4_K_M | GeForce RTX 5070 · 12 GiB | llama.cppbaseline | 96.0-100.2 | 75ms | legacystack comparable250 W capdrv 595 |
| Gemma-4 E2B-it 4 workload shapes | Q4_K_M | Strix Halo · Radeon 8060S · 128 GiB unified (96 GiB VRAM) | llama.cppbaseline | 85.7-89.4 | 102ms | legacystack comparable |
| Qwen2.5-Coder 14B-Instruct 4 workload shapes | AWQ | GeForce RTX 3090 · 24 GiB | vLLMbaseline | 40.7-82.5 | 40ms | legacystack comparable420 W capdrv 590 |
| Qwen2.5 14B-Instruct 4 workload shapes | AWQ | GeForce RTX 3090 · 24 GiB | vLLMbaseline | 40.1-82.3 | 40ms | legacystack comparable420 W capdrv 590 |
| Qwen3.6 35B-A3B-MTP 4 workload shapes | Q4_K_M | Strix Halo · Radeon 8060S · 128 GiB unified (96 GiB VRAM) | llama.cppMTP n=3 | 63.4-70.6 | 192ms | legacystack comparablereasoning |
| Qwen3.6 35B-A3B-MTP 4 workload shapes | Q4_K_M | Strix Halo · Radeon 8060S · 128 GiB unified (96 GiB VRAM) | llama.cppMTP n=2 | 60.8-70.0 | 182ms | legacystack comparablereasoning |
| Gemma-3 4b-it 4 workload shapes | Q4_K_M | Strix Halo · Radeon 8060S · 128 GiB unified (96 GiB VRAM) | llama.cppbaseline | 63.7-66.3 | 212ms | legacystack comparable |
| Qwen3.6 27B-MTP 4 workload shapes | Q4_K_M | GeForce RTX 3090 · 24 GiB | llama.cppMTP n=2 | 29.8-63.7 | 451ms | legacystack comparabledrv 590reasoning |
| Qwen3.6 27B-MTP 4 workload shapes | Q4_K_M | GeForce RTX 3090 · 24 GiB | llama.cppMTP n=3 | 31.1-59.2 | 460ms | legacystack comparabledrv 590reasoning |
| Qwen3.6 27B-MTP 4 workload shapes | Q8_0 | 2× GeForce RTX 3090 · 24 GiB each | llama.cppMTP n=3 | 45.1-57.1 | 443ms | legacystack comparabledrv 590reasoning |
| Gemma-4 E4B-it 4 workload shapes | Q4_K_M | Strix Halo · Radeon 8060S · 128 GiB unified (96 GiB VRAM) | llama.cppbaseline | 53.2-55.3 | 258ms | legacystack comparable |
| Gemma-4 E4B-it 4 workload shapes | Q4_K_M | Strix Halo · Radeon 8060S · 128 GiB unified (96 GiB VRAM) | llama.cppbaseline | 52.1-53.8 | 273ms | legacystack comparable |
| Qwen3.6 27B-MTP 4 workload shapes | Q8_0 | 2× GeForce RTX 3090 · 24 GiB each | llama.cppMTP n=2 | 42.5-53.2 | 442ms | legacystack comparabledrv 590reasoning |
| Qwen3.6 35B-A3B-MTP 4 workload shapes | Q4_K_M | Strix Halo · Radeon 8060S · 128 GiB unified (96 GiB VRAM) | llama.cppbaseline | 52.5-52.9 | 164ms | legacystack comparablereasoning |
| Gemma-4 26B-A4B-it 4 workload shapes | Q4_K_M | Strix Halo · Radeon 8060S · 128 GiB unified (96 GiB VRAM) | llama.cppbaseline | 47.2-52.0 | 443ms | legacystack comparable |
| Qwen3.5 35B-A3B 4 workload shapes | Q4_K_XL | Strix Halo · Radeon 8060S · 128 GiB unified (96 GiB VRAM) | llama.cppbaseline | 48.8-48.9 | 414ms | legacystack comparablereasoning |
| Gemma-4 26B-A4B-it 4 workload shapes | Q4_K_M | Strix Halo · Radeon 8060S · 128 GiB unified (96 GiB VRAM) | llama.cppbaseline | 45.1-47.7 | 444ms | legacystack comparable |
Use filters to narrow this view or open Explorer for row-level data.
Decode tok/s
Headline speed metric
TTFT / TPOT
Latency context
Raw vs workload
Separate comparison contracts
Notes badge key
hardware comparable
Use these rows for GPU-to-GPU comparisons when the model, quant, backend, driver family, power policy, and benchmark shape match closely.
stack comparable
Use these rows to compare a similar software stack. They are useful, but backend, server path, driver, cache, or power settings may still influence the number.
stack realistic
Treat these as real workload measurements, not pure hardware rankings. They include prompt mix, API/server overhead, cache behavior, and local software details.
legacyOlder workload harness row.
350 W capRecorded GPU power limit.
drv 590GPU driver branch.
reasoningReasoning-token model.
Metric guide
Decode tok/s - Generation rate. Raw rows come from the engine benchmark; API rows use token intervals when available.
TTFT - Time to first token. This includes prompt processing and server/API overhead.
TPOT / ITL - Time per output token after the first token. Lower is better.
Raw Engine - llama-bench style cases intended for hardware-normalized comparison across rigs.
Workload / API - Stack-realistic measurements that include backend, server, cache, driver, and prompt behavior.
Power badges - A cap badge shows the recorded power limit. The row metadata records the cap relative to the recorded max.