Benchmarks
Local LLM speed results across models, backends, hardware, and power profiles. Decode tok/s is the headline metric; latency, raw engine runs, and workload context stay visible in their own views.
content/benchmarks/runs/Advanced filters
Power rows are isolated here so normal model rankings are not swamped by intermediate cap sweeps and driver reruns.
| 350M | Q4_K_M | GeForce RTX 3090 · 24 GiBcap 350 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-350w | chat | 940.7 | 9ms |
| 350M | Q4_K_M | GeForce RTX 3090 · 24 GiB450 W maxdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-450w | codegen | 936.3 | 14ms |
| 350M | Q4_K_M | GeForce RTX 3090 · 24 GiB450 W maxdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-450w | agent | 935.5 | 15ms |
| 350M | Q4_K_M | GeForce RTX 3090 · 24 GiBcap 350 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-350w | codegen | 934.6 | 14ms |
| 350M | Q4_K_M | GeForce RTX 3090 · 24 GiBcap 350 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-350w | agent | 927.6 | 17ms |
| 350M | Q4_K_M | GeForce RTX 3090 · 24 GiB450 W maxdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-450w | chat | 925.1 | 8ms |
| 350M | Q4_K_M | GeForce RTX 3090 · 24 GiBcap 350 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-350w | rag | 924.2 | 22ms |
| 350M | Q4_K_M | GeForce RTX 3090 · 24 GiB450 W maxdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-450w | rag | 922.5 | 18ms |
| 1.2B | Q4_K_M | GeForce RTX 3090 · 24 GiB450 W maxdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-450w | chat | 617.7 | 13ms |
| 1.2B-Tool | Q4_K_M | GeForce RTX 3090 · 24 GiB450 W maxdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-450w | chat | 617.3 | 11ms |
| 1.2B | Q4_K_M | GeForce RTX 3090 · 24 GiB450 W maxdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-450w | codegen | 613.1 | 19ms |
| 1.2B-Tool | Q4_K_M | GeForce RTX 3090 · 24 GiB450 W maxdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-450w | codegen | 613.1 | 17ms |
| 1.2B | Q4_K_M | GeForce RTX 3090 · 24 GiB450 W maxdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-450w | rag | 611.6 | 34ms |
| 1.2B-Tool | Q4_K_M | GeForce RTX 3090 · 24 GiB450 W maxdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-450w | rag | 609.8 | 33ms |
| 1.2B | Q4_K_M | GeForce RTX 3090 · 24 GiBcap 350 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-350w | chat | 609.0 | 12ms |
| 1.2B-Tool | Q4_K_M | GeForce RTX 3090 · 24 GiB450 W maxdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-450w | agent | 609.0 | 20ms |
| 1.2B-Tool | Q4_K_M | GeForce RTX 3090 · 24 GiBcap 350 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-350w | chat | 608.6 | 12ms |
| 1.2B | Q4_K_M | GeForce RTX 3090 · 24 GiB450 W maxdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-450w | agent | 607.9 | 20ms |
| 1.2B-Tool | Q4_K_M | GeForce RTX 3090 · 24 GiBcap 350 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-350w | codegen | 605.7 | 17ms |
| 1.2B | Q4_K_M | GeForce RTX 3090 · 24 GiBcap 350 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-350w | codegen | 605.0 | 16ms |
| 1.2B | Q4_K_M | GeForce RTX 3090 · 24 GiBcap 350 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-350w | rag | 602.4 | 47ms |
| 1.2B | Q4_K_M | GeForce RTX 3090 · 24 GiBcap 350 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-350w | agent | 601.7 | 23ms |
| 1.2B-Tool | Q4_K_M | GeForce RTX 3090 · 24 GiBcap 350 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-350w | rag | 601.0 | 34ms |
| 1.2B-Tool | Q4_K_M | GeForce RTX 3090 · 24 GiBcap 350 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-350w | agent | 600.6 | 19ms |
| 8B-A1B | Q4_K_M | GeForce RTX 3090 · 24 GiB450 W maxdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-450w | chat | 439.9 | 21ms |
| 8B-A1B | Q4_K_M | GeForce RTX 3090 · 24 GiB450 W maxdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-450w | codegen | 438.0 | 34ms |
| 8B-A1B | Q4_K_M | GeForce RTX 3090 · 24 GiB450 W maxdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-450w | rag | 436.7 | 52ms |
| 8B-A1B | Q4_K_M | GeForce RTX 3090 · 24 GiB450 W maxdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-450w | agent | 435.2 | 22ms |
| 8B-A1B | Q4_K_M | GeForce RTX 3090 · 24 GiBcap 350 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-350w | chat | 433.7 | 21ms |
| 8B-A1B | Q4_K_M | GeForce RTX 3090 · 24 GiBcap 350 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-350w | codegen | 430.3 | 33ms |
| 8B-A1B | Q4_K_M | GeForce RTX 3090 · 24 GiBcap 350 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-350w | agent | 428.3 | 23ms |
| 8B-A1B | Q4_K_M | GeForce RTX 3090 · 24 GiBcap 350 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-350w | rag | 425.0 | 53ms |
| 2.6B | Q4_K_M | GeForce RTX 3090 · 24 GiB450 W maxdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-450w | chat | 332.6 | 17ms |
| 2.6B | Q4_K_M | GeForce RTX 3090 · 24 GiB450 W maxdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-450w | rag | 329.2 | 55ms |
| 2.6B | Q4_K_M | GeForce RTX 3090 · 24 GiB450 W maxdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-450w | codegen | 328.2 | 32ms |
| 2.6B | Q4_K_M | GeForce RTX 3090 · 24 GiB450 W maxdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-450w | agent | 326.6 | 28ms |
| 2.6B | Q4_K_M | GeForce RTX 3090 · 24 GiBcap 350 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-350w | chat | 324.9 | 18ms |
| 2.6B | Q4_K_M | GeForce RTX 3090 · 24 GiBcap 350 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-350w | rag | 322.4 | 57ms |
| 2.6B | Q4_K_M | GeForce RTX 3090 · 24 GiBcap 350 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-350w | codegen | 321.9 | 33ms |
| 2.6B | Q4_K_M | GeForce RTX 3090 · 24 GiBcap 350 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-350w | agent | 320.5 | 42ms |
| E2B-it | Q4_K_M | GeForce RTX 3090 · 24 GiBcap 350 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-350w | chat | 226.4 | 41ms |
| E2B-it | Q4_K_M | GeForce RTX 3090 · 24 GiB450 W maxdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-450w | chat | 226.2 | 40ms |
| E2B-it | Q4_K_M | GeForce RTX 3090 · 24 GiBcap 350 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-350w | rag | 224.5 | 83ms |
| E2B-it | Q4_K_M | GeForce RTX 3090 · 24 GiBcap 350 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-350w | codegen | 224.3 | 56ms |
| E2B-it | Q4_K_M | GeForce RTX 3090 · 24 GiB450 W maxdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-450w | codegen | 223.7 | 54ms |
| E2B-it | Q4_K_M | GeForce RTX 3090 · 24 GiBcap 350 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-350w | agent | 223.3 | 51ms |
| E2B-it | Q4_K_M | GeForce RTX 3090 · 24 GiB450 W maxdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-450w | agent | 222.5 | 49ms |
| E2B-it | Q4_K_M | GeForce RTX 3090 · 24 GiB450 W maxdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-450w | rag | 222.2 | 82ms |
| 30B-A3B-Instructthink | Q4_K_XL | GeForce RTX 3090 · 24 GiB450 W maxdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-450w | chat | 203.0 | 86ms |
| 30B-A3B-Instructthink | Q4_K_XL | GeForce RTX 3090 · 24 GiBcap 350 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-350w | chat | 201.7 | 89ms |
| 30B-A3B-Instructthink | Q4_K_XL | GeForce RTX 3090 · 24 GiB450 W maxdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-450w | codegen | 199.8 | 125ms |
| 30B-A3B-Instructthink | Q4_K_XL | GeForce RTX 3090 · 24 GiB450 W maxdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-450w | rag | 198.9 | 205ms |
| 30B-A3B-Instructthink | Q4_K_XL | GeForce RTX 3090 · 24 GiB450 W maxdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-450w | agent | 198.6 | 157ms |
| 30B-A3B-Instructthink | Q4_K_XL | GeForce RTX 3090 · 24 GiBcap 350 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-350w | codegen | 197.8 | 120ms |
| 30B-A3B-Instructthink | Q4_K_XL | GeForce RTX 3090 · 24 GiBcap 350 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-350w | rag | 196.9 | 236ms |
| 30B-A3B-Instructthink | Q4_K_XL | GeForce RTX 3090 · 24 GiBcap 350 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-350w | agent | 196.0 | 164ms |
| 4b-it | Q4_K_M | GeForce RTX 3090 · 24 GiB450 W maxdrv 595 | llama.cpp cuda-3e12fbd (cuda) | baseline-pl-450w-595-r3 | chat | 186.7 | 36ms |
| 4b-it | Q4_K_M | GeForce RTX 3090 · 24 GiB450 W maxdrv 595 | llama.cpp cuda-3e12fbd (cuda) | baseline-pl-450w-595 | chat | 186.2 | 37ms |
| 4b-it | Q4_K_M | GeForce RTX 3090 · 24 GiBcap 430 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-430w | chat | 185.2 | 37ms |
| 4b-it | Q4_K_M | GeForce RTX 3090 · 24 GiBcap 410 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-410w | chat | 185.1 | 38ms |
| 4b-it | Q4_K_M | GeForce RTX 3090 · 24 GiB450 W maxdrv 595 | llama.cpp cuda-3e12fbd (cuda) | baseline-pl-450w-595-r2 | chat | 185.0 | 26ms |
| 4b-it | Q4_K_M | GeForce RTX 3090 · 24 GiBcap 420 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-420w | chat | 184.9 | 37ms |
| 4b-it | Q4_K_M | GeForce RTX 3090 · 24 GiB450 W maxdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-450w | chat | 184.7 | 38ms |
| 4b-it | Q4_K_M | GeForce RTX 3090 · 24 GiBcap 400 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-400w | chat | 184.6 | 37ms |
| 4b-it | Q4_K_M | GeForce RTX 3090 · 24 GiBcap 440 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-440w | chat | 184.6 | 39ms |
| 4b-it | Q4_K_M | GeForce RTX 3090 · 24 GiBcap 380 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-380w | chat | 184.0 | 37ms |
| 30B-A3B-Reasoningthink | Q4_K_M | GeForce RTX 3090 · 24 GiB450 W maxdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-450w | chat | 183.9 | 159ms |
| 4b-it | Q4_K_M | GeForce RTX 3090 · 24 GiBcap 350 Wdrv 595 | llama.cpp cuda-3e12fbd (cuda) | baseline-pl-350w-595-r2 | chat | 183.4 | 29ms |
| 30B-A3B-Reasoningthink | Q4_K_M | GeForce RTX 3090 · 24 GiB450 W maxdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-450w | rag | 183.4 | 291ms |
| 4b-it | Q4_K_M | GeForce RTX 3090 · 24 GiBcap 370 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-370w | chat | 183.3 | 37ms |
| 4b-it | Q4_K_M | GeForce RTX 3090 · 24 GiBcap 390 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-390w | chat | 183.3 | 37ms |
| 4b-it | Q4_K_M | GeForce RTX 3090 · 24 GiBcap 360 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-360w | chat | 183.2 | 39ms |
| 4b-it | Q4_K_M | GeForce RTX 3090 · 24 GiB450 W maxdrv 595 | llama.cpp cuda-3e12fbd (cuda) | baseline-pl-450w-595 | codegen | 183.1 | 132ms |
| 30B-A3B-Reasoningthink | Q4_K_M | GeForce RTX 3090 · 24 GiB450 W maxdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-450w | codegen | 183.1 | 191ms |
| 4b-it | Q4_K_M | GeForce RTX 3090 · 24 GiB450 W maxdrv 595 | llama.cpp cuda-3e12fbd (cuda) | baseline-pl-450w-595-r3 | codegen | 182.6 | 119ms |
| 30B-A3B-Reasoningthink | Q4_K_M | GeForce RTX 3090 · 24 GiB450 W maxdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-450w | agent | 182.5 | 208ms |
| 4b-it | Q4_K_M | GeForce RTX 3090 · 24 GiBcap 350 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-350w | chat | 182.2 | 38ms |
| 4b-it | Q4_K_M | GeForce RTX 3090 · 24 GiBcap 340 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-340w | chat | 181.9 | 37ms |
| 4b-it | Q4_K_M | GeForce RTX 3090 · 24 GiB450 W maxdrv 595 | llama.cpp cuda-3e12fbd (cuda) | baseline-pl-450w-595-r3 | rag | 181.9 | 339ms |
| 30B-A3B-Reasoningthink | Q4_K_M | GeForce RTX 3090 · 24 GiBcap 350 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-350w | chat | 181.8 | 145ms |
| 4b-it | Q4_K_M | GeForce RTX 3090 · 24 GiB450 W maxdrv 595 | llama.cpp cuda-3e12fbd (cuda) | baseline-pl-450w-595 | rag | 181.6 | 392ms |
| 4b-it | Q4_K_M | GeForce RTX 3090 · 24 GiB450 W maxdrv 595 | llama.cpp cuda-3e12fbd (cuda) | baseline-pl-450w-595-r2 | codegen | 181.5 | 29ms |
| 4b-it | Q4_K_M | GeForce RTX 3090 · 24 GiB450 W maxdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-450w | codegen | 181.3 | 129ms |
| 4b-it | Q4_K_M | GeForce RTX 3090 · 24 GiB450 W maxdrv 595 | llama.cpp cuda-3e12fbd (cuda) | baseline-pl-450w-595 | agent | 181.3 | 227ms |
| 4b-it | Q4_K_M | GeForce RTX 3090 · 24 GiBcap 430 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-430w | codegen | 181.2 | 135ms |
| 4b-it | Q4_K_M | GeForce RTX 3090 · 24 GiBcap 420 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-420w | codegen | 181.2 | 143ms |
| 4b-it | Q4_K_M | GeForce RTX 3090 · 24 GiBcap 440 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-440w | codegen | 181.1 | 121ms |
| 30B-A3B-Reasoningthink | Q4_K_M | GeForce RTX 3090 · 24 GiBcap 350 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-350w | rag | 181.0 | 328ms |
| 4b-it | Q4_K_M | GeForce RTX 3090 · 24 GiBcap 410 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-410w | codegen | 181.0 | 120ms |
| 4b-it | Q4_K_M | GeForce RTX 3090 · 24 GiBcap 330 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-330w | chat | 180.8 | 39ms |
| 30B-A3B-Reasoningthink | Q4_K_M | GeForce RTX 3090 · 24 GiBcap 350 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-350w | codegen | 180.8 | 204ms |
| 4b-it | Q4_K_M | GeForce RTX 3090 · 24 GiBcap 400 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-400w | codegen | 180.8 | 125ms |
| 4b-it | Q4_K_M | GeForce RTX 3090 · 24 GiB450 W maxdrv 595 | llama.cpp cuda-3e12fbd (cuda) | baseline-pl-450w-595-r3 | agent | 180.8 | 228ms |
| 4b-it | Q4_K_M | GeForce RTX 3090 · 24 GiB450 W maxdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-450w | rag | 180.6 | 339ms |
| 4b-it | Q4_K_M | GeForce RTX 3090 · 24 GiBcap 380 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-380w | codegen | 180.4 | 130ms |
| 4b-it | Q4_K_M | GeForce RTX 3090 · 24 GiBcap 420 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-420w | rag | 180.3 | 424ms |
| 30B-A3B-Reasoningthink | Q4_K_M | GeForce RTX 3090 · 24 GiBcap 350 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-350w | agent | 180.2 | 189ms |
| 4b-it | Q4_K_M | GeForce RTX 3090 · 24 GiB450 W maxdrv 595 | llama.cpp cuda-3e12fbd (cuda) | baseline-pl-450w-595-r2 | rag | 180.2 | 231ms |
| 4b-it | Q4_K_M | GeForce RTX 3090 · 24 GiBcap 370 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-370w | codegen | 180.2 | 129ms |
| 4b-it | Q4_K_M | GeForce RTX 3090 · 24 GiB450 W maxdrv 595 | llama.cpp cuda-3e12fbd (cuda) | baseline-pl-450w-595-r2 | agent | 180.1 | 203ms |
Use these rows for GPU-to-GPU comparisons when the model, quant, backend, driver family, power policy, and benchmark shape match closely.
Use these rows to compare a similar software stack. They are useful, but backend, server path, driver, cache, or power settings may still influence the number.
Treat these as real workload measurements, not pure hardware rankings. They include prompt mix, API/server overhead, cache behavior, and local software details.