Benchmarks
Local LLM speed results across models, backends, hardware, and power profiles. Decode tok/s is the headline metric; latency, raw engine runs, and workload context stay visible in their own views.
1181 source rows1181 matching source rowslatest run May 21, 2026schemas v1-v4source
content/benchmarks/runs/Filters
Advanced filters
Full row-level explorer. This is the place for raw shapes, hardware probes, cache/ppfix rows, dense power caps, and reruns.
| 4b-it | GGUF | raw | hardware comparable | GeForce RTX 5070 · 12 GiBcap 250 Wdrv 595 | llama.cpp llama.cpp-1a68ec9 (cuda) | raw-v4-r2 | mixed_4096_256 | 1 | 177.7 | — | — | — | — | — |
| 4b-it | GGUF | raw | hardware comparable | GeForce RTX 5070 · 12 GiBcap 250 Wdrv 595 | llama.cpp llama.cpp-1a68ec9 (cuda) | raw-v4-r2 | mixed_2048_256 | 1 | 177.7 | — | — | — | — | — |
| 4b-it | GGUF | raw | hardware comparable | GeForce RTX 5070 · 12 GiBcap 250 Wdrv 595 | llama.cpp llama.cpp-1a68ec9 (cuda) | raw-v4-r2 | tg_128 | 1 | 177.1 | — | — | — | — | — |
| 4b-it | GGUF | raw | hardware comparable | GeForce RTX 5070 · 12 GiBcap 250 Wdrv 595 | llama.cpp llama.cpp-1a68ec9 (cuda) | raw-v4-r2 | tg_512 | 1 | 176.5 | — | — | — | — | — |
| 4b-it | GGUF | raw | hardware comparable | GeForce RTX 5070 · 12 GiBcap 250 Wdrv 595 | llama.cpp llama.cpp-1a68ec9 (cuda) | raw-v4-r2 | mixed_1024_16 | 1 | 176.2 | — | — | — | — | — |
| 4b-it | GGUF | raw | hardware comparable | GeForce RTX 5070 · 12 GiBcap 250 Wdrv 595 | llama.cpp llama.cpp-1a68ec9 (cuda) | raw-v4-r2 | mixed_2048_768 | 1 | 175.9 | — | — | — | — | — |
| 4b-it | GGUF | raw | hardware comparable | GeForce RTX 5070 · 12 GiBcap 250 Wdrv 595 | llama.cpp llama.cpp-b9174 (vulkan) | raw-v4-r2 | mixed_1024_16 | 1 | 175.5 | — | — | — | — | — |
| 4b-it | GGUF | raw | hardware comparable | GeForce RTX 5070 · 12 GiBcap 250 Wdrv 595 | llama.cpp llama.cpp-1a68ec9 (cuda) | raw-v4-r2 | mixed_1024_1024 | 1 | 175.1 | — | — | — | — | — |
| 4b-it | GGUF | raw | hardware comparable | GeForce RTX 5070 · 12 GiBcap 250 Wdrv 595 | llama.cpp llama.cpp-1a68ec9 (cuda) | raw-v4-r2 | mixed_64_1024 | 1 | 175.1 | — | — | — | — | — |
| 4b-it | GGUF | raw | hardware comparable | GeForce RTX 5070 · 12 GiBcap 250 Wdrv 595 | llama.cpp llama.cpp-1a68ec9 (cuda) | raw-v4-r2 | tg_1024 | 1 | 175.0 | — | — | — | — | — |
| 4b-it | GGUF | raw | hardware comparable | GeForce RTX 5070 · 12 GiBcap 250 Wdrv 595 | llama.cpp llama.cpp-1a68ec9 (cuda) | raw-v4-r2 | mixed_384_1152 | 1 | 174.7 | — | — | — | — | — |
| 4b-it | GGUF | raw | hardware comparable | GeForce RTX 5070 · 12 GiBcap 250 Wdrv 595 | llama.cpp llama.cpp-b9174 (vulkan) | raw-v4-r2 | tg_128 | 1 | 174.1 | — | — | — | — | — |
| 4b-it | GGUF | raw | hardware comparable | GeForce RTX 5070 · 12 GiBcap 250 Wdrv 595 | llama.cpp llama.cpp-1a68ec9 (cuda) | raw-v4-r2 | mixed_16_1536 | 1 | 173.6 | — | — | — | — | — |
| 4b-it | GGUF | raw | hardware comparable | GeForce RTX 5070 · 12 GiBcap 250 Wdrv 595 | llama.cpp llama.cpp-b9174 (vulkan) | raw-v4-r2 | mixed_2048_256 | 1 | 172.8 | — | — | — | — | — |
| 4b-it | GGUF | raw | hardware comparable | GeForce RTX 5070 · 12 GiBcap 250 Wdrv 595 | llama.cpp llama.cpp-b9174 (vulkan) | raw-v4-r2 | mixed_4096_256 | 1 | 172.6 | — | — | — | — | — |
| 4b-it | GGUF | raw | hardware comparable | GeForce RTX 5070 · 12 GiBcap 250 Wdrv 595 | llama.cpp llama.cpp-b9174 (vulkan) | raw-v4-r2 | tg_512 | 1 | 171.6 | — | — | — | — | — |
| 4b-it | GGUF | raw | hardware comparable | GeForce RTX 5070 · 12 GiBcap 250 Wdrv 595 | llama.cpp llama.cpp-1a68ec9 (cuda) | raw-v4-r2 | mixed_1280_3072 | 1 | 171.1 | — | — | — | — | — |
| 4b-it | GGUF | raw | hardware comparable | GeForce RTX 5070 · 12 GiBcap 250 Wdrv 595 | llama.cpp llama.cpp-b9174 (vulkan) | raw-v4-r2 | mixed_2048_768 | 1 | 169.4 | — | — | — | — | — |
| 4b-it | GGUF | raw | hardware comparable | GeForce RTX 5070 · 12 GiBcap 250 Wdrv 595 | llama.cpp llama.cpp-b9174 (vulkan) | raw-v4-r2 | tg_1024 | 1 | 169.3 | — | — | — | — | — |
| 4b-it | GGUF | raw | hardware comparable | GeForce RTX 5070 · 12 GiBcap 250 Wdrv 595 | llama.cpp llama.cpp-b9174 (vulkan) | raw-v4-r2 | mixed_64_1024 | 1 | 168.3 | — | — | — | — | — |
| 4b-it | GGUF | raw | hardware comparable | GeForce RTX 5070 · 12 GiBcap 250 Wdrv 595 | llama.cpp llama.cpp-b9174 (vulkan) | raw-v4-r2 | mixed_1024_1024 | 1 | 167.8 | — | — | — | — | — |
| 4b-it | GGUF | raw | hardware comparable | GeForce RTX 5070 · 12 GiBcap 250 Wdrv 595 | llama.cpp llama.cpp-b9174 (vulkan) | raw-v4-r2 | mixed_384_1152 | 1 | 167.3 | — | — | — | — | — |
| 4b-it | GGUF | raw | hardware comparable | GeForce RTX 5070 · 12 GiBcap 250 Wdrv 595 | llama.cpp llama.cpp-b9174 (vulkan) | raw-v4-r2 | mixed_16_1536 | 1 | 166.1 | — | — | — | — | — |
| 4b-it | GGUF | raw | hardware comparable | GeForce RTX 5070 · 12 GiBcap 250 Wdrv 595 | llama.cpp llama.cpp-b9174 (vulkan) | raw-v4-r2 | mixed_1280_3072 | 1 | 163.7 | — | — | — | — | — |
| 8b | Q4_K_M | legacy | stack comparable | GeForce RTX 3090 · 24 GiB450 W maxdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-450w | chat | 1 | 127.0 | 42ms | 7.9 | — | — | — |
| 8b | Q4_K_M | legacy | stack comparable | GeForce RTX 3090 · 24 GiB450 W maxdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-450w | codegen | 1 | 124.8 | 121ms | 8.0 | — | — | — |
| 8b | Q4_K_M | legacy | stack comparable | GeForce RTX 3090 · 24 GiB450 W maxdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-450w | rag | 1 | 124.4 | 223ms | 8.0 | — | — | — |
| 8b | Q4_K_M | legacy | stack comparable | GeForce RTX 3090 · 24 GiBcap 350 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-350w | chat | 1 | 124.1 | 45ms | 8.1 | — | — | — |
| 8b | Q4_K_M | legacy | stack comparable | GeForce RTX 3090 · 24 GiB450 W maxdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-450w | agent | 1 | 123.1 | 189ms | 8.1 | — | — | — |
| 4b-it | GGUF | raw | hardware comparable | GeForce RTX 3090 · 24 GiBcap 200 Wdrv 595 | llama.cpp llama.cpp-3e12fbd (cuda) | raw-v4-r2 | tg_128 | 1 | 122.9 | — | — | — | — | — |
| 8b | Q4_K_M | legacy | stack comparable | GeForce RTX 3090 · 24 GiB450 W maxdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-450w | agent | 4 | 122.8 | 6.00s | 8.1 | — | — | — |
| 8b | Q4_K_M | legacy | stack comparable | GeForce RTX 3090 · 24 GiBcap 350 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-350w | codegen | 1 | 122.5 | 110ms | 8.2 | — | — | — |
| 8b | Q4_K_M | legacy | stack comparable | GeForce RTX 3090 · 24 GiBcap 350 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-350w | rag | 1 | 122.1 | 244ms | 8.2 | — | — | — |
| 8b | Q4_K_M | legacy | stack comparable | GeForce RTX 3090 · 24 GiBcap 350 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-350w | agent | 4 | 120.9 | 6.22s | 8.3 | — | — | — |
| 4b-it | GGUF | raw | hardware comparable | GeForce RTX 3090 · 24 GiBcap 200 Wdrv 595 | llama.cpp llama.cpp-3e12fbd (cuda) | raw-v4-r2 | tg_512 | 1 | 120.9 | — | — | — | — | — |
| 8b | Q4_K_M | legacy | stack comparable | GeForce RTX 3090 · 24 GiBcap 350 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-350w | agent | 1 | 120.7 | 173ms | 8.3 | — | — | — |
| 4b-it | GGUF | raw | hardware comparable | GeForce RTX 3090 · 24 GiBcap 200 Wdrv 595 | llama.cpp llama.cpp-3e12fbd (cuda) | raw-v4-r2 | tg_1024 | 1 | 119.9 | — | — | — | — | — |
| 8b | Q4_K_M | legacy | stack comparable | GeForce RTX 5070 · 12 GiBcap 250 Wdrv 595 | llama.cpp b9174 (vulkan) | baseline | chat | 1 | 100.2 | 38ms | 10.0 | — | — | — |
| 8b | Q4_K_M | legacy | stack comparable | GeForce RTX 5070 · 12 GiBcap 250 Wdrv 595 | llama.cpp b9174 (vulkan) | baseline | codegen | 1 | 99.1 | 55ms | 10.1 | — | — | — |
| 8b | Q4_K_M | legacy | stack comparable | GeForce RTX 5070 · 12 GiBcap 250 Wdrv 595 | llama.cpp b9174 (vulkan) | baseline | rag | 1 | 98.1 | 95ms | 10.2 | — | — | — |
| 8b | Q4_K_M | legacy | stack comparable | GeForce RTX 5070 · 12 GiBcap 250 Wdrv 595 | llama.cpp b9174 (vulkan) | baseline | agent | 1 | 96.0 | 159ms | 10.4 | — | — | — |
| 8b | Q4_K_M | legacy | stack comparable | GeForce RTX 3090 · 24 GiBcap 200 Wdrv 590 | llama.cpp 59778f0 (cuda) | baseline | chat | 1 | 77.2 | 49ms | 13.0 | — | — | — |
| 8b | Q4_K_M | legacy | stack comparable | GeForce RTX 3090 · 24 GiBcap 200 Wdrv 590 | llama.cpp 59778f0 (cuda) | baseline | rag | 1 | 74.6 | 201ms | 13.4 | — | — | — |
| 8b | Q4_K_M | legacy | stack comparable | GeForce RTX 3090 · 24 GiBcap 200 Wdrv 590 | llama.cpp 59778f0 (cuda) | baseline | codegen | 1 | 72.2 | 112ms | 13.8 | — | — | — |
| 8b | Q4_K_M | legacy | stack comparable | GeForce RTX 3090 · 24 GiBcap 200 Wdrv 590 | llama.cpp 59778f0 (cuda) | baseline | agent | 1 | 71.5 | 254ms | 14.0 | — | — | — |
| 8b | Q4_K_M | legacy | stack comparable | GeForce RTX 5070 · 12 GiBcap 250 Wdrv 595 | llama.cpp b9174 (vulkan) | baseline | agent | 4 | 59.8 | 5.30s | 16.7 | — | — | — |
| 30b | Q4_K_M | legacy | stack comparable | GeForce RTX 3090 · 24 GiB450 W maxdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-450w | chat | 1 | 43.1 | 74ms | 23.2 | — | — | — |
| 30b | Q4_K_M | legacy | stack comparable | GeForce RTX 3090 · 24 GiB450 W maxdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-450w | codegen | 1 | 42.6 | 214ms | 23.5 | — | — | — |
| 30b | Q4_K_M | legacy | stack comparable | GeForce RTX 3090 · 24 GiB450 W maxdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-450w | rag | 1 | 42.5 | 361ms | 23.5 | — | — | — |
| 30b | Q4_K_M | legacy | stack comparable | GeForce RTX 3090 · 24 GiBcap 350 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-350w | chat | 1 | 42.4 | 76ms | 23.6 | — | — | — |
| 8b | Q4_K_M | legacy | stack comparable | GeForce RTX 3090 · 24 GiBcap 200 Wdrv 590 | llama.cpp 59778f0 (cuda) | baseline | agent | 4 | 42.3 | 2.74s | 23.6 | — | — | — |
| 30b | Q4_K_M | legacy | stack comparable | GeForce RTX 3090 · 24 GiB450 W maxdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-450w | agent | 1 | 42.3 | 280ms | 23.6 | — | — | — |
| 30b | Q4_K_M | legacy | stack comparable | GeForce RTX 3090 · 24 GiB450 W maxdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-450w | agent | 4 | 42.2 | 15.37s | 23.7 | — | — | — |
| 30b | Q4_K_M | legacy | stack comparable | GeForce RTX 3090 · 24 GiBcap 350 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-350w | codegen | 1 | 41.9 | 205ms | 23.9 | — | — | — |
| 30b | Q4_K_M | legacy | stack comparable | GeForce RTX 3090 · 24 GiBcap 350 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-350w | rag | 1 | 41.7 | 339ms | 24.0 | — | — | — |
| 30b | Q4_K_M | legacy | stack comparable | GeForce RTX 3090 · 24 GiBcap 350 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-350w | agent | 1 | 41.6 | 281ms | 24.0 | — | — | — |
| 30b | Q4_K_M | legacy | stack comparable | GeForce RTX 3090 · 24 GiBcap 350 Wdrv 590 | llama.cpp cuda-4f13cb7 (cuda) | baseline-pl-350w | agent | 4 | 41.6 | 15.64s | 24.0 | — | — | — |
| 30b | Q4_K_M | legacy | stack comparable | GeForce RTX 3090 · 24 GiBcap 200 Wdrv 590 | llama.cpp 59778f0 (cuda) | baseline | chat | 1 | 21.5 | 92ms | 46.5 | — | — | — |
| 30b | Q4_K_M | legacy | stack comparable | GeForce RTX 3090 · 24 GiBcap 200 Wdrv 590 | llama.cpp 59778f0 (cuda) | baseline | rag | 1 | 21.0 | 428ms | 47.6 | — | — | — |
| 30b | Q4_K_M | legacy | stack comparable | GeForce RTX 3090 · 24 GiBcap 200 Wdrv 590 | llama.cpp 59778f0 (cuda) | baseline | codegen | 1 | 21.0 | 217ms | 47.6 | — | — | — |
| 30b | Q4_K_M | legacy | stack comparable | GeForce RTX 3090 · 24 GiBcap 200 Wdrv 590 | llama.cpp 59778f0 (cuda) | baseline | agent | 1 | 20.8 | 278ms | 48.1 | — | — | — |
| 30b | Q4_K_M | legacy | stack comparable | GeForce RTX 3090 · 24 GiBcap 200 Wdrv 590 | llama.cpp 59778f0 (cuda) | baseline | agent | 4 | 12.3 | 4.48s | 81.2 | — | — | — |
| 4b-it | GGUF | raw | hardware comparable | GeForce RTX 3090 · 24 GiBcap 200 Wdrv 595 | llama.cpp llama.cpp-3e12fbd (cuda) | raw-v4-r2 | pp_4096 | 1 | — | — | — | — | — | — |
| 4b-it | GGUF | raw | hardware comparable | GeForce RTX 5070 · 12 GiBcap 250 Wdrv 595 | llama.cpp llama.cpp-1a68ec9 (cuda) | raw-v4-r2 | pp_512 | 1 | — | — | — | — | — | — |
| 4b-it | GGUF | raw | hardware comparable | GeForce RTX 5070 · 12 GiBcap 250 Wdrv 595 | llama.cpp llama.cpp-1a68ec9 (cuda) | raw-v4-r2 | pp_1024 | 1 | — | — | — | — | — | — |
| 4b-it | GGUF | raw | hardware comparable | GeForce RTX 5070 · 12 GiBcap 250 Wdrv 595 | llama.cpp llama.cpp-1a68ec9 (cuda) | raw-v4-r2 | pp_2048 | 1 | — | — | — | — | — | — |
| 4b-it | GGUF | raw | hardware comparable | GeForce RTX 5070 · 12 GiBcap 250 Wdrv 595 | llama.cpp llama.cpp-1a68ec9 (cuda) | raw-v4-r2 | pp_4096 | 1 | — | — | — | — | — | — |
| 4b-it | GGUF | raw | hardware comparable | GeForce RTX 5070 · 12 GiBcap 250 Wdrv 595 | llama.cpp llama.cpp-b9174 (vulkan) | raw-v4-r2 | pp_512 | 1 | — | — | — | — | — | — |
| 4b-it | GGUF | raw | hardware comparable | GeForce RTX 5070 · 12 GiBcap 250 Wdrv 595 | llama.cpp llama.cpp-b9174 (vulkan) | raw-v4-r2 | pp_1024 | 1 | — | — | — | — | — | — |
| 4b-it | GGUF | raw | hardware comparable | GeForce RTX 5070 · 12 GiBcap 250 Wdrv 595 | llama.cpp llama.cpp-b9174 (vulkan) | raw-v4-r2 | pp_2048 | 1 | — | — | — | — | — | — |
| 4b-it | GGUF | raw | hardware comparable | GeForce RTX 5070 · 12 GiBcap 250 Wdrv 595 | llama.cpp llama.cpp-b9174 (vulkan) | raw-v4-r2 | pp_4096 | 1 | — | — | — | — | — | — |
| Probe | unknown | probe | hardware comparable | Strix Halo · Radeon 8060S · 128 GiB unified (96 GiB VRAM)cap 0 Wunifieddrv 7 | hardware-probe v3 (rocm) | baseline | static_gpu | 1 | — | — | — | 256 GB/s | — | — |
| Probe | unknown | probe | hardware comparable | Strix Halo · Radeon 8060S · 128 GiB unified (96 GiB VRAM)cap 0 Wunifieddrv 7 | hardware-probe v3 (rocm) | baseline | torch_device_properties | 1 | — | — | — | — | — | — |
| Probe | unknown | probe | hardware comparable | Strix Halo · Radeon 8060S · 128 GiB unified (96 GiB VRAM)cap 0 Wunifieddrv 7 | hardware-probe v3 (rocm) | baseline | hip_device_copy | 1 | — | — | — | — | 106 GB/s | — |
| Probe | unknown | probe | hardware comparable | Strix Halo · Radeon 8060S · 128 GiB unified (96 GiB VRAM)cap 0 Wunifieddrv 7 | hardware-probe v3 (rocm) | baseline | hip_rocblas_hgemm_fp16 | 1 | — | — | — | — | — | 30.3 TF |
| Probe | unknown | probe | hardware comparable | GeForce RTX 3090 · 24 GiBcap 200 Wdrv 595 | hardware-probe v2 (cuda) | baseline | static_gpu | 1 | — | — | — | 936 GB/s | — | — |
| Probe | unknown | probe | hardware comparable | GeForce RTX 3090 · 24 GiBcap 200 Wdrv 595 | hardware-probe v2 (cuda) | baseline | torch_device_properties | 1 | — | — | — | — | — | — |
| Probe | unknown | probe | hardware comparable | GeForce RTX 3090 · 24 GiBcap 200 Wdrv 595 | hardware-probe v2 (cuda) | baseline | torch_device_copy | 1 | — | — | — | — | 391 GB/s | — |
| Probe | unknown | probe | hardware comparable | GeForce RTX 3090 · 24 GiBcap 200 Wdrv 595 | hardware-probe v2 (cuda) | baseline | torch_matmul_fp16 | 1 | — | — | — | — | — | 65.3 TF |
| Probe | unknown | probe | hardware comparable | GeForce RTX 3090 · 24 GiBcap 200 Wdrv 595 | hardware-probe v2 (cuda) | baseline | torch_matmul_bf16 | 1 | — | — | — | — | — | 65.4 TF |
| probe | unknown | probe | hardware comparable | GeForce RTX 3090 · 24 GiBcap 200 Wdrv 595 | hardware-probe torch-2.11.0-cu128 (cuda) | pl-200w | static_gpu | 1 | — | — | — | 936 GB/s | — | — |
| probe | unknown | probe | hardware comparable | GeForce RTX 3090 · 24 GiBcap 200 Wdrv 595 | hardware-probe torch-2.11.0-cu128 (cuda) | pl-200w | torch_device_properties | 1 | — | — | — | — | — | — |
| probe | unknown | probe | hardware comparable | GeForce RTX 3090 · 24 GiBcap 200 Wdrv 595 | hardware-probe torch-2.11.0-cu128 (cuda) | pl-200w | torch_device_copy | 1 | — | — | — | — | 391 GB/s | — |
| probe | unknown | probe | hardware comparable | GeForce RTX 3090 · 24 GiBcap 200 Wdrv 595 | hardware-probe torch-2.11.0-cu128 (cuda) | pl-200w | torch_matmul_fp16 | 1 | — | — | — | — | — | 65.4 TF |
| probe | unknown | probe | hardware comparable | GeForce RTX 3090 · 24 GiBcap 200 Wdrv 595 | hardware-probe torch-2.11.0-cu128 (cuda) | pl-200w | torch_matmul_bf16 | 1 | — | — | — | — | — | 65.4 TF |
| probe | unknown | probe | hardware comparable | GeForce RTX 3090 · 24 GiBcap 300 Wdrv 595 | hardware-probe torch-2.11.0-cu128 (cuda) | pl-300w | static_gpu | 1 | — | — | — | 936 GB/s | — | — |
| probe | unknown | probe | hardware comparable | GeForce RTX 3090 · 24 GiBcap 300 Wdrv 595 | hardware-probe torch-2.11.0-cu128 (cuda) | pl-300w | torch_device_properties | 1 | — | — | — | — | — | — |
| probe | unknown | probe | hardware comparable | GeForce RTX 3090 · 24 GiBcap 300 Wdrv 595 | hardware-probe torch-2.11.0-cu128 (cuda) | pl-300w | torch_device_copy | 1 | — | — | — | — | 391 GB/s | — |
| probe | unknown | probe | hardware comparable | GeForce RTX 3090 · 24 GiBcap 300 Wdrv 595 | hardware-probe torch-2.11.0-cu128 (cuda) | pl-300w | torch_matmul_fp16 | 1 | — | — | — | — | — | 65.4 TF |
| probe | unknown | probe | hardware comparable | GeForce RTX 3090 · 24 GiBcap 300 Wdrv 595 | hardware-probe torch-2.11.0-cu128 (cuda) | pl-300w | torch_matmul_bf16 | 1 | — | — | — | — | — | 65.3 TF |
| probe | unknown | probe | hardware comparable | GeForce RTX 3090 · 24 GiB450 W maxdrv 595 | hardware-probe torch-2.11.0-cu128 (cuda) | pl-450w | static_gpu | 1 | — | — | — | 936 GB/s | — | — |
| probe | unknown | probe | hardware comparable | GeForce RTX 3090 · 24 GiB450 W maxdrv 595 | hardware-probe torch-2.11.0-cu128 (cuda) | pl-450w | torch_device_properties | 1 | — | — | — | — | — | — |
| probe | unknown | probe | hardware comparable | GeForce RTX 3090 · 24 GiB450 W maxdrv 595 | hardware-probe torch-2.11.0-cu128 (cuda) | pl-450w | torch_device_copy | 1 | — | — | — | — | 391 GB/s | — |
| probe | unknown | probe | hardware comparable | GeForce RTX 3090 · 24 GiB450 W maxdrv 595 | hardware-probe torch-2.11.0-cu128 (cuda) | pl-450w | torch_matmul_fp16 | 1 | — | — | — | — | — | 65.4 TF |
| probe | unknown | probe | hardware comparable | GeForce RTX 3090 · 24 GiB450 W maxdrv 595 | hardware-probe torch-2.11.0-cu128 (cuda) | pl-450w | torch_matmul_bf16 | 1 | — | — | — | — | — | 65.4 TF |
| Probe | unknown | probe | hardware comparable | GeForce RTX 5070 · 12 GiBcap 250 Wdrv 595 | hardware-probe v2 (cuda) | baseline | static_gpu | 1 | — | — | — | 672 GB/s | — | — |
| Probe | unknown | probe | hardware comparable | GeForce RTX 5070 · 12 GiBcap 250 Wdrv 595 | hardware-probe v2 (cuda) | baseline | torch_device_properties | 1 | — | — | — | — | — | — |
| Probe | unknown | probe | hardware comparable | GeForce RTX 5070 · 12 GiBcap 250 Wdrv 595 | hardware-probe v2 (cuda) | baseline | torch_device_copy | 1 | — | — | — | — | 271 GB/s | — |
| Probe | unknown | probe | hardware comparable | GeForce RTX 5070 · 12 GiBcap 250 Wdrv 595 | hardware-probe v2 (cuda) | baseline | torch_matmul_fp16 | 1 | — | — | — | — | — | 69.6 TF |
| Probe | unknown | probe | hardware comparable | GeForce RTX 5070 · 12 GiBcap 250 Wdrv 595 | hardware-probe v2 (cuda) | baseline | torch_matmul_bf16 | 1 | — | — | — | — | — | 68.3 TF |
Decode tok/s
Headline speed metric
TTFT / TPOT
Latency context
Raw vs workload
Separate comparison contracts
Notes badge key
hardware comparable
Use these rows for GPU-to-GPU comparisons when the model, quant, backend, driver family, power policy, and benchmark shape match closely.
stack comparable
Use these rows to compare a similar software stack. They are useful, but backend, server path, driver, cache, or power settings may still influence the number.
stack realistic
Treat these as real workload measurements, not pure hardware rankings. They include prompt mix, API/server overhead, cache behavior, and local software details.
legacyOlder workload harness row.
350 W capRecorded GPU power limit.
drv 590GPU driver branch.
reasoningReasoning-token model.
Metric guide
Decode tok/s - Generation rate. Raw rows come from the engine benchmark; API rows use token intervals when available.
TTFT - Time to first token. This includes prompt processing and server/API overhead.
TPOT / ITL - Time per output token after the first token. Lower is better.
Raw Engine - llama-bench style cases intended for hardware-normalized comparison across rigs.
Workload / API - Stack-realistic measurements that include backend, server, cache, driver, and prompt behavior.
Power badges - A cap badge shows the recorded power limit. The row metadata records the cap relative to the recorded max.