Benchmarks

Local LLM speed results across models, backends, hardware, and power profiles. Decode tok/s is the headline metric; latency, raw engine runs, and workload context stay visible in their own views.

1181 source rows48 matching source rowslatest run May 21, 2026schemas v1-v4source content/benchmarks/runs/

Leaderboard Hardware Raw engine Power Explorer

Leaderboard: Curated model rankings using workload-style decode speed at the selected concurrency.

What the tabs show

Leaderboard: Curated model rankings using workload-style decode speed at the selected concurrency.

Hardware: Rig details, drivers, power limits, and hardware microbenchmarks separated from model rankings.

Raw engine: llama-bench style prompt/decode cases for the closest hardware-normalized comparison.

Power: Power-limit sweep rows showing how caps change decode speed and latency.

Explorer: Full row-level dataset with every suite, shape, mode, rerun, and technical metric.

Filters

Advanced filters

One row per model, hardware, backend, quant, and normalized mode. Decode tok/s is shown as a range across workload shapes at the selected concurrency.

Showing top 11 of 11 grouped leaderboard rows from 48 matching source rows.

Notes:hardware comparablematched GPU comparisonstack comparablesimilar stackstack realisticreal workload rowlegacyolder harness350 W caprecorded power limitdrv 590driver branch

Model	Quant	Hardware	Backend / Mode	Decode tok/s	TTFT	Notes
LFM2.5-350M 4 workload shapes	Q4_K_M	GeForce RTX 5070 · 12 GiB	llama.cppbaseline	843.9-905.0	7ms	legacystack comparable250 W capdrv 595
LFM2 1.2B 4 workload shapes	Q4_K_M	GeForce RTX 5070 · 12 GiB	llama.cppbaseline	517.9-539.1	12ms	legacystack comparable250 W capdrv 595
LFM2 1.2B-Tool 4 workload shapes	Q4_K_M	GeForce RTX 5070 · 12 GiB	llama.cppbaseline	517.6-534.8	11ms	legacystack comparable250 W capdrv 595
LFM2 8B-A1B 4 workload shapes	Q4_K_M	GeForce RTX 5070 · 12 GiB	llama.cppbaseline	362.1-372.3	30ms	legacystack comparable250 W capdrv 595
LFM2 2.6B 4 workload shapes	Q4_K_M	GeForce RTX 5070 · 12 GiB	llama.cppbaseline	266.8-273.7	21ms	legacystack comparable250 W capdrv 595
Gemma-4 E2B-it 4 workload shapes	Q4_K_M	GeForce RTX 5070 · 12 GiB	llama.cppbaseline	212.6-220.3	36ms	legacystack comparable250 W capdrv 595
Gemma-3 4b-it 4 workload shapes	Q4_K_M	GeForce RTX 5070 · 12 GiB	llama.cppbaseline	168.4-172.6	48ms	legacystack comparable250 W capdrv 595
Gemma-3 4b-it 4 workload shapes	Q4_K_M	GeForce RTX 5070 · 12 GiB	llama.cppbaseline	156.0-168.8	57ms	legacystack comparable250 W capdrv 595
Gemma-4 E4B-it 4 workload shapes	Q4_K_M	GeForce RTX 5070 · 12 GiB	llama.cppbaseline	124.8-129.8	63ms	legacystack comparable250 W capdrv 595
Qwen2.5-Coder 7B-Instruct 4 workload shapes	Q4_K_M	GeForce RTX 5070 · 12 GiB	llama.cppbaseline	117.6-120.7	59ms	legacystack comparable250 W capdrv 595
granite-4.1 8b 4 workload shapes	Q4_K_M	GeForce RTX 5070 · 12 GiB	llama.cppbaseline	96.0-100.2	75ms	legacystack comparable250 W capdrv 595

Decode tok/s

Headline speed metric

TTFT / TPOT

Latency context

Raw vs workload

Separate comparison contracts

Notes badge key

hardware comparable

Use these rows for GPU-to-GPU comparisons when the model, quant, backend, driver family, power policy, and benchmark shape match closely.

stack comparable

Use these rows to compare a similar software stack. They are useful, but backend, server path, driver, cache, or power settings may still influence the number.

stack realistic

Treat these as real workload measurements, not pure hardware rankings. They include prompt mix, API/server overhead, cache behavior, and local software details.

legacyOlder workload harness row.

350 W capRecorded GPU power limit.

drv 590GPU driver branch.

reasoningReasoning-token model.

Metric guide

Decode tok/s - Generation rate. Raw rows come from the engine benchmark; API rows use token intervals when available.

TTFT - Time to first token. This includes prompt processing and server/API overhead.

TPOT / ITL - Time per output token after the first token. Lower is better.

Raw Engine - llama-bench style cases intended for hardware-normalized comparison across rigs.

Workload / API - Stack-realistic measurements that include backend, server, cache, driver, and prompt behavior.

Power badges - A cap badge shows the recorded power limit. The row metadata records the cap relative to the recorded max.