granite-4.1 8b

Q4_K_M·8B params·GGUF

intelligence: see on Artificial Analysis →

checkpoint: unsloth/granite-4.1-8b-GGUF:Q4_K_M

All runs (10)

Hardware	Backend	Shape	Conc.	Gen tok/s ↓	TTFT	TPOT (ms)	Out tok	Total	VRAM Δ
GeForce RTX 5070 · 11.94 GiB	llama.cpp b9174 (cuda)	codegen	1	98.5	55ms	10.1	738	7.48s	0.000 GiB
GeForce RTX 5070 · 11.94 GiB	llama.cpp b9174 (cuda)	chat	1	97.6	38ms	10.0	100	1.00s	0.000 GiB
GeForce RTX 5070 · 11.94 GiB	llama.cpp b9174 (cuda)	agent	1	94.6	159ms	10.4	500	5.24s	0.000 GiB
GeForce RTX 5070 · 11.94 GiB	llama.cpp b9174 (cuda)	rag	1	91.5	95ms	10.2	79	1.12s	0.000 GiB
GeForce RTX 3090 · 24 GiB	llama.cpp 59778f0 (cuda)	chat	1	74.4	49ms	13.0	100	1.30s	0.000 GiB
GeForce RTX 3090 · 24 GiB	llama.cpp 59778f0 (cuda)	codegen	1	70.7	112ms	13.8	735	10.41s	0.000 GiB
GeForce RTX 3090 · 24 GiB	llama.cpp 59778f0 (cuda)	agent	1	66.6	254ms	14.0	500	7.36s	0.000 GiB
GeForce RTX 3090 · 24 GiB	llama.cpp 59778f0 (cuda)	rag	1	63.8	201ms	13.4	79	1.22s	0.000 GiB
GeForce RTX 5070 · 11.94 GiB	llama.cpp b9174 (cuda)	agent	4	36.9	5.30s	16.7	500	11.52s	0.000 GiB
GeForce RTX 3090 · 24 GiB	llama.cpp 59778f0 (cuda)	agent	4	35.0	2.74s	23.6	500	13.85s	0.000 GiB

Environment

GeForce RTX 3090 · 24 GiB

cpuAMD EPYC 7302P 16-Core Processor

gpuNVIDIA GeForce RTX 3090

archNVIDIA

vram24 GiB (system 64.0 GiB)

power200 W / 450 W max(44% cap)

backendllama.cpp 59778f0 (cuda)

serverlemonade unknown

osUbuntu 24.04 LTS

kernel6.17.13-7-pve

driver590.48.01

python3.12.3

containerizedtrue

runs/cell5

warmups2

endpoint/v1/chat/completions

streamingtrue

GeForce RTX 5070 · 11.94 GiB

cpuAMD Ryzen 9 7900 12-Core Processor

gpuNVIDIA GeForce RTX 5070

archNVIDIA

vram11.94 GiB (system 30.4 GiB)

power250 W / 300 W max(83% cap)

backendllama.cpp b9174 (cuda)

serverlemonade unknown

osCachyOS

kernel7.0.8-1-cachyos

driver595.71.05

python3.14.4

containerizedfalse

runs/cell5

warmups2

endpoint/v1/chat/completions

streamingtrue