Skip to content

LLM inference speed benchmarks

Compare Qwen3.8-27B token generation and prompt processing speeds across tested local AI hardware, with single-request and four-request results.

Qwen3.8-27B

20 prompts, medium reasoning

Token generation, single request

Higher is better
Mac Studio M5 MaxOllama (MLX), nvfp4
74 tok/s
DGX Sparkllama.cpp, NVFP4 LOW
45 tok/s
GTR9 Prollama.cpp Vulkan, UD-Q4_K_XL
28 tok/s
Mac Studio M5 MaxOllama (GGML), UD-Q4_K_XL
27 tok/s

Token generation, four requests (total)

Higher is better
Mac Studio M5 MaxqueuedOllama (MLX), nvfp4
74 tok/s
DGX Sparkllama.cpp, NVFP4 LOW
99 tok/s
GTR9 Prollama.cpp Vulkan, UD-Q4_K_XL
42 tok/s
Mac Studio M5 MaxOllama (GGML), UD-Q4_K_XL
62 tok/s

For configurations marked queued, the server runs one request at a time. Four clients wait on that single stream.

Prompt processing (prefill)

Prefill speed reported by the engine, tok/s, median of 3 runs per prompt length

Prompt length in tokens. Speed in tokens per second.

Scroll horizontally to compare all prompt lengths.