Skip to content

Compare Qwen3.8-27B benchmark scores and inference speed on Olares One, Mac Studio, NVIDIA DGX Spark and Beelink GTR9 Pro.

Qwen's 27B dense model with native image and video understanding. Thinking is on by default and its depth is set per request with reasoning_effort. The context window is 262K tokens, extensible to 1M.

Parameters
27B dense
Official IFBench
79.5
Official GPQA Diamond
89.2

Each machine runs the quantized copy we recommend for it. We time the same benchmarks and compare the scores with the official model card.

IFBench

IFBench is an instruction-following benchmark from Ai2. It tests whether models follow explicit requirements in their responses.

xhigh reasoning, 64k output cap

Evaluation time

Lower is better
Mac Studio M5 MaxOllama (MLX), nvfp4
12.2 h
DGX Sparkllama.cpp, NVFP4 LOW
18.2 h
GTR9 Prollama.cpp Vulkan, UD-Q4_K_XL
33.1 h

Each score is the mean of 1 or 2 or 3 full runs on that machine. Other columns are per-run means. One run varies by about ±2.3 points at this question count, a mean of k runs by about ±2.3/√k (±2.3 for 1, ±1.6 for 2, ±1.3 for 3), so a score above the official one is noise, not a gain. Evaluation time is about questions × output per question ÷ token generation speed, one request at a time. It also includes loading and scoring.

GPQA Diamond

GPQA Diamond is a set of 198 graduate-level science questions in biology, physics and chemistry. The questions are written to be hard to answer even with web search.

xhigh reasoning, 64k output cap

Evaluation time

Lower is better
Mac Studio M5 MaxOllama (MLX), nvfp4
14.9 h
MachineConfigurationScoreOutput per questionToken generationEvaluation time
OfficialModel card ↗Original weights, model card ↗89.2———
Mac Studio M5 MaxOllama (MLX), nvfp4 ↗Ollama (MLX), nvfp4 ↗85.916,176 tok60 tok/s14.9 h

One run varies by about ±2.2 points at this question count, so a score above the official one is noise, not a gain. Evaluation time is about questions × output per question ÷ token generation speed, one request at a time. It also includes loading and scoring.

LLM inference speed

How fast a long prompt is read (prefill), how fast the answer streams out with one request, and the total throughput with four requests at once.

20 prompts, medium reasoning

Token generation, single request

Higher is better
Mac Studio M5 MaxOllama (MLX), nvfp4
74 tok/s
DGX Sparkllama.cpp, NVFP4 LOW
45 tok/s
GTR9 Prollama.cpp Vulkan, UD-Q4_K_XL
28 tok/s
Mac Studio M5 MaxOllama (GGML), UD-Q4_K_XL
27 tok/s

Token generation, four requests (total)

Higher is better
Mac Studio M5 MaxqueuedOllama (MLX), nvfp4
74 tok/s
DGX Sparkllama.cpp, NVFP4 LOW
99 tok/s
GTR9 Prollama.cpp Vulkan, UD-Q4_K_XL
42 tok/s
Mac Studio M5 MaxOllama (GGML), UD-Q4_K_XL
62 tok/s

Prompt processing (prefill)

Prefill speed reported by the engine, tok/s, median of 3 runs per prompt length

Prompt length in tokens. Speed in tokens per second.