Appearance
Compare Qwen3.8-27B benchmark scores and inference speed on Olares One, Mac Studio, NVIDIA DGX Spark and Beelink GTR9 Pro.
Qwen's 27B dense model with native image and video understanding. Thinking is on by default and its depth is set per request with reasoning_effort. The context window is 262K tokens, extensible to 1M.
Each machine runs the quantized copy we recommend for it. We time the same benchmarks and compare the scores with the official model card.
IFBench is an instruction-following benchmark from Ai2. It tests whether models follow explicit requirements in their responses.
xhigh reasoning, 64k output cap
Evaluation time
Lower is better| Machine | Configuration | Score | Output per question | Token generation | Evaluation time |
|---|---|---|---|---|---|
| OfficialModel card ↗ | Original weights, model card ↗ | 79.5 | — | — | — |
| Olares Onellama.cpp, NVFP4 LOW ↗ | llama.cpp, NVFP4 LOW ↗ | 79.9 | 7,875 tok | 75 tok/s | 8.7 h |
| Olares Onellama.cpp, UD-Q4_K_XL ↗ | llama.cpp, UD-Q4_K_XL ↗ | 79.3 | 8,208 tok | 68 tok/s | 10.0 h |
| Mac Studio M5 MaxOllama (MLX), nvfp4 ↗ | Ollama (MLX), nvfp4 ↗ | 79.9 | 9,426 tok | 65 tok/s | 12.2 h |
| DGX Sparkllama.cpp, NVFP4 LOW ↗ | llama.cpp, NVFP4 LOW ↗ | 79.3 | 7,727 tok | 36 tok/s | 18.2 h |
| GTR9 Prollama.cpp Vulkan, UD-Q4_K_XL ↗ | llama.cpp Vulkan, UD-Q4_K_XL ↗ | 82.0 | 8,267 tok | 21 tok/s | 33.1 h |
Each score is the mean of 1 or 2 or 3 full runs on that machine. Other columns are per-run means. One run varies by about ±2.3 points at this question count, a mean of k runs by about ±2.3/√k (±2.3 for 1, ±1.6 for 2, ±1.3 for 3), so a score above the official one is noise, not a gain. Evaluation time is about questions × output per question ÷ token generation speed, one request at a time. It also includes loading and scoring.
GPQA Diamond is a set of 198 graduate-level science questions in biology, physics and chemistry. The questions are written to be hard to answer even with web search.
xhigh reasoning, 64k output cap
Evaluation time
Lower is better| Machine | Configuration | Score | Output per question | Token generation | Evaluation time |
|---|---|---|---|---|---|
| OfficialModel card ↗ | Original weights, model card ↗ | 89.2 | — | — | — |
| Olares Onellama.cpp, NVFP4 LOW ↗ | llama.cpp, NVFP4 LOW ↗ | 88.4 | 14,350 tok | 69 tok/s | 11.5 h |
| Mac Studio M5 MaxOllama (MLX), nvfp4 ↗ | Ollama (MLX), nvfp4 ↗ | 85.9 | 16,176 tok | 60 tok/s | 14.9 h |
One run varies by about ±2.2 points at this question count, so a score above the official one is noise, not a gain. Evaluation time is about questions × output per question ÷ token generation speed, one request at a time. It also includes loading and scoring.
How fast a long prompt is read (prefill), how fast the answer streams out with one request, and the total throughput with four requests at once.
20 prompts, medium reasoning
Prefill speed reported by the engine, tok/s, median of 3 runs per prompt length
Prompt length in tokens. Speed in tokens per second.
| Machine | Configuration | 1k | 4k | 16k | 32k | 64k |
|---|---|---|---|---|---|---|
| Olares Onellama.cpp, NVFP4 LOW ↗ | llama.cpp, NVFP4 LOW ↗ | 2,005 | 2,713 | 2,716 | 2,255 | 1,510 |
| Olares Onellama.cpp, UD-Q4_K_XL ↗ | llama.cpp, UD-Q4_K_XL ↗ | 1,545 | 1,671 | 1,602 | 1,434 | 1,114 |
| DGX Sparkllama.cpp, NVFP4 LOW ↗ | llama.cpp, NVFP4 LOW ↗ | 950 | 1,072 | 1,061 | 1,007 | 877 |
| Mac Studio M5 MaxOllama (MLX), nvfp4 ↗ | Ollama (MLX), nvfp4 ↗ | 975 | 1,016 | 951 | 874 | 737 |
| Mac Studio M5 MaxOllama (GGML), UD-Q4_K_XL ↗ | Ollama (GGML), UD-Q4_K_XL ↗ | 690 | 712 | 677 | 594 | 497 |
| GTR9 Prollama.cpp Vulkan, UD-Q4_K_XL ↗ | llama.cpp Vulkan, UD-Q4_K_XL ↗ | 318 | 326 | 300 | 268 | 223 |