Appearance
Compare Qwen3.8-27B accuracy and evaluation time on Olares One and Mac Studio.
GPQA Diamond is a set of 198 graduate-level science questions in biology, physics and chemistry. The questions are written to be hard to answer even with web search.
We report accuracy: the percentage of four-option multiple-choice questions answered correctly, with no examples in the prompt.
Sorted by Score, highest first.
| Hardware | Model / configuration | Percentage of four-option questions answered correctly, with no examples in the prompt.%Higher is better | Wall-clock time for all 198 questions, including model loading and scoring, with one request at a time.hoursLower is better | Output tokens per second while the model writes the reply.tok/sHigher is better | Output per questionMean output length. Longer replies also add to the evaluation time.tokens |
|---|---|---|---|---|---|
| Olares One | Qwen3.8-27Bllama.cpp, NVFP4 LOW ↗ | 88.38 | 11.5 | 69 | 14,350 |
| Mac Studio M5 Max | Qwen3.8-27BOllama (MLX), nvfp4 ↗ | 85.86 | 14.9 | 60 | 16,176 |
Configurations tested more than once show the mean of their full evaluations. Scores can move between runs.
Published GPQA Diamond scores for the original models, not for the quantizations in this table.