Skip to content

GPQA Diamond benchmark results

Compare Qwen3.8-27B accuracy and evaluation time on Olares One and Mac Studio.

GPQA Diamond is a set of 198 graduate-level science questions in biology, physics and chemistry. The questions are written to be hard to answer even with web search.

We report accuracy: the percentage of four-option multiple-choice questions answered correctly, with no examples in the prompt.

Results across hardware

  • 198 test questions
  • Reasoning effort: xhigh
  • Output limit: 64k tokens

Sorted by Score, highest first.

HardwareModel / configurationPercentage of four-option questions answered correctly, with no examples in the prompt.%Higher is betterWall-clock time for all 198 questions, including model loading and scoring, with one request at a time.hoursLower is betterOutput tokens per second while the model writes the reply.tok/sHigher is betterOutput per questionMean output length. Longer replies also add to the evaluation time.tokens
Olares OneQwen3.8-27Bllama.cpp, NVFP4 LOW ↗88.3811.56914,350
Mac Studio M5 MaxQwen3.8-27BOllama (MLX), nvfp4 ↗85.8614.96016,176
Hardware / modelPercentage of four-option questions answered correctly, with no examples in the prompt.Wall-clock time for all 198 questions, including model loading and scoring, with one request at a time.
Olares OneQwen3.8-27Bllama.cpp, NVFP4 LOW88.38%11.5 h
Token generation
69 tok/s
Output per question
14,350 tokens
Mac Studio M5 MaxQwen3.8-27BOllama (MLX), nvfp485.86%14.9 h
Token generation
60 tok/s
Output per question
16,176 tokens

Configurations tested more than once show the mean of their full evaluations. Scores can move between runs.

Model-card reference scores

Published GPQA Diamond scores for the original models, not for the quantizations in this table.