Skip to content

IFBench benchmark results

Compare IFBench token generation speed, evaluation time, and instruction-following scores across local hardware and model configurations.

IFBench is an instruction-following benchmark from Ai2. It tests whether models follow explicit requirements in their responses.

We use prompt-level strict scoring: a response passes only if it satisfies every requirement in the prompt.

Results across hardware

  • 300 test prompts
  • Reasoning effort: xhigh
  • Output limit: 64k tokens

Sorted by Score, highest first.

HardwareModel / configurationPercentage of prompts that pass every constraint, using prompt-level strict scoring.%Higher is betterWall-clock time for all 300 prompts, including model loading and scoring, with one request at a time.hoursLower is betterOutput tokens per second while the model writes the reply.tok/sHigher is betterOutput per questionMean output length. Longer replies also add to the evaluation time.tokens
Olares OneQwen3.8-Flash-NextStrata, UD-IQ4_XS ↗84.008.8889,334
GTR9 ProQwen3.8-27Bllama.cpp Vulkan, UD-Q4_K_XL ↗82.0033.1218,267
Olares OneQwen3.8-Flash-NextStrata, IQ3_XXS ↗82.008.012411,827
Mac Studio M5 MaxQwen3.8-27BOllama (MLX), nvfp4 ↗79.8912.2659,426
Olares OneQwen3.8-27Bllama.cpp, NVFP4 LOW ↗79.898.7757,875
DGX SparkQwen3.8-27Bllama.cpp, NVFP4 LOW ↗79.3318.2367,727
Olares OneQwen3.8-27Bllama.cpp, UD-Q4_K_XL ↗79.3310.0688,208
Olares OneQwen3.8-Flash-NextStrata, IQ2_XS ↗76.337.714813,669
Hardware / modelPercentage of prompts that pass every constraint, using prompt-level strict scoring.Wall-clock time for all 300 prompts, including model loading and scoring, with one request at a time.
Olares OneQwen3.8-Flash-NextStrata, UD-IQ4_XS84.00%8.8 h
Token generation
88 tok/s
Output per question
9,334 tokens
GTR9 ProQwen3.8-27Bllama.cpp Vulkan, UD-Q4_K_XL82.00%33.1 h
Token generation
21 tok/s
Output per question
8,267 tokens
Olares OneQwen3.8-Flash-NextStrata, IQ3_XXS82.00%8.0 h
Token generation
124 tok/s
Output per question
11,827 tokens
Mac Studio M5 MaxQwen3.8-27BOllama (MLX), nvfp479.89%12.2 h
Token generation
65 tok/s
Output per question
9,426 tokens
Olares OneQwen3.8-27Bllama.cpp, NVFP4 LOW79.89%8.7 h
Token generation
75 tok/s
Output per question
7,875 tokens
DGX SparkQwen3.8-27Bllama.cpp, NVFP4 LOW79.33%18.2 h
Token generation
36 tok/s
Output per question
7,727 tokens
Olares OneQwen3.8-27Bllama.cpp, UD-Q4_K_XL79.33%10.0 h
Token generation
68 tok/s
Output per question
8,208 tokens
Olares OneQwen3.8-Flash-NextStrata, IQ2_XS76.33%7.7 h
Token generation
148 tok/s
Output per question
13,669 tokens

Configurations tested more than once show the mean of their full evaluations. Scores can move between runs.

Model-card reference scores

Published IFBench scores for the original models, not for the quantizations in this table. The model cards do not say which scoring variant they used. This table uses prompt-level strict scoring.