Appearance
Compare IFBench token generation speed, evaluation time, and instruction-following scores across local hardware and model configurations.
IFBench is an instruction-following benchmark from Ai2. It tests whether models follow explicit requirements in their responses.
We use prompt-level strict scoring: a response passes only if it satisfies every requirement in the prompt.
Sorted by Score, highest first.
| Hardware | Model / configuration | Percentage of prompts that pass every constraint, using prompt-level strict scoring.%Higher is better | Wall-clock time for all 300 prompts, including model loading and scoring, with one request at a time.hoursLower is better | Output tokens per second while the model writes the reply.tok/sHigher is better | Output per questionMean output length. Longer replies also add to the evaluation time.tokens |
|---|---|---|---|---|---|
| Olares One | Qwen3.8-Flash-NextStrata, UD-IQ4_XS ↗ | 84.00 | 8.8 | 88 | 9,334 |
| GTR9 Pro | Qwen3.8-27Bllama.cpp Vulkan, UD-Q4_K_XL ↗ | 82.00 | 33.1 | 21 | 8,267 |
| Olares One | Qwen3.8-Flash-NextStrata, IQ3_XXS ↗ | 82.00 | 8.0 | 124 | 11,827 |
| Mac Studio M5 Max | Qwen3.8-27BOllama (MLX), nvfp4 ↗ | 79.89 | 12.2 | 65 | 9,426 |
| Olares One | Qwen3.8-27Bllama.cpp, NVFP4 LOW ↗ | 79.89 | 8.7 | 75 | 7,875 |
| DGX Spark | Qwen3.8-27Bllama.cpp, NVFP4 LOW ↗ | 79.33 | 18.2 | 36 | 7,727 |
| Olares One | Qwen3.8-27Bllama.cpp, UD-Q4_K_XL ↗ | 79.33 | 10.0 | 68 | 8,208 |
| Olares One | Qwen3.8-Flash-NextStrata, IQ2_XS ↗ | 76.33 | 7.7 | 148 | 13,669 |
Configurations tested more than once show the mean of their full evaluations. Scores can move between runs.
Published IFBench scores for the original models, not for the quantizations in this table. The model cards do not say which scoring variant they used. This table uses prompt-level strict scoring.