Skip to content

How we benchmark local AI hardware

We test models the way you would run them at home: download a model, run it on one machine with the engine we would pick for that machine, and time what you actually wait for. The question we answer is "How long will this take on my machine, and is the result still good?" Our focus is the time and output you can expect in everyday use.

Configurations we compare ​

Each machine runs the quantization and engine we recommend for it. Configurations can differ to suit each machine, and the setup is shown under every bar.

We use the same tasks and settings across systems and keep an archived run record for every reported number.

Wait times are wall-clock measurements covering the full process from start to finish.

Language model evaluations ​

We give each model a full set of questions from IFBench, which tests instruction following, and GPQA Diamond, which tests difficult science questions. We measure how long the model takes to complete the questions, including thinking time.

The score checks whether the downloadable, quantized model still answers about as well as the official model card reports. Official scores may come from a different evaluation stack. We use them specifically as a reference for checking the quality of quantized model responses.

Repeated runs and score variation ​

Scores move by a few points between runs. We call differences within that noise "tied".

For 300 IFBench questions, one run varies by about ±2.7 points. Two runs of the same setup produce different outcomes on 35 to 50 questions. We therefore run IFBench two or three times per machine and show the mean:

  • Score: Total correct answers divided by total questions across the runs.
  • Other columns: Per-run means.
  • Variation: The noise of a mean of k runs is about ±2.7/√k.

We also compare machines question by question. Each question's result is averaged across a machine's runs before we calculate the difference between machines. This gives a smaller error than comparing two headline scores.

LLM inference speed ​

The speed benchmark measures how fast a model reads a long prompt, how fast text appears for one request, and the total output when four requests run at once.

  • Prefill: We use prompts of 1k, 4k, 16k, 32k and 64k tokens, with three runs per length. The table shows the median prompt rate reported by the engine itself, using llama.cpp and Ollama timings. The wait before the first token also includes 0.1 to 0.2 s of fixed request overhead from HTTP, templates, tokenization and the first token. This has the greatest effect on the 1k prompt. Client-side times are recorded in each run's files.
  • Decode, 1 stream: We send 20 prompts one at a time and measure tokens per second while the model writes. The client times the streamed reply in the same way for every engine.
  • 4 streams, total: We send the same 20 prompts four at a time and divide all output tokens by the wall-clock time. The multiple shown alongside the result compares this throughput with one stream.

Each machine uses its own speculative decoding, either MTP or a DFlash2 draft model. Speed therefore partly reflects how well the draft predicts the output.

Ollama on the Mac answers one request at a time. Its four-stream result measures four requests waiting in line, and the bar is marked accordingly.

Image generation ​

We use the same prompts and seeds at 1024 × 1024 resolution and measure the time from submitting a prompt to saving the image.

We report the median after the first image. The first image also includes model loading, which is only needed once for the run.

Video generation ​

We measure how long one clip takes with eight generation steps and which clip lengths fit in 24 GB.

The MiniMax H3 time and VRAM calculator predicts other lengths using a fitted model. We check that fit against held-out measurements kept separate from the data used to build it.

Peak compute figures ​

The hardware table lists peak dense INT8 GPU throughput. These specification figures describe the hardware’s theoretical compute ceiling.

  • Derived from vendor figures: NVIDIA publishes an FP4 figure with 2:4 sparsity for each machine. Halving it gives dense FP4. Halving it again gives dense INT8, following the ratio in NVIDIA's architecture whitepapers.
  • Estimated figures: AMD’s published GPU compute figure uses FP16. We estimate Radeon 8060S INT8 throughput from its compute units and clock speed. Apple leaves GPU compute throughput unspecified, so the M5 Max estimate scales a measurement on the 10-core M5 GPU to 40 cores.
  • Comparison scope: We compare dense GPU throughput for the downloaded models, which retain dense weights. Every run here uses the GPU. Sparse throughput, NPU throughput and whole-chip totals, such as AMD’s 126 TOPS, sit outside this comparison.

How close a model gets to peak compute depends on the engine. In llama.cpp prompt processing, the Mac reaches a much larger share of its peak than the NVIDIA machines. ComfyUI on the Mac uses a compute path that bypasses the INT8 hardware.

Memory bandwidth limits token generation in LLMs, making bandwidth the relevant hardware figure for interpreting generation speed.

What we publish ​

We publish selected results from archived benchmark runs, with machine hostnames and private network details removed. The full batch archive is kept separately.