hyperfine: the honest review

Verdict: install it the next time you catch yourself typing time some-command twice and eyeballing the difference — that’s the job hyperfine does properly, with real statistics and a repeat count you didn’t have to think about. But it measures wall-clock time, so the answer is only as honest as the machine you run it on, and it will warn you about that in nearly every sentence. hyperfine runs a command over and over, reports the mean, the standard deviation, and how many times faster one command was than another. We reach for it whenever the question is “is A faster than B,” which is the only benchmarking question most of us actually have.

hyperfine is free and open source (MIT/Apache-2.0). We have no relationship with the project and nothing to sell. Like its Rust-rewrite cousins ripgrep and fd, the catch here isn’t price or telemetry. It’s subtler: the tool is honest to a fault about its own margin of error, and on the shared CI box our robot lives on, that margin never got out of our way. Every number below we captured on that box — an Ubuntu 24.04 GitHub Actions runner, hyperfine 1.18.0 — and the noise is part of the review.

Install

brew install hyperfine        # macOS
sudo apt install hyperfine    # Debian/Ubuntu 24.04+

No rename tax this time — unlike fd shipping as fdfind or bat as batcat, the command on your PATH is hyperfine, the name every tutorial types:

$ hyperfine --version
hyperfine 1.18.0

Why you’d use it instead of time

Point it at a command. It runs it enough times to get a stable number, then reports the spread:

$ hyperfine --warmup 1 'sleep 0.05'
Benchmark 1: sleep 0.05
  Time (mean ± σ):      51.6 ms ±   0.2 ms    [User: 0.5 ms, System: 0.8 ms]
  Range (min … max):    51.1 ms …  52.0 ms    58 runs

That’s the whole pitch over time: time runs once and gives you one sample, which on a busy machine could be anything. hyperfine ran sleep 0.05 fifty-eight times, told you the mean was 51.6 ms with a standard deviation of 0.2 ms, and chose the run count itself. The ± 0.2 ms is the part time can never give you — it’s the difference between “it was 51 ms” and “it’s reliably 51 ms.”

Where it earns its keep is comparing two commands. Ask which is faster at finding a line in a 200,000-line file, grep or awk:

$ seq 1 200000 > nums.txt
$ hyperfine --warmup 3 -N 'grep 199999 nums.txt' 'awk "/199999/" nums.txt'
Benchmark 1: grep 199999 nums.txt
  Time (mean ± σ):       1.7 ms ±   0.0 ms    [User: 1.0 ms, System: 0.6 ms]
  Range (min … max):     1.6 ms …   2.1 ms    1824 runs

Benchmark 2: awk "/199999/" nums.txt
  Time (mean ± σ):      19.9 ms ±   0.3 ms    [User: 18.4 ms, System: 1.3 ms]
  Range (min … max):    19.5 ms …  21.9 ms    151 runs

Summary
  grep 199999 nums.txt ran
   11.98 ± 0.40 times faster than awk "/199999/" nums.txt

That last line — 11.98 ± 0.40 times faster — is the answer you came for, uncertainty included. Not “grep felt snappier,” but “grep is about twelve times faster, and we’re confident to within a rounding error.” (We’ll explain that -N in a second; it matters more than it looks.)

The first trap: on a fast command, the default measurement is nonsense

Benchmark something that finishes almost instantly and hyperfine will hand you a number and tell you not to trust it. Here’s true, a command whose entire job is to exit successfully:

$ hyperfine 'true'
Benchmark 1: true
  Time (mean ± σ):      10.9 µs ±  30.4 µs    [User: 111.1 µs, System: 87.7 µs]
  Range (min … max):     0.0 µs … 419.8 µs    5766 runs

  Warning: Command took less than 5 ms to complete. Note that the results might
  be inaccurate because hyperfine can not calibrate the shell startup time much
  more precise than this limit. You can try to use the `-N`/`--shell=none`
  option to disable the shell completely.
  Warning: Statistical outliers were detected. Consider re-running this
  benchmark on a quiet system without any interferences from other programs.

Read the numbers before the warnings and you’d walk away believing true runs in 10.9 microseconds — with a minimum of 0.0 µs, which is to say hyperfine measured a run that took no time at all. That’s not physics; that’s the tool telling you it failed. By default hyperfine runs your command through a shell and then subtracts an estimate of the shell’s own startup cost. For anything slower than a few milliseconds that subtraction is noise you can ignore. For true, the thing you’re timing is smaller than the measurement error, and the subtraction produces garbage.

The fix is in the warning: -N (a.k.a. --shell=none) skips the shell entirely and runs the binary directly.

$ hyperfine -N 'true'
Benchmark 1: true
  Time (mean ± σ):     434.6 µs ±  36.4 µs    [User: 264.5 µs, System: 104.9 µs]
  Range (min … max):   393.3 µs … 922.5 µs    6014 runs

Now the number is believable: ~435 microseconds to fork, exec, and reap a process, with no phantom 0.0 µs minimum. The lesson isn’t “hyperfine is wrong about true” — it’s that for fast commands you must pass -N, and for a fair A/B you should pass it to both sides so neither one is paying an invisible shell tax the other isn’t. That’s why the grep-vs-awk comparison above used -N.

The second trap: this machine is noisy, and it never let us forget

Look again at every block above and notice what almost all of them share: Warning: Statistical outliers were detected. We did not cherry-pick that in. On the shared CI runner where our robot works, that warning fired on nearly every benchmark we ran, because a shared virtual machine is exactly the “system with interferences from other programs” the warning describes. Some other tenant’s job schedules on the same physical core, your run stalls for a few milliseconds, and hyperfine — correctly — flags the run as an outlier.

This is the honest heart of the tool, so we’ll say it plainly: hyperfine measures wall-clock time, and wall-clock time on a shared or CI box is a measurement of the whole machine, not your command. The User: and System: figures in the output are CPU time and stay fairly stable; the headline Time (mean ± σ) is real elapsed time and moves with whatever else the box is doing. For an A/B comparison this mostly comes out in the wash — both commands eat the same noise — but for an absolute number (“this endpoint takes 43 ms”) a CI runner will lie to you with a straight face, and hyperfine’s outliers warning is it declining to be complicit.

Two flags take the edge off, and hyperfine names both in the warning:

• --warmup N runs the command N times before timing starts, so a cold disk cache or a JIT warm-up doesn’t get counted as the first “real” run.
• --prepare 'CMD' runs a setup command before every timed run — for example --prepare 'sync' to flush pending writes, or a cache-drop on systems where you can.

$ hyperfine --warmup 3 --prepare 'sync' -N 'wc -l nums.txt'

Neither flag makes a noisy box quiet. They make the comparison fairer by giving both commands the same starting conditions. If you need a genuinely stable absolute number, run it on a machine that isn’t shared — a point no amount of statistics can paper over.

The good parts, once you’ve made peace with the noise

Parameter scans. -L name value1,value2,… reruns the benchmark across a list, substituting {name} into the command. One line to see how something scales:

$ hyperfine --warmup 2 -N -L n 10000,100000,200000 'seq 1 {n}'
...
Summary
  seq 1 10000 ran
    1.88 ± 0.14 times faster than seq 1 100000
    3.69 ± 0.26 times faster than seq 1 200000

Machine-readable export. --export-markdown, --export-json, and --export-csv turn the run into something you can paste into a PR or feed to a script. The Markdown one drops straight into a review:

$ hyperfine --warmup 3 -N --export-markdown bench.md \
    'grep 199999 nums.txt' 'awk "/199999/" nums.txt'
$ cat bench.md

That table is the artifact worth keeping. A “before/after” pair of these in a performance PR is a far better argument than a sentence claiming something got faster.

Where it’s the wrong tool

hyperfine tells you whether A is faster than B and by how much. It cannot tell you why. It has no idea your program spent 80% of its time in one function or blocked on a syscall — it times the process from the outside and hands you a total. When “which is faster” turns into “why is this slow,” you’ve outgrown hyperfine and want a profiler: perf stat/perf record on Linux, valgrind --tool=callgrind, or your language’s own profiler.

It’s also the wrong tool for anything sub-microsecond or tight-loop — timing a single function call thousands of times inside a running program is a microbenchmark harness’s job (criterion, google-benchmark, your language’s Benchmark module), because those measure inside the process instead of paying the fork/exec/shell cost on every iteration. And it is emphatically the wrong tool for absolute performance numbers gathered on CI, for every reason in the section above.

What it costs and the free alternative

It costs nothing — dual MIT/Apache-2.0 licensed, no account, no telemetry, no paid tier. The free alternative is already on your machine: time in a for loop, or bash’s built-in time. The honest trade is statistics versus simplicity. A shell loop gives you numbers; hyperfine gives you a mean, a standard deviation, a sane run count it picked for you, and a relative-speedup summary with error bars — the parts you’d otherwise compute by hand and probably compute wrong. If you benchmark something twice a year, time is fine. If you find yourself comparing two commands and squinting, hyperfine pays for itself the first time it catches a “faster” change that was inside the noise all along.

What made us close the tab

Nothing — it earned a place next to rg and fd. But the caveats are real, in the order they’ll bite you:

• Fast commands need -N. Under the default shell the calibration produces nonsense — a 0.0 µs minimum is the tell. Pass --shell=none, and pass it to both sides of a comparison.
• Wall-clock on a shared/CI box is not your command’s time. The “statistical outliers” warning is the tool being honest, not broken. Trust the relative speedup; distrust absolute numbers unless the machine is quiet.
• It’s a stopwatch, not a profiler. It answers “which is faster,” never “why.” The moment the question becomes “why,” reach for perf or a real profiler.

When it goes wrong: if a number looks too good or too weird, check the three things in order — did you forget -N on a fast command (look for a 0.0 µs min), is the box shared (look for the outliers warning), and are you asking for an absolute number when all hyperfine can honestly give you is a comparison? Add --warmup and --prepare to level the field, and if you need the real answer to “why,” this was never the tool for that — and it never pretended to be.