Benchmark builder

When you want to know which of two approaches is faster, a single measurement is not enough: timings vary because of caching, garbage collection, background processes and CPU frequency changes. The benchmark builder lets you record several run times for each task and summarises them with averages and variance so you can compare fairly.

How to use it

1. Create one suite per approach you want to compare, for example Query with index and Query without index.
2. Enter the measured durations for several runs of each, in the same unit.
3. Read the mean and variance for each suite and see which one is fastest and by how much.

Example

Suite A: 120, 118, 125, 119, 121 ms gives a mean of 120.6 ms. Suite B: 98, 140, 101, 97, 99 ms gives a mean of 107 ms but a much larger variance, caused by the 140 ms outlier. B is faster on average, but A is more predictable. Looking at both numbers prevents a misleading conclusion.

Tips for reliable benchmarks

• Warm up first. Discard the first runs, which include JIT compilation, cold caches and connection setup.
• Measure many runs. At least 5 to 10, more for short operations.
• Change one thing at a time so you know what caused the difference.
• Use the same environment for all suites: same machine, same data and same load.
• Watch the variance. If the spread is larger than the difference between means, the result is not conclusive.