Benchmarking
Benchmarking measures how fast code actually runs under controlled, repeatable conditions, using the standard library's timeit module and disciplined methodology to avoid being misled by noise.
Benchmarking answers a narrower question than profiling: not 'where does time go inside this program' but 'how long does this specific piece of code take to run, and is it actually faster than the alternative?' Python's standard library provides the timeit module for this, built to avoid several classic measurement mistakes — but the tool alone doesn't guarantee a valid benchmark; methodology (warm-up, repetition, realistic workloads) matters just as much.
Benchmarking vs Profiling
Profiling (covered separately) breaks down where time is spent inside a program, across many functions, to find hotspots. Benchmarking measures the total wall-clock (or CPU) time of a specific, self-contained piece of code — often to compare two implementations of the same task head to head. They're complementary: profile to find what's slow, benchmark to confirm whether a proposed fix is actually faster.
timeit: The Standard Library's Benchmarking Tool
timeit runs a snippet of code repeatedly and reports the total (or best/average) elapsed time, using a high-resolution timer and disabling the garbage collector during the run by default, both of which reduce noise compared to hand-rolled timing with time.time().
The number= argument controls how many times the statement executes within one timed run; timeit chooses a reasonable default when omitted, but explicit control is usually clearer.
import timeit
duration = timeit.timeit("sum(range(1000))", number=100_000)
print(f"{duration:.4f} seconds for 100,000 runs")Timing Code With Setup
The setup= parameter runs once before timing begins and is not included in the measured time — used for imports or building test data the timed statement depends on.
Since Python 3.5, timeit.timeit also accepts actual callables (not just strings) for both stmt and setup, which avoids needing to write code as text:
import timeit
duration = timeit.timeit(
stmt="sorted(data)",
setup="import random; data = [random.random() for _ in range(1000)]",
number=1000,
)
print(duration)import random
import timeit
def make_data():
return [random.random() for _ in range(1000)]
data = make_data()
duration = timeit.timeit(lambda: sorted(data), number=1000)
print(duration)The timeit Command-Line Interface
timeit is also usable directly from the shell, which is convenient for one-off comparisons without writing a script. The CLI automatically picks a loop count so the total run takes a reasonable amount of time, and reports the best-of-several result.
python -m timeit "sum(range(1000))"
# 100000 loops, best of 5: 12.3 usec per loop
python -m timeit -s "data = list(range(1000))" "sorted(data)"repeat(): Multiple Trials for Statistical Confidence
A single timing run can be skewed by a transient system event (another process using the CPU, an OS scheduling hiccup). timeit.repeat() runs the whole timed measurement multiple times and returns a list of results, letting you take the minimum (the closest to the code's true best-case speed, since noise can only slow a run down, never speed it up below the code's real floor).
import timeit
results = timeit.repeat("sum(range(1000))", number=10_000, repeat=5)
print(results)
print("best:", min(results))Warm-Up Considerations
The very first execution of some code can be slower than subsequent ones for reasons unrelated to the algorithm itself — for example, importing a module, populating internal caches, or (in other Python implementations, notably PyPy) just-in-time compilation warming up. A benchmark that measures only a single, cold first run risks reporting startup cost rather than steady-state performance.
timeit.repeat() naturally provides multiple trials, but for a deliberate warm-up you can run the code once (unmeasured) before starting the timed trials, or simply discard the first result in a repeat() list.
CPython itself does not JIT-compile code (Python 3.13+ has an experimental JIT, still evolving), so classic 'JIT warm-up' concerns are smaller in standard CPython than in PyPy, but caching effects (first file read, first regex compile, lazy imports) still apply and matter.
import timeit
stmt = "expensive_first_call_then_cached()"
setup = "from mymodule import expensive_first_call_then_cached"
timeit.timeit(stmt, setup=setup, number=1) # warm-up, result discarded
results = timeit.repeat(stmt, setup=setup, number=1000, repeat=5)
print("best after warm-up:", min(results))Variance and Statistical Rigor
A benchmark result is a sample, not a fixed truth — rerunning the same code can produce different timings due to system noise (background processes, thermal throttling, other load on a shared machine). Reporting a single number is misleading; reporting the minimum of several repeated trials (as timeit.repeat() facilitates) is the standard practice, since the true 'best case' speed of the code is a more stable, meaningful comparison point than a noisy average.
A large standard deviation relative to the mean is itself informative: it signals the measurement is noisy (perhaps due to system load) and the result should not be trusted as a precise comparison until conditions are controlled better (idle machine, more repeats).
import statistics
import timeit
results = timeit.repeat("sum(range(1000))", number=10_000, repeat=10)
print("min:", min(results))
print("median:", statistics.median(results))
print("stdev:", statistics.stdev(results))Realistic Workloads
A benchmark is only meaningful if its workload resembles the real usage the result will inform. Timing a function on a 10-element list to justify a decision that matters at 10-million-element scale (or vice versa) can produce a conclusion that doesn't hold in practice, because different implementations can have different growth rates (an O(n log n) algorithm that loses to a simpler O(n^2) one only at small n, for example).
Microbenchmark Pitfalls
A microbenchmark isolates a tiny snippet of code, which makes it easy to write but also easy to get misleadingly wrong. Common pitfalls: benchmarking a snippet so small that constant per-call overhead (function call cost, timer resolution) dominates the actual measurement rather than the logic being compared; letting global/module-level state built during setup= accidentally leak into or bias the timed statement; comparing implementations under artificial, uniform conditions (e.g. tiny inputs, no memory pressure, no concurrent load) that don't match production; and being fooled by CPython-specific effects (like small-integer caching or interned strings) that make a microbenchmark faster than the same logic will be on realistic, varied data.
import timeit
# misleading: dict.get vs try/except on a workload where the key almost always exists
# -- results can differ dramatically if the "usually present" assumption doesn't hold
# in the real workload being modeled
setup = "d = {'a': 1}"
try_except = timeit.timeit(
"try:\n d['a']\nexcept KeyError:\n pass",
setup=setup, number=1_000_000,
)
uses_get = timeit.timeit("d.get('a')", setup=setup, number=1_000_000)
print(try_except, uses_get)Comparing Implementations Fairly
When comparing two approaches, keep everything except the thing being measured identical: same input data, same machine, same interpreter, ideally run back-to-back (or interleaved) to average out any drift in system load over time. Report both the minimum and enough context (input size, Python version, hardware) for the comparison to be reproducible and meaningful to someone else.
Third-Party Benchmarking Tools
For benchmarking within a test suite, pytest-benchmark (third-party, installed via pip) integrates statistical benchmarking directly into pytest, automatically handling warm-up, repetition, and producing comparison reports and regression tracking across runs. It's a common choice when benchmarks need to live alongside a project's test suite rather than as standalone scripts.
pip install pytest-benchmarkdef test_sum_performance(benchmark):
result = benchmark(sum, range(1000))
assert result == sum(range(1000))Practical Usage and Best Practices
Use timeit.repeat() and take the minimum, not a single timeit() call. Benchmark on a quiet machine (or account for noise via more repeats), with realistic data sizes and shapes. Warm up caches or lazy imports before timing if that reflects real steady-state usage. State clearly what's being compared and under what conditions, since a benchmark result without that context is not reproducible or trustworthy. Combine with profiling: benchmark first to confirm a real difference exists, profile the slower version to understand why, then benchmark again after a fix to confirm the improvement.
Complete Example
Repeating each variant seven times and comparing the minimums gives a fair, noise-resistant comparison between the two approaches at a realistic input size, rather than trusting a single potentially noisy run of each.
import timeit
def build_with_list_comprehension(n):
return [i * i for i in range(n)]
def build_with_map(n):
return list(map(lambda i: i * i, range(n)))
n = 100_000
setup = f"from __main__ import build_with_list_comprehension, build_with_map; n = {n}"
comprehension_times = timeit.repeat(
"build_with_list_comprehension(n)", setup=setup, number=50, repeat=7,
)
map_times = timeit.repeat(
"build_with_map(n)", setup=setup, number=50, repeat=7,
)
print("list comprehension best:", min(comprehension_times))
print("map best:", min(map_times))Mental Model and Quick Reference
Think of a single timeit() call as one noisy measurement; a trustworthy benchmark is the minimum across several repeated measurements, taken under conditions (input size, warm-up, machine load) that actually resemble what you care about.
timeit.timeit(stmt, setup, number=N) -> total time for N executions
timeit.repeat(stmt, setup, number, repeat=R) -> R independent timing trials
min(results) -> the standard way to summarize repeat() trials
python -m timeit "..." -> quick CLI benchmarking, auto-picks loop count
pytest-benchmark (third-party) -> statistical benchmarking integrated into pytestKey Takeaways
timeit (standard library) is purpose-built for benchmarking small code snippets, using a high-resolution timer and disabling the garbage collector to reduce noise.
Use timeit.repeat() and take the minimum across several trials rather than trusting a single timeit() call, since system noise can only slow a run down, never speed it below the code's real floor.
Warm-up matters when the first execution pays a one-time cost (imports, cache population) not representative of steady-state performance.
Benchmark with realistic input sizes and data shapes; results at one scale don't always generalize to another.
Microbenchmarks are easy to get subtly wrong — watch for per-call overhead dominating the measurement, unrealistic uniform inputs, and CPython-specific effects skewing results.
pytest-benchmark is a third-party tool for integrating rigorous, repeatable benchmarking into a pytest test suite.
import time start = time.time() result = do_work() print(time.time() - start) # single, noisy sample import timeit results = timeit.repeat(lambda: do_work(), number=100, repeat=5) print("best:", min(results))
A single hand-rolled timing measurement is highly susceptible to noise from unrelated system activity; timeit.repeat() with multiple trials, summarized by the minimum, gives a far more reliable result.
results = timeit.repeat("sum(range(1000))", number=1000, repeat=5) print("average:", sum(results) / len(results)) # noise inflates the average results = timeit.repeat("sum(range(1000))", number=1000, repeat=5) print("best:", min(results)) # noise can only slow a run down, not speed it up
Averaging repeated timing trials lets slow outliers (caused by transient system noise) pull the reported number up; the minimum better reflects the code's actual best-case performance, since interference can only add time, never remove it.
Different algorithms have different growth rates; a comparison at an unrealistically small scale can reverse at production scale, so benchmark data should match the real workload.
duration = timeit.timeit("expensive_first_call()", setup="from mymodule import expensive_first_call", number=1) # includes one-time import/cache-population cost in the only measured run timeit.timeit("expensive_first_call()", setup="from mymodule import expensive_first_call", number=1) # warm-up, discarded results = timeit.repeat("expensive_first_call()", setup="from mymodule import expensive_first_call", number=100, repeat=5)
Measuring only a single, cold execution can report one-time setup cost (imports, cache population) rather than the steady-state performance the benchmark is actually meant to characterize.
What is the key difference in purpose between benchmarking and profiling?
Profiling breaks down where time is spent across many functions inside a program to find hotspots; benchmarking measures the total time of a specific, self-contained piece of code, often to compare two implementations directly.
Why does timeit.repeat() summarized by the minimum give a more trustworthy result than averaging the trials or trusting a single run?
System noise (background processes, scheduling) can only slow a run down, never make it faster than the code's actual floor performance; the minimum across several trials is the closest available estimate of that true best-case speed, whereas an average lets slow outliers skew the result upward.
Why might the very first execution of a benchmarked function be slower than later executions, and how do you account for this?
The first call can pay one-time costs like module imports, cache population, or lazy initialization that later calls don't repeat; run the code once (unmeasured) as a warm-up before starting timed trials, or discard the first result from a repeat() list.
Why is benchmarking on a small, unrealistic input size potentially misleading?
Different implementations can have different growth rates (algorithmic complexity), so a comparison that holds at a small scale can reverse at production scale; the benchmark's data should resemble the real workload the result will inform.
What parameter of timeit.timeit() runs code once before timing begins, and why is it not included in the measured duration?
The setup= parameter; it's meant for one-time preparation (imports, building test data) that the code under test depends on but that isn't itself part of what's being measured, so including it in the timed portion would bias the result.
What is pytest-benchmark, and how does it relate to timeit?
It's a third-party pytest plugin that integrates statistical benchmarking (warm-up, repetition, comparison reporting) directly into a test suite; it builds on the same underlying goal as timeit — accurate, repeatable timing — but is meant for benchmarks that live and run alongside a project's tests.
Name two specific ways a microbenchmark can give a misleading result even when using timeit correctly.
Any two of: the snippet is so small that per-call/timer overhead dominates rather than the actual logic; setup= state leaks into or biases the timed statement; the workload/input shape is unrealistic compared to production; CPython-specific effects (like small-integer caching or string interning) make the microbenchmark faster than the same logic would be on real, varied data.
Write code using timeit.repeat() to compare summing a list with a for-loop versus the built-in sum() function, reporting the minimum time for each over 5 repeats of 1000 executions.
import timeit def loop_sum(values): total = 0 for v in values: total += v return total setup = "from __main__ import loop_sum; values = list(range(1000))" loop_times = timeit.repeat("loop_sum(values)", setup=setup, number=1000, repeat=5) builtin_times = timeit.repeat("sum(values)", setup=setup, number=1000, repeat=5) print("loop_sum best:", min(loop_times)) print("sum() best:", min(builtin_times))
Both variants use the same setup (identical data), the same number of executions per trial, and the same number of repeats, isolating the comparison to just the difference between a manual loop and the built-in sum(), and each is summarized by its minimum trial for a noise-resistant comparison.
What does this measure, and why would timing '1 + 1' specifically be a poor way to compare, say, two different sorting algorithms?
It measures the near-negligible cost of evaluating the literal expression 1 + 1 a million times, dominated almost entirely by Python's per-statement execution overhead rather than any meaningful computation. It illustrates that timeit measures exactly the snippet given — to compare sorting algorithms you'd need to time calls to the actual sort functions on representative data, not an unrelated trivial expression.
Write a benchmark comparing string concatenation with '+' in a loop versus ''.join() for building a string from 10,000 short strings, using timeit.repeat with setup and reporting both minimums.
import timeit def concat_with_plus(parts): result = "" for p in parts: result += p return result def concat_with_join(parts): return "".join(parts) setup = ( "from __main__ import concat_with_plus, concat_with_join; " "parts = [str(i) for i in range(10_000)]") plus_times = timeit.repeat("concat_with_plus(parts)", setup=setup, number=50, repeat=5) join_times = timeit.repeat("concat_with_join(parts)", setup=setup, number=50, repeat=5) print("plus best:", min(plus_times)) print("join best:", min(join_times)) join() is expected to win clearly at this scale because repeated '+' concatenation on strings (an immutable type) can require repeatedly allocating and copying into new string objects, while ''.join() builds the result in one pass; the benchmark quantifies that difference on a realistic-sized input rather than assuming it.
What does this print statement's result tell you, and is that conclusion likely to hold for other input sizes without re-benchmarking?
It prints True or False depending on whether the list comprehension's best time was lower than the filter+lambda version's best time (in CPython, the list comprehension is typically faster due to lambda call overhead in filter). Whether this holds at other input sizes isn't guaranteed without re-benchmarking at those sizes, since relative overhead (e.g. per-call lambda cost vs iteration cost) can shift the comparison as data size or shape changes.
import timeit
results = timeit.repeat("1 + 1", number=1_000_000, repeat=3)
print(results)import timeit
setup = "data = list(range(100_000))"
a = timeit.repeat("[x for x in data if x % 2 == 0]", setup=setup, number=100, repeat=5)
b = timeit.repeat("list(filter(lambda x: x % 2 == 0, data))", setup=setup, number=100, repeat=5)
print(min(a) < min(b))