Property-Based Testing

Property-based testing flips example-based testing around: instead of picking specific inputs, you describe a general property that must always hold, and a tool generates hundreds of inputs — including adversarial edge cases — to try to break it.

In ordinary example-based testing, you write specific inputs and their expected outputs by hand. Property-based testing instead asks you to describe a property — an invariant that should hold for any valid input — and lets a testing tool generate many varied inputs automatically, searching for a counterexample. In Python this is done almost entirely through the third-party library Hypothesis, which is not part of the standard library.

What Is Property-Based Testing

A property-based test states a rule that should be true for a whole category of inputs, rather than a single input/output pair. For example: 'sorting a list twice gives the same result as sorting it once' or 'encoding then decoding a value returns the original value'. A property-based testing tool then generates many random (and strategically chosen) inputs matching the described shape and checks the property against each one.

Property-based testing complements example-based tests; it does not replace them. Concrete examples still document expected behavior clearly, while property tests explore the space around them.

Example-Based vs Property-Based Testing

The example-based test proves the function works for one hand-picked list. The property-based test proves (probabilistically, across many generated lists including empty lists, duplicates, negative numbers, and large lists) that sorting is idempotent in general.

# example-based: you choose the input
def test_sort_specific_list():
    assert sorted([3, 1, 2]) == [1, 2, 3]

# property-based: the tool chooses many inputs
from hypothesis import given
from hypothesis import strategies as st

@given(st.lists(st.integers()))
def test_sort_is_idempotent(values):
    once = sorted(values)
    twice = sorted(once)
    assert once == twice

Hypothesis: A Third-Party Tool

Hypothesis is a third-party library, installed with pip install hypothesis, that integrates with pytest (and unittest) via the @given decorator. It is the dominant property-based testing tool in the Python ecosystem, inspired by Haskell's QuickCheck, but it is not part of the standard library and must be installed like any other dependency.

pip install hypothesis

Writing a Basic Hypothesis Test

The @given decorator marks a test function as property-based. Its arguments describe strategies — specifications of what kind of data to generate — and Hypothesis calls the test function repeatedly with generated values matching those strategies.

Running this with pytest executes test_addition_is_commutative many times (100 by default) with different generated integer pairs, including edge values like 0 and very large/negative numbers.

from hypothesis import given
from hypothesis import strategies as st

def add(a: int, b: int) -> int:
    return a + b

@given(st.integers(), st.integers())
def test_addition_is_commutative(a, b):
    assert add(a, b) == add(b, a)

Strategies: Describing the Input Space

A strategy describes the shape and constraints of generated data. Strategies compose: you can build strategies for lists, dictionaries, tuples, and custom objects out of simpler strategies.

from hypothesis import strategies as st

st.integers()                          # any int
st.integers(min_value=0, max_value=100)  # bounded ints
st.text()                              # arbitrary unicode strings
st.lists(st.integers(), min_size=1)     # non-empty lists of ints
st.dictionaries(st.text(), st.integers())
st.tuples(st.integers(), st.booleans())
st.one_of(st.integers(), st.text())     # a value that could be either type

Invariants Worth Testing

Good properties come in a few recurring shapes: round-trip ('decode(encode(x)) == x'), invariance ('sorting twice equals sorting once'), symmetry ('reverse(reverse(x)) == x'), and comparison against a simpler reference implementation (a naive but obviously-correct version of the same function).

import json
from hypothesis import given
from hypothesis import strategies as st

json_values = st.recursive(
    st.none() | st.booleans() | st.floats(allow_nan=False) | st.text(),
    lambda children: st.lists(children) | st.dictionaries(st.text(), children),
)

@given(json_values)
def test_json_round_trip(value):
    assert json.loads(json.dumps(value)) == value

Shrinking and Edge-Case Discovery

When Hypothesis finds an input that breaks a property, it does not just report that large, random-looking input. It automatically shrinks the failing example — searching for the smallest, simplest input that still fails — before reporting it, which makes debugging far easier than a raw random counterexample would be.

Hypothesis found and shrank the failure down to the simplest possible counterexample: an empty list — exactly the edge case a human tester might forget to try by hand.

def buggy_max(values):
    result = values[0]
    for v in values:
        if v > result:
            result = v
    return result  # bug: raises IndexError on an empty list

@given(st.lists(st.integers()))
def test_buggy_max_matches_builtin(values):
    if values:
        assert buggy_max(values) == max(values)

Falsifying example: test_buggy_max_matches_builtin(
    values=[],
)
IndexError: list index out of range

Reproducibility

Hypothesis prints the exact failing example when a test fails, and it also maintains a local example database so that a discovered failure is retried first on subsequent runs, keeping a fixed bug reproducible instead of only appearing occasionally at random.

Practical Usage and When to Reach for It

Property-based testing shines for pure functions with clear mathematical or structural properties: parsers, serializers, data structure operations, mathematical utilities, and anything with an inverse operation. It is less useful for code whose correctness is defined purely by a single expected business rule with no general invariant (in which case an example-based test is simpler and clearer).

Best Practices and Common Pitfalls

Keep strategies as tightly scoped as the real input domain (e.g. bound integers if negative values are meaningless in context) so Hypothesis spends its search budget on realistic inputs. Avoid properties so weak they're trivially true (e.g. 'the function does not crash' is a start, but a property that also checks the actual output is stronger). Combine @given with pytest fixtures carefully — fixtures with function scope are re-run for the whole batch of generated examples, not once per generated value, which can be surprising.

Because Hypothesis calls the test function potentially hundreds of times, avoid slow operations (real network calls, real file I/O) inside a property test.

Complete Example

This single property test — pushing a sequence of values and popping them back in reverse order should reproduce the sequence exactly, leaving the stack empty — replaces dozens of hand-written example cases and would automatically catch an off-by-one error, a wrong pop order, or an issue with duplicate values.

# stack.py
class Stack:
    def __init__(self):
        self._items = []

    def push(self, item):
        self._items.append(item)

    def pop(self):
        return self._items.pop()

    def is_empty(self):
        return len(self._items) == 0


# test_stack.py
from hypothesis import given
from hypothesis import strategies as st
from stack import Stack

@given(st.lists(st.integers()))
def test_push_then_pop_returns_last_pushed(values):
    s = Stack()
    for v in values:
        s.push(v)
    for expected in reversed(values):
        assert s.pop() == expected
    assert s.is_empty()

Mental Model and Quick Reference

Think of Hypothesis as an automated adversary: instead of you guessing which inputs might break your code, you describe the space of valid inputs and Hypothesis actively searches that space for a counterexample, then hands you the smallest one it can find.

@given(strategy1, strategy2, ...)   -> mark a property-based test
st.integers(), st.text(), st.lists(...) -> building-block strategies
st.recursive(...), st.composite     -> building complex/custom strategies
shrinking                          -> automatic reduction to the minimal failing case
example database                    -> failures are remembered and retried first

Key Takeaways

Property-based testing checks a general rule against many generated inputs, instead of checking specific hand-picked input/output pairs.

Hypothesis is the dominant Python tool for this and is a third-party library (pip install hypothesis), not part of the standard library.

Strategies (st.integers, st.lists, st.text, etc.) describe the shape of generated data and compose to build complex inputs.

Good properties include round-trips, invariance under repeated operations, symmetry, and comparison against a reference implementation.

Hypothesis shrinks failing examples down to the smallest reproducible counterexample automatically, which is often exactly the edge case a human would miss.

Property-based tests complement, rather than replace, example-based tests.

@given(st.lists(st.integers())) def test_no_crash(values): sort_algorithm(values) # only checks it doesn't raise; verifies nothing about correctness

@given(st.lists(st.integers())) def test_sort_matches_builtin(values): assert sort_algorithm(values) == sorted(values)

A property that only checks the code doesn't crash is much weaker than one that checks the actual result against a known-correct reference; prefer comparing to a trusted implementation or checking a real invariant.

@given(st.integers()) def test_average(n): avg = compute_average([n, n, n]) assert avg == n # true, but this only ever tests lists of three identical values

@given(st.lists(st.floats(allow_nan=False, allow_infinity=False), min_size=1)) def test_average_within_range(values): avg = compute_average(values) assert min(values) <= avg <= max(values)

A strategy that only generates a narrow, unrealistic shape of input (three copies of one integer) barely exercises the function; describe the strategy to match the real domain of valid inputs.

@given(st.lists(st.integers())) def test_process(values): response = requests.post("https://real-api.example.com/process", json=values) assert response.status_code == 200

@given(st.lists(st.integers())) def test_process(values): result = process_locally(values) # pure function, no network call assert result == expected_from_reference(values)

Hypothesis calls the test function up to hundreds of times per run; real network or disk I/O inside a property test makes the suite slow and flaky, and couples correctness to an external service's availability.

How does property-based testing differ fundamentally from example-based testing?

Example-based testing checks specific, hand-chosen input/output pairs; property-based testing describes a general rule (an invariant) that should hold for a whole category of inputs, and a tool generates many varied inputs to check the rule holds.

What does it mean that Hypothesis is a third-party tool, and what must you do to use it?

It is not part of the Python standard library; it must be installed separately with pip install hypothesis before it can be imported and used with @given.

What is 'shrinking' in Hypothesis, and why is it valuable?

When a property fails, Hypothesis automatically searches for the smallest, simplest input that still causes the failure, rather than reporting the original (possibly large or complex) random input — this makes the failing case much easier to understand and debug.

Give an example of a round-trip property and explain what it verifies.

assert decode(encode(x)) == x for a serialization function verifies that encoding and then decoding a value always reconstructs the original value exactly, for any value the strategy can generate — not just for a few examples chosen by hand.

Why might a property-based test with the strategy st.integers(min_value=0, max_value=10) be less useful than st.integers() for testing a general-purpose function?

A narrowly bounded strategy explores only a small slice of the real input space, potentially missing bugs that only appear with negative numbers, zero, or very large values — the strategy should reflect the actual domain the function must handle correctly.

Why is 'the function does not raise an exception' usually considered a weak property?

It says nothing about whether the output is correct; a function that always returns the wrong answer without crashing would still pass such a test, so properties should check actual expected relationships in the output whenever possible.

What kind of code tends to benefit most from property-based testing?

Pure functions with clear structural or mathematical invariants — parsers, serializers/deserializers, data structure operations, sorting/searching algorithms, and mathematical utilities — where round-trip, invariance, or reference-implementation properties are natural to state.

Write a Hypothesis property test asserting that reversing a list twice returns the original list, for any list of integers.

from hypothesis import given from hypothesis import strategies as st @given(st.lists(st.integers())) def test_double_reverse_is_identity(values): assert list(reversed(list(reversed(values)))) == values st.lists(st.integers()) generates lists of varying length and content (including the empty list), and the property checks that reversing twice always reconstructs the original order for every generated list.

Will this test pass every time it runs, and if not, why?

It will fail intermittently, because st.lists(st.integers()) can generate an empty list, and first_element([]) raises IndexError. Hypothesis will find this counterexample, shrink it to the empty list, and report it as the minimal failing case.

Write a property test that checks a custom clamp(value, low, high) function always returns a result between low and high inclusive, for any integers where low <= high.

from hypothesis import given from hypothesis import strategies as st def clamp(value: int, low: int, high: int) -> int: return max(low, min(value, high)) @given(st.integers(), st.integers(), st.integers()) def test_clamp_stays_in_range(value, low, high): if low > high: low, high = high, low result = clamp(value, low, high) assert low <= result <= high

The test swaps low/high when out of order so the precondition (low <= high) always holds before checking the invariant, then asserts the clamped result always falls within the bounds regardless of what value Hypothesis generates.

Is this property true for all strings Hypothesis can generate, including empty strings and unicode text?

Yes — string concatenation length is additive for any two strings regardless of content, length, or unicode characters, so this property holds for every input Hypothesis generates and the test will always pass.

from hypothesis import given
from hypothesis import strategies as st

def first_element(values):
    return values[0]

@given(st.lists(st.integers()))
def test_first_element(values):
    assert first_element(values) == values[0]

from hypothesis import given
from hypothesis import strategies as st

@given(st.text(), st.text())
def test_concat_length(a, b):
    assert len(a + b) == len(a) + len(b)