Iterator Tools

The standard library's itertools module is a toolbox of fast, memory-efficient building blocks for combining, slicing, and generating iterators - the functional-programming complement to generators.

itertools is a standard library module providing a set of fast, composable functions that return iterators - implementing common iteration patterns (chaining, infinite counting, combinatorics, grouping) without building intermediate lists. Every function is lazy: nothing is computed until you iterate.

Why itertools

Loops that chain sequences, generate infinite counters, or compute permutations are easy to write by hand but easy to get subtly wrong or inefficient. itertools provides tested, C-implemented (in CPython) equivalents that stay lazy and compose cleanly with each other and with generators.

itertools is part of the Python standard library, distinct from third-party functional libraries like more-itertools (which extends it with additional recipes).

chain(): Concatenating Iterables

itertools.chain(*iterables) yields items from each iterable in sequence, as if they were one continuous iterable, without copying any of them into a new structure.

from itertools import chain

a = [1, 2]
b = (3, 4)
c = 'xy'
print(list(chain(a, b, c)))   # [1, 2, 3, 4, 'x', 'y']

islice(): Lazy Slicing

Regular slicing (seq[1:5]) requires a sequence that supports indexing; islice(iterable, start, stop, step) slices any iterable, including infinite ones, without needing __getitem__ or len().

islice consumes the underlying iterator as it slices - it does not support negative indices, since that would require knowing the length in advance.

from itertools import islice, count

print(list(islice(range(20), 2, 10, 3)))   # [2, 5, 8]
print(list(islice(count(0), 5)))            # [0, 1, 2, 3, 4] - first 5 of an infinite counter

Infinite Iterators: count, cycle, repeat

Three functions generate infinite streams, always paired with something that limits consumption (islice, a break, zip with a finite iterable, etc.).

from itertools import count, cycle, repeat, islice

print(list(islice(count(10, 2), 4)))     # [10, 12, 14, 16] - start=10, step=2
print(list(islice(cycle('AB'), 5)))       # ['A','B','A','B','A'] - repeats the sequence
print(list(repeat('x', 3)))                # ['x', 'x', 'x'] - repeat with a limit
print(list(islice(repeat('y'), 2)))        # ['y', 'y'] - repeat() with no limit is infinite too

Combinatorics: combinations, permutations, product

These three generate combinatorial arrangements of input elements, each lazily and without duplicating elements - only tuples of positions/values are produced.

combinations picks subsets without regard to order (and without an element paired with itself twice); permutations considers order significant; product computes the cartesian product, optionally of a single iterable repeated via repeat=n.

A quick reference for how the three combinatorial functions differ in what they allow.

from itertools import combinations, permutations, product

print(list(combinations('ABC', 2)))    # [('A','B'), ('A','C'), ('B','C')] - order doesn't matter, no repeats
print(list(permutations('ABC', 2)))    # [('A','B'), ('A','C'), ('B','A'), ('B','C'), ('C','A'), ('C','B')] - order matters
print(list(product('AB', repeat=2)))   # [('A','A'), ('A','B'), ('B','A'), ('B','B')] - cartesian product

combinations('ABC', 2)   AB AC BC             (unordered, no repeats)
permutations('ABC', 2)   AB AC BA BC CA CB    (ordered, no repeats)
product('AB', repeat=2)  AA AB BA BB          (ordered, repeats allowed)

groupby(): Grouping Consecutive Items

groupby(iterable, key=None) groups *consecutive* items sharing the same key into sub-iterators. It does NOT group across the whole iterable like a dict would - input must already be sorted/grouped by the key for the result to make sense.

The final group of 1 is separate from the first because groupby only merges consecutive equal items - it has no memory of earlier groups.

To group all items sharing a key regardless of position, sort the data by that key first (sorted(data, key=...)) before passing it to groupby.

from itertools import groupby

data = [1, 1, 2, 2, 2, 3, 1]
for key, group in groupby(data):
    print(key, list(group))
# 1 [1, 1]
# 2 [2, 2, 2]
# 3 [3]
# 1 [1]   <- a *second* group of 1s, because it's not adjacent to the first

Consuming groupby Groups Immediately

Each group returned by groupby is itself a lazy iterator that becomes invalid once you advance to the next group - so you must fully consume (or materialize) a group before moving on.

Wrapping each group in list(...) immediately, inside the loop, avoids the trap of storing group iterators to use later - by then they'd be stale.

from itertools import groupby

data = sorted(['apple', 'avocado', 'banana', 'blueberry', 'cherry'], key=lambda w: w[0])
result = {key: list(group) for key, group in groupby(data, key=lambda w: w[0])}
print(result)   # {'a': ['apple', 'avocado'], 'b': ['banana', 'blueberry'], 'c': ['cherry']}

The Recipes Concept

The itertools documentation includes a section of "recipes" - short recommended compositions of the core functions (like pairwise, sliding_window, flatten, unique_everseen) that solve common problems but aren't (or weren't) built-in functions themselves. Some recipes have since been promoted into the module itself.

pairwise started life as a documented recipe and became a real itertools function in 3.10 - a good example of how the recipes concept feeds back into the module over time.

from itertools import pairwise   # promoted to itertools in Python 3.10

print(list(pairwise([1, 2, 3, 4])))   # [(1, 2), (2, 3), (3, 4)]

Composing itertools Functions

The real power of itertools is composition - feeding one lazy tool into another to build a pipeline that never materializes an intermediate list.

from itertools import chain, islice, count

# first 5 even numbers >= 100, followed by the literal values -1, -2
pipeline = islice(chain((n for n in count(100) if n % 2 == 0), [-1, -2]), 7)
print(list(pipeline))   # [100, 102, 104, 106, 108, 110, 112]

Practical Usage

Common real-world uses: chain to treat several lists/files as one stream; islice to peek at or paginate a large/infinite generator; product to enumerate all parameter combinations for testing; combinations to generate candidate pairs (e.g. for comparing all items); groupby after sorting to bucket records by a field, without building a full dict of lists by hand.

Edge Cases

permutations(iterable, r) and combinations(iterable, r) treat elements by position, not by value equality - if the input has duplicate values, the output may contain what look like duplicate tuples. product with an empty iterable among its arguments yields nothing at all.

from itertools import combinations, product

print(list(combinations('AAB', 2)))   # [('A','A'), ('A','B'), ('A','B')] - positional, not value-deduplicated
print(list(product([1, 2], [])))       # [] - any empty input makes the whole product empty

Key Takeaways

itertools is a standard library module of fast, lazy, composable iterator-building functions.

chain concatenates iterables; islice lazily slices any iterable, including infinite ones.

count, cycle, and repeat produce infinite (or repeat-limited) streams; always pair them with a limiter.

combinations, permutations, and product cover the standard combinatorics needs, differing in whether order and repetition matter.

groupby only groups consecutive equal keys - sort first if you need full grouping - and each group must be consumed before moving to the next.

from itertools import groupby data = ['b', 'a', 'b', 'a'] for key, group in groupby(data): print(key, list(group)) # treats each element as its own group - not the grouping most people expect from itertools import groupby data = sorted(['b', 'a', 'b', 'a']) for key, group in groupby(data): print(key, list(group)) # ['a','a'] then ['b','b'] - correctly grouped because input is sorted first groupby only merges consecutive equal items. Sort the data by the grouping key first if you want all matching items grouped together.

groups = [] for key, group in groupby(sorted(data)): groups.append(group) # storing the raw group iterator for g in groups: print(list(g)) # later groups may already be exhausted/invalid groups = [] for key, group in groupby(sorted(data)): groups.append(list(group)) # materialize immediately for g in groups: print(g)

A groupby group is a lazy view tied to the parent iterator's current position; it must be consumed (or copied into a list) before advancing to the next group.

for x in itertools.cycle([1, 2, 3]): print(x) # runs forever - no limiter from itertools import cycle, islice for x in islice(cycle([1, 2, 3]), 10): print(x) # bounded to 10 items count, cycle, and repeat() (without a stop count) are infinite; always combine them with islice, a break, or zip against a finite iterable.

What is the key difference between itertools.combinations and itertools.permutations?

combinations treats order as irrelevant (AB and BA are the same combination, so only one appears) while permutations treats order as significant (both AB and BA appear as distinct results).

Why does groupby sometimes produce more groups than you expect?

Because it only merges consecutive items with equal keys; if equal items are not adjacent in the input, they form separate groups. Sorting by the key first fixes this.

What must you do before consuming the next group from groupby's output?

Fully consume (e.g. list()) the current group, since each group is a lazy view onto the same underlying iterator and becomes invalid once groupby advances past it.

How does itertools.islice differ from ordinary sequence slicing (seq[a:b])?

islice works on any iterable, including ones without __getitem__/len() such as generators or infinite iterators, and does not support negative indices since the length isn't known in advance; ordinary slicing requires a sequence type.

Why must count(), cycle(), and repeat() (without a count) always be paired with something like islice or a break?

Because they produce infinite streams; without a limiting mechanism, iterating over them directly (e.g. in a plain for loop or list()) never terminates.

What does itertools.product([1,2], []) return, and why?

It returns an empty iterator - product computes the cartesian product of all its inputs, and if any input is empty, there are zero possible combinations overall.

Interview: what does the 'recipes' section of the itertools documentation represent, and give an example of one that became a real function?

It's a set of documented, recommended compositions of the core itertools functions for common tasks that aren't built-in themselves; pairwise() is an example that started as a recipe and was promoted into itertools proper in Python 3.10.

Write a function using itertools that returns all unique unordered pairs from a list.

from itertools import combinations def unique_pairs(items: list) -> list[tuple]: return list(combinations(items, 2)) print(unique_pairs([1, 2, 3])) # [(1, 2), (1, 3), (2, 3)] combinations(items, 2) generates every 2-element subset exactly once, with no regard to order and no element paired with itself.

What does this print?

[3, 6, 9, 12] count(1) generates 1, 2, 3, ...; the generator expression filters to multiples of 3 (3, 6, 9, 12, 15, ...); islice takes the first 4 of that lazily generated, otherwise-infinite stream.

Using groupby, write code that groups a list of words by their first letter, given the list is already sorted appropriately.

from itertools import groupby words = ['ant', 'apple', 'bear', 'bee', 'cat'] grouped = {key: list(group) for key, group in groupby(words, key=lambda w: w[0])} print(grouped) # {'a': ['ant', 'apple'], 'b': ['bear', 'bee'], 'c': ['cat']}

Because words is already sorted by first letter, consecutive items share keys correctly; each group is materialized into a list immediately inside the dict comprehension.

from itertools import islice, count
result = list(islice((n for n in count(1) if n % 3 == 0), 4))
print(result)