Generators

Generators let you write iterator logic as plain, readable function code — Python pauses and resumes the function for you, producing values lazily instead of building a whole sequence in memory.

A generator function looks like a normal function but contains at least one yield expression. Calling it doesn't run the body - it returns a generator object, an iterator that runs the function's code incrementally, pausing at each yield and resuming exactly where it left off on the next request for a value.

yield and Generator Functions

Any function containing yield anywhere in its body is a generator function. Calling it does not execute any code immediately - it returns a generator object implementing the iterator protocol.

Calling count_up_to(3) does not run any of the function body yet; the body only advances on each next() call.

def count_up_to(n: int):
    i = 1
    while i <= n:
        yield i
        i += 1

gen = count_up_to(3)
print(type(gen))    # <class 'generator'>
print(next(gen))     # 1
print(next(gen))     # 2
print(next(gen))     # 3
next(gen)             # raises StopIteration

Lazy Computation and State Preservation

Each yield suspends the function, preserving all local variables, the instruction pointer, and any open loops or try blocks - exactly where execution paused. The next next() call resumes right after that yield. Values are produced one at a time, on demand, rather than all at once.

Nothing prints until the first next() call - proof that the function body is genuinely paused, not pre-run.

Because values are produced lazily, a generator can represent an infinite or very large sequence using constant memory - contrast with a list comprehension, which materializes every element up front.

def trace():
    print('start')
    yield 1
    print('resumed after first yield')
    yield 2
    print('resumed after second yield')

g = trace()
print(next(g))   # prints 'start', then 1
print(next(g))   # prints 'resumed after first yield', then 2

for Loops over Generators

Generators are iterators, so for loops, list(), sum(), and every other iteration-consuming construct work on them directly - no special handling needed.

def squares(n: int):
    for i in range(n):
        yield i * i

for sq in squares(5):
    print(sq)   # 0 1 4 9 16

Generator Expressions

A generator expression is the lazy sibling of a list comprehension - same syntax with parentheses instead of brackets - producing a generator object instead of a fully built list.

Prefer a generator expression when you only need to consume values once in a pipeline (e.g. feeding sum(), any(), or another loop) and don't need random access or a stored list.

squares_list = [x * x for x in range(1_000_000)]   # builds 1,000,000 ints now
squares_gen = (x * x for x in range(1_000_000))     # builds nothing yet

print(sum(squares_gen))   # computed lazily, one value at a time, minimal memory

send(): Two-Way Communication

yield is an expression, not just a statement - it can receive a value. generator.send(value) resumes the generator, making that yield expression evaluate to value, then runs until the next yield (or return).

A generator must be 'primed' with next() (or send(None)) once before the first real send(); sending a non-None value to a not-yet-started generator raises TypeError.

def running_total():
    total = 0
    while True:
        amount = yield total
        total += amount

gen = running_total()
next(gen)                 # prime the generator - runs to the first yield, returns 0
print(gen.send(10))        # total becomes 10, yields 10
print(gen.send(5))         # total becomes 15, yields 15

throw() and close()

generator.throw(exc) raises exc at the point of the paused yield, letting the generator handle or propagate it. generator.close() raises GeneratorExit inside the generator, which should let it clean up (often via finally) and stop; further next() calls then raise StopIteration.

Generators used as context-manager or resource cleanup helpers should catch GeneratorExit or use finally so close() (called automatically on garbage collection) releases resources deterministically.

def resource_gen():
    try:
        yield 'using resource'
    except ValueError:
        print('handled a ValueError from outside')
        yield 'recovered'
    finally:
        print('cleaning up')

g = resource_gen()
print(next(g))             # 'using resource'
print(g.throw(ValueError))  # prints 'handled...', then 'recovered'
g.close()                    # prints 'cleaning up'

yield from

yield from subiterable delegates to another iterable (often another generator), yielding all of its values as if they were yielded directly, and also forwarding send(), throw(), and close() calls through to it. It also makes the delegating generator's expression evaluate to the sub-generator's return value.

Without yield from, delegating manually would require an explicit loop re-yielding each value from inner() - yield from does this correctly, including forwarding StopIteration's return value.

def inner():
    yield 1
    yield 2
    return 'inner done'

def outer():
    result = yield from inner()
    yield f'inner returned: {result}'

print(list(outer()))   # [1, 2, 'inner returned: inner done']

return in a Generator

A bare return (or return value) inside a generator function ends iteration - it raises StopIteration, and if a value was given, that value becomes StopIteration.value, retrievable via yield from or by catching the exception manually.

def gen():
    yield 1
    return 'done'

g = gen()
next(g)
try:
    next(g)
except StopIteration as e:
    print(e.value)   # 'done'

Practical Usage: Pipelines and Large Data

Generators shine for processing streams or large files line-by-line, chaining lazy transformation stages, and representing infinite sequences - all without loading everything into memory at once.

This pipeline never holds the whole file in memory - each line flows through all three generator stages before the next line is even read.

def read_lines(path: str):
    with open(path) as f:
        for line in f:
            yield line.rstrip('\n')

def non_empty(lines):
    for line in lines:
        if line.strip():
            yield line

def upper(lines):
    for line in lines:
        yield line.upper()

# Each stage processes one line at a time end-to-end - constant memory
# for stage in upper(non_empty(read_lines('data.txt'))):
#     print(stage)

Mental Model

Think of a generator as a function with a bookmark: calling it doesn't run it, it just creates the bookmark at the top. Each next() moves the bookmark forward to the next yield, handing back one value and freezing everything else exactly in place until asked to continue.

Quick Reference

def f():          generator function (contains yield)
f()               calling it returns a generator object; body not run yet
next(g)            resume until next yield; returns the yielded value
g.send(v)          resume, making the paused yield evaluate to v
g.throw(exc)       raise exc at the paused yield
g.close()          raise GeneratorExit at the paused yield; stops iteration
yield from sub     delegate iteration (and send/throw/close) to sub
return v           ends the generator; StopIteration.value == v
(x for x in it)    generator expression - lazy comprehension

Key Takeaways

A function containing yield is a generator function; calling it returns a generator object without running the body.

yield pauses execution and preserves all local state; the next next() call resumes right after it.

Generators compute lazily, one value at a time, which makes them memory-efficient for large or infinite sequences.

send()/throw()/close() give two-way communication and controlled early termination.

yield from delegates to a sub-iterable, forwarding values and propagating its return value.

def make_list(n): return [i for i in range(n)] data = make_list(10_000_000) # builds the whole list in memory def make_gen(n): for i in range(n): yield i data = make_gen(10_000_000) # produces values lazily, on demand

When you only need to iterate once over a large sequence, a generator avoids materializing the entire thing in memory.

gen = running_total() gen.send(10) # TypeError: can't send non-None value to a just-started generator gen = running_total() next(gen) # prime it first gen.send(10)

A generator must be advanced to its first yield (via next() or send(None)) before you can send() a real value into it.

def gen(): yield 1 yield 2 g = gen() values = list(g) more = list(g) # [] - silently empty, easy to miss def gen(): yield 1 yield 2 g_list = list(gen()) # materialize once if you need it more than once values = g_list more = g_list

Generators are single-use, unlike lists. If you need to iterate multiple times, either call the generator function again or convert the result to a list once.

What happens when you call a generator function?

Nothing in the function body runs; Python immediately returns a generator object. The body only starts executing on the first call to next() (or an equivalent, like iterating in a for loop).

Why are generators more memory-efficient than building a full list for large sequences?

Because they produce values lazily, one at a time, holding only the state needed to compute the next value - rather than allocating storage for every element up front.

What is the difference between yield and return inside a generator?

yield pauses the function and produces a value, preserving state to resume later; return (with or without a value) ends the generator entirely, raising StopIteration, with the return value stored on StopIteration.value.

What does generator.send(value) do, and why must you call next() first?

send(value) resumes the generator and makes the paused yield expression evaluate to value, then runs to the next yield. You must first advance the generator to its initial yield with next() (or send(None)); sending a real value before that raises TypeError since there's no paused yield expression to receive it yet.

What does yield from do that a plain for-loop-with-yield cannot do as concisely?

yield from delegates iteration to a sub-iterable and also transparently forwards send()/throw()/close() calls into it, and makes the yield from expression evaluate to the sub-generator's return value - a manual for loop only forwards yielded values.

Interview: why can a generator represent an infinite sequence safely, while a list comprehension cannot?

A generator computes and yields one value per next() call without ever holding the whole sequence; a list comprehension eagerly builds the entire collection before returning, so an infinite source would run forever without completing (or exhaust memory).

What is the relationship between generator expressions and list comprehensions?

They share nearly identical syntax (parentheses vs brackets) and semantics, but a generator expression produces a lazy generator object instead of eagerly building a list - useful when the values only need to be consumed once.

Write a generator function fibonacci() that yields the Fibonacci sequence indefinitely (0, 1, 1, 2, 3, 5, ...).

def fibonacci(): a, b = 0, 1 while True: yield a a, b = b, a + b from itertools import islice print(list(islice(fibonacci(), 8))) # [0, 1, 1, 2, 3, 5, 8, 13]

The generator never terminates on its own (infinite while True loop), so it must be limited by a consumer such as itertools.islice or a break condition in the caller.

What does this print?

1 11 101 next(g) runs to the first yield, producing 1. g.send(10) makes the first yield evaluate to 10 (so x=10), runs to the second yield with x+1=11. g.send(100) makes that yield evaluate to 100 (y=100), runs to the third yield with y+1=101.

Write a generator flatten(nested) that yields every non-list item from an arbitrarily nested list of lists, using yield from.

from collections.abc import Iterable def flatten(nested): for item in nested: if isinstance(item, list): yield from flatten(item) else: yield item print(list(flatten([1, [2, 3, [4, [5]], 6]]))) # [1, 2, 3, 4, 5, 6] yield from recursively delegates to flatten(item) for nested lists, flattening arbitrary depth while yielding scalar items directly.

def gen():
    x = yield 1
    y = yield x + 1
    yield y + 1

g = gen()
print(next(g))
print(g.send(10))
print(g.send(100))