Process Pools
ProcessPoolExecutor manages a fixed pool of worker processes for you, making it easy to parallelize CPU-bound work across cores without hand-rolling Process objects.
Managing individual Process objects for every unit of work is tedious and wasteful — creating a fresh OS process per task has real startup cost. A process pool solves this by starting a fixed number of worker processes up front and reusing them across many tasks. concurrent.futures.ProcessPoolExecutor is the standard, high-level way to do this in Python.
What Is a Process Pool
A process pool is a set of long-lived worker processes that pull tasks from an internal queue and execute them, returning results back to the caller. Instead of paying process-creation overhead for every task, you pay it once when the pool starts, and each worker is reused for many tasks over its lifetime.
concurrent.futures.ProcessPoolExecutor provides this pattern with a Future-based API shared with ThreadPoolExecutor, so code that submits work and waits for results looks the same whether the underlying pool uses processes or threads — only the concurrency model differs.
ProcessPoolExecutor Basics
.map() distributes the iterable's items across worker processes and returns results in the original order, similar to the builtin map() but running in parallel.
For more control, .submit(fn, *args) returns a Future immediately, letting you submit heterogeneous tasks and collect results as they complete (via as_completed) rather than waiting for all of them in submission order.
from concurrent.futures import ProcessPoolExecutor
def square(n: int) -> int:
return n * n
if __name__ == "__main__":
with ProcessPoolExecutor(max_workers=4) as executor:
results = list(executor.map(square, range(10)))
print(results) # [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]from concurrent.futures import ProcessPoolExecutor, as_completed
def slow_square(n: int) -> int:
return n * n
if __name__ == "__main__":
with ProcessPoolExecutor(max_workers=3) as executor:
futures = {executor.submit(slow_square, n): n for n in range(5)}
for future in as_completed(futures):
n = futures[future]
print(f"square({n}) = {future.result()}")CPU-Bound Workloads
Process pools shine for CPU-bound work: computation-heavy tasks (numeric processing, image/video transforms, compression, parsing) where the bottleneck is the CPU actually executing bytecode. Because each worker process has its own interpreter and its own GIL, ProcessPoolExecutor with max_workers set near the machine's core count can achieve near-linear speedup for embarrassingly parallel CPU-bound tasks — something ThreadPoolExecutor cannot do for pure-Python computation under CPython, because threads there still contend for a single GIL (see the GIL & CPython topic).
For I/O-bound work (network calls, file I/O, waiting on external services), threads or asyncio are usually a better and lighter-weight fit than processes — the bottleneck there is waiting, not computing, so the GIL is not the limiting factor and process overhead is wasted.
Serialization Costs
Every argument sent to a worker and every result sent back must be pickled by the parent and unpickled by the worker (or vice versa) — there is no shared memory by default. For small arguments and results (numbers, short strings, small lists) this cost is negligible. For large objects (big NumPy arrays, large dataframes, deeply nested structures), pickling and unpickling can dominate runtime and even erase the benefit of parallelism.
Each chunk (a large list) must be pickled and sent to a worker; for genuinely huge payloads, consider multiprocessing.shared_memory instead to avoid copying (see the Shared Memory topic).
from concurrent.futures import ProcessPoolExecutor
import time
def process_chunk(chunk: list[int]) -> int:
return sum(x * x for x in chunk)
if __name__ == "__main__":
big_data = list(range(10_000_000))
chunks = [big_data[i::4] for i in range(4)] # 4 large chunks
start = time.perf_counter()
with ProcessPoolExecutor(max_workers=4) as executor:
totals = list(executor.map(process_chunk, chunks))
print(sum(totals), f"in {time.perf_counter() - start:.2f}s")Worker Lifecycle
When a ProcessPoolExecutor is created, it does not necessarily start workers immediately — workers are typically spawned lazily as tasks are submitted, up to max_workers. Once started, a worker process stays alive and keeps pulling tasks from the internal work queue until the pool is shut down (or, since Python 3.11's optional max_tasks_per_child, after handling a configured number of tasks, after which it is recycled).
max_tasks_per_child is useful for isolating memory leaks or corrupted state in long-running worker processes — recycling a worker periodically bounds the damage a single bad task can do to that worker's subsequent tasks.
from concurrent.futures import ProcessPoolExecutor
def work(n: int) -> int:
return n + 1
if __name__ == "__main__":
with ProcessPoolExecutor(max_workers=2, max_tasks_per_child=100) as executor:
results = list(executor.map(work, range(1000)))
print(len(results))Shutdown
Using ProcessPoolExecutor as a context manager (with ... as executor:) ensures .shutdown(wait=True) is called automatically on exit, which waits for all pending tasks to complete and then terminates the worker processes cleanly. Calling .shutdown(wait=False) returns immediately without waiting, and .shutdown(cancel_futures=True) (3.9+) additionally cancels any tasks that haven't started yet.
Forgetting to shut down a pool explicitly (when not using with) can leave zombie worker processes running, or cause the parent program to hang at exit waiting for workers that are still processing a large backlog — always prefer the context-manager form.
Handling Exceptions
Unlike raw multiprocessing.Process, ProcessPoolExecutor does propagate exceptions: if a worker's task raises, calling .result() on the corresponding Future re-raises that exception in the caller (it's pickled and shipped back with the traceback information reconstructed as best as possible).
from concurrent.futures import ProcessPoolExecutor
def risky(n: int) -> int:
if n == 3:
raise ValueError("bad input: 3")
return n * 2
if __name__ == "__main__":
with ProcessPoolExecutor(max_workers=2) as executor:
futures = [executor.submit(risky, n) for n in range(5)]
for f in futures:
try:
print(f.result())
except ValueError as e:
print(f"caught: {e}")Edge Cases and Pitfalls
Only picklable callables and arguments work — bound methods on unpicklable objects, lambdas, and local closures typically fail. Interactive environments (some REPLs, certain notebook setups) can behave inconsistently with spawn-based pools because the target function must be importable from the child, which module-less interactive definitions may not satisfy. Oversubscribing max_workers far beyond the CPU core count for CPU-bound work usually hurts rather than helps, since processes then compete for the same limited cores.
Practical Usage
A common, effective default: set max_workers to os.process_cpu_count() (3.13+) or os.cpu_count() for CPU-bound work, chunk large inputs before submitting (via executor.map(fn, data, chunksize=...)) to reduce per-item serialization overhead, and always wrap top-level pool code in if __name__ == "__main__":.
Complete Example
from concurrent.futures import ProcessPoolExecutor, as_completed
import os
def is_prime(n: int) -> bool:
if n < 2:
return False
for i in range(2, int(n ** 0.5) + 1):
if n % i == 0:
return False
return True
def check(n: int) -> tuple[int, bool]:
return n, is_prime(n)
if __name__ == "__main__":
candidates = range(100_000, 100_050)
with ProcessPoolExecutor(max_workers=os.cpu_count()) as executor:
futures = [executor.submit(check, n) for n in candidates]
primes = sorted(n for future in as_completed(futures)
for n, prime in [future.result()] if prime)
print(primes)Mental Model
Think of a process pool like a small fixed team of contractors rather than hiring and firing a new contractor for every single job. You hand jobs to whichever contractor is free; each does the work independently (their own workspace, no shared desk), and hands finished results back to you. Starting the team costs something upfront, but it's far cheaper per job than repeatedly hiring and dismissing individuals.
Quick Reference
ProcessPoolExecutor(max_workers=N, max_tasks_per_child=M)
executor.map(fn, iterable, chunksize=1) -> results in order
executor.submit(fn, *args) -> Future
future.result() / future.exception() -> blocks for result / error
as_completed(futures) -> yields as each finishes
with ProcessPoolExecutor() as ex: ... -> auto shutdown(wait=True)Key Takeaways
ProcessPoolExecutor reuses a fixed set of worker processes across many tasks, amortizing process-creation overhead.
It suits CPU-bound work, where separate interpreters (and separate GILs) allow genuine multi-core speedup unattainable with threads under CPython.
All arguments and results are pickled across the process boundary — large payloads carry real serialization cost.
Exceptions raised in workers propagate to the caller through future.result(), unlike raw Process objects.
Always use the context-manager form (or explicit .shutdown()) and guard pool-creation code with if __name__ == "__main__":.
with ProcessPoolExecutor(max_workers=64) as ex: ex.map(cpu_bound_fn, huge_range) # 64 workers on an 8-core machine import os with ProcessPoolExecutor(max_workers=os.cpu_count()) as ex: ex.map(cpu_bound_fn, huge_range)
For CPU-bound work, workers beyond the physical core count don't add parallelism -- they just add context-switching and memory overhead, and can slow the workload down.
results = [] for item in huge_list: with ProcessPoolExecutor(max_workers=4) as ex: results.append(ex.submit(process, item).result()) with ProcessPoolExecutor(max_workers=4) as ex: results = list(ex.map(process, huge_list))
Creating a new pool per item pays worker-startup cost repeatedly and processes tasks one at a time anyway; create the pool once and submit all tasks to it.
executor = ProcessPoolExecutor(max_workers=4) future = executor.submit(work, 5) print(future.result()) # executor never shut down -- workers may linger with ProcessPoolExecutor(max_workers=4) as executor: future = executor.submit(work, 5) print(future.result()) # pool automatically shut down on exit
Without the context manager (or an explicit .shutdown()), worker processes may not be cleanly terminated, which can leak resources or cause the program to hang at exit.
Why is ProcessPoolExecutor generally preferred over managing raw Process objects by hand?
It reuses a fixed pool of worker processes across many tasks (avoiding repeated process-creation overhead), provides a Future-based API for submitting work and collecting results, propagates exceptions automatically, and handles clean shutdown as a context manager.
For what kind of workload does a process pool give the biggest benefit, and why?
CPU-bound workloads, because each worker process has its own interpreter and its own GIL, so multiple workers can execute Python bytecode truly in parallel across cores -- something threads under CPython's single GIL cannot do for pure-Python computation.
What cost does every argument and return value incur when crossing into and out of a worker process, and when does it matter most?
Pickling/unpickling (serialization) cost; it matters most with large payloads (big arrays, dataframes, deeply nested structures), where the serialization time can outweigh or erase the parallel speedup.
What does the executor.map chunksize parameter do and why would you increase it?
It groups multiple items into a single unit of work sent to a worker at once, reducing the number of individual submit/serialize round-trips; increasing it helps when tasks are very small and per-task IPC overhead dominates.
How do exceptions raised inside a worker's task reach the calling code?
The exception is captured, pickled, and shipped back with the Future; calling future.result() on that Future re-raises the exception in the caller.
What happens if you set max_workers far higher than the number of physical CPU cores for a CPU-bound task?
Extra worker processes compete for the same limited cores, adding scheduling and memory overhead without increasing effective parallelism, which can make things slower rather than faster.
What does max_tasks_per_child let you do, and why is it useful?
It recycles a worker process after it has handled a given number of tasks; useful for bounding the impact of memory leaks or corrupted state that might otherwise accumulate in a long-lived worker.
Write a function that uses ProcessPoolExecutor to compute the factorial of each number in a list, in parallel.
from concurrent.futures import ProcessPoolExecutor import math def parallel_factorials(numbers: list[int]) -> list[int]: with ProcessPoolExecutor() as executor: return list(executor.map(math.factorial, numbers)) if __name__ == "__main__": print(parallel_factorials([5, 6, 7])) # [120, 720, 5040] executor.map distributes each number to a worker process, which computes math.factorial(n); results come back in the same order as the input list.
What happens when this code runs?
A RuntimeError("boom") is raised when list(results) is evaluated -- executor.map is lazy, and iterating it re-raises any exception a worker encountered for the corresponding input, propagating it to the caller (list() here) rather than swallowing it.
Write code that submits five independent tasks to a process pool and prints each result as soon as it's ready, not necessarily in submission order.
from concurrent.futures import ProcessPoolExecutor, as_completed import time def variable_delay_task(n: int) -> int: time.sleep((5 - n) * 0.1) return n * n if __name__ == "__main__": with ProcessPoolExecutor(max_workers=5) as executor: futures = [executor.submit(variable_delay_task, n) for n in range(5)] for future in as_completed(futures): print(future.result()) as_completed yields each Future as soon as it finishes, regardless of submission order, so tasks that happen to finish sooner (here, higher n, shorter sleep) print first.
from concurrent.futures import ProcessPoolExecutor
def fail_on_two(n):
if n == 2:
raise RuntimeError("boom")
return n
if __name__ == "__main__":
with ProcessPoolExecutor(max_workers=2) as ex:
results = ex.map(fail_on_two, [1, 2, 3])
print(list(results))