Pipeline, Stream, and Composition Paradigms

A pipeline divides work into stages, each consuming input and producing output for the next stage.

15.1 Pipeline programming

A pipeline divides work into stages, each consuming input and producing output for the next stage. Pipelines appear in shell commands, ETL, compilers, image processing, and streaming systems.

Output (Ubuntu 22.04 on WSL2) for a 4-line sample log with two 500 responses:

This illustrative shell pipeline counts lines containing the text 500 . For production log analysis, use a parser that understands the log format rather than assuming a substring always identifies an HTTP 500 status.

cat access.log | grep ' 500 ' | wc -l

15.2 Functional pipelines

Output (CPython 3.13.1, with print(large_totals) appended):

This list comprehension filters orders and projects totals. A pipeline can improve readability when each transformation has a clear name and contract.

Verified run: with print(large_totals) appended, CPython 3.13.1 printed [120, 80] — the orders with totals of at least 80, in their original order.

orders = [
    {"customer": "A", "total": 40},
    {"customer": "B", "total": 120},
    {"customer": "C", "total": 80},
]

large_totals = [
    order["total"]
    for order in orders
    if order["total"] >= 80
]

[120, 80]

15.3 Streaming versus batch processing

Batch: process a bounded collection, often on a schedule.

Stream: process data as it arrives, potentially continuously.

Micro-batch: process small windows or batches to balance latency and throughput.

Streaming systems must define event time versus processing time, late events, ordering, replay, state retention, and delivery semantics.

15.4 Pipeline error handling

Decide whether a failed record stops the entire pipeline, is retried, is quarantined, or is skipped with an audit event. Silent data loss is usually unacceptable for important business workflows.

15.5 Backpressure and bounded buffers

Every stage should have a capacity and overload policy. A slow downstream stage can otherwise cause memory growth or latency spikes.

Exercise

Create a pipeline that validates course data, normalizes fields, rejects invalid rows, deduplicates records, and produces a report. Include a strategy for malformed input and partial failure.

Quiz

? What did the shell pipeline `cat access.log | grep ' 500 ' | wc -l` print for a 4-line log with two HTTP 500 responses?
+ 2 :: A real run on WSL2 printed 2 — each stage transforms the stream for the next.
- 4 :: `grep` keeps only matching lines.
- 500 :: `wc -l` counts lines.
- 0 :: Two lines contain " 500 ".

? Which statements about stream processing are correct? (select all that apply)
+ It processes data as it arrives, potentially forever. :: That is the definition.
+ It must define event time versus processing time and late events. :: Arrival order differs from occurrence order.
- It never needs backpressure. :: Slow stages still need capacity limits.
+ Micro-batching trades latency for throughput. :: Small windows balance the two.

? A record fails validation in an important business pipeline. What is usually unacceptable?
+ Silently dropping it. :: Silent data loss hides errors; quarantine or audit it instead.
- Quarantining it for review. :: That is a valid strategy.
- Retrying it. :: Retrying can be appropriate.
- Stopping the pipeline with a clear error. :: That can be appropriate for critical data.