Pipelines & Orchestration

Reliable, repeatable data flows. Graphs and DAGs, ETL/ELT, incremental loads and CDC, idempotency, orchestration and dbt.

• Graph Theory & Networks — Graphs model networks of all kinds. Learn vertices and edges, directed, undirected and weighted graphs, degree, paths, cycles and connectivity, trees and spanning trees (Kruskal, Prim), shortest paths (Dijkstra), topological sorting, and network flow (max-flow/min-cut) — with a Python Dijkstra implementation.
• Data Pipelines: ETL/ELT, Orchestration & Idempotency — Moving and transforming data reliably. Learn ETL vs ELT, ingestion patterns (full loads, incremental loads with watermarks, change data capture), idempotency and exactly-once effects, backfills, DAGs and orchestration (Airflow, Dagster, Prefect), scheduling and dependencies, retries and alerting, dbt for SQL transformations with tests, the medallion architecture (bronze/silver/gold), data contracts and CI/CD for pipelines — with Python ordering a DAG topologically and running an idempotent incremental load.