Data Engineering roadmap

Data pipelines, warehousing, big data and distributed systems.

Start with Data Foundations & SQL, follow the steps in order, then build a project, prepare for interviews and earn your certificate.

Foundations

  1. Data Foundations & SQL — The data lifecycle and its language. What data engineers do, the lifecycle and modern data stack, set theory behind relations, and SQL from joins to window functions.
  2. Data Modelling — Structure for correctness and speed. Keys and normalisation, dimensional modelling with star schemas, fact grain and slowly changing dimensions.

Core skills

  1. Pipelines & Orchestration — Reliable, repeatable data flows. Graphs and DAGs, ETL/ELT, incremental loads and CDC, idempotency, orchestration and dbt.
  2. Storage & Distributed Compute — Warehouses, lakes and Spark. Columnar formats, partitioning, lakehouse table formats, and distributed processing with MapReduce and Spark.

Advanced

  1. Real-Time Streaming — Data as it happens. Kafka, event time, windows, watermarks, stateful processing and delivery guarantees.
  2. Quality, Governance & DataOps — Data you can trust. Statistics for monitoring, data quality tests, observability, lineage, governance and privacy — plus MLOps for data-driven models.

All roadmaps