Introduction to Data Engineering & the Data Lifecycle

The plumbing behind every dashboard and AI model. Learn what data engineers do and how they differ from data scientists and analysts, the data lifecycle (generation, ingestion, storage, transformation, serving), batch vs streaming, OLTP vs OLAP, structured/semi-structured/unstructured data, the modern data stack (sources, ingestion tools, lakes and warehouses, transformation, orchestration, BI and ML), data volume/velocity/variety, and sizing a platform — with Python estimating ingest volume, storage and throughput.

Data engineering designs and runs the systems that collect, move, store, transform and serve data reliably, so analysts, data scientists, AI models and applications can use it.

Without good data engineering, dashboards show wrong numbers, ML models train on garbage, and companies can't trust their own data. With it, data becomes a dependable product.

An analogy

Data engineers are the water-supply engineers of a company: they build the pipes (pipelines), reservoirs (warehouses and lakes) and treatment plants (transformations and quality checks) so clean water (trusted data) flows to every tap (dashboards, apps, models).

Roles in the data team

Who does what:

• Data engineer — builds pipelines, storage, platforms; ensures reliability and quality
• Analytics engineer — models data in the warehouse (dbt), defines metrics
• Data analyst — answers business questions with SQL and dashboards
• Data scientist / ML engineer — builds models on top of the data
• Platform/infra engineer — clusters, cloud, security, cost

The data lifecycle

Generation (apps, devices, logs, third-party APIs) → ingestion (batch file loads, CDC from databases, streaming events) → storage (lake, warehouse, lakehouse) → transformation (cleaning, joining, aggregating, modelling) → serving (BI, reverse ETL to apps, feature stores for ML, APIs). Cross-cutting 'undercurrents': security, data management and governance, DataOps, orchestration, architecture and cost.

Kinds of data and systems

Structured (tables), semi-structured (JSON, logs, Avro), unstructured (images, PDFs, audio). OLTP databases (PostgreSQL, MySQL) handle many small transactional reads/writes with row storage and normalisation; OLAP warehouses (BigQuery, Snowflake, Redshift, ClickHouse) scan billions of rows for analytics with columnar storage. Batch processing handles data in periodic chunks; streaming processes events continuously with seconds of latency.

The modern data stack

Typical cloud stack: sources → ingestion (Fivetran/Airbyte, Kafka, Debezium CDC) → object storage + table format (S3/GCS/ADLS with Iceberg/Delta/Hudi) and/or warehouse → transformation (SQL/dbt, Spark) → orchestration (Airflow, Dagster, Prefect) → serving (BI tools, reverse ETL, feature stores, vector DBs for GenAI) with observability, catalog and governance. Design principles: reliability, scalability, idempotency, loose coupling, cost awareness and 'data as a product'.

Formulas

• Daily volume: GB/day = events/day × bytes/event/10⁹
• Average throughput: EPS = events/day/86 400
• Storage: storage = daily volume × retention days × replication/compression
• Compression ratio: ratio = raw size/compressed size

Worked example: Daily volume

Given: 20 million events × 1 KB

Find: GB/day.

• 20 × 10⁶ × 1000/10⁹

Answer: 20 GB/day

Worked example: Throughput

Given: 86.4 million events/day

Find: Average EPS.

• 86.4 × 10⁶/86 400

Answer: 1000 events/s

Worked example: OLTP or OLAP?

Given: 'Total sales by region for the last 3 years'

Find: System.

• Scans many rows, few columns

Answer: OLAP warehouse (columnar)

Sizing a data platform

# Sizing a data platform: event volume, storage and throughput
events_per_day = 50_000_000          # app events
bytes_per_event = 800                # JSON
raw_gb_day = events_per_day * bytes_per_event / 1e9
print(f"raw ingest {raw_gb_day:.0f} GB/day = {raw_gb_day*365/1000:.1f} TB/year")
compressed = raw_gb_day / 8          # Parquet + compression ~8x vs JSON
print(f"as compressed Parquet ~ {compressed:.0f} GB/day")
avg_eps = events_per_day / 86400
print(f"average {avg_eps:.0f} events/s, peak (x5) {avg_eps*5:.0f} events/s")
# 3 replicas in the lake, 2 years retention
print(f"2-year storage with 3x replication: {compressed*730*3/1000:.1f} TB")
# raw ingest 40 GB/day = 14.6 TB/year
# as compressed Parquet ~ 5 GB/day
# average 579 events/s, peak (x5) 2894 events/s
# 2-year storage with 3x replication: 10.9 TB

A mid-size app generating 50 million events a day produces 40 GB of raw JSON daily. Converting to columnar Parquet with compression cuts this ~8×, so two years of replicated history fits in ~11 TB. Pipelines must handle peak rates (often 5–10× average), not just the average.

Common mistakes

• Wrong: Running analytics on the production OLTP database. — Right: Replicate to a warehouse/lake for analytics.
• Wrong: Sizing for average load. — Right: Design for peaks and growth.
• Wrong: Storing everything as raw JSON forever. — Right: Use columnar formats, partitioning and retention policies.

Applications

• Business intelligence: Reliable dashboards and KPIs.
• ML platforms: Features and training data.
• Real-time products: Fraud checks, recommendations, tracking.
• Regulatory reporting: Auditable, governed data.

Key takeaways

• Generate → ingest → store → transform → serve.
• OLTP for apps, OLAP for analytics.
• Batch and streaming both matter.
• Size for peaks, formats and retention.

Going further

• Data mesh: Domain-owned data products.
• Lakehouse: Warehouse features on open lake formats.
• FinOps for data: Controlling cloud data costs.