Serialization, Persistence and ORM
Learning Objectives
Distinguish serialization from persistence; DTOs from domain objects.
Handle schema evolution, identity, cycles and versioning.
Validate untrusted input.
Understand ORM concepts: entities, relationships, lazy/eager loading, identity map, unit of work, N+1 queries.
Prerequisites: Modules 18, 51.
Intuition and Definition
Serialization: converting an object graph to bytes/text (JSON, Protobuf, CBOR) for transport or storage.
Persistence: keeping state beyond the process (databases, files).
Object–relational mismatch: objects have identity, inheritance, references and collections; tables have rows, foreign keys and joins.
ORM (object–relational mapper): maps entities to tables (Entity Framework, Hibernate, SQLAlchemy, TypeORM), tracking changes (unit of work), ensuring one object per row per session (identity map), and loading related data eagerly (joins) or lazily (on access).
N+1 problem: loading N parents, then issuing one query per parent for its children — N+1 round trips.
Why It Matters
Data outlives code. Formats must evolve without breaking old data or old clients, and input from outside is untrusted.
Syntax and Language Notes
var json = JsonSerializer.Serialize(dto); // System.Text.Json
var back = JsonSerializer.Deserialize<VehicleDto>(json);// TypeScript types don't validate JSON at run time — use a schema validator (zod, ajv) at the boundary
const data: unknown = JSON.parse(text);Complete Example
Versioned DTO ↔ domain mapping with validation at the boundary, plus an N+1 demonstration on SQLite (Python's built-in sqlite3 with a query counter):
import json, sqlite3
from dataclasses import dataclass
@dataclass(frozen=True)
class Vehicle: # domain object: validated, immutable
vin: str
odometer_km: int
def __post_init__(self):
if len(self.vin) != 17: raise ValueError("VIN must have 17 characters")
if self.odometer_km < 0: raise ValueError("odometer must be >= 0")
def from_dto(raw: str) -> Vehicle: # boundary: untrusted JSON -> domain
d = json.loads(raw)
version = d.get("schema", 1)
if version == 1: # v1 stored miles
km = round(d["odometer_miles"] * 1.609344)
elif version == 2:
km = d["odometer_km"]
else:
raise ValueError(f"unsupported schema {version}")
return Vehicle(str(d["vin"]), int(km))
def to_dto(v: Vehicle) -> str:
return json.dumps({"schema": 2, "vin": v.vin, "odometer_km": v.odometer_km})
print(from_dto('{"vin": "WVWZZZ1KZ6W123456", "odometer_miles": 1000}'))
print(to_dto(from_dto('{"schema": 2, "vin": "WVWZZZ1KZ6W123456", "odometer_km": 52000}')))
for bad in ['{"schema": 2, "vin": "SHORT", "odometer_km": 1}', '{"schema": 9}']:
try: from_dto(bad)
exce
…Vehicle(vin='WVWZZZ1KZ6W123456', odometer_km=1609)
{"schema": 2, "vin": "WVWZZZ1KZ6W123456", "odometer_km": 52000}
rejected: VIN must have 17 characters
rejected: unsupported schema 9
lazy loading: 6 queries
eager join: 1 query, 10 rowsStep-by-Step Walkthrough
from_dto is the boundary: it parses untrusted JSON, upgrades old schema versions (miles → km), and constructs a validated domain object. Invalid data never becomes a Vehicle.
to_dto always writes the latest schema — readers handle old versions, writers produce the new one ("tolerant reader").
The lazy pattern issues 1 query for vehicles + 5 for services = 6 (N+1 with N=5); the join fetches everything in one.
Common Mistakes
Serializing domain objects directly — private invariants and internal fields leak into the API; renames break clients.
No version field in persisted formats.
Deserializing untrusted data into arbitrary types (Java native serialization, Python pickle, .NET BinaryFormatter) — remote code execution. Use data-only formats and explicit DTOs.
Cycles in object graphs that serializers can't handle (or emit infinitely).
N+1 queries hidden behind lazy-loading properties. Long-lived ORM contexts accumulating thousands of tracked entities.
Trade-offs and Alternatives
ORMs speed up CRUD code but hide queries; micro-ORMs (Dapper) and query builders give control; document stores avoid some mapping friction but move integrity rules into code. Protobuf/Avro give compact, schema-checked, evolvable formats; JSON is universal and readable.
Real-World Use
REST/gRPC APIs (DTOs), telemetry backends (versioned message schemas), mobile apps with local SQLite caches, ECU calibration files, event stores.
Practice
Q1. (Easy) What is a DTO?
Answer: A data transfer object — a simple structure shaped for crossing a boundary (API, storage), without domain behaviour.
Q2. (Medium) How do you fix an N+1 query problem?
Answer: Load related data eagerly (join or Include/selectinload), batch the child queries (WHERE vehicle_id IN (...)), or project exactly the data needed.
Q3. (Advanced) Why is Python pickle unsafe for data from users?
Answer: Unpickling can execute arbitrary code (__reduce__), so a crafted payload runs commands on your server. Use JSON (or another data-only format) with schema validation.
Interview Question
Q1. "How do you evolve a persisted message format without breaking old clients?"
Answer: Include a version (or use a self-describing schema like Protobuf with field numbers); only add optional fields with defaults; never reuse or repurpose field ids/names; readers tolerate unknown fields and upgrade old versions on read; writers emit the newest version once all readers understand it; deprecate before removing; and test with stored samples of every historical version.
Quick Quiz
Q1. What is the object–relational mismatch?
Answer: The conceptual gap between object graphs (identity, inheritance, references) and relational tables (rows, keys, joins).
Q2. What does an identity map guarantee?
Answer: At most one in-memory object per database row within a session.
Q3. Where should untrusted input be validated?
Answer: At the boundary, before it becomes a domain object.
Summary
Keep DTOs at boundaries, version formats, validate input, avoid unsafe deserialization.
ORMs help but hide costs like N+1 queries.
Related: Modules 18, 51, 68, 75.