Event-Driven and Distributed Design

Learning Objectives

Use domain and integration events with message brokers.

Handle eventual consistency, idempotent consumers and the outbox pattern.

Understand sagas, the limits of distributed transactions, contract versioning and partial failure.

Prerequisites: Modules 51, 61, 69, 74.

Intuition and Definition

Domain event: something that happened inside a bounded context (in-process).

Integration event: a published message for other services, with a versioned contract.

Broker: Kafka, RabbitMQ, Azure Service Bus, MQTT — decouples producers and consumers in time and space.

Eventual consistency: after an event, other services catch up later, not atomically.

Outbox pattern: write the business change *and* the outgoing event into the same database transaction (an outbox table); a relay publishes from the outbox afterwards — no lost or phantom events.

Idempotent consumer: brokers deliver *at least once*; consumers must tolerate duplicates.

Saga: a long-running business transaction across services as a sequence of local transactions with compensating actions.

A remote object is not a local object (the "fallacies of distributed computing"): the network is unreliable, latency is non-zero, bandwidth is finite, topology changes, and partial failure is normal — a call may have succeeded even though you got a timeout.

Why It Matters

Distributed systems fail in ways in-process code never does. Designs that pretend remote calls are local method calls fail under load and partial outages.

Syntax and Language Notes

Message contracts are versioned like APIs (Avro/Protobuf schemas, JSON with versions); consumers are tolerant readers (ignore unknown fields).

Complete Example

Outbox + idempotent consumer + compensating saga step, with an at-least-once relay that redelivers a message:

import json

class Db:
    def __init__(self): self.orders, self.outbox = {}, []
    def place_order(self, order_id, total):
        # ONE local transaction: business row + outgoing event
        self.orders[order_id] = {"total": total, "status": "PLACED"}
        self.outbox.append({"id": f"evt-{order_id}", "type": "OrderPlaced", "v": 1, "order": order_id, "total": total})

class Inventory:                                          # idempotent consumer
    def __init__(self): self.stock, self.seen = {"PAD": 1}, set()
    def on_order_placed(self, msg):
        if msg["id"] in self.seen:
            return "duplicate ignored"
        self.seen.add(msg["id"])
        if self.stock["PAD"] <= 0:
            return "OUT_OF_STOCK"
        self.stock["PAD"] -= 1
        return "RESERVED"

def relay(db, deliver, duplicate_first=False):            # at-least-once publisher
    results = []
    for msg in list(db.outbox):
        wire = json.dumps(msg)                            # crosses the network as data
        results.append(deliver(json.loads(wire)))
        if duplicate_first:                               # e.g. ack lost -> redelivery
            results.append(deliver(json.loads(w
…

O-1: ['RESERVED', 'duplicate ignored']
O-2: ['OUT_OF_STOCK']
{'O-1': 'PLACED', 'O-2': 'CANCELLED (no stock)'} stock: 0

Step-by-Step Walkthrough

Placing an order writes the order row and the event to the outbox together — if the transaction fails, neither exists; if it commits, the event will eventually be published.

The relay delivers O-1's event twice (simulated lost acknowledgement); the consumer recognises the event id and ignores the duplicate — stock is decremented once.

O-2 finds no stock; there's no distributed rollback, so the order service runs a compensating action (cancel the order) — a saga step.

Messages cross the boundary as serialized data with a type and version, not as object references.

sequenceDiagram
  participant O as Order service
  participant X as Outbox relay
  participant I as Inventory
  O->>O: write order + outbox (1 transaction)
  X->>I: OrderPlaced evt-O-1
  X->>I: OrderPlaced evt-O-1 (redelivery)
  I-->>X: duplicate ignored
  X->>I: OrderPlaced evt-O-2
  I-->>O: OUT_OF_STOCK
  O->>O: compensate: cancel O-2

Common Mistakes

Dual writes (save to DB, then publish to broker) without an outbox — crashes in between lose or duplicate events.

Non-idempotent consumers under at-least-once delivery.

Distributed transactions (2PC) across services — fragile and slow; prefer sagas.

Chatty synchronous calls between services that behave like a distributed monolith.

Breaking event contracts without versioning.

Assuming ordering across partitions/queues.

Trade-offs and Alternatives

Events decouple services and absorb load spikes but bring eventual consistency, harder debugging (correlation ids, tracing) and more infrastructure. Synchronous APIs are simpler when immediate consistency is required and the dependency is reliable.

Real-World Use

E-commerce order flows, payment processing, vehicle telematics (MQTT ingest → Kafka → services), IoT fleets, banking ledgers with event sourcing.

Practice

Q1. (Easy) What delivery guarantee do most brokers give?

Answer: At least once (duplicates possible).

Q2. (Medium) What problem does the outbox pattern solve?

Answer: Atomically recording a state change and the event announcing it, avoiding lost or phantom events from dual writes.

Q3. (Advanced) What is a compensating action, and why is it not a rollback?

Answer: A new business operation that semantically undoes a previous step (cancel order, refund payment). Earlier steps already committed and may have been observed; compensation adds new facts rather than erasing history.

Interview Question

Q1. "A payment call times out. Did the payment happen?" Answer: Unknown — that's partial failure. The request may have failed, may have succeeded with a lost response, or may still be in progress. Design for it: send an idempotency key so retries are safe, query the payment status before retrying or charging again, model the order as "payment pending" until confirmed, and reconcile asynchronously (webhooks/events). Never assume a timeout means failure.

Quick Quiz

Q1. What is a saga?

Answer: A sequence of local transactions across services with compensating actions for failures.

Q2. Why version integration events?

Answer: Producers and consumers deploy independently; contracts must evolve without breaking either side.

Q3. Name one fallacy of distributed computing.

Answer: "The network is reliable" (others: zero latency, infinite bandwidth, secure network, unchanging topology, one administrator, zero transport cost, homogeneous network).

Summary

Use domain/integration events with versioned contracts; outbox for atomic publish; idempotent consumers; sagas instead of 2PC.

Remote calls can partially fail — design for timeouts, retries and reconciliation.

Related: Modules 51, 61, 62, 69, 74.