Event-Driven Architecture at Scale: Producers, Consumers, and Delivery Guarantees
Distinguish producers, consumers, and message brokers in a distributed system, explain the difference between commands and events, and reason correctly about at-most-once, at-least-once, and exactly-once delivery guarantees.
This course has an earlier module on Event-Driven PHP, which covers event loops and event-driven patterns within a single PHP process or application. This lesson is a different, complementary topic: event-driven architecture at the level of an entire distributed system, where separate services — potentially written in different languages, deployed independently, and running on entirely different servers — communicate by producing and consuming messages through a shared broker, rather than by calling each other's code directly. Understanding this architectural style matters increasingly as PHP applications grow past 'one app, one database' into systems made of several cooperating services.
Producers, Consumers, and the Message Broker as Middleman
In an event-driven distributed system, a producer is any piece of code that creates a message describing something — a fact, an instruction, a notification — and sends it onward, without directly calling any other service's code and without waiting for that message to be fully processed. A consumer is a separate piece of code, often running as an entirely different, independently deployed service, that receives messages and acts on them. Crucially, the producer and consumer never talk to each other directly; they only ever talk to a message broker sitting between them.
The broker's job is to receive messages from producers, hold them reliably (usually persisted to disk, so a crash doesn't lose in-flight messages), and deliver them to the appropriate consumer(s). This indirection is the whole point of the architecture: the producer doesn't need to know how many consumers exist, whether they're currently online, or how long they'll take to process a message — it hands the message to the broker and moves on immediately, keeping the producer's own response time fast and its failure modes independent of whatever is happening downstream.
This differs sharply from a direct synchronous call (like a PHP service making an HTTP request straight to another service and waiting for a response): a direct call couples the two services' availability and latency together — if the downstream service is slow or down, the calling request is slow or fails too. Routing that same interaction through a broker decouples them: the producer succeeds the moment the broker accepts the message, and the consumer can process it whenever it's ready, even if that's a few seconds or minutes later, or even if the consumer was briefly offline when the message was sent.
• Producer: creates and sends a message, without knowing or caring who (if anyone) consumes it directly. • Consumer: an independently running service that receives and acts on messages. • Broker: the middleman that receives, durably holds, and delivers messages between producers and consumers. • Key benefit over direct synchronous calls: producer and consumer availability/latency become decoupled from each other.
Commands vs Events: Two Different Kinds of Messages
A command is a message that tells one specific service to do something, and it expects exactly one handler to act on it — 'ChargeCreditCard', 'SendWelcomeEmail', or 'GenerateInvoicePdf' are commands: they name an intended action, are addressed (conceptually) to whichever service owns that responsibility, and it would be a bug, not a feature, if two different services both tried to handle the same command and, say, charged the card twice. Commands model 'do this' — imperative, singular, targeted.
An event is a message that announces something already happened, as a plain statement of fact, with no expectation about who — if anyone — reacts to it. 'OrderPlaced', 'UserRegistered', or 'PaymentReceived' are events: the service that emits 'OrderPlaced' doesn't know or care whether zero, one, or five other services are listening for it, and it's entirely correct for many independent consumers to each react to the same event in their own way — an email service sends a confirmation, an analytics service records the sale, a shipping service starts a fulfillment workflow, all in response to the exact same 'OrderPlaced' event, with none of them aware the others exist. Events model 'this happened' — declarative, plural, broadcast.
Choosing the wrong one is a common architecture mistake: modeling something as a command when it's really a fact multiple services care about (forcing the producer to know about and directly invoke every current and future consumer) tightly couples services together and makes adding a new consumer require changing the producer's code; modeling something as an event when only one specific action must happen (like actually charging a payment) can leave that critical action ambiguous about who's responsible for it, or accidentally handled more than once.
• Command: 'do this' — addressed to one specific handler, expects exactly one action to occur. • Event: 'this happened' — broadcast, any number of subscribers (including zero) may react independently. • Commands couple the producer to knowing what should happen next; events decouple 'something happened' from 'what to do about it'. • Getting this distinction wrong is a common source of accidental coupling or ambiguous responsibility in distributed systems.
A Producer → Broker → Multiple Consumers Flow
A concrete example makes the shape of this clear. When a checkout service finishes processing an order, it publishes a single 'OrderPlaced' event to the broker and its job is done — it does not loop through a list of interested services calling each one. The broker then delivers that same event to every consumer subscribed to it: an email-notifications service that sends the customer a confirmation, an inventory service that decrements stock counts, and an analytics service that records the sale for reporting dashboards.
None of those three consumers know about each other's existence, and none of them are on the checkout service's critical path — if the analytics service is temporarily down, the customer still gets their confirmation email and their order still goes through, because the checkout service's own responsibility ended the moment it successfully published the event. This is a major operational and architectural benefit: a failure or slowdown in one downstream consumer doesn't cascade back and affect the producer or any other consumer.
Adding a new capability later — say, a loyalty-points service that should award points whenever an order is placed — requires no change whatsoever to the checkout service's code. The new service simply subscribes to the existing 'OrderPlaced' event stream. This extensibility without touching the producer is one of the most valuable properties event-driven architecture provides as a system grows more services over time.
• One producer publishes once; the broker fans the same event out to every current subscriber. • Consumers are independent of each other — one failing or lagging doesn't affect the others or the producer. • New consumers can subscribe to an existing event stream without any change to the producer's code. • This fan-out and extensibility is the core structural advantage over direct service-to-service calls.
Delivery Guarantees: At-Most-Once, At-Least-Once, and Why Exactly-Once Is a Myth in Practice
A broker's delivery guarantee describes what happens to a message when something fails partway through — a network blip, a consumer crash, a broker restart. At-most-once delivery means a message is delivered zero or one times, never more: if something fails after the broker sends it but before the consumer finishes processing (or before it acknowledges), the message is simply lost, never retried. This is the cheapest and simplest guarantee, but it's only acceptable for messages where losing one occasionally is truly fine — an ephemeral 'user is typing' indicator, for example, not an order confirmation.
At-least-once delivery means a message is guaranteed to be delivered one or more times — the broker keeps a message around and retries delivery until it receives explicit acknowledgment that the consumer successfully processed it. This is by far the most common guarantee real message brokers provide (RabbitMQ, Kafka, and most managed queue services default to this model), because it never silently loses a message. The tradeoff is right there in the name: 'one or more' means a consumer can, and eventually will, receive the exact same message more than once — for instance if the consumer successfully processes a message but crashes right before sending its acknowledgment back to the broker, the broker has no way to know the work was already done and will redeliver it.
Exactly-once delivery — a message delivered and processed precisely one time, no more, no less — sounds like the obviously ideal guarantee, but it's effectively unachievable across a real distributed system with independent failures, and this is a well-known, load-bearing result in distributed systems theory (related to the wider problem of achieving exactly-once semantics across independent, unreliable components — sometimes summarized as 'the two generals problem' for message delivery specifically). Some systems (Kafka included) offer features marketed as 'exactly-once semantics', but these work by combining at-least-once delivery with deduplication inside a narrow, tightly-controlled scope (like a single Kafka-to-Kafka pipeline) — they don't magically solve the general problem, and the moment a message crosses into an external system (a database write, an HTTP call to a third party, an email being sent) that guarantee no longer holds end-to-end.
The practical, load-bearing takeaway for building real systems is: design for at-least-once delivery plus idempotency in your consumers. Idempotency means processing the same message twice produces the same end result as processing it once — for example, a payment consumer that checks 'has this specific order ID already been charged?' before charging, so a duplicate delivery of the same 'ChargeOrder' message is a harmless no-op rather than a double charge. This single pattern — assume duplicates will happen, and make handling a duplicate safe — is how virtually every reliable distributed system actually deals with the impossibility of true exactly-once delivery.
• At-most-once: delivered 0 or 1 times — simplest, but silently loses messages on failure. Fine only for truly disposable data. • At-least-once: delivered 1+ times — never silently loses a message, but the consumer may see duplicates. The industry default. • Exactly-once: delivered exactly 1 time — effectively unachievable end-to-end across independent, unreliable systems. • Practical solution: design for at-least-once delivery, and make consumers idempotent so duplicate processing is harmless.
How This Differs From the Event-Driven PHP Module
It's worth being explicit about the boundary between this lesson and the earlier Event-Driven PHP module in this course. That module covers event-driven programming inside a single PHP process — event loops (like those in ReactPHP or Amp), non-blocking I/O, and emitting/listening for events within one application's own code, all of which happens on one machine, in one process, with no network hop and no broker involved.
This lesson is about a different scale entirely: distributed, cross-service event architecture, where the 'events' travel over the network through a broker to reach entirely separate services, potentially written in different languages and owned by different teams, with all the failure modes (network partitions, consumer crashes, message duplication) that only exist once you cross a process boundary. A single PHP process's internal event loop has no concept of 'at-least-once delivery' because there's no unreliable network link or separate consumer process involved — that concern only appears once events leave the process.
In practice, a single service might use both: internally, its own code might be structured around events dispatched within a single request (the Event-Driven PHP module's territory), while that same service also publishes some of those same business-meaningful occurrences (like 'OrderPlaced') out to a broker so that other, entirely separate services can react to them (this lesson's territory). The two ideas rhyme — both are about reacting to 'something happened' rather than direct calls — but operate at very different scales with very different failure characteristics.
• Event-Driven PHP module: event loops and listeners within one process, one machine, no network involved. • This lesson: events crossing process/service boundaries through a broker, with real network failure modes. • A single service can use both: internal event dispatch for its own logic, plus publishing events externally for other services. • Delivery guarantees and message duplication only become a concern once events leave a single process — that's the dividing line.
Conceptual Shape of Publishing an Event (Broker-Agnostic)
Including a unique 'id' in the message is what lets a consumer implement idempotent deduplication under at-least-once delivery. The producer returns immediately after the broker accepts the message — it never blocks waiting on consumers.
<?php
declare(strict_types=1);
// This is illustrative pseudocode-shaped PHP showing the SHAPE of
// publishing an event — the exact client API differs per broker
// (RabbitMQ, Kafka, Redis Streams — covered concretely in the next lesson).
function publishEvent(string $eventName, array $payload): void {
$message = [
'event' => $eventName,
'payload' => $payload,
'id' => bin2hex(random_bytes(16)), // unique id, useful for dedup
'occurred_at' => date(DATE_ATOM),
];
$broker->publish('order-events', json_encode($message));
// The producer's responsibility ends here — it does not wait
// for any consumer to process this event.
}
publishEvent('OrderPlaced', ['order_id' => 8823, 'total_cents' => 4599]);Code Examples
The command goes to a dedicated queue meant for exactly one kind of handler to consume once; the event is published to a broadcast-style topic that any number of independent consumers can subscribe to without the producer knowing who they are.
$broker->send(...) for commands vs $broker->publish(...) for events
Because the broker guarantees at-least-once delivery, this same message could arrive twice; recording each processed message_id and checking it first makes a duplicate delivery a harmless no-op instead of a double charge.
Common mistake: Assuming a broker's 'at-least-once' guarantee means 'exactly-once in practice' and skipping this idempotency check is a common, expensive bug — duplicate charges, duplicate emails, and double-counted analytics all trace back to this exact mistake.
<?php
declare(strict_types=1);
// COMMAND: addressed to exactly one handler, expects one action.
$broker->send('payments.charge-card', [
'command' => 'ChargeCreditCard',
'order_id' => 8823,
'amount_cents' => 4599,
]);
// Exactly one payment service instance should pick this up and act.
// EVENT: broadcast fact, any number of subscribers may react.
$broker->publish('order-events', [
'event' => 'OrderPlaced',
'order_id' => 8823,
]);
// Zero, one, or many services (email, inventory, analytics, loyalty)
// may each independently subscribe and react to this.<?php
declare(strict_types=1);
function handleChargeOrderMessage(PDO $pdo, array $message): void {
$orderId = $message['order_id'];
$messageId = $message['id'];
// Idempotency guard: has this exact message already been processed?
$already = $pdo->prepare('SELECT 1 FROM processed_messages WHERE message_id = ?');
$already->execute([$messageId]);
if ($already->fetchColumn()) {
return; // safe no-op: this is a duplicate delivery, already handled
}
// ... actually charge the card via the payment provider ...
$pdo->prepare('INSERT INTO processed_messages (message_id, processed_at) VALUES (?, NOW())')
->execute([$messageId]);
}Producer to Broker to Multiple Independent Consumers
Each consumer subscribes independently; the checkout service never learns how many consumers exist or whether they succeeded.
Checkout Service (producer)
│
│ publish("OrderPlaced", {order_id: 8823})
▼
┌───────────────┐
│ Message Broker │ (durably holds the event, fans it out)
└───────┬───────┘
┌────┼────────────┬───────────────┐
▼ ▼ ▼ ▼
Email Inventory Analytics Loyalty Points
Service Service Service Service (added later,
(sends (decrements (records no producer change
confirm) stock) the sale) needed)Important Rules
Model 'tell one service to do a specific thing' as a command; model 'announce a fact that happened' as an event.
Never assume a broker delivers exactly-once — design consumers to be idempotent under at-least-once delivery instead.
Keep the producer's responsibility limited to successfully handing the message to the broker — never block it on consumer processing.
Include a unique message ID in every published message so consumers can deduplicate.
Remember: an internal, single-process event loop and a distributed, cross-service event architecture are different tools solving different problems.
Order Fulfillment Fan-Out
A large e-commerce platform publishes one OrderPlaced event per order, and over time has grown a dozen independent consumer services subscribed to it — shipping, fraud detection, recommendations, loyalty points, tax reporting, and more — none of which required a single change to the original checkout service's code when they were added.
// Modeling 'ChargeCreditCard' as a broadcast event that any service can subscribe to
// Modeling 'ChargeCreditCard' as a targeted command with exactly one authoritative handler
An action with real-world side effects that must happen exactly once (charging money) needs a single, unambiguous owner — broadcasting it as an event invites the possibility of two different consumers both charging the card.
function handlePaymentMessage($message) { chargeCard($message['order_id']); } // no dedup check function handlePaymentMessage($message) { if (alreadyProcessed($message['id'])) return; chargeCard($message['order_id']); markProcessed($message['id']); }
Under an at-least-once broker guarantee, this exact message can be redelivered after a crash or network blip; without an idempotency check, a redelivered charge message double-charges the customer.
What is the key difference between a command and an event in a message-driven architecture? A. Commands are always sent over HTTP; events are always sent over a broker B. A command tells one specific handler to do something and expects one action; an event announces a fact that any number of subscribers may independently react to C. Events are synchronous; commands are asynchronous D. There is no real difference — the terms are interchangeable
B. A command tells one specific handler to do something and expects one action; an event announces a fact that any number of subscribers may independently react to — Commands are targeted, imperative ('do this'), and expect exactly one handler; events are broadcast, declarative ('this happened'), and may have zero, one, or many independent subscribers.
Why is exactly-once delivery considered effectively unachievable in real distributed systems? A. Because brokers are too slow to guarantee it B. Because independent failures (network issues, crashes) make it impossible to guarantee delivery and processing happen precisely once end-to-end across separate unreliable components C. Because PHP specifically doesn't support it, though other languages do D. It is achievable, and every modern broker guarantees it by default
B. Because independent failures (network issues, crashes) make it impossible to guarantee delivery and processing happen precisely once end-to-end across separate unreliable components — Achieving true exactly-once semantics requires perfect coordination across independent, failure-prone components, which is a well-known unsolved problem in distributed systems; real systems instead combine at-least-once delivery with idempotent processing.
What does it mean for a message consumer to be 'idempotent'? A. It processes messages faster than average B. It can only process one message at a time C. Processing the same message more than once produces the same end result as processing it exactly once D. It never acknowledges messages back to the broker
C. Processing the same message more than once produces the same end result as processing it exactly once — Idempotency means duplicate processing is harmless — e.g. checking whether an order was already charged before charging it — which is the standard way real systems cope with at-least-once delivery's possible duplicates.
How does this lesson's topic differ from the course's earlier Event-Driven PHP module? A. They are the same topic covered twice B. The Event-Driven PHP module covers events within a single process/event loop on one machine; this lesson covers events crossing service boundaries through a broker, with network-related failure modes like duplication C. This lesson replaces the Event-Driven PHP module entirely D. The Event-Driven PHP module only applies to JavaScript, not PHP
B. The Event-Driven PHP module covers events within a single process/event loop on one machine; this lesson covers events crossing service boundaries through a broker, with network-related failure modes like duplication — Event-Driven PHP is about intra-process event loops and listeners with no network involved; this lesson is about distributed, cross-service event architecture where messages travel over a network through a broker and can be delayed, lost, or duplicated.
A new 'send an SMS notification' feature needs to run whenever a UserRegistered fact occurs, and separately, a 'verify the user's email address' action must happen exactly once per registration. Decide whether each should be modeled as a command or an event, and explain why.
Sending an SMS notification in reaction to a registration is naturally an event subscriber: publish a 'UserRegistered' event, and let an SMS-notification service (among possibly others, like an email-welcome service or an analytics service) subscribe to it independently. Verifying the email address, however, is a specific action that should happen exactly once and is owned by one responsible service — model it as a command (e.g. 'SendVerificationEmail') sent to that one service, rather than as a broadcast event multiple services might redundantly act on.
Ask: does this need exactly one specific handler to act, or could any number of independent services reasonably react to the same occurrence?