Warming up the neural circuits...
By the end of this chapter you will:
When your services don't need to know about each other, your system becomes resilient by default. Events make that possible.
When you make a phone call, both parties must be available at the same moment. If the recipient doesn't answer, the communication fails. That's synchronous HTTP — the caller blocks until the callee responds.
The postal service works differently. You write a letter (an event), drop it in the mailbox, and go about your day. The postal system guarantees delivery. The recipient reads it when they're ready. You don't need to know if the recipient is home, awake, or even alive — the letter waits in their mailbox.
Event-driven architecture is the postal service for your backend. The Order service publishes an "OrderPlaced" event and moves on. The Inventory service, the Email service, and the Analytics service each receive that event and act on it independently. If the Email service is down, the event waits in the . The order still succeeds. Each service operates at its own pace, in its own context, with its own data.
But here's the catch: the postal service doesn't tell you if the recipient actually read your letter. Event-driven systems trade immediate consistency for resilience. You need to be comfortable with "the email will be sent within 30 seconds" rather than "the email was sent before the HTTP response returned."
An event describes something that already happened. It's a fact. It cannot be rejected — it's in the past.
// Event: "This happened"
{ type: "order.placed", orderId: "ord_123", amount: 2999, timestamp: "..." }
{ type: "payment.received", paymentId: "pay_456", orderId: "ord_123"
A command is a request for something to happen. It can be rejected.
// Command: "Please do this"
{ type: "placeOrder", orderId: "ord_123", amount: 2999 }
{ type: "chargePayment", paymentId: "pay_456", amount: 2999 }Events are past-tense (OrderPlaced, PaymentReceived). Commands are imperative (PlaceOrder, ChargePayment). This naming distinction alone clarifies your architecture. If you're naming things with past-tense verbs that haven't happened yet, you're confused about whether you're dealing with events or commands.
Events decouple producers from consumers. The Order service publishes order.placed. It doesn't know or care how many consumers exist. Add a new Analytics consumer? No change to the Order service. Commands couple the sender to the receiver — the sender must know who to send the command to.
The broker is the infrastructure that routes events from producers to consumers. The two dominant choices differ fundamentally:
| RabbitMQ | Apache Kafka | |
|---|---|---|
| Model | Smart broker, dumb consumer | Dumb broker, smart consumer |
| Message handling | Broker routes, tracks acknowledgments, handles redelivery | Consumers track their own offset (position) in the log |
| Ordering | Per-queue FIFO | Per-partition strict ordering |
| Replay | Messages gone after consumption | Messages retained (configurable retention, e.g., 7 days); full replay possible |
| Throughput | ~20K msg/s per queue | ~1M+ msg/s per partition |
| Best for | Work queues, RPC, task distribution | Event streaming, event sourcing, high-volume logs |
RabbitMQ example (work queue pattern):
import amqp from 'amqplib';
// Producer
const conn = await amqp.connect('amqp://localhost');
const channel = await conn.createChannel();
await channel.assertQueue(
Kafka example (event streaming):
import { Kafka } from 'kafkajs';
const kafka = new Kafka({ brokers: ['localhost:9092'] });
// Producer
const producer = kafka.producer();
await
Use RabbitMQ when you need complex routing (topic exchanges, header-based routing), per-message acknowledgments, and you don't need event replay. Use Kafka when you need high throughput (millions of events/sec), event replay (for new consumers catching up), or you're building event sourcing. For most applications starting out, RabbitMQ's operational simplicity wins. You can migrate to Kafka later when throughput demands it.
True exactly-once delivery across a network is mathematically impossible (the Two Generals' Problem). What brokers call "exactly-once" is actually at-least-once with idempotent consumers and deduplication. Design your consumers to be idempotent. Even if your broker claims exactly-once, network failures at the edge cases will still produce duplicates.
An idempotent consumer produces the same result whether it processes an event once or ten times:
async function handleOrderPlaced(event: OrderPlacedEvent) {
// Check if we've already processed this event
const alreadyProcessed = await db.query(
'SELECT 1 FROM processed_events WHERE event_id = $1',
[event.id]
);
if (alreadyProcessed
Key: the event processing and the "already processed" record must happen in the same database . Otherwise, a crash between the two steps leaves you with a duplicate.
Your database write and your event publish must be atomic. If you write to the database, then publish to Kafka, and Kafka is down — the database write succeeded but the event is lost. The outbox pattern solves this:
async function createOrder(order: CreateOrderInput) {
const client = await db.connect();
try {
await client.query('BEGIN');
const result = await client.query
The DELETE ... RETURNING pattern ensures at-least-once delivery. For stronger guarantees, use a two-phase approach: mark events as "publishing" → publish → mark as "published."
Event sourcing inverts traditional persistence. Instead of storing the current state (users table with name, email), you store the sequence of events that led to that . To get the current state, replay all events for an aggregate. For performance, you maintain a materialized view (snapshot) and replay only events since the last snapshot.
When to use event sourcing: Auditing (every change is recorded — banks, healthcare), temporal queries ("what was this user's email last January?"), and systems where the event history itself is valuable. Not for apps.
Command Query Responsibility Segregation (CQRS) splits your data model: one model for writes (commands), another for reads (queries). When an order is placed (write side), an event updates the read-side projection. CQRS is powerful but adds complexity. Use it when your read patterns are fundamentally different from your write patterns (e.g., complex search/filtering on data that's written simply).
When a consumer repeatedly fails to process a message, you need somewhere to put the poison message so it doesn't block the queue:
// RabbitMQ: Set up a DLX (Dead Letter Exchange)
await channel.assertExchange('dlx', 'direct', { durable: true });
await channel.assertQueue('email.dlq', { durable: true });
await channel
Monitoring your DLQ is critical. Messages in the DLQ represent lost business events. Set up alerts when DLQ depth > 0.
Events change over time. Your order.placed event from 2023 might not include currency. In 2025, you add international support.
Approach: Backward-compatible schemas (Protobuf)
// order.proto — v1
message OrderPlaced {
string order_id = 1;
int64 amount_cents = 2;
}
// order.proto — v2 (backward-compatible)
message OrderPlaced {
string order_id = 1;
int64 amount_cents = 2;
string currency = 3 [
Protobuf's backward-compatible rules: never change field numbers, never remove required fields, always provide defaults for new fields.
Changing amount_cents from cents to whole dollars in the same field is the worst kind of breaking change. Old consumers read 100 as $1.00; new consumers read 100 as $100.00. If the meaning changes, create a new field. Leave the old one and mark it deprecated.
Netflix's recommendation pipeline is one of the largest event-driven systems in the world. Every user action — play, pause, rate, search, browse — generates an event. These events flow through a Kafka-based pipeline that processes billions of events per day.
The flow:
video.play.started published to Kafka.Why this works: the video.play.started event is a fact. It happened. Each consumer interprets that fact in its own context. The playback service doesn't know about ML training, view history, or analytics. It publishes one event and moves on. New consumers can be added without touching the playback codebase.
Netflix uses Avro with a schema registry. Every event has a schema ID. Consumers resolve the schema at runtime. This lets producers evolve schemas independently of consumers — the registry handles compatibility checks.
| Mistake | Why it's wrong | What to do instead |
|---|---|---|
| Using events for request-response flows | Events are fire-and-forget. If you need an immediate response ("was the payment successful?"), events add latency and complexity | Use synchronous HTTP/gRPC for request-response. Use events for notification and processing |
| Not making consumers idempotent | At-least-once delivery means duplicates happen. Charging a customer twice is unacceptable | Store processed event IDs. Check before processing. Use database transactions to make the check-and-process atomic |
| Publishing events before committing the database transaction | If the DB transaction rolls back, you've published an event about something that never happened — a phantom event | Use the transactional outbox pattern: write the event to the database in the same transaction as the business data |
| Building event sourcing for a CRUD app | Event sourcing doubles your codebase and adds replay complexity for no benefit when you never need audit trails or temporal queries | Use event sourcing only when you need: full audit history, temporal queries, or event replay for new projections |
| Ignoring schema evolution | Your events WILL change. Consumers WILL break if you rename fields or change types without a compatibility strategy | Use Protobuf/Avro with schema registries. Follow backward-compatible evolution rules. Version your event types |
| No dead letter queue | Poison messages block your queue indefinitely. A single bad message can halt processing for all subsequent messages | Configure DLQs for every queue. Monitor DLQ depth. Alert on non-zero. Build tooling to inspect, fix, and replay DLQ messages |
send() call is slow. Kafka producers batch automatically with linger.ms. Set linger.ms: 10 and batch.size: 16384. Latency increases by 10ms; throughput increases 10x.prefetch setting controls how many unacknowledged messages a consumer can have. Set it based on your processing time and CPU cores.gzip, snappy, lz4, zstd. JSON events compress 5-10x with lz4. Set compression.type: lz4 on the producer.Publish and consume with RabbitMQ: Set up a RabbitMQ instance (). Write a producer that publishes user.registered events. Write a consumer that logs the event. Test with multiple consumers on the same queue — observe how RabbitMQ distributes messages round-robin.
Implement an idempotent consumer: Modify the consumer from exercise 1 to be idempotent. Store processed event IDs in a database table. Verify that publishing the same event twice only processes it once.
Implement the transactional outbox pattern: Build an order service that writes orders AND outbox events in a single PostgreSQL transaction. Build an outbox publisher that reads from the outbox table and publishes to RabbitMQ. Test by killing the publisher mid-publish and verifying no events are lost.
Set up Dead Letter Queues: Configure a DLX in RabbitMQ for a consumer that randomly fails 30% of the time. After 3 retries, messages should move to the DLQ. Build a small admin endpoint that lists DLQ messages and allows replaying them.
Build a CQRS order management system: Write side: PlaceOrder command creates an order and publishes order.placed. Read side: two independent projectors — one maintains an order_summaries table for listing, another maintains an order_analytics table for dashboards. Write tests that verify eventual consistency between write and read sides.
Implement event sourcing for a bank account: Model a bank account as a stream of events (AccountOpened, MoneyDeposited, MoneyWithdrawn, AccountClosed). Implement rebuild from events, snapshotting (save state every 100 events), and a balance validator that rejects withdrawals that would cause a negative balance.
Q: What's the difference between an event and a command?
A: An event describes something that happened (past tense, cannot be rejected): OrderPlaced. A command is a request for something to happen (imperative, can be rejected): PlaceOrder. Events decouple producers from consumers; commands couple the sender to a specific receiver.
Q: What is a message broker? Name two popular ones. A: A message broker is that routes messages from producers to consumers, providing buffering, routing, and delivery guarantees. RabbitMQ (AMQP-based, smart broker, work queues) and Apache Kafka (distributed log, high throughput, event replay) are the two most common.
Q: Why do consumers need to be idempotent? A: Because most message brokers provide at-least-once delivery guarantees. Network failures, consumer crashes, and broker redelivery can cause the same message to be delivered multiple times. An idempotent consumer produces the same result whether it processes a message once or ten times.
Q: You're designing an order processing system. The flow is: create order → reserve inventory → charge payment → send confirmation email. Each step is a separate service. How do you ensure consistency without distributed transactions? A: Use the Saga pattern with the transactional outbox. Each service: (1) Performs its local database change AND writes the next event to an outbox table in one transaction. (2) An outbox publisher reads from the outbox and publishes to the message broker. (3) The next service consumes the event. If any step fails, preceding services execute compensating transactions. The system is eventually consistent.
Q: How does Kafka achieve high throughput compared to traditional message queues?
A: Kafka treats messages as an append-only log (sequential disk I/O — very fast). Producers append to the end of a partition; consumers read sequentially from their offset. No random I/O. Kafka batches messages at every layer: producer batches, broker writes batches to disk, consumer fetches batches. Zero-copy transfer (sendfile syscall) moves data from disk to network without copying through user space. Traditional message queues track per-message state (acknowledged, requeued) — this bookkeeping limits throughput.
Q: Your event schema needs to change. You're adding a required field to an existing event type. How do you migrate without downtime? A: You can't add a required field in a backward-compatible way — old consumers will receive events without that field and break. Instead: (1) Add the field as optional with a default value. (2) Update all consumers to handle both formats. (3) Update producers to start including the new field. (4) Monitor that all events now include the field. (5) After confirming (e.g., 30 days of data), update consumers to treat the field as required. (6) Optionally, update the schema to make the field required. Never make a field required in a single deployment — it's a breaking change.
Event-driven architecture trades immediate consistency for resilience and decoupling. Events (past-tense facts) are published by producers and consumed by any number of independent consumers. The transactional outbox pattern solves the atomicity problem between database writes and event publishes. Message brokers sit at the heart: RabbitMQ for work queues and complex routing, Kafka for high-throughput event streaming and replay. Idempotent consumers are non-negotiable — duplicates happen. Schema evolution (Protobuf/Avro with registries) prevents producer changes from breaking consumers. Dead letter queues catch poison messages before they block your entire pipeline. Start simple: a modular can adopt events internally before extracting services.
OrderPlaced), immutable facts. Commands = imperative (PlaceOrder), can be rejected.What tense should event names use?
A: Past tense (e.g., OrderPlaced, PaymentReceived)
What pattern ensures atomicity between a database write and an event publish? A: Transactional outbox pattern
Which message broker is optimized for high-throughput event streaming and replay? A: Apache Kafka
What queue receives messages that failed processing after all retries? A: Dead Letter Queue (DLQ)
True or false: Exactly-once delivery is trivially achievable with modern message brokers. A: False — it requires idempotent consumers. Network edge cases always create duplicates.
What schema format is commonly used with Kafka for backward-compatible event evolution? A: Apache Avro (with Schema Registry) or Protocol Buffers (Protobuf)
In CQRS, what is the read-side component that builds query-optimized data from events called? A: A projector (or projection)