# Chapter 10 — Event-Driven Architecture By the end of this chapter you will: - Model a workflow as events - Use the transactional outbox pattern - Choose Kafka vs Rabbit vs SNS/SQS - Implement a saga that compensates correctly --- ## The why > **When your services don't need to know about each other, your system becomes resilient by default. Events make that possible.** When you make a phone call, both parties must be available at the same moment. If the recipient doesn't answer, the communication fails. That's synchronous HTTP — the caller blocks until the callee responds. The postal service works differently. You write a letter (an event), drop it in the mailbox, and go about your day. The postal system guarantees delivery. The recipient reads it when they're ready. You don't need to know if the recipient is home, awake, or even alive — the letter waits in their mailbox. Event-driven architecture is the postal service for your backend. The Order service publishes an "OrderPlaced" event and moves on. The Inventory service, the Email service, and the Analytics service each receive that event and act on it independently. If the Email service is down, the event waits in the queue. The order still succeeds. Each service operates at its own pace, in its own context, with its own data. But here's the catch: the postal service doesn't tell you if the recipient actually read your letter. Event-driven systems trade immediate consistency for resilience. You need to be comfortable with "the email will be sent within 30 seconds" rather than "the email was sent before the HTTP response returned." --- ## Core concepts ### Events vs Commands: The most important distinction An **event** describes something that already happened. It's a fact. It cannot be rejected — it's in the past. ```typescript // Event: "This happened" { type: "order.placed", orderId: "ord_123", amount: 2999, timestamp: "..." } { type: "payment.received", paymentId: "pay_456", orderId: "ord_123" } { type: "user.registered", userId: "usr_789", email: "user@example.com" } ``` A **command** is a request for something to happen. It can be rejected. ```typescript // Command: "Please do this" { type: "placeOrder", orderId: "ord_123", amount: 2999 } { type: "chargePayment", paymentId: "pay_456", amount: 2999 } ``` Events are past-tense (`OrderPlaced`, `PaymentReceived`). Commands are imperative (`PlaceOrder`, `ChargePayment`). This naming distinction alone clarifies your architecture. If you're naming things with past-tense verbs that haven't happened yet, you're confused about whether you're dealing with events or commands. Events decouple producers from consumers. The Order service publishes `order.placed`. It doesn't know or care how many consumers exist. Add a new Analytics consumer? No change to the Order service. Commands couple the sender to the receiver — the sender must know who to send the command to. ### Message brokers: RabbitMQ vs Kafka The broker is the infrastructure that routes events from producers to consumers. The two dominant choices differ fundamentally: | | RabbitMQ | Apache Kafka | |---|---|---| | **Model** | Smart broker, dumb consumer | Dumb broker, smart consumer | | **Message handling** | Broker routes, tracks acknowledgments, handles redelivery | Consumers track their own offset (position) in the log | | **Ordering** | Per-queue FIFO | Per-partition strict ordering | | **Replay** | Messages gone after consumption | Messages retained (configurable retention, e.g., 7 days); full replay possible | | **Throughput** | ~20K msg/s per queue | ~1M+ msg/s per partition | | **Best for** | Work queues, RPC, task distribution | Event streaming, event sourcing, high-volume logs | **RabbitMQ example (work queue pattern):** ```typescript import amqp from 'amqplib'; // Producer const conn = await amqp.connect('amqp://localhost'); const channel = await conn.createChannel(); await channel.assertQueue('email.notifications', { durable: true }); channel.sendToQueue('email.notifications', Buffer.from(JSON.stringify({ userId: '42', template: 'welcome' })), { persistent: true } ); // Consumer channel.consume('email.notifications', async (msg) => { if (!msg) return; const { userId, template } = JSON.parse(msg.content.toString()); try { await sendEmail(userId, template); channel.ack(msg); } catch (err) { channel.nack(msg, false, true); // Requeue for retry } }, { prefetch: 10 }); ``` **Kafka example (event streaming):** ```typescript import { Kafka } from 'kafkajs'; const kafka = new Kafka({ brokers: ['localhost:9092'] }); // Producer const producer = kafka.producer(); await producer.connect(); await producer.send({ topic: 'orders', messages: [ { key: 'ord_123', value: JSON.stringify({ type: 'order.placed', orderId: 'ord_123' }) } ] }); // Consumer const consumer = kafka.consumer({ groupId: 'email-service' }); await consumer.connect(); await consumer.subscribe({ topic: 'orders', fromBeginning: false }); await consumer.run({ eachMessage: async ({ message }) => { const event = JSON.parse(message.value!.toString()); await sendEmail(event); } }); ``` Use RabbitMQ when you need complex routing (topic exchanges, header-based routing), per-message acknowledgments, and you don't need event replay. Use Kafka when you need high throughput (millions of events/sec), event replay (for new consumers catching up), or you're building event sourcing. For most applications starting out, RabbitMQ's operational simplicity wins. You can migrate to Kafka later when throughput demands it. ### At-least-once vs Exactly-once delivery - **At-most-once:** Fire and forget. The message might be lost. Fast, unreliable. - **At-least-once:** The message will be delivered, but might be delivered multiple times. Requires idempotent consumers. - **Exactly-once:** The message is delivered exactly once. Very hard. Usually requires idempotent consumers anyway. True exactly-once delivery across a network is mathematically impossible (the Two Generals' Problem). What brokers call "exactly-once" is actually at-least-once with idempotent consumers and deduplication. Design your consumers to be idempotent. Even if your broker claims exactly-once, network failures at the edge cases will still produce duplicates. ### Idempotent consumers: Handling duplicates An idempotent consumer produces the same result whether it processes an event once or ten times: ```typescript async function handleOrderPlaced(event: OrderPlacedEvent) { // Check if we've already processed this event const alreadyProcessed = await db.query( 'SELECT 1 FROM processed_events WHERE event_id = $1', [event.id] ); if (alreadyProcessed.rows.length > 0) return; // Skip duplicate // Process the event await db.transaction(async (tx) => { await reserveInventory(tx, event); await tx.query( 'INSERT INTO processed_events (event_id, processed_at) VALUES ($1, NOW())', [event.id] ); }); } ``` Key: the event processing and the "already processed" record must happen in the **same database transaction**. Otherwise, a crash between the two steps leaves you with a duplicate. ### The Transactional Outbox Pattern Your database write and your event publish must be atomic. If you write to the database, then publish to Kafka, and Kafka is down — the database write succeeded but the event is lost. The outbox pattern solves this: ```typescript async function createOrder(order: CreateOrderInput) { const client = await db.connect(); try { await client.query('BEGIN'); const result = await client.query( 'INSERT INTO orders (user_id, amount, status) VALUES ($1, $2, $3) RETURNING id', [order.userId, order.amount, 'pending'] ); const orderId = result.rows[0].id; // The event is part of the same transaction await client.query( `INSERT INTO outbox (aggregate_id, event_type, payload, created_at) VALUES ($1, $2, $3, NOW())`, [orderId, 'order.created', JSON.stringify({ orderId, amount: order.amount })] ); await client.query('COMMIT'); return orderId; } catch (err) { await client.query('ROLLBACK'); throw err; } finally { client.release(); } } // Outbox publisher: reads from outbox table, publishes to broker async function outboxPublisher() { while (true) { const { rows } = await db.query( `DELETE FROM outbox WHERE id IN ( SELECT id FROM outbox ORDER BY id LIMIT 100 ) RETURNING *` ); for (const row of rows) { await kafka.producer().send({ topic: row.event_type.replace('.', '-'), messages: [{ key: row.aggregate_id, value: row.payload }] }); } if (rows.length === 0) await sleep(100); } } ``` The `DELETE ... RETURNING` pattern ensures at-least-once delivery. For stronger guarantees, use a two-phase approach: mark events as "publishing" → publish → mark as "published." ### Event Sourcing: Store events, not state Event sourcing inverts traditional persistence. Instead of storing the current state (`users` table with `name`, `email`), you store the sequence of events that led to that state. To get the current state, replay all events for an aggregate. For performance, you maintain a materialized view (snapshot) and replay only events since the last snapshot. **When to use event sourcing:** Auditing (every change is recorded — banks, healthcare), temporal queries ("what was this user's email last January?"), and systems where the event history itself is valuable. Not for CRUD apps. ### CQRS: Separate reads from writes Command Query Responsibility Segregation (CQRS) splits your data model: one model for writes (commands), another for reads (queries). When an order is placed (write side), an event updates the read-side projection. CQRS is powerful but adds complexity. Use it when your read patterns are fundamentally different from your write patterns (e.g., complex search/filtering on data that's written simply). ### Dead Letter Queues (DLQ) When a consumer repeatedly fails to process a message, you need somewhere to put the poison message so it doesn't block the queue: ```typescript // RabbitMQ: Set up a DLX (Dead Letter Exchange) await channel.assertExchange('dlx', 'direct', { durable: true }); await channel.assertQueue('email.dlq', { durable: true }); await channel.bindQueue('email.dlq', 'dlx', 'email.failed'); await channel.assertQueue('email.notifications', { durable: true, arguments: { 'x-dead-letter-exchange': 'dlx', 'x-dead-letter-routing-key': 'email.failed', 'x-message-ttl': 60000 } }); ``` Monitoring your DLQ is critical. Messages in the DLQ represent lost business events. Set up alerts when DLQ depth > 0. ### Event Schema Evolution Events change over time. Your `order.placed` event from 2023 might not include `currency`. In 2025, you add international support. **Approach: Backward-compatible schemas (Protobuf)** ```protobuf // order.proto — v1 message OrderPlaced { string order_id = 1; int64 amount_cents = 2; } // order.proto — v2 (backward-compatible) message OrderPlaced { string order_id = 1; int64 amount_cents = 2; string currency = 3 [default = "USD"]; // New field with default } ``` Protobuf's backward-compatible rules: never change field numbers, never remove required fields, always provide defaults for new fields. Changing `amount_cents` from cents to whole dollars in the same field is the worst kind of breaking change. Old consumers read `100` as $1.00; new consumers read `100` as $100.00. If the meaning changes, create a new field. Leave the old one and mark it deprecated. --- ## Real-world example: How Netflix uses event-driven for recommendations Netflix's recommendation pipeline is one of the largest event-driven systems in the world. Every user action — play, pause, rate, search, browse — generates an event. These events flow through a Kafka-based pipeline that processes billions of events per day. The flow: 1. **User plays "Stranger Things"** → Event `video.play.started` published to Kafka. 2. **Multiple consumers react independently:** The **view history service** records the play for the "Continue Watching" row. The **recommendation model trainer** feeds the event into ML pipelines that update user preference vectors. The **A/B test analytics service** checks if this play came from a test variant. The **billing service** doesn't care (Netflix doesn't bill per-view), so it never subscribes to this event. 3. **Each consumer operates at its own pace.** The view history update must be near-real-time. The ML model retraining can happen hours later. Why this works: the `video.play.started` event is a fact. It happened. Each consumer interprets that fact in its own context. The playback service doesn't know about ML training, view history, or analytics. It publishes one event and moves on. New consumers can be added without touching the playback codebase. Netflix uses Avro with a schema registry. Every event has a schema ID. Consumers resolve the schema at runtime. This lets producers evolve schemas independently of consumers — the registry handles compatibility checks. --- ## Common mistakes | Mistake | Why it's wrong | What to do instead | |---|---|---| | Using events for request-response flows | Events are fire-and-forget. If you need an immediate response ("was the payment successful?"), events add latency and complexity | Use synchronous HTTP/gRPC for request-response. Use events for notification and async processing | | Not making consumers idempotent | At-least-once delivery means duplicates happen. Charging a customer twice is unacceptable | Store processed event IDs. Check before processing. Use database transactions to make the check-and-process atomic | | Publishing events before committing the database transaction | If the DB transaction rolls back, you've published an event about something that never happened — a phantom event | Use the transactional outbox pattern: write the event to the database in the same transaction as the business data | | Building event sourcing for a CRUD app | Event sourcing doubles your codebase and adds replay complexity for no benefit when you never need audit trails or temporal queries | Use event sourcing only when you need: full audit history, temporal queries, or event replay for new projections | | Ignoring schema evolution | Your events WILL change. Consumers WILL break if you rename fields or change types without a compatibility strategy | Use Protobuf/Avro with schema registries. Follow backward-compatible evolution rules. Version your event types | | No dead letter queue | Poison messages block your queue indefinitely. A single bad message can halt processing for all subsequent messages | Configure DLQs for every queue. Monitor DLQ depth. Alert on non-zero. Build tooling to inspect, fix, and replay DLQ messages | --- ## Production notes - **Run Kafka with at least 3 brokers.** Kafka uses ZooKeeper (or KRaft in newer versions) for leader election. With 1 broker, you have no failover. With 2, you can't form a quorum if one dies. Minimum 3. - **Set retention policies carefully.** Kafka retains messages by time (e.g., 7 days) or size (e.g., 100 GB per topic). If a consumer is down for 8 days, it will miss messages if retention is 7 days. Size your retention to cover your worst-case consumer downtime. - **Monitor consumer lag.** Consumer lag = how far behind a consumer is from the latest message. If lag grows continuously, your consumer can't keep up with production. Set alerts: lag > 10,000 messages or lag > 5 minutes. - **Use schema registries for Avro/Protobuf.** Tools like Confluent Schema Registry enforce compatibility checks. A producer can't publish an incompatible schema change. - **Test your DLQ replay process.** Regular disaster recovery drills: intentionally dead-letter messages, then practice inspecting, fixing, and replaying them. --- ## Performance notes - **Kafka partitions = parallelism.** More partitions = more consumer instances can process in parallel. Rule of thumb: 1 partition per 10 MB/s of throughput. - **Batch publishes for throughput.** Publishing one event per `send()` call is slow. Kafka producers batch automatically with `linger.ms`. Set `linger.ms: 10` and `batch.size: 16384`. Latency increases by 10ms; throughput increases 10x. - **Consumer prefetch and concurrency.** RabbitMQ's `prefetch` setting controls how many unacknowledged messages a consumer can have. Set it based on your processing time and CPU cores. - **Compression.** Kafka supports `gzip`, `snappy`, `lz4`, `zstd`. JSON events compress 5-10x with `lz4`. Set `compression.type: lz4` on the producer. - **Avoid large messages.** Kafka's default max message size is 1 MB. If your payload is large, store it in object storage (S3) and put the URL in the event. --- ## Security notes - **Encrypt messages in transit.** Use TLS between producers, brokers, and consumers. Kafka supports SSL encryption. Never send events over plaintext in production. - **Authenticate producers and consumers.** Kafka supports SASL/SCRAM, mTLS, and Kerberos. At minimum, use SASL/PLAIN with strong passwords. - **Authorize topic access.** Not every service should read every topic. Use Kafka ACLs or RBAC to limit which services can produce to or consume from each topic. - **Sanitize event payloads.** Events often get logged, stored, and replayed. Never include secrets (passwords, API keys, tokens) in event payloads. - **Encrypt event payloads containing PII.** If events contain personally identifiable information, encrypt those fields at the application level before publishing. --- ## Exercises ### Beginner 1. **Publish and consume with RabbitMQ:** Set up a RabbitMQ instance (Docker). Write a producer that publishes `user.registered` events. Write a consumer that logs the event. Test with multiple consumers on the same queue — observe how RabbitMQ distributes messages round-robin. 2. **Implement an idempotent consumer:** Modify the consumer from exercise 1 to be idempotent. Store processed event IDs in a database table. Verify that publishing the same event twice only processes it once. ### Intermediate 3. **Implement the transactional outbox pattern:** Build an order service that writes orders AND outbox events in a single PostgreSQL transaction. Build an outbox publisher that reads from the outbox table and publishes to RabbitMQ. Test by killing the publisher mid-publish and verifying no events are lost. 4. **Set up Dead Letter Queues:** Configure a DLX in RabbitMQ for a consumer that randomly fails 30% of the time. After 3 retries, messages should move to the DLQ. Build a small admin endpoint that lists DLQ messages and allows replaying them. ### Advanced 5. **Build a CQRS order management system:** Write side: `PlaceOrder` command creates an order and publishes `order.placed`. Read side: two independent projectors — one maintains an `order_summaries` table for listing, another maintains an `order_analytics` table for dashboards. Write tests that verify eventual consistency between write and read sides. 6. **Implement event sourcing for a bank account:** Model a bank account as a stream of events (`AccountOpened`, `MoneyDeposited`, `MoneyWithdrawn`, `AccountClosed`). Implement rebuild from events, snapshotting (save state every 100 events), and a balance validator that rejects withdrawals that would cause a negative balance. --- ## Interview questions ### Beginner 1. **Q: What's the difference between an event and a command?** A: An event describes something that happened (past tense, cannot be rejected): `OrderPlaced`. A command is a request for something to happen (imperative, can be rejected): `PlaceOrder`. Events decouple producers from consumers; commands couple the sender to a specific receiver. 2. **Q: What is a message broker? Name two popular ones.** A: A message broker is middleware that routes messages from producers to consumers, providing buffering, routing, and delivery guarantees. RabbitMQ (AMQP-based, smart broker, work queues) and Apache Kafka (distributed log, high throughput, event replay) are the two most common. 3. **Q: Why do consumers need to be idempotent?** A: Because most message brokers provide at-least-once delivery guarantees. Network failures, consumer crashes, and broker redelivery can cause the same message to be delivered multiple times. An idempotent consumer produces the same result whether it processes a message once or ten times. ### Senior 4. **Q: You're designing an order processing system. The flow is: create order → reserve inventory → charge payment → send confirmation email. Each step is a separate service. How do you ensure consistency without distributed transactions?** A: Use the Saga pattern with the transactional outbox. Each service: (1) Performs its local database change AND writes the next event to an outbox table in one transaction. (2) An outbox publisher reads from the outbox and publishes to the message broker. (3) The next service consumes the event. If any step fails, preceding services execute compensating transactions. The system is eventually consistent. 5. **Q: How does Kafka achieve high throughput compared to traditional message queues?** A: Kafka treats messages as an append-only log (sequential disk I/O — very fast). Producers append to the end of a partition; consumers read sequentially from their offset. No random I/O. Kafka batches messages at every layer: producer batches, broker writes batches to disk, consumer fetches batches. Zero-copy transfer (`sendfile` syscall) moves data from disk to network without copying through user space. Traditional message queues track per-message state (acknowledged, requeued) — this bookkeeping limits throughput. 6. **Q: Your event schema needs to change. You're adding a required field to an existing event type. How do you migrate without downtime?** A: You can't add a required field in a backward-compatible way — old consumers will receive events without that field and break. Instead: (1) Add the field as optional with a default value. (2) Update all consumers to handle both formats. (3) Update producers to start including the new field. (4) Monitor that all events now include the field. (5) After confirming (e.g., 30 days of data), update consumers to treat the field as required. (6) Optionally, update the schema to make the field required. Never make a field required in a single deployment — it's a breaking change. --- ## Summary Event-driven architecture trades immediate consistency for resilience and decoupling. Events (past-tense facts) are published by producers and consumed by any number of independent consumers. The transactional outbox pattern solves the atomicity problem between database writes and event publishes. Message brokers sit at the heart: RabbitMQ for work queues and complex routing, Kafka for high-throughput event streaming and replay. Idempotent consumers are non-negotiable — duplicates happen. Schema evolution (Protobuf/Avro with registries) prevents producer changes from breaking consumers. Dead letter queues catch poison messages before they block your entire pipeline. Start simple: a modular monolith can adopt events internally before extracting services. --- ## Quick recall - Events = past tense (`OrderPlaced`), immutable facts. Commands = imperative (`PlaceOrder`), can be rejected. - Transactional outbox: write event to DB in same transaction as business data. Publisher reads outbox → broker. - Idempotent consumers: store processed event IDs. Check before processing. Use DB transactions for atomicity. - RabbitMQ: smart broker, work queues, per-message ACKs. Kafka: distributed log, high throughput, replay. - At-least-once delivery is the realistic guarantee. Exactly-once requires idempotent consumers. - Dead Letter Queue: where poison messages go after repeated failures. Monitor it. Alert on non-zero. - Schema evolution: Protobuf/Avro with backward-compatible rules. Never change field numbers or remove required fields. - CQRS: separate read and write models. Useful when read patterns differ from write patterns. - Event sourcing: store events, compute state by replay. For audit trails and temporal queries. --- ## Quiz 1. **What tense should event names use?** **A: Past tense (e.g., `OrderPlaced`, `PaymentReceived`)** 2. **What pattern ensures atomicity between a database write and an event publish?** **A: Transactional outbox pattern** 3. **Which message broker is optimized for high-throughput event streaming and replay?** **A: Apache Kafka** 4. **What queue receives messages that failed processing after all retries?** **A: Dead Letter Queue (DLQ)** 5. **True or false: Exactly-once delivery is trivially achievable with modern message brokers.** **A: False — it requires idempotent consumers. Network edge cases always create duplicates.** 6. **What schema format is commonly used with Kafka for backward-compatible event evolution?** **A: Apache Avro (with Schema Registry) or Protocol Buffers (Protobuf)** 7. **In CQRS, what is the read-side component that builds query-optimized data from events called?** **A: A projector (or projection)**

Chapter 10 — Event-Driven Architecture

Part of L4