# Chapter 8 — Production Debugging & Incidents By the end of this chapter you will: - Triage an incident in 5 minutes flat - Roll back without making it worse - Write a blameless postmortem - Build a runbook your future self will thank you for --- ## The why > **Production debugging is not about being smart. It's about being methodical under pressure. When the pager goes off at 3 AM, the engineer who follows a checklist outperforms the genius who trusts their gut. Every single time.** A patient arrives at the ER unconscious. The doctor doesn't start by guessing — "maybe it's a rare tropical disease." They follow a protocol: 1. **ABCs — Airway, Breathing, Circulation.** Is the patient breathing? Is their heart beating? These kill in minutes. 2. **Vitals.** Blood pressure, heart rate, oxygen saturation, temperature. These narrow the possibilities. 3. **History.** What happened before they collapsed? Any medications? Any allergies? 4. **Targeted tests.** Blood work, X-ray, CT scan — guided by the vitals and history, not a random fishing expedition. 5. **Treatment.** Stabilize first, diagnose fully later. You don't need to identify the exact virus strain to administer oxygen. The doctor who skips step 1 and orders a full-body MRI kills the patient. The doctor who follows the protocol saves lives. Production incidents follow the same pattern. Your server is the patient. The symptoms: 500 errors, 10-second response times, OOM kills. The protocol: 1. **Triage — is the site down?** Check if users are impacted. This determines urgency. 2. **Vitals.** CPU, memory, disk, network. These narrow the possibilities in 30 seconds. 3. **Recent changes.** What was deployed in the last hour? Any config changes? Any traffic spikes? 4. **Targeted investigation.** Logs, metrics, traces — guided by vitals and recent changes. 5. **Mitigate.** Stop the bleeding first. Root cause analysis comes after users can access the site again. The engineer who opens a code editor during an outage is the doctor ordering an MRI while the patient isn't breathing. --- ## Core concepts ### The incident response protocol: a framework for 3 AM Every incident follows the same lifecycle. Internalizing this structure is what separates senior engineers from everyone else: ``` DETECT → TRIAGE → MITIGATE → DIAGNOSE → RESOLVE → LEARN | | | | | | Alert Is site Stop the Find root Fix the Write fires down? bleeding cause root cause postmortem (blameless) CRITICAL RULE: Never skip MITIGATE to go directly to DIAGNOSE. A mitigated incident with unknown root cause is a nuisance. An unmitigated incident while you're deep in code is a catastrophe. ``` ```bash # Your brain at 3 AM, in script form: # PHASE 1: DETECT (0-1 min) # Alert fires: "api.yourapp.com — 5xx error rate > 5%" # Acknowledge the alert. Don't panic. Open your incident channel. # PHASE 2: TRIAGE (1-5 min) # Answer exactly three questions: # Q1: Is the site completely down or partially degraded? curl -o /dev/null -s -w '%{http_code}\n' https://api.yourapp.com/health # Q2: How many users are affected? (all, some, specific region?) # Q3: When did it start? (check monitoring dashboards) # Declare severity: # SEV1: Site completely down, all users affected → all-hands, war room # SEV2: Major feature broken, some users → on-call engineer + relevant team # SEV3: Minor degradation → ticket for next business day # PHASE 3: MITIGATE (5-15 min) — STOP THE BLEEDING # Options, in order of preference: # 1. Roll back to last known good deploy # 2. Scale up (add more instances if CPU/memory bound) # 3. Feature flag: disable the problematic feature # 4. Shed load: rate-limit non-critical endpoints # 5. Fail over to standby region/database # PHASE 4: DIAGNOSE (as long as needed, but site is up) # Now, with the pressure off, find the root cause. ``` ### The vital signs: what to check first, every time You can diagnose 80% of production issues by checking five things in 30 seconds: ```bash # 1. CPU — is the server pegged at 100%? top -bn1 | head -20 # Look for: %Cpu(s) near 100%, load average > CPU count # Common causes: infinite loop, traffic spike, crypto operation, regex catastrophe # 2. MEMORY — is the OOM killer active? free -h dmesg | grep -i "killed process" | tail -20 # Look for: available memory near 0, swap usage climbing, OOM killer messages # Common causes: memory leak, large file processing, connection pool explosion # 3. DISK — is the disk full? df -h du -sh /var/log/* | sort -rh | head -10 # Look for: any filesystem > 90% full # Common causes: log rotation broken, WAL files piling up, core dumps, Docker images # 4. NETWORK — are connections piling up? ss -s # Socket summary ss -t state time-wait | wc -l # TIME_WAIT connections netstat -i # Network interface errors/drops # Look for: thousands of TIME_WAIT sockets, interface errors, dropped packets # 5. PROCESSES — what's actually running and what state? ps auxf --sort=-%mem | head -20 # Top processes by memory systemctl status # Is your app running? # Look for: zombie processes, defunct processes, unexpected processes ``` Build a script that runs all five checks and outputs a one-page summary. Run it on every server, every 60 seconds, and log the output. When an incident hits, you can compare current state vs. last known good state. The difference IS the problem. This is called a "situational awareness snapshot" and it's the single highest-leverage debugging tool you'll ever build. ### Failure modes: the usual suspects Most production incidents fall into familiar patterns. Recognizing the pattern cuts diagnosis time from hours to minutes: #### OOM (Out of Memory) — the silent killer ```bash # Symptoms: app processes disappearing, random 502 errors, dmesg shows "Killed" # Diagnosis: dmesg | grep -i "out of memory" dmesg | grep -i "killed process node" # Check memory usage over time: # /var/log/syslog or journalctl journalctl -u your-app.service --since "1 hour ago" | grep -i oom # Root causes: # 1. Memory leak — garbage collector can't free memory fast enough # Fix: heap dump analysis (node --inspect, Chrome DevTools) # 2. Large payload processing — reading a 500MB file into memory # Fix: stream processing, chunked uploads # 3. Connection pool explosion — default pool often unlimited # Fix: set max connections in DB pool config # Immediate mitigation: # - Increase swap (temporary bandaid) # - Restart the process (clears leaked memory) # - Reduce worker threads/processes ``` #### Disk Full — the "everything breaks" failure ```bash # Symptoms: 500 errors on writes, "No space left on device", # Postgres refusing connections, Docker pull failures # Quick triage: df -h du -sh /* 2>/dev/null | sort -rh | head -10 # Common culprits: # /var/log — log rotation broken, 50GB of nginx access logs # Fix: sudo logrotate -f /etc/logrotate.conf # /var/lib/postgresql — WAL files accumulating (replication slot issue) # Fix: check pg_replication_slots, remove stale slots # /var/lib/docker — old images and volumes # Fix: docker system prune -a --volumes # /tmp — temp files never cleaned # Fix: tmpfiles.d configuration, or reboot (tmpfs cleared on reboot) # Emergency space recovery: sudo journalctl --vacuum-size=500M # Trim systemd journals sudo apt clean # Clear package cache sudo docker system prune -a -f # Remove unused Docker data (DANGER: removes all stopped containers) ``` #### CPU Spike — the throughput killer ```bash # Symptoms: response times climbing, requests queuing, timeouts cascading # Quick triage: top -bn1 htop # Interactive, shows per-core usage # Find the culprit process: ps aux --sort=-%cpu | head -5 # For Node.js: what's burning CPU? # Send SIGUSR1 to enable debugger: kill -USR1 # Or use clinic.js (flamegraphs): clinic doctor -- node app.js # Common causes: # 1. Infinite loop — regex catastrophic backtracking, while(true) without await # 2. N+1 query explosion — 1 request triggering 10,000 DB queries # 3. Crypto operations — bcrypt with cost factor too high, synchronous crypto # 4. Serialization — JSON.stringify on massive objects # 5. Garbage collection thrashing — allocating and freeing huge objects rapidly ``` #### Connection Pool Exhaustion — the "works fine, then dies" failure ```bash # Symptoms: app works for a while, then all requests timeout. # Restart fixes it temporarily. Pattern repeats every N minutes/hours. # Diagnosis — check DB connections: # PostgreSQL: sudo -u postgres psql -c "SELECT count(*) FROM pg_stat_activity;" sudo -u postgres psql -c "SELECT state, count(*) FROM pg_stat_activity GROUP BY state;" # Look for: idle in transaction (leaked connections, never committed/rolled back) # Look for: active > pool_max (pool exhausted) # Root causes: # 1. Connection leak — acquired connection never released back to pool # Fix: always release in finally block, use try-with-resources patterns # 2. Pool too small for traffic — 10 connections, 200 concurrent requests # Fix: increase pool size, add connection queuing timeout # 3. Slow queries holding connections — 5s query ties up connection for 5s # Fix: query timeout (statement_timeout), query optimization, read replicas ``` ### Debugging tools you need in your arsenal ```bash # === STRACE: trace system calls === # "What is this process actually doing?" sudo strace -p -c # Summary of system calls sudo strace -p -e trace=network # Only network calls sudo strace -p -T # Show time spent in each call # Classic use: process stuck at 100% CPU but not responding # strace shows it's in a tight loop calling gettimeofday() — profiling code gone wrong # === LSOF: list open files === # "What files, sockets, pipes does this process have open?" sudo lsof -p # Everything open by this process sudo lsof -i :5432 # What's connected to Postgres? sudo lsof /var/log/nginx/access.log # What's writing to this file? # Classic use: "Why can't my app connect to the database?" # lsof -i :5432 shows no connections — firewall or DB is down # Classic use: "Why is disk full?" — lsof shows a deleted file still held open # === TCPDUMP: capture network packets === # "What's actually going over the wire?" sudo tcpdump -i eth0 port 443 -w capture.pcap sudo tcpdump -i any host 10.0.1.50 -A # ASCII output, readable HTTP # Classic use: "Is the app sending requests to the old database IP?" # tcpdump shows packets to 10.0.1.50:5432 (old DB) — DNS cache or hardcoded IP # === NETSTAT / SS: socket statistics === ss -tulanp # All listening and established connections ss -s # Summary: TCP, UDP, RAW counts watch -n 1 'ss -s' # Watch socket stats in real-time # Classic use: "Why are we getting connection refused?" # ss shows listen queue full (Recv-Q > 0 on listening socket) # === JOURNALCTL: systemd logs === journalctl -u your-app.service --since "10 minutes ago" -f # Follow journalctl -u your-app.service --since today --no-pager | grep -i error journalctl -u nginx.service -p err # Only errors and above ``` ### Blameless postmortems: learning without fear A postmortem is the single most important artifact of any incident. Its purpose is NOT to assign blame. Its purpose is to ensure the same incident never happens twice. ```markdown # Incident Postmortem: API Latency Spike — May 13, 2026 ## Summary On May 13, 2026, from 14:32 to 15:17 UTC, the user API experienced response times of 3-8 seconds (normal: 80ms). 12% of requests failed with 504 Gateway Timeout. Root cause: a database query introduced in deploy v2.4.1 was missing an index, causing sequential scans on a table with 45 million rows. ## Timeline (UTC) - 14:30 — Deploy v2.4.1 to production (new user search endpoint) - 14:32 — PagerDuty alert: API latency > 2s for 5 minutes - 14:34 — On-call engineer acknowledges alert - 14:38 — Identified `/api/users/search` as the slow endpoint - 14:42 — Rolled back to v2.4.0 - 14:45 — Latency returned to normal - 15:17 — Incident resolved ## Root Cause Query `SELECT * FROM users WHERE last_active > $1 ORDER BY created_at` was scanning the entire `users` table (45M rows) because there was no composite index on `(last_active, created_at)`. The query worked in staging (50K rows) but caused sequential scans in production. ## What Went Well - Alert fired within 2 minutes of degradation - Rollback completed in 8 minutes (automated CI/CD rollback) - No data loss, no incorrect responses served ## What Went Poorly - The new query wasn't tested against a production-sized dataset - No `EXPLAIN ANALYZE` was run before deploy - The deploy happened at 2:30 PM peak traffic, not during low-traffic window ## Action Items 1. [P0] Add composite index: `CREATE INDEX idx_users_active_created ON users(last_active, created_at)` 2. [P1] Add CI step: run `EXPLAIN ANALYZE` (not just `EXPLAIN`) on all new queries against a production-sized anonymized dataset 3. [P1] Move production deploys to 10:00-11:00 UTC (lowest traffic window) 4. [P2] Add Query Plan Review to PR template checklist 5. [P3] Build a `db-review` bot that comments on PRs when queries lack indexes ``` Google's Site Reliability Engineering (SRE) book formalized the blameless postmortem. Key principles: (1) Assume good intent — nobody deployed bad code on purpose. (2) Focus on process failures, not human errors — "the deploy pipeline shouldn't allow an un-indexed query to reach production" not "the developer should have added an index." (3) Action items must be concrete, assigned, and tracked — "be more careful" is not an action item. (4) Share postmortems widely — the whole organization learns from each incident. A postmortem that nobody reads might as well not exist. ### Runbooks: your 3 AM instruction manual A runbook is a step-by-step guide for handling a specific incident. You write it when you're calm so you can follow it when you're not: ```markdown # Runbook: Database Connection Pool Exhaustion ## Symptoms - API returns "too many connections" errors - New requests hang or timeout - `SELECT count(*) FROM pg_stat_activity` exceeds configured max ## Immediate Actions (first 2 minutes) 1. Check pool status: ```bash ssh app-server sudo -u postgres psql -c "SELECT count(*), state FROM pg_stat_activity GROUP BY state;" ``` 2. If `idle in transaction` > 10: kill idle transactions ```sql SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE state = 'idle in transaction' AND age(now(), xact_start) > '5 minutes'; ``` 3. If pool still full: restart the application (clears all connections) ```bash sudo systemctl restart your-app ``` ## Root Cause Investigation (after service restored) - Check deploy log: any ORM/pool config changes? - Check traffic: unusual spike? - Check slow query log: any new slow queries holding connections? - Analyze connection pool metrics from last 24 hours ## Prevention - Add `statement_timeout` to Postgres (kills queries > 10s) - Add connection pool timeout (don't queue requests indefinitely) - Set up alert: warn at 70% pool utilization, page at 90% ``` --- ## Real-world example: How Google SRE handles incidents Google's Site Reliability Engineering teams manage some of the world's largest distributed systems. Their incident response philosophy has become industry standard: **Error budgets:** Google defines an acceptable level of downtime (the "error budget"). For a 99.9% SLA, the error budget is 43 minutes of downtime per month. When the error budget is exhausted, ALL feature development stops until reliability is restored. This creates a natural tension between "ship new features" and "keep the site up" — and prevents the common pattern of shipping features until everything breaks. **The incident commander role:** During a major incident, one person is designated Incident Commander. Their ONLY job is to coordinate — they don't debug, they don't write code, they don't fix configs. They track the timeline, delegate investigation tasks, communicate with stakeholders, and decide when to escalate or declare resolved. This prevents the chaos of five engineers all trying different fixes simultaneously without coordination. **Wheel of Misfortune:** Google runs disaster roleplay exercises where a moderator presents a realistic incident scenario and the on-call engineer must diagnose and mitigate in real-time while the team observes. These are done monthly. The goal: practice incident response in a safe environment so that real incidents feel familiar, not terrifying. **The lesson:** Incident response is a skill, not a talent. You get better at it through practice, protocols, and postmortems — not through raw intelligence. Build your runbooks, run your fire drills, and write your postmortems. When the pager goes off at 3 AM, you'll be following a well-rehearsed script, not improvising in the dark. --- ## Common mistakes | Mistake | Why it's wrong | What to do instead | |---------|---------------|-------------------| | Trying to find root cause before mitigating | Every minute spent debugging is a minute users are down. You can debug after the site is back up | Mitigate first (rollback, scale up, feature flag off). Root cause analysis happens after the incident is contained | | Not communicating during an incident | Stakeholders (CEO, support team, customers) have no idea what's happening. They make their own (usually wrong) assumptions | Post updates every 15-30 minutes in a shared Slack channel or status page. Even "still investigating" is better than silence | | Restarting the server without collecting diagnostic data first | Restart clears memory, kills processes, truncates logs. The evidence of what went wrong is destroyed | Take a snapshot: `top`, `free -h`, `df -h`, `dmesg | tail -50`, `journalctl --since "10 min ago" > incident-snapshot.log`. THEN restart | | Running commands in production without understanding them | `docker system prune -a` deleted the production database volume. `kill -9` on Postgres corrupted the WAL | Read the man page. Understand the consequences. If you're sleep-deprived at 3 AM, ask a colleague to pair on any destructive command | | "Fixing" the symptom instead of the root cause | Adding swap space to "fix" OOM kills just delays the problem. Memory leak is still there | Mitigate symptoms (swap, restart) to restore service. Then INVESTIGATE and fix root cause (heap dump, identify leak) before closing the incident | | Blaming individuals in postmortems | "Alice wrote a bad query" creates a culture of fear. Next time, Alice hides her mistake instead of raising an alarm | Focus on process: "Our CI pipeline didn't catch missing indexes. How do we add that check?" Blame the process, fix the process | | No runbooks for recurring incidents | Every time the database pool fills up, someone rediscovers the fix from scratch. 30 minutes of downtime becomes 30 minutes × N incidents | Write a runbook after the FIRST occurrence. The second time, follow the runbook. The third time, automate the runbook | --- ## Production notes - **Separate your incident communication channel from your debugging channel.** Use one Slack channel for stakeholder updates ("Investigating increased error rates on the payment API. Next update in 15 minutes.") and a separate thread or call for engineers actively debugging. Stakeholders don't need to see your strace output. Engineers don't need stakeholder questions interrupting their flow. - **Automate the first 30 seconds of diagnosis.** A script that runs `top -bn1 | head -5`, `free -h`, `df -h`, `ss -s`, and `systemctl status ` and posts the output to the incident channel when an alert fires. This saves the on-call engineer from SSHing into servers while half-asleep. - **Practice incident response quarterly.** Run a "Wheel of Misfortune" style game day: inject a realistic failure (kill the database, fill the disk, corrupt a config file) and have the on-call engineer respond. Time-to-detection, time-to-mitigation, and communication quality are your metrics. Practice doesn't make perfect — practice makes permanent. - **Track Mean Time To Resolve (MTTR) as a key metric.** Not just uptime. A 99.9% uptime target means nothing if your incidents take 4 hours to resolve each time. Track MTTR per severity level and per incident type. Use this data to prioritize automation: "We spend 40% of incident time on database failover — automate it." - **Every alert must be actionable.** An alert that fires and requires no action trains engineers to ignore alerts. If CPU > 80% for 5 minutes is normal for your workload, don't alert on it. Tune your thresholds. The only thing worse than no alerting is alert fatigue. --- ## Performance notes - **Structured logging unlocks fast diagnosis.** JSON-formatted logs with consistent fields (`timestamp`, `level`, `service`, `traceId`, `userId`, `duration`) let you query with `jq` and aggregate with tools like Loki or Elasticsearch. Plain-text logs require regex parsing under pressure — slow and error-prone. - **Distributed tracing connects the dots across services.** When a request hits your API gateway, calls three microservices, queries two databases, and writes to a queue — standard logging shows 5 separate log lines with no connection. A trace ID links them all. Jaeger, Zipkin, or Datadog APM are worth the setup cost. - **Correlate deploys with incidents automatically.** Your monitoring dashboard should show deploy markers on every graph. If latency spikes exactly at the deploy marker, you know what happened. Tools like Datadog, Grafana with annotations, or a simple deploy webhook that posts to Slack make this trivial. --- ## Security notes - **Never share raw logs, strace output, or tcpdump captures in public channels.** These often contain sensitive data: API keys in query parameters, user emails in HTTP bodies, session tokens in headers. Sanitize before sharing, or share in encrypted channels only. - **Have a secure "break glass" procedure for production access.** During an incident, you might need root access to a server you normally don't touch. This access should: require a second person's approval (even at 3 AM — call them), be time-limited (auto-expire after 1 hour), and be fully audited (every command logged). - **Your incident response playbook is a security document.** If an attacker knows your runbook for "database is down," they know exactly how to extend an outage. Store runbooks in a secure location with access control. Runbook contents are operationally sensitive. - **Postmortems should not contain secrets, PII, or exploitable vulnerability details before they're fixed.** Share the timeline and lessons learned publicly. Share the specific exploit details only after the vulnerability is patched and deployed. --- ## Exercises ### Beginner 1. **Build your first diagnostic script:** Write a bash script that captures the "big five" diagnostics: CPU (top), memory (free), disk (df), network (ss), and processes (ps). It should output a timestamped snapshot to a file. Run it every 60 seconds via cron. Simulate a problem (start a CPU-intensive process) and use the snapshots to identify when the problem started and what changed. 2. **Write a runbook for a common failure:** Pick a failure mode you've encountered (OOM, connection pool exhaustion, disk full). Write a runbook following the template from this chapter: symptoms, immediate actions (first 2 minutes), root cause investigation steps, and prevention measures. Test it: have a colleague follow it to resolve a simulated incident. ### Intermediate 1. **Simulate and resolve incidents:** On a test VPS, inject the following failures one at a time and practice diagnosing and resolving each: (a) Fill the disk to 95% with large files, then identify what's consuming space and clean it. (b) Exhaust Postgres connections by running 200 simultaneous connections, then identify the blocked queries and terminate idle transactions. (c) Simulate a memory leak in a Node.js process, identify it with a heap dump, and fix the code. 2. **Write a blameless postmortem:** After resolving the simulated incidents above, write a full blameless postmortem for one of them. Include: summary, timeline, root cause, what went well, what went poorly, and concrete action items. Trade postmortems with a colleague and critique: are the action items concrete and assigned? Would they actually prevent recurrence? ### Advanced 1. **Design an incident response system:** Design a system that: accepts alerts from multiple sources (monitoring, user reports, support tickets), automatically creates an incident channel (Slack/Discord), posts initial diagnostic snapshots, assigns an incident commander role, tracks the timeline, and generates a postmortem template when resolved. Implement the core workflow as a script or bot. Test it with a simulated incident. 2. **Build a production debugging toolkit:** Create a Docker-based "debug sidecar" container with all the tools from this chapter (strace, lsof, tcpdump, htop, iotop, netstat, jq, curl, dig). It should be deployable alongside any application container with minimal configuration. Include scripts that automate common diagnostic tasks ("diagnose-cpu", "diagnose-memory", "diagnose-disk", "diagnose-network"). Document how to use each tool in an incident scenario. --- ## Interview questions ### Beginner 1. **Q: Your production API is returning 500 errors. What's the first thing you check?** **A:** Check the application logs for the error message and stack trace. Simultaneously, check the server's vital signs using the diagnostic script or manual checks: CPU (is the server overloaded?), memory (is the OOM killer active?), disk (is the disk full?), and network (are connections piling up?). Check if there was a recent deploy — most production incidents are caused by recent changes. The goal is to understand the scope (one endpoint or everything?), the symptom (timeout, connection refused, 500?), and the timing (did it just start? has it been building up?). 2. **Q: What's the difference between mitigating an incident and resolving it?** **A:** Mitigation stops the user impact — the site is back up, errors stop, response times return to normal. Resolution means the root cause is fixed and won't recur. You ALWAYS mitigate first (rollback, scale up, disable feature), then resolve (fix the bug, add the index, patch the vulnerability). A mitigated incident is an inconvenience. An unmitigated incident while you're deep in debugging is a catastrophe. Example: traffic spike causes CPU saturation → mitigate by scaling up (add instances) → resolve by adding rate limiting and optimizing the slow endpoint. 3. **Q: What is a blameless postmortem and why does it matter?** **A:** A blameless postmortem is an incident analysis that focuses on process failures and systemic improvements rather than individual mistakes. It matters because a blame culture causes people to hide mistakes, which prevents the organization from learning. If "Alice wrote a bad query" is the conclusion, the fix is "Alice should be more careful" — which doesn't prevent anyone else from making the same mistake. If "the CI pipeline doesn't validate query plans against production-sized datasets" is the conclusion, the fix is a concrete CI improvement that prevents EVERYONE from making that mistake. Blameless postmortems create psychological safety, which creates honest incident reports, which creates real improvements. ### Senior 1. **Q: You're the on-call engineer. At 3 AM, you get paged for "API completely down." You're 5 minutes into investigation. The CEO messages you: "What's happening? When will it be fixed?" How do you respond?** **A:** Acknowledge immediately — silence is worse than uncertainty. Provide the three things stakeholders need: (1) What we know: "We're investigating a complete API outage. It started at 2:55 AM. All users are affected." (2) What we're doing: "I'm in the incident channel with the database team. We're checking if this is related to the deploy at 2:30 AM." (3) When you'll update them: "I'll update you in 15 minutes, or immediately if we identify the cause." Do NOT speculate on root cause or resolution time — if you say "should be 10 minutes" and it takes an hour, you've destroyed trust. Do NOT stop investigating to write a detailed response — the site being down costs more than the CEO waiting 15 minutes. If you're leading the response, designate someone else as communications lead so you can focus on technical investigation. 2. **Q: A production bug caused data corruption — 500 users have incorrect account balances. How do you handle this technically and organizationally?** **A:** Technically: (1) Stop the bleeding — disable the feature or rollback the code that caused the corruption. (2) Determine the scope — which users, which data, what time range? Query the database for affected records. (3) Restore from backups — you have point-in-time recovery (WAL archiving) or logical backups. Restore the affected data to a staging database, verify correctness, then apply the fix to production. (4) If no backup covers the exact moment: write a corrective script that recalculates balances from the event log (you DO have an immutable event log, right?). Apply it, verify, monitor. Organizationally: (1) Communicate proactively — affected users get an email explaining what happened, what data was affected, what's been corrected, and what they need to do (usually nothing). (2) Be transparent internally — post an incident summary in the company-wide channel. (3) Write a thorough postmortem. (4) If the corruption affected financial data, consult legal — there may be regulatory reporting requirements. (5) The compensation question: if users lost real money (even temporarily), compensate them. If it was a display bug only, explain and apologize. The reputational cost of hiding an incident is always higher than the cost of transparently fixing it. 3. **Q: Design an alerting strategy that balances "catch everything" with "don't wake people up for nothing."** **A:** The strategy uses severity levels and alert routing. **SEV1 (page on-call, any hour):** Complete outage (site returns 5xx for all users), data loss or corruption detected, security breach confirmed. These skip all filters and wake someone up immediately. **SEV2 (page during business hours, notify after-hours):** Major feature broken (payments failing, login not working), latency > 5x baseline for > 10 minutes, error rate > 10%. After hours, these go to a Slack channel and the on-call engineer decides whether to investigate immediately or in the morning. **SEV3 (ticket created, no paging):** Minor degradation, non-critical feature broken, disk > 85%, certificate expiry < 14 days. These create tickets in the engineering backlog. Alert design principles: (1) Alert on symptoms (user-facing problems), not causes (high CPU). "API error rate > 5%" is a symptom. "CPU > 80%" is a cause — it might be fine, it might be a problem. If CPU is always 80%, your alert should be on "API latency > 1s" instead. (2) Include a runbook link in every alert. The alert message should tell the on-call engineer exactly what to check first. (3) Aggregate and deduplicate. If 100 servers all report high CPU simultaneously, send ONE alert ("100 servers affected"), not 100 alerts. (4) Make thresholds dynamic where possible. Don't alert on "error rate > 5%" at 3 AM when traffic is 10 req/min (1 error = 10%). Use "error rate > 5% AND request count > 100/min." (5) Test your alerts quarterly. Intentionally break things and verify the right alerts fire, the right people are notified, and the alerts are actionable. --- ## Summary Production debugging is a discipline, not a talent. The incident response protocol — Detect, Triage, Mitigate, Diagnose, Resolve, Learn — is a structure to follow when your brain is foggy at 3 AM. The five vital signs (CPU, memory, disk, network, processes) diagnose 80% of issues in 30 seconds. Failure modes are predictable: OOM kills, disk full, CPU spikes, connection pool exhaustion. Learn the patterns so you recognize them instantly. Tools like strace, lsof, tcpdump, and journalctl are your surgical instruments — practice with them before you need them. Blameless postmortems transform incidents from sources of fear into sources of learning. Runbooks turn hard-won debugging experience into reusable instructions. The difference between a junior and senior engineer in a crisis isn't coding speed — it's the discipline to check vitals before diving into code, to mitigate before diagnosing, and to communicate clearly under pressure. --- ## Quick recall - Incident protocol: Detect → Triage → Mitigate → Diagnose → Resolve → Learn. NEVER skip mitigation. - Five vital signs: CPU (top), memory (free -h, dmesg), disk (df -h), network (ss -s), processes (ps auxf). - OOM: dmesg shows "Killed process." Fix: heap dump analysis, streaming processing, connection pool limits. - Disk full: df -h, du -sh /*. Common culprits: logs, WAL files, Docker images, temp files. - Connection pool exhaustion: `SELECT count(*), state FROM pg_stat_activity GROUP BY state`. Kill idle transactions. - strace traces system calls. lsof lists open files/sockets. tcpdump captures network packets. journalctl reads logs. - Blameless postmortem: assume good intent, focus on process failures, concrete action items, share widely. - Runbook: symptoms → immediate actions (2 min) → investigation → prevention. Write it after the FIRST incident. - MTTR (Mean Time to Resolve) matters more than uptime percentage. Track it and drive it down. - Every alert must be actionable. Alert fatigue is worse than no alerting. Tune thresholds relentlessly. --- ## Quiz 1. **During an incident, what should you do FIRST?** A) Find the root cause in the code **B) Triage: determine scope, severity, and mitigate if needed** C) Notify the CEO D) Restart the server 2. **What does the OOM killer do?** A) Fixes memory leaks automatically **B) Terminates processes when the system runs out of memory, logging the event in dmesg** C) Restarts the server D) Sends an alert to the engineering team 3. **You're investigating a production incident. What should you do BEFORE restarting the server?** **A) Take a diagnostic snapshot: top, free, df, dmesg, journalctl — restarting destroys evidence** B) Nothing — restarting always fixes the problem C) Email the team D) Run a full database backup 4. **What is the purpose of a blameless postmortem?** A) To identify which engineer caused the incident **B) To learn from the incident and improve processes so it doesn't happen again** C) To document who should be fired D) To satisfy compliance requirements only 5. **Which tool would you use to see what files and network connections a process has open?** A) top B) df **C) lsof** D) free 6. **A runbook should be written:** A) Only for the most severe incidents **B) After the FIRST occurrence of an incident — so the second time follows a tested procedure** C) Never — runbooks create over-reliance D) By the CEO 7. **Why is "CPU > 80%" a bad alert compared to "API latency > 1 second"?** A) CPU is harder to measure **B) CPU > 80% might be normal for your workload and doesn't indicate user impact. Alert on symptoms (user experience), not causes (system metrics)** C) The CPU metric is unreliable D) Latency alerts are required by law

Chapter 8 — Production Debugging & Incidents

Part of L6