nonobie — Real dev skills, explained like a friend would

What you'll learn

By the end of this chapter you will:

Triage an incident in 5 minutes flat
Roll back without making it worse
Write a blameless postmortem
Build a runbook your future self will thank you for

The why

Production debugging is not about being smart. It's about being methodical under pressure. When the pager goes off at 3 AM, the engineer who follows a checklist outperforms the genius who trusts their gut. Every single time.

💡The Emergency Room Doctor Analogy

A patient arrives at the ER unconscious. The doctor doesn't start by guessing — "maybe it's a rare tropical disease." They follow a protocol:

ABCs — Airway, Breathing, Circulation. Is the patient breathing? Is their heart beating? These kill in minutes.
Vitals. Blood pressure, heart rate, oxygen saturation, temperature. These narrow the possibilities.
History. What happened before they collapsed? Any medications? Any allergies?
Targeted tests. Blood work, X-ray, CT scan — guided by the vitals and history, not a random fishing expedition.
Treatment. Stabilize first, diagnose fully later. You don't need to identify the exact virus strain to administer oxygen.

The doctor who skips step 1 and orders a full-body MRI kills the patient. The doctor who follows the protocol saves lives.

Production incidents follow the same pattern. Your server is the patient. The symptoms: 500 errors, 10-second response times, OOM kills. The protocol:

Triage — is the site down? Check if users are impacted. This determines urgency.
Vitals. CPU, memory, disk, network. These narrow the possibilities in 30 seconds.
Recent changes. What was deployed in the last hour? Any config changes? Any traffic spikes?
Targeted investigation. Logs, metrics, traces — guided by vitals and recent changes.
Mitigate. Stop the bleeding first. Root cause analysis comes after users can access the site again.

The engineer who opens a code editor during an outage is the doctor ordering an MRI while the patient isn't breathing.

Core concepts

The incident response protocol: a framework for 3 AM

Every incident follows the same lifecycle. Internalizing this structure is what separates senior engineers from everyone else:

plaintext

DETECT → TRIAGE → MITIGATE → DIAGNOSE → RESOLVE → LEARN
  |         |         |          |          |         |
Alert    Is site   Stop the   Find root   Fix the   Write
fires    down?    bleeding    cause     root cause postmortem
                                                      (blameless)
 
CRITICAL RULE: Never skip MITIGATE to go directly to DIAGNOSE.
A mitigated incident with unknown root cause is a nuisance.
An unmitigated incident while you're deep in code is a catastrophe.

bash

# Your brain at 3 AM, in script form:
 
# PHASE 1: DETECT (0-1 min)
# Alert fires: "api.yourapp.com — 5xx error rate > 5%"
# Acknowledge the alert. Don't panic. Open your incident channel.
 
# PHASE 2: TRIAGE (1-5 min)
# Answer exactly three questions:
# Q1: Is the site completely down or partially degraded?
curl -o /dev/null -s -w '%{http_code}\n' https://api.yourapp.com/health
#

The vital signs: what to check first, every time

You can diagnose 80% of production issues by checking five things in 30 seconds:

bash

# 1. CPU — is the server pegged at 100%?
top -bn1 | head -20
# Look for: %Cpu(s) near 100%, load average > CPU count
# Common causes: infinite loop, traffic spike, crypto operation, regex catastrophe
 
# 2. MEMORY — is the OOM killer active?
free -h
dmesg | grep -i "killed process" | tail -20
# Look for: available memory near 0, swap usage climbing, OOM killer messages
#

The 30-second diagnostic

Build a script that runs all five checks and outputs a one-page summary. Run it on every server, every 60 seconds, and log the output. When an incident hits, you can compare current vs. last known good state. The difference IS the problem. This is called a "situational awareness snapshot" and it's the single highest-leverage debugging tool you'll ever build.

Failure modes: the usual suspects

Most production incidents fall into familiar patterns. Recognizing the pattern cuts diagnosis time from hours to minutes:

OOM (Out of Memory) — the silent killer

bash

# Symptoms: app processes disappearing, random 502 errors, dmesg shows "Killed"
# Diagnosis:
dmesg | grep -i "out of memory"
dmesg | grep -i "killed process node"
 
# Check memory usage over time:
# /var/log/syslog or journalctl
journalctl -u your-app.service --since "1 hour ago" | grep -i oom

Disk Full — the "everything breaks" failure

bash

# Symptoms: 500 errors on writes, "No space left on device", 
# Postgres refusing connections, Docker pull failures
 
# Quick triage:
df -h
du -sh /* 2>/dev/null | sort -rh | head -10
 
# Common culprits:
# /var/log — log rotation broken, 50GB of nginx access logs
#   Fix: sudo logrotate -f /etc/logrotate.conf
#

CPU Spike — the throughput killer

bash

# Symptoms: response times climbing, requests queuing, timeouts cascading
 
# Quick triage:
top -bn1
htop  # Interactive, shows per-core usage
 
# Find the culprit process:
ps aux --sort=-%cpu | head -5
 
# For Node.js: what's burning CPU?
# Send SIGUSR1 to enable debugger: kill -USR1 <pid>
# Or use clinic.js (flamegraphs): clinic doctor -- node app.js
 
#

Connection Pool Exhaustion — the "works fine, then dies" failure

bash

# Symptoms: app works for a while, then all requests timeout.
# Restart fixes it temporarily. Pattern repeats every N minutes/hours.
 
# Diagnosis — check DB connections:
# PostgreSQL:
sudo -u postgres psql -c "SELECT count(*) FROM pg_stat_activity;"
sudo -u postgres psql -c "SELECT state, count(*) FROM pg_stat_activity GROUP BY state;"
 
# Look for: idle in transaction (leaked connections, never committed/rolled back)
#

Debugging tools you need in your arsenal

bash

# === STRACE: trace system calls ===
# "What is this process actually doing?"
sudo strace -p <pid> -c          # Summary of system calls
sudo strace -p <pid> -e trace=network  # Only network calls
sudo strace -p <pid> -T          # Show time spent in each call

Blameless postmortems: learning without fear

A postmortem is the single most important artifact of any incident. Its purpose is NOT to assign blame. Its purpose is to ensure the same incident never happens twice.

markdown

# Incident Postmortem: API Latency Spike — May 13, 2026
 
## Summary
On May 13, 2026, from 14:32 to 15:17 UTC, the user API experienced
response times of 3-8 seconds (normal: 80ms). 12% of requests failed
with 504 Gateway Timeout. Root cause: a database query introduced in
deploy v2.4.1 was missing an index, causing sequential scans on a
table with 45 million rows.
 
## Timeline (UTC)
- 14:30 — Deploy v2.4.1 to production (new user search endpoint)
- 14:32 — PagerDuty alert: API latency > 2s for 5 minutes
- 14:34 — On-call engineer acknowledges alert
- 14:38 — Identified `/api/users/search` as the slow endpoint
- 14:42 — Rolled back to v2.4.0
-

The Google SRE postmortem culture

Google's Site Reliability Engineering (SRE) book formalized the blameless postmortem. Key principles: (1) Assume good intent — nobody deployed bad code on purpose. (2) Focus on process failures, not human errors — "the deploy pipeline shouldn't allow an un-indexed query to reach production" not "the developer should have added an ." (3) Action items must be concrete, assigned, and tracked — "be more careful" is not an action item. (4) Share postmortems widely — the whole organization learns from each incident. A postmortem that nobody reads might as well not exist.

Runbooks: your 3 AM instruction manual

A runbook is a step-by-step guide for handling a specific incident. You write it when you're calm so you can follow it when you're not:

markdown

# Runbook: Database Connection Pool Exhaustion
 
## Symptoms
- API returns "too many connections" errors
- New requests hang or timeout
- `SELECT count(*) FROM pg_stat_activity` exceeds configured max
 
## Immediate Actions (first 2 minutes)
1. Check pool status:
   ```bash
   ssh app-server
   sudo -u postgres psql -c "SELECT count(*), state FROM pg_stat_activity GROUP BY state;"

If idle in transaction > 10: kill idle transactions

sql

SELECT pg_terminate_backend(pid) FROM pg_stat_activity
WHERE state = 'idle in transaction' AND age(now(), xact_start) > '5 minutes';

If pool still full: restart the application (clears all connections)
bash
```
sudo systemctl restart your-app
```

Root Cause Investigation (after service restored)

Check deploy log: any /pool config changes?
Check traffic: unusual spike?
Check slow query log: any new slow queries holding connections?
Analyze connection pool metrics from last 24 hours

Prevention

Add statement_timeout to Postgres (kills queries > 10s)
Add connection pool timeout (don't requests indefinitely)
Set up alert: warn at 70% pool utilization, page at 90%

plaintext

 
---
 
## Real-world example: How Google SRE handles incidents
 
Google's Site Reliability Engineering teams manage some of the world's largest distributed systems. Their incident response philosophy has become industry standard:
 
**Error budgets:** Google defines an acceptable level of downtime (the "error budget"). For a 99.9% SLA, the error budget is 43 minutes of downtime per month. When the error budget is exhausted, ALL feature development stops until reliability is restored. This creates a natural tension between "ship new features" and "keep the site up" — and prevents the common pattern of shipping features until everything breaks.
 
**The incident commander role:** During a major incident, one person is designated Incident Commander. Their ONLY job is to coordinate — they don't debug, they don't write code, they don't fix configs. They track the timeline, delegate investigation tasks, communicate with stakeholders, and decide when to escalate or declare resolved. This prevents the chaos of five engineers all trying different fixes simultaneously without coordination.
 
**Wheel of Misfortune:** Google runs disaster roleplay exercises where a moderator presents a realistic incident scenario and the on-call engineer must diagnose and mitigate in real-time while the team observes. These are done monthly. The goal: practice incident response in a safe environment so that real incidents feel familiar, not terrifying.
 
**The lesson:** Incident response is a skill, not a talent. You get better at it through practice, protocols, and postmortems — not through raw intelligence. Build your runbooks, run your fire drills, and write your postmortems. When the pager goes off at 3 AM, you'll be following a well-rehearsed script, not improvising in the dark.
 
---
 
## Common mistakes
 
| Mistake | Why it's wrong | What to do instead |
|---------|---------------|-------------------|
| Trying to find root cause before mitigating | Every minute spent debugging is a minute users are down. You can debug after the site is back up | Mitigate first (rollback, scale up, feature flag off). Root cause analysis happens after the incident is contained |

Chapter 8 — Production Debugging & Incidents

The why

Core concepts

The incident response protocol: a framework for 3 AM

The vital signs: what to check first, every time

Failure modes: the usual suspects

OOM (Out of Memory) — the silent killer

Disk Full — the "everything breaks" failure

CPU Spike — the throughput killer

Connection Pool Exhaustion — the "works fine, then dies" failure

Debugging tools you need in your arsenal

Blameless postmortems: learning without fear

Runbooks: your 3 AM instruction manual

Root Cause Investigation (after service restored)

Prevention