# Chapter 1 — Linux Basics for Backend By the end of this chapter you will: - Move around a Linux box without Stack Overflow - Read top, htop, and iostat - Tail a journalctl log to find a crash - Edit a systemd unit file --- ## The why > You can build the most elegant backend in the world, but when it's running on a Linux box at 2 AM and you need to figure out why memory spiked, your IDE won't help you. The terminal will. A mechanic doesn't just know how to drive — they know what's under the hood. When a customer says "it's making a rattling noise at 60 km/h", the mechanic pops the hood, listens with a stethoscope, checks the belts, reads the OBD scanner, and diagnoses the root cause. Backend development on a server works the same way. Your API is the car. The Linux box is the engine bay. The commands in this chapter are your stethoscope, your wrench, and your diagnostic scanner. When someone says "the API is slow" or "the server is down", you don't guess — you `top`, you `journalctl`, you `lsof`, and you find the problem. This chapter covers the 20 commands you'll use every single day as a backend developer. Not all 2,000 Linux commands — just the ones that matter for running and debugging backend services in production. --- ## Core concepts ### Connecting to your server: SSH SSH (Secure Shell) is how you get into a remote Linux machine. If you take one thing from this chapter, make it this workflow: ```bash # Connect to a remote server ssh user@your-server-ip # Connect with a specific identity file (key) ssh -i ~/.ssh/my-key.pem ubuntu@ec2-instance.compute.amazonaws.com # Run a single command remotely without opening a shell ssh user@server "systemctl status nginx" # Copy files securely scp ./app.tar.gz user@server:/home/user/deploy/ ``` Set up key-based auth. Passwords over SSH are a liability. Generate a key pair: ```bash ssh-keygen -t ed25519 -C "your-email@example.com" ssh-copy-id user@your-server ``` Now you can connect without typing a password. Keep your private key (`~/.ssh/id_ed25519`) safe — it's the master key to your server. ### What's running? Process management The first question you ask when something feels wrong: "What's actually running?" ```bash # List all processes (a = all users, u = user format, x = no tty) ps aux # Find a specific process ps aux | grep node # Interactive process viewer (real-time) top # Better top — colors, mouse, scrollable htop # Kill a process by PID graceful: kill -15 1234 forceful: kill -9 1234 # Kill by name pkill -f "node server.js" # See process tree (parent-child relationships) pstree -p ``` **Reading `top` output in 30 seconds:** ```bash top - 14:23:11 up 45 days, 2:15, 1 user, load average: 0.85, 0.62, 0.58 Tasks: 156 total, 1 running, 155 sleeping, 0 stopped, 0 zombie %Cpu(s): 12.3 us, 3.1 sy, 0.0 ni, 84.2 id, 0.2 wa, 0.0 hi, 0.2 si, 0.0 st MiB Mem: 7986.2 total, 6234.1 used, 1752.1 free, 234.5 buff/cache ``` What to look for: - **load average**: three numbers = 1, 5, 15 min averages. If > number of CPU cores, the system is overloaded. - **%Cpu(s)**: `us` = user space (your app), `sy` = system/kernel, `wa` = waiting for I/O (disks are slow), `id` = idle. High `wa` means your disk is the bottleneck. - **Mem**: Check `free` — if it's near zero and swap is high, you're out of memory. - **zombie processes**: Should be 0. Zombies are finished processes whose parents haven't cleaned them up. ### Systemd: The service manager systemd is the init system on virtually every modern Linux distribution. It starts, stops, and monitors services (your backend app is a service). ```bash # Check if your app is running systemctl status my-app # Start / Stop / Restart sudo systemctl start my-app sudo systemctl stop my-app sudo systemctl restart my-app # Enable at boot (starts automatically after reboot) sudo systemctl enable my-app # Disable at boot sudo systemctl disable my-app # See all failed services systemctl --failed # List all active services systemctl list-units --type=service --state=running ``` **Writing a systemd unit file:** ```ini title="/etc/systemd/system/my-app.service" [Unit] Description=My Node.js Backend API After=network.target postgresql.service redis.service [Service] Type=simple User=appuser Group=appuser WorkingDirectory=/opt/my-app ExecStart=/usr/bin/node /opt/my-app/dist/server.js Restart=on-failure RestartSec=5 EnvironmentFile=/opt/my-app/.env StandardOutput=journal StandardError=journal # Security hardening NoNewPrivileges=yes PrivateTmp=yes ProtectSystem=strict ReadWritePaths=/opt/my-app/logs /opt/my-app/uploads [Install] WantedBy=multi-user.target ``` After creating or editing a unit file: ```bash sudo systemctl daemon-reload # Reload systemd config sudo systemctl start my-app # Start the service sudo systemctl enable my-app # Auto-start on boot ``` ### Logs: journalctl When your app crashes at 3 AM, logs are your only witness. systemd captures stdout/stderr from your service automatically. ```bash # View logs for a specific service journalctl -u my-app # Follow logs in real-time (like tail -f) journalctl -u my-app -f # Last 100 lines journalctl -u my-app -n 100 # Logs since last boot journalctl -u my-app -b # Logs in a time range journalctl -u my-app --since "2026-05-13 10:00:00" --until "2026-05-13 11:00:00" # Filter by priority (0=emerg, 3=err, 6=info) journalctl -u my-app -p err # Kernel messages only journalctl -k # See how much disk logs are using journalctl --disk-usage ``` When someone says "the API was down 20 minutes ago", run: `journalctl -u my-app --since "20 minutes ago" -p err`. This shows only errors from the relevant time window. This command alone has saved me hours of scrolling through noise. ### Finding stuff: grep and find `grep` searches inside files. `find` searches for files themselves. Master both. ```bash # Find "ERROR" in all .log files, with line numbers grep -rn "ERROR" /var/log/my-app/ # Case-insensitive, show 3 lines of context grep -rni -C 3 "timeout" /opt/my-app/ # Search only .js files grep -rn "TODO" --include="*.js" ./src/ # Invert match (lines that DON'T contain this) grep -v "DEBUG" app.log # Count occurrences grep -c "reference" app.log # Find files modified in the last 7 days find /opt/my-app/logs -type f -mtime -7 # Find files larger than 100MB find / -type f -size +100M 2>/dev/null # Find and delete .tmp files older than 3 days find /tmp -name "*.tmp" -mtime +3 -delete # Find all .js files and search for "import" inside them find ./src -name "*.js" -exec grep -l "import" {} \; ``` ### Disk and memory: df, du, free Your server runs out of disk or memory — it's a matter of when, not if. ```bash # Disk space summary (human-readable) df -h # Check specific partition df -h /var # What's eating my disk? (top-level) du -sh /* 2>/dev/null | sort -hr | head -10 # What's eating THIS directory? du -sh * | sort -hr | head -10 # Memory overview free -h # Memory details (buffers, cache, swap) free -hm ``` If `free -h` shows almost no "available" memory but the system feels fine, check the `buff/cache` column. Linux uses unused RAM as disk cache — it's released when applications need it. Available memory = free + reclaimable cache. The `available` column in `free -h` is the number you actually care about. ### File permissions: chmod, chown Every file and directory has an owner, a group, and three permission sets (read/write/execute for owner, group, and others). ```bash # See permissions ls -la # -rwxr-xr-- 1 appuser appgroup 4096 May 13 14:23 server.js # ^^^ ^^^ ^^^ # owner group others # Change permissions chmod 755 script.sh # rwx r-x r-x chmod 600 .env # rw- --- --- (only owner) chmod 644 public.html # rw- r-- r-- # Change owner chown appuser:appgroup server.js # Recursive (careful!) chown -R appuser:appgroup /opt/my-app/ # Make a script executable chmod +x deploy.sh # Default permissions for new files (set in .bashrc or system-wide) umask 022 # new files: 644, new dirs: 755 ``` Permission numbers demystified: `r=4, w=2, x=1`. So `7 = rwx`, `5 = r-x`, `6 = rw-`. `755` = owner can do everything, group and others can read+execute. ### Networking: curl, lsof, ss ```bash # HTTP requests (the swiss army knife) curl -X GET https://api.example.com/health curl -X POST -H "Content-Type: application/json" -d '{"key":"val"}' https://api.example.com/data curl -v https://api.example.com # verbose (see headers, TLS handshake) curl -I https://api.example.com # headers only # What's listening on which port? ss -tlnp # TCP listening ss -ulnp # UDP listening lsof -i :3000 # What's using port 3000? lsof -i :80 # What's using port 80? # Which process has this file open? lsof /var/log/my-app/app.log # See all files opened by a process lsof -p 1234 # See all network connections for a process lsof -i -a -p 1234 # Download a file wget https://example.com/file.tar.gz ``` ### Package management ```bash # Debian/Ubuntu sudo apt update # Refresh package list sudo apt upgrade # Upgrade all packages sudo apt install nginx # Install sudo apt remove nginx # Remove (keep configs) sudo apt purge nginx # Remove (including configs) apt search "node" # Search packages apt list --installed # What's installed? # RHEL/CentOS/Fedora sudo dnf update sudo dnf install nginx sudo dnf remove nginx dnf search node ``` ### Scheduling: crontab ```bash # Edit your crontab crontab -e # List your crontab crontab -l # Format: minute hour day month weekday command # Every day at 2 AM — run database backup 0 2 * * * /opt/scripts/backup-db.sh >> /var/log/backup.log 2>&1 # Every 5 minutes — health check */5 * * * * curl -s https://myapp.com/health || echo "DOWN" | mail -s "Alert" admin@example.com # Every Monday at 3 AM — cleanup old logs 0 3 * * 1 find /var/log/my-app -name "*.log" -mtime +30 -delete ``` Cron runs with a minimal environment. It doesn't load `.bashrc` or `.profile`. If your script depends on environment variables or a specific PATH, set them explicitly inside the script or at the top of the crontab: `PATH=/usr/local/bin:/usr/bin:/bin`. Always test cron jobs with full paths. --- ## Real-world example: How a 5-person startup debugs a production outage Picture this: A SaaS startup running a Node.js API on an AWS EC2 instance. It's 11 PM. A customer Slack messages: "App is down." The on-call developer SSHes into the box and runs through a mental checklist they've internalized: 1. **`systemctl status api`** — Service is "active (running)" but has been restarting every 30 seconds. The `Restart=on-failure` in the unit file is looping. 2. **`journalctl -u api --since "5 minutes ago" -p err`** — Sees: `FATAL: connection to database timed out`. The app can't reach PostgreSQL. 3. **`systemctl status postgresql`** — PostgreSQL is running. But is it accepting connections? 4. **`ss -tlnp | grep 5432`** — Port 5432 is listening on `127.0.0.1` only. The app's new config changed the DB host to the public IP. PostgreSQL wasn't configured to listen on that interface. 5. **Fix:** Edit `postgresql.conf`, set `listen_addresses = '*'`, restart PostgreSQL. Or (better) revert the app config to use `127.0.0.1`. Total time from alert to fix: 8 minutes. No guessing. No "let me try restarting everything." Just systematic diagnosis with the commands you now know. That's the difference between a developer who panics and a developer who debugs. --- ## Common mistakes | Mistake | Why it's wrong | Fix | |---------|---------------|-----| | Using password SSH instead of keys | Passwords can be brute-forced; keys can't | Use `ssh-keygen -t ed25519` + `ssh-copy-id` | | `kill -9` as first resort | Doesn't let the process clean up (close DB connections, flush logs) | Try `kill -15` first, escalate only if needed | | Running app as root | A bug in your app now has full system access | Create a dedicated `appuser`, use `User=appuser` in systemd unit | | Not setting `Restart=on-failure` in systemd | App crashes once and stays dead until you notice | Add `Restart=on-failure` and `RestartSec=5` to your unit file | | Ignoring `wa` (I/O wait) in top | High `wa` means your disk is the bottleneck, not CPU or RAM | Check disk type (HDD vs SSD), optimize queries, reduce disk writes | | `chmod 777` as a quick fix | Every user on the system can read/write/execute — massive security hole | Use the minimum permissions needed: `600` for secrets, `755` for scripts, `644` for static files | | Editing files directly on the server | Changes lost on next deploy, no version control, no review | Deploy via CI/CD pipeline; server is cattle, not a pet | --- ## Production notes - **Use a non-root user for everything.** Create `appuser` with minimal permissions. The systemd unit should specify `User=appuser`. Your SSH config should disable root login (`PermitRootLogin no` in `/etc/ssh/sshd_config`). - **Set up log rotation.** Logs grow until your disk is full. Configure `logrotate` for your app logs or use `journalctl --vacuum-size=500M` to cap journal size. - **Monitor disk space proactively.** Set up a cron job that alerts you when disk usage exceeds 80%: `df -h / | awk 'NR==2 {print $5}' | tr -d '%'`. - **Keep a runbook.** Document the exact commands to restart each service, check logs, and diagnose common failures. At 2 AM, you won't remember the journalctl flags. ## Performance notes - **`htop` is your first stop for CPU/memory investigation.** Sort by CPU (`P`) or memory (`M`). Look for processes consuming disproportionate resources. - **High `load average` with low CPU usage usually means I/O wait.** Check `iostat -x 1` to see which disk is saturated. - **`free -h` shows `available` memory — that's the real number.** Linux will use spare RAM for cache; it's released instantly when needed. - **Use `nice` and `renice` to deprioritize batch jobs** so they don't starve your API of CPU: `nice -n 19 ./batch-job.sh`. ## Security notes - **Disable password authentication for SSH.** Edit `/etc/ssh/sshd_config`: set `PasswordAuthentication no`, `PermitRootLogin no`, then `systemctl restart sshd`. - **Use `ufw` (Uncomplicated Firewall) to block everything except needed ports.** `ufw allow 22/tcp && ufw allow 80/tcp && ufw allow 443/tcp && ufw enable`. - **`fail2ban` blocks IPs after repeated failed SSH attempts.** Install it, enable the SSH jail — it's a 5-minute setup that stops 99% of brute-force attacks. - **Never hardcode secrets in crontab or scripts.** Use environment files (mode `600`) or a secrets manager. - **Keep the system updated.** `apt update && apt upgrade` weekly. Subscribe to your distro's security mailing list. --- ## Exercises **Beginner:** 1. SSH into a remote machine (or a local VM). Run `htop`, sort by memory, and identify the top 3 memory-consuming processes. Write them down with their PIDs. 2. Create a systemd unit file for a simple Node.js "hello world" HTTP server. Start it, enable it, reboot, and verify it comes back up automatically. **Intermediate:** 1. Your app log at `/var/log/my-app/app.log` is 2 GB. Use `grep` to find all lines containing "ERROR" from the last 24 hours, count them, and save the output to a separate file. Then use `logrotate` to configure automatic rotation. 2. Write a crontab entry that runs a health-check script every minute, and if the health check fails 3 times in a row (track with a temp file), sends an alert. Use only bash and cron — no external monitoring tools. **Advanced:** 1. Simulate a "mystery outage": have a friend break something on a test server (change a permission, fill the disk with a large file, kill a process). Diagnose and fix it using only the commands in this chapter. Document your debugging steps. 2. Write a bash script that runs as a cron job and generates a daily server health report: CPU load average, memory usage, disk usage, top 5 processes by CPU, list of failed systemd services, and recent auth failures from `/var/log/auth.log`. Email the report. --- ## Interview questions **Beginner:** 1. **Q: How do you check what processes are running on a Linux server?** **A:** `ps aux` for a snapshot, `top` or `htop` for real-time monitoring. `ps aux | grep ` to find a specific process. `pstree -p` to see the process tree with parent-child relationships. 2. **Q: What's the difference between `kill -9` and `kill -15`?** **A:** `kill -15` (SIGTERM) is a polite request — the process can catch it, clean up resources, close connections, and exit gracefully. `kill -9` (SIGKILL) is the kernel forcibly terminating the process — no cleanup, no chance to respond. Always try SIGTERM first. 3. **Q: How do you make a service start automatically when the server reboots?** **A:** With systemd: create a `.service` unit file in `/etc/systemd/system/`, then run `systemctl enable `. This creates a symlink that tells systemd to start the service at boot. Without systemd, you'd use crontab's `@reboot` directive or init.d scripts. **Senior:** 1. **Q: Your server's load average is 8.0 but CPU usage is only 15%. What's likely happening, and how do you investigate?** **A:** High load with low CPU typically indicates I/O wait — processes are stuck waiting for disk. Check `top` and look at the `%wa` (I/O wait) column. Run `iostat -x 1` to see which disk device is saturated and at what utilization percentage. Use `iotop` to identify which process is causing the I/O. Common culprits: database without proper indexes causing full table scans, log files on slow HDD, or a backup job reading/writing large files. The fix depends on the root cause — add indexes, move to SSD, or schedule I/O-heavy jobs during low-traffic hours. 2. **Q: How would you design a systemd unit file for a production Node.js application? What security hardening would you include?** **A:** Key elements: `Type=simple` for Node.js (it doesn't fork), `User=appuser` (never root), `Restart=on-failure` with `RestartSec=5`, `EnvironmentFile` for secrets, `StandardOutput/Error=journal` for centralized logging. Security hardening: `NoNewPrivileges=yes` prevents privilege escalation, `PrivateTmp=yes` gives the service its own `/tmp`, `ProtectSystem=strict` makes the entire filesystem read-only except explicitly allowed paths via `ReadWritePaths`, `ProtectHome=yes` hides home directories, `RestrictAddressFamilies=AF_INET AF_INET6` limits network protocols the service can use. Also consider `MemoryMax`, `CPUQuota`, and `LimitNOFILE` to set resource limits. 3. **Q: Walk me through how you'd debug "the API is responding with 502 Bad Gateway" on a server running Nginx as a reverse proxy to a Node.js backend.** **A:** First, check if the backend is running: `systemctl status my-api`. If it's running, check if it's actually listening: `ss -tlnp | grep `. If listening, check if it's healthy: `curl -v http://localhost:/health`. If that works, the issue is between Nginx and the backend — check Nginx config for correct `proxy_pass` URL. Check Nginx error logs: `journalctl -u nginx --since "5 min ago"`. Common causes: backend process is running but hung (accepting connections but not responding), port mismatch in Nginx config, or the backend is running but on a different port after a restart. If the backend isn't running, check its logs with `journalctl -u my-api` for the crash reason. If it's restarting in a loop, check `journalctl -u my-api -p err` for the recurring error — often a misconfigured environment variable or a database connection failure. --- ## Summary Linux is the operating system that runs the internet. As a backend developer, you don't need to be a sysadmin, but you need to be dangerous enough to diagnose problems when they happen. The core workflow is: SSH into the server, check what's running (`ps`, `top`, `systemctl status`), read the logs (`journalctl`), check resources (`df`, `free`, `ss`), and find the problem (`grep`, `find`, `lsof`). systemd manages your services — learn to write a proper unit file with restart policies and security hardening. File permissions (`chmod`, `chown`) protect your secrets. Cron handles scheduled tasks. Master these 20 commands and you'll move from "I need DevOps to check something" to "I've already found and fixed it." ## Quick recall - **SSH**: `ssh user@host`, use keys, not passwords - **Processes**: `ps aux`, `top`/`htop`, `kill -15` before `kill -9` - **systemd**: `systemctl start/stop/status/enable`, unit files in `/etc/systemd/system/` - **Logs**: `journalctl -u -f`, `-p err` for errors, `--since` for time windows - **Search**: `grep -rn "pattern" /path`, `find /path -name "*.log"` - **Permissions**: `chmod 600` for secrets, `755` for scripts, `644` for files - **Disk/Memory**: `df -h`, `du -sh *`, `free -h` - **Network**: `ss -tlnp`, `lsof -i :port`, `curl -v` - **Packages**: `apt update && apt upgrade`, `apt install ` - **Scheduling**: `crontab -e`, format: `min hour day month weekday command` ## Quiz 1. What's the difference between `ps aux` and `top`? **Answer: `ps aux` is a one-time snapshot; `top` updates in real time.** 2. How do you view logs for a specific systemd service in real time? **Answer: `journalctl -u -f`** 3. Your app crashes and you want it to restart automatically. What do you add to the systemd unit file? **Answer: `Restart=on-failure` and `RestartSec=5`** 4. What does a load average of 4.0 mean on a 2-core machine? **Answer: On average, 2 processes are waiting for CPU time — the system is overloaded.** 5. Why should you never run your backend application as root? **Answer: A bug or vulnerability in your app then has unrestricted access to the entire system — any file, any port, any system call.** 6. What's the `wa` column in `top` and why should you care? **Answer: I/O wait — the percentage of time the CPU is idle waiting for disk. High `wa` means your disk is the bottleneck, not your code.** 7. How do you find which process is listening on port 3000? **Answer: `lsof -i :3000` or `ss -tlnp | grep 3000`**

Chapter 1 — Linux Basics for Backend

Part of L6