Chapter 1 — Linux Basics for Backend

The why

You can build the most elegant backend in the world, but when it's running on a Linux box at 2 AM and you need to figure out why memory spiked, your IDE won't help you. The terminal will.

This chapter covers the 20 commands you'll use every single day as a backend developer. Not all 2,000 Linux commands — just the ones that matter for running and debugging backend services in production.

Core concepts

Connecting to your server: SSH

SSH (Secure Shell) is how you get into a remote Linux machine. If you take one thing from this chapter, make it this workflow:

# Connect to a remote server
ssh user@your-server-ip
 
# Connect with a specific identity file (key)
ssh -i ~/.ssh/my-key.pem ubuntu@ec2-instance.compute.amazonaws.com
 
# Run a single command remotely without opening a shell
ssh user@server "systemctl status nginx"
 
# Copy files securely
scp ./app.tar.gz user@server:/home/user/deploy/

Set up key-based auth. Passwords over SSH are a liability. Generate a key pair:

ssh-keygen -t ed25519 -C "your-email@example.com"
ssh-copy-id user@your-server

Now you can connect without typing a password. Keep your private key (~/.ssh/id_ed25519) safe — it's the master key to your server.

What's running? Process management

The first question you ask when something feels wrong: "What's actually running?"

# List all processes (a = all users, u = user format, x = no tty)
ps aux
 
# Find a specific process
ps aux | grep node
 
# Interactive process viewer (real-time)
top
 
# Better top — colors, mouse, scrollable
htop
 
# Kill a process by PID
graceful: kill -15 1234
forceful: kill -9 1234
 

Reading top output in 30 seconds:

top - 14:23:11 up 45 days,  2:15,  1 user,  load average: 0.85, 0.62, 0.58
Tasks: 156 total,   1 running, 155 sleeping,   0 stopped,   0 zombie
%Cpu(s): 12.3 us,  3.1 sy,  0.0 ni, 84.2 id,  0.2 wa,  0.0 hi,  0.2 si,  0.0 st
MiB Mem:   7986.2 total,   6234.1 used,   1752.1 free,    234.5 buff/cache

What to look for:

• load average: three numbers = 1, 5, 15 min averages. If > number of CPU cores, the system is overloaded.
• %Cpu(s): us = user space (your app), sy = system/kernel, wa = waiting for I/O (disks are slow), id = idle. High wa means your disk is the bottleneck.
• Mem: Check free — if it's near zero and swap is high, you're out of memory.
• zombie processes: Should be 0. Zombies are finished processes whose parents haven't cleaned them up.

Systemd: The service manager

systemd is the init system on virtually every modern Linux distribution. It starts, stops, and monitors services (your backend app is a service).

# Check if your app is running
systemctl status my-app
 
# Start / Stop / Restart
sudo systemctl start my-app
sudo systemctl stop my-app
sudo systemctl restart my-app
 
# Enable at boot (starts automatically after reboot)
sudo systemctl enable my-app
 
# Disable at boot
sudo systemctl disable my-app

Writing a systemd unit file:

[Unit]
Description=My Node.js Backend API
After=network.target postgresql.service redis.service
 
[Service]
Type=simple
User=appuser
Group=appuser
WorkingDirectory=/opt/my-app
ExecStart=/usr/bin/node /opt/my-app/dist/server.js
Restart=on-failure
RestartSec=5
EnvironmentFile=/opt/my-app/.env

After creating or editing a unit file:

sudo systemctl daemon-reload    # Reload systemd config
sudo systemctl start my-app     # Start the service
sudo systemctl enable my-app    # Auto-start on boot

Logs: journalctl

When your app crashes at 3 AM, logs are your only witness. systemd captures stdout/stderr from your service automatically.

# View logs for a specific service
journalctl -u my-app
 
# Follow logs in real-time (like tail -f)
journalctl -u my-app -f
 
# Last 100 lines
journalctl -u my-app -n 100
 
# Logs since last boot
journalctl -u my-app -b
 
# Logs in a time range
journalctl -u my-app

Finding stuff: grep and find

grep searches inside files. find searches for files themselves. Master both.

# Find "ERROR" in all .log files, with line numbers
grep -rn "ERROR" /var/log/my-app/
 
# Case-insensitive, show 3 lines of context
grep -rni -C 3 "timeout" /opt/my-app/
 
# Search only .js files
grep -rn "TODO" --include="*.js" ./src/
 
# Invert match (lines that DON'T contain this)

Disk and memory: df, du, free

Your server runs out of disk or memory — it's a matter of when, not if.

# Disk space summary (human-readable)
df -h
 
# Check specific partition
df -h /var
 
# What's eating my disk? (top-level)
du -sh /* 2>/dev/null | sort -hr | head -10
 
# What's eating THIS directory?
du -sh * | sort -hr

File permissions: chmod, chown

Every file and directory has an owner, a group, and three permission sets (read/write/execute for owner, group, and others).

# See permissions
ls -la
# -rwxr-xr--  1 appuser appgroup  4096 May 13 14:23 server.js
#  ^^^ ^^^ ^^^
#  owner group others
 
# Change permissions
chmod 755 script.sh       # rwx r-x r-x
chmod 600 .env            # rw- --- --- (only owner)
chmod 644 public.html     #

Permission numbers demystified: r=4, w=2, x=1. So 7 = rwx, 5 = r-x, 6 = rw-. 755 = owner can do everything, group and others can read+execute.

Networking: curl, lsof, ss

# HTTP requests (the swiss army knife)
curl -X GET https://api.example.com/health
curl -X POST -H "Content-Type: application/json" -d '{"key":"val"}' https://api.example.com/data
curl -v https://api.example.com     # verbose (see headers, TLS handshake)
curl -I https://api.example.com     # headers only
 
# What's listening on which port?
ss -tlnp

Package management

# Debian/Ubuntu
sudo apt update                  # Refresh package list
sudo apt upgrade                 # Upgrade all packages
sudo apt install nginx           # Install
sudo apt remove nginx            # Remove (keep configs)
sudo apt purge nginx             # Remove (including configs)

Scheduling: crontab

# Edit your crontab
crontab -e
 
# List your crontab
crontab -l
 
# Format: minute hour day month weekday command
 
# Every day at 2 AM — run database backup
0 2 * * * /opt/scripts/backup-db.sh >> /var/log/backup.log 2>&1
 
# Every 5 minutes — health check
*/5 * *

Real-world example: How a 5-person startup debugs a production outage

Picture this: A SaaS startup running a Node.js API on an AWS EC2 instance. It's 11 PM. A customer Slack messages: "App is down."

The on-call developer SSHes into the box and runs through a mental checklist they've internalized:

1. systemctl status api — Service is "active (running)" but has been restarting every 30 seconds. The Restart=on-failure in the unit file is looping.

2. journalctl -u api --since "5 minutes ago" -p err — Sees: FATAL: connection to database timed out. The app can't reach PostgreSQL.

3. systemctl status postgresql — PostgreSQL is running. But is it accepting connections?

4. ss -tlnp | grep 5432 — Port 5432 is listening on 127.0.0.1 only. The app's new config changed the DB host to the public IP. PostgreSQL wasn't configured to listen on that interface.

5. Fix: Edit postgresql.conf, set listen_addresses = '*', restart PostgreSQL. Or (better) revert the app config to use 127.0.0.1.

Total time from alert to fix: 8 minutes. No guessing. No "let me try restarting everything." Just systematic diagnosis with the commands you now know. That's the difference between a developer who panics and a developer who debugs.

Common mistakes

Production notes

• Use a non-root user for everything. Create appuser with minimal permissions. The systemd unit should specify User=appuser. Your SSH config should disable root login (PermitRootLogin no in /etc/ssh/sshd_config).
• Set up log rotation. Logs grow until your disk is full. Configure logrotate for your app logs or use journalctl --vacuum-size=500M to cap journal size.
• Monitor disk space proactively. Set up a cron job that alerts you when disk usage exceeds 80%: df -h / | awk 'NR==2 {print $5}' | tr -d '%'.
• Keep a runbook. Document the exact commands to restart each service, check logs, and diagnose common failures. At 2 AM, you won't remember the journalctl flags.

Performance notes

• htop is your first stop for CPU/memory investigation. Sort by CPU (P) or memory (M). Look for processes consuming disproportionate resources.
• High load average with low CPU usage usually means I/O wait. Check iostat -x 1 to see which disk is saturated.
• free -h shows available memory — that's the real number. Linux will use spare RAM for cache; it's released instantly when needed.
• Use nice and renice to deprioritize batch jobs so they don't starve your API of CPU: nice -n 19 ./batch-job.sh.

Security notes

• Disable password authentication for SSH. Edit /etc/ssh/sshd_config: set PasswordAuthentication no, PermitRootLogin no, then systemctl restart sshd.
• Use ufw (Uncomplicated Firewall) to block everything except needed ports. ufw allow 22/tcp && ufw allow 80/tcp && ufw allow 443/tcp && ufw enable.
• fail2ban blocks IPs after repeated failed SSH attempts. Install it, enable the SSH jail — it's a 5-minute setup that stops 99% of brute-force attacks.
• Never hardcode secrets in crontab or scripts. Use environment files (mode 600) or a secrets manager.
• Keep the system updated. apt update && apt upgrade weekly. Subscribe to your distro's security mailing list.

Exercises

Beginner:

1. SSH into a remote machine (or a local VM). Run htop, sort by memory, and identify the top 3 memory-consuming processes. Write them down with their PIDs.
2. Create a systemd unit file for a simple Node.js "hello world" HTTP server. Start it, enable it, reboot, and verify it comes back up automatically.

Intermediate:

1. Your app log at /var/log/my-app/app.log is 2 GB. Use grep to find all lines containing "ERROR" from the last 24 hours, count them, and save the output to a separate file. Then use logrotate to configure automatic rotation.
2. Write a crontab entry that runs a health-check script every minute, and if the health check fails 3 times in a row (track with a temp file), sends an alert. Use only bash and cron — no external monitoring tools.

Advanced:

1. Simulate a "mystery outage": have a friend break something on a test server (change a permission, fill the disk with a large file, kill a process). Diagnose and fix it using only the commands in this chapter. Document your debugging steps.
2. Write a bash script that runs as a cron job and generates a daily server health report: CPU load average, memory usage, disk usage, top 5 processes by CPU, list of failed systemd services, and recent auth failures from /var/log/auth.log. Email the report.

Interview questions

Beginner:

1. Q: How do you check what processes are running on a Linux server? A: ps aux for a snapshot, top or htop for real-time monitoring. ps aux | grep <name> to find a specific process. pstree -p to see the process tree with parent-child relationships.

2. Q: What's the difference between kill -9 and kill -15? A: kill -15 (SIGTERM) is a polite request — the process can catch it, clean up resources, close connections, and exit gracefully. kill -9 (SIGKILL) is the kernel forcibly terminating the process — no cleanup, no chance to respond. Always try SIGTERM first.

3. Q: How do you make a service start automatically when the server reboots? A: With systemd: create a .service unit file in /etc/systemd/system/, then run systemctl enable <service-name>. This creates a symlink that tells systemd to start the service at boot. Without systemd, you'd use crontab's @reboot directive or init.d scripts.

Senior:

1. Q: Your server's load average is 8.0 but CPU usage is only 15%. What's likely happening, and how do you investigate? A: High load with low CPU typically indicates I/O wait — processes are stuck waiting for disk. Check top and look at the %wa (I/O wait) column. Run iostat -x 1 to see which disk device is saturated and at what utilization percentage. Use iotop to identify which process is causing the I/O. Common culprits: database without proper indexes causing full table scans, log files on slow HDD, or a backup job reading/writing large files. The fix depends on the root cause — add indexes, move to SSD, or schedule I/O-heavy jobs during low-traffic hours.

2. Q: How would you design a systemd unit file for a production Node.js application? What security hardening would you include? A: Key elements: Type=simple for Node.js (it doesn't fork), User=appuser (never root), Restart=on-failure with RestartSec=5, EnvironmentFile for secrets, StandardOutput/Error=journal for centralized logging. Security hardening: NoNewPrivileges=yes prevents privilege escalation, PrivateTmp=yes gives the service its own , makes the entire filesystem read-only except explicitly allowed paths via , hides home directories, limits network protocols the service can use. Also consider , , and to set resource limits.

Summary

Linux is the operating system that runs the internet. As a backend developer, you don't need to be a sysadmin, but you need to be dangerous enough to diagnose problems when they happen. The core workflow is: SSH into the server, check what's running (ps, top, systemctl status), read the logs (journalctl), check resources (df, free, ss), and find the problem (grep, find, lsof). systemd manages your services — learn to write a proper unit file with restart policies and security hardening. File permissions (chmod, chown) protect your secrets. Cron handles scheduled tasks. Master these 20 commands and you'll move from "I need DevOps to check something" to "I've already found and fixed it."

Quick recall

• SSH: ssh user@host, use keys, not passwords
• Processes: ps aux, top/htop, kill -15 before kill -9
• systemd: systemctl start/stop/status/enable, unit files in /etc/systemd/system/
• Logs: journalctl -u <service> -f, -p err for errors, --since for time windows
• Search: grep -rn "pattern" /path, find /path -name "*.log"
• Permissions: chmod 600 for secrets, 755 for scripts, 644 for files
• Disk/Memory: , ,

Quiz

1. What's the difference between ps aux and top? Answer: ps aux is a one-time snapshot; top updates in real time.
2. How do you view logs for a specific systemd service in real time? Answer: journalctl -u <service> -f
3. Your app crashes and you want it to restart automatically. What do you add to the systemd unit file? Answer: Restart=on-failure and RestartSec=5
4. What does a load average of 4.0 mean on a 2-core machine? Answer: On average, 2 processes are waiting for CPU time — the system is overloaded.
5. Why should you never run your backend application as root? Answer: A bug or vulnerability in your app then has unrestricted access to the entire system — any file, any port, any system call.
6. What's the wa column in top and why should you care? Answer: I/O wait — the percentage of time the CPU is idle waiting for disk. High wa means your disk is the bottleneck, not your code.
7. How do you find which process is listening on port 3000? Answer: lsof -i :3000 or ss -tlnp | grep 3000