# Chapter 3 — Docker Compose: Multi-Container Apps By the end of this chapter you will: - Compose a full stack of services - Use named volumes for data persistence - Configure healthchecks and dependency ordering - Run the same compose in dev and CI --- ## The why > Your app doesn't run in isolation. It needs a database, a cache, a message queue, maybe a search engine. Docker Compose lets you spin up the entire stack with one command — and tear it down just as easily. An orchestra has violins, cellos, brass, woodwinds, and percussion. Each section can play alone, but the symphony only happens when they play together, in sync, following the same tempo. The conductor doesn't play any instrument — they coordinate: "violins, you enter here; brass, wait for the cellos to finish; percussion, softer in this movement." Docker Compose is the conductor. Each service (API, database, Redis, worker) is a section of the orchestra. Compose coordinates them: which services start first, how they discover each other on the network, where their data lives, and how to check if they're actually ready to play. One `docker compose up` and the entire symphony starts. --- ## Core concepts ### What Docker Compose is Docker Compose is a tool for defining and running multi-container Docker applications. You describe your entire stack in a single YAML file (`docker-compose.yml`) and manage it with a single set of commands. ```bash # Start everything docker compose up -d # See what's running docker compose ps # View logs from all services docker compose logs -f # Stop everything docker compose down # Stop AND delete volumes (reset database) docker compose down -v ``` ### The docker-compose.yml structure Every Compose file has at minimum a `services` section. Networks and volumes are optional but almost always needed. ```yaml title="docker-compose.yml" version: "3.8" services: # Each service becomes a container api: build: . ports: - "3000:3000" environment: - DATABASE_URL=postgres://postgres:password@db:5432/myapp - REDIS_URL=redis://cache:6379 depends_on: db: condition: service_healthy cache: condition: service_started volumes: - ./uploads:/app/uploads restart: unless-stopped db: image: postgres:16-alpine environment: POSTGRES_USER: postgres POSTGRES_PASSWORD: password POSTGRES_DB: myapp volumes: - pgdata:/var/lib/postgresql/data healthcheck: test: ["CMD-SHELL", "pg_isready -U postgres"] interval: 10s timeout: 5s retries: 5 cache: image: redis:7-alpine volumes: - redisdata:/data healthcheck: test: ["CMD", "redis-cli", "ping"] interval: 10s timeout: 3s retries: 5 worker: build: context: . dockerfile: Dockerfile.worker environment: - REDIS_URL=redis://cache:6379 - DATABASE_URL=postgres://postgres:password@db:5432/myapp depends_on: cache: condition: service_healthy db: condition: service_healthy restart: unless-stopped volumes: pgdata: redisdata: ``` Let's break down every piece of this file. ### Services Each entry under `services` defines a container. The key (e.g., `api`, `db`, `cache`) becomes the container name and the DNS hostname on the default network. ```yaml services: api: # Other containers reach this at http://api:3000 db: # Other containers reach this at db:5432 cache: # Other containers reach this at cache:6379 ``` **Using an existing image vs building:** ```yaml # Use a pre-built image from Docker Hub db: image: postgres:16-alpine # Build from a Dockerfile in the current directory api: build: . # Build from a specific Dockerfile worker: build: context: . dockerfile: Dockerfile.worker ``` ### Networks Compose automatically creates a default network for your app. All services join it and can resolve each other by service name. This is Docker's built-in DNS at work. For more complex topologies, define custom networks: ```yaml networks: frontend: backend: services: api: networks: - frontend - backend # Can talk to both frontend and backend db: networks: - backend # Only on backend network — isolated from frontend ``` This is a security pattern: the database is on a private network that only the API can reach. The frontend (Nginx, for example) can't directly talk to the database. ### Volumes Named volumes persist data across `docker compose down` (unless you use `-v`). They're listed at the top level and referenced by services: ```yaml volumes: pgdata: # Declare the volume redisdata: services: db: volumes: - pgdata:/var/lib/postgresql/data # Mount it ``` `docker compose down` stops and removes containers but keeps volumes. `docker compose down -v` also removes volumes — your database is gone. Useful for development resets, dangerous in production. If you're using Compose in production (not ideal, but common for small projects), never use `-v` casually. ### Healthchecks A container can be "running" but not "ready." PostgreSQL starts instantly, but it takes 5-10 seconds to accept connections. Without healthchecks, your API tries to connect too early and crashes. ```yaml healthcheck: test: ["CMD-SHELL", "pg_isready -U postgres"] interval: 10s # Check every 10 seconds timeout: 5s # Fail if check takes > 5 seconds retries: 5 # Mark unhealthy after 5 consecutive failures start_period: 15s # Grace period before checks begin ``` **Common healthcheck commands:** ```yaml # PostgreSQL healthcheck: test: ["CMD-SHELL", "pg_isready -U postgres"] # Redis healthcheck: test: ["CMD", "redis-cli", "ping"] # Custom API endpoint healthcheck: test: ["CMD", "wget", "--spider", "-q", "http://localhost:3000/health"] # MySQL/MariaDB healthcheck: test: ["CMD", "mysqladmin", "ping", "-h", "localhost"] ``` ### depends_on and startup ordering `depends_on` controls startup order, but there's a critical nuance: ```yaml # Basic: api starts AFTER db starts (but db might not be READY yet) depends_on: - db ``` ```yaml # Long syntax with condition: api starts only after db is HEALTHY depends_on: db: condition: service_healthy ``` Without `condition: service_healthy`, `depends_on` only waits for the container to START — not for the service inside to be ready. PostgreSQL's container starts instantly, but the database isn't accepting connections for 5-10 seconds. Your API will crash with "connection refused" unless you use healthcheck-based dependencies. This is the #1 Docker Compose mistake. ### Environment variables Three ways to pass config to services: ```yaml # 1. Inline (fine for non-secret values) environment: - NODE_ENV=production - LOG_LEVEL=info # 2. From an env file (good for development) env_file: - .env # 3. Explicit file path env_file: - ./config/api.env ``` Compose automatically reads variables from a `.env` file in the same directory. You can use these variables in the Compose file with `${VARIABLE}` syntax: `POSTGRES_PASSWORD: ${DB_PASSWORD}`. Keep `.env` in `.gitignore` — never commit secrets. ### Profiles: Optional services Not every service needs to run every time. Profiles let you selectively include services: ```yaml services: api: # No profile — always starts db: # No profile — always starts adminer: # Database GUI — only in development image: adminer profiles: - dev - debug ngrok: # Expose local server to internet — only when testing webhooks image: ngrok/ngrok profiles: - debug ``` ```bash # Start only default services (no profiles) docker compose up -d # Start with dev profile (includes adminer) docker compose --profile dev up -d # Start with both dev and debug profiles docker compose --profile dev --profile debug up -d ``` ### Override files `docker-compose.override.yml` is automatically merged with `docker-compose.yml` (no `-f` flag needed). Use it for development-specific config: ```yaml title="docker-compose.override.yml" services: api: build: target: development # Use dev stage from multi-stage Dockerfile volumes: - .:/app # Bind-mount source for hot reload - /app/node_modules # Anonymous volume — don't override node_modules environment: - NODE_ENV=development - DEBUG=express:* command: npm run dev ``` Production overrides go in a separate file you specify explicitly: ```bash docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d ``` ### Essential Compose commands ```bash # Start in detached mode docker compose up -d # Start and rebuild images docker compose up -d --build # Start specific services docker compose up -d api db # Scale a service (limited — better for dev/testing) docker compose up -d --scale worker=3 # View logs docker compose logs -f # All services docker compose logs -f api # Specific service docker compose logs --tail=50 # Last 50 lines # Execute a command in a running service docker compose exec api sh docker compose exec db psql -U postgres -d myapp # Run a one-off command (new container, removed after) docker compose run --rm api npm run migrate # Restart a single service docker compose restart api # Stop and remove everything docker compose down # Stop, remove, and delete volumes docker compose down -v # List running services docker compose ps # Pull latest images docker compose pull ``` --- ## Real-world example: How a SaaS team onboards developers in 5 minutes A B2B SaaS company with a 12-service architecture (API, 3 workers, PostgreSQL, Redis, Elasticsearch, Nginx, and a few microservices) had a painful onboarding problem. New developers spent their first 2-3 days setting up the local environment: - Install PostgreSQL 15 (but production runs 16 — subtle version bugs) - Install Redis (different config on Mac vs Linux) - Clone 4 separate repos and figure out which port each runs on - Manually create databases, run migrations, seed data - Debug "it works on my machine" issues for days They created a single `docker-compose.yml` in the main repo that defined every service. A `docker-compose.override.yml` added development conveniences (hot reload, debug ports, local volumes). New developer onboarding became: ```bash git clone repo docker compose up -d # Everything starts docker compose run --rm api npm run db:setup # Migrate + seed ``` Five minutes from zero to a fully working stack. The Compose file became the single source of truth for "what does our architecture look like?" — better than any diagram, because it's executable. When they added Elasticsearch to production, they added 8 lines to `docker-compose.yml`. Every developer got Elasticsearch the next time they pulled. No "oh, you need to install Elasticsearch separately, here's a wiki page from 2019 that might still work." --- ## Common mistakes | Mistake | Why it's wrong | Fix | |---------|---------------|-----| | `depends_on` without `condition: service_healthy` | API starts before DB is ready to accept connections | Add healthchecks and use `condition: service_healthy` | | Hardcoding credentials in `docker-compose.yml` | Committed to git — everyone who clones the repo has your DB password | Use `.env` file (gitignored) with `${VARIABLE}` substitution | | Not using named volumes for databases | Data is lost on `docker compose down` | Define volumes at top level and mount them in the service | | Exposing database ports to host in production | `ports: - "5432:5432"` opens your DB to the world if the host firewall is misconfigured | Use internal Docker networks only; no `ports` for databases | | One giant Compose file for everything | Rebuilding the API triggers unnecessary restarts of Redis, DB, etc. | Split into logical Compose files; or use profiles for optional services | | Using `:latest` for database images | A minor version bump can corrupt data or change behavior silently | Pin exact versions: `postgres:16.3-alpine` | | Forgetting `docker compose down -v` resets data | In dev, you accidentally nuke your local database | Use `docker compose down` (no `-v`) for routine stops; `-v` only when you want a fresh start | --- ## Production notes - **Compose is great for development and single-server production.** For multi-server, multi-host deployments, you graduate to Kubernetes or Docker Swarm. Compose's sweet spot is local dev, CI environments, and small production deployments on a single VM. - **Use `restart: unless-stopped`** on every service. This ensures containers come back after a reboot or Docker daemon restart. For critical services, `restart: always` is even stronger. - **Pin image versions with digest in production.** `image: postgres:16.3-alpine@sha256:abc123...` ensures you're running the exact same bits everywhere. - **Monitor with `docker compose ps` in your health-check scripts.** If any service shows `unhealthy` or `restarting`, alert immediately. ## Performance notes - **Use `deploy.resources` to set CPU/memory limits in Compose v3+.** `deploy: resources: limits: { cpus: '1.0', memory: 512M }`. This prevents one container from starving others. - **The `:delegated` mount option** (`./src:/app/src:delegated`) tells Docker that host-to-container sync can be delayed — improves I/O performance on macOS/Windows for development volumes. - **Database initialization scripts** (`/docker-entrypoint-initdb.d/`) run on first container start only. If you need them to re-run, delete the volume first: `docker compose down -v && docker compose up -d`. ## Security notes - **Never expose database ports to the host in production Compose files.** Databases should only be accessible on the internal Docker network. If you need external access, use an SSH tunnel. - **Use `secrets` for sensitive config in Docker Swarm mode.** For standard Compose, use `.env` files with restrictive permissions (`chmod 600 .env`). - **Run containers as non-root even in Compose.** Add `user: "1000:1000"` to service definitions, or handle it in the Dockerfile. - **Use internal networks** to isolate services that don't need to talk to each other. The worker doesn't need to reach the frontend Nginx. --- ## Exercises **Beginner:** 1. Write a `docker-compose.yml` with two services: a Node.js API and a PostgreSQL database. Use environment variables for the database connection. Start both with `docker compose up -d` and verify the API can connect to the database. 2. Add a Redis service to your Compose file. Use `docker compose exec` to connect to Redis (`redis-cli`) and run `PING`. Then use `docker compose logs` to view logs from all three services. **Intermediate:** 1. Add healthchecks to your PostgreSQL and Redis services. Modify the API's `depends_on` to use `condition: service_healthy`. Test that the API waits for the database to be truly ready — artificially delay the DB startup (add a slow init script) and observe. 2. Create a `docker-compose.override.yml` that adds development conveniences: bind-mount source code for hot reload, enable debug ports, and use a `dev` profile for an Adminer (database GUI) service. Start with and without the dev profile. **Advanced:** 1. Build a Compose file for a realistic stack: API, worker (BullMQ + Redis), PostgreSQL, Redis, and Nginx reverse proxy. Configure a custom network so the database is on a backend network only the API and worker can access. Add healthchecks everywhere. Test graceful shutdown: `docker compose down` and verify all services stop cleanly. 2. Write a shell script that uses `docker compose run --rm` to run database migrations, then starts the stack. If migrations fail, the script should NOT start the services. Add this script as a `pre-up` hook or document it as the startup procedure. --- ## Interview questions **Beginner:** 1. **Q: What is Docker Compose and when would you use it?** **A:** Docker Compose is a tool for defining and running multi-container Docker applications. You describe services, networks, and volumes in a YAML file and manage them with `docker compose` commands. Use it for local development, CI testing environments, and small production deployments on a single server. 2. **Q: How do containers in a Compose file discover each other?** **A:** Compose creates a default network. Each service's name becomes a DNS hostname. So `api` can reach `db` at `db:5432`. Docker's built-in DNS resolves service names to container IPs automatically. 3. **Q: What's the difference between `docker compose down` and `docker compose down -v`?** **A:** `down` stops and removes containers and the default network. `down -v` also removes named volumes — all persistent data is deleted. Use `-v` for development resets; avoid it in production unless you're certain. **Senior:** 1. **Q: Explain the problem with `depends_on` without healthchecks, and how you solve it.** **A:** `depends_on` without `condition: service_healthy` only waits for the dependent container to start — not for the service inside to be ready. A PostgreSQL container starts instantly, but the database takes 5-10 seconds to begin accepting connections. If your API starts during that window, it gets "connection refused" and may crash. The fix is: (1) Add a `healthcheck` to the database service (e.g., `pg_isready`), (2) use the long syntax `depends_on: db: condition: service_healthy`. Compose then waits for the healthcheck to pass before starting the API. Additionally, your API should have retry logic with exponential backoff for database connections — healthchecks reduce the problem but robust apps handle transient failures. 2. **Q: How would you structure Compose files for a project with significantly different dev, staging, and production configurations?** **A:** Use a base `docker-compose.yml` with common service definitions. Then create override files: `docker-compose.dev.yml` (bind mounts, hot reload, debug ports, dev profiles), `docker-compose.staging.yml` (CI-appropriate config, test secrets, no volumes for databases), `docker-compose.prod.yml` (pinned image digests, resource limits, restart policies, secrets from external sources). Developers just run `docker compose up -d` (override is auto-applied). CI runs `docker compose -f docker-compose.yml -f docker-compose.staging.yml up -d`. Production runs `docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d`. Key principle: the base file changes rarely; environment-specific differences are isolated in override files. 3. **Q: How do you handle database migrations in a Docker Compose setup — before the API starts serving traffic?** **A:** Several approaches: (1) A separate migration service that runs `docker compose run --rm api npm run migrate` before `docker compose up` — the migration runs to completion, then services start. (2) An init container pattern: add a service with `command: npm run migrate && echo 'done'`, have the API `depends_on` this migration service with `condition: service_completed_successfully` (Compose v2.1+). (3) Run migrations as part of the API's startup script — the API runs migrations on boot before starting the HTTP listener. Approach 1 is simplest and most common. Approach 2 is elegant but requires careful handling of the migration service exiting. Approach 3 is risky in scaled environments (multiple API instances racing to migrate). --- ## Summary Docker Compose turns "I need to start 5 services to work on this feature" into `docker compose up -d`. You define your entire stack — API, database, cache, worker, reverse proxy — in a single YAML file. Services communicate over a shared network using their names as hostnames. Named volumes persist data across restarts. Healthchecks ensure services are truly ready before dependents start. Profiles let you include optional services (admin tools, debug proxies) without bloating the default configuration. Override files separate development concerns (hot reload, debug ports) from production concerns (resource limits, pinned versions). Compose won't replace Kubernetes for large-scale orchestration, but for local development, CI environments, and single-server deployments, it's the tool that makes multi-service development practical. ## Quick recall - **`docker-compose.yml`** defines services, networks, and volumes. - **Service name = DNS hostname** on the default network. - **`depends_on` with `condition: service_healthy`** is essential — without it, you get race conditions on startup. - **Named volumes** persist data; declared at top level, mounted in services. - **`.env` file** provides variable substitution: `${DB_PASSWORD}` keeps secrets out of the Compose file. - **Profiles** (`--profile dev`) include optional services without changing the base file. - **Override files** (`docker-compose.override.yml`) auto-merge; use for dev-specific config. - **`docker compose exec `** runs commands in running containers. - **`docker compose run --rm `** runs one-off commands in fresh containers. - **`docker compose down -v`** nukes volumes — useful for dev resets, dangerous in production. ## Quiz 1. How do containers in a Compose file communicate with each other? **Answer: By using the service name as the hostname — Docker's built-in DNS resolves it to the container's IP on the shared network.** 2. What's the difference between `depends_on: - db` and `depends_on: db: condition: service_healthy`? **Answer: The first only waits for the container to start (not the service inside to be ready). The second waits for the healthcheck to pass.** 3. Where should database credentials be stored instead of hardcoding them in `docker-compose.yml`? **Answer: In a `.env` file (gitignored), referenced in the Compose file with `${VARIABLE}` syntax.** 4. What happens to your database data when you run `docker compose down` vs `docker compose down -v`? **Answer: `down` stops and removes containers but preserves named volumes (data survives). `down -v` also deletes volumes (data is gone).** 5. How do you add an optional service (like a database admin GUI) that doesn't start by default? **Answer: Use `profiles: - dev` on that service, then start with `docker compose --profile dev up -d`.** 6. What's the purpose of `docker-compose.override.yml`? **Answer: It's automatically merged with `docker-compose.yml` and typically contains development-specific overrides like bind mounts, hot reload, and debug ports.** 7. How do you run a one-time database migration using Docker Compose? **Answer: `docker compose run --rm api npm run migrate` — creates a new container, runs the command, removes the container when done.**

Chapter 3 — Docker Compose: Multi-Container Apps

Part of L6