# Chapter 9 — Secrets Management & Encryption By the end of this chapter you will: - Store secrets in a secrets manager, not .env files - Rotate a DB password without downtime - Encrypt PII at rest with envelope encryption - Never commit a secret to git again --- ## The why > The 2024 GitGuardian report found 12.8 million secrets leaked on GitHub — a 28% increase year over year. Database passwords, API keys, private keys, tokens — all committed to repositories, often by accident, sometimes by developers who didn't know better. Once a secret hits a public repository, it's compromised. You cannot "delete" it — bots clone repositories within seconds of a push. This chapter teaches you to manage secrets like a professional. You don't keep your house key taped to the front door. You don't make 50 copies and hand them to everyone. You don't leave them on a park bench. When you lose a key, you change the locks — immediately. Secrets work the same way. Store them in a safe place (secrets manager). Give access to only who needs it (least privilege). Rotate them regularly (change the locks). And if one leaks, revoke it instantly — don't wait. --- ## Core concepts ### The secrets hierarchy — from worst to best | Method | Security | When it leaks | |--------|----------|---------------| | Hardcoded in source code | ❌ Worst | Git push = leaked. Every dev's laptop has a copy. | | `.env` file committed to git | ❌ | Same as above. `.gitignore` mistakes happen. | | `.env` file, not committed | ⚠️ | Leaked via: screenshots, logs, backups, malware on dev machine | | Environment variables (manual) | ⚠️ | Visible in process listing (`ps aux`), leaked in error pages, child processes inherit them | | Environment variables (platform) | ✅ | Vercel/Railway/Heroku env vars — encrypted at rest, access-controlled | | Secrets manager (AWS/GCP/Vault) | ✅✅ | Encrypted, access-controlled, audited, rotated automatically | | Secrets manager + KMS envelope encryption | ✅✅✅ | Best — secrets encrypted with keys that are themselves encrypted | ### Environment variables — the minimum bar ```bash # .env — NEVER commit this file. Add to .gitignore. DATABASE_URL="postgresql://user:password@localhost:5432/mydb" JWT_SECRET="super-secret-key-change-in-production" STRIPE_API_KEY="sk_live_..." # .env.example — DO commit this (without real values) DATABASE_URL="postgresql://user:password@localhost:5432/mydb" JWT_SECRET="generate-with: openssl rand -hex 64" STRIPE_API_KEY="sk_live_..." ``` `.env` files are plaintext on disk. Anyone with file system access can read them. They're better than hardcoding, but for production secrets (database passwords, API keys, signing keys), use a proper secrets manager or your platform's encrypted environment variables. ### Using a secrets manager (Vercel / AWS Secrets Manager / HashiCorp Vault) ```typescript // AWS Secrets Manager example import { SecretsManagerClient, GetSecretValueCommand } from '@aws-sdk/client-secrets-manager'; const client = new SecretsManagerClient({ region: 'us-east-1' }); async function getDatabasePassword(): Promise { const response = await client.send(new GetSecretValueCommand({ SecretId: 'prod/database/password', })); return JSON.parse(response.SecretString!).password; } // Cache the secret — don't fetch on every request let cachedPassword: string | null = null; let cachedAt = 0; async function getDatabaseUrl(): Promise { const now = Date.now(); if (cachedPassword && now - cachedAt < 300_000) { // Cache 5 minutes return `postgresql://user:${cachedPassword}@host/db`; } cachedPassword = await getDatabasePassword(); cachedAt = now; return `postgresql://user:${cachedPassword}@host/db`; } ``` ### Rotating secrets without downtime Rotating a database password while the application is running: ``` 1. Create a NEW database user/password (user_v2 / password_v2) 2. Add the new credentials to the secrets manager (as a second version or new key) 3. Deploy application code that reads BOTH credentials, tries new first, falls back to old 4. Monitor — all connections using new credentials → no errors 5. Remove old credentials from secrets manager 6. Drop the old database user ``` ```typescript // Dual-credential rotation async function createPool() { const [newCreds, oldCreds] = await Promise.all([ getSecret('database/v2').catch(() => null), getSecret('database/v1'), ]); // Try new credentials first if (newCreds) { const pool = new Pool({ connectionString: buildUrl(newCreds) }); try { await pool.query('SELECT 1'); return pool; // New credentials work! } catch { await pool.end(); } } // Fall back to old credentials return new Pool({ connectionString: buildUrl(oldCreds) }); } ``` ### Encryption at rest — protecting PII in the database If your database is compromised (backup leak, SQL injection), encrypted columns protect the data: ```typescript import crypto from 'crypto'; const ALGORITHM = 'aes-256-gcm'; const KEY = Buffer.from(process.env.ENCRYPTION_KEY!, 'hex'); // 32 bytes = 256 bits function encrypt(plaintext: string): { encrypted: string; iv: string; tag: string } { const iv = crypto.randomBytes(16); const cipher = crypto.createCipheriv(ALGORITHM, KEY, iv); let encrypted = cipher.update(plaintext, 'utf8', 'hex'); encrypted += cipher.final('hex'); const tag = cipher.getAuthTag().toString('hex'); return { encrypted, iv: iv.toString('hex'), tag }; } function decrypt(encrypted: string, iv: string, tag: string): string { const decipher = crypto.createDecipheriv(ALGORITHM, KEY, Buffer.from(iv, 'hex')); decipher.setAuthTag(Buffer.from(tag, 'hex')); let decrypted = decipher.update(encrypted, 'hex', 'utf8'); decrypted += decipher.final('utf8'); return decrypted; } ``` ### Envelope encryption (KMS) Encrypting data with AES-256 is good. But where do you store the AES key? If it's in an environment variable, and that leaks, all encrypted data is decryptable. **Envelope encryption** solves this: 1. Generate a **data key** (AES-256) for each record (or each user) 2. Encrypt the data with the data key 3. Encrypt the data key itself with a **master key** stored in AWS KMS / GCP KMS 4. Store the encrypted data + encrypted data key together ```typescript // Simplified envelope encryption with AWS KMS import { KMSClient, GenerateDataKeyCommand, DecryptCommand } from '@aws-sdk/client-kms'; const kms = new KMSClient({ region: 'us-east-1' }); const MASTER_KEY_ID = 'arn:aws:kms:us-east-1:123456789:key/abc-123'; async function encryptWithEnvelope(plaintext: string): Promise<{ encrypted: string; encryptedDataKey: string; iv: string; tag: string }> { // 1. KMS generates a data key (returns plaintext + encrypted versions) const { Plaintext: dataKey, CiphertextBlob: encryptedDataKey } = await kms.send( new GenerateDataKeyCommand({ KeyId: MASTER_KEY_ID, KeySpec: 'AES_256' }) ); // 2. Encrypt data with the data key const iv = crypto.randomBytes(16); const cipher = crypto.createCipheriv('aes-256-gcm', dataKey!, iv); let encrypted = cipher.update(plaintext, 'utf8', 'hex'); encrypted += cipher.final('hex'); const tag = cipher.getAuthTag().toString('hex'); return { encrypted, encryptedDataKey: Buffer.from(encryptedDataKey!).toString('base64'), iv: iv.toString('hex'), tag, }; } ``` **Why envelope encryption matters:** - The master key never leaves KMS. Even if your application server is compromised, the attacker can't decrypt data without calling KMS (which is audited and rate-limited). - You can rotate the master key without re-encrypting all data (re-encrypt only the data keys). - You can revoke access instantly by disabling the KMS key. --- ## Real-world example: The Uber 2016 data breach In 2016, Uber suffered a data breach exposing 57 million user and driver records. The attackers found Uber's AWS credentials **hardcoded in a private GitHub repository**. With those credentials, they accessed Uber's AWS S3 buckets and downloaded unencrypted personal data. **What went wrong:** - Secrets in source code (in a "private" repo — private repos get compromised too) - No encryption at rest on sensitive data - No secret rotation - No anomaly detection on AWS API usage **The fix (implemented after):** All secrets moved to HashiCorp Vault. All PII encrypted with envelope encryption (AWS KMS). Automated secret rotation every 90 days. --- ## Common mistakes | Mistake | Why it's wrong | What to do instead | |---------|---------------|-------------------| | Committing `.env` to git | Secret is permanently in git history. Bots scan for it within seconds. | `.gitignore` the `.env` file. Use `.env.example` with placeholders. | | Hardcoding secrets in source | Same as above — git history preserves it forever | Environment variables minimum. Secrets manager for production. | | Using the same secret everywhere | If dev database password leaks, production is compromised too | Different secrets per environment. Rotate them independently. | | Not rotating secrets | Secrets that never change = secrets that are eventually leaked through some vector | Rotate every 90 days. Automate it. | | Storing encryption keys next to encrypted data | Like locking a door and leaving the key under the mat | Envelope encryption: encrypt keys with KMS | | Plaintext secrets in logs/error messages | `console.log('Connected to', connectionString)` — includes password | Redact sensitive fields in logging. Never log raw connection strings. | --- ## Production notes - **Use `git-secrets` or `detect-secrets` pre-commit hooks.** They scan for patterns that look like API keys, tokens, and passwords. Blocks commits that contain them. - **Rotate secrets on a schedule.** Every 90 days is standard. Automate with your cloud provider's rotation feature (AWS Secrets Manager supports Lambda-based rotation). - **Separate secrets from config.** Config (port numbers, feature flags) can be in environment variables. Secrets (passwords, keys) belong in a secrets manager. ## Performance notes - **Cache secrets in memory.** Fetch from secrets manager on startup + every 5 minutes. Don't fetch on every request. - **KMS calls are rate-limited.** AWS KMS: varies by region, typically 5,000–50,000 requests per second. For envelope encryption, generate a data key once per record (or reuse per user), not per encryption operation. ## Security notes - **Principle of least privilege for secrets.** The application should have access to ONLY the secrets it needs. Database password? Yes. Stripe API key for the billing service? Yes. Stripe API key for the auth service? No. - **Audit secret access.** Cloud providers log every access. Alert on unusual patterns (secret accessed from a new IP, at unusual hours, by an unexpected service). - **`.env` is better than hardcoding but still not secure.** It's plaintext on disk. A compromised server = all secrets exposed. --- ## Exercises ### Beginner 1. Move all secrets from hardcoded strings to a `.env` file. Verify the app still works. Add `.env` to `.gitignore`. Create a `.env.example` for teammates. 2. Install `git-secrets` or `detect-secrets`. Scan your repository for accidentally committed secrets. Fix any findings. ### Intermediate 1. Implement AES-256-GCM encryption/decryption for a `ssn` (Social Security Number) field. Store encrypted in the database. Write a service function that encrypts on write and decrypts on read. 2. Implement a secret rotation script: create a new database user, update the secret in AWS Secrets Manager, restart the application. Verify zero-downtime. ### Advanced 1. Implement envelope encryption using AWS KMS: generate a data key per user, encrypt user PII with the data key, encrypt the data key with KMS. Store both. Write the decryption path that calls KMS to unwrap the data key. 2. Set up HashiCorp Vault (dev mode). Store and retrieve secrets via the Vault API. Implement dynamic database credentials: Vault creates temporary database users on-the-fly for your application. --- ## Interview questions ### Beginner 1. **Why shouldn't you commit `.env` files to git?** Git history is permanent. Even if you delete the file in a later commit, the secret exists in the history forever. Bots scan public repositories for secrets within seconds of a push. 2. **What's the difference between a `.env` file and a secrets manager?** `.env` is a plaintext file on disk — readable by anyone with file system access. A secrets manager encrypts secrets at rest, controls access via IAM, and provides audit logging and automatic rotation. 3. **What is encryption at rest?** Encrypting data before writing it to disk (database, file system) so that physical theft or backup leaks don't expose the plaintext data. ### Senior 4. **Explain envelope encryption and why it's better than encrypting directly with KMS.** Data is encrypted with a data key (AES-256). The data key is encrypted with a KMS master key and stored alongside the data. Benefits: the master key never leaves KMS (uncompromisable), KMS rate limits don't bottleneck your encryption operations, and rotating the master key only requires re-encrypting data keys, not all data. 5. **How do you rotate a database password without downtime?** Create a second database user with the new password. Store both credentials in the secrets manager. Deploy application code that tries both credentials. Monitor for successful connections using the new credentials. Once verified, remove the old credentials and drop the old user. 6. **What would you do if you discovered a production database password committed to a public GitHub repository?** (1) Rotate the password immediately — that's the #1 priority. (2) Audit access logs for unauthorized access using the compromised credentials. (3) Force-push to remove the commit from git history (or better: rotate and accept the history is immutable on public repos). (4) Enable branch protection rules and pre-commit secret scanning. (5) Post-incident review: how did this happen? Add safeguards. --- ## Summary Secrets management is foundation-level security. Never commit secrets to git. Use secrets managers (AWS/GCP/Vault) for production secrets, environment variables as the minimum bar. Rotate secrets on a schedule. Encrypt PII at rest with envelope encryption (KMS). Cache secrets in memory — don't fetch on every request. Separate secrets from config. The difference between a bad day (rotate a leaked secret) and a catastrophic day (data breach + regulatory fine) is how you manage secrets. --- ## Quick recall - `.env` → `.gitignore`. `.env.example` → commit. - Secrets manager (AWS/GCP/Vault) for production. Rotate every 90 days. - Envelope encryption: data key encrypts data, KMS encrypts data key. - Cache secrets in memory. Don't fetch on every request. - Pre-commit hooks: `git-secrets`, `detect-secrets`. - Separate secrets from config. Different secrets per environment. --- ## Quiz 1. What should you do immediately if a production secret is committed to a public repo? **Rotate it — change the credential/API key immediately. Then investigate.** 2. What is envelope encryption? **Data encrypted with a data key. Data key encrypted with a KMS master key. The master key never leaves KMS.** 3. How do you rotate a database password without downtime? **Create a second user, dual-credential application code, verify, remove old credentials.** 4. Why shouldn't you fetch secrets from a secrets manager on every request? **Latency (~50ms per KMS call) and rate limits. Cache in memory, refresh every 5 minutes.** 5. What's the difference between a `.env` file and a `.env.example` file? **`.env` has real secrets (NEVER commit). `.env.example` has placeholder values (DO commit — documentation for teammates).** 6. Why use different secrets for dev/staging/production? **If dev credentials leak, production is still safe. Smaller blast radius.** 7. Name a tool that prevents committing secrets to git. **git-secrets, detect-secrets, Gitleaks, truffleHog, GitHub's built-in secret scanning.**

Chapter 9 — Secrets Management & Encryption

Part of L3