Chapter 9 — Secrets Management & Encryption

The why

The 2024 GitGuardian report found 12.8 million secrets leaked on GitHub — a 28% increase year over year. Database passwords, API keys, private keys, tokens — all committed to repositories, often by accident, sometimes by developers who didn't know better. Once a secret hits a public repository, it's compromised. You cannot "delete" it — bots clone repositories within seconds of a push. This chapter teaches you to manage secrets like a professional.

Core concepts

The secrets hierarchy — from worst to best

Environment variables — the minimum bar

# .env — NEVER commit this file. Add to .gitignore.
DATABASE_URL="postgresql://user:password@localhost:5432/mydb"
JWT_SECRET="super-secret-key-change-in-production"
STRIPE_API_KEY="sk_live_..."
 
# .env.example — DO commit this (without real values)
DATABASE_URL="postgresql://user:password@localhost:5432/mydb"
JWT_SECRET="generate-with: openssl rand -hex 64"
STRIPE_API_KEY

Using a secrets manager (Vercel / AWS Secrets Manager / HashiCorp Vault)

// AWS Secrets Manager example
import { SecretsManagerClient, GetSecretValueCommand } from '@aws-sdk/client-secrets-manager';
 
const client = new SecretsManagerClient({ region: 'us-east-1' });
 
async function getDatabasePassword(): Promise<string

Rotating secrets without downtime

Rotating a database password while the application is running:

1. Create a NEW database user/password (user_v2 / password_v2)
2. Add the new credentials to the secrets manager (as a second version or new key)
3. Deploy application code that reads BOTH credentials, tries new first, falls back to old
4. Monitor — all connections using new credentials → no errors
5. Remove old credentials from secrets manager
6. Drop the old database user

// Dual-credential rotation
async function createPool() {
  const [newCreds, oldCreds] = await Promise.all([
    getSecret('database/v2').catch(() => null),
    getSecret('database/v1')

Encryption at rest — protecting PII in the database

If your database is compromised (backup leak, SQL injection), encrypted columns protect the data:

import crypto from 'crypto';
 
const ALGORITHM = 'aes-256-gcm';
const KEY = Buffer.from(process.env.ENCRYPTION_KEY!, 'hex'); // 32 bytes = 256 bits
 

Envelope encryption (KMS)

Encrypting data with AES-256 is good. But where do you store the AES key? If it's in an environment variable, and that leaks, all encrypted data is decryptable. Envelope encryption solves this:

1. Generate a data key (AES-256) for each record (or each user)
2. Encrypt the data with the data key
3. Encrypt the data key itself with a master key stored in AWS KMS / GCP KMS
4. Store the encrypted data + encrypted data key together

// Simplified envelope encryption with AWS KMS
import { KMSClient, GenerateDataKeyCommand, DecryptCommand } from '@aws-sdk/client-kms';
 
const kms = new KMSClient({ region: 'us-east-1' });
const MASTER_KEY_ID = 'arn:aws:kms:us-east-1:123456789:key/abc-123

Why envelope encryption matters:

• The master key never leaves KMS. Even if your application server is compromised, the attacker can't decrypt data without calling KMS (which is audited and rate-limited).
• You can rotate the master key without re-encrypting all data (re-encrypt only the data keys).
• You can revoke access instantly by disabling the KMS key.

Real-world example: The Uber 2016 data breach

In 2016, Uber suffered a data breach exposing 57 million user and driver records. The attackers found Uber's AWS credentials hardcoded in a private GitHub repository. With those credentials, they accessed Uber's AWS S3 buckets and downloaded unencrypted personal data.

What went wrong:

• Secrets in source code (in a "private" repo — private repos get compromised too)
• No encryption at rest on sensitive data
• No secret rotation
• No anomaly detection on AWS API usage

The fix (implemented after): All secrets moved to HashiCorp Vault. All PII encrypted with envelope encryption (AWS KMS). Automated secret rotation every 90 days.

Common mistakes

Production notes

• Use git-secrets or detect-secrets pre-commit hooks. They scan for patterns that look like API keys, tokens, and passwords. Blocks commits that contain them.
• Rotate secrets on a schedule. Every 90 days is standard. Automate with your cloud provider's rotation feature (AWS Secrets Manager supports Lambda-based rotation).
• Separate secrets from config. Config (port numbers, feature flags) can be in environment variables. Secrets (passwords, keys) belong in a secrets manager.

Performance notes

• Cache secrets in memory. Fetch from secrets manager on startup + every 5 minutes. Don't fetch on every request.
• KMS calls are rate-limited. AWS KMS: varies by region, typically 5,000–50,000 requests per second. For envelope encryption, generate a data key once per record (or reuse per user), not per encryption operation.

Security notes

• Principle of least privilege for secrets. The application should have access to ONLY the secrets it needs. Database password? Yes. Stripe API key for the billing service? Yes. Stripe API key for the auth service? No.
• Audit secret access. Cloud providers log every access. Alert on unusual patterns (secret accessed from a new IP, at unusual hours, by an unexpected service).
• .env is better than hardcoding but still not secure. It's plaintext on disk. A compromised server = all secrets exposed.

Exercises

Beginner

1. Move all secrets from hardcoded strings to a .env file. Verify the app still works. Add .env to .gitignore. Create a .env.example for teammates.
2. Install git-secrets or detect-secrets. Scan your repository for accidentally committed secrets. Fix any findings.

Intermediate

1. Implement AES-256-GCM encryption/decryption for a ssn (Social Security Number) field. Store encrypted in the database. Write a service function that encrypts on write and decrypts on read.
2. Implement a secret rotation script: create a new database user, update the secret in AWS Secrets Manager, restart the application. Verify zero-downtime.

Advanced

1. Implement envelope encryption using AWS KMS: generate a data key per user, encrypt user PII with the data key, encrypt the data key with KMS. Store both. Write the decryption path that calls KMS to unwrap the data key.
2. Set up HashiCorp Vault (dev mode). Store and retrieve secrets via the Vault API. Implement dynamic database credentials: Vault creates temporary database users on-the-fly for your application.

Interview questions

Beginner

1. Why shouldn't you commit .env files to git? Git history is permanent. Even if you delete the file in a later commit, the secret exists in the history forever. Bots scan public repositories for secrets within seconds of a push.

2. What's the difference between a .env file and a secrets manager? .env is a plaintext file on disk — readable by anyone with file system access. A secrets manager encrypts secrets at rest, controls access via IAM, and provides audit logging and automatic rotation.

3. What is encryption at rest? Encrypting data before writing it to disk (database, file system) so that physical theft or backup leaks don't expose the plaintext data.

Senior

4. Explain envelope encryption and why it's better than encrypting directly with KMS. Data is encrypted with a data key (AES-256). The data key is encrypted with a KMS master key and stored alongside the data. Benefits: the master key never leaves KMS (uncompromisable), KMS rate limits don't bottleneck your encryption operations, and rotating the master key only requires re-encrypting data keys, not all data.

5. How do you rotate a database password without downtime? Create a second database user with the new password. Store both credentials in the secrets manager. Deploy application code that tries both credentials. Monitor for successful connections using the new credentials. Once verified, remove the old credentials and drop the old user.

6. What would you do if you discovered a production database password committed to a public GitHub repository? (1) Rotate the password immediately — that's the #1 priority. (2) Audit access logs for unauthorized access using the compromised credentials. (3) Force-push to remove the commit from git history (or better: rotate and accept the history is immutable on public repos). (4) Enable branch protection rules and pre-commit secret scanning. (5) Post-incident review: how did this happen? Add safeguards.

Summary

Secrets management is foundation-level security. Never commit secrets to git. Use secrets managers (AWS/GCP/Vault) for production secrets, environment variables as the minimum bar. Rotate secrets on a schedule. Encrypt PII at rest with envelope encryption (KMS). Cache secrets in memory — don't fetch on every request. Separate secrets from config. The difference between a bad day (rotate a leaked secret) and a catastrophic day (data breach + regulatory fine) is how you manage secrets.

Quick recall

• .env → .gitignore. .env.example → commit.
• Secrets manager (AWS/GCP/Vault) for production. Rotate every 90 days.
• Envelope encryption: data key encrypts data, KMS encrypts data key.
• Cache secrets in memory. Don't fetch on every request.
• Pre-commit hooks: git-secrets, detect-secrets.
• Separate secrets from config. Different secrets per environment.

Quiz

1. What should you do immediately if a production secret is committed to a public repo? Rotate it — change the credential/API key immediately. Then investigate.
2. What is envelope encryption? Data encrypted with a data key. Data key encrypted with a KMS master key. The master key never leaves KMS.
3. How do you rotate a database password without downtime? Create a second user, dual-credential application code, verify, remove old credentials.
4. Why shouldn't you fetch secrets from a secrets manager on every request? Latency (~50ms per KMS call) and rate limits. Cache in memory, refresh every 5 minutes.
5. What's the difference between a .env file and a .env.example file? .env has real secrets (NEVER commit). .env.example has placeholder values (DO commit — documentation for teammates).
6. Why use different secrets for dev/staging/production? If dev credentials leak, production is still safe. Smaller blast radius.
7. Name a tool that prevents committing secrets to git. git-secrets, detect-secrets, Gitleaks, truffleHog, GitHub's built-in secret scanning.