Chapter 3 — Background Jobs & Queues

The why

An HTTP request is a dinner reservation — you have 30 seconds before the table is given away. Background jobs are the kitchen prep that happens hours before the doors open.

Core concepts

Why Not Just setTimeout or Promise.all?

Before job queues, developers reach for in-process solutions:

// Anti-pattern 1: Inline async (blocks response)
app.post('/purchase', async (req, res) => {
  const order = await createOrder(req.body);
  await generateInvoicePDF(order);     //

Fire-and-forget has three fatal flaws: (1) if the process crashes, work is lost forever, (2) if the email API is down, there's no retry, (3) there's no visibility — you don't know if the PDF was generated or why it failed.

BullMQ Architecture

BullMQ uses Redis as its backbone. The architecture has four components:

Producer (API server)        Redis                     Worker (separate process)
┌──────────┐          ┌─────────────────┐          ┌──────────┐
│ Add job  │──RPUSH──►│ waiting (list)  │──LPOP──►│ Process  │
│ to queue │          │ active (list)   │          │ job      │
└──────────┘          │ completed (set) │          └──────────┘
                      │ failed (set)    │
                      │ delayed (zset)  │
                      └─────────────────┘

Producer: Your API server adds jobs to the queue. It resolves immediately after Redis confirms receipt — single-digit milliseconds.

Redis: Stores jobs in lists, sets, and sorted sets. Jobs in waiting are FIFO. Jobs in delayed use a sorted set scored by execution time. Jobs in completed/failed are sets with optional TTL.

Worker: A separate process that pulls jobs from Redis, processes them, and reports results back. Workers can run on different machines — they compete for jobs via Redis atomic operations.

Setting Up BullMQ

import { Queue, Worker, Job } from 'bullmq';
import { createClient } from 'redis';
 
const connection = createClient({ url: process.env.REDIS_URL

import { purchaseQueue } from '../queues/purchase-queue';
 
app.post('/api/purchase', async (req, res) => {
  const order = await createOrder(req.body);

The Job Lifecycle

Every job in BullMQ transitions through states:

┌─────────┐     ┌─────────┐     ┌──────────┐     ┌───────────┐
│ waiting │────►│ active  │────►│completed │  or │  failed   │
└─────────┘     └─────────┘     └──────────┘     └─────┬─────┘
     ▲                                                  │
     │              ┌─────────┐                         │
     └──────────────│ delayed │◄────── retry ────────────┘
                    └─────────┘      (if attempts remain)

• waiting: Job is in the queue, waiting for a worker to pick it up
• active: A worker has claimed the job and is processing it
• completed: The job function returned successfully
• failed: The job threw an error or exceeded timeout
• delayed: Job is scheduled for future execution (retry backoff, or delay option)

import { Worker, Job } from 'bullmq';
import { purchaseQueue } from '../queues/purchase-queue';
import { generateInvoicePDF } from '../services/invoice';
import { sendConfirmationEmail } from '../services/email'

Retry Strategies

Not all failures are equal. A network timeout to the email API should retry. A "user not found" error should NOT retry — it will never succeed.

import { Worker, UnrecoverableError } from 'bullmq';
 
const worker = new Worker('purchase', async (job: Job) => {
  try {
    return await processPurchaseJob(job

Idempotency: The Most Important Concept in Job Queues

Idempotency means processing the same job twice produces the same result as processing it once. Without it, you WILL double-charge customers. Here's why it matters:

A worker pulls a job, processes it, but the Redis connection drops before it can report "completed." Redis still thinks the job is active. After a timeout, another worker picks up the "same" job and processes it again. Now the customer gets two confirmation emails, two invoice PDFs, and potentially two charges.

Solution: Job IDs as idempotency keys.

// The jobId IS the idempotency key
await purchaseQueue.add('send-purchase-emails', data, {
  jobId: `purchase-${order.id}`, // Same order = same jobId
});
 
// If this code runs twice (retry, duplicate webhook),
//

Inside the worker — check before acting:

async function processPurchaseJob(job: Job) {
  const { orderId } = job.data;
 
  // Idempotency check: was this already done?
  const existingInvoice = await db.query(
    'SELECT id FROM invoices WHERE order_id = $1',
    [orderId

Dead Letter Queues

After 5 retries, a job lands in failed. But some failures need human intervention: a payment that partially succeeded, an email to a malformed address, a PDF for a deleted order. You don't want these mixed in with transient network failures.

import { Queue, Worker, QueueEvents } from 'bullmq';
 
const mainQueue = new Queue('purchase', { connection });
const deadLetterQueue = new Queue('

A separate dashboard or admin endpoint lets support staff inspect the DLQ and manually retry or discard jobs.

Cron / Scheduled Jobs

BullMQ supports repeatable jobs — jobs that run on a schedule, powered by Redis sorted sets under the hood:

// Daily cleanup: delete expired tokens
await cleanupQueue.add(
  'cleanup-expired-tokens',
  {},
  {
    repeat: {
      pattern: '0 3 * * *', // Every day at 3 AM
    },
    jobId: 'cleanup-expired-tokens', // Unique — prevents duplicate schedules
  }

Choosing a Queue Backend

Real-world example: How Stripe processes payments asynchronously

Stripe's payment processing pipeline is a masterclass in job queue design. When you call POST /v1/charges, Stripe's API returns in under 200ms with a charge object that has status: "pending". The actual work happens in a series of asynchronous jobs:

The pipeline:

1. API server validates the request and creates a charge record (status: pending).
2. A job is enqueued to the payment-processing queue. The API returns immediately.
3. Worker 1 picks up the job, calls the card network (Visa, Mastercard), gets an authorization.
4. Worker 2 — triggered on authorization success — performs fraud checks (Stripe Radar, ML models).
5. Worker 3 captures the funds, updates the charge status to succeeded, and enqueues a webhook delivery job.
6. Worker 4 delivers the webhook to the merchant's endpoint with exponential backoff (up to 3 days).

Why Stripe's approach matters for your architecture:

• The API is never the bottleneck. The /v1/charges endpoint doesn't wait for fraud analysis (200ms ML inference) or card network round-trips (500ms-2s). It enqueues and returns.
• Stripe uses idempotency keys religiously. Every API request can include an Idempotency-Key header. Stripe stores the response of the first request and replays it for duplicates — safe to retry network failures infinitely.
• Webhook delivery is itself a job queue. Failed webhooks retry with exponential backoff for 3 days. After that, they go into the dashboard as "failed" — Stripe's equivalent of a dead letter queue. Merchants can manually retry from the Dashboard.
• Job observability. Stripe's internal tooling lets engineers see every job's state, retry count, and latency at each step. This visibility is why they can debug payment failures in minutes, not hours.

Common mistakes

Production notes

• Redis persistence is critical. Without appendonly yes in Redis config, a Redis restart wipes all jobs. Use AOF persistence. For the truly paranoid, snapshot (RDB) + AOF together.
• Bull Board for visibility. The bull-board package gives you a web UI to see queue lengths, job statuses, retry counts, and job data. Install it at /admin/queues behind authentication.
• Graceful shutdown of workers. On SIGTERM: stop accepting new jobs (worker.pause()), wait for active jobs to complete (worker.close() with timeout), then exit. Incomplete jobs return to waiting for another worker to pick up.
• Separate Redis for queues vs cache. Your queue Redis should NOT be the same instance as your cache Redis. If cache eviction (allkeys-lru) kicks in, it can evict active jobs. Queue Redis needs noeviction policy.
• Alert on queue depth. If the waiting count exceeds a threshold for > 5 minutes, something is wrong — workers are down or can't keep up. Set PagerDuty alerts on queue metrics.

Performance notes

• Concurrency tuning. Start with concurrency: 10 per worker process. Increase until Redis CPU hits ~70% or worker memory hits your limit. More concurrency ≠ more throughput if your bottleneck is an external API with rate limits.
• Sandboxed processors for CPU-bound work. If your job does heavy computation (image processing, PDF generation), use SandboxedProcessor — it runs the job in a child process, preventing it from blocking the event loop of the main worker.
• Batch job enqueuing. Adding 1,000 jobs one at a time = 1,000 Redis round trips. Use queue.addBulk() to add many jobs in a single pipeline call: await queue.addBulk(jobs.map(data => ({ name: 'task', data }))).
• Rate limit awareness. If your worker calls an API that allows 100 req/s, 50 concurrent workers will hammer it. Use rate-limit per worker or a token bucket in Redis to throttle.
• Job staleness. If a job sits in waiting for > 10 minutes, something is wrong (zero workers, Redis disconnected). Monitor job.timestamp vs current time and alert on stale jobs.

Security notes

• Never put secrets in job data. Job data is stored in Redis as JSON, visible in Bull Board, and logged in error traces. Pass references (IDs, URLs, keys), not API keys or passwords.
• Job data validation. Workers should validate job data with the same rigor as HTTP controllers. A malformed job from a buggy producer shouldn't crash all workers.
• Queue access control. Redis for BullMQ should have a strong password and not be exposed to the public internet. Use a VPC or TLS tunnel. Bull Board must be behind authentication.
• PII in job data. Job data may contain email addresses, user IDs, and order amounts. Set removeOnComplete with a reasonable TTL. Don't retain PII in job data longer than necessary for debugging.
• Worker isolation. If one job type handles sensitive data (payment processing) and another handles public data (thumbnail generation), run them in separate worker processes with different environment configurations.

Exercises

Beginner

1. Move email sending to a queue. Take an Express endpoint that sends a welcome email inline. Create a BullMQ queue and worker. The endpoint adds a job; the worker sends the email. Verify that the API responds in < 50ms while the email sends asynchronously.

2. Add retry logic. Configure the queue with attempts: 3 and exponential backoff. Simulate a failing email API (throw an error 2 times, succeed on the 3rd). Confirm the job retries and eventually completes.

Intermediate

1. Build a dead letter queue. After 3 failed attempts, move the job to a separate purchase-dead-letter queue. Write an admin endpoint that lists DLQ jobs and allows retrying them back to the main queue.

2. Implement idempotency. Create an order processing job that uses orderId as the jobId. Write a test that adds the same job twice — the second add() should return the existing job, not create a duplicate. Inside the worker, add a database check that skips processing if the invoice already exists.

Advanced

1. Rate-limited external API worker. Your worker calls a third-party API limited to 50 requests per second. Implement a token bucket rate limiter in Redis that spans all worker instances. Workers acquire a token before making the API call and release it after. Without it, simultaneous workers exceed the rate limit.

2. Build a cron-based report generator. Create a repeatable job that runs every Monday at 9 AM. It queries the database for last week's orders, generates a CSV report, uploads it to S3, and emails a link to the admin. Handle edge cases: what if the report generation fails? What if it takes longer than a week (overlapping runs)? Use a lock (Redis SET NX) to prevent concurrent executions.

Interview questions

Beginner

Q1: Why move work to a background queue instead of doing it in the HTTP request handler?

Answer: Three reasons: (1) User experience — the API responds in milliseconds instead of seconds. (2) Reliability — if the work fails, the queue retries with backoff; a failed inline handler just returns a 500. (3) Resource isolation — slow work in the API process blocks the event loop for all users. Background workers are separate processes that can be scaled independently.

Q2: What's a job ID and why does it matter?

Answer: A job ID is a unique identifier assigned when a job is added to the queue. In BullMQ, adding a job with the same jobId returns the existing job instead of creating a duplicate. This is the foundation of idempotency — it prevents duplicate work from duplicate webhook calls, retry storms, or buggy producers.

Q3: What happens to a job that fails all its retry attempts?

Answer: In BullMQ, it stays in the failed set. It's not automatically removed or retried. Without a dead letter queue, failed jobs accumulate in the main queue's failed set, mixed with transient failures. The best practice is to listen for failed events and move exhausted jobs to a dedicated dead letter queue for inspection and manual retry.

Senior

Q4: How do you handle a job that takes 10 minutes but your Redis connection drops after 5?

Answer: BullMQ uses Redis locks with a TTL ("stalled check"). When a worker claims a job, it sets a lock with a TTL (default 30s). The worker periodically renews the lock while the job is running. If the lock expires (worker crashed, Redis disconnected), another worker detects the "stalled" job and picks it up. This is why idempotency is critical — the job might be processed twice. The stalled check interval and lock duration are configurable via stalledInterval and lockDuration.

Q5: You're migrating from in-process async to BullMQ. How do you ensure zero data loss during the migration?

Answer: I'd implement a dual-write pattern: (1) Add the queue alongside the existing inline processing (both run). (2) The worker checks a database flag: if the work was already done in-process, skip. (3) Monitor for a week — confirm queue-based results match inline results. (4) Flip a feature flag to route 100% of traffic through the queue. (5) Remove the inline code. The key is that the worker is idempotent from day one — it always checks if the work was already completed.

Q6: Compare BullMQ, RabbitMQ, and SQS for a payment processing pipeline. Which would you choose and why?

Answer: For payment processing where correctness > everything: I'd lean toward SQS + SNS because of zero-ops reliability (AWS manages it, 99.9%+ uptime) and dead letter queue support built in. But SQS doesn't support job progress or delayed jobs natively (you use visibility timeout hacks). For a team already running Redis, BullMQ wins on developer experience — job progress, Bull Board UI, repeatable jobs, and it's all in Node.js (same language as the API). RabbitMQ wins for polyglot teams (Java, Go, Python workers) and complex routing patterns (topic exchanges, header-based routing). My default for a Node.js shop: BullMQ for simplicity, SQS if "we never want to think about Redis again."