Warming up the neural circuits...
By the end of this chapter you will:
If you do everything synchronously, the user waits 8 seconds for a KYC submission that only needs to save 3 fields. If the mail server is down, the entire request fails even though the data was saved correctly.
Imagine a user submits a KYC document. To process it, your app needs to:
If you do all of this synchronously, the user waits up to 8 seconds staring at a loading spinner. If the mail server is down, the entire request fails even though the KYC data was saved correctly.
The solution: Do the slow, unreliable parts asynchronously.
User submits KYC document
→ Save document to DB (10ms)
→ Enqueue "send confirmation email" job ← doesn't wait
→ Enqueue "start vendor KYC" job ← doesn't wait
→ Return 201 Created (immediately)Good candidates for queues:
Bad candidates for queues (do synchronously):
Rule of thumb: "Is the user waiting for this in the HTTP response?" If yes, sync. If no, .
@Injectable()
export class OrdersService {
constructor(
@InjectQueue('email') private emailQueue: Queue,
) {}
async create(user: AuthUser, dto: CreateOrderDto): Promise<
@Processor('email')
export class EmailProcessor {
constructor(private mailerService: MailerService) {}
@Process('order-confirmation')
async handleOrderConfirmation(job: Job) {
const { order_id,
Retries WILL happen. Your job must produce the same result whether it runs once or three times.
// ❌ Not idempotent — creates duplicate notification on every retry
async handleOrderConfirmation(job: Job) {
await Notification.create({
order_id: job.data.order_id,
message: 'Your order is confirmed',
});
}
When a downstream service (e.g. email server) is down, don't retry immediately 5 times in a row. Use exponential backoff:
{
attempts: 5,
backoff: {
type: 'exponential',
delay: 1000, // 1st retry after 1s, 2nd after 2s, 3rd after 4s, 4th after 8s...
}
}This gives the downstream time to recover while still retrying.
If a job fails all 5 attempts, it goes to a "failed" . Set up monitoring to alert on failed jobs:
@OnQueueFailed()
async onFailed(job: Job, error: Error) {
this.logger.error('job.failed', {
queue: job.queue.name,
job_name:
A failed job that nobody monitors is worse than no retries at all — it creates the false impression that something succeeded.
Some work happens on a schedule, not triggered by a user request:
@Injectable()
export class FxRateSyncScheduler {
@Cron('0 * * * *') // every hour at :00
async syncFxRates() {
this.logger.log('fx_rate_sync.started');
const rates = await this.fxVendorService
With multiple pods, every pod runs the cron job simultaneously. This can cause duplicate work. Use a distributed lock (Redis SET NX) to ensure only one pod runs a given job at a time.
| Pattern | Use when |
|---|---|
| Synchronous (in the request) | User is waiting for the result |
| Queue job | Work is slow, unreliable, or doesn't need to block the response |
| Scheduled cron | Work happens on a time interval |
| Exponential backoff | Any job that calls an external service |
| Idempotency check at job start | Any job that can be retried |
| Dead-letter queue | Any job that might fail permanently |
Every job will be retried. Write every job handler to be idempotent — running it twice must produce the same result as running it once. Check if the work was already done before doing it.