Chapter 5 — File Uploads & Object Storage

The why

Your Node.js server is a bicycle courier. Don't ask it to deliver a grand piano. Give the sender the warehouse address and let them drive the truck themselves.

Core concepts

The Multipart Problem: Parsing File Uploads

When a browser uploads a file via <input type="file"> or FormData, it sends a multipart/form-data request. Here's what the raw HTTP body looks like:

POST /upload HTTP/1.1
Content-Type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW
 
------WebKitFormBoundary7MA4YWxkTrZu0gW
Content-Disposition: form-data; name="caption"
 
My vacation photo
------WebKitFormBoundary7MA4YWxkTrZu0gW
Content-Disposition: form-data; name="photo"; filename="beach.jpg"
Content-Type: image/jpeg
 
(binary JPEG data here — could be megabytes)
------WebKitFormBoundary7MA4YWxkTrZu0gW--

Node.js's built-in HTTP parser does NOT parse multipart bodies. You need a middleware like multer:

import express from 'express';
import multer from 'multer';
import path from 'path';
 
const app = express();
 
// Store files on disk (not memory!) for anything > 1MB
const upload = multer(

Pre-Signed URLs: Bypass Your Server Entirely

Pre-signed URLs are the single most important concept in file uploads. They give a client temporary, scoped access to S3 without sharing your AWS credentials.

Upload flow with pre-signed URLs:

1. Client → Server:  POST /api/upload-url  { filename: 'beach.jpg', contentType: 'image/jpeg' }
2. Server → S3:      Generate pre-signed PUT URL (valid 5 min)
3. Server → Client:  { uploadUrl: 'https://s3.../beach.jpg?X-Amz-...', fileKey: 'uploads/abc123/beach.jpg' }
4. Client → S3:      PUT uploadUrl with file bytes (server never sees the file)
5. Client → Server:  POST /api/upload-complete { fileKey: 'uploads/abc123/beach.jpg' }
6. Server:            Validate the file exists in S3, create database record

import { S3Client, PutObjectCommand } from '@aws-sdk/client-s3';
import { getSignedUrl } from '@aws-sdk/s3-request-presigner';
import crypto from 'crypto';
import express from 'express';
 
const s3

app.post('/api/upload-complete', async (req, res) => {
  const { fileKey } = req.body;
 
  // Verify the file actually exists in S3
  try {
    await s3.send

Streaming Uploads Through Your Server (When You Must)

Sometimes you can't use pre-signed URLs: the file needs server-side transformation before storage, you're behind a corporate firewall, or you're not using S3. In these cases, stream — never buffer:

import { PassThrough } from 'stream';
import { Upload } from '@aws-sdk/lib-storage';
import multer from 'multer';
import express from 'express';
 
// Use multer with diskStorage, then stream from disk to S3
app

Resumable Uploads with the Tus Protocol

For large files (videos, datasets, backups) on unreliable connections, you need resumable uploads. The tus protocol (tus.io) is the open standard. It works by splitting the file into chunks and uploading them sequentially, with the ability to resume from where it left off.

import { Server } from '@tus/server';
import { S3Store } from '@tus/s3-store';
import express from 'express';
 
const app = express();
 
const tusServer = new Server

The tus protocol flow:

1. Client sends POST /uploads/files with Upload-Length: 524288000 (500MB) and Upload-Metadata: filename bXl2aWRlby5tcDQ= (base64 encoded).
2. Server responds with Location: /uploads/files/abc123 — a unique upload URL.
3. Client sends PATCH /uploads/files/abc123 with Upload-Offset: 0 and the first chunk.
4. Server stores the chunk, responds with Upload-Offset: 5242880 (5MB received).
5. Connection drops at 200MB. Client sends HEAD /uploads/files/abc123 to check offset.
6. Server responds: Upload-Offset: 209715200 (200MB received).
7. Client resumes: PATCH /uploads/files/abc123 with Upload-Offset: 209715200 and remaining data.

File Validation: Beyond the File Extension

Never trust file.mimetype (it comes from the client) or the file extension. Validate at multiple levels:

import { fileTypeFromFile } from 'file-type';
import sharp from 'sharp';
import path from 'path';
 
interface ValidationResult {
  valid: boolean;
  reason?: string;

Image Processing Pipeline

Once an image is uploaded, you typically need multiple sizes: thumbnail (150px), medium (600px), full (1920px). Sharp is the standard for this in Node.js:

import sharp from 'sharp';
import { S3Client, GetObjectCommand, PutObjectCommand } from '@aws-sdk/client-s3';
import { getSignedUrl } from '@aws-sdk/s3-request-presigner';
 
const IMAGE_SIZES = [
  { name

Virus Scanning

For user-generated content, virus scanning is non-negotiable. ClamAV is the open-source standard:

import { execFile } from 'child_process';
import { promisify } from 'util';
 
const execFileAsync = promisify(execFile);
 
export async function scanFile(filePath: string):

CDN Fronting

Never serve user-uploaded files directly from S3. Put a CDN (CloudFront, Cloudflare, Fastly) in front:

// When generating URLs for clients, use CDN domain, not S3 domain
const cdnUrl = `https://cdn.yourapp.com/${s3Key}`;
 
// S3 objects get Cache-Control metadata
await s3.send(new PutObjectCommand({
  Bucket: BUCKET,
  Key: key,
  Body:

Benefits of CDN fronting:

• Lower latency: files served from edge locations near users
• Reduced S3 costs: S3 egress is expensive ($0.09/GB); CDN egress is cheaper ($0.02-0.08/GB depending on provider)
• DDoS protection: CDNs absorb volumetric attacks
• HTTPS with custom domain: your files are at cdn.yourapp.com, not s3.amazonaws.com

Real-world example: How YouTube and Instagram handle uploads

YouTube's upload pipeline processes 500+ hours of video uploaded every minute. Their architecture reveals patterns you can downscale:

1. Chunked upload with resumability. YouTube's upload client splits videos into chunks (typically 1-8MB each). If your connection drops at 40%, the client resumes from chunk N, not byte 0. This is conceptually the tus protocol, though YouTube uses a proprietary implementation.

2. Processing happens asynchronously. When the upload completes, the video enters a processing pipeline: transcoding to multiple resolutions (144p through 4K), thumbnail generation, content ID matching (copyright detection), and ML-based content moderation. None of this blocks the upload confirmation — the creator sees "Processing... 0%" immediately after upload.

3. Multiple storage tiers. Hot videos (trending, recent uploads from popular creators) are stored on SSD-backed storage for fast access. Warm videos sit on HDD. Cold videos (years old, low views) go to tape archive. This tiering saves millions in storage costs.

Instagram's image pipeline processes 100M+ photos per day:

1. Progressive loading. Instagram serves a 200-byte blurhash placeholder immediately, followed by a low-res version (50KB), then the full-res version. The user sees something instantly even on 3G.

2. Client-side preprocessing. Before upload, the Instagram app resizes and compresses the image on-device. A 12MP phone photo (8MB) becomes a 1080px JPEG (200KB) before it ever touches the network. This reduces upload time and server processing load.

3. CDN multi-tier caching. Profile photos (frequently accessed, small) are cached aggressively at edge. Feed photos (timely, large) have shorter cache TTLs. Stories (ephemeral, 24h life) use the shortest TTLs and may not be cached at all if they're viewed by few people.

Key lesson from both: the upload is just the first step. The real engineering is in the async processing pipeline (transcoding, resizing, moderation) and the delivery architecture (CDN, progressive loading, tiered storage).

Common mistakes

Production notes

• S3 bucket policy: block public access. All buckets should have "Block all public access" enabled. Access files via pre-signed URLs (short-lived, scoped) or CDN signed URLs. Never make an S3 bucket publicly readable.
• S3 lifecycle policies. Configure auto-deletion or archival: temporary uploads (/tmp/) expire after 24 hours; processed uploads transition to Standard-IA after 30 days, Glacier after 90 days. This saves 40-70% on storage costs.
• Multipart upload for large files. S3 supports multipart uploads: split a file into parts, upload in parallel, S3 reassembles. The @aws-sdk/lib-storage Upload class handles this automatically. Essential for files > 100MB.
• CORS on S3 buckets. If clients upload directly to S3 via pre-signed URLs, configure CORS on the bucket to allow your domain. Otherwise browser requests are blocked by same-origin policy.
• Webhook or S3 event notifications. Instead of the client calling /api/upload-complete (which can be spoofed), configure S3 to send an SNS/SQS/Lambda event on s3:ObjectCreated. Your backend processes the file when S3 confirms the object exists.

Performance notes

• Presigned URL generation is sub-millisecond. No cryptographic work — the SDK signs a request with your credentials locally. You can generate 1,000+ per second on a single Node.js process.
• S3 Transfer Acceleration. For users far from your S3 bucket region, enable Transfer Acceleration. It uses CloudFront edge locations to accept uploads and routes over AWS backbone to S3. 50-500% speedup for cross-continent uploads.
• Parallel multipart upload parts. For a 1GB file, split into 100 × 10MB parts and upload 10 parts simultaneously. The @aws-sdk/lib-storage Upload class with queueSize: 10 does this. Tune partSize (5MB-5GB, default 5MB) and queueSize (1-10) per workload.
• Sharp performance tuning. Sharp uses libvips (C library) and is extremely fast — 4-5x faster than ImageMagick and 10x faster than Jimp (pure JS). Process images in parallel with Promise.all across sizes.
• Lazy resize vs eager resize. Don't generate all image sizes on upload. Generate only the most common sizes eagerly (medium). Generate rare sizes (specific crop dimensions for a redesign) lazily on first access, then cache.

Security notes

• Validate pre-signed URL callbacks. An attacker could call /api/upload-complete with any fileKey they guess, claiming to have uploaded it. Always verify the object exists in S3 (HeadObject) and that the userId in the S3 metadata matches the requesting user.
• Content-Disposition for downloads. When serving user files, set Content-Disposition: attachment; filename="..." for downloadable files. Without it, evil.html uploaded by a user renders in the browser — potentially accessing cookies, localStorage, or performing phishing if served from your domain. Serve user content from a separate domain (e.g., usercontent.yourapp.com) to isolate it from your main app's cookies and JavaScript context.
• File type restrictions must be at the infrastructure level. In addition to application-level validation, use an S3 bucket policy that denies PutObject for disallowed content types. Defense in depth: even if your app validation is bypassed, S3 rejects the upload.
• Rate limiting upload endpoints. /api/upload-url is cheap but can be abused. Rate limit by user ID and IP — a free user shouldn't be able to request 10,000 upload URLs per minute (storage costs add up even if files are tiny).
• Scan uploads for malware. ClamAV or a cloud API (VirusTotal, AWS Macie). For high-volume platforms, scanning can be async (enqueue after upload confirmation). If a virus is found, delete the S3 object, ban the user, and log for security review.

Exercises

Beginner

1. Set up multer with disk storage. Create an Express endpoint that accepts a single image upload via multer. Store to /tmp/uploads. Return the filename, size, and mimetype. Test with curl -F "photo=@beach.jpg".

2. Generate a pre-signed S3 upload URL. Using @aws-sdk/s3-request-presigner, write an endpoint that accepts filename and contentType and returns a pre-signed PUT URL valid for 5 minutes. Test by using curl to PUT a file directly to the returned URL.

Intermediate

1. Build an image processing pipeline. After upload, use sharp to generate thumbnail (150×150), medium (600×600), and large (1920×1080) variants. Upload all variants to S3. Store the URLs in a database. Return all URLs to the client.

2. Implement file validation with magic bytes. Write a validation function that checks: file extension (allowlist), MIME type from magic bytes (file-type package), and image validity (can sharp open it?). Reject files that fail any check. Log rejection reasons for security monitoring.

Advanced

1. Build a resumable upload system with tus. Set up a @tus/server with S3 store. Implement a client (browser or Node.js) using tus-js-client that uploads a 500MB file. Simulate a connection failure at 60% and verify the upload resumes correctly.

2. End-to-end upload security. Implement: (a) pre-signed URL generation with user-scoped keys (uploads/{userId}/{uuid}.ext), (b) S3 bucket policy that denies uploads not matching the user's prefix, (c) virus scanning via ClamAV in a worker, (d) automatic deletion of infected files, (e) rate limiting on upload URL generation (10/min per user). Write tests that verify each security layer blocks unauthorized access.

Interview questions

Beginner

Q1: What is a pre-signed URL and why use it instead of uploading through your server?

Answer: A pre-signed URL is a time-limited URL generated by the server using its AWS credentials that grants temporary access to upload or download a specific S3 object. Using it means the client uploads directly to S3 — the server never handles the file bytes. Benefits: (1) server RAM/CPU isn't consumed by file transfer, (2) server bandwidth costs are eliminated, (3) uploads benefit from S3's scale and reliability, (4) the server stays fast for API requests while S3 handles multi-GB uploads.

Q2: How do you validate that an uploaded file is actually an image?

Answer: At minimum, three checks: (1) File extension allowlist (.jpg, .png, .webp) — but this is easily spoofed. (2) Magic bytes — read the first few bytes of the file to determine the actual format (JPEG: FF D8 FF, PNG: 89 50 4E 47). Use the file-type package. (3) Parse the file with an image library (sharp) — if it can't decode it, the file is corrupted or not an image. Optionally, (4) check dimensions (reject unreasonably large sizes to prevent decompression bomb attacks).

Q3: What's the difference between diskStorage and memoryStorage in multer?

Answer: diskStorage writes the uploaded file to a temporary directory on disk. The file is accessed via req.file.path. RAM usage is minimal regardless of file size. memoryStorage stores the entire file in a Buffer in req.file.buffer. RAM usage equals file size — a 500MB upload uses 500MB of heap. Never use memoryStorage in production unless you have a hard file size limit under 5MB and trust your users.

Senior

Q4: How would you design a file upload system that handles 10,000 concurrent uploads of files up to 5GB each?

Answer: The architecture would be: (1) Pre-signed URLs for direct S3 upload — the API servers never touch file bytes. They validate the request (file type, size, user quota) and generate a pre-signed URL in < 5ms. (2) Multipart upload via S3 — files > 100MB use S3 multipart upload (split into 5MB-5GB parts, upload in parallel). (3) S3 event notifications → SQS → workers — when S3 confirms the object is created, an SQS message triggers the processing pipeline (virus scan, transcoding, thumbnail generation). (4) Separate worker pools — CPU-bound workers (video transcoding) scale independently from I/O-bound workers (metadata extraction). (5) CDN for delivery — processed files are served via CloudFront with appropriate cache policies. (6) Rate limiting and quotas — per-user upload limits prevent abuse; storage quotas trigger cleanup of old files.

Q5: A user uploads an HTML file named innocent.jpg. It's served from cdn.yourapp.com. What's the risk and how do you fix it?

Answer: The risk is stored XSS (Cross-Site Scripting). When the browser loads https://cdn.yourapp.com/uploads/innocent.jpg, the server returns the file with Content-Type: image/jpeg (based on the original claimed MIME type). But if the file is actually HTML with <script> tags, some browsers perform "content sniffing" and execute the HTML even with the wrong Content-Type. The attacker's script now runs on cdn.yourapp.com's origin, accessing cookies and localStorage for that domain. Fixes: (1) Serve user content from a completely separate domain (usercontent.yourapp.com) that has no cookies or auth context — this is the critical fix. (2) Set X-Content-Type-Options: nosniff to prevent MIME sniffing. (3) Validate the actual content type via magic bytes, not just the extension. (4) For downloadable files, force Content-Disposition: attachment so the browser downloads rather than renders.

Q6: Compare the tus resumable upload protocol to S3 multipart upload. When would you choose one over the other?

Answer: S3 multipart upload is S3-specific — it splits a file into parts, uploads parts in parallel, and S3 reassembles. It's ideal when: you're already on AWS, the client can use the AWS SDK, and you want S3-managed reliability. Tus is a protocol (open standard) that works with any storage backend (S3, GCS, local disk, etc.). It's ideal when: you need backend portability, the client is a browser using tus-js-client, or you need upload progress that survives page refreshes (tus stores upload state on the server). For a pure AWS shop, S3 multipart with Transfer Acceleration is simpler. For a multi-cloud or browser-heavy application, tus provides a better developer experience and protocol-level resumability without AWS SDK dependencies on the client.