# Chapter 5 — File Uploads & Object Storage By the end of this chapter you will: - Generate S3 pre-signed URLs server-side - Stream uploads directly to object storage - Validate file size and type without trusting the client - Serve images via a CDN --- ## The why > **Your Node.js server is a bicycle courier. Don't ask it to deliver a grand piano. Give the sender the warehouse address and let them drive the truck themselves.** You're moving to a new apartment. You have two options: (A) Ask a friend with a bicycle to carry your couch, bed, and 50 boxes one at a time across town. Each trip takes 30 minutes, your friend is exhausted, and you pay them for 25 trips. (B) Rent a moving truck, drive everything in one trip, and your friend helps unpack at the destination. Option A is proxying file uploads through your Node.js server: the client uploads to your server, your server buffers the entire file in memory, then uploads to S3. A 500MB video upload means 500MB of server RAM consumed, 500MB of ingress bandwidth, 500MB of egress bandwidth, and your event loop blocked during the transfer. Under load, 10 concurrent uploads = 5GB of RAM and your server crashes with an OOM (out of memory) kill. Option B is pre-signed URLs: your server generates a short-lived URL that gives the client direct access to upload to S3. The client uploads directly to S3 — your server never touches the bytes. You validate the request, generate the signed URL in 5ms, and move on. S3 handles the bandwidth, the resumability, and the durability. This chapter covers multipart parsing, pre-signed URLs, the tus resumable upload protocol, file validation beyond file extension, image processing pipelines, and the patterns used by YouTube and Instagram to handle uploads at planetary scale. --- ## Core concepts ### The Multipart Problem: Parsing File Uploads When a browser uploads a file via `` or `FormData`, it sends a `multipart/form-data` request. Here's what the raw HTTP body looks like: ```http POST /upload HTTP/1.1 Content-Type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW ------WebKitFormBoundary7MA4YWxkTrZu0gW Content-Disposition: form-data; name="caption" My vacation photo ------WebKitFormBoundary7MA4YWxkTrZu0gW Content-Disposition: form-data; name="photo"; filename="beach.jpg" Content-Type: image/jpeg (binary JPEG data here — could be megabytes) ------WebKitFormBoundary7MA4YWxkTrZu0gW-- ``` Node.js's built-in HTTP parser does NOT parse multipart bodies. You need a middleware like `multer`: ```typescript title="routes/upload.ts — basic multer setup" import express from 'express'; import multer from 'multer'; import path from 'path'; const app = express(); // Store files on disk (not memory!) for anything > 1MB const upload = multer({ storage: multer.diskStorage({ destination: '/tmp/uploads', filename: (req, file, cb) => { const uniqueSuffix = Date.now() + '-' + Math.round(Math.random() * 1E9); cb(null, file.fieldname + '-' + uniqueSuffix + path.extname(file.originalname)); } }), limits: { fileSize: 10 * 1024 * 1024, // 10MB max files: 1, // Only 1 file per request }, fileFilter: (req, file, cb) => { // Reject non-image files const allowedMimes = ['image/jpeg', 'image/png', 'image/webp', 'image/gif']; if (allowedMimes.includes(file.mimetype)) { cb(null, true); } else { cb(new Error('Only JPEG, PNG, WebP, and GIF are allowed')); } } }); app.post('/upload', upload.single('photo'), (req, res) => { const file = req.file; if (!file) { return res.status(400).json({ error: 'No file uploaded' }); } // file.path = '/tmp/uploads/photo-1681234567890-123456789.jpg' // file.originalname = 'beach.jpg' // file.size = 2456789 // file.mimetype = 'image/jpeg' // Now upload to S3, process, etc. res.json({ message: 'File uploaded', filename: file.filename, size: file.size }); }); ``` `multer.memoryStorage()` stores the entire file in a `Buffer` in RAM. One 500MB upload = 500MB of heap. Three concurrent = 1.5GB, and Node.js crashes with OOM. Always use `diskStorage` for multer and stream to S3 from disk, or better yet, bypass your server entirely with pre-signed URLs. ### Pre-Signed URLs: Bypass Your Server Entirely Pre-signed URLs are the single most important concept in file uploads. They give a client temporary, scoped access to S3 without sharing your AWS credentials. **Upload flow with pre-signed URLs:** ``` 1. Client → Server: POST /api/upload-url { filename: 'beach.jpg', contentType: 'image/jpeg' } 2. Server → S3: Generate pre-signed PUT URL (valid 5 min) 3. Server → Client: { uploadUrl: 'https://s3.../beach.jpg?X-Amz-...', fileKey: 'uploads/abc123/beach.jpg' } 4. Client → S3: PUT uploadUrl with file bytes (server never sees the file) 5. Client → Server: POST /api/upload-complete { fileKey: 'uploads/abc123/beach.jpg' } 6. Server: Validate the file exists in S3, create database record ``` ```typescript title="routes/upload-url.ts — generating pre-signed URLs" import { S3Client, PutObjectCommand } from '@aws-sdk/client-s3'; import { getSignedUrl } from '@aws-sdk/s3-request-presigner'; import crypto from 'crypto'; import express from 'express'; const s3 = new S3Client({ region: process.env.AWS_REGION!, credentials: { accessKeyId: process.env.AWS_ACCESS_KEY_ID!, secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!, }, }); const BUCKET = process.env.S3_BUCKET!; const ALLOWED_TYPES = ['image/jpeg', 'image/png', 'image/webp', 'image/gif', 'application/pdf']; const MAX_SIZE = 50 * 1024 * 1024; // 50MB app.post('/api/upload-url', async (req, res) => { const { filename, contentType, size } = req.body; // 1. Validate if (!ALLOWED_TYPES.includes(contentType)) { return res.status(400).json({ error: `File type ${contentType} not allowed` }); } if (size > MAX_SIZE) { return res.status(400).json({ error: `File too large. Max ${MAX_SIZE / 1024 / 1024}MB` }); } // 2. Generate safe filename (never trust client filename) const ext = filename.split('.').pop()?.toLowerCase(); const safeFilename = `${crypto.randomUUID()}.${ext}`; const key = `uploads/${req.user.id}/${safeFilename}`; // 3. Generate pre-signed URL (valid for 5 minutes) const command = new PutObjectCommand({ Bucket: BUCKET, Key: key, ContentType: contentType, Metadata: { userId: req.user.id, originalName: encodeURIComponent(filename), }, }); const uploadUrl = await getSignedUrl(s3, command, { expiresIn: 300 }); res.json({ uploadUrl, fileKey: key, expiresIn: 300, }); }); ``` ```typescript title="routes/upload-complete.ts — confirming the upload" app.post('/api/upload-complete', async (req, res) => { const { fileKey } = req.body; // Verify the file actually exists in S3 try { await s3.send(new HeadObjectCommand({ Bucket: BUCKET, Key: fileKey, })); } catch { return res.status(400).json({ error: 'File not found in storage' }); } // Create database record const result = await db.query( `INSERT INTO user_files (user_id, s3_key, status, created_at) VALUES ($1, $2, 'uploaded', NOW()) RETURNING id`, [req.user.id, fileKey] ); // Enqueue processing job (thumbnail generation, virus scan, etc.) await fileProcessingQueue.add('process-upload', { fileId: result.rows[0].id, s3Key: fileKey, }); res.json({ fileId: result.rows[0].id, status: 'uploaded', message: 'File is being processed', }); }); ``` ### Streaming Uploads Through Your Server (When You Must) Sometimes you can't use pre-signed URLs: the file needs server-side transformation before storage, you're behind a corporate firewall, or you're not using S3. In these cases, stream — never buffer: ```typescript title="routes/streaming-upload.ts" import { PassThrough } from 'stream'; import { Upload } from '@aws-sdk/lib-storage'; import multer from 'multer'; import express from 'express'; // Use multer with diskStorage, then stream from disk to S3 app.post('/upload-stream', upload.single('file'), async (req, res) => { const file = req.file!; const key = `uploads/${req.user.id}/${crypto.randomUUID()}-${file.originalname}`; // Stream from disk to S3 (never loads entire file into memory) const fileStream = require('fs').createReadStream(file.path); const parallelUploadS3 = new Upload({ client: s3, params: { Bucket: BUCKET, Key: key, Body: fileStream, ContentType: file.mimetype, }, queueSize: 4, // Upload 4 chunks in parallel partSize: 5 * 1024 * 1024, // 5MB parts }); parallelUploadS3.on('httpUploadProgress', (progress) => { console.log(`Uploaded ${progress.loaded} of ${progress.total} bytes`); }); await parallelUploadS3.done(); // Clean up temp file require('fs').unlink(file.path, () => {}); res.json({ key }); }); ``` ### Resumable Uploads with the Tus Protocol For large files (videos, datasets, backups) on unreliable connections, you need resumable uploads. The **tus protocol** (tus.io) is the open standard. It works by splitting the file into chunks and uploading them sequentially, with the ability to resume from where it left off. ```typescript title="tus-server.ts — using @tus/server" import { Server } from '@tus/server'; import { S3Store } from '@tus/s3-store'; import express from 'express'; const app = express(); const tusServer = new Server({ path: '/uploads/files', datastore: new S3Store({ s3ClientConfig: { region: process.env.AWS_REGION!, credentials: { accessKeyId: process.env.AWS_ACCESS_KEY_ID!, secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!, }, }, bucket: process.env.S3_BUCKET!, }), respectForwardedHeaders: true, maxSize: 5 * 1024 * 1024 * 1024, // 5GB onUploadCreate: async (req, res, upload) => { // Validate file metadata before accepting upload const metadata = upload.metadata; if (!metadata?.filename?.endsWith('.mp4')) { throw new Error('Only MP4 files are allowed'); } return { res, metadata }; }, onUploadFinish: async (req, res, upload) => { console.log(`Upload complete: ${upload.id}, size: ${upload.size}`); // Enqueue processing job }, }); app.use('/uploads', tusServer.handle.bind(tusServer)); app.listen(3000); ``` The tus protocol flow: 1. Client sends `POST /uploads/files` with `Upload-Length: 524288000` (500MB) and `Upload-Metadata: filename bXl2aWRlby5tcDQ=` (base64 encoded). 2. Server responds with `Location: /uploads/files/abc123` — a unique upload URL. 3. Client sends `PATCH /uploads/files/abc123` with `Upload-Offset: 0` and the first chunk. 4. Server stores the chunk, responds with `Upload-Offset: 5242880` (5MB received). 5. Connection drops at 200MB. Client sends `HEAD /uploads/files/abc123` to check offset. 6. Server responds: `Upload-Offset: 209715200` (200MB received). 7. Client resumes: `PATCH /uploads/files/abc123` with `Upload-Offset: 209715200` and remaining data. ### File Validation: Beyond the File Extension Never trust `file.mimetype` (it comes from the client) or the file extension. Validate at multiple levels: ```typescript title="lib/file-validation.ts" import { fileTypeFromFile } from 'file-type'; import sharp from 'sharp'; import path from 'path'; interface ValidationResult { valid: boolean; reason?: string; detectedType?: string; } export async function validateUpload(filePath: string, originalName: string): Promise { // 1. Extension check (cursory — easily spoofed) const ext = path.extname(originalName).toLowerCase(); const allowedExtensions = ['.jpg', '.jpeg', '.png', '.webp', '.gif', '.pdf']; if (!allowedExtensions.includes(ext)) { return { valid: false, reason: `Extension ${ext} not allowed` }; } // 2. Magic bytes check (reads file header, not extension) const detectedType = await fileTypeFromFile(filePath); if (!detectedType) { return { valid: false, reason: 'Could not detect file type from content' }; } const allowedMimes = ['image/jpeg', 'image/png', 'image/webp', 'image/gif', 'application/pdf']; if (!allowedMimes.includes(detectedType.mime)) { return { valid: false, reason: `Actual file type ${detectedType.mime} not allowed`, detectedType: detectedType.mime }; } // 3. For images: verify it's a valid, complete image if (detectedType.mime.startsWith('image/')) { try { const metadata = await sharp(filePath).metadata(); if (!metadata.width || !metadata.height) { return { valid: false, reason: 'Invalid or corrupted image' }; } // Reject unreasonably large images (decompression bombs) if (metadata.width * metadata.height > 100_000_000) { // 100 megapixels return { valid: false, reason: 'Image dimensions too large' }; } } catch { return { valid: false, reason: 'Image failed to parse — may be corrupted' }; } } return { valid: true, detectedType: detectedType.mime }; } ``` Every file format has a unique byte signature at the start. JPEG files start with `FF D8 FF`, PNG with `89 50 4E 47`, PDF with `25 50 44 46`. The `file-type` package reads the first few bytes of the file — not the filename — and determines the real type. This catches `virus.exe` renamed to `totally_safe.jpg`. ### Image Processing Pipeline Once an image is uploaded, you typically need multiple sizes: thumbnail (150px), medium (600px), full (1920px). Sharp is the standard for this in Node.js: ```typescript title="workers/image-processor.ts" import sharp from 'sharp'; import { S3Client, GetObjectCommand, PutObjectCommand } from '@aws-sdk/client-s3'; import { getSignedUrl } from '@aws-sdk/s3-request-presigner'; const IMAGE_SIZES = [ { name: 'thumbnail', width: 150, height: 150, fit: 'cover' as const }, { name: 'medium', width: 600, height: 600, fit: 'inside' as const }, { name: 'large', width: 1920, height: 1080, fit: 'inside' as const }, ]; export async function processImage(s3Key: string): Promise> { // Download original from S3 as stream const getCommand = new GetObjectCommand({ Bucket: BUCKET, Key: s3Key }); const response = await s3.send(getCommand); const imageBuffer = await response.Body!.transformToByteArray(); const urls: Record = {}; for (const size of IMAGE_SIZES) { const pipeline = sharp(Buffer.from(imageBuffer)) .resize(size.width, size.height, { fit: size.fit, withoutEnlargement: true }) .jpeg({ quality: 85, progressive: true }) // Convert to progressive JPEG .withMetadata(); // Preserve EXIF (rotation) const outputBuffer = await pipeline.toBuffer(); const variantKey = s3Key.replace( /(\.[^.]+)$/, `-${size.name}$1` ); await s3.send(new PutObjectCommand({ Bucket: BUCKET, Key: variantKey, Body: outputBuffer, ContentType: 'image/jpeg', CacheControl: 'public, max-age=31536000, immutable', // Cache forever })); urls[size.name] = `${process.env.CDN_URL}/${variantKey}`; } return urls; } ``` ### Virus Scanning For user-generated content, virus scanning is non-negotiable. ClamAV is the open-source standard: ```typescript title="workers/virus-scanner.ts" import { execFile } from 'child_process'; import { promisify } from 'util'; const execFileAsync = promisify(execFile); export async function scanFile(filePath: string): Promise<{ safe: boolean; signature?: string }> { try { // clamscan exits with 0 = clean, 1 = virus found, 2 = error const { stdout } = await execFileAsync('clamscan', [ '--no-summary', '--infected', '--stdout', filePath, ]); if (stdout.trim()) { // stdout contains "filepath: VirusName FOUND" const match = stdout.match(/(\S+)\s+FOUND$/); return { safe: false, signature: match?.[1] || 'Unknown' }; } return { safe: true }; } catch (err: any) { // Exit code 1 = virus found if (err.code === 1) { const match = err.stdout?.match(/(\S+)\s+FOUND$/); return { safe: false, signature: match?.[1] || 'Unknown' }; } // Exit code 2 = clamscan error — fail closed (assume unsafe) console.error('Virus scan failed:', err); return { safe: false, signature: 'SCAN_ERROR' }; } } ``` If the virus scanner crashes, times out, or can't read the file, treat it as infected. \"Fail closed\" means blocking on uncertainty. \"Fail open\" means one scanner bug and malicious files sail through. ### CDN Fronting Never serve user-uploaded files directly from S3. Put a CDN (CloudFront, Cloudflare, Fastly) in front: ```typescript // When generating URLs for clients, use CDN domain, not S3 domain const cdnUrl = `https://cdn.yourapp.com/${s3Key}`; // S3 objects get Cache-Control metadata await s3.send(new PutObjectCommand({ Bucket: BUCKET, Key: key, Body: fileStream, CacheControl: 'public, max-age=31536000, immutable', // Cache for 1 year ContentType: mimeType, })); // For authenticated/private files: use CloudFront signed URLs // instead of S3 pre-signed URLs (CloudFront is cheaper for high traffic) ``` Benefits of CDN fronting: - **Lower latency**: files served from edge locations near users - **Reduced S3 costs**: S3 egress is expensive ($0.09/GB); CDN egress is cheaper ($0.02-0.08/GB depending on provider) - **DDoS protection**: CDNs absorb volumetric attacks - **HTTPS with custom domain**: your files are at `cdn.yourapp.com`, not `s3.amazonaws.com` --- ## Real-world example: How YouTube and Instagram handle uploads **YouTube's upload pipeline** processes 500+ hours of video uploaded every minute. Their architecture reveals patterns you can downscale: 1. **Chunked upload with resumability.** YouTube's upload client splits videos into chunks (typically 1-8MB each). If your connection drops at 40%, the client resumes from chunk N, not byte 0. This is conceptually the tus protocol, though YouTube uses a proprietary implementation. 2. **Processing happens asynchronously.** When the upload completes, the video enters a processing pipeline: transcoding to multiple resolutions (144p through 4K), thumbnail generation, content ID matching (copyright detection), and ML-based content moderation. None of this blocks the upload confirmation — the creator sees "Processing... 0%" immediately after upload. 3. **Multiple storage tiers.** Hot videos (trending, recent uploads from popular creators) are stored on SSD-backed storage for fast access. Warm videos sit on HDD. Cold videos (years old, low views) go to tape archive. This tiering saves millions in storage costs. **Instagram's image pipeline** processes 100M+ photos per day: 1. **Progressive loading.** Instagram serves a 200-byte blurhash placeholder immediately, followed by a low-res version (50KB), then the full-res version. The user sees something instantly even on 3G. 2. **Client-side preprocessing.** Before upload, the Instagram app resizes and compresses the image on-device. A 12MP phone photo (8MB) becomes a 1080px JPEG (200KB) before it ever touches the network. This reduces upload time and server processing load. 3. **CDN multi-tier caching.** Profile photos (frequently accessed, small) are cached aggressively at edge. Feed photos (timely, large) have shorter cache TTLs. Stories (ephemeral, 24h life) use the shortest TTLs and may not be cached at all if they're viewed by few people. **Key lesson from both:** the upload is just the first step. The real engineering is in the async processing pipeline (transcoding, resizing, moderation) and the delivery architecture (CDN, progressive loading, tiered storage). --- ## Common mistakes | Mistake | Why it's wrong | What to do instead | |---|---|---| | Buffering entire files in `memoryStorage` | 500MB upload = 500MB RAM; 3 concurrent = OOM crash | Use `diskStorage` or pre-signed URLs; stream, never buffer | | Trusting `file.mimetype` from the client | Client can send `virus.exe` with `Content-Type: image/jpeg` | Validate magic bytes with `file-type` package; check actual file headers | | Proxying uploads through Node.js to S3 | Server pays for ingress AND egress bandwidth; file touches server unnecessarily | Use S3 pre-signed URLs — client uploads directly to S3 | | No file size limit | Attacker uploads a 10GB file, fills your storage, runs up your bill | Set `limits.fileSize` in multer and validate size before generating pre-signed URLs | | Serving files directly from S3 | High egress costs ($0.09/GB); no caching; slow for global users | Put a CDN in front (CloudFront/Cloudflare); set `Cache-Control` headers | | No virus scanning | User-generated content is the #1 malware vector for platforms | Scan with ClamAV (or a cloud API); fail closed if scanner errors | | Using predictable filenames | `user_123_avatar.jpg` allows attackers to enumerate and scrape all user uploads | Use `crypto.randomUUID()` for filenames; never expose sequential IDs in object keys | --- ## Production notes - **S3 bucket policy: block public access.** All buckets should have "Block all public access" enabled. Access files via pre-signed URLs (short-lived, scoped) or CDN signed URLs. Never make an S3 bucket publicly readable. - **S3 lifecycle policies.** Configure auto-deletion or archival: temporary uploads (`/tmp/`) expire after 24 hours; processed uploads transition to Standard-IA after 30 days, Glacier after 90 days. This saves 40-70% on storage costs. - **Multipart upload for large files.** S3 supports multipart uploads: split a file into parts, upload in parallel, S3 reassembles. The `@aws-sdk/lib-storage` Upload class handles this automatically. Essential for files > 100MB. - **CORS on S3 buckets.** If clients upload directly to S3 via pre-signed URLs, configure CORS on the bucket to allow your domain. Otherwise browser requests are blocked by same-origin policy. - **Webhook or S3 event notifications.** Instead of the client calling `/api/upload-complete` (which can be spoofed), configure S3 to send an SNS/SQS/Lambda event on `s3:ObjectCreated`. Your backend processes the file when S3 confirms the object exists. --- ## Performance notes - **Presigned URL generation is sub-millisecond.** No cryptographic work — the SDK signs a request with your credentials locally. You can generate 1,000+ per second on a single Node.js process. - **S3 Transfer Acceleration.** For users far from your S3 bucket region, enable Transfer Acceleration. It uses CloudFront edge locations to accept uploads and routes over AWS backbone to S3. 50-500% speedup for cross-continent uploads. - **Parallel multipart upload parts.** For a 1GB file, split into 100 × 10MB parts and upload 10 parts simultaneously. The `@aws-sdk/lib-storage` Upload class with `queueSize: 10` does this. Tune `partSize` (5MB-5GB, default 5MB) and `queueSize` (1-10) per workload. - **Sharp performance tuning.** Sharp uses libvips (C library) and is extremely fast — 4-5x faster than ImageMagick and 10x faster than Jimp (pure JS). Process images in parallel with `Promise.all` across sizes. - **Lazy resize vs eager resize.** Don't generate all image sizes on upload. Generate only the most common sizes eagerly (medium). Generate rare sizes (specific crop dimensions for a redesign) lazily on first access, then cache. --- ## Security notes - **Validate pre-signed URL callbacks.** An attacker could call `/api/upload-complete` with any `fileKey` they guess, claiming to have uploaded it. Always verify the object exists in S3 (`HeadObject`) and that the `userId` in the S3 metadata matches the requesting user. - **Content-Disposition for downloads.** When serving user files, set `Content-Disposition: attachment; filename="..."` for downloadable files. Without it, `evil.html` uploaded by a user renders in the browser — potentially accessing cookies, localStorage, or performing phishing if served from your domain. Serve user content from a separate domain (e.g., `usercontent.yourapp.com`) to isolate it from your main app's cookies and JavaScript context. - **File type restrictions must be at the infrastructure level.** In addition to application-level validation, use an S3 bucket policy that denies `PutObject` for disallowed content types. Defense in depth: even if your app validation is bypassed, S3 rejects the upload. - **Rate limiting upload endpoints.** `/api/upload-url` is cheap but can be abused. Rate limit by user ID and IP — a free user shouldn't be able to request 10,000 upload URLs per minute (storage costs add up even if files are tiny). - **Scan uploads for malware.** ClamAV or a cloud API (VirusTotal, AWS Macie). For high-volume platforms, scanning can be async (enqueue after upload confirmation). If a virus is found, delete the S3 object, ban the user, and log for security review. --- ## Exercises ### Beginner 1. **Set up multer with disk storage.** Create an Express endpoint that accepts a single image upload via `multer`. Store to `/tmp/uploads`. Return the filename, size, and mimetype. Test with `curl -F "photo=@beach.jpg"`. 2. **Generate a pre-signed S3 upload URL.** Using `@aws-sdk/s3-request-presigner`, write an endpoint that accepts `filename` and `contentType` and returns a pre-signed PUT URL valid for 5 minutes. Test by using `curl` to PUT a file directly to the returned URL. ### Intermediate 1. **Build an image processing pipeline.** After upload, use `sharp` to generate thumbnail (150×150), medium (600×600), and large (1920×1080) variants. Upload all variants to S3. Store the URLs in a database. Return all URLs to the client. 2. **Implement file validation with magic bytes.** Write a validation function that checks: file extension (allowlist), MIME type from magic bytes (`file-type` package), and image validity (can `sharp` open it?). Reject files that fail any check. Log rejection reasons for security monitoring. ### Advanced 1. **Build a resumable upload system with tus.** Set up a `@tus/server` with S3 store. Implement a client (browser or Node.js) using `tus-js-client` that uploads a 500MB file. Simulate a connection failure at 60% and verify the upload resumes correctly. 2. **End-to-end upload security.** Implement: (a) pre-signed URL generation with user-scoped keys (`uploads/{userId}/{uuid}.ext`), (b) S3 bucket policy that denies uploads not matching the user's prefix, (c) virus scanning via ClamAV in a worker, (d) automatic deletion of infected files, (e) rate limiting on upload URL generation (10/min per user). Write tests that verify each security layer blocks unauthorized access. --- ## Interview questions ### Beginner **Q1: What is a pre-signed URL and why use it instead of uploading through your server?** **Answer:** A pre-signed URL is a time-limited URL generated by the server using its AWS credentials that grants temporary access to upload or download a specific S3 object. Using it means the client uploads directly to S3 — the server never handles the file bytes. Benefits: (1) server RAM/CPU isn't consumed by file transfer, (2) server bandwidth costs are eliminated, (3) uploads benefit from S3's scale and reliability, (4) the server stays fast for API requests while S3 handles multi-GB uploads. **Q2: How do you validate that an uploaded file is actually an image?** **Answer:** At minimum, three checks: (1) File extension allowlist (`.jpg`, `.png`, `.webp`) — but this is easily spoofed. (2) Magic bytes — read the first few bytes of the file to determine the actual format (JPEG: `FF D8 FF`, PNG: `89 50 4E 47`). Use the `file-type` package. (3) Parse the file with an image library (`sharp`) — if it can't decode it, the file is corrupted or not an image. Optionally, (4) check dimensions (reject unreasonably large sizes to prevent decompression bomb attacks). **Q3: What's the difference between `diskStorage` and `memoryStorage` in multer?** **Answer:** `diskStorage` writes the uploaded file to a temporary directory on disk. The file is accessed via `req.file.path`. RAM usage is minimal regardless of file size. `memoryStorage` stores the entire file in a `Buffer` in `req.file.buffer`. RAM usage equals file size — a 500MB upload uses 500MB of heap. Never use `memoryStorage` in production unless you have a hard file size limit under 5MB and trust your users. ### Senior **Q4: How would you design a file upload system that handles 10,000 concurrent uploads of files up to 5GB each?** **Answer:** The architecture would be: (1) **Pre-signed URLs for direct S3 upload** — the API servers never touch file bytes. They validate the request (file type, size, user quota) and generate a pre-signed URL in < 5ms. (2) **Multipart upload via S3** — files > 100MB use S3 multipart upload (split into 5MB-5GB parts, upload in parallel). (3) **S3 event notifications → SQS → workers** — when S3 confirms the object is created, an SQS message triggers the processing pipeline (virus scan, transcoding, thumbnail generation). (4) **Separate worker pools** — CPU-bound workers (video transcoding) scale independently from I/O-bound workers (metadata extraction). (5) **CDN for delivery** — processed files are served via CloudFront with appropriate cache policies. (6) **Rate limiting and quotas** — per-user upload limits prevent abuse; storage quotas trigger cleanup of old files. **Q5: A user uploads an HTML file named `innocent.jpg`. It's served from `cdn.yourapp.com`. What's the risk and how do you fix it?** **Answer:** The risk is stored XSS (Cross-Site Scripting). When the browser loads `https://cdn.yourapp.com/uploads/innocent.jpg`, the server returns the file with `Content-Type: image/jpeg` (based on the original claimed MIME type). But if the file is actually HTML with `

Chapter 5 — File Uploads & Object Storage

Part of L4