# Chapter 3 — Streaming AI Responses By the end of this chapter you will: - Stream LLM output via SSE - Handle abort signals when the user navigates away - Buffer tokens for smoother UX - Compare SSE to WebSocket for AI streaming --- ## The why > **Users will wait 8 seconds for a page to load. They will not wait 8 seconds staring at a spinner while an LLM "thinks." Streaming cuts perceived latency from seconds to milliseconds — the first token appears almost instantly, and the user reads along as the model generates. This isn't a UX nice-to-have; it's the difference between a product that feels alive and one that feels broken.** You walk into a sushi restaurant. You have two options: **Option A — The "batch" restaurant:** You sit down. The chef disappears into the kitchen. Twenty minutes later, he emerges holding a platter with all 12 pieces of your omakase. The sushi is excellent, but you spent 20 minutes staring at an empty table, wondering if your order was forgotten. You check your phone. You get restless. The food arrives and it's great — but the experience felt broken. **Option B — The conveyor belt (kaitenzushi):** You sit down at a counter with a moving belt. Within 30 seconds, the first plate glides past — a piece of salmon nigiri. You grab it, eat it, and 15 seconds later a tuna roll arrives. Then shrimp. Then tamago. You're continuously engaged. The entire meal still takes 20 minutes from first bite to last, but you never waited more than 30 seconds between bites. The experience feels fast, responsive, alive. Streaming AI responses is the conveyor belt. The total generation time is the same as the batch approach — the model still needs time to produce all 500 tokens. But instead of waiting for the full response, you pipe each token to the client as it's generated. The user sees "Salmon..." at 200ms, "Salmon nigiri is..." at 400ms, and reads along while the rest generates. Perceived latency: 200ms. Actual latency: 8 seconds. That's the power of streaming. --- ## Core concepts ### Server-Sent Events (SSE): the protocol that powers AI streaming SSE is a simple HTTP-based protocol for server-to-client streaming. Unlike WebSockets (bidirectional), SSE is unidirectional — the server pushes data to the client over a single, long-lived HTTP connection. This simplicity is exactly what makes it perfect for AI streaming: the client sends a request (the prompt), the server streams the response (generated tokens), the connection closes when generation finishes. ```typescript // Minimal SSE endpoint in Next.js App Router // app/api/chat/stream/route.ts export async function POST(req: Request) { const { messages } = await req.json(); // Create a ReadableStream that the client will consume via EventSource const stream = new ReadableStream({ async start(controller) { const encoder = new TextEncoder(); try { // Simulate streaming tokens (in production, this comes from an LLM) const tokens = ['Hello', '!', ' How', ' can', ' I', ' help', ' you', ' today', '?']; for (const token of tokens) { // SSE format: "data: \n\n" const sseChunk = `data: ${JSON.stringify({ token })}\n\n`; controller.enqueue(encoder.encode(sseChunk)); // Simulate generation delay — in production, the LLM dictates timing await new Promise(resolve => setTimeout(resolve, 100)); } // Send the [DONE] event to signal completion controller.enqueue(encoder.encode('data: [DONE]\n\n')); controller.close(); } catch (error) { controller.error(error); } } }); return new Response(stream, { headers: { 'Content-Type': 'text/event-stream', 'Cache-Control': 'no-cache', 'Connection': 'keep-alive', 'X-Accel-Buffering': 'no' // Disable Nginx buffering for this endpoint } }); } ``` SSE is just a specific text format over HTTP: `data: \n\n`. Each event is separated by two newlines. The `data:` prefix is required. You can optionally include `event:`, `id:`, and `retry:` fields. For AI streaming, `data:` is all you need. The `[DONE]` message is a convention (not part of the SSE spec) popularized by OpenAI — it signals that the stream is complete and the client should close the connection. ### Parsing LLM streaming responses: three providers, three event shapes Remember the provider abstraction from Chapter 1? The streaming layer is where the differences between providers become most visible. OpenAI, Anthropic, and Gemini each emit completely different streaming event types: ```typescript // src/ai/streaming/openai-stream.ts import OpenAI from 'openai'; export async function* streamOpenAI(messages: ChatMessage[]): AsyncGenerator { const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY! }); const stream = await openai.chat.completions.create({ model: 'gpt-4o', messages: messages.map(m => ({ role: m.role, content: m.content })), stream: true }); for await (const chunk of stream) { const delta = chunk.choices[0]?.delta?.content; if (delta) yield delta; // OpenAI also emits usage info in the final chunk when streaming if (chunk.choices[0]?.finish_reason) { console.log('Stream finished:', chunk.choices[0].finish_reason); } } } // src/ai/streaming/anthropic-stream.ts import Anthropic from '@anthropic-ai/sdk'; export async function* streamAnthropic( systemPrompt: string, messages: ChatMessage[] ): AsyncGenerator { const anthropic = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY! }); const stream = await anthropic.messages.create({ model: 'claude-sonnet-4-20250514', system: systemPrompt, messages: messages.map(m => ({ role: m.role as 'user' | 'assistant', content: m.content })), max_tokens: 1024, stream: true }); for await (const event of stream) { // Anthropic emits different event types: message_start, content_block_start, // content_block_delta, content_block_stop, message_delta, message_stop if (event.type === 'content_block_delta' && event.delta.type === 'text_delta') { yield event.delta.text; } // Track usage from message_delta events if (event.type === 'message_delta') { console.log('Usage:', event.usage); } } } // src/ai/streaming/gemini-stream.ts import { GoogleGenAI } from '@google/genai'; export async function* streamGemini(messages: ChatMessage[]): AsyncGenerator { const genai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY! }); const response = await genai.models.generateContentStream({ model: 'gemini-2.5-flash', contents: messages.map(m => ({ role: m.role === 'assistant' ? 'model' : 'user', parts: [{ text: m.content }] })) }); for await (const chunk of response) { const text = chunk.candidates?.[0]?.content?.parts?.[0]?.text; if (text) yield text; } } ``` Streaming doesn't reduce your API bill. You pay for the same number of tokens whether you stream or not. In fact, streaming can cost MORE per request because the model might generate slightly longer responses (it can't "plan ahead" for conciseness the way it can in batch mode). The benefit is purely UX: users perceive faster responses, which reduces bounce rates and increases engagement. For internal batch jobs (summarizing 10,000 support tickets overnight), use non-streaming — nobody's watching. ### The unified streaming adapter: one `AsyncGenerator` to rule them all Just like Chapter 1's `LLMProvider` interface unified `chat()` calls, we need a unified streaming interface: ```typescript // src/ai/streaming/unified-stream.ts import { ChatMessage } from '../types'; export interface StreamingProvider { stream(messages: ChatMessage[], systemPrompt?: string): AsyncGenerator; } export interface StreamEvent { type: 'token' | 'error' | 'done'; data?: string; usage?: { promptTokens: number; completionTokens: number }; finishReason?: string; } export async function* unifiedStream( provider: 'openai' | 'anthropic' | 'gemini', messages: ChatMessage[], systemPrompt?: string ): AsyncGenerator { try { switch (provider) { case 'openai': { const stream = streamOpenAI(messages); for await (const token of stream) { yield { type: 'token', data: token }; } yield { type: 'done' }; break; } case 'anthropic': { const stream = streamAnthropic(systemPrompt || '', messages); for await (const token of stream) { yield { type: 'token', data: token }; } yield { type: 'done' }; break; } case 'gemini': { const stream = streamGemini(messages); for await (const token of stream) { yield { type: 'token', data: token }; } yield { type: 'done' }; break; } } } catch (error) { yield { type: 'error', data: error instanceof Error ? error.message : 'Streaming failed' }; } } ``` ### AbortController: respecting user attention The single most important streaming feature is the ability to cancel mid-generation. When a user navigates away, clicks "stop generating", or closes the tab, you need to abort the upstream LLM request immediately. Otherwise, you're paying for tokens nobody will ever read. ```typescript // src/ai/streaming/abortable-stream.ts export async function* abortableStream( provider: 'openai' | 'anthropic' | 'gemini', messages: ChatMessage[], signal: AbortSignal ): AsyncGenerator { // Check for abort before we even start if (signal.aborted) { yield { type: 'done' }; return; } // Set up abort handler const onAbort = () => { console.warn('Stream aborted by client — cancelling LLM request'); }; signal.addEventListener('abort', onAbort, { once: true }); try { for await (const event of unifiedStream(provider, messages)) { // Check abort between each token — stop yielding immediately if (signal.aborted) { yield { type: 'done' }; return; } yield event; } } finally { signal.removeEventListener('abort', onAbort); } } // Usage in API route: export async function POST(req: Request) { const { messages } = await req.json(); const signal = req.signal; // Request object has a built-in AbortSignal! const stream = new ReadableStream({ async start(controller) { const encoder = new TextEncoder(); try { for await (const event of abortableStream('openai', messages, signal)) { if (signal.aborted) { controller.close(); return; } if (event.type === 'token') { controller.enqueue(encoder.encode(`data: ${JSON.stringify({ token: event.data })}\n\n`)); } else if (event.type === 'error') { controller.enqueue(encoder.encode(`data: ${JSON.stringify({ error: event.data })}\n\n`)); controller.close(); return; } else if (event.type === 'done') { controller.enqueue(encoder.encode('data: [DONE]\n\n')); controller.close(); return; } } } catch (error) { controller.error(error); } } }); return new Response(stream, { headers: { 'Content-Type': 'text/event-stream', 'Cache-Control': 'no-cache', 'Connection': 'keep-alive' } }); } ``` ### Backpressure: when the client can't keep up SSE over HTTP handles backpressure naturally — the TCP connection's flow control prevents the server from overwhelming the client's receive buffer. But there's a subtlety with AI streaming: the LLM is generating tokens at its own pace (maybe 50 tokens/second), and your server is forwarding them to the client. If the client's connection is slow (mobile network, congested WiFi), tokens queue up in the server's send buffer. The ReadableStream API handles this with its internal queuing strategy: ```typescript const stream = new ReadableStream({ async start(controller) { for await (const token of tokens) { // controller.enqueue returns when the token is in the stream's internal queue // If the queue is full (backpressure), it waits until there's room // This naturally slows the producer to match the consumer's speed controller.enqueue(encoder.encode(sseChunk)); } controller.close(); } }, { // High water mark: maximum chunks in the internal queue before backpressure kicks in highWaterMark: 10 // Buffer up to 10 SSE chunks (adjust based on chunk size) }); ``` Don't fight backpressure — use it. If the client is slow, let tokens queue up server-side (up to your buffer limit). This prevents the LLM from generating 1,000 tokens for a user who's already on a different page (AbortController will cancel it). The `highWaterMark` setting controls how many tokens you're willing to buffer before the LLM pauses. Set it lower (5-10) for chat UIs where users read tokens as they arrive. Set it higher (50-100) for audio/text-to-speech pipelines where smooth playback matters more than latency. ### Frontend token-by-token rendering The backend streams tokens. The frontend renders them. Here's how the client consumes an SSE stream with proper cleanup, reconnection handling, and smooth rendering: ```typescript // client/useStreamingChat.ts import { useState, useRef, useCallback } from 'react'; export function useStreamingChat() { const [response, setResponse] = useState(''); const [isStreaming, setIsStreaming] = useState(false); const abortControllerRef = useRef(null); const sendMessage = useCallback(async (userMessage: string) => { // Reset state setResponse(''); setIsStreaming(true); // Create abort controller for this request const abortController = new AbortController(); abortControllerRef.current = abortController; try { const res = await fetch('/api/chat/stream', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ messages: [{ role: 'user', content: userMessage }] }), signal: abortController.signal }); if (!res.ok || !res.body) { throw new Error(`HTTP ${res.status}`); } const reader = res.body.getReader(); const decoder = new TextDecoder(); let buffer = ''; while (true) { const { done, value } = await reader.read(); if (done) break; // Decode the chunk and append to buffer buffer += decoder.decode(value, { stream: true }); // SSE messages are separated by \n\n. Process complete messages. const lines = buffer.split('\n\n'); buffer = lines.pop() || ''; // Keep the incomplete line in buffer for (const line of lines) { if (!line.startsWith('data: ')) continue; const data = line.slice(6); // Remove "data: " prefix if (data === '[DONE]') { setIsStreaming(false); return; } try { const parsed = JSON.parse(data); if (parsed.token) { // Append token to response — React re-renders with each token setResponse(prev => prev + parsed.token); } else if (parsed.error) { setResponse(prev => prev + `\n\n[Error: ${parsed.error}]`); setIsStreaming(false); return; } } catch { // Skip malformed SSE messages } } } } catch (error: any) { if (error.name === 'AbortError') { setResponse(prev => prev + '\n\n[Generation stopped]'); } else { setResponse(prev => prev + `\n\n[Error: ${error.message}]`); } } finally { setIsStreaming(false); abortControllerRef.current = null; } }, []); const stopGeneration = useCallback(() => { abortControllerRef.current?.abort(); }, []); return { response, isStreaming, sendMessage, stopGeneration }; } ``` SSE messages are delimited by `\n\n` (two newlines). But a chunk from the stream might split a message in the middle. The `buffer` pattern above handles this: accumulate incoming bytes, split on `\n\n`, keep the last partial line in the buffer for the next chunk. Without this, you'll lose tokens that happen to span chunk boundaries — users see words mysteriously missing from the response. ### SSE vs WebSocket for AI streaming Developers often ask: "Why SSE and not WebSocket for AI streaming?" Here's the decision framework: | Feature | SSE | WebSocket | |---------|-----|-----------| | **Direction** | Server → Client only | Bidirectional | | **Protocol** | Plain HTTP (works through all proxies) | Upgrade from HTTP (some proxies block it) | | **Reconnection** | Automatic (`EventSource` API) | Manual (implement yourself) | | **Binary data** | Text only (base64 for binary) | Text and binary frames | | **Complexity** | Minimal — ReadableStream + fetch | Socket lifecycle, ping/pong, reconnection logic | | **Load balancer support** | Works with all HTTP load balancers | Requires sticky sessions or WebSocket-aware LB | Use SSE when the client sends one request and the server streams a response (AI chat, real-time logs, progress updates). Use WebSocket when the communication is bidirectional over a long period (collaborative editing, multiplayer games, live dashboards with server-push AND user interactions). For AI streaming specifically: SSE is the right choice 95% of the time. --- ## Real-world example: How ChatGPT's streaming UX became the industry standard ChatGPT's streaming implementation is the reference architecture that every AI product now follows. When you type a message into ChatGPT, here's what happens: **The backend pipeline:** (1) Your message arrives at an API gateway. (2) The gateway routes to a model inference server (OpenAI's custom infrastructure, not the public API). (3) The model generates tokens one at a time. (4) Each token is immediately pushed through the SSE connection. (5) The frontend renders each token as it arrives with a subtle "cursor blink" animation at the end of the text — this animation is the critical UX detail that signals "more is coming." **The frontend rendering strategy:** ChatGPT doesn't re-render the entire message on each token (that would be expensive for long responses). Instead, it maintains a text buffer and appends to the DOM incrementally. The "thinking" dots appear while the first token is being generated. The message container auto-scrolls as new tokens appear. The "stop generating" button is always visible during streaming — clicking it sends an abort signal to the backend. **The key UX insight that most clones miss:** The cursor animation. ChatGPT shows a blinking `▊` at the end of the streaming text. This tiny detail tells the user "the model is still working, more text is coming." Without it, users hit a natural pause in the response (the model generated a period and is thinking about the next sentence) and assume the generation is complete. They start reading, then new text appears and disorients them. The blinking cursor prevents this confusion. Cursor implements streaming similarly but adds code-block awareness: when streaming code (inside ``` blocks), it renders syntax-highlighted tokens as they arrive. The code block's background appears immediately when ` ``` ` is detected, and syntax highlighting updates incrementally. This attention to detail — making streaming feel like magic, not like a loading bar — is what separates polished AI products from demos. --- ## Common mistakes | Mistake | Why it's wrong | What to do instead | |---------|---------------|-------------------| | Not handling `AbortController` on the backend | When a user navigates away, the LLM keeps generating tokens you're paying for. For a 1,000-token response, that's ~$0.01 wasted. Across 100K users, that's real money | Pass `req.signal` to your streaming generator. Check `signal.aborted` between every token. Cancel the upstream LLM stream immediately on abort | | Forgetting `X-Accel-Buffering: no` behind Nginx | Nginx buffers responses by default. Your "streaming" endpoint sends the entire response in one chunk after generation completes — exactly the behavior you're trying to avoid | Set `proxy_buffering off;` in the Nginx location block for streaming endpoints, or send `X-Accel-Buffering: no` header from your application. Test with `curl -N` to verify chunked delivery | | Using non-streaming mode for chat interfaces | Users stare at a spinner for 8 seconds. They assume the app is broken. Bounce rate spikes. Your support inbox fills with "is the AI not working?" tickets | Always use streaming for interactive chat. The only acceptable use of non-streaming is batch processing and background jobs where no user is waiting | | Not handling partial SSE messages on the frontend | Network packets don't align with SSE message boundaries. A chunk might contain `data: {"token": "Hel` and the next chunk `lo"}\n\n`. Without buffering, you lose the first half | Implement the buffer pattern: accumulate chunks, split on `\n\n`, keep the last partial line. Only process complete SSE messages | | Keeping the connection alive indefinitely without a timeout | A stalled LLM (rate limited, provider outage) holds the HTTP connection open forever. Your server runs out of file descriptors. Other users can't connect | Set a timeout on the streaming loop: 30 seconds for standard requests, 120 seconds for long-form generation. Send a friendly "Generation is taking longer than expected..." message before timing out | | Not logging streaming errors separately from batch errors | Streaming failures (mid-generation crash, network drop) are different from batch failures (bad request, auth error). Without separate logging, you can't distinguish "the model crashed" from "the user closed the tab" | Log `stream_aborted`, `stream_error`, and `stream_completed` events separately. Track abort rate — if 40% of streams are aborted, your responses might be too slow or too long | | Emitting `[DONE]` without closing the connection | The EventSource API keeps the connection open, waiting for more events. Memory leaks on both client and server | After sending `[DONE]`, call `controller.close()` on the server. On the client, check for `[DONE]` and close the reader | --- ## Production notes - **Monitor stream health metrics separately from application metrics.** Track: stream start rate, stream completion rate (completed without abort), median time-to-first-token (TTFT), median tokens-per-second during streaming, abort rate (user-initiated vs. error), and stream error rate by provider. A spike in TTFT means the LLM is slow — switch providers. A spike in abort rate means users aren't finding answers fast enough — optimize prompts or reduce response length. - **Implement a token buffer for smoother rendering.** Tokens arrive from the LLM at irregular intervals — sometimes 3 tokens in 50ms, then nothing for 200ms. Rendering each token immediately creates a jittery, distracting UX. Buffer tokens for 30-50ms: collect incoming tokens, then flush the buffer either when it reaches 5 tokens or 50ms elapses. This smooths the rendering without introducing perceptible delay. - **Set connection limits on streaming endpoints.** Streaming connections are long-lived and consume server resources (one TCP connection, one stream object, memory for buffering). If you have 1,000 concurrent streaming connections, that's 1,000 open file descriptors. Configure your server's max connections based on available memory: roughly 2-4MB per streaming connection (including buffers). At 4GB of available RAM, cap at 500-1000 concurrent streams. Implement a queue or 503 "Service Unavailable" when at capacity. - **Use HTTP/2 for multiplexing multiple streams.** HTTP/1.1 limits browsers to 6 concurrent connections per domain. If your app has 6 streaming AI responses plus API calls, you hit the limit. HTTP/2 multiplexes many streams over a single TCP connection — no more head-of-line blocking. Most hosting platforms (Vercel, Cloudflare) enable HTTP/2 by default for HTTPS traffic. - **Log token-level telemetry during streaming.** For each streamed response, log: `stream_id`, `provider`, `model`, `ttft_ms` (time to first token), `total_tokens`, `duration_ms`, `tokens_per_second`, `aborted` (boolean), `finish_reason`. This data reveals: which provider is fastest for your use case, whether your prompts generate overly long responses, and which users consistently abort mid-generation (they're not getting what they need). ## Performance notes - **Time-to-first-token (TTFT) is the most important streaming metric.** Users perceive latency based on when they see the first token, not the last. Target TTFT < 500ms. If TTFT > 1 second, users get restless. Factors affecting TTFT: prompt length (longer prompts = more prefill computation), model size (larger models start slower), provider load (shared infrastructure can queue). Cache system prompts to reduce prefill time. Use smaller/faster models for simple queries where TTFT matters more than answer quality. - **Tokens-per-second (TPS) determines reading comfort.** Users read at ~5-8 tokens/second (English). If your stream delivers 50 TPS, tokens pile up faster than anyone can read — the scrollbar races ahead and users feel overwhelmed. If it delivers 5 TPS, it's comfortable but borderline slow. Target 20-30 TPS: fast enough to feel snappy, slow enough to read. Some providers let you control this (Anthropic doesn't, OpenAI doesn't directly, but you can add artificial delays between tokens if the LLM is too fast). - **Gzip compression on SSE streams is usually counterproductive.** SSE events are tiny (10-50 bytes each). Gzip works best on larger payloads — for tiny chunks, the compression overhead often exceeds any savings. Plus, Gzip adds buffering latency (the compressor waits for more data). Unless your tokens are very large (streaming paragraphs, not tokens), disable compression for SSE endpoints. - **Connection reuse saves TLS handshake overhead.** If your frontend opens a new HTTP connection for every streaming request, you pay the TLS handshake cost (50-200ms) on every message. Configure HTTP/2 or HTTP/1.1 keep-alive so subsequent streaming requests reuse the same TCP+TLS connection. This cuts 50-200ms from every TTFT after the first message in a session. - **Prefetch embeddings and context BEFORE starting the stream.** If your streaming endpoint also does vector search (fetching relevant context for the prompt), do that BEFORE creating the stream. The user waits for the first token — don't make them wait for your database query PLUS the LLM. Precompute everything the LLM needs, then start the stream with all context ready. ## Security notes - **Validate `Origin` and `Host` headers on streaming endpoints.** SSE endpoints are vulnerable to Cross-Site WebSocket Hijacking (CSWSH) — a malicious site can open an EventSource to your streaming endpoint and receive tokens intended for another user if your auth is cookie-based. Set `Access-Control-Allow-Origin` to your specific domain (not `*`), and validate the `Origin` header server-side. For sensitive AI responses, use token-based auth (Authorization header) instead of cookies. - **Implement per-user concurrent stream limits.** A single user shouldn't be able to open 50 simultaneous streaming connections — each one costs you money (LLM tokens) and server resources. Limit to 3-5 concurrent streams per user. Return 429 if exceeded. This also prevents a buggy client from accidentally spamming streams. - **Stream content through a moderation filter.** Offensive or dangerous LLM outputs stream token-by-token — the first few tokens might look innocent, and the problematic content appears later. Implement a sliding window content filter: buffer the last N tokens (e.g., 50), run moderation on the buffer periodically (every 20 tokens). If problematic content is detected, abort the stream, send a moderation notice, and do NOT bill the user for the remaining tokens. OpenAI and Anthropic have built-in content filters, but Gemini's is less strict — always add your own layer. - **Sanitize SSE data to prevent injection.** If your SSE events contain user-generated content (e.g., streaming a response that includes the user's name), validate and escape the data before encoding as JSON. A malicious username like `{"token": "fake"}\n\n` could inject fake SSE events. Always use `JSON.stringify()` to serialize — never concatenate strings into the SSE format without proper serialization. - **Rate-limit streaming connections separately from regular API calls.** Streaming endpoints consume resources differently — one streaming connection might last 30 seconds and generate thousands of tokens. Regular API endpoints complete in milliseconds. Use separate rate limit buckets: 100 regular requests/minute + 10 streaming requests/minute for free users. The cost and resource difference justifies separate limits. --- ## Exercises ### Beginner 1. **Build a minimal SSE server:** Create a Next.js API route at `/api/stream/demo` that streams the numbers 1 through 10 with a 500ms delay between each. The client should display each number as it arrives. Use `ReadableStream` and the SSE wire format. Verify with `curl -N http://localhost:3000/api/stream/demo` — you should see each number appear incrementally. 2. **Consume SSE in the browser:** Write a React component that calls your demo endpoint and renders each number as it arrives. Implement the buffer pattern for partial SSE messages. Add a "Start" and "Cancel" button. The cancel button should use `AbortController` to stop the stream. Verify: click "Start", see numbers 1-5, click "Cancel", verify no more numbers appear. ### Intermediate 1. **Stream real LLM tokens from OpenAI:** Modify your SSE endpoint to call OpenAI's streaming API (`stream: true`). Pipe the tokens through your SSE stream to the frontend. Implement: (1) `AbortController` integration — aborting on the client cancels the OpenAI stream, (2) Token counting — track and display tokens generated so far, (3) Error handling — if OpenAI returns an error mid-stream, send an error event and close gracefully. Test by sending a prompt that generates ~200 tokens and clicking cancel halfway through. 2. **Multi-provider streaming with failover:** Extend the streaming endpoint to support multiple providers. The client sends `{ provider: 'openai' }`. If OpenAI fails (simulate by using an invalid API key), automatically fall back to Gemini and start streaming from Gemini instead. The client should receive tokens without knowing which provider is active. Log the failover event. Test by configuring OpenAI to fail and verifying the stream continues from Gemini seamlessly. ### Advanced 1. **Build a streaming proxy with token-level telemetry:** Create a streaming proxy layer that sits between your frontend and any LLM provider. The proxy: (1) Accepts SSE streams from OpenAI/Anthropic/Gemini, (2) Logs every token with microsecond timestamps (TTFT, inter-token intervals), (3) Detects anomalies: stalled streams (no tokens for 5+ seconds), rate drops (TPS falls below 50% of running average), (4) Can inject system messages mid-stream ("Switching to a faster model...") and hot-switch to a different provider without the client noticing. Write integration tests that simulate provider failures and verify the proxy maintains a seamless token flow. 2. **Implement a token budgeting streaming gateway:** Design a system where each user has a monthly token budget. The streaming gateway: (1) Checks remaining budget before starting the stream, (2) Estimates total cost based on prompt tokens + expected completion, (3) Streams tokens while decrementing the budget in real-time (Redis atomic counters), (4) If the budget is exhausted mid-stream, gracefully ends the response with "You've reached your monthly AI limit. [X] tokens remaining will be available on [date]." (5) Handles concurrent streams from the same user sharing the same budget — uses Redis transactions or Lua scripts for atomicity. Load-test with 1,000 concurrent users streaming simultaneously against a shared Redis budget counter. --- ## Interview questions ### Beginner 1. **Q: What is Server-Sent Events (SSE) and how does it differ from a regular HTTP response?** **A:** SSE is a protocol where the server keeps the HTTP connection open and pushes data to the client incrementally. Unlike a regular HTTP response (request → full response → connection close), SSE uses `Content-Type: text/event-stream` and the connection stays open. The server writes `data: \n\n` events, and the client receives them as they arrive. SSE is simpler than WebSocket — it's plain HTTP, works through all proxies, and the browser's `EventSource` API handles automatic reconnection. The key difference from regular HTTP: the response is never "complete" until the server explicitly closes the connection or sends a termination event. 2. **Q: Why is streaming important for AI chat applications?** **A:** LLMs generate tokens sequentially, one at a time. A 500-token response takes ~8 seconds to generate fully. Without streaming, the user stares at a spinner for 8 seconds, then sees the full response at once — this feels broken. With streaming, the user sees the first token in ~200ms and reads along as the rest generates. The total time is the same, but the perceived latency drops from 8 seconds to 200ms. This dramatically improves user engagement and reduces abandonment. Streaming is the difference between "this AI is slow" and "this AI is thinking and responding in real time." 3. **Q: What does `AbortController` do in the context of AI streaming?** **A:** `AbortController` provides a way to cancel an in-flight request. In AI streaming: (1) The client creates an `AbortController` and passes its `signal` to the `fetch()` call. (2) When the user clicks "stop generating" or navigates away, the client calls `controller.abort()`. (3) The `fetch` promise rejects with an `AbortError`. (4) On the server, `req.signal.aborted` becomes `true` — the streaming loop checks this between tokens and stops making expensive LLM calls. Without `AbortController`, the LLM keeps generating tokens that nobody will read, wasting money and compute. The backend MUST check `signal.aborted` between every token — not just at the start. ### Senior 1. **Q: Design a streaming system that can handle 10,000 concurrent AI chat sessions. What are the bottlenecks and how do you address them?** **A:** The bottlenecks, in order: (1) **LLM API rate limits:** OpenAI's GPT-4o has ~500 RPM for typical accounts. 10,000 concurrent streams would require 20+ API keys or enterprise tier. Solution: implement a provider pool with multiple API keys (rotating, load-balanced) and multi-provider routing (OpenAI + Anthropic + Gemini in parallel). (2) **Server connection limits:** 10,000 long-lived SSE connections consume file descriptors and memory. Each connection = ~2-4MB (buffers + stream object + TCP state) = 20-40GB RAM. Solution: horizontal scaling — 10 instances handling 1,000 streams each behind a load balancer. Use HTTP/2 multiplexing to reduce per-connection overhead. (3) **Network egress:** 10,000 streams at 30 tokens/second = 300,000 tokens/second. At ~4 bytes/token (SSE overhead), that's ~1.2MB/s egress. Most cloud providers handle this comfortably, but monitor bandwidth costs. (4) **Token cost:** 10,000 concurrent streams × 30 tokens/sec × $0.01/1K tokens = $0.0003 per second per stream × 10,000 = $3/second = $10,800/hour. This is the real bottleneck. Solution: aggressive caching, smart model routing (cheap models for simple queries), and per-user token budgets. (5) **State management:** Each stream needs conversation history. Storing 10,000 active conversations in Redis is fine (1-2GB). But persisting completed conversations to the database at 10,000/hour requires a write-optimized pipeline (batch inserts, async writes). 2. **Q: Your streaming AI responses are jittery — tokens arrive in bursts of 5-10, then a 500ms gap, then another burst. Users complain it's "distracting." How do you fix this?** **A:** This is called "token bursting" and it happens because LLMs generate tokens in parallel batches internally but the API emits them sequentially. The fix: implement a client-side token buffer with a fixed emission rate. (1) Receive tokens from the LLM as fast as they arrive. Push them into a client-side queue. (2) Create a "token metronome" — a timer that fires every 30-50ms. Each tick, pop one token from the queue and render it. (3) If the queue is empty when the timer fires, don't render anything — just wait for the next tick. (4) If the queue grows beyond a threshold (e.g., 50 tokens), increase the emission rate (fire every 20ms) to drain the backlog without overwhelming the reader. (5) For the "cursor blink" animation at the end: when the queue empties but the stream hasn't received `[DONE]`, show the blinking cursor. When `[DONE]` arrives, hide the cursor. This provides a smooth, typewriter-like reading experience regardless of the LLM's bursty generation pattern. ChatGPT and Claude both implement variants of this — that's why their output feels smooth. 3. **Q: How do you handle a scenario where the LLM starts streaming, but 5 seconds in, the output becomes clearly wrong or hallucinated? Can you "undo" tokens that have already been sent to the user?** **A:** You can't unsend tokens — once they're rendered in the user's browser, they're seen. But you can implement correction patterns: (1) **Real-time fact-checking:** Run a lightweight validation model (or regex checks for factual claims) on the streaming buffer every 10 tokens. If a hallucination is detected (e.g., the model claims "React 21 was released in 2025"), inject a correction event: `data: {"type": "correction", "original": "React 21...", "corrected": "React 19..."}`. The frontend strikes through the original text and replaces it. (2) **Streaming rewinds:** The LLM provider (specifically Anthropic's extended thinking) can internally "think again" and revise. If the API supports it, detect the revision event and send a `data: {"type": "rewind", "tokens": 15}` event — the frontend removes the last 15 tokens and continues from there. (3) **Post-generation correction:** If the full response is hallucinated but detected after completion, send a follow-up event that replaces the entire message. The frontend shows a "Correcting response..." transition. (4) **The practical answer:** For most products, the best approach is prevention, not correction. Use RAG (Chapter 4) to ground responses in real data. Validate critical claims (dates, version numbers, API names) against a knowledge base. Show confidence indicators during streaming. And accept that LLMs sometimes hallucinate — build your UX to gracefully handle correction rather than trying to achieve 100% accuracy in real-time. --- ## Summary Streaming transforms AI from a "submit and wait" experience into a real-time conversation. SSE (Server-Sent Events) is the protocol of choice — simple, HTTP-native, and sufficient for unidirectional token streaming. The backend wraps LLM provider streams in an `AsyncGenerator` that yields tokens uniformly regardless of provider. `AbortController` is not optional — it saves money and server resources when users navigate away. Backpressure, handled by the `ReadableStream` queuing strategy, prevents servers from overwhelming slow clients. The frontend consumes SSE with a buffer pattern (split on `\n\n`, handle partial messages) and renders tokens incrementally. The key performance metric is time-to-first-token (TTFT), not total generation time. Production systems need: stream health monitoring, connection limits, HTTP/2 for multiplexing, per-user concurrent stream caps, and content moderation on the streaming buffer. The products that do streaming well (ChatGPT, Claude, Cursor) all share the same secret: a blinking cursor at the end of streaming text that signals "more is coming." It's the smallest detail with the biggest UX impact. --- ## Quick recall - SSE = `Content-Type: text/event-stream` + `data: \n\n` over a long-lived HTTP connection. Simpler than WebSocket, works everywhere. - Wrap LLM streams in `AsyncGenerator` — each provider's streaming event format is different, but the generator yields plain text tokens uniformly. - `AbortController` on both client AND server. Client aborts → `req.signal.aborted` → server stops LLM stream → stop spending money. - Backpressure via `ReadableStream`'s `highWaterMark`. Set to 10 for chat UIs, 50-100 for audio pipelines. Lets the stream naturally slow to match consumer speed. - Frontend SSE parsing: accumulate chunks → split on `\n\n` → keep partial lines in buffer. Without this, tokens split across chunk boundaries are lost. - Time-to-first-token (TTFT) < 500ms is the target. Users perceive latency based on first token, not last. Precompute context (embeddings, DB queries) before starting the stream. - Token buffer for smooth rendering: collect tokens for 30-50ms, flush when buffer reaches 5 tokens or timer fires. Eliminates jittery burst rendering. - Connection limits: 500-1000 concurrent streams per server (2-4MB each). Return 503 when at capacity. Use HTTP/2 for multiplexing. - Per-user concurrent stream cap: 3-5 streams max. Prevents accidental (or malicious) resource exhaustion. - Blinking cursor `▊` at end of streaming text = the single most important UX detail. Signals "more is coming" during natural pauses in generation. --- ## Quiz 1. **What protocol does AI streaming typically use?** A) WebSocket B) **SSE (Server-Sent Events)** C) gRPC streaming D) HTTP/1.1 long-polling 2. **What does `AbortController` prevent in an AI streaming context?** A) The LLM from generating incorrect answers **B) Wasting money on tokens generated after the user navigates away or clicks "stop"** C) Rate limiting on the API D) The server from crashing 3. **What's the correct SSE wire format for sending a token "Hello"?** A) `token: Hello` B) `event: token\ndata: Hello` **C) `data: {"token": "Hello"}\n\n`** D) `Hello` 4. **Why do you need a buffer when parsing SSE on the frontend?** A) To compress the data **B) Network chunks don't align with SSE message boundaries — a message might be split across two chunks, and without buffering you lose the first/last part** C) To slow down rendering D) To encrypt the tokens 5. **What is TTFT and why does it matter?** **A) Time To First Token — the delay before the user sees the first word. Users perceive latency based on this, not total generation time** B) Total Token Flow Time — the full generation duration C) Token Transfer Fault Tolerance — error recovery D) Time To Finish Training — model warm-up time 6. **When should you use WebSocket instead of SSE for AI features?** A) Always — WebSocket is newer and better B) When you need the server to push data to the client **C) When communication is bidirectional over a long session — SSE is server→client only** D) When you're using HTTP/1.1 7. **What Nginx setting prevents SSE streaming from being accidentally buffered?** A) `gzip on;` **B) `proxy_buffering off;` in the location block for streaming endpoints** C) `server_tokens off;` D) `worker_connections 4096;`

Chapter 3 — Streaming AI Responses

Part of L7