Chapter 3 — Streaming AI Responses

The why

Users will wait 8 seconds for a page to load. They will not wait 8 seconds staring at a spinner while an LLM "thinks." Streaming cuts perceived latency from seconds to milliseconds — the first token appears almost instantly, and the user reads along as the model generates. This isn't a UX nice-to-have; it's the difference between a product that feels alive and one that feels broken.

Core concepts

Server-Sent Events (SSE): the protocol that powers AI streaming

SSE is a simple HTTP-based protocol for server-to-client streaming. Unlike WebSockets (bidirectional), SSE is unidirectional — the server pushes data to the client over a single, long-lived HTTP connection. This simplicity is exactly what makes it perfect for AI streaming: the client sends a request (the prompt), the server streams the response (generated tokens), the connection closes when generation finishes.

// Minimal SSE endpoint in Next.js App Router
// app/api/chat/stream/route.ts
export async function POST(req: Request) {
  const { messages } = await req.json();
 
  // Create a ReadableStream that the client will consume via EventSource
  const stream = new

Parsing LLM streaming responses: three providers, three event shapes

Remember the provider abstraction from Chapter 1? The streaming layer is where the differences between providers become most visible. OpenAI, Anthropic, and Gemini each emit completely different streaming event types:

// src/ai/streaming/openai-stream.ts
import OpenAI from 'openai';
 
export async function* streamOpenAI(messages: ChatMessage[]): AsyncGenerator<string> {
  const openai = new OpenAI({ apiKey: process

The unified streaming adapter: one AsyncGenerator to rule them all

Just like Chapter 1's LLMProvider interface unified chat() calls, we need a unified streaming interface:

// src/ai/streaming/unified-stream.ts
import { ChatMessage } from '../types';
 
export interface StreamingProvider {
  stream(messages: ChatMessage[], systemPrompt?: string): AsyncGenerator<StreamEvent>;
}
 
export

AbortController: respecting user attention

The single most important streaming feature is the ability to cancel mid-generation. When a user navigates away, clicks "stop generating", or closes the tab, you need to abort the upstream LLM request immediately. Otherwise, you're paying for tokens nobody will ever read.

// src/ai/streaming/abortable-stream.ts
export async function* abortableStream(
  provider: 'openai' | 'anthropic' | 'gemini',
  messages: ChatMessage[],
  signal: AbortSignal
): AsyncGenerator<StreamEvent>

Backpressure: when the client can't keep up

SSE over HTTP handles backpressure naturally — the TCP connection's flow control prevents the server from overwhelming the client's receive buffer. But there's a subtlety with AI streaming: the LLM is generating tokens at its own pace (maybe 50 tokens/second), and your server is forwarding them to the client. If the client's connection is slow (mobile network, congested WiFi), tokens queue up in the server's send buffer.

The ReadableStream API handles this with its internal queuing strategy:

const stream = new ReadableStream({
  async start(controller) {
    for await (const token of tokens) {
      // controller.enqueue returns when the token is in the stream's internal queue
      // If the queue is full (backpressure), it waits until there's room
      // This naturally slows the producer to match the consumer's speed
      controller

Frontend token-by-token rendering

The backend streams tokens. The frontend renders them. Here's how the client consumes an SSE stream with proper cleanup, reconnection handling, and smooth rendering:

// client/useStreamingChat.ts
import { useState, useRef, useCallback } from 'react';
 
export function useStreamingChat() {
  const [response, setResponse] = useState('');
  const [isStreaming,

SSE vs WebSocket for AI streaming

Developers often ask: "Why SSE and not WebSocket for AI streaming?" Here's the decision framework:

Real-world example: How ChatGPT's streaming UX became the industry standard

ChatGPT's streaming implementation is the reference architecture that every AI product now follows. When you type a message into ChatGPT, here's what happens:

The backend pipeline: (1) Your message arrives at an API gateway. (2) The gateway routes to a model inference server (OpenAI's custom infrastructure, not the public API). (3) The model generates tokens one at a time. (4) Each token is immediately pushed through the SSE connection. (5) The frontend renders each token as it arrives with a subtle "cursor blink" animation at the end of the text — this animation is the critical UX detail that signals "more is coming."

The frontend rendering strategy: ChatGPT doesn't re-render the entire message on each token (that would be expensive for long responses). Instead, it maintains a text buffer and appends to the DOM incrementally. The "thinking" dots appear while the first token is being generated. The message container auto-scrolls as new tokens appear. The "stop generating" button is always visible during streaming — clicking it sends an abort signal to the backend.

The key UX insight that most clones miss: The cursor animation. ChatGPT shows a blinking ▊ at the end of the streaming text. This tiny detail tells the user "the model is still working, more text is coming." Without it, users hit a natural pause in the response (the model generated a period and is thinking about the next sentence) and assume the generation is complete. They start reading, then new text appears and disorients them. The blinking cursor prevents this confusion.

Cursor implements streaming similarly but adds code-block awareness: when streaming code (inside blocks), it renders syntax-highlighted tokens as they arrive. The code block's background appears immediately when ` ` is detected, and syntax highlighting updates incrementally. This attention to detail — making streaming feel like magic, not like a loading bar — is what separates polished AI products from demos.

Common mistakes

Production notes

• Monitor stream health metrics separately from application metrics. Track: stream start rate, stream completion rate (completed without abort), median time-to-first-token (TTFT), median tokens-per-second during streaming, abort rate (user-initiated vs. error), and stream error rate by provider. A spike in TTFT means the LLM is slow — switch providers. A spike in abort rate means users aren't finding answers fast enough — optimize prompts or reduce response length.
• Implement a token buffer for smoother rendering. Tokens arrive from the LLM at irregular intervals — sometimes 3 tokens in 50ms, then nothing for 200ms. Rendering each token immediately creates a jittery, distracting UX. Buffer tokens for 30-50ms: collect incoming tokens, then flush the buffer either when it reaches 5 tokens or 50ms elapses. This smooths the rendering without introducing perceptible delay.
• Set connection limits on streaming endpoints. Streaming connections are long-lived and consume server resources (one TCP connection, one stream object, memory for buffering). If you have 1,000 concurrent streaming connections, that's 1,000 open file descriptors. Configure your server's max connections based on available memory: roughly 2-4MB per streaming connection (including buffers). At 4GB of available RAM, cap at 500-1000 concurrent streams. Implement a queue or 503 "Service Unavailable" when at capacity.
• Use HTTP/2 for multiplexing multiple streams. HTTP/1.1 limits browsers to 6 concurrent connections per domain. If your app has 6 streaming AI responses plus API calls, you hit the limit. HTTP/2 multiplexes many streams over a single TCP connection — no more head-of-line blocking. Most hosting platforms (Vercel, Cloudflare) enable HTTP/2 by default for HTTPS traffic.
• Log token-level telemetry during streaming. For each streamed response, log: stream_id, provider, model, ttft_ms (time to first token), total_tokens, duration_ms, , (boolean), . This data reveals: which provider is fastest for your use case, whether your prompts generate overly long responses, and which users consistently abort mid-generation (they're not getting what they need).

Performance notes

• Time-to-first-token (TTFT) is the most important streaming metric. Users perceive latency based on when they see the first token, not the last. Target TTFT < 500ms. If TTFT > 1 second, users get restless. Factors affecting TTFT: prompt length (longer prompts = more prefill computation), model size (larger models start slower), provider load (shared infrastructure can queue). Cache system prompts to reduce prefill time. Use smaller/faster models for simple queries where TTFT matters more than answer quality.
• Tokens-per-second (TPS) determines reading comfort. Users read at ~5-8 tokens/second (English). If your stream delivers 50 TPS, tokens pile up faster than anyone can read — the scrollbar races ahead and users feel overwhelmed. If it delivers 5 TPS, it's comfortable but borderline slow. Target 20-30 TPS: fast enough to feel snappy, slow enough to read. Some providers let you control this (Anthropic doesn't, OpenAI doesn't directly, but you can add artificial delays between tokens if the LLM is too fast).
• Gzip compression on SSE streams is usually counterproductive. SSE events are tiny (10-50 bytes each). Gzip works best on larger payloads — for tiny chunks, the compression overhead often exceeds any savings. Plus, Gzip adds buffering latency (the compressor waits for more data). Unless your tokens are very large (streaming paragraphs, not tokens), disable compression for SSE endpoints.
• Connection reuse saves TLS handshake overhead. If your frontend opens a new HTTP connection for every streaming request, you pay the TLS handshake cost (50-200ms) on every message. Configure HTTP/2 or HTTP/1.1 keep-alive so subsequent streaming requests reuse the same TCP+TLS connection. This cuts 50-200ms from every TTFT after the first message in a session.
• Prefetch embeddings and context BEFORE starting the stream. If your streaming endpoint also does vector search (fetching relevant context for the prompt), do that BEFORE creating the stream. The user waits for the first token — don't make them wait for your database query PLUS the LLM. Precompute everything the LLM needs, then start the stream with all context ready.

Security notes

• Validate Origin and Host headers on streaming endpoints. SSE endpoints are vulnerable to Cross-Site WebSocket Hijacking (CSWSH) — a malicious site can open an EventSource to your streaming endpoint and receive tokens intended for another user if your auth is cookie-based. Set Access-Control-Allow-Origin to your specific domain (not *), and validate the Origin header server-side. For sensitive AI responses, use token-based auth (Authorization header) instead of cookies.
• Implement per-user concurrent stream limits. A single user shouldn't be able to open 50 simultaneous streaming connections — each one costs you money (LLM tokens) and server resources. Limit to 3-5 concurrent streams per user. Return 429 if exceeded. This also prevents a buggy client from accidentally spamming streams.
• Stream content through a moderation filter. Offensive or dangerous LLM outputs stream token-by-token — the first few tokens might look innocent, and the problematic content appears later. Implement a sliding window content filter: buffer the last N tokens (e.g., 50), run moderation on the buffer periodically (every 20 tokens). If problematic content is detected, abort the stream, send a moderation notice, and do NOT bill the user for the remaining tokens. OpenAI and Anthropic have built-in content filters, but Gemini's is less strict — always add your own layer.
• Sanitize SSE data to prevent injection. If your SSE events contain user-generated content (e.g., streaming a response that includes the user's name), validate and escape the data before encoding as JSON. A malicious username like {"token": "fake"}\n\n could inject fake SSE events. Always use JSON.stringify() to serialize — never concatenate strings into the SSE format without proper serialization.
• Rate-limit streaming connections separately from regular API calls. Streaming endpoints consume resources differently — one streaming connection might last 30 seconds and generate thousands of tokens. Regular API endpoints complete in milliseconds. Use separate rate limit buckets: 100 regular requests/minute + 10 streaming requests/minute for free users. The cost and resource difference justifies separate limits.

Exercises

Beginner

1. Build a minimal SSE server: Create a Next.js API route at /api/stream/demo that streams the numbers 1 through 10 with a 500ms delay between each. The client should display each number as it arrives. Use ReadableStream and the SSE wire format. Verify with curl -N http://localhost:3000/api/stream/demo — you should see each number appear incrementally.

2. Consume SSE in the browser: Write a React component that calls your demo endpoint and renders each number as it arrives. Implement the buffer pattern for partial SSE messages. Add a "Start" and "Cancel" button. The cancel button should use AbortController to stop the stream. Verify: click "Start", see numbers 1-5, click "Cancel", verify no more numbers appear.

Intermediate

1. Stream real LLM tokens from OpenAI: Modify your SSE endpoint to call OpenAI's streaming API (stream: true). Pipe the tokens through your SSE stream to the frontend. Implement: (1) AbortController integration — aborting on the client cancels the OpenAI stream, (2) Token counting — track and display tokens generated so far, (3) Error handling — if OpenAI returns an error mid-stream, send an error event and close gracefully. Test by sending a prompt that generates ~200 tokens and clicking cancel halfway through.

2. Multi-provider streaming with failover: Extend the streaming endpoint to support multiple providers. The client sends { provider: 'openai' }. If OpenAI fails (simulate by using an invalid API key), automatically fall back to Gemini and start streaming from Gemini instead. The client should receive tokens without knowing which provider is active. Log the failover event. Test by configuring OpenAI to fail and verifying the stream continues from Gemini seamlessly.

Advanced

1. Build a streaming proxy with token-level telemetry: Create a streaming proxy layer that sits between your frontend and any LLM provider. The proxy: (1) Accepts SSE streams from OpenAI/Anthropic/Gemini, (2) Logs every token with microsecond timestamps (TTFT, inter-token intervals), (3) Detects anomalies: stalled streams (no tokens for 5+ seconds), rate drops (TPS falls below 50% of running average), (4) Can inject system messages mid-stream ("Switching to a faster model...") and hot-switch to a different provider without the client noticing. Write integration tests that simulate provider failures and verify the proxy maintains a seamless token flow.

2. Implement a token budgeting streaming gateway: Design a system where each user has a monthly token budget. The streaming gateway: (1) Checks remaining budget before starting the stream, (2) Estimates total cost based on prompt tokens + expected completion, (3) Streams tokens while decrementing the budget in real-time (Redis atomic counters), (4) If the budget is exhausted mid-stream, gracefully ends the response with "You've reached your monthly AI limit. [X] tokens remaining will be available on [date]." (5) Handles concurrent streams from the same user sharing the same budget — uses Redis transactions or Lua scripts for atomicity. Load-test with 1,000 concurrent users streaming simultaneously against a shared Redis budget counter.

Interview questions

Beginner

1. Q: What is Server-Sent Events (SSE) and how does it differ from a regular HTTP response? A: SSE is a protocol where the server keeps the HTTP connection open and pushes data to the client incrementally. Unlike a regular HTTP response (request → full response → connection close), SSE uses Content-Type: text/event-stream and the connection stays open. The server writes data: <content>\n\n events, and the client receives them as they arrive. SSE is simpler than WebSocket — it's plain HTTP, works through all proxies, and the browser's EventSource API handles automatic reconnection. The key difference from regular HTTP: the response is never "complete" until the server explicitly closes the connection or sends a termination event.

2. Q: Why is streaming important for AI chat applications? A: LLMs generate tokens sequentially, one at a time. A 500-token response takes ~8 seconds to generate fully. Without streaming, the user stares at a spinner for 8 seconds, then sees the full response at once — this feels broken. With streaming, the user sees the first token in ~200ms and reads along as the rest generates. The total time is the same, but the perceived latency drops from 8 seconds to 200ms. This dramatically improves user engagement and reduces abandonment. Streaming is the difference between "this AI is slow" and "this AI is thinking and responding in real time."

3. Q: What does AbortController do in the context of AI streaming? A: AbortController provides a way to cancel an in-flight request. In AI streaming: (1) The client creates an AbortController and passes its signal to the fetch() call. (2) When the user clicks "stop generating" or navigates away, the client calls . (3) The rejects with an . (4) On the server, becomes — the streaming loop checks this between tokens and stops making expensive LLM calls. Without , the LLM keeps generating tokens that nobody will read, wasting money and compute. The backend MUST check between every token — not just at the start.

Senior

1. Q: Design a streaming system that can handle 10,000 concurrent AI chat sessions. What are the bottlenecks and how do you address them? A: The bottlenecks, in order: (1) LLM API rate limits: OpenAI's GPT-4o has ~500 RPM for typical accounts. 10,000 concurrent streams would require 20+ API keys or enterprise tier. Solution: implement a provider pool with multiple API keys (rotating, load-balanced) and multi-provider routing (OpenAI + Anthropic + Gemini in parallel). (2) Server connection limits: 10,000 long-lived SSE connections consume file descriptors and memory. Each connection = ~2-4MB (buffers + stream object + TCP state) = 20-40GB RAM. Solution: horizontal scaling — 10 instances handling 1,000 streams each behind a load balancer. Use HTTP/2 multiplexing to reduce per-connection overhead. (3) Network egress: 10,000 streams at 30 tokens/second = 300,000 tokens/second. At ~4 bytes/token (SSE overhead), that's ~1.2MB/s egress. Most cloud providers handle this comfortably, but monitor bandwidth costs. (4) Token cost: 10,000 concurrent streams × 30 tokens/sec × $0.01/1K tokens = $0.0003 per second per stream × 10,000 = $3/second = $10,800/hour. This is the real bottleneck. Solution: aggressive caching, smart model routing (cheap models for simple queries), and per-user token budgets. (5) State management: Each stream needs conversation history. Storing 10,000 active conversations in Redis is fine (1-2GB). But persisting completed conversations to the database at 10,000/hour requires a write-optimized pipeline (batch inserts, async writes).

2. Q: Your streaming AI responses are jittery — tokens arrive in bursts of 5-10, then a 500ms gap, then another burst. Users complain it's "distracting." How do you fix this? A: This is called "token bursting" and it happens because LLMs generate tokens in parallel batches internally but the API emits them sequentially. The fix: implement a client-side token buffer with a fixed emission rate. (1) Receive tokens from the LLM as fast as they arrive. Push them into a client-side queue. (2) Create a "token metronome" — a timer that fires every 30-50ms. Each tick, pop one token from the queue and render it. (3) If the queue is empty when the timer fires, don't render anything — just wait for the next tick. (4) If the queue grows beyond a threshold (e.g., 50 tokens), increase the emission rate (fire every 20ms) to drain the backlog without overwhelming the reader. (5) For the "cursor blink" animation at the end: when the queue empties but the stream hasn't received [DONE], show the blinking cursor. When [DONE] arrives, hide the cursor. This provides a smooth, typewriter-like reading experience regardless of the LLM's bursty generation pattern. ChatGPT and Claude both implement variants of this — that's why their output feels smooth.

Summary

Streaming transforms AI from a "submit and wait" experience into a real-time conversation. SSE (Server-Sent Events) is the protocol of choice — simple, HTTP-native, and sufficient for unidirectional token streaming. The backend wraps LLM provider streams in an AsyncGenerator that yields tokens uniformly regardless of provider. AbortController is not optional — it saves money and server resources when users navigate away. Backpressure, handled by the ReadableStream queuing strategy, prevents servers from overwhelming slow clients. The frontend consumes SSE with a buffer pattern (split on \n\n, handle partial messages) and renders tokens incrementally. The key performance metric is time-to-first-token (TTFT), not total generation time. Production systems need: stream health monitoring, connection limits, HTTP/2 for multiplexing, per-user concurrent stream caps, and content moderation on the streaming buffer. The products that do streaming well (ChatGPT, Claude, Cursor) all share the same secret: a blinking cursor at the end of streaming text that signals "more is coming." It's the smallest detail with the biggest UX impact.

Quick recall

• SSE = Content-Type: text/event-stream + data: <json>\n\n over a long-lived HTTP connection. Simpler than WebSocket, works everywhere.
• Wrap LLM streams in AsyncGenerator<string> — each provider's streaming event format is different, but the generator yields plain text tokens uniformly.
• AbortController on both client AND server. Client aborts → req.signal.aborted → server stops LLM stream → stop spending money.
• Backpressure via ReadableStream's highWaterMark. Set to 10 for chat UIs, 50-100 for audio pipelines. Lets the stream naturally slow to match consumer speed.
• Frontend SSE parsing: accumulate chunks → split on \n\n → keep partial lines in buffer. Without this, tokens split across chunk boundaries are lost.
• Time-to-first-token (TTFT) < 500ms is the target. Users perceive latency based on first token, not last. Precompute context (embeddings, DB queries) before starting the stream.
• Token buffer for smooth rendering: collect tokens for 30-50ms, flush when buffer reaches 5 tokens or timer fires. Eliminates jittery burst rendering.
• Connection limits: 500-1000 concurrent streams per server (2-4MB each). Return 503 when at capacity. Use HTTP/2 for multiplexing.
• Per-user concurrent stream cap: 3-5 streams max. Prevents accidental (or malicious) resource exhaustion.
• Blinking cursor ▊ at end of streaming text = the single most important UX detail. Signals "more is coming" during natural pauses in generation.

Quiz

1. What protocol does AI streaming typically use? A) WebSocket B) SSE (Server-Sent Events) C) gRPC streaming D) HTTP/1.1 long-polling

2. What does AbortController prevent in an AI streaming context? A) The LLM from generating incorrect answers B) Wasting money on tokens generated after the user navigates away or clicks "stop" C) Rate limiting on the API D) The server from crashing

3. What's the correct SSE wire format for sending a token "Hello"? A) token: Hello B) event: token\ndata: Hello C) data: {"token": "Hello"}\n\n D) <sse><token>Hello</token></sse>

4. Why do you need a buffer when parsing SSE on the frontend? A) To compress the data B) Network chunks don't align with SSE message boundaries — a message might be split across two chunks, and without buffering you lose the first/last part C) To slow down rendering D) To encrypt the tokens

5. What is TTFT and why does it matter? A) Time To First Token — the delay before the user sees the first word. Users perceive latency based on this, not total generation time B) Total Token Flow Time — the full generation duration C) Token Transfer Fault Tolerance — error recovery D) Time To Finish Training — model warm-up time

6. When should you use WebSocket instead of SSE for AI features? A) Always — WebSocket is newer and better B) When you need the server to push data to the client C) When communication is bidirectional over a long session — SSE is server→client only D) When you're using HTTP/1.1

7. What Nginx setting prevents SSE streaming from being accidentally buffered? A) gzip on; B) proxy_buffering off; in the location block for streaming endpoints C) D)