# Chapter 2 — Embeddings & Vector Databases By the end of this chapter you will: - Generate and store embeddings - Run a cosine-similarity search in pgvector - Choose between pgvector, Pinecone, Qdrant, Weaviate - Index vectors at scale with HNSW / IVFFlat --- ## The why > **Full-text search finds the word "cat" in your documents. Semantic search finds "feline", "kitten", "tabby", and "the animal that knocked over my coffee" — even when none of those words appear in the query. Embeddings are the technology that makes this possible, and vector databases make it fast enough for production.** You walk into a massive library — 10 million books, no digital catalog. You ask the librarian: "I'm looking for books about how to train a dog to stop barking at the mailman." A traditional librarian (full-text search) flips through an index card, finds every book containing the exact words "dog", "train", "barking", and "mailman", and hands you a stack. Most of them are novels where a mailman owns a barking dog. Not useful. An embedding-powered librarian has done something different. She spent her career reading every single book and placing them on shelves by *meaning*, not by title. Books about dog training are near books about pet behavior. Books about postal workers are on a different floor entirely. When you ask your question, she doesn't search for words — she walks to the shelf where "canine behavior modification" books live, then drifts slightly toward the "home security + dogs" section. The first 10 books she hands you are exactly what you need. The "shelf position" in this analogy is an embedding — a list of 1,536 numbers that represents what a piece of text is *about*. Books near each other on the shelf (vectors with high cosine similarity) share meaning. The librarian's mental map of the library is a vector database. Your job as a backend engineer is to build that librarian. --- ## Core concepts ### What embeddings actually are: from text to numbers An embedding is a dense vector — a fixed-length array of floating-point numbers — that represents the semantic meaning of a piece of text. OpenAI's `text-embedding-3-small` produces 1,536-dimensional vectors. Every word, sentence, paragraph, or document you feed it gets mapped to a point in 1,536-dimensional space. The magic: semantically similar texts are close together in that space. Here's the concrete reality. These two sentences: ``` "The cat sat on the mat." "A feline rested on the rug." ``` Produce vectors that are very close together (cosine similarity > 0.9). These two: ``` "The cat sat on the mat." "Quarterly earnings exceeded analyst expectations by 14%." ``` Produce vectors that are far apart (cosine similarity < 0.2). The model learned these relationships by reading billions of text pairs during training. ```typescript // Generate embeddings with OpenAI import OpenAI from 'openai'; const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY }); async function embed(text: string): Promise { const response = await openai.embeddings.create({ model: 'text-embedding-3-small', // 1536 dimensions, $0.02/1M tokens input: text }); return response.data[0].embedding; // Returns: [0.023, -0.451, 0.892, ..., 0.134] (1536 numbers) } const catVector = await embed('The cat sat on the mat.'); const financeVector = await embed('Quarterly earnings exceeded expectations.'); // These two vectors will have low cosine similarity — different topics ``` You can't visualize 1,536 dimensions — humans max out at 3. But you can think of it as 1,536 different "aspects" of meaning. Dimension 1 might loosely correlate with "animate vs. inanimate". Dimension 342 might correlate with "formal vs. casual tone". Dimension 891 might capture "technical vs. non-technical". The model discovers these dimensions automatically during training — we don't label them. What matters is that two texts with similar meanings have similar values across most dimensions, so their vectors point in roughly the same direction. ### Cosine similarity: measuring "closeness" between vectors The standard distance metric for embeddings is cosine similarity — it measures the angle between two vectors, not the straight-line distance. This matters because vector magnitude (length) can vary with text length, but direction captures meaning more reliably. ```typescript function cosineSimilarity(a: number[], b: number[]): number { if (a.length !== b.length) { throw new Error('Vectors must have the same dimension'); } let dotProduct = 0; let magnitudeA = 0; let magnitudeB = 0; for (let i = 0; i < a.length; i++) { dotProduct += a[i] * b[i]; magnitudeA += a[i] * a[i]; magnitudeB += b[i] * b[i]; } magnitudeA = Math.sqrt(magnitudeA); magnitudeB = Math.sqrt(magnitudeB); if (magnitudeA === 0 || magnitudeB === 0) return 0; return dotProduct / (magnitudeA * magnitudeB); // Range: -1 (opposite meaning) to 1 (identical meaning). 0 = unrelated. } // Example: comparing our vectors const similarity = cosineSimilarity(catVector, financeVector); console.log(`Similarity: ${similarity.toFixed(4)}`); // ~0.15 — very different topics ``` Euclidean distance (straight-line distance between points) is sensitive to vector magnitude. A long document might have a larger-magnitude embedding than a short one on the same topic, making Euclidean distance misleadingly large. Cosine similarity ignores magnitude and only considers direction — it answers "are these about the same thing?" regardless of how verbose each text is. For embeddings, always use cosine similarity (or its equivalent, inner product on normalized vectors). ### pgvector: Vector search inside PostgreSQL pgvector is a PostgreSQL extension that adds a `vector` data type and indexing. The killer feature: your vectors live in the same database as your application data. No separate service, no data sync, no eventual consistency headaches. You can join vector search results with regular SQL tables in a single query. ```sql -- Enable the extension (run once per database) CREATE EXTENSION IF NOT EXISTS vector; -- Create a table with a vector column CREATE TABLE document_chunks ( id SERIAL PRIMARY KEY, document_id INTEGER NOT NULL, chunk_text TEXT NOT NULL, chunk_index INTEGER NOT NULL, embedding VECTOR(1536), -- matches text-embedding-3-small dimensions metadata JSONB DEFAULT '{}', created_at TIMESTAMPTZ DEFAULT NOW() ); -- Create an HNSW index for fast approximate nearest neighbor search CREATE INDEX ON document_chunks USING hnsw (embedding vector_cosine_ops); ``` Inserting embeddings from your Node.js application: ```typescript import { createClient } from '@supabase/supabase-js'; const supabase = createClient( process.env.SUPABASE_URL!, process.env.SUPABASE_SERVICE_ROLE_KEY! ); async function indexDocument(documentId: number, text: string) { const chunks = chunkText(text, 500); // Split into ~500-token chunks const rows = []; for (let i = 0; i < chunks.length; i++) { const embedding = await embed(chunks[i]); rows.push({ document_id: documentId, chunk_text: chunks[i], chunk_index: i, embedding: embedding // pgvector accepts number[] directly }); } // Batch insert — way faster than individual inserts const { error } = await supabase.from('document_chunks').insert(rows); if (error) throw error; } ``` Semantic search query — the entire point of all this setup: ```typescript async function semanticSearch(query: string, limit: number = 10) { // 1. Embed the user's query const queryEmbedding = await embed(query); // 2. Find the most similar chunks using cosine distance const { data, error } = await supabase.rpc('match_document_chunks', { query_embedding: queryEmbedding, match_threshold: 0.7, // Only return results with similarity > 0.7 match_count: limit }); if (error) throw error; return data; } // The PostgreSQL function (create this in Supabase SQL editor): // CREATE OR REPLACE FUNCTION match_document_chunks( // query_embedding VECTOR(1536), // match_threshold FLOAT, // match_count INT // ) RETURNS TABLE ( // id INT, // document_id INT, // chunk_text TEXT, // similarity FLOAT // ) LANGUAGE plpgsql AS $$ // BEGIN // RETURN QUERY // SELECT // dc.id, // dc.document_id, // dc.chunk_text, // 1 - (dc.embedding <=> query_embedding) AS similarity // FROM document_chunks dc // WHERE 1 - (dc.embedding <=> query_embedding) > match_threshold // ORDER BY dc.embedding <=> query_embedding // LIMIT match_count; // END; // $$; ``` pgvector uses `<=>` for cosine distance (not similarity). Distance = 0 means identical vectors. Distance = 1 means orthogonal (unrelated). Distance = 2 means opposite. To convert to similarity: `1 - distance`. The `vector_cosine_ops` index operator class tells the HNSW index to optimize for cosine distance queries. ### IVFFlat vs HNSW: choosing the right index You have two main indexing options in pgvector, and the choice matters at scale: | Index | How it works | Build time | Query speed | Memory | Best for | |-------|-------------|-----------|-------------|--------|----------| | **IVFFlat** | Clusters vectors into `k` lists. At query time, searches only the nearest cluster(s) | Fast | Moderate (scans ~5-10% of data) | Lower | Datasets under 500K vectors; frequent inserts | | **HNSW** | Builds a multi-layer graph. At query time, greedily traverses from coarse to fine layers | Slow (but build-once) | Very fast (scans <1% of data) | Higher | Datasets over 1M vectors; read-heavy workloads | ```sql -- IVFFlat: good for moderate datasets with frequent inserts CREATE INDEX ON document_chunks USING ivfflat (embedding vector_cosine_ops) WITH (lists = 100); -- Rule of thumb: lists = rows / 1000 (for 100K rows, use 100 lists) -- HNSW: better for large, read-heavy datasets CREATE INDEX ON document_chunks USING hnsw (embedding vector_cosine_ops) WITH (m = 16, ef_construction = 200); -- m: max connections per node (higher = more accurate, more memory) -- ef_construction: search depth during build (higher = better graph, slower build) ``` IVFFlat requires a training step — it needs to see representative data to build clusters. If you create the index on an empty table and then insert data, the index is useless (all vectors land in one cluster). Always insert at least 10% of your expected data before creating an IVFFlat index, or use HNSW which doesn't need training. ### Choosing a vector database: pgvector vs Pinecone vs the rest You don't always need to reach for Pinecone. Here's a practical decision framework: | Database | Type | Best for | Cost | Operational complexity | |----------|------|----------|------|----------------------| | **pgvector** | PostgreSQL extension | Teams already on Postgres; <10M vectors; need joins with relational data | $0 (open source) | Low (it's just Postgres) | | **Pinecone** | Managed vector DB | >10M vectors; need sub-10ms queries at scale; don't want to manage infra | ~$70/month for 1M vectors | Zero (fully managed) | | **Qdrant** | Self-hosted or cloud | High-dimensional vectors; filtered search (combine vector + metadata filters) | $0 (self-hosted) or $25/month (cloud) | Moderate | | **Weaviate** | Self-hosted or cloud | Hybrid search (vector + keyword); built-in vectorization | $0 (self-hosted) or $25/month (cloud) | Moderate | | **Chroma** | Embedded / local | Prototyping; local development; single-user apps | $0 | Minimal (pip install) | For 90% of applications, pgvector is the right choice. You already have Postgres. Your vectors can participate in SQL joins with user data, permissions, and metadata. You get ACID guarantees. Your vectors are backed up with the rest of your database. Only migrate to a dedicated vector DB when you hit pgvector's scaling limits (~10-50M vectors depending on hardware) or need specialized features like multi-tenant isolation or GPU-accelerated indexing. ### Hybrid search: when vector search alone isn't enough Pure semantic search has a blind spot: exact keyword matching. A user searching for "RFC 7231" should find the actual RFC document, not a blog post about HTTP semantics that happens to be semantically similar. Hybrid search combines vector similarity (semantic) with keyword relevance (lexical) for the best of both worlds: ```sql -- Hybrid search: combine vector similarity + full-text keyword relevance CREATE OR REPLACE FUNCTION hybrid_search( query_text TEXT, query_embedding VECTOR(1536), match_count INT DEFAULT 10, vector_weight FLOAT DEFAULT 0.7, -- How much to trust vector vs keyword keyword_weight FLOAT DEFAULT 0.3 ) RETURNS TABLE ( id INT, chunk_text TEXT, vector_similarity FLOAT, keyword_score FLOAT, combined_score FLOAT ) LANGUAGE plpgsql AS $$ BEGIN RETURN QUERY SELECT dc.id, dc.chunk_text, 1 - (dc.embedding <=> query_embedding) AS vector_similarity, COALESCE(ts_rank(to_tsvector('english', dc.chunk_text), plainto_tsquery('english', query_text)), 0) AS keyword_score, (vector_weight * (1 - (dc.embedding <=> query_embedding))) + (keyword_weight * COALESCE(ts_rank(to_tsvector('english', dc.chunk_text), plainto_tsquery('english', query_text)), 0)) AS combined_score FROM document_chunks dc WHERE 1 - (dc.embedding <=> query_embedding) > 0.5 -- Vector must be somewhat relevant ORDER BY combined_score DESC LIMIT match_count; END; $$; ``` Use hybrid search when users search for: product codes ("SKU-44921"), error messages ("ECONNREFUSED 127.0.0.1:5432"), proper names ("Kubernetes HorizontalPodAutoscaler"), or version numbers ("React 19.1"). These queries benefit from exact matching. Pure vector search works best for natural language questions ("how do I fix a database connection error?"). --- ## Real-world example: How Perplexity built their answer engine on vector search Perplexity AI doesn't just send your question to an LLM. Their pipeline: (1) Your question is embedded and used to search a real-time web index (billions of pages). (2) The top 10-20 most relevant pages are retrieved via semantic + keyword hybrid search. (3) Those pages are re-ranked using a cross-encoder model that reads the full text of each page (not just the embedding). (4) The top 5 pages are injected into the LLM context as sources. (5) The LLM generates an answer with inline citations. The vector database is the retrieval engine that narrows billions of candidates to 20. The re-ranker is the quality filter that picks the best 5. The LLM is the synthesizer. Each stage of this pipeline reduces the candidate set by orders of magnitude while increasing quality. You can't just embed everything and call it a day — the retrieval quality determines the answer quality. Cursor takes a similar approach for their codebase RAG: they embed every file in your project, but they don't search every file on every query. They maintain a dependency graph + recently edited files list to scope the search. Your `package.json` and recently opened files influence which vectors get searched. This is "context-aware retrieval" — using metadata to narrow the vector search scope before running similarity. It's faster AND more relevant than a global vector search. --- ## Common mistakes | Mistake | Why it's wrong | What to do instead | |---------|---------------|-------------------| | Using Euclidean distance instead of cosine similarity | Euclidean distance penalizes long text (bigger magnitude vectors). Two paragraphs about the same topic can appear "far apart" if one is longer | Use cosine similarity (`<=>` with `1 - distance` in pgvector). It ignores magnitude and captures directional similarity | | Not chunking documents before embedding | Embedding a 50-page document produces a vector that represents the "average meaning" — useless for finding specific paragraphs | Chunk documents into 256-512 token pieces with overlap (10-20%). Each chunk gets its own embedding. The right chunk gets retrieved, not the document average | | Embedding the same text for every query | You're paying for embeddings API calls on every search. At 10,000 queries/day, that's 10,000 embedding calls × $0.02/1M tokens | Embed document chunks once at index time. Embed the user's query at search time. Cache query embeddings if users repeat searches | | Creating an IVFFlat index on an empty table | IVFFlat needs training data to build clusters. An empty-table index puts everything in one bucket — queries scan all rows | Insert at least 10% of expected data before building IVFFlat, or use HNSW (no training needed). After bulk inserts, run `REINDEX INDEX CONCURRENTLY` | | Storing embeddings as `double precision[]` instead of `vector` | You lose pgvector's optimized storage (compressed binary), index support, and distance operators. Queries are 10-100x slower | Always use the `vector(N)` data type from pgvector. It stores vectors as half-precision floats (2 bytes each) instead of 8 bytes for double precision | | Not normalizing chunk text before embedding | "PostgreSQL" vs "postgresql" vs "Postgres" produce different embeddings for the same concept. Punctuation and whitespace variance add noise | Normalize text: lowercase, trim whitespace, collapse newlines. For code, consider stripping comments. Test with a few samples: do similar documents get similar similarity scores? | | Setting `match_threshold` to 0 without testing | Every query returns results — even when none are relevant. Users get confidently wrong answers backed by irrelevant "sources" | Set a minimum similarity threshold (0.7 for most use cases). If no chunks exceed the threshold, tell the user "I couldn't find relevant information" instead of fabricating an answer from bad context | --- ## Production notes - **Re-index after bulk inserts.** If you insert 100K new chunks into an HNSW-indexed table, the new vectors exist in the table but not in the index graph. Queries will miss them. Run `REINDEX INDEX CONCURRENTLY your_hnsw_index` after large batch inserts. For continuous ingestion, HNSW supports incremental insertion (new vectors get added to the graph), but the graph quality degrades over time — schedule periodic reindexing (weekly for high-churn datasets). - **Monitor index size and query performance.** pgvector indexes can grow large. An HNSW index on 1M vectors (1536 dimensions) is ~6GB. Monitor `pg_total_relation_size('your_index_name')` and query latency. If queries slow down, check: (1) Is the index bloated? `REINDEX`. (2) Did you run `ANALYZE document_chunks` recently? The query planner needs fresh statistics. (3) Is `ef_search` high enough? Increase it for better recall at the cost of speed: `SET hnsw.ef_search = 100;`. - **Use connection pooling for embedding API calls.** The OpenAI embeddings endpoint has rate limits (3,000 RPM for text-embedding-3-small). At scale, batch your embedding calls: the API accepts up to 2,048 texts per request. One batch call with 2,048 texts uses 1 RPM, not 2,048. Also implement the same TokenBucket rate limiter from Chapter 1 for embedding calls. - **Separate your embedding compute from your application servers.** At 10M+ chunks being embedded, you're making millions of API calls. Run embedding generation as a background job (BullMQ, Temporal, or a simple queue). The application server stores the text, enqueues an embedding job, and the worker processes the queue. This keeps your API response times fast and lets you retry failed embedding calls without user impact. - **Version your embedding model.** If you switch from `text-embedding-3-small` (1536 dims) to `text-embedding-3-large` (3072 dims), your entire vector column becomes incompatible. Store the embedding model name in a metadata column. When you migrate models, create a new column (`embedding_v2 VECTOR(3072)`) and backfill gradually. Run both columns in parallel during the migration window. ## Performance notes - **HNSW with `ef_search=40` gives 95% recall at 1ms on 1M vectors.** The default `ef_search` is 40. Increase to 100 for 99% recall at ~3ms. Decrease to 20 for 90% recall at ~0.5ms. Test with your own data — recall requirements depend on your use case. A search engine can tolerate 90% recall; a medical diagnosis system cannot. - **Batch your embedding API calls.** OpenAI's API accepts up to 2,048 input texts per request. Embedding 10,000 chunks as 5 batch calls (2,048 each) is 100x faster than 10,000 individual calls due to network overhead. The cost is identical — pricing is per-token, not per-request. Most SDKs support batched embeddings: `openai.embeddings.create({ model, input: [text1, text2, ...] })`. - **pgvector query performance scales sub-linearly with HNSW.** On 100K vectors: ~0.3ms. On 1M vectors: ~1ms. On 10M vectors: ~3-5ms. The graph structure means query time grows logarithmically, not linearly. This is why HNSW is the default choice for production — IVFFlat queries grow closer to linearly with dataset size. - **Pre-filter with metadata before vector search.** If your user only has access to documents in "workspace-42", filter by `workspace_id` FIRST, then run vector similarity on the filtered set. pgvector supports this natively: `WHERE workspace_id = 42 ORDER BY embedding <=> query_vector LIMIT 10`. The Postgres planner applies the metadata filter before the index scan, dramatically reducing the search space. - **Cache embeddings for frequently accessed content.** If you have a set of "popular documents" that appear in 80% of searches, cache their embeddings in Redis. The embedding itself is ~6KB (1536 floats). At 10,000 popular documents, that's 60MB — easily fits in Redis. Saves API costs and reduces search latency by skipping the embedding step entirely. ## Security notes - **Embedding vectors can leak sensitive information.** Researchers have demonstrated "embedding inversion attacks" — reconstructing original text from its embedding vector with surprising accuracy. If your vector database is compromised, attackers might partially reconstruct your documents. For highly sensitive content (PII, trade secrets, legal documents), consider: (1) encrypting vectors at rest, (2) using on-premises embedding models (no API calls to third parties), (3) storing vectors in a separate, access-controlled database schema. - **Row-Level Security works with pgvector.** You can attach RLS policies to your `document_chunks` table so users can only search within documents they own. The vector similarity operator respects RLS — queries automatically filter to accessible rows before computing distances. This is a huge advantage of pgvector over dedicated vector databases where you'd need to implement access control at the application layer. - **Validate embedding dimensions match your index.** If you accidentally embed text with `text-embedding-3-large` (3072 dims) and try to insert into a `VECTOR(1536)` column, pgvector throws an error. But worse: if you truncate to 1536 dimensions, your vectors are semantically corrupted — the first 1536 dimensions of a 3072-dim vector don't represent the same meaning. Always validate dimensions at insert time with a CHECK constraint: `CONSTRAINT valid_embedding CHECK (vector_dims(embedding) = 1536)`. - **Rate-limit embedding generation to prevent abuse.** A malicious user submitting 1,000 unique long-form texts for embedding could cost you $10 in API calls. Implement the same per-user token budget pattern from Chapter 1. Different quota for embedding vs. chat: embeddings are cheaper per call but can be submitted in high volume. --- ## Exercises ### Beginner 1. **Generate your first embeddings:** Install `openai` npm package. Write a script that takes 5 sentences (mix of related and unrelated), generates embeddings via `text-embedding-3-small`, and computes pairwise cosine similarity. Print a similarity matrix. Verify that "The weather is sunny today" and "It's raining cats and dogs" have higher similarity than either with "PostgreSQL is a relational database." 2. **Set up pgvector locally:** Install pgvector on your local Postgres (or use Supabase's free tier). Create a `documents` table with a `VECTOR(1536)` column. Insert 3 manually created test documents (embed them in your script). Write a query that accepts a text query, embeds it, and returns the most similar document using the `<=>` operator. ### Intermediate 1. **Build a document chunker with overlap:** Write a `chunkText(text: string, chunkSize: number, overlap: number): string[]` function. It should split text into chunks of `chunkSize` tokens (estimate 4 chars = 1 token), with `overlap` tokens shared between consecutive chunks. Test with a 5,000-word article. Ensure no sentence is cut mid-way — chunks should break at paragraph or sentence boundaries. Use `Intl.Segmenter` for sentence boundary detection. 2. **Implement hybrid search in pgvector:** Create a `hybrid_search` PostgreSQL function (like the one in this chapter) that combines `<=>` cosine distance with `ts_rank` full-text search. Test with a dataset of 100 tech articles. Compare pure vector search vs. hybrid search for queries like "fix ECONNREFUSED error in Node.js" and "how does React useEffect work". Which approach finds more relevant results for each query type? ### Advanced 1. **Build a multi-tenant semantic search API:** Design a system where multiple organizations upload documents and search within their own data. Implement: (1) pgvector with RLS so org-1 can't search org-2's documents, (2) an API endpoint `POST /search` that accepts `{ query, orgId }`, embeds the query, and returns chunks with similarity scores > 0.7, (3) a dashboard that shows per-org embedding costs and search volume. Load-test with 10 organizations, 1,000 documents each, and 100 concurrent searches. 2. **Evaluate and compare embedding models:** Take a dataset of 1,000 question-answer pairs (use the MS MARCO or Natural Questions dataset). Embed all questions with: OpenAI `text-embedding-3-small`, OpenAI `text-embedding-3-large`, and Cohere `embed-english-v3.0`. For each model, compute the retrieval accuracy: for each question, find the top-5 most similar questions (excluding itself). Check if the correct answer document appears in the top-5. Compare: recall@5, cost per 1M tokens, and embedding latency. Write a report recommending which model to use for a production Q&A system. --- ## Interview questions ### Beginner 1. **Q: What is an embedding in the context of LLMs?** **A:** An embedding is a fixed-length array of floating-point numbers (typically 768, 1536, or 3072 dimensions) that represents the semantic meaning of text. Words, sentences, or documents with similar meanings produce vectors that are close together in vector space (high cosine similarity). Embeddings are generated by specialized models like OpenAI's `text-embedding-3-small` — these models are trained to map semantically similar texts to nearby points. You can then search for "concepts" rather than exact keywords: a search for "vehicle maintenance" might return documents about "car repair" even though they share no words. 2. **Q: What's the difference between full-text search and semantic search?** **A:** Full-text search matches exact keywords (with stemming and fuzzy matching). It finds documents containing the words you typed. Semantic search matches meaning — it finds documents about the concept you described, even if they use completely different words. A full-text search for "database performance" finds documents containing those words. A semantic search might find documents about "query optimization", "index tuning", and "slow PostgreSQL debugging" — none of which contain the exact phrase "database performance." Semantic search is broader but can miss exact matches (like error codes). Hybrid search combines both. 3. **Q: What is cosine similarity and why is it used instead of Euclidean distance for embeddings?** **A:** Cosine similarity measures the cosine of the angle between two vectors — it ranges from -1 (opposite direction) to 1 (same direction), with 0 meaning orthogonal (unrelated). It's preferred over Euclidean distance because it ignores vector magnitude (length). A long document and a short document about the same topic should be considered similar — their vectors point in the same direction but have different magnitudes. Euclidean distance would penalize the length difference; cosine similarity correctly identifies the semantic similarity. In pgvector, `1 - (a <=> b)` converts cosine distance to cosine similarity. ### Senior 1. **Q: You're building a codebase search tool (like GitHub's semantic code search). The codebase has 500,000 files totaling 50GB of text. Design the indexing and retrieval architecture.** **A:** The architecture has four layers: (1) **Chunking:** Split each file into semantic units — functions, classes, or logical blocks — using AST-aware chunking (Tree-sitter for each language). A 1,000-line file becomes ~20 chunks. Avoid splitting mid-function. 500K files × 20 chunks = 10M chunks. (2) **Embedding:** Use a code-aware embedding model (e.g., OpenAI's `text-embedding-3-small` or Voyage AI's `voyage-code-2`). Batch-embed 10M chunks via background workers. At 2,048 chunks per batch, that's ~5,000 API calls = $50 in embedding costs. Store embeddings in pgvector with HNSW index (m=16, ef_construction=200). Index size: 10M × 1536 dimensions × 2 bytes = ~30GB. (3) **Retrieval:** User query → embed query → HNSW similarity search with metadata pre-filtering (language, repository, recently edited files). Return top 50 chunks. (4) **Re-ranking:** Use a lightweight cross-encoder or LLM to re-rank the top 50 chunks by relevance to the exact query. Return top 10. This two-stage retrieval (fast vector → slow re-rank) gives 95%+ precision while keeping query latency under 500ms. (5) **Cost optimization:** Cache embeddings for files that haven't changed. Only re-embed on git push. Store file hashes to detect changes. Index updates are incremental — only changed files get re-indexed. 2. **Q: How do you evaluate whether your semantic search is actually returning good results?** **A:** You need quantitative metrics and human evaluation. For quantitative: (1) Build a test set of 500+ queries with known relevant documents (humans label them). (2) Measure Recall@K — for each query, what percentage of known relevant documents appear in the top K results? Recall@10 = 85% means 85% of relevant docs appear in the top 10. (3) Measure MRR (Mean Reciprocal Rank) — where does the first relevant result appear? MRR = 0.5 means the first relevant result is typically at position 2. (4) Measure NDCG@10 — graded relevance that accounts for position (relevant docs at position 1 are better than at position 10). For human evaluation: (1) Side-by-side comparisons — show two sets of search results (different chunking strategies, different models) and ask evaluators "which is better?" (2) Track real user behavior — click-through rate on search results, time-to-next-search (did they find what they needed?), and "was this helpful?" thumbs-up/down. The combination of offline metrics (fast, repeatable) and online metrics (real user behavior) tells you if your semantic search is actually working. 3. **Q: Your vector search is fast at 10K documents but slows to 5 seconds at 10M documents. Walk through your debugging and optimization process.** **A:** First, confirm the index is being used: `EXPLAIN ANALYZE SELECT ...`. Look for "Index Scan using hnsw_index" vs "Seq Scan" (sequential scan means the index is broken or not being chosen). Common causes: (1) **Index not created or dropped:** `SELECT * FROM pg_indexes WHERE tablename = 'document_chunks'`. (2) **HNSW parameters too conservative:** `m=16, ef_construction=200` gives good recall at build time; `ef_search=40` at query time might be too low for 10M vectors — the search stops early and misses good candidates. Increase `SET hnsw.ef_search = 200;` and re-test. (3) **Index bloat:** After many inserts/updates/deletes, the HNSW graph has dead nodes. `REINDEX INDEX CONCURRENTLY hnsw_index` rebuilds the graph. (4) **Query planner choosing wrong plan:** `ANALYZE document_chunks` updates table statistics. If Postgres thinks the table has 1,000 rows but it actually has 10M, it might choose a sequential scan. (5) **Hardware limits:** 10M vectors × 1536 dims × 2 bytes = 30GB for the index alone. If your server has 16GB RAM, the index doesn't fit in memory — every query hits disk. Solution: more RAM, or partition the data (by customer, date range, or category) so each partition's index fits in memory. (6) **Metadata filter applied AFTER vector search:** If you filter by `workspace_id` and there's no index on `workspace_id`, Postgres might scan all 10M rows, compute distances, then filter — the worst case. Create a composite approach: `CREATE INDEX ON document_chunks (workspace_id)` so the planner filters first, then runs vector search on the smaller set. --- ## Summary Embeddings transform text into fixed-length vectors that capture semantic meaning — similar concepts cluster together in high-dimensional space. pgvector brings vector search into PostgreSQL, letting you run semantic similarity queries alongside regular SQL with ACID guarantees. Cosine similarity (not Euclidean distance) is the standard metric — it measures the angle between vectors, ignoring magnitude differences from text length. HNSW indexing provides sub-millisecond approximate nearest-neighbor search that scales logarithmically with dataset size. Hybrid search combines vector similarity (finds related concepts) with keyword relevance (finds exact matches like error codes) for the best of both worlds. Chunk documents before embedding — embedding an entire 50-page PDF produces a useless "average meaning" vector. For most applications, pgvector is the right choice over dedicated vector databases — you already have Postgres, and vectors alongside relational data unlock powerful join+search patterns. Production systems need: reindexing after bulk inserts, embedding model versioning, RLS for multi-tenant access control, and careful evaluation of search quality using Recall@K and human judgment. --- ## Quick recall - Embeddings = fixed-length vectors (1536 floats) that represent semantic meaning. Similar texts → nearby vectors. Generated by models like `text-embedding-3-small`. - Cosine similarity measures the angle between vectors. Ignore magnitude (text length). Use `1 - (a <=> b)` in pgvector. - pgvector adds `VECTOR(N)` type + HNSW/IVFFlat indexes to PostgreSQL. Vectors live alongside relational data — JOIN away. - HNSW: graph-based index, sub-ms queries at 10M scale, no training needed, higher memory. IVFFlat: cluster-based, needs training data, lower memory. - Chunk documents into 256-512 token pieces with 10-20% overlap. Embed each chunk separately. Query embedding → most similar chunk gets retrieved. - Hybrid search = vector similarity (0.7 weight) + full-text ts_rank (0.3 weight). Best for queries mixing concepts and exact terms. - Batch embed: OpenAI accepts 2,048 texts per API call. 1 batch call costs the same as 1 text call — 2,048x throughput gain. - `match_threshold: 0.7` filters out irrelevant results. Return "no relevant information found" instead of bad context poisoning your LLM. - pgvector RLS works natively: `WHERE workspace_id = 42 ORDER BY embedding <=> query_vector`. Access control at the database level. - Evaluate search quality with Recall@K, MRR, and NDCG. Complement with human side-by-side comparisons and real user click-through data. --- ## Quiz 1. **What does an embedding vector represent?** A) The exact text of a document, compressed **B) The semantic meaning of text as a point in high-dimensional space** C) The word count and sentence structure of a document D) The language the text is written in 2. **Why use cosine similarity instead of Euclidean distance for comparing embeddings?** A) Cosine similarity is faster to compute **B) Cosine similarity ignores vector magnitude — long and short texts about the same topic are correctly identified as similar** C) Euclidean distance only works with 2D vectors D) Cosine similarity is required by pgvector 3. **What happens if you embed a 50-page document as a single vector?** A) You get a highly accurate representation of the entire document **B) You get a vector representing the "average meaning" — useless for finding specific paragraphs during search** C) The embedding API returns an error D) pgvector automatically chunks it 4. **Which pgvector index type does NOT require a training step?** A) IVFFlat **B) HNSW** C) B-tree D) GIN 5. **What is hybrid search?** A) Running two vector searches in parallel **B) Combining vector similarity (semantic) with keyword relevance (lexical) for better search results** C) Using two different embedding models D) Searching across two databases simultaneously 6. **At what dataset size should you consider migrating from pgvector to a dedicated vector database like Pinecone?** A) 1,000 vectors B) 100,000 vectors **C) 10-50 million vectors, or when you need specialized features pgvector doesn't support** D) Always use Pinecone — pgvector isn't for production 7. **How do you prevent one user from searching another user's documents in pgvector?** A) Use a separate database per user B) Encrypt all vectors **C) Use PostgreSQL Row-Level Security (RLS) — vector similarity queries automatically respect RLS policies** D) Filter results in application code after retrieval

Chapter 2 — Embeddings & Vector Databases

Part of L7