Chapter 2 — Embeddings & Vector Databases

The why

Full-text search finds the word "cat" in your documents. Semantic search finds "feline", "kitten", "tabby", and "the animal that knocked over my coffee" — even when none of those words appear in the query. Embeddings are the technology that makes this possible, and vector databases make it fast enough for production.

Core concepts

What embeddings actually are: from text to numbers

An embedding is a dense vector — a fixed-length array of floating-point numbers — that represents the semantic meaning of a piece of text. OpenAI's text-embedding-3-small produces 1,536-dimensional vectors. Every word, sentence, paragraph, or document you feed it gets mapped to a point in 1,536-dimensional space. The magic: semantically similar texts are close together in that space.

Here's the concrete reality. These two sentences:

"The cat sat on the mat."
"A feline rested on the rug."

Produce vectors that are very close together (cosine similarity > 0.9). These two:

"The cat sat on the mat."
"Quarterly earnings exceeded analyst expectations by 14%."

Produce vectors that are far apart (cosine similarity < 0.2). The model learned these relationships by reading billions of text pairs during training.

// Generate embeddings with OpenAI
import OpenAI from 'openai';
 
const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
 
async function embed(text: string)

Cosine similarity: measuring "closeness" between vectors

The standard distance metric for embeddings is cosine similarity — it measures the angle between two vectors, not the straight-line distance. This matters because vector magnitude (length) can vary with text length, but direction captures meaning more reliably.

function cosineSimilarity(a: number[], b: number[]): number {
  if (a.length !== b.length) {
    throw new Error('Vectors must have the same dimension');
  }

pgvector: Vector search inside PostgreSQL

pgvector is a PostgreSQL extension that adds a vector data type and indexing. The killer feature: your vectors live in the same database as your application data. No separate service, no data sync, no eventual consistency headaches. You can join vector search results with regular SQL tables in a single query.

-- Enable the extension (run once per database)
CREATE EXTENSION IF NOT EXISTS vector;
 
-- Create a table with a vector column
CREATE TABLE document_chunks (
  id SERIAL PRIMARY KEY,
  document_id INTEGER NOT NULL,
  chunk_text TEXT NOT NULL,
  chunk_index INTEGER NOT NULL,
  embedding VECTOR(1536),  

Inserting embeddings from your Node.js application:

import { createClient } from '@supabase/supabase-js';
 
const supabase = createClient(
  process.env.SUPABASE_URL!,
  process.env.SUPABASE_SERVICE_ROLE_KEY!
);
 
async function indexDocument(documentId

Semantic search query — the entire point of all this setup:

async function semanticSearch(query: string, limit: number = 10) {
  // 1. Embed the user's query
  const queryEmbedding = await embed(query);
 
  // 2. Find the most similar chunks using cosine distance
  const { data,

IVFFlat vs HNSW: choosing the right index

You have two main indexing options in pgvector, and the choice matters at scale:

-- IVFFlat: good for moderate datasets with frequent inserts
CREATE INDEX ON document_chunks
  USING ivfflat (embedding vector_cosine_ops)
  WITH (lists = 100);
-- Rule of thumb: lists = rows / 1000 (for 100K rows, use 100 lists)
 
-- HNSW: better for large, read-heavy datasets
CREATE INDEX ON document_chunks
  USING hnsw (embedding vector_cosine_ops)
  WITH (m = 16, ef_construction = 200);
--

Choosing a vector database: pgvector vs Pinecone vs the rest

You don't always need to reach for Pinecone. Here's a practical decision framework:

Hybrid search: when vector search alone isn't enough

Pure semantic search has a blind spot: exact keyword matching. A user searching for "RFC 7231" should find the actual RFC document, not a blog post about HTTP semantics that happens to be semantically similar. Hybrid search combines vector similarity (semantic) with keyword relevance (lexical) for the best of both worlds:

-- Hybrid search: combine vector similarity + full-text keyword relevance
CREATE OR REPLACE FUNCTION hybrid_search(
  query_text TEXT,
  query_embedding VECTOR(1536),
  match_count INT DEFAULT 10,
  vector_weight FLOAT DEFAULT 0.7,  -- How much to trust vector vs keyword
  keyword_weight FLOAT DEFAULT 0.3
) RETURNS TABLE

Real-world example: How Perplexity built their answer engine on vector search

Perplexity AI doesn't just send your question to an LLM. Their pipeline: (1) Your question is embedded and used to search a real-time web index (billions of pages). (2) The top 10-20 most relevant pages are retrieved via semantic + keyword hybrid search. (3) Those pages are re-ranked using a cross-encoder model that reads the full text of each page (not just the embedding). (4) The top 5 pages are injected into the LLM context as sources. (5) The LLM generates an answer with inline citations.

The vector database is the retrieval engine that narrows billions of candidates to 20. The re-ranker is the quality filter that picks the best 5. The LLM is the synthesizer. Each stage of this pipeline reduces the candidate set by orders of magnitude while increasing quality. You can't just embed everything and call it a day — the retrieval quality determines the answer quality.

Cursor takes a similar approach for their codebase RAG: they embed every file in your project, but they don't search every file on every query. They maintain a dependency graph + recently edited files list to scope the search. Your package.json and recently opened files influence which vectors get searched. This is "context-aware retrieval" — using metadata to narrow the vector search scope before running similarity. It's faster AND more relevant than a global vector search.

Common mistakes

Production notes

• Re-index after bulk inserts. If you insert 100K new chunks into an HNSW-indexed table, the new vectors exist in the table but not in the index graph. Queries will miss them. Run REINDEX INDEX CONCURRENTLY your_hnsw_index after large batch inserts. For continuous ingestion, HNSW supports incremental insertion (new vectors get added to the graph), but the graph quality degrades over time — schedule periodic reindexing (weekly for high-churn datasets).
• Monitor index size and query performance. pgvector indexes can grow large. An HNSW index on 1M vectors (1536 dimensions) is ~6GB. Monitor pg_total_relation_size('your_index_name') and query latency. If queries slow down, check: (1) Is the index bloated? REINDEX. (2) Did you run ANALYZE document_chunks recently? The query planner needs fresh statistics. (3) Is ef_search high enough? Increase it for better recall at the cost of speed: SET hnsw.ef_search = 100;.
• Use connection pooling for embedding API calls. The OpenAI embeddings endpoint has rate limits (3,000 RPM for text-embedding-3-small). At scale, batch your embedding calls: the API accepts up to 2,048 texts per request. One batch call with 2,048 texts uses 1 RPM, not 2,048. Also implement the same TokenBucket rate limiter from Chapter 1 for embedding calls.
• Separate your embedding compute from your application servers. At 10M+ chunks being embedded, you're making millions of API calls. Run embedding generation as a background job (BullMQ, Temporal, or a simple queue). The application server stores the text, enqueues an embedding job, and the worker processes the queue. This keeps your API response times fast and lets you retry failed embedding calls without user impact.
• Version your embedding model. If you switch from text-embedding-3-small (1536 dims) to (3072 dims), your entire vector column becomes incompatible. Store the embedding model name in a metadata column. When you migrate models, create a new column () and backfill gradually. Run both columns in parallel during the migration window.

Performance notes

• HNSW with ef_search=40 gives 95% recall at 1ms on 1M vectors. The default ef_search is 40. Increase to 100 for 99% recall at ~3ms. Decrease to 20 for 90% recall at ~0.5ms. Test with your own data — recall requirements depend on your use case. A search engine can tolerate 90% recall; a medical diagnosis system cannot.
• Batch your embedding API calls. OpenAI's API accepts up to 2,048 input texts per request. Embedding 10,000 chunks as 5 batch calls (2,048 each) is 100x faster than 10,000 individual calls due to network overhead. The cost is identical — pricing is per-token, not per-request. Most SDKs support batched embeddings: openai.embeddings.create({ model, input: [text1, text2, ...] }).
• pgvector query performance scales sub-linearly with HNSW. On 100K vectors: ~0.3ms. On 1M vectors: ~1ms. On 10M vectors: ~3-5ms. The graph structure means query time grows logarithmically, not linearly. This is why HNSW is the default choice for production — IVFFlat queries grow closer to linearly with dataset size.
• Pre-filter with metadata before vector search. If your user only has access to documents in "workspace-42", filter by workspace_id FIRST, then run vector similarity on the filtered set. pgvector supports this natively: WHERE workspace_id = 42 ORDER BY embedding <=> query_vector LIMIT 10. The Postgres planner applies the metadata filter before the index scan, dramatically reducing the search space.
• Cache embeddings for frequently accessed content. If you have a set of "popular documents" that appear in 80% of searches, cache their embeddings in Redis. The embedding itself is ~6KB (1536 floats). At 10,000 popular documents, that's 60MB — easily fits in Redis. Saves API costs and reduces search latency by skipping the embedding step entirely.

Security notes

• Embedding vectors can leak sensitive information. Researchers have demonstrated "embedding inversion attacks" — reconstructing original text from its embedding vector with surprising accuracy. If your vector database is compromised, attackers might partially reconstruct your documents. For highly sensitive content (PII, trade secrets, legal documents), consider: (1) encrypting vectors at rest, (2) using on-premises embedding models (no API calls to third parties), (3) storing vectors in a separate, access-controlled database schema.
• Row-Level Security works with pgvector. You can attach RLS policies to your document_chunks table so users can only search within documents they own. The vector similarity operator respects RLS — queries automatically filter to accessible rows before computing distances. This is a huge advantage of pgvector over dedicated vector databases where you'd need to implement access control at the application layer.
• Validate embedding dimensions match your index. If you accidentally embed text with text-embedding-3-large (3072 dims) and try to insert into a VECTOR(1536) column, pgvector throws an error. But worse: if you truncate to 1536 dimensions, your vectors are semantically corrupted — the first 1536 dimensions of a 3072-dim vector don't represent the same meaning. Always validate dimensions at insert time with a CHECK constraint: CONSTRAINT valid_embedding CHECK (vector_dims(embedding) = 1536).
• Rate-limit embedding generation to prevent abuse. A malicious user submitting 1,000 unique long-form texts for embedding could cost you $10 in API calls. Implement the same per-user token budget pattern from Chapter 1. Different quota for embedding vs. chat: embeddings are cheaper per call but can be submitted in high volume.

Exercises

Beginner

1. Generate your first embeddings: Install openai npm package. Write a script that takes 5 sentences (mix of related and unrelated), generates embeddings via text-embedding-3-small, and computes pairwise cosine similarity. Print a similarity matrix. Verify that "The weather is sunny today" and "It's raining cats and dogs" have higher similarity than either with "PostgreSQL is a relational database."

2. Set up pgvector locally: Install pgvector on your local Postgres (or use Supabase's free tier). Create a documents table with a VECTOR(1536) column. Insert 3 manually created test documents (embed them in your script). Write a query that accepts a text query, embeds it, and returns the most similar document using the <=> operator.

Intermediate

1. Build a document chunker with overlap: Write a chunkText(text: string, chunkSize: number, overlap: number): string[] function. It should split text into chunks of chunkSize tokens (estimate 4 chars = 1 token), with overlap tokens shared between consecutive chunks. Test with a 5,000-word article. Ensure no sentence is cut mid-way — chunks should break at paragraph or sentence boundaries. Use Intl.Segmenter for sentence boundary detection.

2. Implement hybrid search in pgvector: Create a hybrid_search PostgreSQL function (like the one in this chapter) that combines <=> cosine distance with ts_rank full-text search. Test with a dataset of 100 tech articles. Compare pure vector search vs. hybrid search for queries like "fix ECONNREFUSED error in Node.js" and "how does React useEffect work". Which approach finds more relevant results for each query type?

Advanced

1. Build a multi-tenant semantic search API: Design a system where multiple organizations upload documents and search within their own data. Implement: (1) pgvector with RLS so org-1 can't search org-2's documents, (2) an API endpoint POST /search that accepts { query, orgId }, embeds the query, and returns chunks with similarity scores > 0.7, (3) a dashboard that shows per-org embedding costs and search volume. Load-test with 10 organizations, 1,000 documents each, and 100 concurrent searches.

2. Evaluate and compare embedding models: Take a dataset of 1,000 question-answer pairs (use the MS MARCO or Natural Questions dataset). Embed all questions with: OpenAI text-embedding-3-small, OpenAI text-embedding-3-large, and Cohere embed-english-v3.0. For each model, compute the retrieval accuracy: for each question, find the top-5 most similar questions (excluding itself). Check if the correct answer document appears in the top-5. Compare: recall@5, cost per 1M tokens, and embedding latency. Write a report recommending which model to use for a production Q&A system.

Interview questions

Beginner

1. Q: What is an embedding in the context of LLMs? A: An embedding is a fixed-length array of floating-point numbers (typically 768, 1536, or 3072 dimensions) that represents the semantic meaning of text. Words, sentences, or documents with similar meanings produce vectors that are close together in vector space (high cosine similarity). Embeddings are generated by specialized models like OpenAI's text-embedding-3-small — these models are trained to map semantically similar texts to nearby points. You can then search for "concepts" rather than exact keywords: a search for "vehicle maintenance" might return documents about "car repair" even though they share no words.

2. Q: What's the difference between full-text search and semantic search? A: Full-text search matches exact keywords (with stemming and fuzzy matching). It finds documents containing the words you typed. Semantic search matches meaning — it finds documents about the concept you described, even if they use completely different words. A full-text search for "database performance" finds documents containing those words. A semantic search might find documents about "query optimization", "index tuning", and "slow PostgreSQL debugging" — none of which contain the exact phrase "database performance." Semantic search is broader but can miss exact matches (like error codes). Hybrid search combines both.

3. Q: What is cosine similarity and why is it used instead of Euclidean distance for embeddings? A: Cosine similarity measures the cosine of the angle between two vectors — it ranges from -1 (opposite direction) to 1 (same direction), with 0 meaning orthogonal (unrelated). It's preferred over Euclidean distance because it ignores vector magnitude (length). A long document and a short document about the same topic should be considered similar — their vectors point in the same direction but have different magnitudes. Euclidean distance would penalize the length difference; cosine similarity correctly identifies the semantic similarity. In pgvector, 1 - (a <=> b) converts cosine distance to cosine similarity.

Senior

1. Q: You're building a codebase search tool (like GitHub's semantic code search). The codebase has 500,000 files totaling 50GB of text. Design the indexing and retrieval architecture. A: The architecture has four layers: (1) Chunking: Split each file into semantic units — functions, classes, or logical blocks — using AST-aware chunking (Tree-sitter for each language). A 1,000-line file becomes ~20 chunks. Avoid splitting mid-function. 500K files × 20 chunks = 10M chunks. (2) Embedding: Use a code-aware embedding model (e.g., OpenAI's text-embedding-3-small or Voyage AI's voyage-code-2). Batch-embed 10M chunks via background workers. At 2,048 chunks per batch, that's ~5,000 API calls = $50 in embedding costs. Store embeddings in pgvector with HNSW index (m=16, ef_construction=200). Index size: 10M × 1536 dimensions × 2 bytes = ~30GB. (3) Retrieval: User query → embed query → HNSW similarity search with metadata pre-filtering (language, repository, recently edited files). Return top 50 chunks. (4) Re-ranking: Use a lightweight cross-encoder or LLM to re-rank the top 50 chunks by relevance to the exact query. Return top 10. This two-stage retrieval (fast vector → slow re-rank) gives 95%+ precision while keeping query latency under 500ms. (5) Cost optimization: Cache embeddings for files that haven't changed. Only re-embed on git push. Store file hashes to detect changes. Index updates are incremental — only changed files get re-indexed.

2. Q: How do you evaluate whether your semantic search is actually returning good results? A: You need quantitative metrics and human evaluation. For quantitative: (1) Build a test set of 500+ queries with known relevant documents (humans label them). (2) Measure Recall@K — for each query, what percentage of known relevant documents appear in the top K results? Recall@10 = 85% means 85% of relevant docs appear in the top 10. (3) Measure MRR (Mean Reciprocal Rank) — where does the first relevant result appear? MRR = 0.5 means the first relevant result is typically at position 2. (4) Measure NDCG@10 — graded relevance that accounts for position (relevant docs at position 1 are better than at position 10). For human evaluation: (1) Side-by-side comparisons — show two sets of search results (different chunking strategies, different models) and ask evaluators "which is better?" (2) Track real user behavior — click-through rate on search results, time-to-next-search (did they find what they needed?), and "was this helpful?" thumbs-up/down. The combination of offline metrics (fast, repeatable) and online metrics (real user behavior) tells you if your semantic search is actually working.

Summary

Embeddings transform text into fixed-length vectors that capture semantic meaning — similar concepts cluster together in high-dimensional space. pgvector brings vector search into PostgreSQL, letting you run semantic similarity queries alongside regular SQL with ACID guarantees. Cosine similarity (not Euclidean distance) is the standard metric — it measures the angle between vectors, ignoring magnitude differences from text length. HNSW indexing provides sub-millisecond approximate nearest-neighbor search that scales logarithmically with dataset size. Hybrid search combines vector similarity (finds related concepts) with keyword relevance (finds exact matches like error codes) for the best of both worlds. Chunk documents before embedding — embedding an entire 50-page PDF produces a useless "average meaning" vector. For most applications, pgvector is the right choice over dedicated vector databases — you already have Postgres, and vectors alongside relational data unlock powerful join+search patterns. Production systems need: reindexing after bulk inserts, embedding model versioning, RLS for multi-tenant access control, and careful evaluation of search quality using Recall@K and human judgment.

Quick recall

• Embeddings = fixed-length vectors (1536 floats) that represent semantic meaning. Similar texts → nearby vectors. Generated by models like text-embedding-3-small.
• Cosine similarity measures the angle between vectors. Ignore magnitude (text length). Use 1 - (a <=> b) in pgvector.
• pgvector adds VECTOR(N) type + HNSW/IVFFlat indexes to PostgreSQL. Vectors live alongside relational data — JOIN away.
• HNSW: graph-based index, sub-ms queries at 10M scale, no training needed, higher memory. IVFFlat: cluster-based, needs training data, lower memory.
• Chunk documents into 256-512 token pieces with 10-20% overlap. Embed each chunk separately. Query embedding → most similar chunk gets retrieved.
• Hybrid search = vector similarity (0.7 weight) + full-text ts_rank (0.3 weight). Best for queries mixing concepts and exact terms.
• Batch embed: OpenAI accepts 2,048 texts per API call. 1 batch call costs the same as 1 text call — 2,048x throughput gain.
• match_threshold: 0.7 filters out irrelevant results. Return "no relevant information found" instead of bad context poisoning your LLM.
• pgvector RLS works natively: WHERE workspace_id = 42 ORDER BY embedding <=> query_vector. Access control at the database level.
• Evaluate search quality with Recall@K, MRR, and NDCG. Complement with human side-by-side comparisons and real user click-through data.

Quiz

1. What does an embedding vector represent? A) The exact text of a document, compressed B) The semantic meaning of text as a point in high-dimensional space C) The word count and sentence structure of a document D) The language the text is written in

2. Why use cosine similarity instead of Euclidean distance for comparing embeddings? A) Cosine similarity is faster to compute B) Cosine similarity ignores vector magnitude — long and short texts about the same topic are correctly identified as similar C) Euclidean distance only works with 2D vectors D) Cosine similarity is required by pgvector

3. What happens if you embed a 50-page document as a single vector? A) You get a highly accurate representation of the entire document B) You get a vector representing the "average meaning" — useless for finding specific paragraphs during search C) The embedding API returns an error D) pgvector automatically chunks it

4. Which pgvector index type does NOT require a training step? A) IVFFlat B) HNSW C) B-tree D) GIN

5. What is hybrid search? A) Running two vector searches in parallel B) Combining vector similarity (semantic) with keyword relevance (lexical) for better search results C) Using two different embedding models D) Searching across two databases simultaneously

6. At what dataset size should you consider migrating from pgvector to a dedicated vector database like Pinecone? A) 1,000 vectors B) 100,000 vectors C) 10-50 million vectors, or when you need specialized features pgvector doesn't support D) Always use Pinecone — pgvector isn't for production

7. How do you prevent one user from searching another user's documents in pgvector? A) Use a separate database per user B) Encrypt all vectors C) Use PostgreSQL Row-Level Security (RLS) — vector similarity queries automatically respect RLS policies D) Filter results in application code after retrieval