# Project P4 — RAG Support Copilot By the end of this chapter you will: - Build the full pipeline: ingest docs → chunk → embed → pgvector - Answer with citations and refusal behavior when retrieval is weak - Evaluate quality with a golden dataset and track it over changes - Operate within explicit cost and latency budgets --- ## The project goal Build a production-style support copilot that answers questions using your own documentation. Functional requirements: 1. Ingest markdown/MDX docs from a content folder. 2. Chunk documents, embed them, and store vectors in pgvector. 3. Retrieve relevant chunks and generate responses with source citations. 4. Refuse confidently when retrieval confidence is weak. 5. Evaluate quality on a golden set and track metrics over revisions. Non-functional requirements: - p95 latency target <= 2.5 seconds for standard queries, - explicit per-request token and cost budgets, - structured telemetry for incident diagnosis, - deterministic runbook another engineer can execute. --- ## What success looks like Minimum acceptance criteria: 1. `/ask` endpoint returns grounded answer with citations. 2. `/reindex` pipeline ingests docs and updates vector index safely. 3. Golden evaluation suite reports retrieval and answer metrics. 4. Policy guardrails block unsafe or out-of-scope responses. 5. Cost/latency dashboards show stable behavior under test load. Show the evaluator and telemetry screens in your demo, not just one successful answer. That proves engineering maturity. --- ## Recommended architecture ```text apps/rag-copilot/ app/ api/ ask.py reindex.py health.py rag/ chunking.py embeddings.py retrieval.py generation.py citations.py guardrails.py eval/ golden_cases.json runner.py metrics.py db/ schema.sql pgvector_repo.py core/ config.py logging.py telemetry.py rate_limit.py scripts/ ingest_docs.py run_eval.py tests/ test_retrieval.py test_generation_contract.py test_guardrails.py ``` Subsystem responsibilities: - `chunking`: split documents into context-preserving segments, - `embeddings`: model calls and embedding cache policy, - `retrieval`: vector search, thresholding, and reranking, - `generation`: prompt assembly and response shaping, - `guardrails`: policy validation and fallback behavior, - `eval`: repeatable quality scoring. --- ## Data model blueprint ```sql create extension if not exists vector; create table if not exists rag_chunks ( id bigserial primary key, source_id text not null, source_path text not null, section text, chunk_text text not null, chunk_checksum text not null, embedding_model text not null, embedding vector(1536) not null, created_at timestamptz default now() ); create index if not exists rag_chunks_embedding_hnsw on rag_chunks using hnsw (embedding vector_cosine_ops); create index if not exists rag_chunks_source_path_idx on rag_chunks (source_path); ``` Keep `chunk_checksum` for cheap change detection during reindex operations. --- ## Implementation phases Phase 1: Ingestion and indexing - Read docs from configured folders. - Split with structure-aware chunking and overlap. - Generate embeddings with model-versioned cache keys. - Upsert vectors into pgvector with metadata. Phase 2: Retrieval and answer generation - Embed query and retrieve top-k chunks. - Apply threshold and optional reranker. - Build prompt with source-tagged context. - Return structured answer: `summary`, `citations`, `confidence`. Phase 3: Guardrails and fallback behavior - Validate input length and domain scope. - Block prompt injection patterns where possible. - Refuse or ask clarifying questions on low-confidence retrieval. - Enforce JSON output schema before returning responses. Phase 4: Evaluation and observability - Add golden-set evaluator with retrieval recall and citation coverage. - Add answer correctness scoring (human rubric or LLM judge + spot checks). - Log request-level telemetry: model, tokens, latency, cache hit, fallback reason. - Add release gate thresholds before deploy. Phase 5: Release and operations - Containerize app and wire environment config. - Run migration/index checks on deploy. - Add dashboards and alert rules for cost/latency/quality drift. - Document incident and rollback runbook. --- ## Example endpoint contract ```python from pydantic import BaseModel, Field class AskRequest(BaseModel): question: str = Field(min_length=6, max_length=1200) top_k: int = Field(default=5, ge=1, le=10) class Citation(BaseModel): source_id: str source_path: str excerpt: str class AskResponse(BaseModel): summary: str confidence: float = Field(ge=0, le=1) citations: list[Citation] refused: bool = False ``` Contract-first schemas reduce downstream integration bugs. --- ## Command runbook ```bash # start services docker compose up -d postgres # install and run app pip install -r requirements.txt uvicorn app.main:app --reload # initialize schema/index psql "$DATABASE_URL" -f app/db/schema.sql # ingest docs python scripts/ingest_docs.py --source ../../content --module Python-Mastery # ask one question curl -s -X POST http://localhost:8000/ask \ -H "Content-Type: application/json" \ -d '{"question":"What is our refund timeline?","top_k":5}' # run evaluation suite python scripts/run_eval.py --cases app/eval/golden_cases.json ``` --- ## Quality gates | Gate | Requirement | |---|---| | Retrieval recall@5 | >= 0.80 on golden set | | Citation coverage | >= 0.90 | | Refusal correctness | >= 0.95 on out-of-scope cases | | p95 latency | <= 2500 ms in staging profile | | Cost budget | <= configured cap per request/session | | Regression gate | No score drop beyond tolerated delta | --- ## Common failure modes | Failure mode | Symptom | Fix | |---|---|---| | Stale index after doc updates | Answers cite outdated policy | Add checksum-driven incremental reindex job | | High retrieval noise | Fluent but wrong answers | Tighten threshold and rerank top candidates | | Empty retrieval too often | Frequent unnecessary refusals | Improve chunking and expand top-k carefully | | Cost spikes | Sudden token spend jump | Enforce hard token caps and route cheaper model for low-risk tasks | | Prompt injection exposure | Unsafe instruction following | Strict system constraints + sanitize and isolate context | --- ## Extension ideas 1. Add hybrid retrieval (vector + lexical BM25) with weighted fusion. 2. Add reranker model for top-20 to top-5 precision boost. 3. Add tenant-aware filtering for multi-organization deployments. 4. Add feedback loop to capture bad answers as future eval cases. 5. Add asynchronous bulk-reindex worker with progress events. --- ## ADR sample ### ADR-001: pgvector in PostgreSQL vs dedicated vector database - Context: Team already operates PostgreSQL and needs low operational overhead. - Decision: Use pgvector for initial production stage with HNSW indexing. - Consequences: - Positive: simpler infra stack and transactional metadata co-location. - Negative: fewer specialized retrieval features than dedicated vector platforms. - Revisit trigger: sustained corpus/query scale exceeds agreed latency budget. ### ADR-002: LangChain orchestration vs hand-rolled pipeline - Context: Need fast iteration plus explicit control over core retrieval logic. - Decision: Use LangChain for prompt/retriever composition, keep chunking/index logic explicit in local modules. - Consequences: - Positive: faster development while preserving critical transparency. - Negative: additional dependency surface and abstraction learning curve. - Revisit trigger: framework abstraction starts hiding needed low-level control. --- ## Interview defense prompts 1. Why did you choose your chunk size and overlap settings? 2. How do you detect and handle low-confidence retrieval safely? 3. Which quality metric did you optimize first, and why? 4. How do you control cost growth without sacrificing user experience? 5. What would break first if query traffic increases 10x? --- ## Summary - P4 proves you can build and operate a grounded AI assistant, not only demo one. - Retrieval quality, citation integrity, and refusal correctness are as important as fluent language generation. - Evaluation gates and telemetry make prompt/model iteration safe and measurable. - The project is complete when another engineer can run your pipeline from the runbook and reproduce results. ## Self-Check Quiz

Project P4 — RAG Support Copilot

Part of L4