Chapter 5 — AI Agents & Multi-Agent Systems

The why

An LLM that only chats is a librarian who only reads aloud. An LLM with agent capabilities is a librarian who walks to the shelves, pulls the book, cross-references three other volumes, updates the catalog, and emails you a summary — all without you giving a second instruction. Tool use, function calling, and planning are what transform a text generator into a system that actually does things.

Core concepts

The agent loop: LLM + tools + memory + planning

Every AI agent, from a 50-line script to Devin, follows the same fundamental loop. Understanding this loop is the difference between building something that works once and something that works at 3 AM without human intervention.

┌─────────────────────────────────────────────────────┐
│                   THE AGENT LOOP                      │
│                                                       │
│  1. USER INPUT → "What's the top-selling product     │
│     in our European warehouses this quarter?"        │
│                         │                             │
│                         ▼                             │
│  2. LLM DECIDES → "I need to call get_sales_data()   │
│     with region='EU' and period='Q2_2026'"           │
│                         │                             │
│                         ▼                             │
│  3. TOOL EXECUTION → Your function runs, returns     │
│     [{product: "Widget X", revenue: 450000}, ...]    │
│                         │                             │
│                         ▼                             │
│  4. LLM EVALUATES → "Got 12 rows. Widget X is #1.    │
│     Should I also check inventory levels?"           │
│                         │                             │
│                         ▼                             │
│  5. MORE TOOLS? → Yes → call check_inventory()        │
│                  → No  → format final answer          │
│                         │                             │
│                         ▼                             │
│  6. FINAL RESPONSE → Natural language answer +        │
│     optional structured data                          │
└─────────────────────────────────────────────────────┘

Here's the minimal TypeScript agent loop — this is the engine that powers everything from customer support bots to autonomous coding agents:

// === agent-loop.ts ===
// The core loop: 60 lines that turn an LLM into an agent
 
interface Tool {
  name: string;
  description: string;
  parameters: Record<string, unknown>;
  execute: (args: Record

Tool/function calling: the bridge between language and action

Function calling is the mechanism that lets an LLM say "I need to run this specific function with these specific arguments" instead of just "you should probably check the database." The LLM doesn't execute code — it outputs structured JSON that your application executes.

// === tool-definitions.ts ===
// Define tools with JSON Schema — the LLM uses these to decide what to call
 
const tools: Tool[] = [
  {
    name: "search_knowledge_base",
    description:
      "Search the internal knowledge base for articles matching a query. " +
      "Use this when users ask about company policies, product docs, or historical decisions.",
    parameters: {
      type

ReAct pattern: Reason, then Act, then Observe

The ReAct (Reasoning + Acting) pattern interleaves thought with action. Instead of the LLM planning everything upfront, it thinks about what it knows, decides on one action, observes the result, then thinks again. This is the pattern behind most production agents.

// === react-pattern.ts ===
// The ReAct system prompt — teaches the LLM to think-then-act
 
const REACT_SYSTEM_PROMPT = `You are an AI assistant that solves problems step by step.
 
For every user request, follow this pattern:
 
1. THOUGHT: Analyze what you know and what you need to find out.
   Write your reasoning as if thinking aloud.
2. ACTION: If you need information, call the appropriate tool.
   If you have enough information, provide the final answer.
3. OBSERVATION: After a tool returns results, evaluate them.
   Are they sufficient? Is something missing? Should you try a different approach?
 
IMPORTANT RULES:
- Never assume information you haven't retrieved via a tool.
- If a tool returns an error, try an alternative approach before giving up.
- If you've tried 3 different approaches and all failed, explain what you tried and ask the user for guidance.
- Cite sources when using tool results.`;

Planning: CoT, ToT, and structured reasoning

Chain of Thought (CoT) is the simplest form of planning — the LLM writes out its reasoning steps before answering. You enable it with a system prompt that says "think step by step" or by showing examples of step-by-step reasoning. For agents, CoT is essential: the LLM needs to explain WHY it's calling a tool, not just call it.

Tree of Thoughts (ToT) extends CoT by exploring multiple reasoning paths simultaneously. The LLM generates several possible next steps, evaluates each, and pursues the most promising one. This is computationally expensive (you're making multiple LLM calls per step) but dramatically improves accuracy on tasks that require exploration — debugging complex code, solving math problems, or planning multi-step workflows.

// === tree-of-thoughts.ts ===
// Simplified ToT: generate N candidate next steps, score them, pick the best
 
async function treeOfThoughtsStep(
  problem: string,
  candidates: number = 3
): Promise<string> {
  // Step 1: Generate candidates
  const brainstormPrompt

Multi-agent collaboration

Single agents hit a ceiling. A coding agent that also manages your calendar and answers support tickets becomes confused — the system prompt grows to 5000 tokens, tool definitions balloon, and the agent starts mixing contexts. Multi-agent architectures solve this by giving each agent a focused role.

// === multi-agent-orchestrator.ts ===
// A coordinator agent delegates to specialist agents
 
interface SpecialistAgent {
  name: string;
  description: string;
  capabilities: string[];
  run: (input: string) => Promise

Real-world example: How Cognition built Devin

Cognition's Devin is the most prominent AI coding agent. It doesn't just suggest code — it plans features, writes code across multiple files, runs terminal commands, debugs test failures, and iterates until tests pass. The architecture reveals patterns every backend engineer should understand.

The execution environment is a sandbox. Devin doesn't run commands on your machine. It spins up a container with a shell, code editor, and browser — every tool (write_file, run_test, search_web, git_commit) operates inside this sandbox. This is the only safe way to give an LLM shell access. The sandbox has network restrictions, filesystem quotas, and a time limit per session. If Devin runs rm -rf /, it nukes its own container — not your laptop.

Planning before coding. Devin writes a plan before writing code. This isn't a gimmick — it's the ReAct pattern at scale. The planner agent breaks the feature into subtasks, the coder agent implements each subtask, and the reviewer agent checks the output. Failed tests loop back to the coder. This three-agent loop can run for 20-30 iterations on a complex feature.

The same pattern works at smaller scale. CrewAI and AutoGen are open-source frameworks that implement multi-agent orchestration. CrewAI defines agents with roles, goals, and tools — then runs them sequentially or in parallel. AutoGen (from Microsoft) adds conversation patterns: two agents can debate a solution, or a manager agent can delegate to worker agents. These frameworks aren't magic — they're the agent loop with better abstractions, retry logic, and conversation management.

Common mistakes

Production notes

• Version your tool definitions. When you change a tool's parameters (add a required field, rename an argument), old conversations in progress will break because the LLM generated tool calls with the old schema. Use a version prefix in tool names (search_kb_v2) and keep old versions alive for a deprecation window while in-flight conversations drain.
• Implement a tool timeout per invocation. A single tool execution that hangs (database connection pool exhausted, external API down) will stall the entire agent loop. Wrap every tool.execute() call in a timeout promise. If the tool doesn't respond in N seconds, return an error to the LLM so it can try an alternative approach.
• Use a dedicated agent service, not your main API. Agent workloads are bursty and CPU-intensive (multiple LLM calls + tool execution per request). Running them in your main API process will cause latency spikes for regular traffic. Deploy agents as a separate service with its own scaling rules.
• Add human-in-the-loop for destructive actions. Any tool that modifies production data, sends customer-facing messages, or triggers a deployment should require human approval before execution. The agent proposes the action; a human clicks "approve" in a Slack message or dashboard. This adds friction intentionally — the cost of an incorrect automated action is always higher than the cost of a 30-second approval delay.
• Track agent-specific metrics. Beyond standard API metrics, track: agent success rate (did it complete the task?), average iterations per task, tool call success rate (what percentage of tool calls returned useful results vs errors), and abandonment rate (tasks where maxIterations was hit). These metrics reveal whether your tool descriptions, system prompt, or tool implementations need improvement.

Performance notes

• Parallelize independent tool calls. If the LLM requests three tool calls in one response and those tools don't depend on each other (e.g., fetch user profile + fetch team members + fetch recent orders), execute them in parallel with Promise.all(). This cuts iteration time from 3× tool_latency to max(tool_latency).
• Cache tool results within a session. If the agent calls get_user(userId=42) three times in one conversation, the second and third calls are wasted. Cache tool results in a Map keyed by toolName + JSON.stringify(args). Invalidate the cache when a write operation changes the underlying data.
• Stream the agent's thinking to the user. Nothing is worse than staring at a spinner for 30 seconds while the agent works. Use Server-Sent Events to stream each step: "🔍 Searching knowledge base...", "📊 Analyzing 12 results...", "✍️ Generating response...". This makes the latency feel like progress.
• Use cheaper models for planning, expensive models for final output. The planning/decision step (which tool to call next?) can use a fast model like GPT-4o-mini or Claude Haiku. Save the expensive model (GPT-4o, Claude Opus) for the final synthesis step where quality matters most. This cuts per-request cost by 40-60% with minimal quality impact.
• Set a per-task token budget, not just per-request. An agent that makes 10 LLM calls at 2000 tokens each costs more than a single 4000-token call. Track cumulative tokens across the entire agent session. Alert if a single user request consumes more than your threshold (e.g., 20K tokens).

Security notes

• Never pass raw user input to shell or database tools. Even if the LLM is supposed to generate safe commands, treat every tool argument as potentially malicious. Use parameterized queries for databases. Escape shell arguments. Validate file paths against an allowlist of directories. The LLM is not a security boundary — your tool implementations are.
• Restrict tool access by agent role. A customer-facing support agent should never have access to internal admin tools (delete user, view billing details, modify system config). Implement this at the tool registry level: each agent gets a subset of all available tools.
• Sanitize tool results before they reach the LLM. If a database query returns PII (email addresses, phone numbers, SSNs), that data is now in the LLM's context window and potentially in the provider's logs. Redact PII from tool results before passing them to the LLM, or use a self-hosted model for sensitive workloads.
• Rate-limit agent requests more aggressively than regular API requests. An agent request can trigger 5-20 LLM calls + database queries + external API calls. One malicious user making 10 agent requests per second could cost you hundreds of dollars per hour. Use a token bucket or sliding window rate limiter with a low threshold for agent endpoints.
• Implement a kill switch per agent session. If you detect anomalous behavior (the agent is calling the same tool in a loop, accessing unexpected resources, or generating suspicious output), you need to terminate that session immediately — not just stop processing but revoke any temporary credentials or tokens issued for that session.

Exercises

Beginner

1. Build a single-tool agent: Write an agent that has exactly one tool — get_current_weather(city: string). The tool should return hardcoded weather data for 5 cities. Test that the agent: (a) calls the tool when asked about weather, (b) returns a natural language response incorporating the tool's data, and (c) doesn't call the tool when asked a non-weather question like "who won the world cup?"

2. Add iteration limits: Take the single-tool agent from exercise 1 and deliberately create a scenario where the tool returns confusing data (e.g., { error: "ambiguous city name" }). Verify that your agent: (a) retries at most N times (where N is your maxIterations), (b) eventually returns a graceful failure message to the user, and (c) logs each failed attempt with the tool name and error.

Intermediate

1. Build a ReAct agent with 3 tools: Create an agent with these tools: search_docs(query), list_available_apis(), and get_api_schema(apiName). The agent should help developers find the right API for their use case. Test with: "I need to send emails programmatically — what API should I use and what parameters does it need?" The agent should: search docs for "email", list available APIs, find the email API, retrieve its schema, and present the answer.

2. Implement multi-agent delegation: Build a coordinator agent that delegates to two specialist agents: a "translator" agent (detects language, translates to English) and a "summarizer" agent (takes English text and returns a 3-bullet summary). Test with a French paragraph. The coordinator should route French text → translator → summarizer → final output. Log the handoff between agents so you can trace the full pipeline.

Advanced

1. Build an agent with a budget tracker: Extend any agent from the exercises above with a cost-tracking middleware. Track: tokens consumed per LLM call, total tokens per session, estimated cost (using current OpenAI/Anthropic pricing), and tool execution time. Add a configurable budget limit — when the session exceeds the limit, the agent should gracefully stop and return what it has so far. Test with a budget of $0.05 and verify the agent stops at the limit.

2. Implement a human-in-the-loop approval flow: Build an agent that can propose database schema changes. When the agent wants to execute ALTER TABLE, it should not run the command. Instead, it should: (a) output the proposed SQL, (b) explain what the change does and why it's needed, (c) wait for a "approved" or "rejected" response (simulated via a callback function or a mock HTTP endpoint), and (d) only execute on approval. The approval mechanism should have a 5-minute timeout — if no response, the agent should abandon the change and explain why.

Interview questions

Beginner

1. Q: What's the difference between an LLM and an AI agent? A: An LLM (Large Language Model) takes text input and produces text output — it's a completion engine. An AI agent wraps an LLM in a loop that can: call external tools (APIs, databases, file systems), maintain memory across multiple turns, plan multi-step workflows, and evaluate its own outputs. An LLM tells you HOW to query a database. An agent queries the database, analyzes the result, and tells you what it found. The agent loop (decide → act → observe → decide) is what separates them.

2. Q: What is function calling in the context of LLMs? A: Function calling (also called tool use) is a capability where an LLM outputs structured JSON specifying which function to call and with what arguments, instead of (or in addition to) natural language text. The LLM doesn't execute the function — your application reads the JSON, executes the function, and feeds the result back to the LLM. The LLM uses the function's name, description, and parameter schema (provided in the API request) to decide when and how to call it. It's the bridge between "the AI thinks something" and "the AI does something."

3. Q: Why do you need a maxIterations limit in an agent loop? A: Without a limit, an agent can enter an infinite loop if: a tool keeps returning unexpected results, the LLM gets confused and retries the same failing approach, or two tools create a dependency cycle (tool A's output makes the LLM call tool B, whose output makes it call tool A again). Each iteration costs money (LLM API call) and time. A maxIterations limit (typically 5-15) acts as a circuit breaker — when hit, the agent summarizes what it found and stops. In production, you should also track cost and abort if the session exceeds a dollar threshold.

Senior

1. Q: You're building a customer support agent that can refund orders, cancel subscriptions, and modify account details. How do you prevent the agent from taking incorrect destructive actions? A: There are four layers of defense. First, tool design: destructive tools should require explicit confirmation parameters (e.g., confirmed: boolean that must be true) so the LLM can't accidentally trigger them. Second, human-in-the-loop: any tool that modifies production data, moves money, or sends customer-facing communications should queue the action for human approval rather than executing immediately. Third, validation at the tool level: before executing, validate that the order ID exists, the refund amount matches the original payment, and the user has permission to perform the action. Fourth, audit logging: every destructive action should log who requested it (end user + agent session ID), what was proposed, who approved it, and the final result. These logs are critical for both debugging and compliance. The principle: the LLM proposes, the application validates, the human approves.

2. Q: How would you design a multi-agent system where agents need to share intermediate results without blowing up the context window? A: Don't pass raw outputs between agents — pass summaries and structured data. Implement a shared "blackboard" (a key-value store like Redis) where agents write their findings in a structured format. Each agent reads only the keys relevant to its task. For example, the "code reviewer" agent writes { file: "auth.ts", issues: [...], score: 7 } to the blackboard. The "test generator" agent reads only the issues key to generate targeted tests. The coordinator maintains a lightweight context — just the task description and references to blackboard keys. This keeps each agent's context window focused on its specific job. Additionally, implement context window monitoring: if an agent's accumulated messages approach the limit, summarize older messages before adding new ones.

3. Q: An agent in production is behaving inconsistently — sometimes it's brilliant, sometimes it calls the wrong tools or produces garbage output. How do you systematically debug this? A: Start with observability. You need structured logs for every agent session containing: the full message history (system prompt + user messages + assistant responses + tool calls + tool results), which model was used with what parameters, and the final output. Reproduce the failing session by replaying the exact message history — this tells you whether the issue is deterministic (same input → same bad output) or stochastic (temperature-related variance). If deterministic, inspect the tool descriptions and system prompt: is there ambiguity that could cause the LLM to pick the wrong tool? If stochastic, lower the temperature and add more explicit instructions in the tool descriptions. Next, implement evaluation: for each agent task type, maintain a set of 20-50 test cases with expected tool calls and outputs. Run these evals on every prompt or tool description change. Finally, consider a "grader" agent — a separate, cheaper LLM that reviews the main agent's tool calls and flags suspicious ones (e.g., "agent called delete_user for a request about changing email").

Summary

AI agents transform LLMs from text generators into systems that act. The agent loop — decide, call tools, observe results, decide again — is the universal pattern. Function calling bridges language and action by letting the LLM output structured JSON that your application executes. The ReAct pattern interleaves reasoning with action, producing more reliable results than planning everything upfront. Multi-agent architectures split complex tasks across specialist agents with focused tools and prompts. The two hard problems in production agents are reliability (making the right tool calls consistently) and safety (preventing destructive actions). Solve reliability with great tool descriptions, structured logging, and eval suites. Solve safety with tool-level validation, human-in-the-loop for destructive actions, and per-agent tool restrictions. Agents are powerful but expensive — every iteration costs money, so budget tracking and iteration limits are not optional.

Quick recall

• Agent = LLM + tools + memory + planning loop. The LLM decides what to do, tools execute it, the loop continues until the task is done or the limit is hit.
• Function calling: The LLM outputs structured JSON specifying which function to call with which arguments. Your code executes it, not the LLM.
• ReAct pattern: Thought → Action → Observation → Thought. Interleave reasoning with tool calls. Better than upfront planning.
• Tool descriptions are prompt engineering. The LLM decides which tool to call based on the description. Write them like you're explaining to a colleague.
• maxIterations is mandatory. Without it, buggy tools or confusing results cause infinite loops that burn money.
• Multi-agent = divide and conquer. Each specialist agent gets a focused role, minimal tools, and a role-specific system prompt.
• Human-in-the-loop for destructive actions. The LLM proposes; a human approves. The cost of a 30-second delay is always less than the cost of a wrong automated action.
• Log everything. Tool calls, arguments, results, errors, timing. You cannot debug an agent you cannot observe.
• Cheap models for planning, expensive models for final output. GPT-4o-mini for "which tool next?" GPT-4o for "write the final answer." Cuts cost 40-60%.
• Tool results are untrusted input. Validate, sanitize, and truncate everything before it reaches the LLM or your database.

Quiz

1. What is the core loop that every AI agent follows? A) Ask user → respond → ask user → respond B) Decide → call tools → observe results → decide again C) Train → validate → test → deploy D) Receive request → query database → format response → send

2. In function calling, who executes the function — the LLM or your application code? A) The LLM executes it in a sandbox B) Your application code executes it — the LLM only outputs which function to call and with what arguments C) A cloud function provided by OpenAI D) The user's browser

3. Why is maxIterations essential in an agent loop? A) It's required by the OpenAI API B) To prevent infinite loops that burn API budget — an agent with buggy tools can call them forever C) To comply with GDPR D) To make the agent faster

4. What does the ReAct pattern stand for? A) React + Act — a UI framework for agents B) Reasoning + Acting — interleave thought with tool calls C) Reactive + Active — two agent modes D) Read + Act — ingest then execute

5. When should you use multi-agent architecture instead of a single agent? A) Always — more agents are always better B) Only when the task fits in one prompt C) When different roles need different tools, prompts would conflict, or tasks can run in parallel D) Only for frontend applications

6. What's the most important defense for agents that can perform destructive actions? A) A longer system prompt B) Human-in-the-loop approval — the agent proposes, a human approves before execution C) Faster API calls D) Using a larger model

7. How do you reduce per-request cost in an agent system? A) Use the most expensive model for everything for best quality B) Use cheaper/faster models for planning/decision steps, expensive models only for final synthesis C) Skip tool validation D) Remove all logging