Part 3 — Domain 3: Integration (19%) — the heaviest domain
Prep-course lesson: Enterprise Integration & Production. Official tasks: capability bloat · authN/authZ gaps · accuracy–latency trade-offs · observability at scale · RAG pipelines (chunking/indexing) · retrieval matched to data shape and query pattern · integration mechanism (MCP, API/CLI, agent-to-agent) · progressive discovery vs. monolithic context.
Topics: 3.1 Choosing the integration mechanism · 3.2 Tool configuration, capability bloat, progressive discovery · 3.3 Authentication, authorization, identity boundaries · 3.4 RAG pipeline design · 3.5 Retrieval strategy by data shape · 3.6 Reliability, accuracy–latency, rate limits · 3.7 Observability at scale
3.1 Choosing the integration mechanism: MCP, direct API/CLI, agent-to-agent
1. Objectives. Select among MCP, direct API/CLI tool calls, and agent-to-agent protocols for a given system, and defend the choice.
2. Prerequisites. Tool use (function calling); JSON schemas; MCP intro course.
3. Beginner. Tool use: you describe functions to Claude; Claude asks you to call them; you run them and return results. MCP (Model Context Protocol) is an open standard that lets Claude-capable clients discover and call tools/resources/prompts from MCP servers in a uniform way. Direct API/CLI means your own code (or a shell tool) calls a system without an MCP layer. Agent-to-agent protocols let independently owned agents delegate to each other.
4. Advanced. Decision criteria:
- MCP when many clients/agents need the same integration, tools should be discoverable and reusable, a vendor already ships an MCP server, or you want a standard auth/transport story. Cost: another hop to operate/secure; tool descriptions consume context; third-party servers are a supply-chain risk.
- Direct API / tool function when it's one app calling one internal service, you need tight control over validation/latency, or the surface is tiny. Lowest overhead; least reusable.
- CLI/shell for developer/agent workflows on trusted machines where mature CLIs exist (e.g., git, cloud CLIs). Powerful but broad blast radius; sandbox it.
- Agent-to-agent when another team/vendor owns an autonomous capability behind its own agent; you delegate a goal instead of calling a function. Cost: opaque behavior, harder SLAs/evals, trust and identity propagation.
Transport matters: MCP over HTTP can use the spec's OAuth-based authorization; local stdio servers rely on environment credentials and are outside that authorization flow.
5. Steps. (1) List capabilities needed and their owners. (2) Count consumers (one app vs. many). (3) Check for existing MCP servers and their trust level. (4) Decide per integration; document in an ADR. (5) Define the auth path end-to-end (3.3). (6) Load-test and token-count tool definitions.
6. Diagram.
flowchart TD
N[Need: Claude must use system X] --> A{Many consumers or<br/>existing MCP server?}
A -- yes --> M[MCP server<br/>+ gateway/authZ]
A -- no --> B{Other team's autonomous agent?}
B -- yes --> AA[Agent-to-agent<br/>delegate goal, verify result]
B -- no --> C{Trusted dev environment,<br/>mature CLI?}
C -- yes --> CLI[CLI tool, sandboxed]
C -- no --> D[Direct API tool<br/>with schema validation]
7. Scenario. A company has a CRM, an ITSM, and a document store, used by a support agent, a sales assistant, and an internal chatbot. Expose each system once as an MCP server behind an auth gateway, instead of three bespoke integrations per app.
8. Recommended/alternatives. MCP for shared enterprise systems; direct tools for one-off internal calls. Change if only one consumer exists and the MCP hop adds unacceptable latency or ops burden.
9. Security. Treat MCP servers like any dependency: allow-list, pin versions, review tool descriptions (they're prompt-injection vectors), and enforce authZ server-side.
10. Performance. Each tool definition costs tokens; each hop adds latency; cache tool lists where possible.
11. Mistakes. "MCP everywhere" by default; installing unvetted servers; assuming MCP authorization is mandatory or covers tool-level data permissions (it doesn't — your tools must still authorize).
12. Scenario question. Three apps need the same ticketing integration; two other teams will add apps next quarter. → Build one MCP server with centralized auth rather than N direct integrations.
13. Revision. Reuse → MCP; one-off → direct; trusted dev → CLI; delegated goal → A2A; authZ is always yours.
14. Docs. MCP specification (modelcontextprotocol.io, including the authorization section); Intro to MCP course; Tool use docs.
3.2 Tool configuration, capability bloat, and progressive discovery
1. Objectives. Detect over-tooled agents; apply least capability; choose progressive discovery vs. a monolithic tool/context load.
2. Prerequisites. 3.1; 2.3.
3. Beginner. Every tool you give an agent adds description tokens, more choices (more mistakes), and more ways to cause harm. Capability bloat = tools or permissions the agent doesn't need.
4. Advanced. Symptoms: wrong tool selection, long prompts, slow responses, and unexpected actions. Remedies: (a) remove unneeded tools (the official Sample 1 logic: removal beats logging or confirmations); (b) scope by role/task — different agents/workflows get different tool sets; (c) consolidate overlapping tools and write precise, non-overlapping descriptions; (d) progressive discovery — expose a small set of core tools plus a discovery mechanism (search tools, skills, on-demand MCP tool loading) so detail enters context only when needed; (e) design tool outputs to be small and high-signal. Monolithic loading is acceptable when the toolset is small, stable, and always relevant; it fails as the catalog grows.
5. Steps. (1) Inventory tools and map each to a real user need. (2) Remove or gate the rest. (3) Measure tool-selection accuracy in evals. (4) If the catalog > ~dozens or varies by task, introduce discovery. (5) Re-test token cost and accuracy.
6. Diagram.
flowchart LR
subgraph Monolithic
A1[Agent] --- T1[60 tool definitions<br/>always in context]
end
subgraph Progressive
A2[Agent] --- C[5 core tools]
A2 --> D[search_tools / skill index]
D -.loads on demand.-> X[Relevant 3 tools]
end
7. Scenario. Support agent has read-ticket, draft-reply, issue-refund, delete-account. Staff only need the first two. Remove the others from that agent entirely; create a separate, approval-gated workflow for refunds.
8. Recommended/alternatives. Least-capability config + progressive discovery as catalogs grow. Alternative: keep all tools and add guard prompts (prompts aren't enforcement). Change if the role truly needs the capability → keep it behind authZ + approval.
9. Security. Capability reduction shrinks attack surface; it's a preventive control, stronger than detective (logging) or compensating (confirmation) controls.
10. Cost. Fewer tool definitions → fewer input tokens; with caching, tool lists can sit in the stable prefix.
11. Mistakes. "Just in case" tools; overlapping descriptions; returning huge tool results; adding confirmation dialogs instead of removing the tool.
12. Scenario question. An agent has 45 tools and often picks the wrong one. → Reduce/scope tools per task and consider on-demand discovery, then re-run tool-selection evals.
13. Revision. Least capability; remove > monitor; discover on demand; small tool outputs.
14. Docs. Tool use docs; Agent Skills (progressive disclosure); MCP tools spec.
3.3 Authentication, authorization, and identity boundaries
1. Objectives. Identify authN/authZ gaps in Claude integrations; design identity propagation; apply least privilege.
2. Prerequisites. OAuth 2.x basics, RBAC/ABAC.
3. Beginner. AuthN = who are you? AuthZ = what may you do? The model is not a security boundary: it can be tricked, so access decisions must be made by your code/services.
4. Advanced. Principles: (1) Authorize at the tool/service, using the end user's identity (on-behalf-of / token exchange), not a shared super-user service account — otherwise the agent becomes a confused deputy. (2) Validate token audience/scope per resource. (3) Enforce row/document-level entitlements before retrieval returns content. (4) Short-lived, narrowly scoped credentials; never put secrets in prompts or logs. (5) Separate identities for the agent, each worker, and human approvers; record both user and agent identity in audit logs. (6) For MCP over HTTP, use the spec's OAuth-based authorization (protected-resource metadata, resource-bound tokens); remember MCP authorization is optional at the protocol level and doesn't replace a tool's own data permissions; stdio servers use environment credentials. Common gaps to spot: shared API keys in the agent; tools that accept user_id as a model-supplied argument; broad scopes "for convenience"; no tenant isolation in vector stores; credentials visible in tool results.
5. Steps. (1) Draw identities and trust boundaries. (2) For each tool: required permission, who can invoke, which identity executes. (3) Implement OBO/token exchange. (4) Add server-side validation of every model-supplied parameter. (5) Test with a low-privilege user and cross-tenant attempts. (6) Log decisions.
6. Diagram.
sequenceDiagram participant U as User participant App as App (authN) participant C as Claude participant B as Tool broker (authZ) participant S as Backend API U->>App: Request (user token) App->>C: Prompt (no secrets) C-->>App: tool_use(get_case, case_id) App->>B: call + user identity B->>B: check scope, entitlements, args B->>S: call with user-scoped token S-->>B: data (filtered) B-->>App: result App->>C: tool_result
7. Scenario. Agent uses one admin token for the CRM. A junior rep asks about a VIP account they can't normally see; the agent returns it. Fix: user-scoped tokens; CRM enforces entitlements.
8. Recommended/alternatives. OBO with per-tool authZ. Alternative: "tell the model to respect permissions" — not acceptable. Change if the backend can't accept user tokens → put an authorization proxy in front.
9. Governance. Owner: security/IAM + service owner. Evidence: access-test results, audit logs showing user+agent IDs.
10. Performance. Token exchange adds latency — cache short-lived tokens safely.
11. Mistakes. Shared service accounts; trusting model-provided IDs; entitlement checks after retrieval; secrets in system prompts.
12. Scenario question. A tool takes tenant_id from the model's arguments. → Derive tenant from the authenticated session; ignore/verify model-supplied values.
13. Revision. The model isn't a boundary; user-scoped identity; server-side checks; filter before context.
14. Docs. MCP authorization spec; Tool use best practices; Prep Lesson 2–3.
3.4 Designing a RAG pipeline: chunking and indexing
1. Objectives. Design ingestion, chunking, embedding, indexing, and refresh for a RAG system; diagnose stale/irrelevant retrieval.
2. Prerequisites. Embeddings and vector search basics.
3. Beginner. RAG = retrieve relevant passages from your knowledge base and give them to Claude with the question, so answers are grounded in your data and can cite sources.
4. Advanced. Pipeline: ingest (connectors, parsing PDFs/tables) → clean/normalize → chunk → embed → index (+ metadata: source, ACL, date, version) → retrieve → rerank → assemble context → generate with citations → evaluate/monitor. Chunking: match chunk size to question granularity; respect structure (headings, tables, code); use modest overlap; attach parent/section context so a chunk is understandable alone; consider contextual enrichment (prepend a short document/section description) to reduce ambiguity. Indexing: store ACL metadata for pre-filtering; version documents and re-index on change; delete tombstoned docs; keep embedding model/version consistent (mixed embeddings corrupt similarity). Refresh failures are a classic root cause: confident wrong answers right after a document refresh → investigate retrieval/indexing first (the official Sample 3).
5. Steps. (1) Inventory sources, formats, owners, ACLs, update frequency. (2) Choose chunking per doc type. (3) Build a labeled query→relevant-chunk set. (4) Measure retrieval (recall@k, precision) separately from generation quality. (5) Add reranking/hybrid search if recall is weak. (6) Automate re-index with versioning and health checks.
6. Diagram.
flowchart LR S[Sources] --> P[Parse/clean] --> CH[Chunk + metadata + ACL] CH --> E[Embed] --> IX[(Index)] Q[Query + user identity] --> F[Filter by ACL] --> R[Retrieve top-k] --> RR[Rerank] IX --> R RR --> G[Claude: answer + citations] G --> M[Monitor: retrieval hit rate, citation validity]
7. Scenario. HR knowledge assistant over policies updated monthly; after a refresh, answers cite old leave policy. Root cause: failed re-index left stale chunks plus mixed embedding versions.
8. Recommended/alternatives. Structure-aware chunking + hybrid retrieval + rerank + ACL pre-filter. Alternative: long-context stuffing of the whole corpus (works for small static corpora; poor for large/changing/ACL-bound data). Change if corpus is small enough to cache whole.
9. Security. Enforce entitlements in retrieval; treat retrieved text as untrusted (indirect injection); redact sensitive fields at ingest where required.
10. Cost/latency. Embedding + search + rerank add latency; cache frequent queries; cap k.
11. Mistakes. One-size chunking; no metadata; no eval of retrieval; silent index failures; no citations validation.
12. Scenario question. Answers are fluent but wrong after an ingestion job ran. → Check index freshness, chunk integrity, embedding consistency, before touching prompts or the model.
13. Revision. Ingest→chunk→embed→index→retrieve→rerank→generate; test retrieval separately; version and re-index.
14. Docs. Anthropic docs on embeddings/retrieval patterns and citations; Prep Lesson 2.
3.5 Matching retrieval strategy to data shape and query pattern
1. Objectives. Choose among semantic, keyword/BM25, hybrid, metadata-filtered, structured (SQL/graph), and agentic search.
2. Prerequisites. 3.4.
3. Beginner. Different data and questions need different retrieval. Exact IDs and error codes want keyword search; fuzzy conceptual questions want semantic search; "total sales by region" wants SQL, not text retrieval.
4. Advanced. Map data shape (unstructured prose, semi-structured docs, tables, code, knowledge graph, time series) × query pattern (lookup, conceptual, multi-hop, aggregation, comparative):
| Query/data | Best-fit retrieval |
|---|---|
| Exact identifiers, codes, names | Keyword/BM25 or exact filter |
| Conceptual questions over prose | Semantic (embeddings) + rerank |
| Mixed | Hybrid (keyword + semantic) |
| Aggregations over tables | Text-to-SQL / structured API with validated queries |
| Relationships/multi-hop | Graph queries or iterative (agentic) retrieval |
| Code repos | Search tools (grep/symbol) + file reads, often agentic |
| Small, stable corpus | Cached full-context |
Agentic search lets Claude issue and refine queries via tools; it improves multi-hop recall at the cost of latency, tokens, and variance. Progressive discovery (search → skim → open) beats pre-stuffing for large corpora.
5. Steps. (1) Classify top 50 real queries by pattern. (2) Choose the retrieval per class; route if classes differ. (3) Evaluate retrieval per class. (4) For SQL, use read-only roles, schema allow-lists, row limits.
6. Diagram.
flowchart TD
Q[Query] --> R{Router}
R -- identifier --> K[Keyword / exact]
R -- conceptual --> V[Semantic + rerank]
R -- aggregate --> SQL[Validated read-only SQL]
R -- multi-hop --> AG[Agentic iterative search]
7. Scenario. Finance analyst assistant: "What was Q3 EMEA revenue?" (SQL), "Summarize risks in the 10-K" (semantic), "Find error E-4021 runbook" (keyword).
8. Recommended/alternatives. Router + per-class retrieval. Alternative: one vector index for everything (poor for IDs and aggregates).
9. Security. Text-to-SQL: least-privilege DB role, parameterization, allow-listed tables, query validation/limits.
10. Cost. Hybrid + rerank costs more per query but often cheaper than model retries/bad answers.
11. Mistakes. Using vectors for exact lookups; letting the model write unrestricted SQL; ignoring metadata filters.
12. Scenario question. Users search by invoice numbers and get unrelated semantic matches. → Add keyword/exact matching (hybrid).
13. Revision. Shape × pattern → strategy; hybrid for mixed; SQL for aggregates; agentic for multi-hop.
14. Docs. Prep Lesson 2; Tool use docs.
3.6 Reliability, accuracy–latency trade-offs, rate limits, and cost control
1. Objectives. Justify configuration decisions balancing accuracy, latency, and cost; implement timeouts, retries, rate-limit handling, and graceful degradation.
2. Prerequisites. HTTP semantics; 2.1–2.4.
3. Beginner. More thinking, bigger models, and more retrieval improve accuracy but slow responses and increase cost. Systems must also survive failures and traffic spikes.
4. Advanced — levers: model tier; thinking/effort settings (on models that support them); retrieval depth; verification passes; streaming (improves perceived latency; doesn't reduce total time); prompt caching; batch API (asynchronous, 50% cost reduction; wrong for interactive); max_tokens; parallel tool calls. Reliability patterns: per-call timeouts; retries with exponential backoff + jitter for transient errors (e.g., 429, 5xx) honoring retry-after; idempotency keys for write tools so retries don't duplicate side effects; circuit breakers; fallback tiers (smaller model, cached answer, human handoff); queueing and backpressure; cost budgets (per request, per user, per day) with kill-switches. Rate limits are measured in RPM, input and output tokens per minute (token-bucket behavior), so bursts can trigger 429s even under average limits. The Python/TypeScript SDKs include built-in retry logic.
5. Steps. (1) Define SLOs (p95 latency, availability, cost/request). (2) Budget latency across steps. (3) Add timeouts/retries/idempotency. (4) Define degradation modes. (5) Load-test, including 429 behavior. (6) Alert on SLO burn.
6. Diagram.
flowchart LR R[Request] --> Q[Queue + budget check] Q --> C[Claude call<br/>timeout + retry/backoff] C -- 429/5xx --> B[Backoff + jitter] --> C C -- exhausted --> F[Fallback: smaller model / cached / human] C -- ok --> V[Validate] --> O[Respond]
7. Scenario. Month-end spike triggers 429s; refunds retried and duplicated. Fix: backoff with jitter, idempotency keys on the refund tool, queue with concurrency cap, and batch for non-urgent summaries.
8. Recommended/alternatives. Layered resilience + caching/batch where fit. Alternative: raise rate limits only (doesn't fix duplicates or cost). Change if SLA demands sub-second and the fallback quality is unacceptable → pre-compute.
9. Security/governance. Fail closed for authorization/safety checks (deny when the checker is down); fail open/soft only for non-critical enrichments.
10. Cost. Track tokens per request; alert on anomalies; use the token-counting endpoint for estimates.
11. Mistakes. Retrying non-idempotent writes; infinite retries; no timeout on tool calls; using batch for chat; streaming as a "fix" for slow backends.
12. Scenario question. Retried refund calls created duplicate refunds. → Idempotency keys on write operations, not fewer retries alone.
13. Revision. SLOs → budgets → timeouts/retries/idempotency → degradation; safety fails closed.
14. Docs. Rate limits: https://docs.claude.com/en/api/rate-limits · Batch processing · Streaming · Errors (API reference).
3.7 Observability and monitoring at scale
1. Objectives. Select logging/tracing/metrics strategies for LLM systems; handle privacy, volume, and cost.
2. Prerequisites. Distributed tracing basics.
3. Beginner. You can't fix what you can't see. For LLM apps, capture what went in, what came out, which tools ran, how long, how many tokens, and whether users were satisfied.
4. Advanced. Instrument at three levels: request trace (trace ID across gateway → prompt assembly → retrieval → model → tools → response; span per step), metrics (latency p50/p95/p99, time-to-first-token, tokens in/out, cache hit rate, error and 429 rates, tool success rate, cost/request, escalation rate), and quality signals (user feedback, human-review outcomes, automated eval scores on sampled traffic, citation validity). At scale: sample traces (always keep errors, low-confidence, flagged cases), redact/tokenize PII before logging, retention limits, and separate operational logs from audit logs (immutable, with user+agent identity, tool, parameters (redacted), decision, outcome). Track drift: input distribution, retrieval quality, and model behavior after provider updates. Treat logged tool parameters as sensitive.
5. Steps. (1) Define a trace schema and IDs. (2) Add spans and metrics. (3) Redaction at the edge. (4) Dashboards per SLO. (5) Alerts on quality and cost, not just uptime. (6) Replay capability: store inputs needed to reproduce failures within policy.
6. Diagram.
flowchart LR A[App spans] --> OT[Trace pipeline] M[Metrics] --> OT Q[Sampled eval scores<br/>+ user feedback] --> OT OT --> D[Dashboards + alerts] OT --> AU[(Immutable audit log<br/>redacted)]
7. Scenario. Support agent costs doubled; traces show a new prompt version added a 6k-token example block uncached — found via tokens/request and cache-hit-rate metrics.
8. Recommended/alternatives. Trace + metrics + sampled quality evals. Alternative: log everything raw (privacy/cost problem) or only uptime metrics (blind to quality).
9. Governance. Audit evidence supports compliance (D5); access to logs is itself controlled.
10. Cost. Observability volume can rival inference cost — sample and tier storage.
11. Mistakes. Logging PII; no trace linking retrieval to answers; only monitoring HTTP 200s; no alert on cost anomalies.
12. Scenario question. Quality complaints rise but dashboards are green. → Add quality monitoring (sampled evals, feedback, retrieval metrics), not just availability.
13. Revision. Trace, measure, sample, redact, alert on quality/cost, keep audit logs separate.
14. Docs. Prep Lesson 2; Usage and cost reporting in the Console/Admin API docs.