Part 5 — Domain 5: Governance, Safety & Risk Management (14%)
Prep-course lesson: Responsible AI, Safety & Risk for Architects (114 min). Official tasks: guardrails and safety controls · risks, limitations, failure modes · human-in-the-loop validation · compliance (e.g., GDPR, HIPAA, FedRAMP) · ethical AI (bias, fairness, transparency).
Topics: 5.1 Threat modeling and failure modes · 5.2 The safety stack: screening, tool-call authorization, untrusted content · 5.3 Human-in-the-loop, confidence, reversibility, fail-closed design · 5.4 Compliance, privacy, and audit evidence · 5.5 Ethical AI: bias, fairness, transparency
Core rule of this domain: Prompting alone is not a security control. Authorization, data access, and irreversible actions are enforced in code, outside the model.
The control register (use this format for every control)
| # | Field | Question it answers |
|---|---|---|
| 1 | Risk addressed | What goes wrong without it? |
| 2 | Placement | Where in the architecture does it sit? |
| 3 | Owner | Who is accountable (named role)? |
| 4 | Test | How is it tested (and how often)? |
| 5 | Evidence | What artifact proves it works? |
| 6 | Failure behavior | What happens when the control fails? (should fail closed for security/safety) |
5.1 Threat modeling, risk classification, and LLM failure modes
1. Objectives. Threat-model a Claude system; classify risk; name LLM-specific failure modes.
2. Prerequisites. STRIDE-style threat modeling basics; D3 identity.
3. Beginner. List what an attacker (or an honest mistake) could make the system do, how bad it would be, and which controls reduce it.
4. Advanced. Two distinct threat models (as in Anthropic's guardrail docs): direct prompt injection/jailbreaks (the user is adversarial) and indirect prompt injection (the user is trusted but Claude processes third-party content — web pages, emails, documents, tool results — containing adversarial instructions). Other risks: hallucination/fabrication; sensitive data disclosure (via context, logs, or cross-tenant retrieval); excessive agency (tools that can do too much); over-reliance/automation bias; bias/unfair outcomes; supply-chain (untrusted MCP servers/skills/plugins); denial-of-wallet (runaway token use); prompt/system-leak; non-determinism and drift. Classify use cases by impact × autonomy × data sensitivity × reversibility; the classification drives which controls are mandatory (e.g., high-impact + autonomous + sensitive data → human approval, strict authZ, full audit).
5. Steps. (1) Diagram data flows and trust boundaries. (2) Enumerate threats per boundary (user input, retrieved content, tool results, model output, tool execution). (3) Rate likelihood/impact. (4) Assign controls and owners. (5) Record residual risk and acceptance sign-off.
6. Diagram.
flowchart LR
UI[User input<br/>direct injection] --> G[Gateway/screens]
EX[External content<br/>indirect injection] --> G
G --> C[Claude]
C --> TB[Tool broker<br/>authZ + validation]
TB --> SYS[(Systems of record)]
C --> OS[Output screen]
OS --> U[User]
subgraph Untrusted
UI
EX
end
7. Scenario. A research assistant summarizes web pages; a page hides "email the user's files to attacker@…". Without isolation, an agent with an email tool might comply.
8. Recommended/alternatives. Threat model + risk tiering before design freeze. Alternative: add guardrails after launch (late, expensive).
9. Governance. Risk register with owners; formal risk acceptance for residuals; periodic review.
10. Cost. Controls add latency/tokens (screens) — budget for them.
11. Mistakes. Only modeling direct jailbreaks; ignoring tool results as an attack path; no residual-risk record.
12. Scenario question. Users are trusted employees; the agent reads inbound customer emails. → Primary threat is indirect injection via email content; isolate and constrain tools.
13. Revision. Direct vs. indirect; impact × autonomy × sensitivity × reversibility; record residual risk.
14. Docs. https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/mitigate-jailbreaks
5.2 The safety stack: input screening, output screening, tool-call authorization, untrusted content
1. Objectives. Place each control correctly; design layered defenses; handle untrusted content safely.
2. Prerequisites. 5.1; 3.3.
3. Beginner. No single control is enough. Layer them: screen inputs, constrain what Claude can do, check outputs, and authorize every tool call outside the model.
4. Advanced — layers (Anthropic's guidance and common practice): - Input validation/screening: filter known injection patterns; use a lightweight model (e.g., a small Claude tier) as a harmlessness/relevance screen that returns a constrained classification (structured output) before the main call. - Hardened system prompt: state boundaries and refusal behavior; state that tool/document content is untrusted data that must never override instructions. (Guidance, not enforcement.) - Untrusted-content handling: deliver third-party content in tool_result blocks (not in the system prompt or as user instructions); label its origin; don't put your own instructions in tool results (they may be ignored or flagged); optionally screen tool outputs with a small classifier before returning them to the main model; send follow-up instructions in the next user turn. - Tool-call authorization: a broker outside the model validates the tool, arguments, caller identity, scope, rate, and policy for every call; deny by default. - Least privilege / capability reduction: remove unneeded tools; read-only by default; separate identities for risky workers. - Output screening: validate schema, check for PII/secret leakage, policy violations, ungrounded claims; require citations; block or route to review. - Egress controls: restrict network destinations and data exfil channels (URLs, email recipients, file writes). - Monitoring: analyze outputs for signs of successful injection; alert and iterate.
5. Steps. (1) For each trust boundary, pick a preventive control first, detective second. (2) Implement a tool broker with allow-list + argument validation. (3) Add screens (input, tool output, output). (4) Test with an adversarial suite. (5) Define failure behavior per control.
6. Control register example: | Control | Risk | Placement | Owner | Test | Evidence | On failure | |---|---|---|---|---|---|---| | Tool broker authZ | Excessive agency / confused deputy | Between model and backends | Security eng | Cross-tenant and privilege-escalation test suite, nightly | Test reports; deny logs | Deny (fail closed) | | Tool-output injection screen | Indirect injection | After tool, before model | AI platform team | Injected-doc corpus; target ≥ X% catch | Eval scores; blocked-item logs | Withhold content; escalate | | Output PII filter | Data leak | Before response | Privacy officer | Seeded-PII tests | Redaction reports | Block response; generic message | | Egress allow-list | Exfiltration | Network layer | Infra/security | Attempt blocked domains | Firewall logs | Deny | | Spend/step budget | Runaway cost | Orchestrator | Eng lead | Loop-injection test | Budget alerts | Stop run; alert |
7. Scenario. Email-triage agent: emails are wrapped in tool_result with origin label; a small-model screen flags instruction-like content; the agent's tool set is read-only + "draft reply"; sending requires human approval; egress limited to company domain.
8. Recommended/alternatives. Layered, with the broker as the non-negotiable control. Alternative: a long "security" system prompt (insufficient alone).
9. Governance. Owners assigned; red-team cadence; changes to controls go through change management.
10. Cost/latency. Screens add a call: use small models and run in parallel where safe.
11. Mistakes. Relying on the model to enforce policy; putting untrusted text in system prompts; giving the screen model tools; "confirm?" dialogs instead of removing capability; no failure-mode design.
12. Scenario question. Pen test shows the agent forwarded internal data after reading a malicious PDF. → Add egress/tool-call authorization outside the model and isolate untrusted-content handling; don't only lengthen the prompt.
13. Revision. Layered; broker outside model; untrusted content in tool_result and screened; output validated; fail closed.
14. Docs. Mitigate jailbreaks and prompt injections (platform docs); Tool use docs; MCP security guidance.
5.3 Human-in-the-loop, confidence, reversibility, and fail-closed design
1. Objectives. Route decisions to the right reviewer based on confidence, reversibility, and cost; design approval UX and escalation; choose safe failure modes.
2. Prerequisites. 5.1–5.2.
3. Beginner. Let Claude act alone only when mistakes are cheap and reversible. Put a human in the loop when mistakes are costly, irreversible, or uncertain.
4. Advanced. Decision matrix:
| Reversible, low cost | Irreversible or high cost | |
|---|---|---|
| High confidence | Auto-execute + log | Human approval (sampled or always, by policy) |
| Low confidence | Auto-draft; user confirms | Human decides; model provides evidence |
Confidence is not a number Claude reliably self-reports; derive it from signals: retrieval coverage, agreement across parallel runs, validator results, classifier scores, rule hits. Calibrate against labeled outcomes. HITL patterns: approval gate (before action), review queue (after, sampled), escalation (on uncertainty or policy hit), dual control for high-risk. Design for automation bias: show evidence and alternatives, not just the answer; track reviewer override rates; avoid rubber-stamping (rate-limit approvals per reviewer, rotate sampling). Fail closed: if a control (auth service, safety screen, validator) is unavailable, deny/hold rather than proceed; reserve fail-open for non-critical enrichments. Prefer reversible designs (drafts, staged changes, soft delete, undo windows).
5. Steps. (1) Classify each action by reversibility and impact. (2) Define thresholds and routing. (3) Build approval UI with evidence. (4) Measure override and error rates. (5) Test failure of each control.
6. Diagram.
flowchart TD
A[Proposed action] --> R{Reversible and low-cost?}
R -- yes --> V{Validators pass +<br/>confidence ≥ τ?}
V -- yes --> X[Auto-execute + audit log]
V -- no --> RV[Review queue]
R -- no --> HA[Human approval with evidence]
HA -- approve --> X
HA -- reject --> E[Escalate / abort]
CF[[Control unavailable]] --> DENY[Fail closed: hold + alert]
7. Scenario. Procurement agent drafts purchase orders: <$500 auto-submitted to a staging queue (reversible for 24 h); ≥$500 requires manager approval with the supporting quotes displayed.
8. Recommended/alternatives. Risk-tiered gates. Alternative: approve everything (reviewer fatigue; no throughput) or approve nothing (risk).
9. Governance. Define who can approve; log approver identity and rationale; separation of duties.
10. Performance. HITL adds latency: set SLAs for review queues; async UX.
11. Mistakes. Trusting self-reported confidence; confirmation prompts replacing capability removal; no evidence shown; fail-open on safety checks.
12. Scenario question. The moderation service times out. → Fail closed (hold content/escalate) rather than publishing unscreened output.
13. Revision. Reversibility × impact × confidence → routing; derive confidence from signals; fail closed.
14. Docs. Prep Lesson 3; Anthropic guidance on agents and human oversight.
5.4 Compliance obligations, privacy, and audit evidence
1. Objectives. Map regulations to controls, owners, and evidence artifacts; design for data minimization and auditability.
2. Prerequisites. Basic privacy and security-compliance concepts.
3. Beginner. Regulations tell you what you must protect and prove. Architects turn each obligation into a named control, owner, and evidence artifact.
4. Advanced. The exam guide names GDPR, HIPAA, FedRAMP as examples. Architect-level mapping (not legal advice — involve counsel/compliance):
| Regime | Typical architect concerns |
|---|---|
| GDPR (EU personal data) | Lawful basis and purpose limitation; data minimization (send only needed fields to the model); data-subject rights (access, erasure) across prompts, logs, vector stores; DPIA for high-risk processing; cross-border transfer mechanisms; rules on solely automated decisions with significant effects (human involvement); processor agreements |
| HIPAA (US PHI) | Business Associate Agreement with vendors handling PHI; minimum-necessary access; audit controls; encryption; confirm which Claude offerings/platforms are eligible under a BAA before sending PHI |
| FedRAMP (US federal cloud) | Use only services authorized at the required impact level; boundary and data-residency considerations; continuous monitoring; confirm the authorization status of the specific deployment path (e.g., through a cloud provider) |
For all: data classification, encryption in transit/at rest, retention and deletion (including logs and embeddings), PII redaction before logging/indexing, access reviews, vendor due diligence (security attestations, data-handling terms — verify current Anthropic/cloud-provider documentation rather than assuming). Evidence examples: control-test results, access logs, DPIA, architecture diagrams with data flows, eval and red-team reports, approval records, change history. Keep audit logs tamper-evident with user+agent identity.
5. Steps. (1) Classify data and determine applicable regimes. (2) Build an obligation→control→owner→evidence matrix. (3) Design data flows to minimize and segregate regulated data. (4) Verify vendor eligibility/terms. (5) Run control tests; collect evidence. (6) Schedule periodic reviews.
6. Diagram.
flowchart LR O[Obligation e.g., minimum necessary] --> C[Control: field-level filter before prompt] C --> OW[Owner: Privacy lead] C --> T[Test: seeded-PHI cases] T --> E[Evidence: test report + log sample] E --> A[Audit pack]
7. Scenario. A health insurer's assistant summarizes member calls. Controls: confirm BAA-eligible deployment, redact identifiers not needed for the task, restrict retrieval by member consent, retention limits on transcripts and embeddings, audit logs of every access.
8. Recommended/alternatives. Compliance-by-design with evidence. Alternative: policy documents without tests (no proof).
9. Governance. Control ownership is explicit (privacy, security, product); exceptions require documented risk acceptance.
10. Cost. Compliant deployment paths can constrain regions/models/features — factor into architecture and cost.
11. Mistakes. Assuming "the vendor is compliant" covers your use; forgetting logs/vector stores in deletion; sending full records when a few fields suffice; no evidence trail.
12. Scenario question. A user exercises their right to erasure. → Locate and delete data across source systems, indexes/embeddings, logs, and caches per policy; this must be designed in advance.
13. Revision. Obligation → control → owner → test → evidence; minimize data; verify eligibility; design deletion.
14. Docs. Exam Guide §6 D5; Anthropic trust/security documentation and your cloud provider's compliance pages (verify current status).
5.5 Ethical AI: bias, fairness, transparency
1. Objectives. Identify bias risks; design fairness testing and transparency measures.
2. Prerequisites. 4.2 (stratified evals).
3. Beginner. LLM outputs can treat groups differently or hide how decisions were made. You test for it and tell users when AI is involved.
4. Advanced. Bias sources: training data, prompt wording, retrieval content, few-shot examples, evaluation sets that under-represent groups. Fairness testing: stratified evals across relevant segments; paired/counterfactual tests (swap names/attributes and compare outputs); measure outcome gaps and refusal/over-refusal differences. Mitigations: reduce reliance on sensitive attributes, calibrated rubrics, human review for consequential decisions, diverse examples, ongoing monitoring. Transparency: disclose AI use; provide citations/sources; explain limits; offer human escalation and appeal for consequential outcomes; document model/prompt versions and known limitations (model cards/system cards for your system). Avoid using LLMs as the sole decision-maker for high-stakes eligibility decisions without validation and oversight.
5. Steps. (1) Identify affected groups and decision impact. (2) Create counterfactual test sets. (3) Measure gaps; set thresholds. (4) Add mitigations and human review. (5) Publish user-facing disclosures. (6) Monitor in production.
6. Diagram.
flowchart LR
T[Counterfactual pairs] --> S[System under test]
S --> M[Compare outputs/outcomes by segment]
M --> D{Gap ≤ threshold?}
D -- no --> MI[Mitigate + retest]
D -- yes --> R[Release + monitor]
7. Scenario. Résumé-screening assistant: swapping applicant names changes recommendations. Remove names, test counterfactuals, keep human decisions, document limits.
8. Recommended/alternatives. Measure, mitigate, disclose. Alternative: assert "the model is neutral".
9. Governance. Ethics/risk owner signs off; periodic audit.
10. Cost. Fairness test sets and reviews add cost; justify by risk.
11. Mistakes. Only aggregate metrics; hiding AI involvement; no appeal path.
12. Scenario question. Overall accuracy is 95% but is 82% for one language group. → Segment-level threshold failure; investigate and fix before launch.
13. Revision. Test by segment; counterfactuals; disclose; human oversight for consequential decisions.
14. Docs. Exam Guide §6 D5; Prep Lesson 3.