Part 4 — Domain 4: Evaluation, Testing & Optimization (16%)
No dedicated prep-course lesson; the course objective "build evaluations as acceptance criteria and use them as the gate before any model or architecture change" touches it. Official tasks: define metrics (accuracy, latency, cost, safety, security) · eval datasets and test frameworks with mixed methodologies · A/B testing and iteration · diagnose issues (prompt failure, hallucination, model mismatch) · optimize tokens/latency/cost-performance · monitor via logging and observability.
Topics: 4.1 Success criteria and metrics · 4.2 Eval datasets and mixed-method frameworks · 4.3 Regression gates, A/B testing, release decisions · 4.4 Diagnosing failures and optimizing
4.1 Defining success criteria and metrics
1. Objectives. Convert business goals into specific, measurable criteria across accuracy, latency, cost, safety, and security; set thresholds and tolerances.
2. Prerequisites. D1 success-criteria step.
3. Beginner. Before building, decide what "good" means in numbers. Anthropic's docs say good criteria are specific, measurable, achievable, and relevant, and that most use cases need multidimensional evaluation.
4. Advanced. A criteria set for a support-reply drafter might be: task fidelity (≥ 92% of replies judged correct and policy-compliant on the golden set), safety (0 critical policy violations in 1,000 adversarial prompts), latency (p95 < 3 s), cost (≤ $0.02 per reply), consistency (same-intent paraphrases get equivalent answers), and escalation rate bounds. Include non-functional gates (cost, latency) and safety/security gates (injection resistance, data-leak tests) as pass/fail, not "nice to have". Distinguish leading indicators (offline eval scores) from lagging (CSAT, deflection, rework). Set a minimum acceptable and a target; define how much regression is tolerable per metric. Avoid unrealistic criteria beyond current frontier capability.
5. Steps. (1) Interview for outcomes and failure costs. (2) Write 5–8 criteria with owner, metric, threshold, measurement method. (3) Agree on them with stakeholders (D6). (4) Freeze them as the acceptance contract.
6. Template. | Dimension | Metric | Method | Threshold | Gate? | Owner | |---|---|---|---|---|---| | Accuracy | % correct vs. golden | Code + LLM-judge + human sample | ≥ 92% | Yes | Product | | Safety | Critical violations / 1,000 adversarial | Red-team set | 0 | Yes | Security | | Latency | p95 end-to-end | Load test | < 3 s | Yes | Eng | | Cost | $/request | Usage logs | ≤ $0.02 | Yes | Finance/Eng | | Fairness | Metric gap across segments | Stratified eval | ≤ X pts | Yes | Risk |
7. Scenario. A legal-summary tool optimizes ROUGE but lawyers complain of missed obligations. Criteria should measure obligation recall (rubric/human-labeled), not text overlap.
8. Recommended/alternatives. Multi-metric acceptance contract. Alternative: single "accuracy" number (hides cost, safety, tail failures).
9. Governance. Safety and security criteria are owned outside the build team; thresholds are recorded as evidence.
10. Cost/latency. Report cost and latency distributions, not only means.
11. Mistakes. Vague criteria ("good answers"); metrics that don't reflect user value; no tail-risk metrics; changing thresholds after seeing results.
12. Scenario question. Stakeholders want "99% accurate." → Clarify per task, per segment, measured how, against which golden set, and what the failure cost is.
13. Revision. Specific, measurable, multidimensional, gated, agreed up front.
14. Docs. https://platform.claude.com/docs/en/test-and-evaluate/define-success
4.2 Evaluation datasets and mixed-method test frameworks
1. Objectives. Build representative eval sets; choose code-based, LLM-based, and human grading appropriately; evaluate retrieval, generation, tools, and end-to-end separately.
2. Prerequisites. 4.1.
3. Beginner. An eval is a set of test inputs with expected outputs or grading rules. You run your system on them to get scores you can compare across versions.
4. Advanced. Dataset design: sample from real traffic (with consent/redaction), cover each intent/segment, include edge cases (ambiguous inputs, adversarial/jailbreak attempts, irrelevant or harmful input, long/short, multilingual, missing data), hard negatives, and known past incidents. Use stratified sampling so rare-but-critical cases aren't drowned. Maintain golden (human-verified) sets, plus a larger synthetic set (Claude can help generate variations from a seed set — verify quality). Keep a held-out set that prompt authors don't tune against.
Grading methods (mix them): - Code-graded (exact match, schema validity, unit tests, regex, tool-call correctness) — fast, cheap, deterministic; best where an objective answer exists. - LLM-graded (rubric-based judge) — scalable for open-ended quality; give a clear rubric, ask for reasoning before the verdict, use a different/stronger judge where possible, calibrate against human labels, and watch for judge bias (verbosity, position, self-preference). - Human-graded — gold standard for subjective/high-stakes; expensive; use for calibration and sampling.
Component-level evals: retrieval (recall@k, precision), tool selection and argument correctness, generation quality/faithfulness (are claims supported by retrieved text?), safety (injection and policy tests), and end-to-end task success. Component evals let you localize failures.
5. Steps. (1) Draft 50–100 cases per major intent from real data. (2) Add edge/adversarial sets. (3) Define graders per criterion. (4) Calibrate LLM judge vs. human on ~100 samples; track agreement. (5) Automate runs in CI. (6) Refresh the set from production failures.
6. Diagram.
flowchart LR
P[Prod traffic + incidents] --> D[Eval dataset<br/>golden + adversarial + held-out]
D --> R[Run system version N]
R --> G1[Code graders]
R --> G2[LLM judge]
R --> G3[Human sample]
G1 & G2 & G3 --> S[Scorecard by criterion + segment]
S --> GATE{Meets thresholds?}
7. Scenario. RAG assistant scores 94% overall but 61% on questions about policies updated this quarter. Stratified results exposed the freshness issue masked by the average.
8. Recommended/alternatives. Mixed-method, stratified. Alternative: only vibes/manual spot checks (non-repeatable), or only LLM-judge (uncalibrated).
9. Security. Include prompt-injection, data-exfiltration, and authorization-bypass cases as first-class tests; protect eval data containing PII.
10. Cost. LLM-judge runs cost tokens: sample, cache static rubric/prefix, use batch for large offline runs.
11. Mistakes. Tuning on the test set; unrepresentative data; unreliable judge; averaging away tail failures; no ownership of dataset upkeep.
12. Scenario question. LLM-judge scores rise but human reviewers disagree. → Recalibrate the judge/rubric against human labels; add human sampling.
13. Revision. Representative + edge + held-out; code/LLM/human mix; component and end-to-end; stratify.
14. Docs. https://platform.claude.com/docs/en/test-and-evaluate/develop-tests · Console Evaluation Tool docs.
4.3 Regression gates, A/B testing, and release decisions
1. Objectives. Make evals the gate for any prompt/model/architecture change; run A/B tests and canaries safely; decide ship/hold/rollback.
2. Prerequisites. 4.1–4.2.
3. Beginner. Any change — new prompt, new model, new retrieval setting — can break something. A regression gate runs the eval suite and blocks release if thresholds are missed. A/B testing compares versions on live traffic.
4. Advanced. Pipeline: change → offline eval (full suite + safety + cost/latency) → compare against baseline with tolerance per metric → shadow or canary (small % traffic) → monitor guardrail metrics → staged rollout → rollback criteria pre-defined. Offline wins don't guarantee online wins; use online A/B for user-visible outcomes (resolution rate, CSAT, edit distance by humans). Use sufficient sample sizes and avoid peeking; segment results. Model upgrades and provider changes are treated identically to code changes. Keep prompt/config versioned and linked to the eval run that approved it.
5. Steps. (1) CI job runs evals on every prompt/config/model change. (2) Block on gate failures. (3) Canary with automated rollback on guardrail breach. (4) A/B on business metrics. (5) Record decisions (ADR/changelog) with eval IDs.
6. Diagram.
flowchart LR C[Change: prompt/model/retrieval] --> E[Offline evals] E -->|pass| SH[Shadow / canary 5%] E -->|fail| X[Block + fix] SH -->|guardrails ok| AB[A/B on business metrics] SH -->|breach| RB[Auto rollback] AB --> PR[Promote + record evidence]
7. Scenario. Team wants to move to a newer, cheaper model. Offline: accuracy equal, cost −40%, but safety set shows two new failures on indirect-injection cases. Gate blocks; mitigation added; re-run passes; canary then rollout.
8. Recommended/alternatives. Eval-gated, canaried release. Alternative: "it looked fine in the playground" (no evidence).
9. Governance. Gate results are audit evidence; sign-offs from risk/security for safety metrics.
10. Cost/latency. Gate on cost and p95 along with quality, to avoid "better but unaffordable".
11. Mistakes. No baseline; averaging hides regressions; one-metric gates; no rollback plan; peeking at A/B results early.
12. Scenario question. A prompt tweak improves average quality but one customer segment worsens. → Gate on per-segment thresholds; investigate before shipping.
13. Revision. Evals gate every change; canary; A/B for business outcomes; rollback predefined; evidence recorded.
14. Docs. Prep Lesson 2 (evaluation gates); Console Evaluation Tool.
4.4 Diagnosing failures and optimizing performance
1. Objectives. Localize failures to prompt, retrieval, tool, model-mismatch, or hallucination causes; optimize tokens, latency, and cost.
2. Prerequisites. 4.2, D2, D3.
3. Beginner. When answers are wrong, ask where in the pipeline it went wrong before changing anything.
4. Advanced — a diagnostic ladder:
1. What changed? (deploy, prompt, model, index refresh, data, traffic) — recent change is the prime suspect.
2. Inspect the trace: was the right context retrieved? Were tools called correctly? Was the output truncated (max_tokens)?
3. Retrieval problem (stale/irrelevant/missing chunks) → fix indexing/chunking/filters, not the prompt.
4. Prompt failure (ambiguous instructions, conflicting rules, missing format/edge-case examples) → clarify, add examples, test.
5. Hallucination (unsupported claims) → require citations from provided text, allow "I don't know", verify claims against sources, reduce irrelevant context, add output validation.
6. Model mismatch (task exceeds model capability or needs different features) → test a stronger tier or decompose the task.
7. Tool/integration problems (schema drift, bad arguments, timeouts).
Only after localizing do you change one variable and re-run evals.
Optimization levers: reduce input tokens (prune context, concise tool outputs), cache static prefixes, route to cheaper models, batch non-interactive work, cap output length, parallelize independent calls, stream for perceived latency, shorten chains. Always re-run the quality gate after optimizing.
5. Steps. Reproduce with a saved trace → bisect (retrieval, prompt, model, tools) → fix → add a regression test from the failure → deploy via gate.
6. Diagram.
flowchart TD
F[Wrong/low-quality output] --> W{Recent change?}
W -- yes --> RV[Review change / rollback]
W -- no --> T[Inspect trace]
T --> R{Right context retrieved?}
R -- no --> IX[Fix retrieval/index]
R -- yes --> P{Prompt unambiguous + examples?}
P -- no --> PR[Fix prompt]
P -- yes --> H{Unsupported claims?}
H -- yes --> G[Grounding + citations + validation]
H -- no --> M[Try stronger model / decompose]
7. Scenario (mirrors official Sample 3's logic). Confident-but-wrong answers right after a document refresh, model and latency unchanged → retrieval/indexing first. Not weights, temperature, or shrinking context.
8. Recommended/alternatives. Evidence-led root cause. Alternative: "try a bigger model" first (costly, may not fix retrieval).
9. Security. Include security regressions (new injection success) in diagnostics; preserve evidence.
10. Cost/latency. Compare quality-per-dollar before and after each optimization.
11. Mistakes. Changing several things at once; fixing prompts for retrieval bugs; no regression test added; optimizing without a quality gate.
12. Scenario question. p95 latency doubled after adding few-shot examples. → Check token growth and caching; move examples into a cached static prefix or trim.
13. Revision. What changed → trace → localize → single change → regress-test → gate.
14. Docs. Reduce latency (platform docs); Reduce hallucinations (platform docs); Prompt caching.