Part 9d — Question Bank: Domain 4, Evaluation, Testing & Optimization (Q75–Q98)
Original practice questions; not official or leaked. Objectives quoted from Exam Guide v1.0 §6 D4.
Q75 · D4 · Objective: Define evaluation metrics (accuracy, latency, cost, safety, security) · Difficulty: Intermediate
Scenario: A sponsor says the assistant must be "99% accurate." Question: What is the best architect response? A. Agree and promise 99% overall. B. Clarify what "accurate" means per task and segment, how it will be measured and on which golden set, and what an error costs; then record thresholds as acceptance criteria. C. Refuse to give any target. D. Use the vendor's benchmark score as the target. Correct answer: B Explanation: Vague targets need operational definitions and failure-cost context before they become acceptance criteria. Why not the others: A over-commits blindly; C is unhelpful; D isn't specific to your use case. Architectural principle: Specific, measurable, relevant criteria. Ref: https://platform.claude.com/docs/en/test-and-evaluate/define-success
Q76 · D4 · Objective: Define evaluation metrics (accuracy, latency, cost, safety, security) · Difficulty: Advanced
Scenario: A candidate release scores 92% on task accuracy (threshold 90%) but yields three critical policy violations in the 500-case adversarial set (threshold: zero). Question: What is the right decision? A. Ship; accuracy passes. B. Ship with a note to monitor. C. Average the metrics and ship if the mean exceeds 90. D. Block the release: safety is a hard gate; fix and re-run. Correct answer: D Explanation: Safety criteria are pass/fail gates, not tradeable against accuracy. Why not the others: A–C treat a gate as optional. Architectural principle: Multidimensional criteria with non-negotiable safety gates. Ref: Prep Lesson 2–3
Q77 · D4 · Objective: Define evaluation metrics (accuracy, latency, cost, safety, security) · Difficulty: Advanced
Scenario: The dashboard shows mean cost per request of $0.004, within budget, but finance sees occasional $0.50 requests. Question: What should the metrics include? A. Percentiles and distribution of cost and tokens (e.g., p95/p99, max), with alerts and caps on outliers. B. Only the mean. C. Only the minimum. D. Only monthly totals. Correct answer: A Explanation: Tails drive budget and reliability problems that means hide. Why not the others: B–D hide outliers. Architectural principle: Measure distributions, not just averages. Ref: Prep Lesson 2
Q78 · D4 · Objective: Define evaluation metrics (accuracy, latency, cost, safety, security) · Difficulty: Intermediate
Scenario: A legal-summary tool scores well on text-overlap metrics but lawyers say it misses obligations. Question: What is the best change? A. Add more overlap metrics. B. Increase summary length. C. Add a metric aligned to user value — e.g., obligation recall against human-labeled obligations — and use it as a gate. D. Ignore lawyers' feedback. Correct answer: C Explanation: Metrics must reflect the outcome users care about. Why not the others: A and B don't measure what matters; D ignores the signal. Architectural principle: Choose metrics that map to real-world success. Ref: Define success criteria docs
Q79 · D4 · Objective: Define evaluation metrics (accuracy, latency, cost, safety, security) · Difficulty: Intermediate
Question (Select TWO): Which belong in the acceptance criteria for a customer-facing agent? A. Task accuracy measured per customer segment on a golden set. B. The model's public leaderboard rank. C. The count of critical safety violations on an adversarial test set. D. The length of the system prompt. E. The number of tools configured. Correct answers: A, C Explanation: Segment-level quality and safety gate results are outcome-oriented, measurable criteria. Why not the others: B, D, and E are not outcomes for your use case. Architectural principle: Evaluate outcomes in your context. Ref: Define success criteria docs
Q80 · D4 · Objective: Design evaluation datasets and test frameworks using mixed methodologies · Difficulty: Intermediate
Scenario: The eval set consists of 100 clean, typical questions. In production, users send typos, off-topic requests, and adversarial prompts. Question: What should you add? A. More typical questions of the same kind. B. Only synthetic data. C. Nothing; the set is representative enough. D. Edge cases: ambiguous, malformed, irrelevant, multilingual, and adversarial inputs, plus past incidents. Correct answer: D Explanation: Evals must cover the failure modes seen or expected in production. Why not the others: A, B, and C leave gaps. Architectural principle: Evals include edge and adversarial cases. Ref: https://platform.claude.com/docs/en/test-and-evaluate/develop-tests
Q81 · D4 · Objective: Design evaluation datasets and test frameworks using mixed methodologies · Difficulty: Advanced
Scenario: The team iterates on prompts while reporting results on the same 150-case set, and scores keep rising, but production quality doesn't improve. Question: What is the likely problem and fix? A. The set is too small to matter. B. Overfitting to the tuning set; keep a held-out set that prompt authors don't tune against and refresh it from production. C. The model is degrading. D. The graders are too strict. Correct answer: B Explanation: Repeated tuning against a visible set inflates scores. Why not the others: A, C, and D don't explain the divergence. Architectural principle: Held-out evaluation sets. Ref: Prep Lesson 2
Q82 · D4 · Objective: Design evaluation datasets and test frameworks using mixed methodologies · Difficulty: Advanced
Scenario: Overall accuracy is 94%, but accuracy for questions about policies updated this quarter is 61%. Question: What approach exposed this and should be standard? A. Stratified evaluation by segment (e.g., document age, intent, language) with per-segment thresholds. B. Averaging across all cases. C. Reporting only the best-performing segment. D. Using a larger sample of easy cases. Correct answer: A Explanation: Averages hide weak segments. Why not the others: B–D conceal the issue. Architectural principle: Stratify and gate by segment. Ref: Prep Lesson 2
Q83 · D4 · Objective: Design evaluation datasets and test frameworks using mixed methodologies · Difficulty: Intermediate
Scenario: You need to scale the evaluation of open-ended response helpfulness and tone across thousands of outputs. Question: Which method is best? A. Exact string match. B. Count of words. C. An LLM judge with a clear rubric, calibrated against a human-labeled sample, with periodic human spot checks. D. No evaluation. Correct answer: C Explanation: Rubric-based LLM grading scales subjective criteria when calibrated. Why not the others: A and B don't measure quality; D is unacceptable. Architectural principle: Mixed methods: code, LLM, human. Ref: Develop test cases docs
Q84 · D4 · Objective: Design evaluation datasets and test frameworks using mixed methodologies · Difficulty: Advanced
Scenario: An LLM judge consistently prefers longer answers, even when human reviewers prefer concise ones. Question: What is the best mitigation? A. Trust the judge. B. Revise the rubric to penalize unnecessary length, randomize presentation order in pairwise comparisons, and recalibrate against human labels, tracking agreement. C. Make all answers longer. D. Remove human review. Correct answer: B Explanation: Judges have biases (e.g., verbosity); rubric design and calibration reduce them. Why not the others: A, C, and D amplify the bias. Architectural principle: Calibrate automated graders. Ref: Prep Lesson 2
Q85 · D4 · Objective: Design evaluation datasets and test frameworks using mixed methodologies · Difficulty: Intermediate
Scenario: A pipeline extracts invoice totals and dates into JSON, and reference values are known. Question: What is the best grading method? A. Human review of every output. B. An LLM judge reading each output. C. A user satisfaction survey. D. Code-based checks: schema validity and exact value matches (with normalization). Correct answer: D Explanation: Objective, structured outputs should be graded deterministically — fast, cheap, reproducible. Why not the others: A and B are slower and costlier; C is indirect. Architectural principle: Use the cheapest reliable grader that fits. Ref: Develop test cases docs
Q86 · D4 · Objective: Design evaluation datasets and test frameworks using mixed methodologies · Difficulty: Advanced
Scenario: A RAG agent with tools gets low end-to-end scores, and the team can't tell where the failures originate. Question: Which testing structure helps most? A. Component-level evals (retrieval recall, tool selection and arguments, generation faithfulness) plus end-to-end task success. B. Only end-to-end scoring. C. Only manual review of ten examples. D. Only latency monitoring. Correct answer: A Explanation: Component evals localize failures. Why not the others: B–D can't localize. Architectural principle: Evaluate components and the whole. Ref: Exam Guide D4
Q87 · D4 · Objective: Conduct A/B testing and iterative improvements · Difficulty: Intermediate
Scenario: A new prompt wins on offline evals. Leadership wants to know if it improves resolution rates for real customers. Question: What is the best next step? A. Ship to everyone and compare to last quarter. B. Declare victory based on offline results. C. Canary or A/B test on a share of live traffic against the baseline with guardrail metrics and predefined success and rollback criteria. D. Ask the sales team. Correct answer: C Explanation: Online testing measures business outcomes; guardrails protect users. Why not the others: A confounds; B skips online validation; D isn't evidence. Architectural principle: Offline gate, then online experiment. Ref: Prep Lesson 2
Q88 · D4 · Objective: Conduct A/B testing and iterative improvements · Difficulty: Advanced
Scenario: A cheaper model matches accuracy but fails two indirect-prompt-injection cases that the incumbent handles. Question: What should the release process do? A. Block promotion, add mitigations or adjust the design, and re-run the gate. B. Promote because costs drop 40%. C. Remove the injection tests. D. Promote to production but hide the failures from the report. Correct answer: A Explanation: Safety/security gates must pass even when cost improves. Why not the others: B–D bypass or hide the gate. Architectural principle: Evals gate model changes. Ref: Prep Lesson 2–3
Q89 · D4 · Objective: Conduct A/B testing and iterative improvements · Difficulty: Advanced
Scenario: A prompt tweak raises average quality by 2 points but drops one customer segment by 6 points. Question: What is the best decision rule? A. Ship; the average improved. B. Ship and fix later. C. Ignore the segment. D. Apply per-segment thresholds in the gate, investigate the regression, and ship only when no segment breaches tolerance. Correct answer: D Explanation: Segment regressions can be harmful or unfair even when averages improve. Why not the others: A–C hide harm. Architectural principle: Gate on segments. Ref: Prep Lesson 2
Q90 · D4 · Objective: Conduct A/B testing and iterative improvements · Difficulty: Advanced
Scenario: Three days into a planned two-week A/B test, the new variant looks better and a manager wants to stop and ship. Question: What is the sound practice? A. Pre-register the sample size or duration and decision criteria, and avoid stopping early on interim peeks unless using a valid sequential method or a safety breach. B. Stop immediately; early wins are real. C. Extend indefinitely. D. Pick the metric that looks best. Correct answer: A Explanation: Peeking inflates false positives; predefine analysis plans. Why not the others: B invites false conclusions; C and D are unprincipled. Architectural principle: Experimental discipline. Ref: Prep Lesson 2
Q91 · D4 · Objective: Conduct A/B testing and iterative improvements · Difficulty: Intermediate
Scenario: An analyst edited the live prompt in a vendor console; quality dropped and no one can tell what changed. Question: What control prevents this? A. Ask analysts to be careful. B. Disable all monitoring. C. Manage prompts as versioned artifacts in source control with PR review, eval gates, and deployment by version, with console edits restricted. D. Keep a private copy of the prompt. Correct answer: C Explanation: Change control enables audit, rollback, and gates. Why not the others: A, B, and D don't control change. Architectural principle: Evals as release gate with version control. Ref: Prep Lesson 2; D7
Q92 · D4 · Objective: Diagnose system issues (prompt failure, hallucinations, model mismatch) · Difficulty: Intermediate
Scenario: The assistant ignores a length limit that appears at the end of a 3,000-word prompt which also contains conflicting formatting instructions. Question: What is the likely failure type and fix? A. Model mismatch; buy a bigger model. B. Prompt failure; resolve contradictions, simplify and reorder the prompt, and test against the eval set. C. Retrieval failure; re-index. D. Network failure; retry. Correct answer: B Explanation: Contradictory, overlong prompts cause instruction-following issues. Why not the others: The symptoms point to the prompt, not other layers. Architectural principle: Localize before changing. Ref: Prompt engineering overview
Q93 · D4 · Objective: Diagnose system issues (prompt failure, hallucinations, model mismatch) · Difficulty: Advanced
Scenario: When retrieval returns nothing relevant, the assistant invents a plausible policy. Question: Which mitigation set is best? A. Increase temperature. B. Remove the retrieval step. C. Shorten the answer. D. Instruct the model to answer only from provided sources and say "I don't know" otherwise, require citations, validate that citations exist in retrieved text, and route no-evidence cases to a fallback. Correct answer: D Explanation: Grounding, abstention, and verification reduce fabrication. Why not the others: A increases randomness; B removes grounding; C doesn't address fabrication. Architectural principle: Ground, abstain, verify. Ref: Reduce hallucinations docs
Q94 · D4 · Objective: Diagnose system issues (prompt failure, hallucinations, model mismatch) · Difficulty: Expert
Scenario: Quality shifted overnight, with no code changes. Your service references a floating model alias. Question: What is the best long-term fix? A. Pin explicit model versions, and promote new versions only after passing evals with a canary and rollback. B. Ask the provider to freeze models. C. Stop using the API. D. Add more examples to every prompt. Correct answer: A Explanation: Floating references can change behavior without your deployment; pinning plus gates restores control. Why not the others: B is not in your control; C is disproportionate; D doesn't prevent drift. Architectural principle: Control change: pin and gate. Ref: Model deprecations and versioning docs
Q95 · D4 · Objective: Diagnose system issues (prompt failure, hallucinations, model mismatch) · Difficulty: Advanced
Question (Select TWO): Confident wrong answers began after a nightly index refresh. Which are the best first diagnostic steps? A. Check index freshness, document counts, and sample the retrieved chunks for known queries. B. Raise generation temperature. C. Compare retrieval results for a fixed set of test queries before and after the refresh. D. Rewrite the system prompt. E. Replace the generation model. Correct answers: A, C Explanation: The refresh is the change; test retrieval directly. Why not the others: B, D, and E alter unrelated layers. Architectural principle: Start where the change occurred. Ref: Prep Lesson 2
Q96 · D4 · Objective: Optimize token usage, latency, and cost-performance trade-offs · Difficulty: Advanced
Scenario: Cost per request doubled after a prompt edit. Tokens per request rose and cache read tokens fell to near zero. Question: What is the most probable cause? A. Users asked longer questions. B. The provider raised prices. C. The edit changed prompt ordering or inserted volatile content early, breaking the cached prefix. D. The eval set grew. Correct answer: C Explanation: The dropped cache reads point to a broken prefix. Why not the others: The symptoms specifically implicate caching. Architectural principle: Monitor cache hit rate and tokens per request. Ref: Prompt caching docs
Q97 · D4 · Objective: Optimize token usage, latency, and cost-performance trade-offs · Difficulty: Intermediate
Scenario: A workflow calls three independent tools sequentially, adding latency. Question: What is the best optimization? A. Add a longer prompt. B. Run the independent tool calls in parallel, then continue. C. Use a smaller context window. D. Remove tools at random. Correct answer: B Explanation: Parallelizing independent calls reduces wall-clock time. Why not the others: A, C, and D don't address sequential latency. Architectural principle: Parallelize independent work. Ref: Reduce latency docs
Q98 · D4 · Objective: Monitor system performance using logging and observability tools · Difficulty: Advanced
Scenario: Running an LLM judge over every production output is too expensive. Question: Which approach keeps quality visibility at manageable cost? A. Stop quality monitoring. B. Judge only outputs from the most expensive customers. C. Judge only on weekends. D. Sample live traffic (oversampling flagged or low-confidence cases), use a smaller judge or batch processing where appropriate, cache the stable rubric, and track trends and alerts. Correct answer: D Explanation: Sampling and cost-aware grading preserves signal. Why not the others: A–C create blind spots. Architectural principle: Cost-aware quality monitoring. Ref: Batch processing docs; Prep Lesson 2