AI Evals for Product Managers
How to measure, improve and safely launch AI features. 10 classes from basic to advanced, with templates, a metrics cheat sheet and interview questions.
AI Evals for Product Managers
How to measure, improve and safely launch AI features: success criteria, golden datasets, LLM-as-judge, error analysis, RAG and agent evals, safety and red-teaming, A/B tests and monitoring, tools, templates and 15 interview questions.
What an eval is, in one line
An eval is a repeatable test that measures how well an AI feature does its job on a fixed set of examples, with a clear way to score each answer. It's the quality bar, the launch gate and the product spec for AI, and in AI teams the PM owns it.
The 10 classes
What an eval is, and why PMs own it
Understand evals as the unit test, the quality bar and the product spec for AI features.
- An eval = fixed examples + a scoring method + a pass bar.
- PMs define what 'good' means; evals turn that into numbers.
- Run evals on every change, not once.
Defining success: criteria, rubrics and failure modes
Turn a fuzzy goal ('helpful answers') into criteria anyone can score the same way.
- Prefer binary pass/fail criteria with clear definitions.
- Mark severity: one critical failure can block launch.
- Write the bar in the PRD before running the eval.
Building the eval dataset (golden set)
Create a test set that represents real users, including the hard and risky cases.
- Real user data + expert questions + synthetic variations + adversarial cases.
- Cover slices (intent, difficulty, language, risk), not just averages.
- 100–300 examples for a launch decision; keep a held-out set.
Grading methods: code, humans and LLM-as-judge
Choose the cheapest grader that measures what matters, and know when to use humans.
- Use code checks wherever possible; they are cheap and exact.
- LLM judges scale open-ended grading but must be calibrated against humans.
- One criterion per judge, binary verdict, reasoning first.
Error analysis: finding what actually breaks
Read outputs systematically, name the failure modes and fix the biggest ones first.
- Read real outputs before building metrics.
- Open coding → categories → counts → fix the top 3.
- Every fixed failure becomes a permanent test case.
Evaluating RAG (retrieval-augmented generation)
Measure retrieval and generation separately, so you know which half to fix.
- Score retrieval and generation separately.
- Faithfulness (no unsupported claims) is the key RAG safety metric.
- Use the 2×2 to decide whether to fix search or the prompt.
Evaluating agents and tool use
Measure multi-step AI agents on outcomes, the path they took, cost and safety.
- Evaluate outcome, trajectory and efficiency.
- Use sandboxes and check final state with code.
- Run tasks several times; reliability matters as much as average success.
Safety, responsible AI and red-teaming
Test for harm, privacy leaks, bias and jailbreaks before attackers and journalists do.
- Safety bars are usually zero-tolerance; one leak can block launch.
- Test over-refusal too: an AI that refuses everything is useless.
- Red-team findings become permanent eval cases.
Online evals: A/B tests, monitoring and regression gates
Connect offline scores to real user value and catch quality drops in production.
- Offline evals gate releases; online evals prove value.
- Use implicit signals (edits, re-asks, escalations), not only thumbs.
- Automate regression gates and live sampling.
Eval ops: tools, cost, the AI PRD and your capstone
Run evals as a product practice: tooling, budgets, documentation and team rituals.
- Start in a spreadsheet; adopt tools when volume grows.
- Put evals in the PRD: criteria, data, graders, bars, monitoring.
- Make evals a weekly team habit, not a one-off project.
Interview questions
What is an eval and why does a PM care?
A repeatable test of AI quality on fixed examples with a scoring method. PMs define 'good', set launch bars and use evals to make model/cost trade-offs.
How would you evaluate an AI customer-support bot?
Define criteria (correct, grounded, private, resolves, escalates, tone); build a sliced golden set from real tickets; grade with code + calibrated LLM judges; set bars; then measure resolution rate, CSAT and escalations online.
What is LLM-as-judge and its risks?
Using a model to grade outputs against a rubric. Risks: position, verbosity and self-preference bias, leniency. Mitigate with binary criteria, swapping order, a different judge model, and calibration against human labels (kappa ≥ 0.6).
How many examples do you need?
30–50 to find obvious issues, 100–300 for a launch decision, more for fine comparisons. Margin of error ≈ 1.96√(p(1−p)/n).
How do you evaluate a RAG system?
Separately: retrieval (recall@k, MRR) and generation (faithfulness, relevance, citation accuracy). Use the 2×2 of right/wrong docs vs right/wrong answer to locate the fix.
How do you evaluate an AI agent?
Outcome (task success via end-state checks), trajectory (right tools and arguments, no forbidden actions) and efficiency (cost, steps, latency), in a sandbox, with repeated runs for reliability.
Offline vs online evals?
Offline: fixed dataset before release, used as gates. Online: real users after release via A/B tests, implicit signals and sampled judging.
A new model is 40% cheaper. Do you switch?
Run the full eval suite and slices; check critical criteria and safety; compare cost per successful task, not per call; A/B test on a slice of traffic with guardrails.
