An LLM evaluation pipeline produces misleading conclusions and hides critical failures when teams report a single uncalibrated average score before locking the task set, golden dataset error bars, inter-grader agreement, and disaggregated failure slices. In agent evaluation, a task (problem or test case) is a single test with defined inputs and success criteria, and each attempt at a task is a trial because model outputs vary between runs.
For engineering teams building evaluation suites across our LLM Reliability coverage—while budgeting production trace ingestion and retention separately through the AI Agent Observability Cost Comparison Worksheet—this guide provides the task-set specifications, uncertainty and sample-size equations, chance-corrected grader agreement formulas, and subgroup disaggregation rules required before trusting a headline metric.
| Evaluation Element | Primary Layer | Grounded Engineering Specification |
|---|---|---|
| Evaluation ElementGrader and transcript | Primary LayerTrial scoring | Grounded Engineering Specification A grader scores some aspect of an agent's performance with assertions (checks), a transcript (trace or trajectory) records the trial's outputs and tool calls, and the outcome is the final environment state. |
| Evaluation ElementHarness and suite | Primary LayerExecution runner | Grounded Engineering Specification An evaluation harness runs evals end-to-end (providing tools, running tasks concurrently, recording steps, grading outputs, and aggregating results), while an evaluation suite is a collection of tasks. |
| Evaluation ElementCode-based graders | Primary LayerDeterministic checks | Grounded Engineering Specification Agent evaluations combine code-based, model-based, and human graders; code-based graders (string match, binary tests, static analysis, outcome and tool verification) are fast, cheap, objective, and reproducible, but brittle to valid variations. |
| Evaluation ElementModel and human graders | Primary LayerCalibrated judgment | Grounded Engineering Specification Model-based graders (rubric scoring, natural language assertions, pairwise comparison, reference-based, multi-judge consensus) are flexible and scalable but non-deterministic and require calibration with human graders, who provide gold-standard quality and inter-annotator agreement. |
| Evaluation ElementCapability vs regression | Primary LayerSuite lifecycle | Grounded Engineering Specification Capability evals start at a low pass rate on tasks the agent struggles with, whereas regression evals should have a nearly 100% pass rate to protect against backsliding; high-pass capability evals can graduate into a continuous regression suite. |
Defining the LLM Evaluation Task Set and Trial Consistency
Non-deterministic generation requires explicit trial aggregation metrics rather than single-run point checks.
The pass@k metric measures the likelihood that an agent gets at least 1 correct solution in k attempts; as k increases, pass@k rises, and a score of 50% pass@1 means the model succeeds at half the tasks on its first try.
The pass^k metric measures the probability that all k trials succeed: for a 75% per-trial success rate across 3 trials, the probability of passing all three is (0.75)³ ≈ 42%, and while pass@k and pass^k are identical at k=1, by k=10 pass@k approaches 100% while pass^k falls to 0%.
A grader scores some aspect of an agent's performance with assertions (checks), a transcript (trace or trajectory) records the trial's outputs and tool calls, and the outcome is the final environment state.
Locking a reliable LLM evaluation dataset requires six task-quality and harness gates before interpreting model scores:
- For an initial evaluation dataset, 20-50 simple tasks drawn from real failures is a practical starting point because early system changes have a large effect size where small sample sizes suffice (an 80/20 approach).
- A good evaluation task is one where two domain experts independently reach the same pass/fail verdict, and with frontier models a 0% pass rate across many trials (0%
pass@100) is most often a signal of a broken task rather than an incapable agent. - Balanced problem sets must test both cases where a behavior should occur and where it shouldn't to avoid class-imbalanced evals that cause one-sided optimization (undertriggering vs. overtriggering).
- Each trial in an evaluation harness should be isolated by starting from a clean environment because shared state (leftover files, cached data, git history, or resource exhaustion) causes correlated failures or artificially inflates scores.
- LLM-as-judge graders should be calibrated with human experts, given an instruction to return
'Unknown'when lacking information to avoid hallucinations, and separated into isolated LLM-as-judge calls per rubric dimension. - On CORE-Bench, Claude Opus 4.5 initially scored 42% due to rigid grading (penalizing
'96.12'when expecting'96.124991...'), ambiguous task specs, and stochastic tasks; after fixing bugs and using a less constrained scaffold, Opus 4.5's score jumped to 95%.
Open-source evaluation frameworks map these task, solver, and scorer primitives directly into executable code:
| Framework Component | Execution Scope | Verified Architecture and Built-in Scorers |
|---|---|---|
| Framework ComponentInspect Task model | Execution ScopeDataset and solver | Verified Architecture and Built-in Scorers In the UK AI Security Institute's Inspect framework, an evaluation Task combines three components: a Dataset (samples with input and target columns), a Solver (producing an answer per sample), and a Scorer (evaluating the output). |
| Framework ComponentInspect text scorers | Execution ScopeDeterministic matching | Verified Architecture and Built-in Scorers Inspect provides built-in text-matching scorers including |
| Framework ComponentInspect model scorers | Execution ScopeModel-graded QA | Verified Architecture and Built-in Scorers Inspect includes built-in model-graded scorers |
| Framework ComponentPydantic Evals | Execution ScopeTrajectory grading | Verified Architecture and Built-in Scorers Pydantic Evals follows a code-first approach where datasets, experiments, tasks, cases, and evaluators are defined in Python, grading both an agent's final outputs and its trajectory (the sequence and arguments of its tool calls). |
| Framework ComponentEleutherAI harness | Execution ScopeIntegrity and logging | Verified Architecture and Built-in Scorers EleutherAI's Language Model Evaluation Harness provides over 60 standard academic benchmarks for LLMs with publicly available prompts, and supports |
Golden Sample Sizing, Clustered Standard Errors, and Paired Power Math
Reporting raw benchmark percentages without standard errors conceals sampling noise and within-cluster correlation.
For independent evaluation questions, the Central Limit Theorem standard error of the mean score is SE_C.L.T. = sqrt(Var(s)/n), which simplifies for binary (0 or 1) scores to SE_Bernoulli = sqrt(s_bar * (1 - s_bar) / n), while SE_Bernoulli is conservative (too wide) when scores take fractional values like F1.
Evan Miller suggests reporting n and the SE in parentheses beside the mean; a 95% CI may be computed as CI_95% = s_bar ± 1.96 × SE_C.L.T.
When each evaluation question is resampled (answered) K times and scored by the question-level mean of those K answers, the conditional variance becomes Var(s_i) = sigma_i^2 / K; pooling all KN answers as if independent is inconsistent because multiple answers to the same question violate independent draws.
When multiple evaluation questions share a reading passage (or are translated variants of one item), naive independent-draw formulas underestimate uncertainty:
- When evaluation questions are grouped into clusters
cwith question scoress_{i,c}and independent draws of clusters, the cluster-adjusted standard error of the mean score isSE_clustered = (SE_C.L.T.^2 + (1 / n^2) * sum_c sum_i sum_{j != i} (s_{i,c} - s_bar)(s_{j,c} - s_bar))^{1/2}. - On real-world evaluations using Anthropic models (Table 4), clustered standard errors can be over 3X larger than naive standard errors: on DROP,
SE_clusteredis 1.34 vs.SE_C.L.T.of 0.44 (ratio 3.05); on RACE-H, 0.51% vs. 0.46% (ratio 1.10); and on MGSM, 1.62% vs. 0.86% (ratio 1.88). - Under the law of total variance, the variance of an eval mean score decomposes as
Var(mu_hat) = (Var(x) + E[sigma_i^2]) / n, whereVar(x)is the immutable cross-question variance of conditional means andE[sigma_i^2]is the expected within-question generation variance. - When comparing two models A and B evaluated on the exact same set of
nquestions, using the paired difference scores_{A-B,i} = s_{A,i} - s_{B,i}yields a paired standard errorSE_{A-B,paired} = sqrt(Var(s_{A-B})/n)that reduces estimator variance whenever question scores are positively correlated across models. - In power analysis for paired model comparisons, detecting an absolute score difference of
delta = 0.03with 80% statistical power (beta = 0.20), a 5% false-positive rate (alpha = 0.05),sigma_A^2 = sigma_B^2 = 0, andomega^2 = 1/9requires at leastn = (z_{0.025} + z_{0.20})^2 (1/9) / (0.03)^2 ≈ 969independent questions.
Inter-Grader Agreement and LLM-as-a-Judge Calibration
Raw percentage agreement between raters fails to account for skewed label distributions where two annotators or judges agree by chance.
Cohen's kappa measures inter-annotator agreement between two annotators on a classification problem and is defined as kappa = (p_o - p_e) / (1 - p_e), where p_o is the empirical probability of agreement (observed agreement ratio) and p_e is the expected agreement under per-annotator empirical priors.
The Cohen's kappa statistic is symmetric (swapping y1 and y2 does not change the value), supports optional 'linear' or 'quadratic' weighting, and returns a float between -1.0 and 1.0, where the maximum value means complete agreement, the minimum value means complete disagreement, and 0.0 indicates no agreement beyond chance.
- In scikit-learn's worked example for
cohen_kappa_scoreon categorical label arrays wherey1 = ['negative', 'positive', 'negative', 'neutral', 'positive']andy2 = ['negative', 'positive', 'negative', 'neutral', 'negative'],cohen_kappa_score(y1, y2)returns 0.6875. - In
statsmodels.stats.inter_rater.cohens_kappa, a 2-Dim square array with one rater in rows and the second rater in columns is used to compute simple kappa or weighted kappa using Cicchetti-Allison ('ca'or'linear') or Fleiss-Cohen ('fc'or'quadratic') weights. - For more than two raters,
statsmodels.stats.inter_rater.fleiss_kappatakes a 2-D table (subjects in rows, categories in columns containing the number of raters who assigned each category);method='fleiss'uses the sample margin as the chance benchmark, whereasmethod='randolph'(or'uniform') assumes a uniform distribution across categories and satisfies Randolph's kappa >= Fleiss' kappa.
A good evaluation task is one where two domain experts independently reach the same pass/fail verdict, and with frontier models a 0% pass rate across many trials (0% pass@100) is most often a signal of a broken task rather than an incapable agent.
When calibrating automated LLM-as-a-judge evaluators against human experts, engineering teams must test for systematic judge biases and criteria drift:
| Calibration Dimension | Evaluation Setup | Verified Empirical Measurement and Alignment Rule |
|---|---|---|
| Calibration DimensionMT-bench structure | Evaluation SetupMulti-turn benchmark | Verified Empirical Measurement and Alignment Rule MT-bench is a benchmark consisting of 80 high-quality multi-turn questions across 8 common categories of user prompts (writing, roleplay, extraction, reasoning, math, coding, knowledge I STEM, and knowledge II humanities/social science), with 10 multi-turn questions for each category. |
| Calibration DimensionJudge architectures | Evaluation SetupScoring modes | Verified Empirical Measurement and Alignment Rule MT-bench defines 3 LLM-as-a-judge variations: Pairwise comparison (presenting two answers to determine which is better or a tie, though the number of possible pairs grows quadratically), Single answer grading (assigning a score directly to a single answer), and Reference-guided grading (providing a reference solution when applicable). |
| Calibration DimensionPosition bias | Evaluation SetupSwapped assistant order | Verified Empirical Measurement and Alignment Rule In MT-bench position-bias testing (Table 2), where consistency is the percentage of cases a judge gives the same verdict when swapping the order of two assistants, under the default prompt Claude-v1 achieved 23.8% consistency (75.0% biased toward first), GPT-3.5 achieved 46.2% consistency (50.0% biased toward first), and GPT-4 achieved 65.0% consistency (30.0% biased toward first). |
| Calibration DimensionVerbosity bias | Evaluation SetupRepetitive-list attack | Verified Empirical Measurement and Alignment Rule To test verbosity bias (when an LLM judge favors longer, verbose responses over shorter accurate ones), Zheng et al. applied a 'repetitive list' attack to 23 MT-bench answers (Table 3): Claude-v1 and GPT-3.5 both showed a 91.3% failure rate, whereas GPT-4 showed an 8.7% failure rate. |
| Calibration DimensionSelf-enhancement bias | Evaluation SetupWin-rate shift | Verified Empirical Measurement and Alignment Rule In MT-bench self-enhancement bias analysis comparing model win rates (without ties) under LLM judges vs. humans, GPT-4 favored itself with a 10% higher win rate and Claude-v1 favored itself with a 25% higher win rate, while GPT-3.5 did not favor itself. |
| Calibration DimensionMath reference grading | Evaluation SetupPrompt intervention | Verified Empirical Measurement and Alignment Rule On 10 math questions with swapped positions (20 evaluations total, Table 4), GPT-4's judge failure rate dropped from 14/20 (70%) under the Default prompt to 6/20 under Chain-of-Thought (CoT) and 3/20 (15%) under Reference-guided grading. |
| Calibration DimensionTie vs non-tie baselines | Evaluation SetupAgreement setups | Verified Empirical Measurement and Alignment Rule Zheng et al. define agreement between two types of judges as the probability of randomly selected individuals (not identical) of each type agreeing on a randomly selected question, reporting setup S1 (which includes non-tie, tie, and position-inconsistent votes, random baseline R = 33%) and setup S2 (non-tie votes only, random baseline R = 50%). |
| Calibration DimensionHuman expert agreement | Evaluation SetupTurn one evaluation | Verified Empirical Measurement and Alignment Rule On MT-bench First Turn (Table 5a, with 58 expert-level human labelers), under setup S2 (non-tie, R = 50%), GPT-4 pairwise (G4-Pair) and single-answer (G4-Single) both reached 85% agreement with humans compared to 81% human-human agreement; under setup S1 (with ties, R = 33%), G4-Pair reached 66% and G4-Single reached 60% with humans compared to 63% human-human agreement. |
| Calibration DimensionCriteria drift | Evaluation SetupRubric externalization | Verified Empirical Measurement and Alignment Rule In EvalGen, Shankar et al. identify 'criteria drift' in LLM evaluation: users need criteria to grade LLM outputs, but grading outputs helps users define criteria, creating a catch-22 where externalizing evaluation criteria depends on observing and grading outputs. |
| Calibration DimensionValidator coverage | Evaluation SetupAssertion alignment | Verified Empirical Measurement and Alignment Rule Across 84 medical LLM outputs (68% good, 27 bad outputs) and 100 product LLM outputs (51% good, 49 bad outputs), providing EvalGen with 16 human-graded outputs per pipeline produced 4 product assertions achieving 0.73 coverage (73%) at 0.39 FFR vs. SPADE's 9 assertions achieving 0.49 coverage (49%). |
Disaggregating Failure Slices and Pre-Deployment Controls
Collapsing heterogeneous prompt categories, user cohorts, or tool trajectories into a single overall score conceals concentrated subgroup failures.
In Fairlearn, disaggregated metrics evaluate model performance on slices of data via fairlearn.metrics.MetricFrame, which recommends including count() alongside base performance metrics (such as recall_score or false_positive_rate) to help identify subgroups where small sample counts make random noise significant.
In Fairlearn, MetricFrame exposes overall metrics on the entire dataset via MetricFrame.overall, subgroup slice metrics via MetricFrame.by_group, worst and best slice performance via group_min() and group_max(), and slice disparities via MetricFrame.difference() and MetricFrame.ratio() (either between extreme groups by default or with method='to_overall').
To prevent benchmark gaming and unverified pre-deployment claims, apply the six governance and measurement controls from the NIST Artificial Intelligence Risk Management Framework Generative AI Profile:
- NIST Generative AI Profile Action MS-1.1-003 directs teams to disaggregate evaluation metrics by demographic factors to identify discrepancies across diverse populations, and Action MS-1.3-003 directs teams to verify that those conducting structured human feedback exercises are not directly involved in system development tasks for the same GAI model.
- NIST Generative AI Profile Action MS-2.5-001 directs teams to avoid extrapolating GAI system performance or capabilities from narrow, nonsystematic, and anecdotal assessments, and Action MS-2.8-002 directs teams to document the instructions given to data annotators or AI red-teamers.
- NIST Generative AI Profile Action MS-2.11-001 directs teams to document assumptions and limitations of benchmarks—including any actual or possible training/test data cross contamination relative to the deployment environment—and Action MS-2.11-002 directs teams to measure GAI performance across demographic groups and subgroups, including subsampling deployment traffic for manual annotation.
- NIST Generative AI Profile Action MS-2.13-001 directs teams to create measurement error models for pre-deployment metrics to demonstrate construct validity by measuring or estimating, and documenting, biases or statistical variance in applied metrics or structured human feedback processes.
- Test datasets commonly used to benchmark or validate GAI models can contain label errors, and inaccuracies in these benchmark labels can impact the stability or robustness of benchmarks used during model selection.
- Pre-deployment TEVV benchmark datasets and laboratory or in silico techniques may not extrapolate well to real-world conditions, and gaps between benchmarks and real-world use of GAI systems are exacerbated by prompt sensitivity and broad heterogeneity of contexts of use.
Reproducible Evaluation Worksheet: Task Definitions, Golden Samples, and Rater Counts
Before publishing an internal model comparison or release gate, record your task set definitions, golden sample counts, and grader headcount across the three intake blocks below:
| Worksheet Block | Required Input | Verification Rule Before Aggregating Scores |
|---|---|---|
| Worksheet BlockTask set specification | Required InputTask count and balance | Verification Rule Before Aggregating Scores Start with 20-50 simple tasks drawn from real failures (an 80/20 approach), and test both cases where a behavior should occur and where it shouldn't to avoid undertriggering vs. overtriggering. |
| Worksheet BlockTrial consistency gate | Required InputTrial count and isolation | Verification Rule Before Aggregating Scores Start each trial from a clean environment, and distinguish |
| Worksheet BlockGolden sample sizing | Required InputSample size and clusters | Verification Rule Before Aggregating Scores Report |
| Worksheet BlockRater count and agreement | Required InputAnnotators and judges | Verification Rule Before Aggregating Scores Record human rater count and independent expert verdict agreement, compute 2-rater |
| Worksheet BlockFailure slice breakdown | Required InputSubgroup metrics | Verification Rule Before Aggregating Scores Disaggregate evaluation metrics on slices of data via |
Once your team has locked task definitions, golden sample error bars, and rater agreement counts, compare your production trace retention and evaluator token budgets in the AI Agent Observability Cost Comparison Worksheet, review our evidence standards under Methodology, and explore related engineering sessions in LLM Reliability.
Sources
- Anthropic Engineering, Demystifying evals for AI agents.
- Evan Miller (Anthropic), Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations.
- Lianmin Zheng et al. (LMSYS and UC Berkeley), Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.
- Shreya Shankar et al. (UC Berkeley), Who Validates the Validators — Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences.
- scikit-learn Developers, sklearn.metrics.cohen_kappa_score Documentation.
- statsmodels Developers, statsmodels.stats.inter_rater.cohens_kappa and statsmodels.stats.inter_rater.fleiss_kappa Documentation.
- Fairlearn Developers, Performing a Fairness Assessment (MetricFrame User Guide).
- UK AI Security Institute and Meridian Labs, Inspect AI Documentation (Overview and Scorers).
- Pydantic AI, Pydantic Evals Documentation.
- EleutherAI, Language Model Evaluation Harness (lm-evaluation-harness README).
- National Institute of Standards and Technology (NIST), Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile.