Evaluating AI-assisted systems

An evaluation should establish what an AI-assisted system can be relied on to do, for whom, and under which operating conditions. That requires an account of acceptable work, observations that can establish whether the requirements were met, and a method for estimating how often they are met across the intended population. When the system acts, the evidence must also cover its authority, the state it changes, and what happens after an interruption or a failed attempt.

These obligations extend beyond the model. A correct recommendation can reach the wrong person. A checker can identify an error that the application nevertheless delivers. A service can produce excellent answers for the requests it completes while leaving difficult requests in an unstaffed queue. Evaluation must preserve those differences because each supports a different operating decision.

This paper develops a connected method for evaluating assistants, predictive systems, fixed workflows, and agents. It begins with requirements and reference evidence, then examines evaluator validity, sampling, statistical uncertainty, and the transition from a judgment to an enforced decision. It concludes with repetition, recovery, adversarial tests, and the record needed to justify a bounded operating scope. The technical core is the validity of the evaluator: what its observations establish, how its errors can be measured, and when those measurements transfer to the work being considered.

A hypothetical customer-support refund service provides a recurring example. It reads a request and supporting records, proposes a resolution, and may execute an authorized refund or route the case to staff. Its rules and all numerical illustrations in this paper are constructed. They explain evaluation methods without claiming deployment results. Document analysis and prediction provide contrasting examples where completion may stop at a recommendation and where outcomes arrive later.

The resulting evidence can supply the defect rates, completion rates, labor requirements, and uncertainty needed for an economic comparison. Pricing those alternatives requires additional judgments about value and consequences. Here the concern is how to establish the quantities and acceptance evidence on which that comparison depends.

1. Specify the work and the authority being evaluated

The first evaluation decision is the boundary of the work. A task describes what a particular request requires. A system is the configured combination of models, application code, data, tools, checks, and people that attempts it. A workflow follows the request through those components to its eventual disposition. Changing the boundary changes what counts as completion and which failures remain visible.

For the refund example, a recommendation service might finish by identifying the relevant transaction, explaining the applicable policy, and presenting a supported proposed resolution to an authorized employee. An execution service also owes evidence that the permitted payment occurred, that it went to the right recipient, that no duplicate payment was made, and that the customer received the required confirmation. A correct proposed amount establishes only part of the second service's obligation.

The specification should identify eligible users, permissible inputs, the source versions that govern the decision, and the actions the service may take. It should state which missing information requires clarification, which conflicts require a human decision, and when closure without action is acceptable. These decisions determine the evaluation population before performance is measured. A case remains eligible when it later becomes difficult, requires review, or exhausts the attempt budget.

Several units need distinct identifiers. A request can produce multiple candidates. Each candidate may receive several checker judgments. An accepted candidate may trigger more than one action, and the resulting state may change after a retry or repair. A rate per candidate, a rate per request, and a rate per action answer different questions. Linking these records allows the evaluation to reconstruct a request without counting each repair as a new successful service delivery.

Requirements need observable acceptance conditions

Substantive correctness concerns the meaning and completeness of the result. In the example, the proposed resolution must concern the right transaction and apply the relevant refund policy to the available facts. A schema can establish that an amount is numeric. It cannot establish that the amount is owed. An exact match to one reference explanation can reject a different explanation that reaches the same supported conclusion.

Required procedure concerns obligations that cannot be inferred from the endpoint alone. If the service must check for an existing refund before initiating another, a final balance does not establish that this check occurred. If an employee must approve an exception, the evidence must identify the approver, the decision, and the operation approved. Grade mandatory constraints while allowing alternative paths that satisfy them. Requiring a particular sequence of tool calls is justified only when the sequence itself is part of the requirement.

Authority concerns who may cause which changes. The service owner's permission to automate a class of refunds, the customer's request, and an employee's approval have different roles. The operating policy must connect them to the specific account, transaction, amount, and action. An approval to investigate a disputed payment is not automatically approval to issue it. Substantive correctness cannot compensate for an action outside the granted scope.

Table 1 connects requirements to observations. Each row concerns a property of the hypothetical service, rather than a preferred kind of grader.

Table 1. Observations needed for selected acceptance conditions in the hypothetical refund service. A producer's completion message cannot substitute for the records named in the table.

Requirement	Evidence needed	What the evidence leaves open
Supported proposed resolution	Request, relevant records, applicable policy, and proposed decision	Whether the decision was authorized or executed
Required approval	Approval record bound to the actual transaction and action	Whether another path bypassed approval
Permitted execution	Enforcement decision, authenticated actor, and actual operation	Whether the external system completed the operation
Correct resulting state	Before-and-after records, transaction identity, and reconciliation	Whether an earlier unauthorized action occurred
Valid closure without payment	Evidence that no payment is owed or the request is already satisfied, plus permitted closure reason	Whether a refusal concealed an unresolved obligation

Some requirements apply to every resolved request. Others concern the service over a population, such as turnaround time, staff workload, or the share completed by a deadline. Record which is which. A mandatory permission check does not become optional because the average quality score is high. Conversely, a stylistic preference need not become a binary release condition when the service permits reasonable variation.

Benchmark evidence supplies a starting population and scoring contract

A benchmark contributes evidence about its stated task distribution, configured system, execution conditions, and grading rule. Its relevance depends on how those conditions relate to the intended work. It may expose a missing capability, suggest a candidate configuration, or supply challenging cases. It cannot supply observations that its experiment never collected.

Two studies illustrate the measurement issue in opposite directions. Cao and colleagues' procedure-aware study of simulated support interactions identifies violations within endpoint successes. Its manual airline audit examines 131 flagged successes, validating most flags; that design does not estimate violations missed among unflagged runs. AgentRewardBench documents an exact-string grader rejecting a correct national-park answer with an explanation. Its grader comparisons also change the ordering of some agents relative to expert judgments. These results support inspecting the acceptance rule and the evaluator, with no universal claim that benchmarks raise or lower scores. $$1, 2$$

The same discipline applies locally. A local test becomes relevant through its requirements, population, and observations. Merely replacing public tasks with company documents does not establish reference quality, representative sampling, or a valid grader.

2. Build reference judgments that can be challenged

A reference judgment is the basis against which a candidate or checker is assessed. It can combine an authoritative record, an executable invariant, a domain expert's interpretation, and observations of the outcome. It should be inspectable enough that another qualified reviewer can understand both the conclusion and the evidence supporting it.

For the hypothetical refund service, the reference packet would include the authenticated request, transaction history, applicable policy version, relevant communications, and evidence of any prior refund. A reviewer should be able to distinguish a wrong decision from a decision that cannot yet be established because the required facts are missing. That distinction prevents the evaluation from rewarding confident guesses or treating every incomplete record as a model error.

Reference construction begins with concrete cases. Inspect common requests, known failures, valid alternatives, and consequential exceptions. Write down the defect in terms of the requirement violated: wrong transaction, unsupported policy interpretation, omitted evidence, missing approval, duplicate execution, or premature closure. Keep the observed defect separate from a proposed cause. An omitted term might arise from retrieval, parsing, generation, or a misleading source; the output alone may not identify which component caused it.

Group recurring observations into a taxonomy whose categories support different repairs or operating choices. Categories need not be mutually exclusive. One request can contain an incorrect amount and an unauthorized payment. Store both attributes, then define an explicit rule if an aggregate report needs one primary failure or a highest-severity category. Forcing every case into one exclusive label can hide the co-occurrence that makes a failure consequential.

The label instructions should define the unit being judged, the evidence reviewers may consult, accepted alternatives, severity criteria, and how to handle uncertainty. Include positive and negative examples near the decision boundary. A rule such as "all material claims are supported" needs a definition of materiality and a way to locate the supporting source. Otherwise reviewers may apply incompatible standards while appearing to use the same rubric.

Independent assessment and adjudication serve different purposes

Have reviewers assess an overlapping sample independently before discussing it. Where practical, conceal the producer's identity and the checker's verdict until the initial judgment is recorded. This reduces a source of influence and preserves evidence about the instruction set. It does not guarantee that the reviewers' errors are statistically independent.

Adjudication then examines the disagreement. It can discover an overlooked source, correct a mislabeled reference, clarify an ambiguous requirement, or identify a legitimate range of answers. Preserve the original judgments and the reason for the adjudicated decision. A consensus label without that record erases information about where the standard was difficult to apply.

Some disagreements should remain unresolved. A policy conflict may require a service owner's decision; an unavailable record may make the outcome unknowable; a subjective criterion may allow several defensible judgments. Record these cases explicitly. Excluding them from the reported rate changes the population. Depending on the decision, report separate resolved-case estimates, bounds that place unresolved cases on either side of the criterion, or results under alternative defensible labels.

Cohen's kappa summarizes agreement beyond the agreement expected from two annotators' marginal label frequencies. For observed agreement $a_{o}$ and chance-expected agreement $a_{e}$, it is

$$\kappa = \frac{a_{o} - a_{e}}{1 - a_{e}}.$$

It measures consistency under that comparison, not whether either reviewer is correct. Two reviewers can share a mistaken interpretation, while a rare defect can make an agreement coefficient sensitive to the label marginals. Report the underlying label table, disagreements by class, and the adjudication method alongside any coefficient. There is no universal kappa threshold that validates a reference set or a deployment. $$3$$

Once a checker is assessed against an adjudicated reference, the question changes to classification validity relative to that reference. Sensitivity and specificity are then more informative than an agreement coefficient. Those quantities still inherit reference errors. Calling a dataset "gold" does not remove that dependence.

Preserve the independence of the final assessment

Cases used to develop requirements and labels are development evidence. Cases used to tune a judge prompt or select a threshold are calibration evidence. A final assessment should use cases that did not influence those choices, with the configured evaluator fixed before its results are inspected. A subsequent change prompted by those results requires a new assessment or an explicit statement that the set has become development material.

The split must match the claim. Randomly separating near-duplicate pages from the same contract across development and test sets can leave the same underlying problem on both sides. If the intended claim concerns new customers, document families, or time periods, split at that level and preserve the grouping in uncertainty estimates. A representative current-intake audit and a deliberately difficult regression set can coexist, but their scores describe different populations.

3. Establish validity when no complete answer key exists

The absence of a single correct answer does not remove the possibility of evaluation. It changes which properties can be established and how much judgment is needed. A contract memo can have several acceptable phrasings while still being wrong about a clause, omitting a material exception, or applying an obsolete policy. A prediction can be evaluated against later events even though no individual forecast is certain. An executed transaction can be reconciled against external state.

The design question is which observations distinguish an acceptable result from a defective one. The evaluator may need an original document, a policy effective date, an account record, or an outcome collected months later. A more capable model cannot infer an unobserved database mutation from a completion message. A human reviewer cannot reliably adjudicate an omitted clause from an extract that already omitted it.

Table 2 summarizes several forms of reference evidence. Their strengths are property-specific. No ordering by technology or reviewer type establishes a universal ranking of validity or cost.

Table 2. Forms of evidence and their identification limits. Methods can be combined; none establishes properties absent from its observations or specification.

Reference evidence	Property it can establish under stated conditions	Principal limit
Executable invariant or recomputation	A specified relation holds for the supplied inputs and implementation	A wrong specification, shared parsing error, or missing requirement can survive
Authoritative source or record	Agreement with a relevant source version or externally recorded state	Authority, identity, completeness, and freshness must themselves be checked
Expert judgment under a rubric	A reasoned assessment where interpretation or acceptable variation matters	Expertise, observations, instructions, incentives, and disagreements affect the label
Model-assisted judgment	A scalable assessment of a defined property from its visible evidence	Local validity and presentation sensitivity require measurement
Delayed observed outcome	What happened to cases with observed follow-up	Missing outcomes, intervention, timing, and confounding limit interpretation

Source fidelity and completeness require different tests

A document assistant can faithfully reproduce an incorrect source. It can also accurately summarize the passages it retrieved while failing to retrieve a passage that changes the answer. Evaluate source authority, retrieval coverage, and use of the retrieved evidence separately when each affects the requirement.

For a contract memo, an assessor might first establish which executed version governs, then identify the clauses required to answer the user's question, and finally assess whether the memo represents those clauses and exceptions correctly. Checking that every sentence has a citation addresses one part of the last step. It does not establish that the cited material is sufficient or that material omissions are absent.

Component checks are valuable because they locate different repair opportunities. They should be connected to an end-to-end acceptance judgment. A high retrieval score cannot compensate for a wrong final conclusion, and an accidentally correct conclusion does not establish that a defective retrieval procedure will generalize. The component measurements support diagnosis of where requirements failed; the end-to-end measurement establishes whether the service met the request.

Model assistance needs validation for the intended judgment

Model judges can classify defects, rank two candidates, assess rubric dimensions, or contribute predictions to an aggregate estimator. These uses require different validation. Agreement with pairwise preferences does not establish the probability of catching a material defect. A useful estimate of average quality does not identify which individual outputs may be delivered.

RAGChecker illustrates the distinction. Its meta-evaluation relates metric differences to human preferences on 280 response pairs. Annotators receive GPT-4 critiques as part of the reference procedure. That setup can inform comparative assessment and component diagnosis; it supplies neither an unaided expert reference nor a measured runtime catch rate for a new workflow. The effect of the critiques on labels remains unmeasured in that experiment. $$4$$

Written rubrics should expose their decisions through cited observations and concise reasons that can be checked. A plausible explanation is supporting material, not a substitute for the observation it describes. Retain an "insufficient evidence" outcome when the evaluator lacks the facts needed to decide, and specify how that outcome affects delivery and the evaluation denominator.

Presentation can change a model judgment. Zheng and colleagues tested order and verbosity effects in their model-judge experiments. Panickssery and colleagues separately studied self-preference on summarization tasks. These are reasons to examine identity cues, order, length, and rubric wording in the local evaluator. They do not prescribe a fixed penalty for every model or show that a prompt change removes all bias. $$5, 6$$

A validation set should contain valid alternatives as well as characteristic errors of the actual producer. Swapping candidate order can reveal order sensitivity; paraphrasing a correct answer can reveal an overly narrow rubric; inserting an unsupported but fluent claim can test whether the judge uses the source. These are challenge tests. Performance on them becomes a population estimate only when their sampling mechanism supports that interpretation.

If a judge supplies a numerical defect probability, calibration asks whether candidates assigned a probability near a given value actually have that defect frequency in the target population. Examine this relationship on held-out references, with enough cases to show its uncertainty. Threshold selection is a separate choice about which scores trigger action. A score calibrated on a balanced challenge set need not be a calibrated probability on ordinary intake, and a well-calibrated aggregate can still conceal a poorly served subgroup.

Delayed outcomes add evidence and introduce selection

Later outcomes can test a prediction or a service result that was uncertain at the time of the decision. Record the outcome definition, observation horizon, and missing follow-up. A support case that never reopens may be resolved, or the customer may have stopped trying. Silence alone cannot distinguish those explanations. An adjudicated claim may provide a stronger reference for some fields while remaining open to correction or appeal.

For predictive systems, distinguish the accuracy and calibration of forecasts from the quality of actions selected using them. A decision can change the outcome being predicted. If staff act on high-risk cases, their later outcomes reflect both the original risk and the intervention. Labels observed only for approved applications or investigated alerts leave other cases selectively unobserved. Evaluating a policy's effect may require a randomized comparison, a suitable observational design, or a narrower claim about observed cases.

Where no reference identifies a consequential property, record that limit at the decision it affects. The result may support a recommendation boundary, a human investigation, or a restricted operating population. It need not support automatic execution merely because several imperfect signals agree.

4. Measure checker errors against the intended defect population

Define a material defect as the positive class. Let $D = 1$ mean that a candidate has the specified defect, and let $F = 1$ mean that the checker flags it. The defect definition and candidate stage must remain fixed throughout the comparison. An initial proposal, a repaired proposal, and a delivered result are different populations.

Sensitivity is the probability of a flag given a defect. Specificity is the probability of no flag given a candidate without that defect:

$$s = P(F = 1 \mid D = 1),\quad\quad c = P(F = 0 \mid D = 0).$$

Their complements describe missed defects and flags on candidates without the specified defect. For a class-specific analysis, a negative label on one class does not establish overall acceptability. In the binary aggregate illustrations below, $D$ covers any material defect under the stated acceptance conditions. Flag precision, $P(D = 1 \mid F = 1)$, asks how often a flag is justified. The defect rate among unflagged candidates, $P(D = 1 \mid F = 0)$, asks how much defective work remains in that group, when it is nonempty. These conditional probabilities have different denominators. A checker can have high sensitivity and still create a substantial review burden when defects are rare and false flags are frequent.

Specify how indeterminate judgments, timeouts, and missing checker results enter the analysis. If the operating rule treats them as reasons to flag and withhold a candidate, the binary classification can reflect that rule while retaining the original status. Do not silently discard them or count them as successful approvals. Where a binary mapping is unsuitable, report a separate indeterminate category and its operational outcome.

The validation report should show the confusion table, sample counts, uncertainty, and consequential defect classes. An overall accuracy score can be dominated by acceptable cases. A sensitivity estimate based on arithmetic errors supplies little evidence about missing approvals if none were represented. If no naturally occurring defect of a class was audited, the natural-error sensitivity for that class has not been estimated.

In practice, the audit observes reference judgments rather than an independently visible true label for every case. Report sensitivity and specificity against that reference, together with how it was established. The stronger interpretation as detection of actual defects requires an adequate reference for the property in question. A checker can appear wrong because the reference rejects valid alternatives, or appear correct because both share the same source error.

Offline evaluation and operating checks have different roles

Use $q_{E}$ for the sensitivity of an evaluator used to measure candidates offline, and $q_{P}$ for the sensitivity of a checker used in operation, each on a stated defect population. The offline evaluator might use a complete case packet and extensive adjudication. The operating checker might have a shorter context, a time limit, and no access to later outcomes. Even identical code can perform differently under those observations and conditions.

Validated sensitivity and specificity, together with adequate reference evidence, determine how well an offline evaluator can estimate defects. Those measurements do not protect a live delivery unless the operating process uses a suitable control. A high $q_{P}$ means that the operating checker detects defects under its tested conditions. Whether those flags prevent delivery, trigger successful repair, or accumulate in a queue remains a separate measurement.

The familiar expression $p\left( 1 - q_{P} \right)$ describes defective unflagged candidates per initial candidate only under a defined model: initial defect prevalence is $p$, $q_{P}$ is sensitivity on those defects, and flags are classified before any relevant repair or selection. To interpret the expression as faulty deliveries per initial candidate, flagged defects must actually be withheld and the model must account for any other paths or newly introduced errors. It is not automatically the defect rate among deliveries.

If delivery occurs exactly when the checker leaves a candidate unflagged, with no repair or bypass, the latter conditional rate is

$$P(D = 1 \mid F = 0) = \frac{p(1 - s)}{p(1 - s) + (1 - p)c}.$$

The denominator is the fraction left unflagged, including acceptable candidates. Once review, repair, rejection, or multiple checks change the delivery population, estimate the actual final rate from the recorded workflow. A simple formula should not silently substitute for the system that was operated.

Natural defects and planted challenges answer different questions

Naturally occurring candidates reveal errors generated by the configured system on a sampled population. They support estimates of defect prevalence and, with appropriate audits, checker sensitivity on those errors. Rare but serious failures may be absent from an affordable sample.

Planted challenges make specified failures observable. They can test a wrong transaction identifier, an omitted condition, a stale approval, or a duplicated operation. Construct them to preserve the cues and observations available in operation. An obvious artificial typo may be much easier to detect than the producer's characteristic mistake.

A challenge result estimates detection under the challenge construction. It does not establish how frequently that error occurs naturally, and its sensitivity need not transport to natural instances of the same broad class. Report challenge coverage separately from natural-error rates. Both are valuable: one examines mechanisms that must work, while the other estimates performance on the stated intake.

Additional reviewers contribute through the errors they catch

Organizational separation, provider diversity, and statistical independence describe different things. A separate review team can use the same mistaken policy interpretation. Two model providers can share failure patterns. A deterministic recomputation can repeat the producer's parsing mistake if both receive the same incorrect input.

Kim and colleagues found correlated model errors, including across providers. Their judge analysis used a model's own multiple-choice answers as substitute reference labels. That experiment supports examining shared mistakes; it does not supply a production sensitivity estimate for a source-grounded rubric judge. $$7$$

On audited defective candidates, measure which defects each check misses and which remaining defects an added check catches. Multiplying two marginal miss rates assumes an appropriate independence relation. Measuring their joint misses does not require that assumption. Full independence is unnecessary for a second check to help: it can add value by detecting some consequential errors left by the first. Conversely, high agreement between checks may reflect redundancy rather than validity.

Human review belongs in the same analysis. Its observations, instructions, time budget, tools, authority, and workload affect what it can establish. Evaluate the review process that will actually run, including rushed and ambiguous cases when those conditions are part of the intended service. Neither human status nor a previous catch rate establishes performance under a new workload.

5. Sample the population whose performance you intend to report

An audit needs a sampling frame: the inventory of cases from which observations can be selected. Preserve eligible requests even when no candidate is produced, and preserve candidates that are withheld as well as those delivered. Flag status, delivery status, repair status, and final request disposition are separate attributes. Crossing them can reveal behavior hidden by an audit of successful deliveries alone.

Auditing only flags estimates how often flags are justified. It cannot identify missed defects among unflagged candidates. Auditing only deliveries can estimate delivery quality under a suitable design, but it cannot by itself measure false rejection or the burden placed on withheld work. Sampling repaired candidates without their initial versions conceals first-attempt defects and the effort needed to correct them.

Disproportionate sampling can allocate more review to consequential or rare groups. Keep known, positive inclusion probabilities for every group that contributes to the population claim. Weight the observations back to the target population, and retain separate class or segment estimates where aggregation would conceal an operating limitation. If a subgroup has no chance of audit, weighting cannot recover its unobserved performance.

A weighted audit of initial candidates

Consider a constructed pool of 10,000 initial refund proposals, one per eligible request. A fixed checker flags 1,000. Auditors randomly sample within the flagged and unflagged strata, without replacement, using a reference assumed correct for this illustration. All numbers in Table 3 are hypothetical.

Table 3. Hypothetical disproportionate audit of initial candidates. Each audited flagged candidate represents five candidates; each audited unflagged candidate represents 20. These are checker-validation quantities before repair or delivery.

Checker stratum	Candidates in pool	Candidates audited	Audited defects	Estimated defects in pool
Flagged	1,000	200	80	400
Unflagged	9,000	450	9	180
Total	10,000	650	89	580

The estimated initial defect prevalence is $(400 + 180)/10,000 = 5.8\%$. The raw audit fraction, $89/650 = 13.69\%$, overrepresents the flagged group and does not estimate the pool's prevalence. The estimated confusion table contains 400 true positives, 600 false positives, 180 false negatives, and 8,820 true negatives.

Sensitivity is therefore $400/580 = 68.97\%$, specificity is $8,820/9,420 = 93.63\%$, and flag precision is $400/1,000 = 40\%$. Among unflagged candidates, the estimated defect rate is $180/9,000 = 2\%$. The same 180 candidates represent $1.8\%$ of the entire initial pool. Neither rate describes final deliveries unless delivery follows the restricted no-repair rule described earlier.

For stratum weights $W_{h} = N_{h}/N$ and audited defect fractions ${\overline{D}}_{h}$, the direct estimate is

$$\widehat{p} = \sum_{h}^{}W_{h}{\overline{D}}_{h}.$$

Its uncertainty must follow the sampling design. Under the stated independent stratified sampling and fixed reference labels, an approximate 95% design interval for the illustration is 4.51% to 7.09%. Sensitivity has a wider approximate design interval, 55.1% to 82.9%, because uncertainty about missed defects materially changes its denominator. These intervals describe audit uncertainty for this finite pool. They do not include reference mistakes, future intake changes, or untested classes of defect. A binomial interval calculated as though the expanded counts were independently audited observations would overstate precision.

A convenience sample of interesting failures cannot become a representative audit merely by assigning weights afterward. Inclusion probabilities must come from a defensible design or an explicit, justified selection model. Missing records and incomplete adjudication require their own accounting. If reviewers cannot finish the difficult cases, a complete-case estimate can be biased even when the initial sampling was random.

Preserve dependence and the source of uncertainty

Requests can share a customer, document family, source system, reviewer, or outage. Repeated trials of one task also share task difficulty. Treating every observation as an independent new case can produce narrow intervals that omit the main source of variation. Use the natural grouping when designing samples and when resampling for uncertainty, and distinguish variation across cases from variation across repeated execution.

A paired comparison of two systems on the same eligible cases can estimate the difference more efficiently than two unrelated samples. Preserve the pairing in the analysis. Report which cases changed from acceptable to defective, which improved, and whether the difference depends on a consequential subgroup. Match the service boundary, reference method, time horizon, and resource constraints before interpreting the difference as evidence for changing the operating system.

6. Correct measured rates only under an identified observation model

An automated evaluator can label a large pool cheaply enough to be attractive, while making enough mistakes to bias the estimated defect rate. A correction requires a model of those mistakes and evidence for its parameters. It does not arise from agreement with another imperfect checker alone.

Let $p = P(D = 1)$ be defect prevalence in a specified candidate population, $r = P(F = 1)$ the evaluator's flag rate, $s$ its sensitivity, and $c$ its specificity. Conditioning on whether a candidate is defective gives

$$r = sp + (1 - c)(1 - p).$$

The first term is the share correctly flagged. The second is the share of acceptable candidates incorrectly flagged. Rearranging yields the Rogan-Gladen correction:

$$p = \frac{r + c - 1}{s + c - 1}.$$

Rogan and Gladen analyzed this screening-test prevalence correction in 1978. Rosin and colleagues provide a later treatment of estimation, uncertainty, and population standardization. The expression follows directly from the binary observation model; applying it requires that the sensitivity and specificity used describe the population whose flag rate is being corrected. $$8, 9$$

For a separate hypothetical example, suppose the population flag rate is 6%, sensitivity is 80%, and specificity is 98%. The corrected prevalence is $(0.06 + 0.98 - 1)/(0.80 + 0.98 - 1) = 5.13\%$. This is a point calculation under assumed error rates, not an estimated production result or an uncertainty interval.

Identification depends on reference quality and transport

Sensitivity requires audited defective cases; specificity requires audited acceptable cases. A representative probability audit can estimate both, although rare defects make sensitivity expensive to estimate precisely. An enriched validation set can contain more defective cases, but its defect mix, severity, and evidence quality must support transport to the target population. Artificially balanced labels alone do not establish that transport.

If the checker is excellent on numeric errors and poor on omitted exceptions, a change in the composition of defects can change sensitivity even when overall defect prevalence stays constant. Different document formats can change specificity. Stratification and standardization can address measured differences when relevant strata are represented. They cannot identify performance in an absent subgroup or repair unmeasured differences by formula. $$9$$

The correction treats the reference used to estimate $s$ and $c$ as adequate for identifying $D$. If the reference is fallible and its errors are unknown, the observed agreement table does not generally identify the true prevalence and both instruments' error rates. Additional information is needed, such as stronger audits, verified invariants, repeat assessments under justified models, or explicit bounds on reference error. A latent-label model can help only through assumptions that must themselves be stated and examined.

A small denominator makes the correction unstable

The denominator $J = s + c - 1$ measures the separation between the checker's flag probabilities on defective and acceptable candidates. When $J = 0$, the flag rate contains no information about prevalence under this model. Near zero, small errors in the flag rate or calibration parameters produce large changes in the corrected estimate. A checker with $J < 0$ can be systematically reversed in the mathematical model, but empirical use still requires stable, validated rates after any relabeling.

For $J > 0$, a feasible population flag rate lies between $1 - c$ and $s$. Sampling error, unsuitable calibration rates, or population change can put the observed rate outside that range and produce an estimate below zero or above one. Do not silently clip it and report a precise corrected percentage. Inspect the conflict and use an appropriate constrained likelihood or other bounded analysis, with its assumptions visible. $$9$$

Uncertainty must include the flag-rate sample and the data estimating sensitivity and specificity. A bootstrap should resample those sources under their actual sampling design, preserving overlap and dependence when the same cases contribute to several estimates. Likelihood-based or Bayesian approaches can handle boundaries and uncertain references, but their intervals inherit the specified model and, where applicable, priors. Resampling only the flag-rate data while treating estimated checker accuracy as known understates uncertainty.

Rogan-Gladen does not correct arbitrary selection bias, missing labels, a drifting evaluator, or inconsistent defect definitions. If its error rates come from the same weighted audit that already estimates prevalence, applying the formula simply reconstructs that estimated confusion table. It adds no independent evidence and does not justify treating the inputs as independent.

Direct audits and model-assisted estimates can be preferable

When a probability audit can directly estimate the target defect rate with acceptable uncertainty, it is often the clearest result to report. The weighted audit in Table 3 already does so. There is no need to reconstruct the answer through transported sensitivity and specificity.

Predictions can also assist aggregate estimation without being treated as true labels. For a fixed finite pool, let $z_{i}$ be a prediction available for every candidate, let $S$ be a probability audit with inclusion probabilities $\pi_{i} > 0$, and let audited references provide $D_{i}$. A difference estimator adds an audited correction to the prediction average:

$${\widehat{p}}_{assist} = \frac{1}{N}\sum_{i = 1}^{N}z_{i} + \frac{1}{N}\sum_{i \in S}^{}\frac{D_{i} - z_{i}}{\pi_{i}}.$$

For fixed predictions, the weighted residual estimates the full pool's prediction error. The estimator is design-unbiased conditional on the finite pool and the reference labels, because each case's expected audit weight is one. Precision improves only when the residuals and sampling design make the correction easier to estimate than the labels themselves. Using the same audited labels to fit predictions requires an appropriate splitting, cross-fitting, or dependence treatment.

Prediction-powered inference develops related model-assisted estimation and uncertainty methods. ARES applies this approach to aggregate assessment of retrieval-augmented generation. Its reported transfer is limited to the settings tested, with failures on larger shifts involving language, code, and extraction. This supports a role for automated judgments in measurement, provided the audit and inference design are valid. An interval for a population mean still does not authorize individual deliveries. $$10, 11$$

7. Account for every eligible request and its remaining obligations

Candidate validation establishes what proposed work contains and what checks detect. The service evaluation must then follow the requests that generated those candidates. Choose an intake window and a follow-up horizon that give each request the intended observation period. Record the final disposition at that horizon, while preserving later resolutions separately.

A full intake ledger includes automated completion, completion with human involvement, valid rejection, justified no-action, handoffs, requests for information, and unresolved work. A reported disposition is a claim to audit. A case marked "duplicate" is a successful no-action closure only when the evidence establishes that the obligation has already been met. A refusal can be a correct completed decision when the service owes an eligibility determination; it is unfinished work when the service still owes an investigation or remedy.

Table 4 gives the accounting structure for the hypothetical refund service. Each request occupies one final-disposition category. Assign one primary pending category by the next required actor or a declared priority rule, and keep any additional blockers as attributes. Human interventions, checker flags, candidate revisions, and harmful actions remain attributes or events attached to that request.

Table 4. Full eligible-intake accounting. The categories describe service status, not independent proof of quality. A completed handoff can itself be a valid outcome only when that is the explicitly evaluated service boundary.

Disposition at the follow-up horizon	Evidence needed for the outcome	Accounting treatment
Payment completed through the automated route	Accepted decision, authorized execution, reconciliation, and required confirmation	Closed; audit all acceptance conditions
Payment completed with human involvement	Same conditions, with review, repair, or approval recorded	Closed; include all human effort
Closed without a new payment	Evidence that no action was needed or an authorized rejection completed the obligation	Closed only if the reason is valid
Awaiting review or repair	Queue status and responsible destination	Unfinished, even if a defect was correctly detected
Awaiting customer information	Material information request and outstanding obligation	Unfinished unless the defined service contract validly closes it
Handoff still awaiting resolution	Receiving party, acceptance of the handoff, and unresolved obligation	Unfinished for an end-to-end resolution service

Let $N$ be all eligible requests, $C$ reported closures, and $G$ closures that meet the complete acceptance conditions, all for the same intake window and horizon. Reported closure coverage is $C/N$, closure quality is $G/C$ when $C > 0$, and successful-completion coverage is $G/N$. Their relationship explains why excellent quality on a small completed subset can coexist with poor service coverage. If only a sample is audited, estimate $G$ under that sampling design and carry its uncertainty into the reported rates.

Defective closures include wrongful no-action and wrongful rejection, as well as incorrect delivered changes. The share of eligible requests with a defective closure and the defect rate among closures use different denominators. Unfinished requests require a separate report even when no faulty output has been delivered.

Also measure requests affected by harmful or unauthorized actions across the entire intake, including repaired and unfinished cases. A corrected final balance does not erase an earlier unauthorized payment or disclosure. Count an affected request once for a request-level incidence rate; report action counts separately when repeated actions matter. A rate per action requires an inventory of actions, including those during failed and unfinished requests.

Human fallback is part of the operating system. Record active labor for review, investigation, correction, approval, and communication across all dispositions. Completed fallback cases alone do not reveal the labor spent on unresolved cases or on apparently automatic closures. Record customer clarification effort separately, and distinguish active effort from elapsed waiting time.

The comparator needs the same accounting. Comparing the AI service's audited closures with the incumbent's unaudited throughput mixes quality standards and boundaries. A credible comparison follows comparable intake through both configured services, including the incumbent's errors, human work, and unresolved obligations. Where a shadow run cannot reproduce real queue behavior or customer responses, limit the comparison accordingly.

8. Connect a checker judgment to enforcement and recovery

Detection, enforcement, correction, and audit have different causal roles. Detection identifies a possible defect. Enforcement prevents a prohibited transition or delivery. Correction changes the candidate or repairs state. An audit observes whether the relevant requirements were met. They can use some of the same records while still requiring separate evidence.

Suppose the operating checker detects a proposed refund to the wrong transaction. An enforced refusal, supported by external-state evidence that the operation did not occur, establishes prevention of that write. If the case then waits indefinitely for staff, the request remains unfinished. If staff resolve the identity, approve the corrected operation, and the service reconciles the resulting payment, the request may complete. If the application logs the flag and pays anyway, detection succeeded while enforcement failed.

Record these paths explicitly. A flag count measures none of the final outcomes by itself. A tool receipt establishes the event described by that tool, subject to its semantics and reliability. An acknowledgment that a request was accepted is different from evidence of a committed transaction, and a committed transaction is different from full service acceptance.

The production gate must bind its decision to the operation that will execute. Tests should attempt missing approval, stale approval, a substituted transaction, a changed amount, and an alternative write route. Inspect external state even after a reported refusal. Otherwise an interface can appear to deny a write while a side effect has already occurred elsewhere.

Routing changes both risk and workload

Selective prediction studies the tradeoff between the fraction of cases a system answers and the error risk on that answered subset. In a workflow, an acceptance threshold similarly changes automatic coverage and the composition of the cases retained. Geifman and El-Yaniv formalize this risk-coverage relationship under stated sampling assumptions. $$12$$

For a fixed population, let $A = 1$ indicate automatic acceptance under a particular policy. Coverage is $P(A = 1)$, and selective defect risk is $P(D = 1 \mid A = 1)$ when coverage is positive. Plot or tabulate these quantities across thresholds using held-out references and uncertainty. State whether $D$ describes the accepted proposal or the final delivered result. A confidence score or a model's verbal certainty becomes a useful routing variable only through its observed relationship to the relevant defects.

The service also needs the outcomes of cases with $A = 0$. Raising the threshold may reduce automatic delivery risk while increasing waiting time, missed deadlines, and human workload. A referral that reaches a reviewer who lacks the source document, permission, or available time does not supply the expected protection. Measure review completion, review errors, and the unresolved queue under the proposed volume.

Clarification should be evaluated in the same way. Determine whether the question obtains information that changes the decision, whether necessary questions were missed, and whether the request was eventually resolved. Asking more questions can improve information while imposing avoidable customer effort. Question precision or blocker recall can diagnose routing, but neither is a final completion rate or a checker sensitivity estimate.

Correct no-action needs affirmative evidence. FixedBench's coding experiments show that instructions encouraging abstention can reduce unnecessary edits on resolved issues while increasing abstention on partially repaired issues that still need work. The transferable evaluation requirement is to label both kinds of case, rather than equating restraint with success. $$13$$

9. Evaluate repetition under the actual trial and recovery protocol

Repeated sampling, dependent workflow steps, adaptive retries, and successive production requests create different experiments. They should not share a reliability formula merely because each involves several attempts or actions.

For a task $t$, let $\rho_{t}$ be its success probability under a fixed protocol. With conditionally independent, identically distributed trials of that same task, the chance that all $k$ trials succeed is $\rho_{t}^{k}$, and the chance that at least one succeeds is $1 - \left( 1 - \rho_{t} \right)^{k}$. The original tau-bench paper uses these constructions, averaged across tasks, for ${pass}^{k}$ and $pass@k$, respectively. $$14$$

The distinction is operational. At-least-one success measures the availability of a satisfactory candidate under the reference evaluator. Delivering it requires a selector that can recognize it from the information available in operation. A hidden answer key used to score the experiment does not become an available production selector. Evaluate the actual selection rule and include the attempts, checks, timeouts, and human effort it consumes.

Heterogeneous tasks change the repeated-success quantity

In a hypothetical population with half the tasks succeeding with probability 0.9 and half with probability 0.1, repeating one randomly selected task twice gives an all-success probability of

$$\frac{{0.9}^{2} + {0.1}^{2}}{2} = 0.41.$$

Drawing a new task independently for each of the two trials gives

$$\left( \frac{0.9 + 0.1}{2} \right)^{2} = 0.25.$$

The first quantity is $E_{t}\left\lbrack \rho_{t}^{k} \right\rbrack$; the second is $\left( E_{t}\left\lbrack \rho_{t} \right\rbrack \right)^{k}$ under a stable mixture and independent new draws. Their difference comes from shared task difficulty in the first experiment. The convexity inequality between them is not a general upper or lower bound for production, where outages, shared documents, workload, or changing arrivals can create other dependence.

With $n$ independent trials for a task and $c_{t}$ observed successes, the standard combinatorial estimates are

$$\widehat{{pass}^{k}} = \frac{\binom{c_{t}}{k}}{\binom{n}{k}},\quad\quad\widehat{pass@k} = 1 - \frac{\binom{n - c_{t}}{k}}{\binom{n}{k}},\quad\quad n \geq k.$$

Take a binomial coefficient to be zero when its upper count is smaller than $k$. Average per-task estimates using the task weights appropriate to the intended population. These formulas estimate the stated trial protocol; uncertainty still needs to account for task sampling and repeated execution. An adaptive retry that changes its prompt after a failure is a different policy, not another independent draw from the original protocol.

Workflow chains require conditional probabilities and state

For a chain that truly requires every step to succeed, its all-success probability factors into conditional probabilities of each step succeeding given the earlier successes. Multiplying marginal step-success rates is justified only under stronger assumptions. The familiar ${0.95}^{15} \approx 46.3\%$ calculation describes 15 independent steps with the same 95% success probability. It is an illustration, not a forecast for an observed workflow.

Many workflows allow repair, so final completion does not require every initial step to succeed. Their evaluation should follow the configured policy through detection, revision, retries, and escalation, then assess the resulting state and elapsed effort. A model that repeatedly makes the same omission may gain little from another identical attempt, while a tool outage may require recovery at a different component.

State-changing retries need special treatment. If a payment commits but its response is lost, another payment attempt can duplicate the effect. The system should preserve the operation identity and reconcile external state before deciding whether another write is needed. Test interruption before dispatch, during execution, after commitment, and during confirmation. Include concurrent changes and delayed or repeated responses when the supported environment permits them.

A compensating action may restore a balance while leaving a notification, disclosure, fee, or obligation that cannot be undone. Record those consequences separately from successful technical rollback. Recovery quality concerns the full required outcome and the remaining effects, not merely whether a second API call returned success.

An all-success monthly batch probability is relevant when one failure defeats the whole batch. Other services are better described by per-case completion, incident counts, severity, and recovery time. Choose the repeated-use quantity from the service obligation instead of treating ${pass}^{k}$ as the universal operating objective.

10. Test authority and adversarial behavior within a defined scope

An acting system needs controls that operate independently of its explanation of what it intends to do. Bind tool permissions to the authenticated actor, target records, permitted operations, and required approvals. Where practical, constrain the maximum consequence of a single action and the cumulative effect of repeated actions. A per-call limit alone may leave repeated small actions unrestricted.

Evaluation should test the prohibited paths as well as the intended one. Attempt an unauthorized target, field, or operation; attempt to reuse approval for changed content; and attempt a route that avoids the normal checker. Assess refusal, external state, logs, and the remaining service obligation. Test controls in the environment that determines access, rather than assuming that a policy sentence in the producer's prompt enforces them.

Untrusted documents, messages, web pages, and tool results can contain instructions intended to redirect a system. An adversarial evaluation therefore needs a threat model: what the attacker controls, what they know, how many attempts and adaptations are allowed, and which outcomes constitute attacker success. A benchmark score on ordinary requests cannot estimate those outcomes.

AgentDojo separates benign task utility, utility under attack, and attacker success in a defined tool-using environment. That structure is useful for local tests because a defense can reduce unauthorized behavior by blocking legitimate work as well. Report both effects and preserve the attack conditions. Passing a fixed set of attacks supports a claim about that set and configuration, not immunity to new attacks. $$15$$

Protect the evaluation process itself. Keep withheld reference answers and hidden scoring material inaccessible to the producer, and protect authoritative state and evaluation records from unauthorized alteration. Separate producer-generated evidence from records maintained by the environment. Test whether logs and approvals remain attributable after interruption or a denied action. A system that can edit the record used to assess it can compromise both operational control and the validity of the evaluation.

Severity and reversibility determine the evidence needed before expanding authority. A wrong internal recommendation that is readily corrected has different consequences from a payment, disclosure, or external commitment. Record the affected party, recovery possibilities, detection delay, and plausible maximum exposure. These assessments can justify hard prohibitions or narrower permissions even when an average expected-loss calculation looks favorable. They are operating judgments that require an accountable owner; the statistical method does not choose them.

11. Make uncertainty relevant to the operating decision

An estimate becomes decision evidence when the remaining uncertainty is small enough for the proposed scope and consequences. Set the acceptance criteria before inspecting the final results. Specify whether the decision concerns average completion, a consequential defect class, an upper risk bound, or a mandatory per-case condition. An overall confidence interval does not establish that each subgroup or failure class meets its own requirement.

Consider a hypothetical audit finding two defective outcomes among 400 independent cases. The point estimate is 0.5%, but the two-sided 95% Wilson interval is approximately 0.14% to 1.80%. A symmetric interval near zero can extend below the feasible range and misrepresent uncertainty. This result does not establish a defect rate below 1%. $$16$$

If a predeclared decision instead uses a one-sided 95% upper bound and observes zero defects in $n$ independent Bernoulli cases, the exact bound satisfies $\left( 1 - p_{U} \right)^{n} = 0.05$:

$$p_{U} = 1 - {0.05}^{1/n}.$$

The bound first falls below 1% at 299 cases with zero observed defects. That arithmetic assumes a fixed tested configuration, stable sampling population, adequate reference labels, and the stated sampling and stopping plan. It does not establish security against adversarial inputs, detectability of every error class, or a lifetime failure rate. Sampling without replacement from a known finite pool calls for a corresponding finite-population calculation.

A threshold of 1% has no inherent claim to acceptability. The service owner must relate any bound to the type of defect, affected users, volume, reversibility, and authority constraints. A rare error may still be unacceptable if the permitted action has severe consequences. A broader rate may be manageable when the outcome is a reversible recommendation with a functioning review route. The evaluation should make that reasoning visible rather than supplying an arbitrary universal gate.

Distinguish sampling uncertainty from limits of the evidence

A narrow interval can coexist with a wrong reference, a missing subgroup, or an invalid observation model. Increasing the sample size addresses sampling variability under the design; it does not correct those problems. Report uncertainty from measurement and unresolved labels separately where it cannot be credibly folded into one interval.

Repeatedly trying thresholds or systems and reporting only the best final score adds selection. Tuning against the same holdout eventually turns it into development data. Reserve a final assessment, account for the comparisons that were searched, and use an appropriate sequential method if the plan permits repeated decisions as data accumulate. Repeatedly checking an ordinary fixed-sample interval and stopping when it passes does not preserve its stated guarantee.

For rare failures, combine evidence with different purposes. Natural audits estimate occurrence and residual performance on intake. Planted faults test specific detection and control mechanisms. Interruption scenarios test recovery. Adversarial cases examine a defined attack scope. Keep their denominators and conclusions separate. A pooled percentage would conceal which risks were observed naturally and which were introduced by design.

When evidence is sparse, choose the decision it can support. Retain recommendations if execution has not been validated. Restrict automatic processing to a supported document family or transaction class. Reduce volume if review capacity is the limiting condition. Those choices can permit useful operation while preserving the unresolved questions for the next evaluation.

12. Keep a versioned evaluation and operating record

The record should allow another qualified person to reconstruct the claim being made, the evidence supporting it, and the decision it authorized. It belongs to a specific configuration and operating scope. A concise public evaluation card can summarize it, while the underlying record preserves the methods and cases needed for review.

Table 5 describes the information that connects evaluation to operation. The fields are a record structure, not a claim that every system requires the same evaluation burden.

Table 5. Contents of a versioned evaluation and operating record. The record supports a bounded decision; a card or completed field is not independent evidence that the requirement was met.

Record component	Information needed to interpret the decision
Requirements and authority	Eligible users and work; substantive, procedural, and state conditions; permitted actions; hard constraints; owner of the decision
Configuration and observations	Producer, prompts, parsers, retrieval, tools, checker, permissions, and policy versions; what each evaluator can see
Population and references	Intake period, follow-up horizon, unit of analysis, exclusions, sampling probabilities, reference sources, adjudication, and unresolved labels
Methods and outcomes	Initial and repaired candidates; delivered and withheld work; complete intake dispositions; class and severity rates; uncertainty and dependence assumptions
Resources and review	Attempts, compute, tool use, elapsed time, operational labor, customer effort, evaluation effort, and remaining queue
Decision and change triggers	Supported scope, prohibited or withheld uses, responsible owner, evidence gaps, monitoring signals, and conditions requiring reassessment

Separate the costs of building, measuring, and operating

Evaluator-build costs include requirements work, reference construction, instrumentation, fixtures, and initial validation. Offline-evaluation costs include running comparisons, adjudicating samples, and estimating uncertainty. Operating costs include production checks, review, repair, clarification, monitoring, and incident recovery. Record shared resources once and state how they are allocated when comparing alternatives.

These distinctions matter when choosing the measurement system. An expensive offline expert audit may be appropriate for validating a cheaper operating check. A costly reference set may support many later comparisons. A low-cost automated judgment may still create a large review queue through false flags. Record usage and active labor, as well as monetary prices, so changing prices or staffing assumptions can be applied without pretending the underlying measurements changed.

Kapoor and colleagues' work on agent evaluation makes inference cost and holdout design explicit parts of comparison. For an operating service, extend that boundary to the human and recovery work required to finish requests. A quality comparison under a generous research budget and a comparison under a fixed service budget can both be legitimate, provided their claims remain distinct. $$17$$

Update the record when its supporting conditions change

Reassessment should follow the mechanism that changed. A new parser may alter which information reaches both producer and checker. A policy revision may change reference labels. A new tool permission expands possible consequences. A different customer mix changes the prevalence and composition of errors. A growing review queue changes completion and exposure even if checker sensitivity remains stable.

Preserve an independent audit stream alongside operational flags. Otherwise the monitoring system may observe only the errors it already recognizes. Retain regression cases that exposed failures, while keeping them separate from a sample used to estimate current intake performance. When the record is updated, identify which conditions were rechecked and which evidence is carried forward with justification. Avoid interpreting a smaller current sample as confirmation of every historical claim.

NIST's AI Risk Management Framework 1.0 provides voluntary lifecycle guidance organized around Govern, Map, Measure, and Manage. The evaluation record described here is compatible with that approach. Following it does not itself establish certification, legal compliance, or permission to operate. Those are separate decisions with their own requirements. $$18$$

The operating decision should state which requests the system may complete, which actions it may execute, what evidence is required before delivery, and how exceptions will be resolved. Its justification combines requirement-specific observations, validated checks, a defensible sampling design, and demonstrated control and recovery behavior. When those conditions change, the record identifies which part of the permission to operate needs reconsideration.

References

The versions below identify the sources used for this treatment. Primary-source checks were conducted on October 29, 2026. Historical model scores are not presented as current capability estimates.

1. Cao, H., Driouich, I., and Thomas, E. (2026). Beyond Task Completion: Revealing Corrupt Success in LLM Agents through Procedure-Aware Evaluation. arXiv:2603.03116v1, March 3. Sections 4-5 and Table 4. Preprint. https://arxiv.org/html/2603.03116v1

2. Lù, X. H., et al. (2025). AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories. arXiv:2504.08942v2, October 6. Sections 3-5 and Appendix A.2. https://arxiv.org/html/2504.08942v2

3. Cohen, J. (1960). "A Coefficient of Agreement for Nominal Scales." Educational and Psychological Measurement, 20(1), 37-46. https://doi.org/10.1177/001316446002000104. See also Cohen's retrospective, Current Contents, January 20, 1986: https://garfield.library.upenn.edu/classics1986/A1986AXF2600001.pdf

4. Ru, D., et al. (2024). RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation. arXiv:2408.08067v2, August 17. Section 4.2 and Appendices C and H. https://arxiv.org/html/2408.08067v2

5. Zheng, L., et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS Datasets and Benchmarks. Inspected arXiv:2306.05685v4, December 24, Sections 3.3-3.4 and Appendix D. https://arxiv.org/html/2306.05685v4

6. Panickssery, A., Bowman, S. R., and Feng, S. (2024). LLM Evaluators Recognize and Favor Their Own Generations. NeurIPS. https://proceedings.neurips.cc/paper_files/paper/2024/hash/7f1f0218e45f5414c79c0679633e47bc-Abstract-Conference.html

7. Kim, E. M., Garg, A., Peng, K., and Garg, N. (2025). Correlated Errors in Large Language Models. ICML, PMLR 267, 30038-30066. https://proceedings.mlr.press/v267/kim25e.html. Inspected text: https://arxiv.org/html/2506.07962v1

8. Rogan, W. J., and Gladen, B. (1978). "Estimating Prevalence from the Results of a Screening Test." American Journal of Epidemiology, 107(1), 71-76. https://doi.org/10.1093/oxfordjournals.aje.a112510

9. Rosin, S. P., Shook-Sa, B. E., Cole, S. R., and Hudgens, M. G. (2023). "Estimating SARS-CoV-2 seroprevalence." Journal of the Royal Statistical Society Series A, 186(4), 834-851. Sections 2-3. https://doi.org/10.1093/jrsssa/qnad068

10. Angelopoulos, A. N., Bates, S., Fannjiang, C., Jordan, M. I., and Zrnic, T. (2023). "Prediction-powered inference." Science, 382, 669-674. Inspected arXiv:2301.09633v4, November 9, Sections 1.2-1.4. https://arxiv.org/abs/2301.09633v4

11. Saad-Falcon, J., Khattab, O., Potts, C., and Zaharia, M. (2024). ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems. NAACL, 338-354. https://aclanthology.org/2024.naacl-long.20/. Inspected text: https://arxiv.org/html/2311.09476v2

12. Geifman, Y., and El-Yaniv, R. (2017). Selective Classification for Deep Neural Networks. NeurIPS 30. Sections 2-3. https://proceedings.neurips.cc/paper_files/paper/2017/file/4a8423d5e91fda00bb7e46540e2b0cf1-Paper.pdf

13. Gloaguen, T., Mündler, N., Müller, M., Raychev, V., and Vechev, M. (2026). Coding Agents Don't Know When to Act. arXiv:2605.07769v1, May 8. Sections 2 and 3.2. Preprint. https://arxiv.org/html/2605.07769v1

14. Yao, S., Shinn, N., Razavi, P., and Narasimhan, K. (2024). tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv:2406.12045v1, June 17. Section 3. https://arxiv.org/html/2406.12045v1

15. Debenedetti, E., et al. (2024). AgentDojo: A Dynamic Environment to Evaluate Attacks and Defenses for LLM Agents. arXiv:2406.13352v1, June 19. Sections 3 and 4.3. https://arxiv.org/html/2406.13352v1

16. Wilson, E. B. (1927). "Probable Inference, the Law of Succession, and Statistical Inference." Journal of the American Statistical Association, 22(158), 209-212. https://doi.org/10.1080/01621459.1927.10502953. Original-paper scan: https://jhanley.biostat.mcgill.ca/c607/ch08/wilson_jasa_1927.pdf

17. Kapoor, S., Stroebl, B., Siegel, Z. S., Nadgir, N., and Narayanan, A. (2024). AI Agents That Matter. arXiv:2407.01502v1, July 1. Sections 2, 4-6, and Appendix B. https://arxiv.org/html/2407.01502v1

18. National Institute of Standards and Technology (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST AI 100-1, January. https://doi.org/10.6028/NIST.AI.100-1