In this website paper
AI changes the cost of producing work, the effort needed to make that work usable, and the range of activities an organization can undertake. Its economic value depends on how those changes fit together. A fast draft can release expert time, create another review queue, or enable work that was previously uneconomic. A more expensive model can reduce total cost if it prevents enough correction. An effective assistant can become an ineffective delegated system when it receives authority that its checks cannot support.
This paper develops a way to compare complete operating plans across conventional software, predictive machine learning, generative assistance, bounded delegation, and human work. It connects production and checking costs with remaining losses, useful capacity, investment choices, and organizational learning. The comparison begins with the outcome people need and follows the resources required to deliver it. It then examines which uncertainties could change the choice, how to obtain proportionate evidence, and how operation can improve or weaken the original investment case.
The framework is an economic synthesis, illustrated with explicitly hypothetical calculations. It does not assume that verification is always the largest cost, that automation should extend as far as technology permits, or that a productivity percentage from another organization supplies a local return estimate. The central question is whether a feasible change improves the value of completed work after its full consequences are included.
1. Define the valuable completed outcome
The decision unit should correspond to something the organization can use: a customer issue resolved, a software change integrated and operating as required, a contract provision fit for its intended agreement, or a correctly handled request. Generated text, extracted fields, and proposed actions are intermediate outputs when further work is necessary to realize the claimed benefit. Measuring them can diagnose a process, but their production rate does not by itself measure the process's value.
Completion requires an outcome definition, a quality requirement, and a time boundary. For a support request, an answer sent within a minute may be useful only if it resolves the problem or initiates the correct next action. For a backend change, passing a local test suite might be an acceptance condition, while the economic claim concerns a reliable change used in production. If the claim includes reduced outages or faster customer onboarding, follow-up must extend far enough to observe those consequences. A local activity can be complete while the business outcome remains pending.
Some valuable outcomes involve declining to act. Correctly rejecting an invalid request, identifying that a proposed change is unnecessary, or leaving a record unchanged because the evidence does not support an update can complete the assigned work. These dispositions need affirmative evidence that they were appropriate. A model's refusal or an operator's abandonment is not automatically a justified rejection. A request awaiting missing evidence remains unresolved until the applicable completion condition is met.
This distinction matters for denominators. Start from the incoming workload, identified independently of the method's confidence. Track correct completion on the first route, completion after correction or fallback, justified rejection or no-action, incorrect disposition, and unresolved work. Attempts belong to tasks; several attempts do not become several business outcomes. Work that cannot use the proposed method remains in the operating plan, along with the resources and delays of its alternative route.
Passing a check is an observable event. Correctness is a property established against the relevant evidence and requirements. The two diverge when a check misses a defect or rejects an acceptable result. An accounting system may accept a syntactically valid record with the wrong amount. A test suite may pass code that implements an incomplete specification. Human approval can also be mistaken. Economic measurement needs both the recorded disposition and evidence about whether it was warranted.
The incumbent deserves the same examination. Existing processes have transcription errors, inconsistent judgments, unnecessary escalation, delayed decisions, and unfinished work. Their controls also consume resources. Comparing a candidate with an error-free human process overstates its disadvantage; comparing it with an expensive manual process while ignoring an available software improvement overstates its advantage. The baseline is the actual operating route and the feasible improvements to it.
Benefit can change along several dimensions. A method might deliver the same service at lower cost, improve the quality of the service, reduce delay, reach more people, or make a new activity feasible. Those are different claims. For a fixed backlog, earlier completion may have value without increasing the eventual number completed. For new demand, more capacity creates benefit only if the additional work is worth doing and can pass through the remaining process. Generating material that no one uses can consume real resources while adding little benefit.
Before estimating returns, therefore, state what qualifies as completed work, which population is included, how long outcomes will be followed, and whose costs and benefits are counted. These choices establish the comparison that every subsequent equation requires. They also prevent a technically successful subtask from silently becoming a claim about an entire service.
2. Compare feasible operating alternatives
An operating plan combines a method with inputs, interfaces, permissions, checks, staffing, recovery, and maintenance. Comparing model names leaves most of that plan unspecified. Two plans using the same model can differ substantially because one supplies complete evidence and bounded tools while the other relies on users to reconstruct context and detect failures.
Conventional software is a candidate when the procedure can be specified and maintained economically. Its cost includes discovering the rules, handling exceptions, testing interfaces, and updating the implementation when the work changes. Determinism makes execution reproducible under fixed conditions; it does not establish that the encoded rule is correct or that its inputs are complete.
Predictive machine learning can make a classification, estimate, or ranking economical when representative examples and meaningful targets exist. Its operating plan includes label creation, threshold selection, monitoring, and a route for cases where predictions do not justify action. The relevant quality is performance on the cases assigned to the model, including changes in their distribution. A high aggregate score can coexist with costly errors in a small but consequential group.
Generative assistance can help people interpret material, draft alternatives, or assemble a candidate result. Its value depends on how much productive work it removes and how much checking, correction, and coordination it creates. Assistance may be worthwhile even when every result receives expert review. Reading and improving a draft can cost less than producing one from scratch, and the draft can expand the range of options the expert considers.
Bounded delegation assigns software authority to perform specified actions under specified conditions. It is an allocation of responsibility that can use rules, predictive models, generative models, or a combination. Delegation can remove repeated approvals, but it also requires enforceable limits, observability, and a functioning exception route. A system permitted to propose a change has a different loss distribution from one permitted to deploy it.
Human and hybrid plans remain alternatives throughout the analysis. A specialist may handle unfamiliar cases while software handles a stable subset. A team may change its intake requirements or remove a redundant handoff before introducing a model. An outsourced service can be another comparator if its coverage, controls, and retained internal work are included. The comparison should not force every case through the same technology to make the plan appear uniform.
Two task properties help identify plausible combinations: how well the production procedure is understood and how well acceptable outcomes can be assessed. They guide investigation rather than determine a winner.
| Production procedure | Acceptance evidence | Economic implication |
|---|---|---|
| Stable and specifiable | Direct tests or authoritative records | Ordinary software is a strong candidate; compare its build and maintenance cost with assisted execution. |
| Requires search or discovery | Results can be checked against explicit requirements | Candidate generation and selection may reduce search cost, provided selection and recovery are effective. |
| Stable in outline | Quality requires contextual judgment | Templates or software can prepare work; assess which judgments still need people and what evidence they need. |
| Requires discovery | Outcomes are partly subjective or delayed | Assistance may expand options; assess the value of the decision process and constrain consequential actions. |
All shortlisted plans need a common workload, outcome definition, service requirement, horizon, and cost boundary. A candidate that automates easy requests and sends difficult requests to staff should be compared with the incumbent serving the same combined population. Report costs by route, then combine them using the expected case mix. Otherwise a difference in selection can be mistaken for a difference in efficiency.
Feasibility comes before ranking expected returns. Required data must be available on usable terms; the proposed actions must be authorized; staffing and fallback must support the workload; and the investment must fit available funding. Some losses or rights cannot be traded away for a larger expected financial benefit. A plan that fails these conditions needs a narrower scope or a different design before its average return becomes decision-relevant.
Time and scale can change the preferred method. A substantial fixed investment may be justified for repeated stable work but unattractive for a short project. A general model can reduce initial specification effort while creating recurring costs. A well-maintained parser can serve a stable subset cheaply while remaining unsuitable for new formats. Use demand scenarios rather than assuming that every fixed cost will be spread over the most optimistic volume.
3. Account for production, checking, and remaining losses
The basic comparison separates the resources used to perform work from losses that remain after those resources have been used. Checking costs money even when the candidate is correct. Correction consumes resources when a problem is found. An escaped defect may create an additional loss after acceptance. Each belongs in the plan, but the same consequence should appear only once.
For one incoming task under plan m, write expected economic cost as
(1)
Here c is initial production or execution cost; a is the allocation of specification, inputs and context, integration, maintenance, and governance; v is routine checking; R is expected additional handling, including correction, false rejection, retries, and fallback; and Z is remaining loss. All terms refer to the same incoming population and horizon. The allocation in a must identify its volume basis. A full investment calculation can instead retain fixed costs at their actual dates.
This framework has a familiar economic antecedent. Townsend's costly-state-verification model considers contracting when another party's realized endowment can be observed only by paying a verification cost. It establishes the relevance of costly observation to organizational choices. The AI workflow accounting here is an application of that general concern, rather than a result derived from Townsend's particular contract model.
The narrower production-plus-verification model is useful for understanding one mechanism. Suppose an initial candidate has a consequential defect with probability p, a check catches a defective candidate with probability q, and an uncaught defect causes loss L. If additional handling is absent or already included in the resource terms, expected cost becomes
(2)
The catch rate q is conditional on a defect existing. The multiplication p(1-q) therefore does not require the generator and checker to make independent errors. It describes the joint probability of a defect and a failure to stop it. The simple form does require a loss definition consistent with the errors that escape. It also needs explicit treatment of any work triggered by detection.
The cost comparison holds gross intended benefit fixed, with any differences in defective delivery or delay either ruled out or counted once as remaining loss. Under those conditions, the lower-cost feasible plan is preferable. If one plan has lower coverage, different service, or additional useful output beyond what the loss account represents, equal benefit cannot be assumed. A plan can justify a higher cost by delivering sufficiently valuable additional outcomes. Calling equal benefit a conservative assumption would obscure these possibilities; it is a condition to check.
Cost categories and accounting boundaries
| Cost category | Resources to include | Boundary to preserve |
|---|---|---|
| Establish the plan | Specification, data preparation, labels, integration, evaluation setup, training, and transition | Charge prospective investment once; retain sunk costs separately. |
| Supply and execute work | Input preparation, context access, licenses, compute, tools, and initial human work | Count the entire incoming workload and every charged attempt. |
| Check and resolve | Automated checks, expert review, false rejection, correction, rechecking, escalation, and fallback | Separate routine checking from additional handling; include other departments. |
| Maintain the operation | Monitoring, incident investigation, version changes, access administration, governance, and fallback readiness | Distinguish recurring work from reusable investment without charging either twice. |
| Bear remaining consequences | Residual defects, missed deadlines, failed recovery, and lost use | Exclude costs already counted above and losses already deducted from benefits. |
Setup includes the time needed to determine what the work requires. A model that can produce an answer immediately may still need an expensive process for assembling valid inputs, obtaining permissions, and defining acceptable results. Existing data and infrastructure can reduce new expenditure while consuming scarce capacity. Their economic opportunity cost depends on the use displaced, rather than the amount originally spent acquiring them.
Sunk costs do not become avoidable because a new method is attractive. Past spending on the incumbent is irrelevant to the forward choice, while its future support and replacement costs remain relevant. For a prospective investment, record actual expenditure at the time it occurs. If a per-task illustration allocates that investment over N tasks, do not also subtract the full investment when aggregating those same task costs.
Labor needs two accounts. A resource account describes hours by role, including training, review, correction, and work transferred elsewhere. A cash account records changes in payroll, overtime, contractors, licenses, and other expenditure. Valuing an hour at a loaded rate can support a resource comparison; it does not show that cash spending falls when the hour is released.
Released time creates economic benefit through an identifiable use. It may reduce paid overtime, defer credible future hiring, clear a valuable backlog, improve service, or enable other work. Small fragments of time can be difficult to consolidate. More capacity also needs demand and complementary capacity downstream. Count the portion used for additional output separately from the portion that reduces expenditure. The same hours cannot receive full credit in both categories.
Remaining loss must use the same boundary. If repairing a defect costs an hour, and that hour is in R, do not charge it again inside Z. A separate missed deadline or unrecovered payment can still belong in Z. If benefit is already measured as net realized value after defects and delay, subtracting the same losses again understates value. One consistent convention is gross intended benefit, less resource costs, less the residual consequences that prevent that benefit from being fully realized.
Unequal benefits and a multi-period decision
For a candidate relative to the incumbent, an incremental present-value comparison is
(3)
F is initial investment; B is gross intended benefit before the defect and delay consequences recorded in Z; X is recurring cash expenditure; O is opportunity cost not already counted in X or B; Z is remaining loss under that accounting convention; and d is the discount rate for periods of the stated length. Each delta is candidate minus incumbent. Positive incremental value supports the candidate only among feasible plans. Expected losses are uncertain economic quantities, not a prediction of the cash paid in any particular period.
Time enters through several channels. Transition consumes resources before benefits begin. Delayed service can lower B or create a separately recorded loss. Waiting can tie up working capital or scarce staff availability. Benefits that arrive earlier have a different present value. Specify the channel rather than adding a generic time-saving value to every elapsed-hour reduction.
Some benefits are better retained in their natural units. If an organization must meet a service standard, compare the cost of feasible plans meeting it. Report quality, coverage, and delay alongside financial value where credible conversion is unavailable. Assigning an invented dollar value to a consequential outcome does not improve the decision.
Uncertain volumes, implementation effort, exception rates, and usable capacity should remain visible in scenarios or ranges. These variables can move together: a difficult workload may increase execution cost, review time, and residual loss simultaneously. Adding independent best-case assumptions can produce a plan that no plausible operating condition would deliver. Scenario analysis should preserve such dependencies and identify where the preferred alternative changes.
4. Measure what checking changes
Verification establishes specified properties using available evidence. It can involve software, execution, a second model, a person, or several stages. Its economic value depends on which errors it catches, the consequences it prevents, and the costs it creates. An approval step without sufficient information, time, or authority may offer little protection, while a narrow automated check can be highly effective for the property it actually establishes.
A schema checks structure and permitted values. Reconciliation checks agreement with a reference. A test checks behavior under particular conditions. A reviewer can interpret context or assess whether a rule applies. None should be credited with establishing a property beyond its evidence. A valid citation can identify a source without showing that the source supports the claim. A generated test can repeat the same misunderstanding as the generated implementation.
Sensitivity, specificity, and corrective handling
Define a defect as a positive case. Sensitivity q is the probability of flagging a defective candidate. Specificity is the probability of passing an acceptable candidate. Let f be the false-rejection rate, equal to one minus specificity. These definitions must accompany results because some model-verification papers label a correct answer as the positive case, reversing the terminology.
Consider a deliberately simplified binary route. Every initial candidate is either acceptable or defective. Flagged defects are corrected successfully at expected cost h, including rechecking. Falsely rejected acceptable candidates are restored at cost r. There are no unresolved cases, and no further defects are introduced during repair. Then
(4)
All three probabilities p, q, and f lie between zero and one; the cost and loss inputs are nonnegative and expressed in the same monetary units. This equation includes the cost of detecting a real error and the cost of challenging a correct result. If repair can fail, an abstention creates a separate route, or a rejection prevents valuable work permanently, use those route probabilities and outcomes explicitly in Equation 1. The binary equation is a transparent illustration, not a substitute for the observed process.
False rejection can matter especially when valid work is common. A more restrictive gate may stop additional defects while sending many acceptable cases into delay or rework. Its economics depend on both conditional rates and the prevalence of defects. Neither a high overall agreement rate nor a low approval rate identifies an effective checker.
Auditing accepted work estimates the prevalence of defects among accepted cases. It does not, by itself, estimate sensitivity, because it omits the defective cases the gate stopped. Estimating q requires adjudicated cases across the relevant accepted and rejected routes, or a representative set of known defects. Known-defect challenges can diagnose coverage, but an artificially easy challenge set does not estimate performance on live failures. The evidence needed for the economic model is therefore broader than a sample of successful-looking outputs.
Error classes and severity
A single p, q, and L can conceal the errors that control the decision. A checker may reliably catch malformed fields while missing a rare authorization error. A human may recognize an implausible result while missing a plausible omission. Report sensitivity, false rejection, and consequences by classes that differ materially in severity or detectability.
More generally, remaining expected loss is
(5)
The classes j must represent mutually exclusive loss states or genuinely additive, nonoverlapping consequences. If several defects cause the same incident, adding their full incident losses would double count. The severity term is conditional on escape: the defects that survive checking may be more consequential than the defects encountered on average.
Loss depends on the action and its surroundings. A wrong internal draft can be corrected before use; a wrong deployment can interrupt other services. Access limits, staging, transaction caps, reconciliation, rollback, and rapid detection can reduce consequences. Reversibility is a matter of time, cost, and completeness. Undoing a database write may leave customer communication, operational disruption, or leaked information unreversed. Human approval is fallible, and its performance needs measurement; its presence does not guarantee recovery.
Dependence and severe-loss constraints
Different model names or providers do not establish independent errors. Kim and colleagues' ICML 2025 study found substantial overlap in wrong answers across models, including across architectures and providers. Its measurements concern particular evaluation settings, not every production checker, but they directly challenge using provider diversity as evidence of independent protection.
A second checker can nevertheless add value. Measure its detection rate on defects missed by the first checker. The combined escape probability is the initial defect probability multiplied by the first miss probability and the second miss probability conditional on the first miss. This is a conditional-probability calculation; replacing the last quantity with an unconditional score is the unsupported step. Execution, authoritative records, and expert judgment can contribute different evidence while still sharing incomplete specifications or bad inputs.
Dependence across tasks creates another exposure. A stale policy, missing context field, or faulty integration can affect many outputs together. For additive losses, the expected total still equals the sum of expected item losses. Correlation changes the distribution of that total, including the probability of a concentrated incident. It can also change the loss itself when many failures overwhelm recovery capacity. A per-item expected loss can therefore underdescribe what the organization must be able to absorb.
Expected value ranks tradeoffs within an authorized and tolerable set of actions. It does not set that set. An organization may impose a maximum transaction size, require specified controls, cap the probability of loss beyond a threshold, or examine the average loss in the worst part of the distribution. These constraints need defensible evidence and explicit owners. A fitted tail model does not create permission to expose people or resources to harms the organization has no authority to accept.
Potentially adversarial inputs also need a separate place in the loss scenarios. If an outside party can influence documents, messages, or tool results that affect proposed actions, evaluate what unauthorized behavior that influence could induce. Routine quality measurements on benign work do not estimate this exposure. Bound the accessible resources and permitted actions, and examine detection and recovery under the relevant attack conditions. Do not assume that a single ordinary-workload defect rate remains valid under deliberate manipulation.
NIST's AI Risk Management Framework 1.0 is a voluntary framework that distinguishes risk management from prescribing an organization's risk tolerance. It emphasizes context, applicable requirements, measurement, and continuing management. That distinction is useful here: the economic model informs a decision while the relevant authorities determine its permissible conditions.
5. Two checking regimes produce different economics
Checking conditions differ even when the same model produces the initial candidate. A bounded backend change may have executable requirements and reusable tests. A legal clause may require an expert to interpret the agreement, the transaction, and the applicable rules. Neither domain has a single verification regime: software includes difficult semantic and security questions, while legal work includes routine checks that can be automated.
The following examples illustrate different checking conditions. Every number is invented. They are not client cases, forecasts, or estimates from productivity studies. Each comparison covers 1,000 eligible tasks over twelve operating months, with setup allocated across those tasks. All resource costs are valued economically, including human time, and are not presented as cash savings. Both plans are assumed feasible for the stated task class, with the same intended benefit and deadline. Differences in final defects are priced in remaining loss. There are no additional unresolved cases or unpriced delay effects.
The overhead allocation a includes prospective specification, context preparation, integration, maintenance, and governance for both alternatives. Detected defects are corrected at cost h, and false rejections are resolved at cost r, including rechecking. Those resolutions succeed under the illustration's assumptions. Escaped-loss L excludes these already counted costs. These deliberately simple routes make the arithmetic transparent; a real plan should use its observed branches.
A backend change with effective executable checks
Suppose a narrowly scoped change has a meaningful integration suite, a review interface, and a controlled deployment route. The AI-assisted plan costs less to produce and check, although its initial defect rate is higher. Both plans face a $5,000 loss per escaped consequential defect. Correcting a detected defect costs $40, and resolving a false rejection costs $20.
| Input or expected cost per incoming change | Human production | AI-assisted production |
|---|---|---|
| Initial production c | $120.00 | $15.00 |
| Other cost allocation a | $10.00 | $10.00 |
| Routine checking v | $30.00 | $20.00 |
| Defect probability p | 10% | 15% |
| Catch rate q | 75% | 95% |
| False-rejection rate f | 1% | 2% |
| Detected-defect correction p × q × h | $3.00 | $5.70 |
| False-rejection handling (1 − p) × f × r | $0.18 | $0.34 |
| Remaining loss p × (1 − q) × L | $125.00 | $37.50 |
| Total expected economic cost | $288.18 | $88.54 |
The AI-assisted plan costs $199.64 less per incoming change under these assumptions. Its lower production cost contributes $105, while lower checking and residual loss contribute further savings. It also incurs more correction and false-rejection handling. The comparison therefore credits the useful checks while retaining the work they generate.
Holding its other inputs fixed, the AI plan breaks even at a catch rate of about 68.17%. At its assumed 95% catch rate, routine checking could rise to $219.64 before the economic costs match. These thresholds include the increase in correction cost when more defects are caught. They identify sensitivity to assumptions; they do not establish that an arbitrarily cheaper checker would retain the same catch rate or that a feasible quality constraint permits operating at the financial threshold.
The reusable integration suite is an investment, and the overhead allocation includes its relevant cost. Its existence is not enough to justify the assumed sensitivity. Evidence would need to show which consequential defects it catches on this task class, what review adds, and whether repair and deployment preserve the result. An untested requirement remains outside the suite's demonstrated coverage.
A legal clause requiring contextual expert review
Now consider preparing a clause for an agreement where the reviewer must assess contextual consistency and material legal implications. Both plans face a hypothetical $50,000 loss per escaped defect. Correcting a detected defect costs $150, and resolving a false rejection costs $100. The AI plan has lower drafting cost but requires more first-pass review and has a higher initial consequential-defect rate. The overhead figures include the same categories, with a larger assumed allocation for the AI plan's integration and context work.
| Input or expected cost per incoming clause | Human production | AI-assisted production |
|---|---|---|
| Initial production c | $400.00 | $20.00 |
| Other cost allocation a | $25.00 | $40.00 |
| Routine checking v | $300.00 | $360.00 |
| Defect probability p | 8% | 20% |
| Catch rate q | 90% | 90% |
| False-rejection rate f | 0.5% | 1% |
| Detected-defect correction p × q × h | $10.80 | $27.00 |
| False-rejection handling (1 − p) × f × r | $0.46 | $0.80 |
| Remaining loss p × (1 − q) × L | $400.00 | $1,000.00 |
| Total expected economic cost | $1,136.26 | $1,447.80 |
The $380 drafting advantage is insufficient in this configuration. The AI plan's total is $311.54 higher, largely because more consequential defects survive the assumed review. At the stated checking cost, it would break even at a catch rate of about 93.12%. Holding the catch rate at 90%, its checking cost would have to fall to $48.46 to match the human plan.
These are alternative thresholds, not simultaneous requirements. If an improved evidence package raised the catch rate to 95% without other changes, total cost would become $949.30. That would leave $186.96 per clause for additional checking resources before matching the human plan. Whether such a change is achievable is an empirical question. Neither the example nor the presence of expert review establishes that it is impossible.
The two examples teach a conditional comparison. Effective tests can make search and generation economical; contextual review can leave a larger burden after drafting becomes cheaper. The difference can also come from input quality, task difficulty, or integration. A stronger generator might improve the legal plan more than another review stage, while improving the incumbent's tools might reduce the apparent backend advantage.
Checking regimes guide investigation
| Checking conditions | Likely design opportunity | Question that still needs evidence |
|---|---|---|
| Low cost and effective for material defects | Reuse executable checks or authoritative comparisons | Which important defects remain outside their coverage? |
| High cost and effective | Reduce evidence-search and review effort; allocate scarce experts | Does assistance remove enough production work to cover the remaining review? |
| Low cost and weak | Improve evidence, restrict scope, or tolerate only bounded consequences | Does the check change consequential error rates enough to justify its presence? |
| Delayed or intrinsically incomplete | Stage actions, retain reversibility, and evaluate outcomes over time | What exposure is acceptable before stronger evidence becomes available? |
6. Choose the next investment conditionally
A cost breakdown identifies where resources go; it does not identify the highest-return improvement. A large checking budget may be difficult to reduce, while a modest input-cleaning investment prevents many errors. A small execution bill can still justify a model upgrade if that upgrade substantially improves outcomes. The next decision should compare achievable changes, including their implementation and maintenance costs.
For checking intensity alpha between zero and one, let v(alpha) be routine checking cost, q(alpha) its sensitivity, and f(alpha) its false-rejection rate. Hold the generator, workload, loss, and conditional handling costs fixed. In the binary example, an interior cost minimum satisfies
(6)
The left side is the marginal reduction in escaped loss, net of the correction cost created by finding more defects. The right side is the marginal checking cost plus the change in false-rejection handling. The familiar condition pLq' = v' is a restricted case with no incremental repair or false-rejection effects. Delay and changes in delivered benefit require their own terms when they matter.
This equality is a first-order condition, not a general optimum or a deployment rule. It requires differentiable functions and an interior solution. A minimum can instead occur at an endpoint, at a discontinuous operating choice, or at a binding risk constraint. The curvature of the functions determines whether a stationary point is a minimum. If an inexpensive check catches a particular error class completely, its discrete return can be more informative than pretending review effort varies smoothly.
For a general investment x, compare its incremental effect on delivered benefit, resource cost, and remaining loss. In the simplified p(1-q)L term, reducing p by a small amount is worth approximately (1-q)L times that reduction, while increasing q is worth approximately pL times that increase before handling effects. These are conditional sensitivities. They do not say how much either parameter will respond to a dollar of investment.
The alternatives often move several parameters together. Better inputs can reduce defects and reviewer search time. A stronger model can cost more per attempt while producing fewer retries, clearer evidence, or fewer false alarms. A typed interface can prevent a malformed action without establishing that the action is substantively appropriate. A shorter approval route can reduce delay without changing candidate correctness. A repair to a shared data source may improve every method under consideration, including the incumbent.
The backend illustration makes these interactions concrete. Suppose four separate changes are available, each evaluated against the AI-assisted baseline of $88.54. The inputs below are hypothetical, and every change preserves the other assumptions of that example. Their incremental costs already include allocated implementation and upkeep.
| Proposed change | Assumed effect | Added cost per task | Net expected reduction per task |
|---|---|---|---|
| Improve the generator | p falls from 15% to 10% | $8.00 | $6.38 |
| Improve the checker | q rises from 95% to 97% | $5.00 | $9.88 |
| Improve inputs and context | p falls to 12%; routine checking falls by $2 | $3.00 | $7.63 |
| Change the handoff | Routine checking falls by $4 without changing detection | $1.50 | $2.50 |
The checker produces the largest modeled net cost reduction per task among these four changes. The input improvement produces more benefit per added dollar. Which ranking matters depends on the investment constraint and the opportunities available. If the checker's improvement cannot be reproduced, or if a generator change prevents an expensive defect class that the aggregate p conceals, the comparison changes. It should also include whether the improvement can be maintained at the proposed workload.
Useful observations connect interventions to mechanisms. Inspect whether errors originate in missing inputs, misunderstood requirements, generation, or execution. Measure which checks catch which errors, how long the evidence search takes, and where work waits after passing checks. Compare candidates on the same cases where possible. A sequential before-and-after improvement can be informative, but simultaneous changes in case mix, staff experience, and tooling make attribution harder.
Shared investments create additional choices. A source registry, access service, or review interface can support several workflows. Evaluate the portfolio-level incremental benefit and cost, including the work needed to adapt it to each use. Allocating a shared platform's cost differently across teams does not change the organization's total economics. Nor does an attractive average allocation establish that the next workflow will cover its own incremental cost.
The objective is to locate a credible improvement in the whole plan. Checking, generation, inputs, integration, and work allocation remain competing and sometimes complementary uses of the same investment budget.
7. Price repeated attempts and checkpointed work
Repeated generation can increase the chance that a useful candidate exists. The organization still needs a way to identify it, complete any remaining work, and prevent an incorrect selection from being acted on. Candidate coverage and delivered success are different quantities.
If each of k independent candidates for the same task succeeds with probability s, the probability that at least one succeeds is
(7)
This calculation assumes the same success probability and independence for those attempts. Across heterogeneous tasks, average coverage is an average of task-specific probabilities, not necessarily the formula evaluated at average accuracy. Shared missing evidence or a systematic misunderstanding can make retries fail together. In such cases, measure the actual joint outcomes or model their dependence rather than applying the independent formula.
Let u be the probability that selection delivers a correct candidate conditional on at least one correct candidate being available. If the selector only chooses among generated candidates and cannot repair them, delivered correctness is . An abstaining selector also affects completion coverage. A selector that repairs candidates is a different process whose additional work and outcomes belong in the accounting.
For a hypothetical task with s = 0.60 and three independent candidates, coverage is 93.6%. If selection succeeds only 80% of the time when a correct candidate exists, delivered correctness is 74.88%. The gap consists of cases where usable work was generated but not successfully delivered. Repeating a weak selection procedure can therefore consume substantial resources without converting all the available candidate quality into useful work.
Brown and colleagues' 2024 repeated-sampling study illustrates this distinction on coding, mathematical, and formal-proof tasks. Coverage increased with additional samples, while the tested majority-vote and reward-model selectors failed to exploit all of that coverage on the math tasks. The paper also documented imperfections in software test suites. These findings support separate measurement of generation and selection; they supply no current universal sampling budget or workflow return.
Costs include every candidate generated, every candidate evaluated, orchestration, time spent waiting, final validation, and any fallback. A procedure that launches all k candidates pays for all of them even if one would have sufficed. A sequential procedure that stops early needs the distribution of attempts actually used. For independent attempts with a perfect immediate success check, , and a positive integer limit k, expected attempts are . At s = 0, all k attempts are used. An imperfect stopping rule requires a different calculation because it can stop incorrectly or continue after a correct result.
Using the same s = 0.60 and k = 3, expected sequential attempts are 1.56. If each attempt plus its perfect check costs $7 and fallback after three failures costs $30, expected resource cost is $12.84: $10.92 for attempts and $1.92 for fallback. Parallel generation of all three costs $22.92 including the same fallback. These are separate hypothetical designs with equal assumed success after fallback. Parallel work could justify its additional cost through lower elapsed time; the calculation has assigned no such benefit.
Repeated-use reliability asks another question. Under independence and a constant delivered success rate s, the probability that all n uses succeed is s to the power n. That can be relevant for a sequence that must succeed without intervention, but a service's economics may instead depend on defect counts, their severity, recovery, and concentrated incidents. The probability of an entirely clean batch does not replace those quantities. Dependence can alter both the all-success and all-failure probabilities, so it should not be summarized as a universal downward multiplier on reliability.
Checkpoints change observation and recovery
Splitting a task into smaller pieces does not automatically improve the final result. If each piece remains unobserved and errors propagate, a new boundary can add coordination cost without reducing failure. Decomposition can also lose context or introduce incompatible intermediate decisions.
A checkpoint becomes valuable when it establishes a useful property, detects a relevant failure before further work is committed, or enables a cheaper recovery. In a software workflow, validating an interface before generating dependent changes can prevent a mistaken assumption from spreading. In document work, confirming that the governing amendment has been identified can prevent repeated analysis of obsolete terms. The gain comes from the information and recovery opportunity at the boundary.
An additional checkpoint has a cost: checking effort, latency, state management, and possible false rejection. Its benefit is avoided downstream rework or loss, adjusted for the errors it actually catches. A checkpoint placed after an irreversible action may help learning and recovery but cannot prevent that action. A rollback can save work only if the retained state is valid and the process can resume without duplicating actions.
The economical checkpoint pattern therefore depends on where errors originate, how far they propagate, what can be checked, and what recovery costs. Compare a complete process with and without the proposed checkpoint, including successful runs. No constant per-step hazard model is required to justify this analysis, and no such model establishes that more fragmentation is always better.
8. Allocate human work where it changes outcomes
Human involvement has several economic roles: producing work, supplying missing context, interpreting requirements, reviewing candidates, authorizing actions, and recovering failures. Combining them into a single review-time number hides the choices. An expert who resolves an ambiguous requirement contributes something different from a person confirming that an already established condition has been met.
The relevant allocation depends on comparative performance within the cases each route receives. A model's confidence can help route work only if it predicts consequential outcomes on that workload. An expert's average accuracy is equally insufficient when the expert receives only unusual cases. Estimate human performance, time, and error costs on the actual delegated subset and under the proposed information presentation.
Agarwal, Moehring, and Wolitzky's August 2026 working-paper revision studies these choices in an incentivized online fact-checking experiment. Selective automation and delegation performed well in that setting. AI assistance added little on the delegated cases, while assistance improved accuracy when automation was unavailable. The result concerns a particular classification problem and policy model; it does not establish a general preference for unassisted humans or a rule for multi-step delegated agents.
The handoff can itself change behavior. In Jabbour and colleagues' revised 2026 clinical vignette study, withholding selected predictions partly mitigated the harm from inaccurate AI suggestions, while treatment false negatives remained elevated on the subset designated for withholding. The study used 259 clinicians and 45 cases, with a withholding mechanism constructed using known diagnoses and controlled noise. It was not a prospective clinical routing deployment. Its relevant implication is to measure the proposed human fallback with its actual signals and instructions.
Useful collaboration does not require independent human and model errors. It requires an arrangement that improves the relevant outcomes enough to justify its costs. A reviewer can catch some errors despite sharing others, and assistance can help a person acquire evidence or formulate alternatives. Conversely, displaying a confident recommendation before the person examines the evidence can alter the judgment being purchased. The combined process needs evaluation rather than an assumed sum of individual abilities.
Real oversight requires access to the necessary evidence, enough time to examine it, and authority to change the outcome. The interface should make consequential discrepancies inspectable and make uncertainty actionable. A confidence score without a tested relationship to error severity can distract from missing evidence. An approval button is useful only if the process gives the reviewer a practical opportunity to withhold approval, request information, or escalate.
Review also has opportunity cost. Assigning an expert to routine confirmation can displace difficult cases, training, or productive work. Concentrating review on higher-value interventions may improve the plan, but the routing policy must account for the failures it does not recognize. Some randomly selected audits can provide information about apparently easy cases that would otherwise remain unseen.
Sampling is a legitimate design choice under dependent production failures when the sampling design and inference match the question. A random sample of individual items from a fixed, defined lot can support finite-population inference even if a common cause produced its defects. A sample from one convenient batch cannot establish performance across suppliers, days, policy versions, or future conditions. For those questions, sample the relevant groups, preserve inclusion probabilities, and use uncertainty estimates that reflect clustering and drift. Acceptance sampling also requires a decision rule for what happens when a lot fails, such as reinspection or rerouting; NIST's account treats that rule as part of the plan.
Sampling after release primarily supplies monitoring and learning. It prevents an immediate harmful action only when the work can be held, recalled, or otherwise controlled before the consequence occurs. The economic comparison therefore includes detection delay and the cost of handling a rejected batch. A small audit cannot provide strong assurance about extremely rare severe errors, even if it finds none.
Human allocation should leave the organization able to complete the work when the primary route is unavailable. That requires current expertise, access, and capacity on the fallback route. A nominal manual fallback staffed by people who no longer know the procedure or who cannot absorb the volume is a contingency in name only. Maintaining that capability has a cost and may also support learning during ordinary operation.
9. Review capacity determines throughput and delay
Checking is a service process with arrivals, work in progress, and departures. Lower production cost can increase the number of candidates submitted for review. If review capacity remains unchanged, the extra candidates may create a backlog instead of more completed work. Even when average incoming work is below average capacity, variability can produce substantial waiting.
For a stable system with consistent population boundaries, Little's law relates the average number of items present Q to throughput lambda and average time in the system W:
(8)
This is an accounting relationship for flow, not a prediction that automation will create a particular delay. The system boundary must match on both sides: include waiting plus service in both Q and W, or queueing alone in both. Backlogs, abandoned items, and work diverted elsewhere need consistent treatment. Little's original result does not require a particular service-time distribution.
A more restrictive model illustrates why operating close to capacity can be expensive. Assume one continuously available reviewer, independent Poisson arrivals, independent exponentially distributed review times, first-come-first-served service, an unlimited queue, and no abandonment or interruptions. If arrival rate lambda is below service rate mu, mean waiting time before review is
(9)
Suppose average review takes six minutes, giving a service rate of ten items per hour. The model produces the following hypothetical results.
| Arrivals per hour | Utilization | Mean wait before review | Mean time including review |
|---|---|---|---|
| 8 | 80% | 24 minutes | 30 minutes |
| 9 | 90% | 54 minutes | 60 minutes |
An increase of one arrival per hour doubles mean elapsed time in this example without changing active review time. Real operations have shifts, priorities, batch arrivals, varied expertise, and difficult cases with long service times. Their delays need observations or a model that represents those conditions. The illustration establishes why average minutes saved in production cannot be carried directly into an end-to-end service claim.
Several responses can be economic. Reduce unnecessary submissions, provide better evidence, prevent defects upstream, change the routing policy, add capacity, or schedule work to respect deadlines. Faster review is useful only if it preserves required detection and handling. A reduction in average review duration can also conceal a longer tail of difficult cases or more reopening later.
Pressure from a queue may cause reviewers to rush, but a fall in catch rate is a behavioral hypothesis to measure. It is not a consequence of queueing algebra. Compare error detection and correction by workload, case difficulty, reviewer, and time available. A larger count of escaped errors can arise from higher volume even when the underlying rate is unchanged.
Capacity planning should include outages and common-cause incidents. A route that comfortably handles ordinary exceptions may fail when an interface change sends most arrivals to people. Reserves, restricted intake, staged operation, and a tested recovery plan can be worth paying for because they limit the consequences of those concentrated events. Their value belongs in the complete plan, alongside the ordinary-day utilization.
10. Investment and learning change the economics over time
Operating cost and performance can evolve after launch. Staff learn interfaces, input quality improves, corrections become reusable, and the organization discovers which work to assign to each route. Models, policies, customers, and source systems also change. A static per-task estimate is a starting point for an operating forecast, not a description of permanent performance.
Complementary organizational investment
The productivity J-curve literature explains how poorly measured investments in organizational capabilities can distort measured productivity. Resources used to build intangible assets can depress measured productivity initially; their later contribution can make growth appear stronger when those inputs remain incompletely counted. Brynjolfsson, Rock, and Syverson develop this mechanism and examine historical computer hardware and software investment. Applying it to an AI workflow is an inference about required complements, not evidence that a particular pilot will become profitable.
The relevant complements can include reliable records, clearer responsibility, system access, integration, training, and changed handoffs. Some support checking; others improve production or the usefulness of the result. If a customer-support assistant lacks current product information, more review may be an expensive response to a data-maintenance problem. If an accurate recommendation waits for authority no one owns, the problem is the decision process. Treating all complementary work as verification obscures these differences.
An initial performance shortfall can have several explanations. The technology may be unsuitable, the required complements may be absent, the organization may still be learning, or the benefit may have been overvalued. Each predicts different evidence. Integration failures should decline after a specific repair; interface learning should reduce active time on comparable cases; insufficient demand should leave released capacity unused even after technical performance improves. A general expectation of future learning cannot distinguish these explanations.
Investment should follow a specified mechanism and a plausible response. A proposal to improve context needs evidence that missing context causes material failures. A training investment needs a plausible route from the training to better decisions or faster work. A redesign needs to identify the handoff or responsibility it changes. Further expenditure is justified by its expected incremental contribution, including what would happen if the incumbent received the same improvement.
Corrections become assets only through reuse
A corrected output first completes the current task. It becomes a reusable asset only if the correction is captured accurately, associated with the relevant conditions, maintained, and used in future work. It might become a training example, an acceptance test, a retrieval source, a rule, or an improved instruction. Those uses require additional curation and implementation. The original correction effort should be counted once; the separate cost of making it reusable should also be counted.
Reuse can reduce future defect rates, checking effort, or recovery cost. It can also propagate an incorrect correction or hard-code a temporary condition. Feedback collected only from escalated cases describes the escalation policy's selection, while accepted errors may remain invisible. A system trained on approvals can learn the gate's preferences without improving the property the organization intended to measure.
Useful feedback therefore needs an evidentiary basis beyond the score being optimized. Retain source evidence, investigate disagreements, and test changes on work that did not supply the improvement signal. Where outcomes arrive later, connect them back to the original task. Independent adjudication can help, but useful feedback need not be statistically independent of all prior model behavior. Its value depends on what additional information it provides and how it changes future decisions.
A better dashboard is ambiguous until its measurement process is understood. Reported success can rise because actual work improved, easier cases were selected, the checker became more permissive, or difficult cases disappeared into unresolved work. It can fall because detection improved. There is no general law that dashboard bias must be positive or grow over time. Compare the dashboard with audits and downstream outcomes under a stable definition, and explain changes in the population or measurement.
The organizational asset is a collection of capabilities: maintained evidence, valid tests, integration knowledge, useful corrections, trained staff, and recovery procedures. A catch rate is a performance measure produced by those capabilities on a specified workload. Treating q itself as a stock hides changes in task mix and error type. The asset can improve while a more difficult workload lowers measured q, or measured q can improve while valuable expertise is lost.
Preserve expertise and develop new practitioners
Maintaining current expertise and creating future expertise are separate investments. An experienced reviewer may retain judgment through difficult cases while junior staff lose opportunities to learn routine production. Conversely, assistance may give novices worked examples and feedback that accelerate useful learning. Reduced practice, better instruction, changed task difficulty, and altered responsibility can operate simultaneously.
Beane's study of robotic surgery provides a bounded warning about task access. Its comparative ethnography and interview research found that the technology restricted trainees' participation and made established learning routes insufficient in that setting. It does not establish a general rate of AI-induced skill loss. The organizational question it raises is whether the redesigned process still gives learners practice with feedback and progressively greater responsibility.
Evidence can also point toward learning gains. The customer-support study discussed below reports an outage analysis consistent with some transfer of learning, although the estimates were noisy and lacked individual resolution outcomes. It provides a reason to examine learning, rather than assume either guaranteed improvement or inevitable dependence.
An organization can examine these mechanisms through comparable unaided tasks, supervised recovery exercises, review of unfamiliar cases, and observation of how novices progress. Producing a result, detecting a defect, and repairing a failed process are different capabilities. Measure them separately. Retaining a person in a nominal review role does not establish that all three are maintained.
Purposeful practice has an economic cost, including expert attention and work performed more slowly while someone learns. It can also preserve future production, checking, and fallback capacity. The appropriate amount depends on hiring prospects, turnover, the persistence of the work, and the consequences of losing expertise. A staffing plan that removes apprenticeship without providing another route should include the resulting uncertainty about future capability.
Separate learning from changing conditions
Track performance by comparable case groups, system version, and staff experience. A decline in correction time may reflect better tools, a changing workload, or the departure of difficult cases. Improvements observed only on a repeatedly used evaluation set may reflect adaptation to that set. Use fresh representative work to test whether the improvement transfers.
The same discipline applies to deterioration. An increase in errors may reflect source drift, a new interface, a model update, heavier queues, or lost skill. Identify the mechanism before purchasing a remedy. A model replacement cannot restore missing permissions; a training course cannot repair stale records. Maintaining a record of changes and their observed effects turns operation into evidence for subsequent investment decisions.
11. What productivity studies establish
Published studies show that AI assistance can improve measured work and can also make particular work slower or less accurate. Their differences are informative because interventions, populations, and outcome definitions vary. None of the studies below estimates every term of a complete investment-and-operation comparison.
| Study and setting | Source-reported result | Scope needed to interpret it |
|---|---|---|
| Brynjolfsson, Li, and Raymond, QJE 2025; 5,172 support workers | Approximately 15% more resolutions per hour | Staggered-rollout analysis at one firm's support operation; the resolution metric covers the subset with quality data. |
| Peng et al., 2023; 95 programmers randomized | 55.8% less completion time among completers; 95% interval 21%-89% | One JavaScript HTTP-server task with 12 visible tests; broader code quality was not assessed. |
| Cui et al., February 2025; 4,867 developers across three firms | 26.08% higher weekly pull-request counts; standard error 10.3 percentage points | Pooled weighted instrumental-variable estimate using randomized access to induce adoption; a work-volume proxy. |
| Dell'Acqua et al., September 2023; 758 consultants | 12.2% more subtasks within the tested frontier; about 19 percentage points lower correctness outside it | Separate constructed consulting exercises using historical GPT-4; the two outcomes are different measures. |
| Becker et al., July 2025 revision; 16 developers and 246 tasks | 19% longer completion time; 95% interval 2%-39% longer | Task-randomized permission on familiar mature repositories with early-2025 tools; active developer time. |
The support study measures an operational result beyond response generation. Workers could edit or ignore suggestions, and gains differed by experience and initial skill. Less experienced workers improved speed and quality, while the highest-skilled workers saw small speed gains and small quality declines. Its rollout mainly occurred in 2020-2021. These features matter more for transfer than the publication year alone.
The coding studies use different outcomes. A timed task with fixed tests, weekly pull-request counts, and active work on familiar repositories measure different parts of software production. Their percentages cannot be averaged into a coherent general rate. More pull requests can reflect more useful work or a change in how work is divided; faster completion can leave other reviewers' work unchanged.
METR's 2025 outcome included the participating developer's work before and after pull-request review, with some post-review times imputed. It excluded other reviewers' effort and review waiting. The study did not isolate checking as the causal source of the slowdown. Its February 2026 follow-up judged greater speedups likely but described the new magnitude evidence as weak because participation, submitted tasks, payment, completion, and concurrent-agent time reporting had changed.
A later METR survey, published in May 2026, reported positive perceived gains among technical workers. It used a convenience sample and had roughly a 2% response rate to email invitations. These self-reports add information about perceived use and value, while leaving the relationship to realized causal gains unresolved. The historical slowdown and the later perceptions should each retain their own design limits.
The evidence suggests several questions for transfer. Does the proposed use supply missing expertise, or interrupt people who already know the task well? Are requirements and evidence available to the tool? Does the outcome measure include review and subsequent use? Is the additional work valuable, and can downstream capacity absorb it? Have the method, interface, and workload changed since the study? These questions generate local hypotheses rather than a blanket prediction about novices, experts, or agents.
Benchmark performance can help shortlist technically plausible methods. It does not estimate local setup cost, the value of output, the effectiveness of the intended checks, or the resources required to maintain the operation. Likewise, a successful local pilot does not establish an enduring return if it excludes difficult work or relies on unusually attentive support. Evidence should be matched to the economic quantity it can actually inform.
12. Make the investment decision with proportionate evidence
The practical process begins with a small number of complete alternatives, a common outcome definition, and a credible picture of the incumbent. It becomes more detailed where uncertainty can change the choice. A low-cost reversible adjustment does not need the same evidence as a major integration or delegated authority over consequential actions.
Screen alternatives and identify decision thresholds
Use observed volumes, work samples, staffing, existing failures, and available price information to establish plausible ranges. Include transition, maintenance, and excluded cases. Separate estimates grounded in operation from supplier claims and assumptions. A screening estimate is useful if it rules out an unattractive option or identifies what a pilot must resolve; it need not pretend to provide a precise return.
Calculate the thresholds that matter. The legal example identifies the catch rate or checking cost at which its preference changes. A capacity investment may instead depend on how much released time can be used productively. A high-fixed-cost method may depend on future volume. Identify which quantities are close enough to their thresholds that better information could alter the decision.
The value of additional evidence depends on that possibility. If a feasible low-cost option remains preferable across credible assumptions, a broad experiment may add little. If a large irreversible commitment depends on an unmeasured review burden, a targeted pilot can be valuable. The evidence itself has a cost, and a pilot can create reusable integration or measurement assets, but neither benefit should be presumed.
Measure complete routes under comparable conditions
For shortlisted plans, record incoming work, all attempts, routing, active effort, elapsed time, accepted outcomes, justified rejection or no-action, correction, and unresolved cases. Link those records to charges and relevant downstream use. Keep system versions and material changes. Use the same follow-up window for all plans, and report work still pending at the end rather than treating it as free or absent.
Random assignment of comparable work can help estimate causal differences where practical. The assignment unit should reflect the operation: shared reviewers, queues, or learning can make one task's treatment affect another. Teams, shifts, or periods may be more suitable than individual requests, while creating their own requirements for adequate comparison. If randomization is impractical, matched workloads or before-and-after data can still inform a decision, with the assumptions about staffing, seasonality, case mix, and learning stated explicitly.
Measure the gate as well as the work. Audit accepted cases for escaped defects and rejected cases for false rejection or justified disposition. Include enough adjudicated defective cases to estimate relevant catch rates. Oversample rare, costly classes when needed for diagnosis, and use the appropriate weights when estimating population results. Keep diagnostic challenge performance distinct from the prevalence of failures in ordinary operation.
A zero-error result still has uncertainty. Under a stable independent Bernoulli model with accurate audits, zero defects in 300 randomly selected cases gives a one-sided 95% upper bound of about 0.994% on the defect probability. It does not establish a rate of zero or certify safety against an extremely rare severe event. For a fixed finite lot, use the finite-population model; for clustered work, changing conditions, or imperfect audits, use an analysis that represents those features. The numerical bound above is a calculation under its stated assumptions, not a general sample-size recommendation.
Source quality also affects measured performance. An evaluator can share the system's omissions, and a convenient reference answer can be wrong. Where economic ranking depends on a small difference, examine whether adjudication errors could reverse it. Where monetary severity is poorly known, use scenarios and enforceable exposure limits rather than assigning a precise average loss unsupported by observations.
Expand in stages and retain the option to change course
Before expanding, test whether review, fallback, and downstream capacity can support the proposed volume and difficult periods. A pilot may operate with immediate expert attention that will not exist at scale. Include the resources required to preserve its conditions, or evaluate the scaled design directly. A promising unit-cost estimate is incomplete if its assumed service level cannot be delivered.
Staging creates an option to learn before committing more resources. That option is valuable when the next stage can resolve an uncertainty that matters and when the organization can still change or stop the plan. Its cost includes delayed benefits, duplicate operation, and repeated integration. An endless pilot can consume those costs without producing decision-relevant evidence, so each stage should have a defined investment question and a completion criterion.
During operation, update forecasts using comparable evidence. Separate gains from tool changes, better inputs, staff learning, and altered case mix where the design permits. Reassess assumptions after material changes in authority, workload, model behavior, or staffing. A low-cost method may become expensive as exceptions grow; an initially costly method may improve after a specific reusable investment. Both are reasons to revise the plan rather than defend its original estimate.
The resulting decision should state which operating plan is preferred, which requirements it meets, the incremental resources it needs, and the assumptions that could reverse the choice. It should also identify who owns the unresolved work and the authority to limit or stop automatic action. That gives an organization a basis for choosing, operating, and improving AI-assisted work while keeping cost, risk, and learning connected to delivered outcomes.
References
Townsend, Robert M. 1979. "Optimal Contracts and Competitive Markets with Costly State Verification." Journal of Economic Theory 21(2): 265-293. DOI: 10.1016/0022-0531(79)90031-0. Federal Reserve primary record and abstract.
Kim, Elliot Myunghoon, Avi Garg, Kenny Peng, and Nikhil Garg. 2025. "Correlated Errors in Large Language Models." Proceedings of the 42nd International Conference on Machine Learning, PMLR 267: 30038-30066. Proceedings record; inspected preprint, version 1, June 9, 2025.
National Institute of Standards and Technology. 2023. Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST AI 100-1, January. Official framework.
Brown, Bradley, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher Ré, and Azalia Mirhoseini. 2024. "Large Language Monkeys: Scaling Inference Compute with Repeated Sampling." arXiv:2407.21787v1, July 31. Inspected version.
Agarwal, Nikhil, Alex Moehring, and Alexander Wolitzky. 2026. "Designing Human-AI Collaboration: A Sufficient-Statistic Approach." NBER Working Paper 33949; author revision dated August 20, 2026. NBER record; author PDF.
Jabbour, Sarah, David Fouhey, Nikola Banovic, Stephanie D. Shepard, Ella Kazerooni, Michael W. Sjoding, and Jenna Wiens. 2026. "Selective Prediction Reduces the Negative Effects of Automation Bias Overall but Increases False Negatives." arXiv:2508.07617v2, August 11. Inspected version.
NIST/SEMATECH. e-Handbook of Statistical Methods, sections 6.2.1-6.2.2, acceptance sampling and sampling-plan types. NIST, Dataplot, Single Sample Acceptance Plan, updated December 11, 2023.
Little, John D. C. 1961. "A Proof for the Queuing Formula: L = λW." Operations Research 9(3): 383-387. DOI: 10.1287/opre.9.3.383; accessible journal scan.
Brynjolfsson, Erik, Daniel Rock, and Chad Syverson. 2021. "The Productivity J-Curve: How Intangibles Complement General Purpose Technologies." American Economic Journal: Macroeconomics 13(1): 333-372. DOI and published abstract. Earlier version: NBER Working Paper 25148, October 2018, institution-hosted PDF.
Beane, Matthew. 2019. "Shadow Learning: Building Robotic Surgical Skill When Approved Means Fail." Administrative Science Quarterly 64(1): 87-123. Published online January 9, 2018. DOI and publisher abstract.
Brynjolfsson, Erik, Danielle Li, and Lindsey Raymond. 2025. "Generative AI at Work." Quarterly Journal of Economics 140(2): 889-942. DOI: 10.1093/qje/qjae044; author-hosted published PDF.
Peng, Sida, Eirini Kalliamvakou, Peter Cihon, and Mert Demirer. 2023. "The Impact of AI on Developer Productivity: Evidence from GitHub Copilot." arXiv:2302.06590v1, February 13. Inspected version.
Cui, Kevin Zheyuan, Mert Demirer, Sonia Jaffe, Leon Musolff, Sida Peng, and Tobias Salz. 2025. "The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers." Working paper, February. Inspected author PDF.
Dell'Acqua, Fabrizio, Edward McFowland III, Ethan Mollick, Hila Lifshitz-Assaf, Katherine C. Kellogg, Saran Rajendran, Lisa Krayer, François Candelon, and Karim R. Lakhani. 2023. "Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality." Harvard Business School Working Paper 24-013, September 22. Inspected working paper.
Becker, Joel, Nate Rush, Beth Barnes, and David Rein. 2025. "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity." arXiv:2507.09089v2, July 25. Inspected version.
Becker, Joel, Nate Rush, Tom Cunningham, David Rein, and Khalid Mahamud. 2026. "We Are Changing Our Developer Productivity Experiment Design." METR, February 24. Investigators' update.
METR. 2026. "Measuring the Self-Reported Impact of Early-2026 AI on Technical Worker Productivity." May 11. Research report.