Designing and evaluating customer interventions

Customer interventions change what a business does for, offers to, or asks of a customer. A service call, an acquisition offer, a different onboarding sequence, and a new price each create a decision about who is eligible, what happens next, and which outcomes justify the change. A useful evaluation connects those choices to evidence about customer behavior and incremental contribution.

This resource develops that connection from a proposed action to a tested operating policy. It explains when prediction or an established rule is sufficient, how to design a credible comparison, when heterogeneous treatment effects help, and how to assess a policy under contact, budget, and capacity constraints. It includes a hypothetical expansion-offer experiment and a worksheet for specifying an intervention before implementation.

The companion paper, Customer analytics: prediction, experimentation and business decisions, covers the connected account of acquisition, activation, retention, expansion, customer value, and implementation. Here the starting point is a particular intervention decision. The result may be a simple rule, a broadly available service change, a targeted offer, or a decision to leave the current process in place.

1 Define the intervention and the decision population

Write the decision in operational terms before choosing an estimator. For example: at an account's first eligible expansion opportunity, should the business assign an offer of an additional service with a credit, provide the same information at the ordinary price, or make no additional contact? Specify the account, offer terms, channel, timing, duration, and ordinary services that continue in every alternative. "Run a retention campaign" leaves too much unspecified to interpret a treatment effect.

Choose a unit that connects assignment to the business outcome. A business account may contain several users, a household several customer IDs, and a prospective company several sales contacts. Randomizing emails independently can give the same economic relationship conflicting treatments. The analytical unit need not be the event that happens to produce a row in the data.

Eligibility must be determined using information available at assignment. Define exclusions, prior contacts, consent, existing contracts, product availability, and any service obligations. Record the assignment time and the outcome horizon from that time. Eligibility based on subsequently opening a message or remaining active would select people using behavior the intervention could change.

The counterfactual is the specified alternative action or policy. "No added contact" ordinarily preserves access to support, billing messages, and existing product functionality. A comparison with the current sales process answers a different question from a comparison with no proactive sales activity. Both can be useful. The choice depends on the decision management can actually make.

Table 1 illustrates how the same sequence applies across customer decisions. The examples are design possibilities, not evidence that the proposed interventions work.

Table 1. Intervention design follows the decision, assignment unit, and operating constraints. A customer-level experiment is appropriate only when it captures the relevant exposure and spillovers.

Application	Action and comparator	Unit and design consequence	Outcome and constraint
Acquisition	Additional sales call versus the existing follow-up process	Prospect company; link contacts and avoid assigning different salespeople to the same prospect	Contribution from new business after sales effort; constrained call capacity
Activation	Assisted setup offer versus ordinary onboarding	Account at a defined setup stage; preserve necessary technical support	Successful setup and later paid contribution; assistance slots and completion delays
Retention	Proactive service review versus ordinary renewal service	Account or household; align assignment with renewal and prior outreach	Renewal and contribution over a stated period; contact limits and service obligations
Expansion and offers	Add-on credit versus no additional offer, or a message without the credit	Account; separate the value of contact from the value of the incentive if that distinction matters	Additional relationship contribution after credit, returns, and displaced purchases
Service changes	Different support routing versus the current queue	Customer, support team, or queue; shared capacity can affect untreated customers	Resolution, waiting time, repeat contacts, and contribution; minimum service standards
Pricing	A defined price or price schedule versus the current schedule	Account, market, or time block; arbitrage and shared demand can create interference	Contribution, demand, and customer outcomes; permitted prices, inventory, and commitments

Separate outcomes from effects and decision value

Let X denote information available before assignment, A the assigned action, and Y(a) the outcome over the chosen horizon if action a were assigned. Action 0 is the stated comparator. A response prediction estimates an outcome under the conditions represented in its training data. A conditional average treatment effect (CATE) compares two potential outcomes:

$$\tau_{a}(x)\ = \ E\lbrack Y(a)\ - \ Y(0)\ |\ X\ = \ x\rbrack$$

For a binary purchase outcome, this is a difference in purchase probabilities for customers with characteristics X = x. Only one potential outcome is observed for each customer in a particular decision. A model estimates a conditional average; it does not recover a person's unobserved alternative history. $$1, 9$$

The familiar four response types describe possible pairs of binary potential outcomes: purchase under both actions, purchase only under the intervention, purchase under neither, or purchase only under the comparator. These are latent categories for a particular action and horizon. An estimated positive effect does not identify someone as an observed "persuadable," and a negative estimate does not reveal an individually known "sleeping dog." Nor do these categories describe every consequence of an action: a customer could purchase under both actions while changing basket size, support demand, or future behavior.

Define G(a) as contribution over the same horizon under action a, after the costs included in the decision. The economic effect is:

$$\Delta_{a}(x)\ = \ E\lbrack G(a)\ - \ G(0)\ |\ X\ = \ x\rbrack$$

Specify whether contribution includes discounts, fulfillment, returns, payment costs, incremental service work, contact attempts, and customer remediation. Discount later cash flows when the horizon warrants it. Past acquisition spending is sunk when comparing actions for an existing customer; new acquisition spending belongs in an acquisition decision. Customer lifetime value provides a forecast of a relationship's discounted future contribution. It does not establish what a new intervention adds. $$3$$

Monetizable harms belong in the economic outcome where they can be measured credibly. Complaints, unwanted contact, unfair access, and failures to provide required service may also impose constraints that cannot be traded away for a larger revenue estimate. State those limits separately rather than assigning an arbitrary dollar value to every customer outcome.

2 Choose evidence that can answer the proposed decision

A forecast can support staffing, inventory, renewal planning, or the scheduling of an established response. A prediction of a likely service fault can prioritize inspection; a confirmed fault can trigger a repair whose success is directly verifiable. These uses do not require fitting a new uplift model. Claims that the resulting process improves retention or contribution still need evidence for that additional outcome.

An existing rule may already have credible evidence for its action, population, and operating conditions. If a transparent eligibility rule captures the relevant variation, comparing that rule with current practice may settle the decision. When effects are sufficiently similar and contact is inexpensive, offering an intervention broadly can also be preferable to individualized targeting. Low delivery cost alone does not rule out customer harm, attention costs, or interference with other work.

Historical response records describe behavior under the historical policy. A churn model trained while account managers intervene preferentially on some accounts predicts outcomes partly shaped by that intervention. It need not estimate untreated churn risk. Similarly, the high observed conversion of contacted prospects can reflect how salespeople selected them. Before treating a historical difference as an effect, reconstruct who could receive each action and why they did.

Ascarza's retention research demonstrates the practical relevance of this distinction. In two field experiments, customers with the highest predicted churn risk were not necessarily those most responsive to the studied intervention. This supports testing a targeting rule against the available action. It supplies no universal ranking of risk models and treatment-effect models, or transferable efficiency percentage. $$2$$

Prefer random assignment when it is feasible

Randomization creates a known assignment mechanism. Within the experimental population and any randomization blocks, assignment is independent of potential outcomes by design. An offer, invitation, service process, or policy can often be randomized even when actual customer uptake cannot. Preserve the assignment record and evaluate the assigned process.

The experiment should include the populations for which future decisions are contemplated. If only the highest-risk customers are eligible for an initial trial, its evidence does not establish the effect for lower-risk customers. A risk filter can be a sensible scope restriction, but it is also a boundary on what the experiment learns.

When all customers must receive a service, randomize a permissible addition, timing variation, or delivery method if that addresses the question. A trial should not withhold contractual or necessary service to create a convenient control group. Where an effective action is already established, compare feasible alternatives or an improvement to the current policy.

Make observational identification explicit

Observational evidence can support causal estimation when the assignment mechanism and assumptions are credible. Three requirements organize the assessment. Consistency means the recorded action corresponds to the intervention being evaluated, including relevant versions. Exchangeability, often expressed as no unmeasured confounding conditional on X, means the recorded variables account for the common causes of assignment and outcome. Positivity means each action being compared was possible for the relevant customer profiles. $$1, 13$$

Consider discounts negotiated by account managers. If a private conversation about an imminent competitor switch drives both the discount and subsequent departure, a CRM model lacking that information leaves confounding unresolved. If every large account receives a discount, no observed untreated comparison exists for that group. A more flexible model may produce a number there, but the comparison depends on extrapolation.

Document the assignment process with the people who made the decisions. Check weighted covariate balance, inspect action probabilities and effective sample sizes, and identify populations excluded for lack of support. Balance addresses measured variables. Sensitivity analyses ask how strong omitted confounding or alternative assumptions would need to be to change the conclusion; they do not certify that confounding is absent.

Other assignment mechanisms can justify different designs. A threshold rule may support regression discontinuity near its cutoff when customers cannot precisely manipulate the assignment score and other determinants of the outcome change smoothly. A rollout with credible untreated comparison groups may support a difference-in-differences design when their untreated outcome trends would have been comparable. Anticipation and spillovers also need to be addressed in the design. An instrument requires an assignment-specific causal argument. Choose these approaches because the operating process supports their assumptions, and state the population and effect they identify. Merely having dates, a cutoff, or an available instrument-shaped variable is insufficient. $$5, 6, 18$$

If the necessary assumptions are implausible, retain the records for forecasting, accounting, and designing a future comparison. Narrowing the decision or introducing supported randomized variation is often more informative than estimating a complex effect from an unexplained assignment process.

3 Design the experiment around the operating process

An experiment plan should specify an intervention that can be delivered, an outcome that can mature, and a comparison that can inform an operating choice. Randomize after eligibility is established and before the intervention can affect any selection variable. Where baseline size, renewal month, or geography matters, randomize within prespecified blocks and use an analysis that respects the allocation probabilities.

Compare assigned counts with the intended allocation, test identifier stability, and confirm that outcomes can be linked in both arms. A sample-ratio mismatch can reveal an assignment or logging defect. Logging whether a control unit would have triggered an intervention helps define comparable opportunities, provided the trigger is unaffected by treatment. It does not reveal a counterfactual customer outcome. Microsoft's experimentation guidance discusses these engineering checks and the need to examine shared infrastructure. $$4$$

Set a comparator that matches the policy question

Keep ordinary activities explicit. Ascarza's professional-membership experiment compared a renewal letter plus a thank-you gift with the renewal letter alone. A comparator can therefore include meaningful customer contact. $$2$$ Likewise, a new expansion policy might be tested against the existing sales rule, with a separate no-additional-contact arm only if that comparison is useful and feasible.

Competing campaigns require a deliberate choice. Suppress the same campaigns in both arms when estimating the isolated addition, randomize combinations when interactions matter, or let background activity continue while defining the effect as the intervention's addition to that background. If one campaign displaces another, the displacement is part of the policy contrast. Control exposure does not always bias an effect downward; its consequences depend on what the comparison actually contains.

Choose units and timing that address spillovers

Household communication, company accounts, referrals, shared inventory, and service queues can make one customer's assignment affect another's outcome. Cluster randomization can contain some of those effects by assigning a household, account group, store, or market together. It changes the effective number of independent units and requires inference that respects the randomization and within-cluster dependence. Cross-cluster spillovers can remain. Designs for interference distinguish direct effects from effects on others and from the total effect of an allocation. $$5$$

A lightly loaded service pilot may not estimate rollout at full capacity. If additional outreach congests support, both treated and untreated accounts can experience a changed service. Consider randomizing whole queues or operating periods, or testing treatment saturation explicitly. A switchback across time blocks requires attention to calendar patterns and effects that carry into later blocks. A promotional price can change purchasing after the offer ends, so an arbitrary short washout does not create independent observations.

Keep repeat eligibility under control. For a one-time test, assign a stable customer identity once and define when the person can reenter. For a program of repeated contacts, the treatment is a sequence or decision rule over time. A whole-program holdout can estimate the effect of that package. Randomizing at successive eligible occasions can answer more specific questions, but must record the history, probabilities, and outcome windows.

Prior contact can change activity that drives later contact. That activity is then both a consequence of earlier treatment and a determinant of later treatment. Ordinary adjustment can block part of the effect or introduce bias. Longitudinal approaches, including the g-formula and marginal structural models, require sequential identification and support for the histories being evaluated. A single-contact CATE model does not automatically identify a repeated-contact policy. $$7$$

Preserve assignment when delivery or uptake fails

Let assignment be A and actual receipt be D. The primary intention-to-treat comparison estimates the effect of assigning the offer or process, including its delivery failures, refusals, and uptake under the tested implementation. Keep assigned customers in their groups even if an email bounces or a service appointment is declined. Those events explain implementation performance; filtering them out changes the comparison.

Comparing customers who received or accepted treatment with those who did not generally loses randomization. Under additional assumptions, assignment can act as an instrument for receipt. For a binary setting, a complier effect can be obtained from the assignment effect on the outcome divided by its effect on receipt, provided assignment affects outcomes only through that receipt, shifts receipt, satisfies monotonicity, and meets the other design assumptions. Monotonicity means assignment does not cause anyone to forgo treatment they would otherwise receive. The resulting effect applies to customers whose receipt changes with assignment. An invitation that influences behavior without uptake can violate the exclusion restriction. $$6$$

For most implementation decisions, the assignment effect is directly relevant: management can choose whether to operate the offer process. A treatment-received estimate is useful only if its population and assumptions answer an additional decision.

Plan precision and follow-up before reading results

Set the smallest economically useful difference and the uncertainty management can tolerate. Sample size depends on the outcome variance, effect size, allocation, clustering, planned comparisons, and analysis. For independent equal-sized arms with a continuous outcome and similar variance, the usual normal approximation is proportional to variance divided by the squared effect to be detected. Reducing the detectable effect substantially can therefore require substantially more observations. The underlying normal-test relationship is described in the NIST handbook. $$17$$

Power a net-contribution decision for the distribution of net contribution. Purchase rates can be easier to estimate precisely while the profit comparison remains uncertain. Unequal allocation, rare outcomes, heavy-tailed spending, small numbers of clusters, and subgroup interactions require their own calculations. Prespecified adjustment for predictive pretreatment variables may improve precision. Estimate its likely contribution from appropriate historical data rather than assuming a fixed benefit.

Use a calendar that includes enrollment and complete follow-up. A thirty-day outcome needs thirty days after the final enrollment, plus any justified reconciliation period. Incomplete follow-up is not a zero outcome. Track missing outcomes and linkage failures by assigned arm. Weighting or imputation requires defensible missingness assumptions; show sensitivity where plausible missing outcomes could change the decision. $$1$$

Prespecify the primary outcome, economic decision threshold, customer guard measures, and stopping rule. Repeatedly checking an ordinary fixed-sample significance test and stopping when it becomes favorable changes its error rate. Use a planned sequential method if early stopping is required. $$19$$ Multiple offers and exploratory segments also create multiplicity. A nonsignificant result can reflect insufficient precision; failure to detect harm does not establish safety. For a customer guard measure, define the maximum acceptable deterioration and assess uncertainty relative to that bound.

4 Estimate heterogeneity when it can change the action

Begin with the average assignment effect and its uncertainty in the eligible population. If the business can offer only one action broadly, that result may answer the immediate question. Heterogeneity matters when feasible customer characteristics could justify different actions, or when limited resources make prioritization consequential.

Start with a small number of prespecified segments grounded in the intervention mechanism. A setup service may be more useful for accounts with a particular integration difficulty; an expansion offer may differ by current capacity usage. Compare effects directly across segments. One significant estimate and one nonsignificant estimate do not themselves establish a difference between the effects.

Regression with treatment interactions can estimate these comparisons, with regularization or partial pooling when many small segments would otherwise be noisy. For binary outcomes, distinguish the modeled scale from the decision scale: an interaction on log odds is not itself a difference in purchase probabilities. Evaluate predicted outcomes under both actions and express the contrast on the scale used for the decision.

When observed variation is more complex, flexible methods may help. The choice depends on support, signal, imbalance, and the structure of the outcome and effect functions. Baseline outcome and action-probability models are often called nuisance models because they support effect estimation without being its target. Table 2 summarizes what additional methods contribute, without prescribing an algorithm from a sample-count threshold.

Table 2. Methods add flexibility to an identified comparison. None resolves missing action support or unmeasured confounding by itself. Meta-learner definitions follow Künzel et al.; residualized estimation follows Nie and Wager; forest inference follows Wager and Athey. $$9--12$$

Approach	Contribution to the analysis	Limitation that matters for a decision
Prespecified strata or interaction model	Estimates effects for actionable groups; provides an interpretable baseline	Small groups and many interactions create unstable comparisons; shrinkage and independent confirmation matter
S- or T-learner	Uses one outcome model with action as an input, or separate models by action; compares their predictions	Outcome fit need not imply accurate differences; regularization and unequal data can distort the effect
X-learner	Models imputed effect differences and combines estimates; can exploit some imbalanced data settings	Advantages depend on the response and effect functions; imbalance does not excuse absent overlap
R-learner and orthogonal-score methods	Separates baseline outcome and assignment patterns from the effect-estimation problem	Local protection against nuisance-estimation error still requires identification, support, and suitable estimation quality
Causal forest	Learns nonlinear effect variation through local comparisons across an ensemble of partitions	Honest fitting and inference depend on sampling, overlap, and forest conditions; sparse regions remain uncertain

The S-learner can shrink treatment variation too strongly; the T-learner can estimate two complex outcome surfaces whose errors dominate their difference. The X-learner uses estimated counterfactual outcomes as intermediate quantities, not newly observed individual effects. R-learning fits outcome residuals against assignment residuals to focus estimation on treatment variation. Orthogonality reduces sensitivity to small nuisance-estimation errors under its conditions. It is not a general claim that either nuisance model may be arbitrarily wrong. $$9, 10$$

Keep discovery and evaluation separate

Three separations address different problems. Honest tree estimation uses different outcome observations to choose partitions and estimate effects within them. Cross-fitting trains nuisance models on other folds before constructing each observation's causal score. Independent policy evaluation uses outcomes that did not select the model, segments, thresholds, or action rule. Honest subgroup methods and double/debiased machine learning motivate the first two separations. Neither automatically supplies the third. $$8, 11, 12$$

Regularization controls the variance created by searching many weak patterns. Assess whether the learned differences produce a stable decision at the intended capacity. If small changes in training data cause large changes in who receives a costly action, simplify the policy, collect more informative evidence, or limit the deployment population. Instability near a threshold is less consequential when either action has similar value, but more consequential when an error can harm customers.

A model explanation can describe which inputs influence an estimated effect or assignment. It cannot show that those inputs caused responsiveness, that confounding was removed, or that a particular customer would benefit. The related explainability paper develops these interpretation limits. Here the operative evidence is the design, assumptions, and performance of the resulting policy.

5 Convert estimated effects into a feasible policy

A policy maps the available customer information and operating state to an allowable action. For each action, compare expected contribution with the same comparator, horizon, and cost boundary. Estimate the effect on contribution directly when the action changes purchase amount, margin, refunds, or service use. Multiplying a conversion effect by a fixed margin is valid only when those simplifying assumptions fit the intervention.

For multiple actions, include no additional contact or no change as an explicit alternative. If a service call, a credit, and an information message are all feasible, each needs its own supported comparison. The action with the largest estimated conversion effect may have lower net contribution or consume capacity that is more valuable elsewhere.

With equal action costs, one action, and no other constraints, a rule can select the highest positive estimated net effects up to capacity. Unequal resource use changes the problem. A simple value-to-cost ranking need not maximize total value when actions are indivisible, customers have several alternatives, or several budgets bind. Specify the allocation problem as maximizing total estimated incremental contribution subject to eligible actions, at most one selected action per decision, contact limits, and resource constraints. Constrained policy learning formalizes choosing rules within an allowed policy class. Its guarantees are relative to that class and rely on identified effects and statistical assumptions. $$13$$

Keep cash costs and capacity use distinct. An outreach attempt may consume a fixed slot, while a credit consumes cash only if redeemed. Use expected redemption costs for expected-value comparisons, and a reservation, cap, or other explicit rule when a budget must never be exceeded. If reaching a spending cap changes which later customers receive an offer, that behavior becomes part of the policy to evaluate.

Estimation uncertainty also needs a decision rule. An interval spanning zero does not automatically require inaction, and a small p-value does not justify an uneconomic offer. Management can compare expected value, downside exposure, customer harms, reversibility, and the value of further information. Conservative lower-bound allocation is one possible choice; its treatment of multiple comparisons and shared uncertainty must match the analysis. Avoid assuming that adding individual lower confidence bounds produces a valid portfolio bound.

Constraints determine what can be selected even if effects were known exactly. Estimation error determines how reliable the value estimates are. Keep these sources of limitation separate. A known profitable action may be infeasible because a customer has opted out or capacity is exhausted. An operationally feasible action may have uncertain benefit. Eligibility and service rights are not confidence thresholds.

Fairness and legal requirements also shape the feasible policy. Define permitted uses of customer attributes, service entitlements, and allocation criteria before optimization. Review who gains access to an offer or assistance and how customer outcomes differ across relevant groups. Assign responsibility for checking the proposed policy against the rules that apply to its product and jurisdiction. A favorable average contribution estimate does not resolve those questions.

Specify the executable details: when eligibility is rechecked, how ties are broken, which action is chosen if a channel is unavailable, when queued actions expire, and who can override a decision. Evaluate the actual fallback. Replacing a failed low-cost message with an untested call changes both the intervention and its economics.

6 Evaluate a proposed policy on unused outcomes

Evaluation should compare the candidate with a meaningful baseline, such as the existing rule, no added contact, a uniform allocation at the same capacity, or a prespecified segment policy. Use the same eligible population, horizon, resource limits, and accounting. Report value per eligible customer and total value at the planned scale. Value per contacted customer alone can conceal poor coverage or a different contact rate.

For a fixed policy π in a one-decision setting, define its value as:

$$V(\pi)\ = \ E\lbrack G(\pi(X))\rbrack$$

The incremental value is the difference from the comparator policy's value. This includes customers for whom the policy selects no change. A held-out randomized dataset can estimate a proposed policy even if that exact targeting rule has never operated, provided the data contain supported actions for the customers the rule would assign. This is off-policy evaluation. It estimates a policy under the stated data-generating conditions, not its realized production result. $$14$$

Combine action probabilities with outcome information

Inverse probability weighting gives more weight to records whose observed action was unlikely under the logging policy but would be selected by the target policy. A doubly robust estimator combines this correction with an outcome model. For a fixed deterministic policy, a useful form is:

$${\widehat{V}}_{DR}(\pi)\ = \ \frac{1}{n}\sum_{i\ = \ 1}^{n}{\psi_{i}(\pi)}$$

$$\psi_{i}(\pi)\ = \ {\widehat{m}}_{\pi(X_{i})}(X_{i})\ + \ \frac{1\{ A_{i}\ = \ \pi(X_{i})\}}{{\widehat{e}}_{A_{i}}(X_{i})}\lbrack G_{i}\ - \ {\widehat{m}}_{A_{i}}(X_{i})\rbrack$$

Here $\psi_{i}(\pi)$ is each evaluation record's contribution to the estimate; n is the evaluation sample size; ${\widehat{m}}_{a}(x)$ estimates expected net contribution for action a; and ${\widehat{e}}_{a}(x)$ is its historical assignment probability. The indicator is one when the observed action matches the target policy, and $G_{i}$ is the observed net contribution. Use known randomization probabilities when available. The first term predicts the policy outcome; the second corrects it using observed residuals on matching actions. $$14$$

Under identification and regularity conditions, the double robustness property permits consistency when the action-probability model or the action-specific outcome models are correctly specified. It does not remove hidden confounding, poor support, dependence, or transport assumptions. Cross-fitting can help when nuisance models are learned, while valid uncertainty additionally depends on their estimation quality and the analysis design. $$11, 14$$

A historical deterministic policy may never offer a credit to a particular class of accounts. Evaluating a target policy that does so requires unsupported extrapolation. Restrict the claim to a supported population or collect evidence for the missing action. Where probabilities are merely small, inspect weight concentration and effective sample size. Clipping weights trades variance for bias; disclose the rule and its sensitivity rather than treating clipped estimates as an unchanged answer.

These formulas describe a single decision without relevant interference. A batch optimizer can make an account's assignment depend on other accounts and the day's capacity. A threshold learned for one population can exceed capacity in another. Freeze and evaluate the complete selection procedure at the intended batch size and workload, or define a policy whose expected and hard capacity limits are explicit. Sequential actions and evolving customer states need an evaluation method for that history, rather than repeated application of a one-step formula.

Separate ranking quality from value at an operating point

An uplift curve ranks customers using a score fixed before evaluation, then estimates cumulative additional outcomes as the selected share grows. Qini and area under the uplift curve (AUUC) summarize that ranking across selection depths. State the normalization, random-targeting reference, and assignment correction: names and scaling vary between implementations. Ranking metrics can compare a risk score, a hand-built rule, and an effect model on the same evidence. $$15, 16$$

A high ranking score does not determine the value at the contact rate the business can use. Curves can cross; customer margins, delivery costs, and constraints can differ. Evaluate the net value of the actual rule at its operating capacity and attach uncertainty. Check effect calibration in prespecified score bands if the numerical effects determine spending. Classification accuracy and the area under the receiver operating characteristic curve (ROC AUC) can diagnose outcome models, but do not validate treatment-effect estimates.

Selecting the best threshold, offer, or model on a test set and quoting its test value reuses the outcomes for selection. Reserve an independent final evaluation set, use an appropriate nested procedure, or confirm a frozen policy in a later randomized trial. Nuisance cross-fitting alone does not address this selection bias. If an analysis evaluates a learning procedure across folds, distinguish that result from evaluation of the final policy retrained on all data.

For a fixed candidate and comparator, estimate their value difference using the same evaluation observations and account for covariance. Cluster-aware or design-based uncertainty must follow the assignment and dependence structure. A bootstrap of independent rows is inappropriate for clustered customers or repeated contacts. Uncertainty about a newly optimized policy can be more difficult than uncertainty about a fixed policy; repeatedly selecting and scoring the winner within a naive bootstrap is not a universal solution.

Check whether the evidence transfers

Random splits can support within-experiment estimation when independence and feature timing are appropriate. They do not inherently leak future information. A separate later-period test asks whether the policy transfers to future customers, prices, acquisition channels, and delivery conditions. Construct every feature as it was available at the decision time, fit only with matured training outcomes, and use grouping that reflects the intended deployment task.

Compare support and delivery by cohort and relevant customer groups. Reweighting can address changes in the distribution of measured characteristics under additional stability assumptions. It cannot establish an effect for a newly introduced action or repair a changed intervention mechanism. A policy that works during a discount season may need new evidence after ordinary prices return.

7 Worked example of an expansion credit experiment

Consider a wholly hypothetical test of an offer for an additional service. Ten thousand eligible accounts are randomly assigned, with 5,000 in each arm, to no additional offer or an offer with a $6 credit. Accounts can buy the service without the offer. Ordinary service remains available in both arms. The decision is whether to continue this offer process and whether further evidence warrants targeting it.

Each account can make one qualifying purchase within thirty days. Without the credit, a purchase contributes $40 after ordinary variable costs. Every qualifying purchase in the offer arm receives the $6 credit automatically, regardless of whether the message is opened. Assignment creates a $0.20 contact-processing cost even if delivery fails. These assumptions deliberately separate the cost of the offer from the incremental purchases it might create.

Table 3. Invented trial counts and fixed economics illustrate an assignment-based comparison. No customer experiment was conducted. The arithmetic excludes fixed program costs, later-period effects, spillovers, and any additional harms; a real evaluation must assess those boundaries.

Hypothetical measure	No additional offer	Offer with credit
Accounts assigned	5,000	5,000
Accounts with a qualifying purchase	500	650
Purchase rate per assigned account	10.0%	13.0%
Contribution per purchase after credit	$40	$34
Contact cost per assigned account	$0	$0.20
Total contribution after contact cost	$20,000	$21,100
Contribution per assigned account	$4.00	$4.22

The estimated purchase effect is 3 percentage points. At 5,000 assignments, that corresponds to 150 additional purchases relative to an equally sized comparator group. The additional contribution is $1,100, or $0.22 per assignment:

$$0.13\ \times \ (\$ 40\ - \ \$ 6)\ - \ 0.10\ \times \ \$ 40\ - \ \$ 0.20\ = \ \$ 0.22$$

The credit costs $3,900 across all 650 treated purchases. Calculating $40 times the 150 additional purchases and subtracting only contact costs would report $5,000. It would omit the credit expense on the whole treated purchasing group. The difference matters even though no individual can be identified as someone who would have bought anyway.

Under the illustration's independent Bernoulli assumptions, a large-sample 95% interval for the purchase effect is approximately 1.75 to 4.25 percentage points. Applying the same uncertainty calculation to the net-contribution outcomes gives approximately -$0.24 to $0.68 per assigned account. Purchase lift is distinguishable from zero under this model, while economically positive contribution remains uncertain. The interval does not include uncertainty in the assumed margins or omitted consequences.

Suppose delivery logs additionally report that messages reached 4,000 of the 5,000 offer assignments. The primary denominator remains 5,000. The automatic credit, attempted contact, and failed delivery are all part of the assigned process. Comparing only reached accounts with all controls would answer a selected, nonrandomized question.

An operating decision should consider the $0.22 estimate alongside its uncertainty, fixed costs, customer measures, and the cost of obtaining better information. A positive point estimate alone does not establish that rollout is worthwhile. Nor does this pooled result show which accounts should receive the offer. If development data suggest a segment rule, freeze the segment definition and offer terms, then evaluate that rule on independent supported data or in a subsequent policy trial.

8 Connect assignment to delivery and sustained measurement

The data path should preserve the decision that was made and what happened afterward. Begin with a stable customer or account identity and a unique decision ID. Snapshot eligibility, permitted actions, relevant contact history, and features available at assignment. Record policy and model versions, the selected action, the comparator, randomization block, and the probability used by the assignment mechanism.

The execution record should connect that decision to attempts, delivery, acceptance, completion, and any override. Recheck opt-outs and eligibility before execution, deduplicate queued tasks, and expire stale actions. If a customer opts out after assignment, honor the opt-out and retain the assignment record for the primary analysis where data use permits. Do not silently move the customer to control. Record competing contacts and fallback actions so the implemented treatment remains interpretable.

Join outcomes at the prespecified horizon, with explicit maturity and linkage status. Preserve no-purchase outcomes as well as purchases, and reconcile credits, refunds, service effort, and action costs. A missing transaction link is not evidence that no purchase occurred. The commercial and finance owners need an agreed contribution definition; the delivery owner needs a record of what was actually attempted and completed.

Maintain two views of performance. Operational monitoring covers identity linkage, eligibility, queue delay, delivery failure, capacity, opt-outs, and outcome completeness. Causal measurement compares assigned policies or actions. A rising conversion rate, stable score distribution, or well-calibrated prediction does not establish incremental benefit.

A sustained randomized holdout can measure the effect of a program through changing campaigns and delayed outcomes. Define which discretionary activities the holdout excludes and which ordinary services continue. A program-level holdout estimates the package against that comparator; a campaign-level experiment estimates a particular addition. Layering experiments requires explicit membership rules and accounting for interactions. There is no universal holdout percentage, and a group permanently denied all marketing is not required for every causal question.

Revisit the policy when delivery changes, a new action is introduced, support disappears, or customer effects plausibly shift. Diagnose the cause before retraining. A contact-rule change can alter exposure, a new channel can alter the population, and a queue bottleneck can alter the intervention itself. When long-term outcomes remain immature, report the shorter outcome actually measured. Treat engagement or early spending as a surrogate for later value only with evidence that the proxy supports that intervention decision.

9 A worksheet for an accountable intervention decision

Complete this worksheet before launch and update it when the policy changes. Its purpose is to expose decisions that would otherwise remain implicit. The responsible owners can replace the prompts with agreed specifications; no modeling platform is required.

Table 4. Reusable intervention design worksheet. Thresholds, horizons, and permissible actions are contextual choices to agree before observing favorable results.

Design field	Specification to complete
Decision and owner	State the decision, the current policy, and who can authorize continuation, revision, or withdrawal. Identify the delivery and analytical owners.
Action and comparator	Define offer terms, service, channel, timing, duration, and ordinary activities preserved. Include no contact or no change when feasible.
Unit and eligibility	Name the customer, account, household, or cluster; eligibility date; exclusions; consent; prior exposure; and reentry rule.
Evidence and assignment	Record randomization unit, blocks, probabilities, and allocation. For observational analysis, document assignment, measured confounders, support, and the identifying assumptions.
Outcome and horizon	Define the primary outcome, assignment-time origin, follow-up window, reconciliation delay, and treatment of censoring, attrition, and missing links.
Economics	Define contribution, credit or discount rules, delivery and service costs, fixed costs, discounting, and which harms are monetized or constrained separately.
Capacity and permitted actions	Specify cash and staff limits, inventory, contact caps, competing campaigns, customer rights, and applicable fairness and legal restrictions. Define reservation and fallback behavior.
Precision and analysis	Set the useful effect or economic threshold, uncertainty target, sample calculation, multiplicity approach, stopping rule, and customer guard measures.
Estimation and policy	State the average effect and any actionable heterogeneity needed. Identify the simple baseline, model, thresholds, tie handling, and policy version to freeze.
Independent validation	Reserve outcomes unused in selection. Specify policy-value comparison, support checks, interval method, later-period evaluation, and any prospective policy trial.
Execution and outcome records	Assign responsibility for decision IDs, feature snapshots, action attempts, delivery, overrides, opt-outs, competing contacts, costs, and matured outcomes.
Operating decision and follow-up	State the evidence that would justify keeping, expanding, revising, or stopping the policy; the decision date; the accountable owner; and the sustained measurement design.

The final operating decision should identify the policy, population, horizon, estimated contribution, uncertainty, and customer limits it relies on. Where evidence is inconclusive, state which uncertainty could change the choice and whether further measurement is worth its cost. Where the current rule is adequate, retain it. Where a new policy is supported, preserve the assignment and outcome records needed to detect when its effects or operating conditions change.

Related reading

Customer analytics: prediction, experimentation and business decisions supplies the broader lifecycle account. The explainability resource addresses what model interpretations can establish. The general evaluation article develops common questions about measurement validity and operating outcomes; the economics article develops whole-workflow cost, risk, and investment comparisons. This resource supplies the intervention-specific design and analysis needed to use those connections.

References

$$1$$ Hernán, M. A., and Robins, J. M. (2015). Causal Inference. Part I. Author manuscript, August 27, 2015. Source copy

$$2$$ Ascarza, E. (2018). Retention Futility: Targeting High-Risk Customers Might Be Ineffective. Journal of Marketing Research, 55(1), 80-98. Source copy | DOI

$$3$$ Gupta, S., Lehmann, D. R., and Stuart, J. A. (2004). Valuing Customers. Journal of Marketing Research, 41(1), 7-18. Linked author manuscript revised February 2003. Source copy | DOI

$$4$$ Machmouchi, W., Gupta, S., Zhang, R., and Fabijan, A. (2020). Patterns of Trustworthy Experimentation: Pre-Experiment Stage. Microsoft Experimentation Platform, July 31. Source copy

$$5$$ Hudgens, M. G., and Halloran, M. E. (2008). Toward Causal Inference With Interference. Journal of the American Statistical Association, 103(482), 832-842. Source copy | DOI

$$6$$ Angrist, J. D., Imbens, G. W., and Rubin, D. B. (1996). Identification of Causal Effects Using Instrumental Variables. Journal of the American Statistical Association, 91(434), 444-455. Source copy | DOI

$$7$$ Robins, J. M., Hernán, M. A., and Brumback, B. (2000). Marginal Structural Models and Causal Inference in Epidemiology. Epidemiology, 11(5), 550-560. Source copy | DOI

$$8$$ Athey, S., and Imbens, G. W. (2016). Recursive Partitioning for Heterogeneous Causal Effects. Proceedings of the National Academy of Sciences, 113(27), 7353-7360. Author manuscript: arXiv:1504.01132v3, December 30, 2015. Source copy | DOI

$$9$$ Künzel, S. R., Sekhon, J. S., Bickel, P. J., and Yu, B. (2019). Metalearners for Estimating Heterogeneous Treatment Effects Using Machine Learning. Proceedings of the National Academy of Sciences, 116(10), 4156-4165. Author manuscript: arXiv:1706.03461v6, April 2019. Source copy | DOI

$$10$$ Nie, X., and Wager, S. (2021). Quasi-Oracle Estimation of Heterogeneous Treatment Effects. Biometrika, 108(2), 299-319. Published online September 24, 2020; author manuscript: arXiv:1712.04912v4, August 6, 2020. Source copy | DOI

$$11$$ Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., and Robins, J. (2018). Double/Debiased Machine Learning for Treatment and Structural Parameters. The Econometrics Journal, 21(1), C1-C68. Inspected revision: arXiv:1608.00060v7, November 3, 2024. Source copy | DOI

$$12$$ Wager, S., and Athey, S. (2018). Estimation and Inference of Heterogeneous Treatment Effects Using Random Forests. Journal of the American Statistical Association, 113(523), 1228-1242. Author manuscript: arXiv:1510.04342v4, July 2017. Source copy | DOI

$$13$$ Athey, S., and Wager, S. (2021). Policy Learning With Observational Data. Econometrica, 89(1), 133-161. Author manuscript: arXiv:1702.02896v6, September 4, 2020. Source copy | DOI

$$14$$ Dudík, M., Langford, J., and Li, L. (2011). Doubly Robust Policy Evaluation and Learning. Proceedings of the 28th International Conference on Machine Learning, 1097-1104. Author manuscript: arXiv:1103.4601v2, May 6, 2011. Source copy

$$15$$ Surry, P. D., and Radcliffe, N. J. (2011). Quality Measures for Uplift Models. Author-hosted working paper. Source copy

$$16$$ Yadlowsky, S., Fleming, S., Shah, N., Brunskill, E., and Wager, S. (2025). Evaluating Treatment Prioritization Rules via Rank-Weighted Average Treatment Effects. Journal of the American Statistical Association, 120(549), 38-51. Published online October 11, 2024. Source copy | DOI

$$17$$ NIST/SEMATECH (n.d.). Sample Sizes Required. e-Handbook of Statistical Methods, section 7.2.2.2. Accessed October 29, 2026. Source copy

$$18$$ Imbens, G. W., and Wooldridge, J. M. (2009). Recent Developments in the Econometrics of Program Evaluation. Journal of Economic Literature, 47(1), 5-86. Inspected working paper: IZA Discussion Paper 3640, August 2008. Source copy | DOI

$$19$$ Johari, R., Koomen, P., Pekelis, L., and Walsh, D. (2022). Always Valid Inference: Continuous Monitoring of A/B Tests. Operations Research, 70(3), 1806-1821. Published online August 10, 2021. Source copy | DOI