In this website paper
Key takeaways
- Begin with a decision. Specify whose behavior matters, what can be changed, when an action must occur and which outcome would justify its cost.
- Match the estimate to the question. Prediction supports forecasting and operational prioritization. Causal estimates assess what a particular intervention changes. Customer value connects future behavior to economic consequences.
- Define value precisely. State the customer unit, horizon, margin, service costs and discounting convention. Separate observed contribution from projected value and an existing relationship’s value from the incremental value of an action.
- Treat the policy as part of the system. Eligibility, customer preferences, capacity, competing actions and uncertainty determine which recommendations can be executed. A transparent rule or service repair may be the appropriate solution.
- Match evaluation to the decision and claim. Validate forecasts and assess their use; verify repairs against the service objective. Claims about intervention benefits need causal evidence and policy evaluation. Assess realized business outcomes and operating costs in each case.
Customer data becomes useful through specific commercial decisions
Customer analytics connects information about customers to decisions about acquisition, onboarding, service, renewal and growth. The commercial problem is to allocate attention and resources while improving the value customers receive and the contribution the business earns.
A useful implementation therefore needs more than a score. It needs an eligible population, information available at a defined decision time, feasible actions, an accountable owner and an outcome that can be measured after those actions. This makes it possible to decide whether a forecast, an experiment, a targeting model or a straightforward operating change would help.
Consider Harbor, a wholly fictional subscription scheduling service for small businesses. It acquires accounts through several channels, helps new customers configure their calendars, sells quarterly subscriptions and offers additional locations and seats. Every Harbor example below is independently constructed and hypothetical. Its lifecycle illustrates how these decisions connect without implying that every business follows the same funnel.
Acquisition, activation, retention and expansion need different targets
Lifecycle decision map. Each stage has a decision and a set of actions the business can actually take.
| Stage | Decision and available actions |
|---|---|
| Acquisition | Allocate spending and sales effort across channels or prospects; change qualification, messaging or whether to contact. |
| Activation | Help a new customer reach a useful outcome; change setup, provide guidance or route a service request. |
| Retention | Address an approaching renewal or service problem; repair, assist, change the offer or leave ordinary service unchanged. |
| Expansion | Match additional products or capacity to customer needs; recommend, demonstrate, adjust the plan or make no offer. |
Acquisition: connect incoming customers to later contribution
Before acquisition, Harbor may know the campaign, referral source, requested business use and information a prospect supplied. It cannot use subsequent product behavior to decide whom to acquire. Once a trial begins, the available information changes and supports a different decision.
A lead score can rank prospects by their probability of becoming customers under the historical sales process. Allocating additional sales calls requires a further comparison: which prospects benefit from that contact, what the resulting business is worth and how much sales capacity it uses. A high conversion probability alone does not settle that allocation.
Harbor should follow acquisition cohorts through activation, paid conversion, service demand and contribution. A channel with inexpensive signups can be unattractive if those accounts require extensive support and rarely become paying customers. A channel with expensive signups can be viable when subsequent contribution supports the cost. These are accounting and forecasting comparisons; increasing a channel’s budget still requires evidence about the additional customers that spending would generate.
A practical first step is to reconcile campaign spending with account creation and subsequent outcomes, identifying unattributed accounts explicitly. Observed channel differences can guide investigation and experiment design without being interpreted as the causal return to reallocating money.
Activation: identify the first useful outcome and remove obstacles
For Harbor, activation could mean publishing a working calendar and completing a first genuine booking. Merely logging in is easier to count but less closely connected to the product’s purpose. Useful information includes elapsed time since signup, completed setup steps, integration errors and explicit requests for help.
Suppose a new account cannot publish its calendar because of an integration error. A successful ordinary operating path is straightforward: detect the error, open a service task, repair the configuration, confirm that publishing works and close the task. This accomplishes a verifiable service objective. Whether the repair also increases paid conversion or renewal is a separate outcome to measure.
For customers without an explicit fault, Harbor could test a revised setup sequence or an offer of onboarding help. The initial outcome might be successful activation, followed by paid conversion and contribution. An association between early activity and later retention does not establish that inducing more activity will improve retention.
Retention: distinguish the event, its cause and the available response
Harbor can define nonrenewal at a contractual renewal date, with an explicit payment grace period. Predicting it may use tenure, usage, billing events and unresolved service issues observed before outreach begins. Voluntary cancellation, failed payment and business closure should remain distinguishable because the available responses differ.
Business models change the target. In a contractual service, cancellation or nonrenewal can be observed. In noncontractual retail, silence may represent a long gap between purchases. Fader and Hardie’s contractual valuation research makes this distinction explicit. A retailer may need a purchase-timing forecast rather than a universal binary churn label.
Expansion: measure additional contribution across the relationship
Harbor might identify accounts approaching a location or seat limit, then offer a relevant demonstration or plan comparison. Information available at that point includes current capacity, historical usage, expressed needs and unresolved problems.
Evaluate the resulting contribution across the account. An upgrade can displace another purchase, increase support costs or accelerate spending that would otherwise happen later. Customer suitability and communication preferences constrain the offer. Successful expansion can also originate from a customer’s request, with ordinary software routing the request to the right person.
Acquisition determines Harbor’s incoming customer mix. Activation and service shape the relationships on which later retention and expansion decisions depend, while those decisions compete for staff time and budget. Assess changes across stages using compatible account units, horizons and contribution definitions, so a local gain can be weighed against its consequences elsewhere.
Measurement, prediction and customer value serve different purposes
Descriptive measurement establishes what happened: which cohorts converted, how long activation took, how much service work was required and which accounts renewed. Its immediate contribution may be discovering a broken process or an inconsistent outcome definition.
Prediction estimates future outcomes from available information under specified operating conditions. Harbor can forecast renewals for revenue planning and incoming service demand for staffing. Where a response is already established, a prediction can help schedule it or prioritize a constrained queue. Prediction does not need to estimate a new intervention effect every time it is used. A model trained under an existing outreach policy reflects that history; it does not automatically recover the risk customers would face without outreach.
The distinction matters when the business changes its behavior toward customers:
| Estimate | Question answered |
|---|---|
| Renewal or purchase probability | What outcome should be expected under the conditions represented by the prediction? |
| Incremental intervention effect | How would the expected outcome change if this eligible customer profile received a specified action rather than the comparator? |
A high churn probability does not establish that a discount or service call will prevent departure. Conversely, high predicted customer value can describe someone who will remain without additional attention. Eva Ascarza’s “Retention Futility” used two field experiments, in telecommunications and professional membership, to show that customers ranked highest by churn risk were not necessarily the most responsive to the studied interventions. Its targeting comparisons used experimental data; they are not evidence of a universal commercial uplift.
Customer value needs an economic definition and an observation horizon
Customer lifetime value is expected discounted future contribution from a relationship. Gupta, Lehmann and Stuart’s “Valuing Customers” connects future margins, retention and acquisition economics. An operational definition should specify the currency, account or household unit, valuation date and included costs. Revenue must be reduced by relevant delivery, payment, returns and service costs. Historical acquisition expenditure is sunk for an existing-customer intervention, although it remains relevant when evaluating acquisition economics.
For a fixed twelve-month decision horizon, call the estimate “expected twelve-month contribution.” If reporting lifetime value, disclose how the unobserved tail is extrapolated and how later cash flows are discounted. Match the discount rate to the time period. Keep program costs separate when they are not already included in contribution, avoiding double counting.
Model structure should reflect the relationship. A survival or renewal model may suit contractual subscriptions. The BG/NBD model of Fader, Hardie and Lee provides a foundation for noncontractual repeat purchasing, using transaction history and observation duration to model purchasing and latent departure. Purchase counts still require a monetary component to support value estimates. The paper also explicitly assumes broadly stable future marketing activity when interpreting its forecasts.
Incomplete follow-up matters. An account observed for three months has not supplied a completed twelve-month outcome. Pfeifer and Bang explain why averaging value observed so far across ongoing and completed relationships biases mean lifetime-value estimates. For a fixed horizon, use cohorts with complete observation through that horizon, or methods that explicitly handle censoring and its assumptions. A customer need not have left for a fixed-horizon outcome to be complete.
Uncertainty also comes from future behavior, estimated parameters and structural changes in price, service or customer mix. Show how plausible alternatives affect the decision. Fader and Hardie’s contractual analysis demonstrates that changing cohort retention can arise through customer composition alone. A rising retention curve is therefore insufficient evidence that a new program worked.
Experiments establish what an action changes
For an onboarding test, Harbor can define eligible new accounts before assignment and randomly allocate them to ordinary onboarding or an additional offer of help. Comparing outcomes by assigned group estimates the effect of deploying that offer process, including the fact that some customers will ignore it.
Keep those customers in the primary comparison. Restricting analysis to people who open a message or accept help selects on behavior after assignment. Gordon and colleagues’ comparison of advertising measurement approaches distinguishes randomized assignment from actual exposure. In their Facebook studies, observational estimates often failed to recover experimental effects despite extensive customer covariates. The finding concerns those designs and data, but illustrates why adjustment cannot be assumed to remove selection.
Define the comparator carefully. “No additional outreach” can preserve ordinary billing, support and product access. It does not mean customers receive no marketing elsewhere or cannot be affected by other participants. Shared business accounts, referral activity and competition for support capacity can create spillovers. Choose a randomization unit and design that address the mechanism; an individual holdout may not measure the value of rolling out a program to everyone.
Before launch, specify the primary outcome, observation window, useful effect size, sample requirement, customer-outcome limits and analysis plan. For Harbor, those limits could cover complaints, unwanted contact or a decline in customers’ ability to complete a booking. Check assignment integrity and missing outcomes. Microsoft’s pre-experiment engineering guidance describes logging whether control requests would have triggered a feature, checking sample-ratio mismatches and examining shared infrastructure for contamination. Such logging identifies comparable opportunities; it does not observe customers’ counterfactual outcomes.
Observational causal analysis depends on treatment assignment
Historical records can support causal estimation when the assignment mechanism is defensible. For a comparison based on observed characteristics, the central requirements are that relevant common causes of treatment and outcome are measured, comparable profiles have a real chance of receiving either action, and each action has a sufficiently consistent meaning. These are identification conditions, not achievements of a particular model.
If account managers reserve discounts for customers whose private conversations reveal imminent departure, ordinary CRM fields may omit a decisive confounder. If every high-value account receives a discount, the data provide no untreated counterpart for that group. Restricting the target population, gathering missing assignment information or introducing randomized variation can be more useful than changing algorithms. Regression discontinuities or other quasi-experimental designs can sometimes help, but require their own assignment-specific assumptions.
Heterogeneous effects describe averages for customer profiles
A treatment-effect model estimates how an intervention’s average effect varies with information available before treatment. Athey and Imbens’ research on heterogeneous causal effects develops methods that separate subgroup discovery from effect estimation. This helps control the optimism created by searching for attractive subgroups.
The output is a conditional average treatment effect. Ordinary campaign data cannot reveal both outcomes for the same person. Labels such as “persuadable” or “would buy anyway” refer to conceptual potential-outcome categories, not individually observable customer types. A conditional difference in purchase probabilities is causal only under an appropriate design and identification assumptions.
A decision policy combines effects, economics and constraints
A policy maps available information to a feasible action. For Harbor, candidates might include an onboarding guide, a service review, a plan discussion or no additional contact. A discount is another option when appropriate, and no discount remains a meaningful choice.
The economic comparison is the expected contribution under each action relative to its comparator, less incremental delivery cost. When the action changes usage, margins or service workload, estimate those consequences as well. Multiplying a churn score by customer value leaves the intervention’s effectiveness unspecified.
Choose within eligibility, preferences, contact limits, staff availability and budget. Shared constraints mean the individually most attractive action cannot always be selected for every account. Capacity is an upper bound, not a requirement to contact customers whose expected incremental value is negative.
Athey and Wager’s “Policy Learning With Observational Data” formalizes learning assignment rules within a constrained policy class. Its guarantees require identified causal effects and additional statistical conditions; its benchmark is the best policy in the specified class.
A useful operating illustration appears in Uber’s May 2026 account of its Tarot targeting platform. The architecture separates eligibility, estimates based on historical experiments, budget pacing, constrained assignment and downstream execution. Configurable schedules initiate targeting runs. This illustrates system responsibilities; the account does not establish a transferable profit gain.
Where estimated differences are small relative to uncertainty, prefer a simpler policy or a limited experiment that could change the decision. A service-wide defect may warrant a product fix. A short, transparent eligibility rule may capture the actionable variation sufficiently well.
Hypothetical calculation: three policies for the same outreach capacity
Harbor has 400 eligible accounts approaching quarterly renewal, divided into four observable segments of 100. It can assign at most 100 standardized proactive service reviews, costing $10 per assigned account, regardless of uptake or renewal. No discount is offered.
A renewed account contributes $300 during the next quarter, after ordinary variable service costs but before this outreach cost. Nonrenewal contributes zero. Ordinary billing and support continue in every policy. All figures are expected values, with no spillovers, capacity displacement or effects beyond the quarter.
Synthetic inputs. These segment-level causal probabilities are assumed known for illustration, not estimated from a dataset.
| Segment, 100 accounts each | Renewal without added review | Renewal when review is offered |
|---|---|---|
| A | 40% | 42% |
| B | 65% | 80% |
| C | 85% | 87% |
| D | 95% | 93% |
The transparent baseline is a uniform lottery selecting 100 accounts, averaging 25 from each segment. Risk-based prioritization selects all 100 A accounts. The intervention policy selects all 100 B accounts, which have the greatest positive incremental contribution.
Expected net additional contribution per assigned account equals $300 multiplied by the change in renewal probability, minus $10. This is -$4 for A, $35 for B, -$4 for C and -$16 for D.
Synthetic policy comparison. Changes use all 400 eligible accounts as the common population and no added outreach as the comparator.
| Policy | Additional renewals | Net additional contribution |
|---|---|---|
| Uniform lottery, 100 offers | 4.25 | $275 |
| Highest churn risk, 100 A offers | 2 | -$400 |
| Intervention policy, 100 B offers | 15 | $3,500 |
For B, 100 offers create 15 expected additional renewals: 15 × $300 - 100 × $10 = $3,500. Risk targeting increases expected renewals, but their contribution fails to cover delivery costs. Fractional renewals are expectations across repeated equivalent cohorts.
These results follow from the invented inputs. A real implementation must estimate the effects and validate the policy on separate evidence. Development, integration and maintenance costs are excluded here and belong in the assessment of organizational benefit.
Implementation requires a reliable path from customer events to action
For Harbor, the operating unit is the business account, even when several users interact with the product. The implementation must connect account identity, users, subscriptions, payments and service cases without silently merging unrelated customers. Event time and the time information became available both matter.
Construct each historical decision as it could have been made then. A cancellation reason entered later, a backfilled account status or a service resolution recorded after outreach is unavailable to the original decision. Kaufman, Rosset and Perlich’s research on leakage grounds feature legitimacy in the prediction task and its timing. Uber’s feature-consistency engineering account describes logging the feature values actually consumed during inference to improve subsequent training consistency.
The central operational record should connect:
- Eligibility and decision: account, decision time, available features, permitted actions, score and policy versions, assigned action, and assignment probability when randomized.
- Delivery and exposure: CRM or service task, responsible owner, attempts, actual delivery, acceptance, overrides, failures and competing contacts.
- Outcomes and costs: renewal or conversion window, outcome maturity, payments, refunds, service effort and incremental action expense.
Assignment, delivery and uptake should remain separate. A queued review can fail to reach a customer; an accepted review may remain unfinished. Systems should prevent duplicate actions, apply customer preferences at execution time and preserve the reason for manual overrides. Required access, permitted data uses, retention periods and deletion handling should be agreed with the people responsible for customer data.
A successful routine is a nightly eligible-account extract, a versioned decision rule, a deduplicated service queue, completion records and a later join to matured renewal outcomes. The service owner confirms delivery; the commercial owner decides whether the measured result justifies continuation. Investigate missing identities and treatment records. Preserve known randomized assignments even when delivery or uptake is uncertain. When outcomes cannot be linked, assess missingness by group and limit conclusions accordingly.
Batch processing fits work assigned for the next business day. A response to an in-session setup error may require immediate execution. Choose latency from the window in which an action remains useful, balancing freshness, reliability and operating cost. Neither execution cadence establishes the quality of the decision.
Evaluation must follow the decision into production
The required evidence depends on the decision and the claim. Harbor’s service-demand forecast needs forecast validation and an assessment of how it supports staffing. Its calendar repair needs verification that customers can publish again. A claim that an intervention improves conversion, renewal or contribution additionally needs causal evidence and evaluation of the policy used to assign it.
When using predictions, validate them for the intended task. Reconstruct earlier decision dates, train on outcomes available then and test on later periods with sufficient follow-up. Keep tuning and calibration separate from the final evaluation. Evaluate new-account generalization separately when that is the intended deployment task. Historical rows from the same customer are not interchangeable independent observations.
Use ranking measures when assessing a queue, and probability checks when estimates enter expected-value calculations. Niculescu-Mizil and Caruana demonstrate why good classification or ROC performance does not guarantee calibrated probabilities. Check predicted probabilities against observed frequencies in relevant ranges and cohorts, using enough data to express uncertainty. Evaluate contribution forecasts against matured economic outcomes as well.
When assigning interventions, evaluate the policy. Compare candidate rules at a common eligible population, capacity, horizon and cost definition. Offline policy-value estimation needs data that support the proposed actions. Dudik, Langford and Li combine outcome models and action probabilities in doubly robust evaluation; missing action support and unrecorded confounding remain consequential. Qini measures and the area under the uplift curve (AUUC) summarize how well a model ranks customers by intervention effect. The underlying curves compare estimated incremental outcomes as successively larger shares of the ranked population are selected. They require a valid evaluation design and do not establish ROI.
Assess the operating result in every implementation. Check whether the forecast was used or the service objective was achieved, then assess the realized business outcomes and operating costs. To claim incremental benefits from an intervention policy, test the frozen policy against the existing operating policy or a justified holdout. Report incremental contribution or profit, uncertainty and customer outcomes such as complaints and opt-outs. Reconcile delivery rates, displaced work, additional staffing and system costs to assess realized organizational benefit.
Early signals can support monitoring while renewal outcomes mature. A Netflix study of surrogate outcomes examines using shorter observations to anticipate later experimental decisions. Such use requires evidence that the proxy captures the intervention’s relevant effects. Predictive association alone does not turn engagement into measured long-term value.
Monitor identity coverage, feature freshness, action execution, customer mix and outcome rates, with calibration checks where probabilities are used. For interventions, preserve a measurement design that can detect changed treatment effects. A new price, acquisition channel or outreach policy can change what historical data predict. Diagnose whether deterioration comes from data, delivery, population change or the intervention before deciding to retrain.
Language models can help process customer information
Potential language-model components include extracting an issue from a service conversation, summarizing account history or drafting an approved message. For Harbor, a summary could help a service specialist prepare a review. Evaluate its factual accuracy, omitted details and usefulness against the actual task, and preserve the source text and model version when extracted fields enter analytics.
Keep probabilities and economic decisions tied to validated data and methods. Specialized predictive models can estimate renewal or value; experiments can identify intervention effects; ordinary code can enforce eligibility, permissions, budgets and delivery rules. Customer-support assistance is an adjacent operational capability. Its effect on retention requires evidence linking the changed support process to subsequent customer outcomes.
Begin with a complete, bounded operating decision
An effective first delivery could be a reliable activation measure, a repaired onboarding path or a renewal queue whose actions and outcomes can be traced. Choose a decision with a named owner, feasible alternatives and an outcome the organization can observe. Establish a baseline, identify the uncertainty that could change the decision and collect the evidence needed to resolve it. Expand when the implemented result justifies the additional scope.