In this website paper
How to read model structure, feature attributions, effect plots and counterfactual examples, and connect them to the decisions a business actually makes.
Key takeaways
- Identify the question. Explaining a predictive score, investigating an error, allocating service and identifying an effective intervention require different evidence.
- Preserve the model’s units. Contributions in an additive logistic model sum in log-odds. Probability changes depend on the starting score and cannot be read directly from those contributions.
- Define the reference. Feature attributions depend on the background population, the treatment of missing features and the output being explained. Computational exactness applies to that definition.
- Distinguish input changes from actions. A counterfactual profile can cross a model threshold while remaining infeasible, outside the customer’s control or unsupported as a causal intervention.
- Match evidence to the claim. Local agreement, stability, source support and execution records each establish something specific. None alone establishes that a model or business policy is sound.
A model assigns a high score to a customer-service case. An analyst wants to understand its calculation; a manager wants to know why the case received priority; an adviser wants to know what might prevent escalation. A feature-importance chart can appear to answer all three. Usually, it addresses only part of the first.
The practical task is to identify what an explanation represents and whether that representation answers the question at hand. This article develops one hypothetical service-prioritization example, then uses it to interpret common explanation methods. The aim is to make model outputs investigable without converting predictive associations into causal claims or attributing a business decision entirely to a score.
A predictive score and a service allocation have different explanations
Hypothetical setting. At 09:00, a service operation scores unresolved cases, with one case per customer. The intended prediction target is whether the customer will request a supervisor within the next seven days under the historical service process. All features describe information available at scoring time. The model and every numerical value below are constructed for illustration; there is no fitted dataset or measured predictive performance.
The service policy considers open cases whose customers accept callbacks. Cases scoring at least 60% are ranked by descending score, with longer unresolved time and then ticket identifier breaking ties. Up to 20 receive priority callback tasks. Our example case has a 69.0% score and a stipulated rank of 18 among qualifying cases, so the policy selects it. Task creation records that a callback task was queued; a later completion event would establish that a call occurred.
Changing capacity to 15, before allocation and with the same candidate pool, would leave this case unselected at the same score. An explanation of its high score cannot account for that difference. The reason for selection includes eligibility, the cutoff, competing cases, capacity and tie handling. Investigating an actual service action also requires the policy and data in force at the time, plus any override and evidence of execution.
The prediction target needs equal care. Risk estimated under a historical service process already reflects the service customers received. It does not identify which customer would benefit most from an additional callback. That requires evidence about the consequences of the callback. Explaining the score can expose an association with earlier service activity, but cannot supply the missing intervention comparison.
An additive model is interpretable only on its defined scale
Define a small logistic model. Let h be hours since the unresolved case opened, and n the number of subsequent contacts about that issue during the preceding seven days, excluding the initial contact. Both are nonnegative; n is an integer. Transform them into w = min(h, 72)/24 and c = min(n, 3). Let o indicate an active outage affecting the case and r indicate that the customer has acknowledged a documented resolution plan. Both indicators are either zero or one.
z = -1.50 + 0.80w + 0.50c
+ 0.70o - 0.60r
p = 1 / (1 + exp(-z))
The additive score z is log-odds; p is the corresponding probability-scale output. Consider h = 36 hours, n = 2, o = 1 and r = 1.
The declared model adds contributions in log-odds
| Component | Input and calculation | Log-odds |
|---|---|---|
| Reference intercept | All four transformed inputs zero | -1.50 |
| Unresolved time | 0.80 × (36/24) | +1.20 |
| Repeat contacts | 0.50 × 2 | +1.00 |
| Active outage | 0.70 × 1 | +0.70 |
| Acknowledged plan | -0.60 × 1 | -0.60 |
| Case score | Intercept plus all four terms | 0.80 |
Reference output: 18.2%. Case output: 69.0%.
The baseline is the specified all-zero profile: a newly opened case with no repeat contacts, no active outage and no acknowledged plan. Its 18.2% output is neither a population average nor an observed escalation rate. Centering inputs elsewhere would move values between the intercept and components while preserving predictions.
The coefficients describe transformed units. Within the first 72 hours, each additional hour contributes 0.80/24 log-odds. Time beyond 72 hours adds nothing further. The probability change from adding a contribution d is σ(z + d) - σ(z), where σ is the logistic function. It depends on the starting score. Applying σ separately to the terms would not produce additive probability contributions.
Dependence between inputs does not invalidate this calculation. Linear models do not require statistically independent predictors. Exact collinearity prevents unique unpenalized coefficient identification; near collinearity can make estimated coefficients unstable. Holding one input fixed while changing another may nevertheless describe a combination scarcely represented in the data. The statsmodels discussion of rank deficiency separates these numerical issues from merely obtaining a fitted result. Statistical intervals additionally require justified sampling and variance assumptions; the declared coefficients here have no sampling intervals.
Intrinsic structure exposes a computation that still needs interpretation
Intrinsic versus post-hoc describes how an explanation relates to the model. Local versus global describes its scope. Our equation is an intrinsic global specification; Figure 1 is an intrinsic local decomposition. A surrogate can approximate a complex model locally, while a collection of local attributions can summarize its behavior across a population. These dimensions do not form a quality ranking.
A linear or logistic model is easier to inspect when its inputs have meaningful units and few transformations. Its full specification still includes encoding, missing-value handling, feature construction and any calibration after scoring. A sparse equation over opaque embeddings may reveal little to a service manager. A short decision-tree path can reproduce one case’s routing while leaving behavior on other branches unexplained.
Generalized additive models replace straight-line terms with learned component functions. For a binary outcome, those functions can add on a log-odds scale before the inverse link produces a probability. Models with pairwise interactions also include functions of two inputs. In our setting, such a term could let unresolved time matter differently during an outage.
Explainable boosting machines learn nonlinear component functions through restricted boosting and can include selected interactions. Their components can be plotted and looked up for an individual case. An interaction contribution belongs to the pair unless a further allocation rule divides it. Showing only the largest terms, omitting interactions or hiding a downstream transformation makes the displayed explanation incomplete. Many terms and intricate curves also impose a substantial reading burden.
An interpretable model is attractive when its representation matches domain questions and its predictive performance is adequate for the intended decision. Compare plausible alternatives under a common protocol; do not assume a universal accuracy penalty for transparency. Rudin’s 2019 argument challenges that assumed tradeoff. It does not establish that every transparent model will match every complex alternative.
A more complex model can be justified when its improvement matters for the service allocation and persists in relevant evaluation. That choice increases the work needed to investigate behavior and communicate limits. Conversely, a perfectly legible model can use an unsuitable target, learn an association produced by earlier service decisions or support a poor allocation policy. Structural visibility helps uncover those problems; it does not settle them.
Feature attribution allocates a difference relative to a reference
For a complex model, a common question is why this output differs from a reference output. SHAP assigns contributions using a Shapley-value construction. For each feature, it averages the change from adding that feature across different subsets of already available features, using specified weights. Baseline plus contributions reconstructs the explained output when the computation satisfies the relevant additivity condition.
The decisive choice is the value assigned to a subset. A model expecting every feature cannot ordinarily accept “the outage feature is unavailable” as a complete input. An explanation therefore defines how to average over, replace or otherwise represent the missing information. This explanatory absence is separate from the model’s treatment of genuinely missing operational data.
A conditional definition averages the model output given the observed subset under a reference joint distribution. It asks what learning these feature values tells us about the output. Estimating the missing features requires a conditional distribution; Aas, Jullum and Løland develop approximations for dependent inputs. The distribution estimate can be wrong even when the subsequent Shapley arithmetic is exact.
A marginal replacement definition fixes the observed subset and takes the other inputs from background cases. This preserves dependence within the replaced subset but breaks dependence between replaced and fixed inputs. It probes the model on those recombinations. Conditional attribution can credit an input unused by the model because it predicts another input the model uses. Marginal replacement can evaluate implausible combinations. Janzing, Minorics and Blöbaum explain why these definitions answer different questions, and why manipulating computational inputs does not identify the consequences of changing the corresponding real-world attributes.
The background also changes the comparison. Comparing this case with all open cases and comparing it with cases already affected by outages can produce different feature contributions without changing the prediction. Feature grouping matters too: explaining individual encoded columns and treating an original variable’s columns as a group define different allocation problems.
Check the estimator and the output scale
Kernel SHAP estimates contributions through specially weighted regression over feature subsets. Naive exhaustive enumeration involves 2M subsets for M features; practical estimation can sample subsets. The usual KernelExplainer background replacement is not a general estimator of arbitrary conditional distributions. Extensions that estimate those distributions can incorporate dependence. Sampling budget, regularization and the replacement mechanism affect the estimate.
Tree SHAP uses tree structure to compute attributions efficiently. TreeExplainer’s documented modes include background-based interventional calculations and tree-path-dependent calculations using recorded sample counts along paths. The modes define different ways of integrating out absent inputs. Tree-path sample counts are not a general estimate of the observational conditional distribution.
Exactness depends on the explained function, algorithm and dependence mode. The cited paper gives exact calculations for trees and sums of trees, but uses a compositional approximation for nonlinear output transformations (Methods 10.2). A decomposition that adds up to a transformed output does not by itself establish exact Shapley values for that function. Check the guarantee for the particular implementation and output; computational exactness does not establish a causal effect on customers.
Finally, name the output: raw margin (the score before the output transformation), probability or another quantity. A probability decomposition has contributions in probability units; a log-odds decomposition does not. Explaining a raw log-odds function also differs from taking the logit of an expected probability. Averaging and nonlinear transformation generally do not commute. A useful attribution display identifies the model, output scale, reference population, feature grouping, dependence treatment and estimator. Its numbers describe those feature contributions; they do not give an unconditional reason for the customer’s behavior.
Local approximations and rules describe a selected neighborhood
LIME generates perturbed inputs, obtains the original model’s outputs and fits a small surrogate using proximity weights and a complexity penalty. Its coefficients describe the chosen surrogate around the case. Change the representation, distance measure, sampling distribution or sparsity constraint and the explanation can change. Inspect its agreement with the original model on relevant nearby inputs, including where the service threshold is crossed. A surrogate’s simplicity does not establish local fidelity.
Anchors instead seeks an if-then rule under which a classifier usually retains its prediction. It reports precision and coverage under a specified perturbation distribution, with sampling-based confidence that precision exceeds a chosen threshold. Here precision measures agreement with the model, not correctness against observed customer outcomes. A narrow rule can have high precision and little coverage. An anchor does not establish that each condition is necessary, or explain what happens outside its covered region.
Both methods can help an analyst compress behavior into something inspectable. The relevant question is whether the sampled neighborhood corresponds to plausible service cases and the comparison the analyst wants to make. A convenient perturbation distribution is an experimental choice, not a definition of which customer changes are possible.
Effect plots show how a model responds across input values
A partial dependence plot, or PDP, replaces one feature with each value on a grid, predicts for reference cases and averages the results. It answers a population-average question about that modified-input calculation. Statistical independence is not required to define the calculation. Dependence can make its evaluations poorly supported: assigning very short unresolved time to cases with many repeat contacts may combine inputs seldom seen together.
Individual conditional expectation, or ICE, retains a separate curve for each reference case. Its average yields the corresponding PDP. Individual curves can expose differing responses hidden by the average, but still involve holding other inputs fixed. Show where actual inputs occur, and inspect joint support before treating the rest of a curve as representative.
Accumulated local effects, or ALE, average small prediction differences within intervals occupied by the relevant cases, accumulate those differences and center the result. This reduces the broad recombination problem of PDP. Binning, sparse intervals and interactions still affect the estimate. Accumulation across a range does not validate a feasible journey through that range with all other inputs unchanged. Categorical inputs also need a defensible comparison or ordering.
For our declared additive log-odds score, the unresolved-time component is already known: 0.80 log-odds per day, capped at three days. Its intrinsic shape is the direct answer to a question about that term. On the probability scale, curves for different cases can bend differently because of the logistic transformation, even though the score has no interaction terms. Statements about interactions must therefore name the plotted scale.
Use PDP to examine an average response when its reference distribution and recombinations serve the question; use ICE to investigate variation across cases; use ALE when local differences better respect the available support. Include data density, units and relevant interactions. None of these model-response plots alone estimates how customers would react to an intervention.
Crossing a model threshold does not establish feasible recourse
For this customer, changing the outage indicator from one to zero while holding every other input fixed removes 0.70 log-odds. The score becomes 0.10, and the output becomes 52.5%, below the 60% priority threshold. This is a valid input counterfactual for the declared function.
The score and callback capacity can change selection separately
Model output (%) · Vertical mark = 60% threshold
Current case69.0%
Rank 18 · Capacity 20 · Selected for a callback task
Outage input set to zero52.5%
Other inputs fixed · Below threshold · Not selected
Capacity reduced before allocation69.0%
Same pool and rank 18 · Capacity 15 · Not selected
0%100%
The customer cannot clear an outage by changing a form field. Actually restoring service is an organizational intervention that can change subsequent contacts, perceptions and service activity. Its effect on supervisor requests remains unknown. The 16.5 percentage-point reduction is a difference between two model outputs, not an estimate of the intervention’s effect on this person.
Define the desired result before searching: access to a callback, a lower escalation score, or fewer actual escalations. Each requires a different target and supporting evidence.
Counterfactual search typically balances reaching a target output with distance, cost or sparsity. DiCE adds diversity so a user can inspect alternative profiles. The original paper explicitly discusses infeasible combinations produced by independently varying causally related features. Multiple alternatives do not by themselves make any alternative actionable.
Work on algorithmic recourse instead evaluates permissible actions and their consequences under an assumed causal model. Applying that idea here requires specifying who can act, immutable history, dependencies, time, costs and constraints. Changing a mistaken outage record is a data correction; restoring service is an operational action. Acknowledging a plan is not equivalent to receiving an effective solution.
Recourse also depends on the policy. A fixed score threshold is easier to describe than a rank cutoff that moves with other cases and staffing. An attribution plot cannot establish the unobserved outcome for the same customer under a different intervention. Explain which comparison is calculable, which actions are feasible and which consequences remain uncertain.
Explanation tests are informative only when their interventions are defined
An explanation should be tested against its claim. For native terms, check that the complete calculation reproduces the implemented output, including transformations and calibration. For a surrogate, measure agreement on a separately sampled, relevant neighborhood. For an attribution, examine estimator variability and sensitivity to justified background choices. Reconstructing a score is a useful numerical check; it cannot validate the choice of reference or the model’s business meaning.
Deletion tests remove or replace supposedly important inputs; sufficiency tests retain a selected subset and replace the rest. Specify whether the measured response is a probability, a class or an action. A class can remain unchanged despite a substantial score movement. Zeroing a field, drawing from a background population and sampling conditionally create different interventions on the input.
A large change can reflect useful information removed, unrealistic masked cases, or both. Redundant inputs can preserve an output after an informative variable is removed. Hooker and colleagues’ 2019 work examined this distribution-shift problem in image models and introduced a retraining-based test. Retraining changes the fitted model being evaluated, so it answers a different question from deleting inputs at inference time.
Slack and colleagues’ 2020 study constructed classifiers that behaved differently on ordinary cases and explanation perturbations, misleading the tested LIME and Kernel SHAP procedures. This demonstrates a vulnerability of those procedures under the adversarial construction. It is not a measured failure rate for ordinary explanations or all SHAP algorithms.
Stability is similarly conditional. Repeat an explanation with changed sampling seeds, plausible nearby cases or a refitted model, and identify which source of variation is being examined. A constant explanation would be perfectly stable and useless. Desirable stability means that irrelevant changes do not alter the explanation while material changes in model behavior remain visible.
For communication, test whether an adviser can distinguish the probability score from priority status, or an analyst can recognize an impossible counterfactual. These tasks assess comprehension more directly than whether a chart appears persuasive.
Rationales, citations and traces provide different kinds of evidence
A language model might turn the service records into a fluent explanation of the predictive score and callback selection. Check that description against the actual model inputs, complete score decomposition, policy rules and execution records. It should distinguish the 69.0% output from selection under the 20-task capacity, and a queued task from a completed call.
Whether the language model faithfully reports the influences on its own answer generation is a different question. Fluency and factual correctness do not establish that internal account. In Turpin and colleagues’ 2023 experiments, prompt biases affected answers from the tested GPT-3.5 and Claude 1.0 models without being acknowledged. Chen and colleagues’ 2025 experiments found related omissions in hint-influenced answers from Claude 3.7 Sonnet and DeepSeek R1. These are dated, task-specific findings.
Editing a displayed rationale after generation does not intervene on the original computation. Editing reasoning text before generating the final answer tests dependence on the modified sequence. An unchanged answer can reflect recomputation, correction of the inserted error or another sufficient route. Lanham and colleagues study such interventions and their limitations. A useful test compares controlled modifications and repeated outcomes; one unchanged answer does not prove fabrication.
A citation identifies material against which a claim can be checked. Separate answer correctness, support in the cited passage and causal reliance on that passage. Research on retrieval-augmented attributions makes this distinction explicit. Removing one cited passage from the model’s context can leave redundant evidence, while regeneration can vary even without the edit. The result therefore needs a controlled interpretation.
An execution trace can show recorded calls, arguments, responses and observed events. Trace instrumentation defines which operations are captured. Those records can connect a prediction to policy execution, but do not expose every internal computation or guarantee that unrecorded steps were absent. Exact replay additionally depends on retained versions, inputs, external state and nondeterminism; even fixed seeds do not guarantee reproducibility across environments, as PyTorch documents. Captured requests and outputs can contain sensitive data, requiring deliberate collection, access and retention choices.
Choose the explanation by the decision it must support
| Question | Method and output | Principal limit |
|---|---|---|
| How was this score calculated? | Intrinsic structure: terms, transforms and link | Computational visibility does not establish validity or causal meaning. |
| How is a difference from a reference allocated? | SHAP: baseline and feature contributions | Depends on the output, background and missing-feature definition. |
| What approximates nearby behavior? | LIME: a weighted local surrogate | Fidelity applies to the chosen neighborhood and representation. |
| Under what rule does the label persist? | Anchors: rule, precision and coverage | Agreement under a sampling distribution is not outcome accuracy. |
| How does the model respond across values? | PDP, ICE or ALE: response curves | Support, averaging, interactions and output scale constrain interpretation. |
| What input profile changes the result? | Counterfactual search: alternative profiles | Feasibility and intervention effects require additional evidence. |
| Why did the organization select this case for action? | Policy and selection evidence: eligibility, threshold, ranking, capacity, overrides and execution records | Selection explains priority; execution records distinguish a queued task from a completed action. Neither establishes benefit. |
| Which feasible intervention would improve actual outcomes? | Intervention evidence: feasible actions and comparisons of their consequences | A changed input profile or predictive association alone does not identify an intervention effect. |
Comparison of explanation questions, outputs and limits
For the service reviewer, a useful explanation states the scoring time, input facts, probability and relevant contributions, then separately identifies the allocation rule and queued action. A model debugger needs the full computation and comparisons that can reveal errors. An intervention designer needs feasible actions and evidence about their consequences. Selecting the explanation begins by deciding which of those jobs it must do.