First, the key terms.
- Transparency
- Visibility into relevant design, data, behavior or process under a specified meaning.
- Post-hoc explanation
- An explanation produced after a model’s output, potentially by a separate method.
- Fidelity
- How accurately an explanation reflects the behavior it claims to describe.
The idea, explained
Begin by specifying who needs to understand what and for which decision. A developer debugging a model, an auditor examining group harms and an applicant contesting a rejection need different information. A simple model with many engineered or opaque variables can be difficult to understand. A post-hoc explanation approximates or describes behavior after the fact; it is not automatically the model’s internal mechanism or proof of causation. Lipton’s analysis cautions against treating the word interpretable as self-evident.
State the explanatory question
Ask whether the goal is debugging, checking consistency, enabling an appeal or understanding a causal mechanism. These goals require different evidence. A feature-attribution display may help one task while failing to justify a consequential policy.
Inspect assumptions and stability
Explanations can depend on the background data, feature representation and treatment of correlated variables. Small input changes or different methods may alter the story. Check whether a local approximation remains valid in the neighborhood where it is used.
Connect explanation to action
An affected person needs a way to correct errors or contest the criterion, not just a technical score breakdown. A suggested change should be feasible and legitimate. Altering a proxy to improve a score does not prove a causal improvement in the desired outcome.
Follow a worked example.
A hypothetical applicant is told that an employment-gap feature reduced their score. That message might identify a model association, but it does not show that the gap is job-relevant or that the underlying employment record is correct.
Walk it through, one step at a time
- Identify the source of the explanation: the original model, a local surrogate or a later narrative generator.
- Test whether the explanation consistently reflects actual model behavior and whether correlated features change the attribution.
- Offer record correction and a substantive review of the criterion. Do not equate explaining a harmful rule with justifying it.
Where people get stuck
A plausible natural-language rationale can be generated without faithfully representing how the system reached its result.
Now try it yourself.
1. Check the core idea
Does a linear model automatically provide a sufficient explanation for every user?
Reveal the answer & reasoning
No. Feature meanings, transformations, scale, number of inputs and the user’s task matter. Simplicity of mathematical form is not the same as useful understanding.
2. Apply what you learned
Can an explanation establish that a feature causes success at work?
Reveal the answer & reasoning
Not from predictive attribution alone. A causal claim needs suitable design and assumptions beyond explaining a model’s associations.