What Makes a Good AI Explanation?

Explanation is not the same as interpretation

An explanation is the objective, system-generated account of how an output was produced. Interpretation is the human act of applying context, expertise and judgement to that account. The two are connected, but they are not interchangeable.

For example, an Explainable Neural Network (XNN) may report that a liquidity module contributed negatively, a revenue-stability interaction contributed positively and a threshold rule activated. That is explanation data. A credit analyst may interpret the combination as evidence that the applicant can service current obligations but remains vulnerable to short-term cash-flow variation.

Separating the two protects accuracy. The model should expose its computation; the stakeholder should apply the domain meaning for which they are responsible.

Five qualities of a useful explanation

Faithfulness. The explanation should correspond to the components that produced the result. Model-native rules and attributions are useful because they arise from the XNN execution path rather than from a separate surrogate.

Completeness for purpose. Complete does not always mean displaying every internal object. A consumer explanation and a technical audit have different requirements. The underlying record should retain the contributing components, while each view presents the information necessary for its audience and task.

Clarity. Terms should be defined, direction should be explicit and the reader should be able to distinguish the baseline, positive contributions, negative contributions and final output. A list of numbers without scale or context is not an explanation.

Traceability. A reader should be able to move from the presented factor to the feature, module, partition or rule that supports it. Model version, input context and query metadata may also be necessary where decisions can be challenged or audited.

Relevance. The explanation should answer the question being asked. "How was this calculated?" requires a different view from "Why was this case declined?" or "What could change the outcome?"

Explain, filter and interpret

Hybrid Intelligence uses a three-stage pattern:

1. Explain: generate the model-native record, including the prediction, attributions, activations and relevant rules.
2. Filter: group, rank, summarise or augment the explanation according to the purpose and permissions of the audience.
3. Interpret: allow the stakeholder to apply domain knowledge, policy and situational context.

Filtering must preserve lineage. If several interaction contributions are rolled up into a feature-level factor, the rollup should remain reconcilable with the underlying module attributions. If technical detail is omitted from a consumer view, it should still be available to authorised reviewers.

Audience changes presentation, not model history

A domain expert may need exact conditions and local expressions. An operational user may need the top factors, relevant rule and permitted next step. An executive may need aggregate patterns across a portfolio. A regulator or internal auditor may need version information, activation paths and evidence that a result has not changed since it was recorded.

These are different explanation products derived from one execution history. The underlying prediction should not be rewritten to suit a preferred narrative.

Concise without becoming shallow

Conciseness is valuable when it removes repetition or low-impact detail. It becomes risky when it hides interaction effects, uncertainty, thresholds or dependencies that materially alter meaning.

A strong layered explanation begins with the outcome and principal factors, then allows the reader to open the module, rule or attribution detail. This keeps the first view usable while preserving technical depth.

Verification and justification

An explanation can show which components contributed and how they combined. Audit metadata and Unique Verification Codes (UVCs) can support integrity checks over selected model and query information. Neither mechanism proves that the business policy is appropriate or that the outcome is fair. Those conclusions require evaluation against external requirements and domain evidence.

Good explanation design therefore states the scope of the claim. "This factor contributed +0.12 to the model score" is a computational statement. "This was a fair decision" is a normative conclusion that needs additional analysis.

Why it matters

Poor explanations create two opposite risks: technical detail without meaning, or friendly language without evidence. A layered, traceable explanation provides a usable account for the immediate reader while preserving the model history needed for challenge, validation and governance.