What are Explainable Neural Networks?
Explainable Neural Networks (XNNs) are modular predictive models that build explanation into their computation. Each module represents a feature or interaction, each partition defines a region of behaviour, and each active rule produces a contribution that can be traced to the final output.
Explanation as part of prediction
Post-hoc explainability methods analyse a trained black-box model from the outside. They can be useful, but the explanation remains a separate approximation of the original computation. XNNs take a different architectural approach: the components used to produce the prediction are themselves organised for inspection.
An XNN does not store all behaviour in one undifferentiated set of hidden parameters. It decomposes the task into modules, partitions and local rules. This creates an explicit relationship between the input region selected by the model, the mathematical function applied there and the contribution added to the result.
Modules represent features and interactions
An XNN module is responsible for a feature or a combination of features. A simple module may represent Age. An interaction module may represent Age × Education. Modules can also be connected into a wider acyclic structure where the output of one component becomes an input to another.
The modular design matters because feature interactions are not left implicit across a dense network. A technical reviewer can inspect the module representing the interaction, see how its behaviour changes across values and measure its contribution to individual predictions.
Partitions organise local behaviour
Within a module, partitions divide the relevant feature space into non-overlapping regions. Inputs with similar predictive behaviour are grouped together; materially different behaviour is separated. The resulting hierarchy can be explored from broader patterns to finer distinctions.
For a single feature, partitions may correspond to ranges such as Age ≤ 29.5 and Age > 29.5. For an interaction, a partition may combine conditions such as Age ≤ 29.5 AND Education > 8. Only one partition within a module activates for a given query.
Partitions provide a local context. Instead of forcing one equation to describe all possible values, the model can use a simple local expression within each consistent region.
Rules connect conditions to contributions
Each partition has an IF, THEN rule. The IF expression defines the conditions under which the partition is active. The THEN expression is generally a linear or polynomial function over the relevant transformed inputs.
For example:
IF Age ≤ 29.5 THEN contribution = θ0 + θ1(Age) + θ2(Age²)
The coefficients are learnt, but the variables and mathematical form remain explicit. In an interaction module, the expression may include terms for each feature and a combined term such as Age × Education.
The output of the active rule is the module attribution. Attributions from the active modules are added to a background value. The result is then passed through an identity, sigmoid or softmax function according to the prediction task.
The module execution path
For each query, an XNN performs a repeatable sequence:
- Receive the transformed values relevant to each module.
- Evaluate the module's partition conditions.
- Activate one non-overlapping partition per module.
- Execute the partition's THEN expression.
- Produce the module attribution.
- Combine attributions with the background value.
- Apply the task-specific output function.
Because the same sequence supplies both prediction and explanation data, the result can be connected to the exact modules, partitions and rules used during inference.
From module attribution to feature explanation
Interaction modules create a practical explanation challenge: the module has one total attribution, but a reader may want to know how much each feature contributed. XNN explanations can decompose the module attribution into feature-level values while preserving the total. Contributions from all modules involving the same feature can then be aggregated into a feature rollup.
This supports several views without changing the underlying computation. A rule view shows the active logic. A module view shows feature or interaction behaviour. A feature rollup offers a concise list of the original inputs that pushed the prediction up or down.
Why it matters
XNNs turn model behaviour into addressable components. That supports case-level explanation, targeted diagnostics, rule inspection, feature-interaction analysis and comparison across versions. The architecture does not remove the need for validation; it makes the object being validated more directly inspectable.