UMNAI
Collection 3

How XNN Predictions are Built

Explainable Neural Network (XNN) predictions are additive before the final task-specific transformation. Each active module rule produces an attribution, those contributions are combined with a background value, and an identity, sigmoid or softmax function produces the model output.

By Angelo Dalli7 min read
z = B0 + Σ Mm(x) ŷ = σ(z)Background+0.10Module M1+0.32Module M2-0.22Module M3+0.15ΣSumσ(z)Task functionScoreAdditive before the task function
A background value plus signed module contributions are summed, then transformed by an identity, sigmoid or softmax function into the task output.

The common structure

Although regression and classification return different forms of output, their explanations share a common structure. Each XNN module evaluates the query, selects one partition and executes its local rule. The resulting module attribution contributes to the model score.

At a high level:

z = B0 + Σ Mm(x)

ŷ = σ(z)

B0 is the background value, Mm(x) is the attribution produced by module m, and σ is the output function.

This structure makes the components additive on the pre-activation scale. It also provides a direct reconciliation route from the active modules to the output.

The background value

The background value is the common starting point or intercept. In a regression model it may resemble an average baseline. A positive attribution raises the prediction relative to the baseline; a negative attribution reduces it.

The background should not be described as an explanation for the individual case. It is the reference from which case-specific contributions are measured.

Regression

For regression, the output function is the identity: the combined value is returned directly.

If a property model has a background value of £334,000 and the active modules sum to £635,000, the prediction is £969,000. If the module sum is -£56,000, the prediction is £278,000.

Raw regression attributions inherit the unit of the target, making them readily interpretable. They still require context: a £20,000 contribution may be material in one valuation range and minor in another.

Binary classification

For binary classification, the combined score is passed through a sigmoid to produce a value between zero and one. A threshold then maps the value to a class. The default may be 0.5, but production thresholds should reflect the model evaluation and decision objective.

It is important to distinguish three objects:

  • the additive pre-sigmoid score;
  • the post-sigmoid output;
  • the class selected by the threshold.

A module attribution is generally not a direct percentage-point change in the final probability. Its effect depends on the combined score and the non-linear sigmoid.

Multi-class classification

In multi-class classification, the model produces a score for each class. Softmax converts the vector into outputs that sum to one, and the class with the highest value is selected.

Each active rule can contain one expression per output class. A module may therefore support one class while opposing another. A useful explanation presents the full class vector and class-specific contributions where the distinction is material.

Rules behind the numbers

The module attribution is produced by the active partition's THEN expression. The IF condition defines the local region, for example a numeric range or categorical combination. The THEN expression may include linear, quadratic and interaction terms.

This local structure allows the same feature to behave differently in different regions while retaining explicit logic. A threshold in the partition hierarchy may create a visible change in the attribution curve.

Feature and interaction views

The model is additive by module, not necessarily by original feature. Interaction modules need decomposition before their contributions can be included in a feature rollup.

The module view is the most direct representation of the computation. The feature rollup is a derived but reconcilable presentation designed for readability. Both should remain available for technical review.

From prediction to operational decision

The model may return a score or class, but the operational system may apply a policy threshold, evidence requirement or manual review step. The product interface should record these separately. A changed business threshold does not alter the underlying XNN prediction; it changes how the organisation acts on it.

Why it matters

Understanding the output construction prevents attribution and probability from being confused. It also enables reconciliation, threshold governance and consistent explanation across regression, binary and multi-class applications.

Continue the journey

Continue with Model Lifecycle.