Where a deep neural network is selected for a high-risk system, the AISDP must include a candid assessment of the explainability limitations, distinct from the technical description of the explanation method itself. This assessment is a compliance-critical artefact because it sets the foundation for the compensating controls that the oversight framework must provide.
The assessment should address fidelity risk: the extent to which the post-hoc explanation method reflects the model’s actual reasoning rather than producing a plausible but potentially misleading narrative. LIME, for instance, fits a local linear model around each prediction point; if the true decision boundary is highly non-linear in that region, the LIME explanation may be unfaithful. Stability risk is also relevant, covering whether the explanation changes if the input is perturbed slightly. An explanation method that produces materially different attributions for nearly identical inputs undermines operator confidence and complicates audit.
Coverage limitations should be documented. Not every prediction may receive a full explanation due to computational cost constraints. The AISDP must state the explanation coverage: the proportion of predictions receiving full explanations, the method used for the remainder (such as top-three features only), and the computational overhead involved. For systems processing thousands of predictions daily, generating full SHAP explanations for every case may be infeasible; practical approaches include full explanations for a random sample, lightweight explanations for all predictions, and pre-computed explanations for common input patterns.
The assessment should conclude with the compensating controls selected. These typically include enhanced human oversight (more intensive review, longer dwell times, calibration cases), output validation against known-good references, constrained output spaces, and explanation quality monitoring in production (periodic fidelity testing, explanation pattern monitoring, and human evaluation sampling). The AI Governance Lead reviews and accepts the residual explainability risk with a formal sign-off, retained in the evidence pack.
Key outputs
- Explainability limitations assessment document
- Compensating controls specification
- AI Governance Lead acceptance of residual explainability risk