FMEA — Structured Failure Mode Analysis

Failure Mode and Effects Analysis (FMEA), following the IEC 60812 framework, is the workhorse of AI risk identification. For each system component (data pipeline, feature engineering, model inference, post-processing, user interface), the team enumerates the ways the component can fail, the effects of each failure, and the severity of those effects.

In an AI context, failure modes extend beyond traditional software failures. They include data drift, concept drift, adversarial manipulation, distributional shift in input data, label noise propagation, and emergent biases. The practical approach starts with the system’s architecture diagram, walking through each component and asking three questions at each node: what can go wrong with the data entering this component, what can go wrong with the processing within this component, and what can go wrong with the output leaving this component?

Each failure mode is assigned a Risk Priority Number (RPN): Severity (1–10) × Occurrence (1–10) × Detectability (1–10). RPNs above the defined threshold (often 100 on a 1,000-point scale, though this is system-specific) trigger mandatory mitigation. The completed FMEA worksheet is a Module 6 evidence artefact. Dedicated software (Relyence, Jama Connect) provides structured worksheets with automatic RPN calculation; spreadsheet templates serve smaller teams.

Key outputs

• Component-by-component FMEA with AI-specific failure modes
• RPN scoring (Severity × Occurrence × Detectability) per failure mode
• Mandatory mitigation for RPNs above threshold
• Module 6 AISDP evidence

Stakeholder Consultation

Risks to fundamental rights are frequently invisible from within the engineering team. Structured consultation with deployers, affected persons, civil society representatives, domain experts, and legal advisors surfaces risks that technical analysis alone will miss. The AI System Assessor schedules consultations at multiple points in the development lifecycle; a single pre-deployment review is insufficient.

Consultations are documented with attendees, the questions posed, the responses received, and the actions taken in response. Each stakeholder concern is traced to a risk register entry or a documented rationale for exclusion. Generic stakeholder statements are insufficient; the consultation must produce actionable insights that inform the risk assessment.

Stakeholder recruitment should prioritise persons and groups who will be directly affected by the system’s decisions. For a recruitment screening system, this includes job applicants, HR professionals, diversity officers, and employment law specialists. For a credit scoring system, this includes consumer advocates, financial inclusion organisations, and data protection specialists. The AI System Assessor documents the recruitment methodology and any limitations on representativeness.

Key outputs

• Structured stakeholder consultation at multiple lifecycle points
• Documentation (attendees, questions, responses, actions)
• Traceability from each concern to the risk register or exclusion rationale
• Module 6 AISDP evidence

Regulatory Gap Analysis

The team systematically maps the system’s characteristics against every obligation in Articles 8 through 15, identifying areas where current design, testing, or operational practice falls short. Each gap becomes a risk entry in the risk register. The gap analysis should also consider the Master Compliance Questionnaire, working through each applicable question to ensure comprehensive coverage.

Credo AI’s policy engine can automate the Article-to-control mapping and flag unmatched requirements. For manual execution, a structured walkthrough of the Articles proceeds requirement by requirement, asking for each: is this requirement addressed in the current system design? If so, where is the evidence? If not, what is the gap and what is the associated risk?

The regulatory gap analysis is particularly valuable early in the development lifecycle, when gaps can be addressed through design changes rather than retrospective remediations. It should be repeated after any substantial modification to confirm that no new gaps have been introduced.

Key outputs

• Systematic Article 8–15 mapping against current system state
• Gap-to-risk-entry traceability
• Master Compliance Questionnaire walkthrough
• Module 6 AISDP evidence

Adversarial Red-Teaming

A dedicated adversarial testing programme subjects the system to deliberate misuse scenarios, input manipulation, data poisoning attempts, and social engineering of the human oversight layer. Red-team exercises are conducted by the Technical SME with personnel who were not involved in the system’s development, ensuring fresh perspectives and reduced confirmation bias.

MITRE ATLAS provides the threat taxonomy, cataloguing known adversarial techniques against AI systems (evasion, poisoning, extraction, inference) with real-world case studies. The red team works through the ATLAS matrix, assessing which techniques are applicable and attempting to execute them in a controlled environment. Microsoft’s PyRIT automates portions of this for LLM-based systems; for non-LLM systems, the red team manually crafts adversarial inputs.

Each red-team finding becomes a risk entry recording the attack description, the severity, and the required mitigation. Red-teaming for risk identification purposes (this article) is distinct from the cybersecurity red-team exercises described in the security section; the former is oriented towards understanding the risk landscape, the latter towards testing the controls that mitigate those risks.

Key outputs

• Adversarial red-teaming by independent personnel
• MITRE ATLAS threat taxonomy coverage
• Per-finding risk register entries
• Module 6 AISDP evidence

Horizon Scanning

Horizon scanning reviews incidents, enforcement actions, and published risk assessments from comparable AI systems in the same or adjacent domains. A recruitment screening system provider should review enforcement actions against algorithmic hiring tools, published bias audits of similar systems, and academic literature on fairness in automated selection. This method identifies risks that the organisation might not have encountered internally.

Three curated sources provide the foundation: the OECD AI Policy Observatory for regulatory developments, the Stanford HAI tracker for policy and research, and the AI Incident Database (maintained by the Responsible AI Collaborative) for real-world incident scenarios. The AI System Assessor performs horizon scanning at least quarterly and before every risk register review.

Each horizon scanning finding is assessed for its relevance to the system’s specific risk profile. A finding from a comparable system in the same domain warrants a risk register entry; a finding from a dissimilar system in a different domain may warrant a watching brief. The AI Governance Lead reviews horizon scanning findings as part of the quarterly governance review.

Key outputs

• Quarterly horizon scanning using OECD, Stanford HAI, and AI Incident Database
• Relevance assessment per finding
• Risk register entries or watching briefs as appropriate
• Module 6 AISDP evidence