Intent Alignment Dashboards — Real-Time vs AISDP Thresholds

The monitoring layer provides dashboards that show the system’s current behaviour relative to its documented intended purpose. If the AISDP declares that the system must achieve AUC-ROC ≥ 0.80 and fairness ratios ≥ 0.90, the dashboard displays these metrics in real time with a clear indication of whether the system is within specification.

Intent alignment dashboards serve two audiences. The engineering team needs operational visibility into the system’s technical metrics: latency, throughput, error rates, feature distributions, and model performance. The governance team needs compliance visibility: are all declared thresholds being met, when was the last threshold breach, and what was the response? A layered monitoring approach addresses both audiences from the same underlying data.

At the base layer, infrastructure monitoring (Prometheus with Grafana, or Datadog) tracks system health. Above it, ML-specific monitoring (Evidently AI, NannyML, Fiddler AI, or Arize AI) tracks feature drift, prediction drift, performance estimation, and fairness metrics. Above that, the governance layer aggregates the ML metrics into compliance-relevant summaries. This layered approach ensures that each audience sees the information it needs without being overwhelmed by detail intended for others.

Key outputs

• Intent alignment dashboards with real-time threshold comparison
• Layered monitoring stack (infrastructure, ML-specific, governance)
• Dashboard access configured for engineering and governance audiences
• Module 12 documentation of monitoring architecture

Statistical Anomaly Detection & Severity-Based Escalation

Statistical anomaly detection algorithms identify unusual patterns in the system’s inputs, outputs, or operational metrics. Anomalies may indicate data quality problems, model degradation, adversarial inputs, or infrastructure failures. The monitoring layer runs these algorithms continuously on production data.

When anomalies are detected, severity-based escalation ensures they receive an appropriate response. Three severity tiers are defined. Low-severity anomalies are logged and reviewed during the next scheduled monitoring cycle. Medium-severity anomalies trigger immediate alerts to the technical team and are investigated within a defined timeframe. High-severity anomalies trigger automatic escalation to the AI Governance Lead and may activate break-glass mechanisms if the anomaly indicates an imminent risk to affected persons.

The escalation thresholds are calibrated to balance sensitivity against alert fatigue. Thresholds that are too sensitive will flood the team with false positives, eroding responsiveness. Thresholds that are too permissive will miss genuine problems. The calibration is documented in the AISDP and reviewed periodically based on operational experience. The anomaly detection methods, escalation rules, and response procedures collectively form a critical part of the post-market monitoring plan documented in Module 12.

Key outputs

• Anomaly detection algorithms configured for inputs, outputs, and operational metrics
• Three-tier severity classification with defined escalation paths
• Calibrated alert thresholds with documented rationale
• Module 6 and Module 12 AISDP evidence

Multi-Dimensional Drift Monitoring (Input, Output, Fairness, Error, Override)

Single-dimension monitoring may miss drift that manifests across multiple dimensions without crossing any individual threshold. The monitoring layer tracks drift across five dimensions simultaneously: input feature distributions, output score distributions, fairness metrics, error rates, and operator override rates.

Each dimension provides a different perspective on the system’s behavioural stability. Input drift (monitored at Layer 2) indicates that the population the system serves is changing. Output drift (monitored at Layer 4) indicates that the system’s decisions are shifting. Fairness drift indicates that the system’s impact on different subgroups is evolving. Error rate changes indicate that the system’s accuracy is degrading. Override rate changes indicate that human operators are finding the system’s recommendations less reliable.

Correlating drift signals across dimensions yields diagnostic insight. Input drift accompanied by output drift suggests the model is responding appropriately to a changing population. Output drift without input drift suggests the model itself is changing, perhaps through an undetected configuration change or a feedback loop effect. The monitoring layer should include specific checks for feedback loops, where the system’s outputs influence the data that is subsequently used to evaluate or retrain the system. These checks require comparing the training data distribution against the production data distribution while controlling for the system’s own influence.

Key outputs

• Five-dimensional drift monitoring configuration
• Cross-dimensional correlation analysis
• Feedback loop detection methodology
• Module 12 post-market monitoring evidence

Feedback Loop Detection — Training vs Production Distribution

Feedback loops occur when the system’s outputs influence the data that is subsequently used to evaluate or retrain the system. A recruitment screening system that reduces the diversity of candidates presented to hiring managers may, over time, generate training data that reflects the system’s own biases rather than genuine hiring outcomes. This self-reinforcing dynamic can amplify small initial biases into significant disparate impact.

Detecting feedback loops requires comparing the system’s training data distribution against the production data distribution while controlling for the system’s own influence on that distribution. If the production data is increasingly shaped by the system’s past decisions, the distributions will converge in ways that mask real-world changes. The monitoring layer includes specific statistical tests for this convergence pattern.

When a feedback loop is detected, the response may involve collecting external validation data that is not influenced by the system’s outputs, adjusting the retraining methodology to account for the system’s selection effects, or pausing automated retraining until the loop is broken. The detection methodology, response procedures, and any corrective actions taken are documented in AISDP Module 12 and feed into the risk management system documented in Module 6.

Key outputs

• Feedback loop detection methodology (training vs production distribution comparison)
• Statistical tests for distribution convergence
• Response procedures for confirmed feedback loops
• Module 6 and Module 12 AISDP evidence