Monitoring¶
To trust a model to power mission-critical operations, you must have confidence in all aspects of model deployment. By closely tracking the performance of models in production, you can identify potential issues before they impact business operations. Monitoring ranges from whether the service is reliably providing predictions in a timely manner and without errors to ensuring the predictions themselves are reliable.
The predictive performance of a model typically starts to diminish as soon as it’s deployed. For example, someone might be making live predictions on a dataset with customer data, but the customer’s behavioral patterns might have changed due to an economic crisis, market volatility, natural disaster, or even the weather. Models trained on older data that no longer represents the current reality might not just be inaccurate, but irrelevant, leaving the prediction results meaningless or even harmful. Without dedicated production model monitoring, the user cannot know or detect when this happens. If model accuracy starts to decline without detection, the results can impact a business, expose it to risk, and destroy user trust.
The endpoints in this section cover the many aspects of model monitoring.
| Topic | Description |
|---|---|
| Monitoring jobs | Define and run batch monitoring jobs against external data sources. |
| Accuracy | Analyze deployment accuracy by comparing predictions to actual outcomes over time. |
| Batch monitoring | Review monitoring statistics grouped by prediction batch instead of time. |
| Bias and fairness | Evaluate fairness metrics and bias across protected features in production. |
| Actuals configuration | Configure actuals ingestion and deployment monitoring settings. |
| Custom metrics | Define, submit, and track custom metrics for deployment performance. |
| Custom scoring metrics | Create and manage custom scoring metrics for model evaluation. |
| Data exports | Export prediction and monitoring data for downstream analysis. |
| Data drift | Monitor data drift and feature distribution changes in production deployments. |
| Service health | Track deployment health metrics such as latency, throughput, and errors. |
| Tracking agents | Install and configure the monitoring agent for external deployment tracking. |