Configure challengers¶
DataRobot can securely store prediction request data at the row level for deployments (not supported for external model deployments). This setting must be enabled on the Settings > Challengers tab for any deployment using the Challengers tab. In addition to enabling challenger analysis, access to stored prediction request rows enables you to thoroughly audit the predictions and use that data to troubleshoot operational issues. For instance, you can examine the data to understand an anomalous prediction result or why a dataset was malformed.
You may have enabled prediction row storage during deployment creation, under the Challenger Analysis section. Once enabled, prediction requests made for the deployment are collected by DataRobot. Prediction explanations are not stored. Contact your DataRobot representative to learn more about data security, privacy, and retention measures or to discuss prediction auditing needs.
Important
Prediction requests are only collected if the prediction data is in a valid data format interpretable by DataRobot, such as CSV or JSON. Failed prediction requests with a valid data format are also collected (i.e., missing input features).
On a deployment's Challengers Settings page, you can configure the following settings to store prediction request data at the row level and replay predictions on a schedule:
| Field | Description |
|---|---|
| Prediction Data Collection | |
| Enable prediction row storage | Enables prediction data storage, a setting required to score predictions made by the challenger and compare performance with the deployed model. |
| Challengers | |
| Enable challenger analysis | Enables the use of challenger models, allowing you to compare models post-deployment and replace the champion model if necessary. |
| Replay Schedule | |
| Automatically replay challengers | Enables a recurring, scheduled challenger replay on stored predictions for retraining. |
| Replay time | Displays the selected replay time in UTC. |
| Local replay time | Displays the selected replay time converted to local time. If the selected replay time in UTC results in the replay time on a new day, a warning appears. |
| Frequency / Time | Configures the replay schedule, selecting from the following options:
|
| Use advanced scheduler | Configures the replay schedule using values for the following advanced options:
|
Note
The Time setting applies across all days selected. In other words, you cannot set checks to occur every 12 hours on Saturday and every 2 hours on Monday.