Console¶

The NextGen DataRobot Console provides critical AI Production features in a refreshed, modern user interface. This updated layout provides a seamless transition from model experimentation in Workbench and registration in Registry to model monitoring and management through deployments in Console.

Dashboard and overview¶

Topic	Describes how to
Dashboard	Navigate the deployment Dashboard, the central hub for deployment management activity.
Overview tab	Navigate and interact with the Overview tab, providing a model- and environment-specific summary that describes the deployment, including the information you supplied when creating the deployment and any model replacement activity.
Deployment actions	Manage a deployment with the settings and controls available in the actions menu.
Deployment reports	Generate a deployment report to summarize the details of a deployment, such as its owner, how the model was built, the model age, and the humility monitoring status.

Monitoring¶

Topic	Describes how to
Service health	Track model-specific deployment latency, throughput, and error rate.
Data drift	Monitor model accuracy based on data distribution.
Accuracy	Analyze the performance of a model over time.
Fairness	Monitor deployments to recognize when protected features fail to meet predefined fairness criteria.
Usage	Track prediction processing progress for use in accuracy, data drift, and predictions over time analysis.
Custom metrics	Create and monitor custom business or performance metrics or add pre-made metrics.
Data exploration	Export a deployment's stored prediction data, actuals, and training data to compute and monitor custom business or performance metrics.
Monitoring jobs	Monitor deployments running and storing feature data and predictions outside of DataRobot.
Deployment reports	Generate a deployment report to summarize the details of a deployment, such as its owner, how the model was built, the model age, and the humility monitoring status.

Predictions¶

Topic	Describes how to
Make predictions	Make predictions with large datasets, providing input data and receiving predictions for each row in the output data.
Prediction API	Adapt downloadable DataRobot Python code to submit a CSV or JSON file for scoring and integrate it into a production application via the Prediction API.
Monitoring	Access monitoring snippets for agent-monitored external models deployed in Console.
Prediction intervals	For time series deployments, enable and configure prediction intervals returned alongside the prediction response of deployed models.
Prediction jobs	View and manage prediction job definitions for a deployment.

Mitigation¶

Topic	Describes how to
Challengers	Compare model performance post-deployment.
Retraining	Define the retraining settings and then create retraining policies.
Humility	Monitor deployments to recognize, in real-time, when the deployed model makes uncertain predictions or receives data it has not seen before.

Settings¶

Topic	Describes how to
Set up service health monitoring	Enable segmented analysis to assess service health, data drift, and accuracy statistics by filtering them into unique segment attributes and values.
Set up data drift monitoring	Enable data drift monitoring on a deployment's Data Drift Settings tab.
Set up accuracy monitoring	Enable accuracy monitoring on a deployment's Accuracy Settings tab.
Set up fairness monitoring	Enable fairness monitoring on a deployment's Fairness Settings tab.
Set up custom metrics monitoring	Enable custom metrics monitoring by defining the "at risk" and "failing" thresholds for the custom metrics you created.
Set up humility rules	Enable humility monitoring by creating rules that enable models to recognize, in real-time, when they make uncertain predictions or receive data they have not seen before.
Configure challengers	Enable challenger comparison by configuring a deployment to store prediction request data at the row level and replay predictions on a schedule.
Configure retraining	Enable Automated Retraining for a deployment by defining the general retraining settings and then creating retraining policies.
Configure predictions settings	Review the Predictions Settings tab to view details about your deployment's prediction data or, for deployed time series models, enable prediction intervals in the prediction response.
Set up timeliness tracking	Enable timeliness indicators show if the prediction or actuals upload frequency meets the standards set by your organization.
Enable data exploration	Enable data exploration to compute and monitor custom business or performance metrics.
Configure deployment notifications	Enable personal notifications to trigger emails for service health, data drift, accuracy, and fairness monitoring.

Feature considerations¶

When curating a prediction request/response dataset from an external source:

Include the 25 most important features.
Follow the CSV file size requirements.
For classification projects, classes must have a value of 0 or 1, or be text strings.

Additionally, note that:

Self-Managed AI Platform only: By default, the 25 most important features and the target are tracked for data drift.
The Make Predictions tab is not available for external deployments.
DataRobot deployments only track predictions made against dedicated prediction servers by deployment_id.
- To be analyzed by model management, other prediction methods should record requests and predictions to a CSV file. Then, upload the file to DataRobot as an external deployment.
- As of Self-Managed AI Platform version 7.0, the previously deprecated endpoints using project_id and model_id instead of deployment_id, return HTTP 404 Not found (unless otherwise configured with a DataRobot representative).
The first 1,000,000 predictions per deployment per hour are tracked for data drift analysis and computed for accuracy. Further predictions within an hour where this limit has been reached are not processed for either metric. However, there is no limit on predictions in general.
If you score larger datasets (up to 5GB), there will be a longer wait time for the predictions to become available, as multiple prediction jobs must be run. If you choose to navigate away from the predictions interface, the jobs will continue to run.
After making prediction requests, it can take 30 seconds or so for data drift and accuracy metrics to update. Note that the speed at which the metrics update depends on the model type (e.g., time series), the deployment configuration (e.g., segment attributes, number of forecast distances), and system stability.
DataRobot recommends that you do not submit multiple prediction rows that use the same association ID—an association ID is a unique identifier for a prediction row. If multiple prediction rows are submitted, only the latest prediction uses the associated actual value. All prior prediction rows are, in effect, unpaired from that actual value. Additionally, all predictions made are included in data drift statistics, even the unpaired prediction rows.
If you want to write back your predictions to a cloud location or database, you must use the Prediction API.

Time series deployments¶

To make predictions with a time series deployment, the amount of history needed depends on the model used:
- Traditional time series (ARIMA family) models require the full history between training time and prediction time. DataRobot recommends scoring these models with the Prediction API.
- All other time series models only require enough history to fill the feature derivation window, which varies by project. For cross series, all series must be provided at prediction time.
Both categories of models are supported for real-time predictions, with a maximum payload size of 50 MB.
ARIMA family and non-ARIMA cross-series models do not support batch predictions.
All other time series models support batch predictions. For multiseries, input data must be sorted by series ID and timestamp.
There is no data limit for time series batch predictions on supported models other than a single series cannot exceed 50 MB.
When scoring regression time series models using integrated enterprise databases, you may receive a warning that the target database is expected to contain the following column, which was not found: DEPLOYMENT_APPROVAL_STATUS. The column, which is optional, records whether the deployed model has been approved by an administrator. If your organization has configured a deployment approval workflow, you can:
- Add the column to the target database.
- Redirect the data to another column by using the columnNamesRemapping parameter.
After taking either of the above actions, run the prediction job again, and the approval status appears in the prediction results. If you are not recording approval status, ignore the message, and the prediction job continues.
To ensure DataRobot can process your time series data for deployment predictions, configure the dataset to meet the following requirements:
- Sort prediction rows by their timestamps, with the earliest row first.
- For multiseries, sort prediction rows by series ID and then by timestamp.
- There is no limit on the number of series DataRobot supports. The only limit is the job timeout. For more information, see the batch prediction limits.
For dataset examples, see the requirements for the scoring dataset.

Multiclass deployments¶

Multiclass deployments of up to 100 classes support monitoring for target, accuracy, and data drift.
Multiclass deployments of up to 100 classes support retraining.
Multiclass deployments created before Self-Managed AI Platform version 7.0 with feature drift enabled don't have historical data for feature drift of the target; only new data is tracked.
DataRobot uses holdout data as a baseline for target drift. As a result, for multiclass deployments using certain datasets, rare class values could be missing in the holdout data and in the baseline for drift. In this scenario, these rare values are treated as new values.

Challengers¶

To enable Challengers and replay predictions against them, the deployed model must support target drift tracking and not be a Feature Discovery or Unstructured custom inference model.
To replay predictions against Challengers, you must be in the organization associated with the deployment. This restriction also applies to deployment owners.

Prediction results cleanup¶

For each deployment, DataRobot periodically performs a cleanup job to delete the deployment's predicted and actual values from its corresponding prediction results table in Postgres. DataRobot does this to keep the size of these tables reasonable while allowing you to consistently generate accuracy metrics for all deployments and schedule replays for challenger models without the danger of hitting table size limits.

The cleanup job prevents a deployment from reaching its "hard" limit for prediction results tables; when the table is full, predicted and actual values are no longer stored, and additional accuracy metrics for the deployment cannot be produced. The cleanup job triggers when a deployment reaches its "soft" limit, serving as a buffer to prevent the deployment from reaching the "hard" limit. The cleanup prioritizes deleting the oldest prediction rows already tied to a corresponding actual value. Note that the aggregated data used to power data drift and accuracy over time are unaffected.

Managed AI Platform¶

Managed AI Platform users have the following hourly limitations. Each deployment is allowed:

Data drift analysis: 1,000,000 predictions or, for each individual prediction instance, 100MB of total prediction requests. If either limit is reached, data drift analysis is halted for the remainder of the hour.
Prediction row storage: the first 100MB of total prediction requests per deployment per each individual prediction instance. If the limit is reached, no prediction data is collected for the remainder of the hour.