MLOps FAQ¶

DataRobot MLOps supports three types of model for deployment:

• DataRobot models built with AutoML and deployed directly to the inventory
• Custom inference models assembled in the Custom Model Workshop
• External models registered as model packages and monitored by the MLOps agent.

To make predictions with a deployment, navigate to the Predictions tab. From there, you can use the predictions interface to drag and drop prediction data and return prediction results. Supported models can download and configure Scoring Code from a deployment. External models can score datasets in batches on a remote environment with the Portable Prediction Server. For a code-centric experience, use the provided Python Scoring Code, which contains the commands and identifiers needed to submit a CSV or JSON file for scoring with the Prediction API.

Models that run on your own infrastructure (outside of DataRobot) may be run in different environments and can have differing deployment permissions and approval processes. For example, while any user may have permission to deploy a model to a test environment, deployment to production may require a strict approval workflow and only be permitted by those authorized to do so. Prediction environments support this deployment governance by grouping deployment environments and supporting grouped deployment permissions and approval workflows. They indicate the platform used in your external infrastructure (AWS, Azure, Snowflake, etc.) and the model formats it supports.

To activate the Accuracy tab for deployments, you must first select an association ID, a foreign key that links predictions with future results (referred to as actuals or outcome data). In the Settings > Data tab for a deployment, the Inference section has a field for the column name containing the association IDs. Enter the column name here, and then, after making predictions, add actuals to the deployment to generate accuracy statistics.

Data Drift refers to changes in the distribution of prediction data versus training data. Data Drift alerts indicate that the data you are making predictions on looks different from the data the model used for training. DataRobot uses PSI or "Population Stability Index" to measure this. Models themselves cannot drift; once they are fit, they are static. Sometimes the term "Model Drift" is used to refer to drift in the predictions, which simply indicates that the average predicted value is changing over time.

The Service Health, Data Drift, and Accuracy summaries in the deployment inventory provide an at-a-glance indication of health and accuracy for all deployed models. To view this more detailed information for an individual model, click on the model in the inventory list. For more information about interpreting the color indicators, reference the documentation.

Prediction data needs to be provided in a CSV or JSON file. For more information, reference the documentation for the DataRobot Prediction API.

To replace a model, use the Replace model functionality found in the Actions menu for a deployment. Note that DataRobot issues a warning if the replacement model differs from the current model in either of these ways:

• Feature names do not match.
• Feature names match but have different data types in the replacement model.

Humility monitoring, available from a deployment's Humility tab, allows you to configure rules that allow models to be capable of recognizing, in real-time, when they make uncertain predictions or receive data they have not seen before. Unlike data drift, model humility does not deal with broad statistical properties over time—it is instead triggered for individual predictions, allowing you to set desired behaviors with rules that depend on different triggers. Humility rules help to identify and handle data integrity issues during monitoring and to better identify the root cause of unstable predictions.

In order to add a challenger model to compare against your deployed model, you must be an MLOps user, and you must enable the Challengers tab. To do so, select Settings > Data in your deployment. Under Data Drift in the right pane, toggle Enable prediction rows storage and click Save change(s). This setting is required for you to compare challenger models to the champion model. When you select a deployment, you can now select the Challengers tab.

The Portable Prediction Server (PPS) is a DataRobot execution environment for DataRobot model packages (.mlpkg files) distributed as a self-contained Docker image. The PPS can be run disconnected from the main installation environments. Once started, the image serves HTTP API via the :8080 port. In order to use it, you create an external deployment, create an external prediction environment for your infrastructure, download the PPS Docker image, and download the model package. This configuration allows you to run PPS outside of DataRobot but still have access to insights and statistics from your deployment in the application.

They serve different purposes and, while they both allow you to take your models out of DataRobot, they differ in how you interface with the model and in the number of models they support:

• Portable Prediction Server: Supports almost every Leaderboard model by exporting a self-contained Docker image to host a remote DataRobot execution environment for one or more DataRobot model packages (MLPKG files). The PPS exposes a REST API for real-time predictions, batch predictions, and prediction explanations; therefore, you must ensure your use case supports calling that API.

• Scoring Code: Supports qualifying Leaderboard models by exporting the model as a Java Archive (JAR) file, allowing you to integrate your inference model directly into your Java (and Python) applications. Scoring code can also be called from a CLI in a standalone manner. For more information on which models don't support Scoring Code export, see the feature considerations.