Skip to content

Click in-app to access the full platform documentation for your version of DataRobot.

Monitor an external model with the MLOps agent

With DataRobot MLOps you can register an external model, create an external prediction environment, and deploy the model to the external prediction environment you registered. Next, you can install and configure the monitoring agent alongside the external model, establishing a deployment scenario for that external model. Once installed and configured, the monitoring agent allows you to monitor models running externally as an MLOps deployment so that you can take advantage of DataRobot's powerful MLOps model management tools to monitor accuracy and data drift, prediction distribution, latency, and more.

To install the MLOps agent to monitor an external model running in a prediction environment outside of DataRobot, follow the workflow outlined below:

graph TB
  A[Decide to monitor an existing external model]:::link --> B[Register an external model package]:::link
  classDef link color: blue;
  click A "./ext-cus-model-ext-env.html#decide-to-monitor-an-existing-external-model"
  click B "./ext-cus-model-ext-env.html#register-an-external-model"
  B --> C{Create an external prediction environment?}
  C -->|No|E[Deploy the model to an external prediction environment]:::link
  click E "ext-cus-model-ext-env.html#deploy-the-model-package-to-an-external-prediction-environment"
  C --> |Yes|D[Add an external prediction environment]:::link
  click D "./ext-cus-model-ext-env.html#add-an-external-prediction-environment"
  D --> E
  E --> F[Obtain the MLOps agent tarball and your API key]:::link
  click E "./ext-cus-model-ext-env.html#download-the-mlops-agent-tarball"
  F --> G[Install and configure the monitoring agent]:::link
  click G "./ext-cus-model-ext-env.html#install-and-configure-the-mlops-agent"
  G --> H[Configure monitoring agent and MLOps library communication]:::link
  click H "./ext-cus-model-ext-env.html#configure-the-mlops-agent-and-library-spooler"

Decide to monitor an existing external model

The monitoring agent is a solution for monitoring external models on your infrastructure while reporting statistics to DataRobot MLOps. The API used by the monitoring agent allows you to request specific data to report to a deployment you create in DataRobot. For more information, see the MLOps agent overview.

MLOps agent overview

Register an external model package

To report predictions metrics to an MLOps deployment in DataRobot, you must first register an external model's details as a model package in the DataRobot Model Registry; then, you can create an external MLOps deployment.

Register an external model package

Optional: Add an external prediction environment

To create an external deployment, you need an external prediction environment. Create an external prediction environment if you don't already have one in DataRobot.

Add an external prediction environment

Deploy the model to an external prediction environment

To associate a model running externally with the external model package registered in the Model Registry, you must deploy the model from the Model Registry to an external prediction environment. After deploying this model externally, you can obtain the Model ID and Deployment ID from the external deployment's Overview tab. The monitoring agent uses the Model ID and Deployment ID to report an external model's data to the deployment in DataRobot MLOps.

Deploy a model to an external prediction environment

Obtain the MLOps agent tarball and your API key

The monitoring agent is a solution for monitoring external models on your infrastructure while reporting statistics to DataRobot MLOps. In the monitoring agent's configuration file, you must provide your MLOps URL and an API key. API keys are the preferred method for authenticating requests to the DataRobot API; they replace the legacy API token method.

To use the monitoring agent, you must obtain the MLOps agent tarball and an API key from DataRobot's developer tools.

Get started with the MLOps agent

Install and configure the MLOps agent

To monitor externally deployed models, you must implement the following software components included in the MLOps agent tarball download:

  • The MLOps library: Provides an API to communicate an external model's prediction data to the associated deployment in DataRobot (the external deployment of an external model you created earlier). The function calls provided by the MLOps library allow you to request specific data that you want to report, including prediction time, the number of predictions, and other metrics and deployment statistics. The MLOps library writes this data to a spooler (or buffer) channel, from which the data can then be sent to DataRobot MLOps by either the monitoring agent or other MLOps library method calls. Libraries are available in Python 2, Python 3, and Java.

  • The MLOps agent: Monitors the spooler (or buffer) channel in a location you define when you configure MLOps agent and library communication. The MLOps agent reads data from the spooler and reports to the the associated deployment in DataRobot (the external deployment of an external model you created earlier). Depending on your configuration, the agent can read and report this data manually or automatically.

Install and configure the MLOps agent

Configure the MLOps agent and library spooler

The MLOps library communicates with the monitoring agent through a spooler, so it's essential that the library and the agent have matching spooler configurations. Some spooler configuration settings are required, and some are optional. You can configure these settings programmatically; settings configured through environment variables take precedence over those defined in configuration files.

Configure the monitoring agent and MLOps library spooler


Updated June 22, 2022
Back to top