Skip to content

Click in-app to access the full platform documentation for your version of DataRobot.

MLOps agent

Availability information

The MLOps agent is a feature exclusive to DataRobot MLOps. Contact your DataRobot representative for information on enabling it.

DataRobot MLOps provides powerful tools for tracking and managing models for prediction. But what if you already have—or need to have—deployments running in your own environment? How can you monitor external models that may have intermittent or no connectivity and so may report predictions sporadically?

The MLOps agent allows you to monitor and manage external models—those running outside of DataRobot MLOps. With this functionality, predictions and information from these models can be reported as part of MLOps deployments. You can use the same powerful model management tools to monitor accuracy, data drift, prediction distribution, latency, and more, regardless of where the model is running. Data provided to DataRobot MLOps provides valuable insight into the performance and health of those externally deployed models.

The MLOps agent provides:

  • The ability to manage, monitor, and get insight from all model deployments in a single system
  • API and communications constructed to ensure little or no latency when monitoring external models
  • Support for deployments that are always connected to the network and the MLOps system, as well as partially or never-connected deployments
  • The MLOps library (available in Python and Java) which can be used to monitor models written natively in those languages or to report the input and output of a model artifact in any language
  • Configuration with the Portable Prediction Server

Reference the following sections for more information about using the MLOps agent:

Components of the MLOps agent

When you enable the MLOps agent, you have access to the agent installation and MLOps components, all packaged within a single tarball. The image below illustrates the roles of these components in enabling DataRobot MLOps to manage external models.

Component Description
External model External models are machine learning models running outside of DataRobot, within your environment. The deployments (running in Python or Java) score data and generate predictions along with other information, such as the number of predictions created and the length of time to create each.
DataRobot MLOps library The MLOps library, available in Python (v2 and v3) and Java, provides APIs to report prediction data and information from a specified deployment (identified by deployment ID and model ID). Supported library calls for the MLOps client let you specify which data to report to the MLOps service, including prediction time, number of predictions, and other metrics and deployment statistics.
Buffer The library-provided APIs pass messages to a configured buffer.
MLOps agent The MLOps agent detects data written to the target buffer location and reports it to the MLOps service.
DataRobot MLOps service If the agent is running as a service, it retrieves the data as soon as it’s available; otherwise, it retrieves prediction data when it is run manually.

If models are running in isolation and disconnected from the network, the MLOps library will not have networked access from the buffer directory. For these deployments, you can manually copy prediction data from the buffer location via USB drive as needed. The agent then accesses that data as configured and reports it to the MLOps service.

Additional agent configuration settings specify where to read data from and report data to, how frequently to report the data, and so forth. The flexible MLOps agent design ensures support of a variety of deployment and environment requirements.

Finally, from the deployment inventory you can view your deployments and view and manage prediction statistics and metrics.

MLOps agent tarball

You can download the MLOps agent tarball from two locations:

The MLOps agent tarball contains the MLOps libraries for you to install. See MLOps agent and prediction reporting setup to configure the agent.

Note

You can also download the MLOps Python libraries from the public Python Package Index site. Download and install the DataRobot MLOps metrics reporting library and the DataRobot MLOps Connected Client. These pages include instructions for installing the libraries.

In addition to the MLOps library, the tarball includes Python and Java API examples and accompanying datasets to:

  • Create a deployment that generates (example) predictions for both regression and classification models.
  • Report metrics from deployments using the MLOps library.

The tarball also includes scripts to:

  • Start and stop the agent, as well as retrieve current agent status.
  • Create a remote deployment that uploads a training dataset and returns the deployment ID and model ID for the deployment.

How the agent works

This section outlines the basic workflow for using the MLOps agent from different environments.

Using DataRobot MLOps:

  1. Use the Model Registry to create a model package with information about your model's metadata.
  2. Deploy the model package. Create a deployment to display metrics about the running model.
  3. Use the deployment Predictions tab to view a code snippet demonstrating how to instrument your prediction code with the agent to report metrics.

Using a remote deployment environment:

  1. Install the DataRobot MLOps agent.
  2. Use the MLOps library to report metrics from your prediction code as demonstrated by the snippet. The MLOps library buffers the metrics in a spooler (i.e. Filesystem, Rabbit MQ, Kafka, among others), which enables high throughput without slowing down the deployment.
  3. The MLOps agent forwards the metrics to DataRobot MLOps.
  4. You can view the reported metrics via the DataRobot MLOps Deployment inventory.


Updated January 7, 2022
Back to top