MLOps library and agent spooler configuration¶

The MLOps library communicates to the agent through a spooler, so it is important that the agent and library spooler configurations match. When configuring the MLOps agent and library's spooler settings, some settings are required, and some are optional (optional settings are identified in each table under Optional configuration). The required settings can be configured programmatically or through the environment variables documented in the General configuration and Spooler-specific configurations sections. If you configure any settings programmatically and by defining an environment variable, the environment variable takes precedence.

MLOps agent and library communication can be configured to use any of the following spoolers:

• Filesystem

• Amazon SQS

• RabbitMQ

• Google Cloud Pub/Sub

• Apache Kafka

• Azure Event Hubs

• DataRobot API

MLOps agent configuration¶

When running the monitoring agent as a separate service, specify the spooler configuration in mlops.agent.conf.yaml by uncommenting the channelConfigs section and entering the required configs. For more information on setting the channelConfigs see Configure the monitoring agent.

MLOps library configuration¶

The MLOps library can be configured programmatically or by using environment variables. To configure the spooler programmatically, specify the spooler during the MLOps init call; for example, to configure the filesystem spooler using the Python library:

mlops = MLOps().set_filesystem_spooler("your_spooler_directory").init()

Equivalent interfaces exist for other spooler types.

To configure the MLOps library and agent using environment variables, see the General configuration and Spooler-specific configurations sections.

General configuration¶

Use the following environment variables to configure the MLOps agent and library and to select a spooler type:

After setting a spooler type, you can configure the spooler-specific environment variables.

Spooler-specific configurations¶

Depending on the MLOPS_SPOOLER_TYPE you set, you can provide configuration information as environment variables unique to the supported spoolers.

Filesystem¶

Use the following environment variable to configure the FILESYSTEM spooler:

1
2
3
4
5
6

docker run \
-p 8080:8080 \
-v <local path to model package folder>/:/tmp/ \
-e MLOPS_SPOOLER_TYPE="FILESYSTEM" \
-e MLOPS_FILESYSTEM_DIRECTORY="/tmp/" \
datarobot-portable-prediction-api

Amazon SQS¶

When using Amazon SQS as a spooler, you can provide your credential set in either of two ways:

• Set your credentials in the AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_REGION or AWS_DEFAULT_REGION environment variables. Only AWS software packages use these credentials; DataRobot doesn't access them.

• If you are in an AWS environment, create an AWS IAM (Identity and Access Management) role for credential authentication.

Use one of the following environment variables to configure the SQS spooler:

RabbitMQ¶

Use the following environment variables to configure the RABBITMQ spooler:

Google Cloud Pub/Sub¶

When using Google Cloud PUBSUB as a spooler, you must provide the appropriate credentials in the GOOGLE_APPLICATION_CREDENTIALS environment variable. Only Google Cloud software packages use these credentials; DataRobot doesn't access them.

Use the following environment variables to configure the PUBSUB spooler:

Apache Kafka¶

Use the following environment variables to configure the Apache KAFKA spooler:

DataRobot API¶

The process to configure the DataRobot API spooler is different than typical spooler configuration. Usually, the monitoring agent connects to the spooler, gathers information, and sends that information to DataRobot MLOps. Using the DataRobot API, you do not actually connect to a spooler, and the calls you make to the MLOps library are unchanged. The calls do not go to a spooler or the monitoring agent, and instead go directly to DataRobot MLOps via HTTPS. In this case, you do not need to configure a complex spooler and monitoring agent.

Use the following parameters to configure the DataRobot Python API spooler:

Azure Event Hubs¶

DataRobot allows you to use Microsoft Azure Event Hubs as a monitoring agent spooler by leveraging the existing Kafka spooler type. To set this up, see Using Azure Event Hubs from Apache Kafka applications.

To use Azure Event Hubs as a spooler, you need to set up authentication for the monitoring agent and MLOps library using one of these methods:

• SAS-based authentication

• Azure Active Directory OAuth 2.0

SAS-based authentication for Event Hubs¶

To use Event Hubs SAS-based authentication for the monitoring agent and MLOps library, set the following environment variables using the example shell fragment below:

# Azure recommends setting the following values; see:
# https://docs.microsoft.com/en-us/azure/event-hubs/apache-kafka-configurations
export MLOPS_KAFKA_REQUEST_TIMEOUT_MS='60000'
export MLOPS_KAFKA_SESSION_TIMEOUT_MS='30000'
export MLOPS_KAFKA_METADATA_MAX_AGE_MS='180000'

# Common configuration variables for both Java- and Python-based libraries.
export MLOPS_KAFKA_BOOTSTRAP_SERVERS='XXXX.servicebus.windows.net:9093'
export MLOPS_KAFKA_SECURITY_PROTOCOL='SASL_SSL'
export MLOPS_KAFKA_SASL_MECHANISM='PLAIN'

# The following setting is specific to the Java SDK (and the monitoring agent daemon)
export MLOPS_KAFKA_SASL_JAAS_CONFIG='org.apache.kafka.common.security.plain.PlainLoginModule required username="$ConnectionString" password="Endpoint=sb://XXXX.servicebus.windows.net/;SharedAccessKeyName=XXXX;SharedAccessKey=XXXX";'

# For the Python SDK, you will need the following settings (in addition to the common ones above)
export MLOPS_KAFKA_SASL_USERNAME='$ConnectionString'
export MLOPS_KAFKA_SASL_PASSWORD='Endpoint=sb://XXXX.servicebus.windows.net/;SharedAccessKeyName=XXX;SharedAccessKey=XXXX'

Azure Active Directory OAuth 2.0 for Event Hubs¶

DataRobot supports Azure Active Directory OAuth 2.0 for Event Hubs authentication. To use this authentication method, you must create a new Application Registration with the necessary permissions over your Event Hubs Namespace (i.e., Azure Event Hubs Data Owner). See Authenticate an application with Azure AD to access Event Hubs resources for details.

To use Event Hubs Azure Active Directory OAuth 2.0 authentication, set the following environment variables using the example shell fragment below:

# Azure recommends setting the following values; see:
# https://docs.microsoft.com/en-us/azure/event-hubs/apache-kafka-configurations
export MLOPS_KAFKA_REQUEST_TIMEOUT_MS='60000'
export MLOPS_KAFKA_SESSION_TIMEOUT_MS='30000'
export MLOPS_KAFKA_METADATA_MAX_AGE_MS='180000'

# Common configuration variables for both Java- and Python-based libraries.
export MLOPS_KAFKA_BOOTSTRAP_SERVERS='XXXX.servicebus.windows.net:9093'
export MLOPS_KAFKA_SECURITY_PROTOCOL='SASL_SSL'
export MLOPS_KAFKA_SASL_MECHANISM='OAUTHBEARER'

# The following setting is specific to the Java SDK (and the tracking-agent daemon)
export MLOPS_KAFKA_SASL_JAAS_CONFIG='org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required aad.tenant.id="XXXX" aad.client.id="XXXX" aad.client.secret="XXXX";'
export MLOPS_KAFKA_SASL_LOGIN_CALLBACK_CLASS='com.datarobot.mlops.spooler.kafka.ActiveDirectoryAuthenticateCallbackHandler'

# For the Python SDK, you will need the following settings (in addition to the common ones above)
export MLOPS_KAFKA_SASL_OAUTHBEARER_CONFIG='aad.tenant.id=XXXX-XXXX-XXXX-XXXX-XXXX, aad.client.id=XXXX-XXXX-XXXX-XXXX-XXXX, aad.client.secret=XXXX'

Dynamically load required spoolers in a Java application¶

To configure Monitoring Agent spoolers using third-party code, you can dynamically load a separate JAR file for the required spooler. This configuration is required for the Amazon SQS, RabbitMQ, Google Cloud Pub/Sub, and Apache Kafka spoolers. The natively supported file system spooler is configurable without loading a JAR file.

Include spooler dependencies in the project object model¶

To use a third-party spooler in your MLOps Java application, you must include the required spoolers as dependencies in your POM (Project Object Model) file, along with datarobot-mlops:

<properties>
    <mlops.version>8.3.0</mlops.version>
</properties>

    <dependency>
        <groupId>com.datarobot</groupId>
        <artifactId>datarobot-mlops</artifactId>
        <version>${mlops.version}</version>
    </dependency>
    <dependency>
        <groupId>com.datarobot</groupId>
        <artifactId>spooler-sqs</artifactId>
        <version>${mlops.version}</version>
    </dependency>
    <dependency>
        <groupId>com.datarobot</groupId>
        <artifactId>spooler-rabbitmq</artifactId>
        <version>${mlops.version}</version>
    </dependency>
    <dependency>
        <groupId>com.datarobot</groupId>
        <artifactId>spooler-pubsub</artifactId>
        <version>${mlops.version}</version>
    </dependency>
    <dependency>
        <groupId>com.datarobot</groupId>
        <artifactId>spooler-kafka</artifactId>
        <version>${mlops.version}</version>
    </dependency>

Provide an executable JAR file for the spooler¶

The spooler JAR files are included in the MLOps agent tarball. They are also available individually as downloadable JAR files in the public Maven repository for the DataRobot MLOps Agent.

To use a third-party spooler with the executable agent JAR file, add the path to the spooler to the classpath:

java ... -cp path/to/mlops-agent-8.2.0.jar com.datarobot.mlops.agent.Agent

java ... -cp path/to/mlops-agent-8.3.0.jar:path/to/spooler-kafka-8.3.0.jar com.datarobot.mlops.agent.Agent

The start-agent.sh script provided as an example automatically performs this task, adding any spooler JAR files found in the lib directory to the classpath. If your spooler JAR files are in a different directory, set the MLOPS_SPOOLER_JAR_PATH environment variable.