# Deploy and monitor Spark models with DataRobot MLOps
URL: https://docs.datarobot.com/en/docs/classic-ui/integrations/azure/spark-deploy-and-monitor.html

> Deploy and monitor Spark models with DataRobot MLOps

# Deploy and monitor Spark models with DataRobot MLOps

This page shows how to use DataRobot's Monitoring Agent (MLOps agent) to manage and monitor models from a central dashboard without deploying them within MLOps.

You will explore how to manage and monitor remote models—models that are not running within DataRobot MLOps—deployed on your own infrastructure. Common examples are serverless deployments (AWS Lambda, Azure Functions) or deployments on Spark clusters (Hadoop, Databricks, AWS EMR).

The sections below show how to take a DataRobot model to be deployed on a Databricks cluster and monitor this model with DataRobot MLOps in a central dashboard. This dashboard covers all of your models, regardless of where they were developed or deployed. This approach works for any model that runs within a Spark cluster.

## Create a model

In this section, you are creating a model with DataRobot AutoML, then importing it into your Databricks cluster, using the [Lending Club dataset](https://datarobot-doc-assets.s3.us-east-1.amazonaws.com/10K_Lending_Club_Loans.csv).

> [!NOTE] Note
> If you already have a regression model that runs in your Spark cluster, you can skip this step and proceed to [Install the MLOps monitoring agent and library](https://docs.datarobot.com/en/docs/classic-ui/integrations/azure/spark-deploy-and-monitor.html#install-the-mlops-monitoring-agent-and-library).

1. To upload the training data to DataRobot, do either of the following:
2. Enterloan_amtas your target (1) (what you want to predict) and clickStart(2) to run Autopilot.
3. After Autopilot finishes, clickModels(1) and select a model with theSCORING CODElabel (2) at the top of the Leaderboard (3).
4. Under the model you selected, clickPredict(1) and clickDownloads(2) to access theScoring Code JARdownload. NoteThe ability to download Scoring Code for a model from the Leaderboard depends on the MLOps configuration for your organization.
5. ClickDownload(3) to start the JAR file download.

For more information, see the documentation on [Scoring Code](https://docs.datarobot.com/en/docs/classic-ui/predictions/port-pred/scoring-code/index.html).

## Deploy a model

To install Scoring Code, you can install the previously downloaded JAR file, along with Spark Wrapper, on the Databricks cluster as shown below.

1. ClickClustersto open the cluster settings.
2. Select the cluster to which you'd like to deploy the DataRobot model.
3. Click theLibrariestab.
4. ClickInstall New.
5. In theInstall Librarydialog box, with theLibrary Sourceset toUploadand theLibrary Typeset toJAR, drag-and-drop the Scoring Code JAR file (e.g.,5ed68d70455df33366ce0508.jar),
6. ClickInstall.

Once the install is complete, repeat the same steps and install Spark Wrapper, which you can [download here](https://github.com/datarobot-community/ai_engineering/blob/master/mlops/DRMLOps_Spark_examples/scoring-code-spark-api.jar), or pull the latest version of it directly from [Maven](https://mvnrepository.com/artifact/com.datarobot/scoring-code-spark-api_2.4.3).

## Install the MLOps monitoring agent and library

Remote models do not directly communicate with DataRobot MLOps.
Instead, the communication is handled via DataRobot MLOps monitoring agents, which support many spooling mechanisms (e.g., flat files, AWS SQS, RabbitMQ).
These agents are typically deployed in the external environment where the model is running.

Libraries are available for all common programming languages to simplify communication with the DataRobot MLOps monitoring agent. The model is instructed to talk to the agent with the help of the MLOps library. The agent then collects all metrics from the model and relays them to the MLOps server and dashboards.

In this example, the runtime environment is Spark. Therefore, you will install the MLOps library to your Spark cluster (Databricks) in the same way you installed the model itself previously (in Deploy a Model). You will also install the MLOps monitoring agent in an Azure Kubernetes Service (AKS) cluster alongside RabbitMQ, which is used as your queuing system.

This process assumes that you are familiar with Azure Kubernetes Service and the Azure CLI. For more information, see [Microsoft's Quick Start Tutorial](https://docs.microsoft.com/en-us/azure/aks/kubernetes-walkthrough).

### Create an AKS cluster

1. If you don't have a running AKS cluster, create one, as shown below: RESOURCE_GROUP=ai_success_engCLUSTER_NAME=AIEngineeringDemo

azakscreate\--resource-group$RESOURCE_GROUP\--name$CLUSTER_NAME\-sStandard_B2s\--node-count1\--generate-ssh-keys\--service-principalXXXXXX\--client-secretXXXX\--enable-cluster-autoscaler\--min-count1\--max-count2
2. Start the Kubernetes dashboard: azaksbrowse--resource-group$RESOURCE_GROUP--name$CLUSTER_NAME​

### Install RabbitMQ

There are many ways to deploy applications. The most direct way is via the Kubernetes dashboard.

To install RabbitMQ:

1. ClickCREATE > CREATE AN APP(1).
2. On theCREATE AN APPpage (2), specify the following: FieldValue3App namee.g.,rabbitmqdemo4Container imagee.g.,rabbitmq:latest5Number of podse.g.,16ServiceExternal7PortandTarget port5672and567215672and15672
3. ClickDEPLOY(8).

### Download the MLOps monitoring agent

To download the MLOps monitoring agent directly from your DataRobot cluster:

1. In the upper-right corner of DataRobot, click your profile icon (or the default avatar).
2. ClickAPI keys and tools.
3. UnderExternal Monitoring Agent, click the download icon.

### Install the MLOps monitoring agent

You can install the agent anywhere; however, for this process, you will install it alongside RabbitMQ.

1. Copy the monitoring agent tarball you downloaded in the previous section to the container where RabbitMQ is running. To do this, run the following command: NoteYou may need to replace the filename of the tarball in the example below. kubectlcpdatarobot-mlops-agent-6.1.0.tar.gzdefault/rabbitmq-649ccbd8cb-qjb4l:/opt
2. ClickPods(1) and click the containerName(2) to connect to the CLI of the container.
3. ClickExec(3) to start the container.
4. In the container's CLI, begin to configure the agent. Review the tarball name and, if necessary, update the filename in the following commands, and then run them: cd/opt&&tar-xvzfmlops-agent-6.1.0.tar&&cdmlops-agent-6.1.0/conf
5. In the directory, update themlops.agent.conf.yamlconfiguration file to point to your DataRobot MLOps instance and message queue.
6. To update the configuration and run the agent, you must install Vim and Java with the following commands: apt-getupdate&&apt-getinstallvim&&apt-getinstalldefault-jdk
7. In this example, you are using RabbitMQ and the DataRobot managed AI Platform solution, so you must configure themlopsURLandapiToken(1) and thechannelConfigs(2) as shown below. NoteYou can obtain yourapiTokenfrom theDeveloper Tools.
8. Before starting the agent,  you can enable the RabbitMQ Management UI and create a new user to monitor queues: ### Enable the RabbitMQ UIrabbitmq-pluginsenablerabbitmq_management&&### Add a user via the CLIrabbitmqctladd_user<username><yourpassword>&&rabbitmqctlset_user_tags<username>administrator&&rabbitmqctlset_permissions-p/<username>".*"".*"".*"
9. Now that RabbitMQ is configured and the updated configuration is saved, switch to the/bindirectory and start the agent: cd../bin&&./start-agent.sh
10. Confirm that the agent is running correctly by checking its status: ./status-agent.sh
11. To ensure that everything is running as expected, check the logs located in the/logsdirectory.

### Install the MLOps library in a Spark cluster

First, [Download the library from here](https://github.com/datarobot-community/ai_engineering/blob/master/mlops/DRMLOps_Spark_examples/datarobot-mlops-6.2.0.jar).

To install the library in a Spark cluster (Databricks):

1. ClickClustersto open the cluster settings.
2. Select the cluster to which you'd like to deploy the DataRobot model.
3. Click theLibrariestab.
4. In theInstall Librarydialog box, with theLibrary Sourceset toUploadand theLibrary Typeset toJAR, drag-and-drop the MLOps JAR file (e.g.,MLOps.jar).
5. ClickInstall.

## Run your Model

Now that all the prerequisites are in place, run your model to make predictions:

```
// Scala example (see also PySpark example in notebook references at the bottom)

// 1) Use local DataRobot Model for Scoring
import com.datarobot.prediction.spark.Predictors

// referencing model_id, which is the same as the generated filename of the JAR file
val DataRobotModel = com.datarobot.prediction.spark.Predictors.getPredictor("5ed68d70455df33366ce0508")

// 2) read the scoring data
val scoringDF = sql("select * from 10k_lending_club_loans_with_id_csv")

// 3) Score the data and save results to spark dataframe
val output = DataRobotModel.transform(scoringDF)

// 4) Review/consume scoring results
output.show(1,false)
```

To track the actual scoring time, wrap the scoring command, so the updated code would look like the following:

```
// to track the actual scoring time

def time[A](f: => A): Double = {
  val s = System.nanoTime
  val ret = f
  val scoreTime = (System.nanoTime-s)/1e6 * 0.001
  println("time: "+ scoreTime+"s")
  return scoreTime
}

// 1) Use local DataRobot Model for Scoring
import com.datarobot.prediction.spark.Predictors

// referencing model_id, which is the same as the generated filename of the JAR file
val DataRobotModel = com.datarobot.prediction.spark.Predictors.getPredictor("5ed708a8fca6a1433abddbcb")

// 2) read the scoring data
val scoringDF = sql("select * from 10k_lending_club_loans_with_id_csv")

val scoreTime = time {

  // Score the data and save results to spark dataframe
  val scoring_output = DataRobotModel.transform(scoringDF)
  scoring_output.show(1,false)
  scoring_output.createOrReplaceTempView("scoring_output")
}
```

## Report usage to MLOps via monitoring agents

After using the model to predict the loan amount of an application, you can report the telemetrics around these predictions to your DataRobot MLOps server and dashboards. To do this, see the commands in the following sections.

### Create an external deployment

Before you can report scoring details, you must create an external deployment within DataRobot MLOps. This only has to be done once and can be done via the UI in DataRobot MLOps:

1. ClickModel Registry(1), clickModel Packages(2), and then clickNew external model package(3).
2. Specify a package name and description (1 and 2), upload the corresponding training data for drift tracking (3), and identify the model location (4), target (5), environment (6), and prediction type (7), then clickCreate package(8).
3. After creating the external model package, note the model ID in the URL, as shown below (blurred in the image for security purposes).
4. ClickDeployments(1) and clickCreate new deployment(2). Once the deployment is created, theDeployments > Overviewpage is shown.
5. On theOverviewpage, copy the deployment ID (from the URL).

Now that you have your model ID and deployment ID, you can report the predictions in the next section.

### Report prediction details

To report prediction details to DataRobot, run the following code in your Spark environment. Make sure you update the input parameters.

```
import com.datarobot.mlops.spark.MLOpsSparkUtils
val channelConfig = "OUTPUT_TYPE=RABBITMQ;RABBITMQ_URL=amqp://<<RABBIT HOSTNAME>>:5672;RABBITMQ_QUEUE_NAME=mlopsQueue"

MLOpsSparkUtils.reportPredictions(

                scoringDF, // spark dataframe with actual scoring data
                "5ec3313XXXXXXXXX", // external DeploymentId
                "5ec3313XXXXXXXXX", // external ModelId
                channelConfig, // rabbitMQ config
                scoringTime, // actual scoring time
                Array("PREDICTION"), //target column
                "id" // AssociationId
                )
```

### Report actuals

When you get actual values, you can report them to track accuracy over time.

Report actuals using the function below:

```
import com.datarobot.mlops.spark.MLOpsSparkUtils

val actualsDF = spark.sql("select id as associationId, loan_amnt as actualValue, null as timestamp from actuals")

MLOpsSparkUtils.reportActuals(
      actualsDF,
      deploymentId,
      ModelId,
      channelConfig
    )
```

Even though you deployed a model outside of DataRobot on a Spark cluster (Databricks), you can monitor it like any other model to track service health, data drift, and actuals in one central dashboard.

For complete sample notebooks with code snippets for Scala and PySpark, go to [the DataRobot Community GitHub](https://github.com/datarobot-community/ai_engineering/tree/master/mlops/DRMLOps_Spark_examples).

---
