End-to-end ML workflow with Databricks¶

Access this AI accelerator on GitHub

DataRobot features an in-depth API that allows data scientists to produce fully automated workflows in their coding environment of choice. This accelerator shows how to pair the power of DataRobot with the Spark-backed notebook environment provided by Databricks.

In this notebook you'll see how data acquired and prepared in a Databricks notebook can be used to train a collection of models on DataRobot. You'll then deploy a recommended model and use DataRobot's exportable Scoring Code to generate predictions on the Databricks Spark cluster.

This accelerator notebook covers the following activities:

• Acquiring a training dataset.
• Building a new DataRobot project.
• Deploying a recommended model.
• Scoring via Spark using DataRobot's exportable Java Scoring Code.
• Scoring via DataRobot's Prediction API.
• Reporting monitoring data to the MLOps agent framework in DataRobot.
• Writing results back to a new table.