Python modeling workflow overview¶
This code example outlines how to use DataRobot's Python API client to train and experiment with models. It also offers ideas for integrating DataRobot with other products via the API.
Specifically, you will:
- Create a project and run Autopilot.
- Experiment with feature lists, modeling algorithms, and hyperparameters.
- Choose the best model.
- Perform an in-depth evaluation of the selected model.
- Deploy a model into production in a few lines of code.
Data used for this example¶
This walkthrough uses a synthetic dataset that illustrates a credit card company’s anti-money laundering (AML) compliance program, with the intent of detecting the following money-laundering scenarios:
- A customer spends on the card, but overpays their credit card bill and seeks a cash refund for the difference.
- A customer receives credits from a merchant without offsetting transactions, and either spends the money or requests a cash refund from the bank.
A rule-based engine is in place to produce an alert when it detects potentially suspicious activity consistent with the scenarios above. The engine triggers an alert whenever a customer requests a refund of any amount. Small refund requests are included because they could be a money launderer’s way of testing the refund mechanism or trying to establish refund requests as a normal pattern for their account.
The target feature is SAR, suspicious activity reports. It indicates whether or not the alert resulted in an SAR after manual review by investigators, which means that this project is a binary classification problem. The unit of analysis is an individual alert, so the model will be built on the alert level. Each alert will get a score ranging from 0 to 1, indicating the probability of being an alert leading to an SAR. The data consists of a mixture of numeric, categorical, and text data.
Setup¶
Import Libraries¶
In [31]:
Copied!
import datarobot as dr
from datarobot_bp_workshop import Visualize, Workshop
import matplotlib.pyplot as plt
import pandas as pd
%matplotlib inline
import time
import warnings
import graphviz
import plotly.express as px
import seaborn as sns
warnings.filterwarnings("ignore")
w = Workshop()
# wider .head()s
pd.options.display.width = 0
pd.options.display.max_columns = 200
pd.options.display.max_rows = 2000
sns.set_theme(style="darkgrid")
import datarobot as dr
from datarobot_bp_workshop import Visualize, Workshop
import matplotlib.pyplot as plt
import pandas as pd
%matplotlib inline
import time
import warnings
import graphviz
import plotly.express as px
import seaborn as sns
warnings.filterwarnings("ignore")
w = Workshop()
# wider .head()s
pd.options.display.width = 0
pd.options.display.max_columns = 200
pd.options.display.max_rows = 2000
sns.set_theme(style="darkgrid")
Connect to DataRobot¶
Read more about different options for connecting to DataRobot from the client.
In [ ]:
Copied!
# If the config file is not in the default location described in the API Quickstart guide, '~/.config/datarobot/drconfig.yaml', then you will need to call
# dr.Client(config_path='path-to-drconfig.yaml')
# If the config file is not in the default location described in the API Quickstart guide, '~/.config/datarobot/drconfig.yaml', then you will need to call
# dr.Client(config_path='path-to-drconfig.yaml')
Select a training dataset¶
In [ ]:
Copied!
# To read from a local file, uncomment and use:
# df = pd.read_csv('./data/DR_Demo_AML_Alert.csv')
# To read from an s3 bucket:
df = pd.read_csv("https://s3.amazonaws.com/datarobot_public_datasets/DR_Demo_AML_Alert.csv")
df.head()
# To read from a local file, uncomment and use:
# df = pd.read_csv('./data/DR_Demo_AML_Alert.csv')
# To read from an s3 bucket:
df = pd.read_csv("https://s3.amazonaws.com/datarobot_public_datasets/DR_Demo_AML_Alert.csv")
df.head()
In [20]:
Copied!
# To view target distribution:
df_target_summary = (
pd.DataFrame(df["SAR"].value_counts())
.reset_index()
.rename(columns={"index": "SAR", "SAR": "Count"})
)
ax = sns.barplot(x="SAR", y="Count", data=df_target_summary)
for index, row in df_target_summary.iterrows():
ax.text(row.SAR, row.Count, round(row.Count, 2), color="black", ha="center")
plt.show()
# To view target distribution:
df_target_summary = (
pd.DataFrame(df["SAR"].value_counts())
.reset_index()
.rename(columns={"index": "SAR", "SAR": "Count"})
)
ax = sns.barplot(x="SAR", y="Count", data=df_target_summary)
for index, row in df_target_summary.iterrows():
ax.text(row.SAR, row.Count, round(row.Count, 2), color="black", ha="center")
plt.show()
Create a project and train models with Autopilot¶
In [ ]:
Copied!
# Create a project by uploading data. This will take a few minutes.
project = dr.Project.create(
sourcedata=df,
project_name="DR_Demo_API_alert_AML_{}".format(pd.datetime.now().strftime("%Y-%m-%d %H:%M")),
)
# Set the project's target and initiate Autopilot in Quick mode.
# Wait for Autopilot to finish. You can set verbosity to 0 if you do not wish to see progress updates.
project.analyze_and_model(target="SAR", worker_count=-1)
# Open the project's Leaderboard to monitor the progress in UI.
project.open_in_browser()
# Create a project by uploading data. This will take a few minutes.
project = dr.Project.create(
sourcedata=df,
project_name="DR_Demo_API_alert_AML_{}".format(pd.datetime.now().strftime("%Y-%m-%d %H:%M")),
)
# Set the project's target and initiate Autopilot in Quick mode.
# Wait for Autopilot to finish. You can set verbosity to 0 if you do not wish to see progress updates.
project.analyze_and_model(target="SAR", worker_count=-1)
# Open the project's Leaderboard to monitor the progress in UI.
project.open_in_browser()
Retrieve and review results from the Leaderboard¶
In [33]:
Copied!
def get_top_of_leaderboard(project, verbose=True):
# A helper method to assemble a dataframe with Leaderboard results and print a summary:
leaderboard = []
for m in project.get_models():
leaderboard.append(
[
m.blueprint_id,
m.featurelist.id,
m.id,
m.model_type,
m.sample_pct,
m.metrics["AUC"]["validation"],
m.metrics["AUC"]["crossValidation"],
]
)
leaderboard_df = pd.DataFrame(
columns=[
"bp_id",
"featurelist",
"model_id",
"model",
"pct",
"validation",
"cross_validation",
],
data=leaderboard,
)
if verbose == True:
# Print a Leaderboard summary:
print("Unique blueprints tested: " + str(len(leaderboard_df["bp_id"].unique())))
print("Feature lists tested: " + str(len(leaderboard_df["featurelist"].unique())))
print("Models trained: " + str(len(leaderboard_df)))
print("Blueprints in the project repository: " + str(len(project.get_blueprints())))
# Print the essential information for top models, sorted by accuracy from validation data:
print("\n\nTop models on the Leaderboard:")
leaderboard_top = (
leaderboard_df[leaderboard_df["pct"] == 64]
.sort_values(by="cross_validation", ascending=False)
.head()
.reset_index(drop=True)
)
display(leaderboard_top.drop(columns=["bp_id", "featurelist"], inplace=False))
# Show blueprints of top models:
for index, m in leaderboard_top.iterrows():
Visualize.show_dr_blueprint(dr.Blueprint.get(project.id, m["bp_id"]))
return leaderboard_top
leaderboard_top = get_top_of_leaderboard(project)
def get_top_of_leaderboard(project, verbose=True):
# A helper method to assemble a dataframe with Leaderboard results and print a summary:
leaderboard = []
for m in project.get_models():
leaderboard.append(
[
m.blueprint_id,
m.featurelist.id,
m.id,
m.model_type,
m.sample_pct,
m.metrics["AUC"]["validation"],
m.metrics["AUC"]["crossValidation"],
]
)
leaderboard_df = pd.DataFrame(
columns=[
"bp_id",
"featurelist",
"model_id",
"model",
"pct",
"validation",
"cross_validation",
],
data=leaderboard,
)
if verbose == True:
# Print a Leaderboard summary:
print("Unique blueprints tested: " + str(len(leaderboard_df["bp_id"].unique())))
print("Feature lists tested: " + str(len(leaderboard_df["featurelist"].unique())))
print("Models trained: " + str(len(leaderboard_df)))
print("Blueprints in the project repository: " + str(len(project.get_blueprints())))
# Print the essential information for top models, sorted by accuracy from validation data:
print("\n\nTop models on the Leaderboard:")
leaderboard_top = (
leaderboard_df[leaderboard_df["pct"] == 64]
.sort_values(by="cross_validation", ascending=False)
.head()
.reset_index(drop=True)
)
display(leaderboard_top.drop(columns=["bp_id", "featurelist"], inplace=False))
# Show blueprints of top models:
for index, m in leaderboard_top.iterrows():
Visualize.show_dr_blueprint(dr.Blueprint.get(project.id, m["bp_id"]))
return leaderboard_top
leaderboard_top = get_top_of_leaderboard(project)
Unique blueprints tested: 15 Feature lists tested: 4 Models trained: 28 Blueprints in the project repository: 81 Top models on the Leaderboard:
| model_id | model | pct | validation | cross_validation | |
|---|---|---|---|---|---|
| 0 | 61ae672ab9e0c15c325b05b2 | RandomForest Classifier (Gini) | 64.0 | 0.94577 | 0.945790 |
| 1 | 61ae73aa2a2a4649c04e2c73 | RandomForest Classifier (Gini) | 64.0 | 0.94598 | 0.945712 |
| 2 | 61ae69df4e24ec75154c24ee | AVG Blender | 64.0 | 0.94573 | 0.945318 |
| 3 | 61ae65a640d62771a45b0594 | eXtreme Gradient Boosted Trees Classifier with... | 64.0 | 0.94675 | 0.945166 |
| 4 | 61ae65a540d62771a45b0592 | RandomForest Classifier (Gini) | 64.0 | 0.94542 | 0.944690 |
Experiment to get better results¶
When you run a project using Autopilot, DataRobot first creates blueprints based on the characteristics of your data and puts them in the Repository. Then, it chooses a subset from these to train; when training completes, these are the blueprints you’ll find on the Leaderboard. After the Leaderboard is populated, it can be useful to train some of those blueprints that DataRobot skipped. For example, you can try a more complex Keras blueprint like Keras Residual AutoInt Classifier using Training Schedule (3 Attention Layers with 2 Heads, 2 Layers: 100, 100 Units). In some cases, you may want to directly access the trained model through R and retrain it with a different feature list or tune its hyperparameters.
Find blueprints trained for the project from the Repository¶
In [ ]:
Copied!
blueprints = project.get_blueprints()
# After retrieving the blueprints, you can search for a specific blueprint
# In the example below, search for all models that have "Gradient" in their name
models_to_run = []
for blueprint in blueprints:
if "Gradient" in blueprint.model_type:
models_to_run.append(blueprint)
blueprints = project.get_blueprints()
# After retrieving the blueprints, you can search for a specific blueprint
# In the example below, search for all models that have "Gradient" in their name
models_to_run = []
for blueprint in blueprints:
if "Gradient" in blueprint.model_type:
models_to_run.append(blueprint)
In [ ]:
Copied!