How-to: Anti-money laundering (AML) alert scoring¶

This walkthrough provides step-by-step instructions for building a model that uses historical data, including customer and transactional information, to identify which alerts resulted in a Suspicious Activity Report (SAR). The model can then be used to assign a suspicious activity score to future alerts and improve the efficiency of an AML compliance program using rank ordering by score.

For a deeper dive into the use case, see the Anti-Money Laundering (AML) Alert Scoring guide and review it alongside working through this walkthrough.

Assets for download¶

To follow this walkthrough, download the dataset that will be used to train and evaluate the model below. The dataset contains a sample of alerts from a financial institution.

Download dataset

Building a model¶

Now that the data has been uploaded and analyzed, it is time to build a model.

1. Click Data actions > Start modeling.

   

2. In the Set up new experiment window, specify SAR in the Target feature field.

   

3. Leave the remaining fields at their defaults and click Next.

   

   Note

   For more details on the additional settings, see Start modeling setup.

4. Leave all partitioning changes fields at their defaults and click Start modeling.

5. DataRobot begins building the models.

   

6. After a few moments, the Model Leaderboard appears and indicates the training progress.

   

   Model build time

   Model build time can vary depending on the size of the dataset. When it completes, the Workers pane displays No jobs currently running.

For details on how to assess the various models after they are built, see Compare models.

Model evaluation and interpretation¶

Now that a set of models are ready for analysis, select the top model and explore its details. DataRobot flags the most accurate model as Prepared for deployment in the Model Leaderboard.

Click the model to view more detailed information about it. Use the tabs in the Details pane to explore various insights, as highlighted below.

DataRobot provides a variety of tools that can be used to assess why a particular alert was flagged as suspicious. Review the following sections to understand how to use the most relevant ones for this use case.

Blueprint¶

From the model details page, click Details > Blueprint to reveal the model blueprint—the pipeline of preprocessing steps, modeling algorithms, and post-processing steps used to create the model.

Feature Impact¶

Click Explanations > Feature Impact and then Compute to see the association between each feature in the dataset and the target.

DataRobot identifies the top three most impactful features (which enable the machine to differentiate SAR from non-SAR alerts) as total merchant credit in the last 90 days, number refund requests by the customer in the last 90 days, and total refund amount in the last 90 days.

Feature Effects¶

To understand the direction of impact and the SAR risk at different levels of the input feature, DataRobot provides partial dependence graphs on the Feature Effects tab. Click Explanations > Feature Effects and then Compute to depict how the likelihood of being a SAR changes when the input feature takes different values.

In this example, the total merchant credit amount in the last 90 days (on the x-axis) is the most impactful feature, but the SAR risk (on the y-axis) is not linearly increasing when the amount increases. The chart shows that the SAR risk remains relatively low when the amount is below $1000, surges significantly when the amount is above $1000, and then slows when the amount approaches $1500.

Individual Prediction Explanations¶

To view a breakdown of how each feature in the dataset contributes to the overall prediction outcome, click Explanations > Individual Prediction Explanations. This tab shows the explanations for each alert scored and prioritized by the machine learning model.

The image above demonstrates a sample of five random predictions from the model. You can click Predictions to sample below the chart to adjust the total number of predictions to be used to generate the chart.

Click on a specific prediction in the predictions list to see which features contributed to the prediction. In the example below, the prediction with ID=1789 has a very high likelihood of being a suspicious activity (prediction=91.3%) based on the total merchant credit amount in the last 90 days.

Word Cloud¶

Click Explanations > Word Cloud to explore how text fields affect predictions. The Word Cloud uses a color spectrum to indicate the word's impact on the prediction. In this example, red words indicate the alert is more likely to be associated with a SAR. Larger words indicate words that occur more frequently in the dataset.

Click a word in the cloud to see the details about that particular word.

Evaluate accuracy¶

In addition to providing insights into the model's performance, DataRobot provides tools to evaluate the model's accuracy. This section describes several of the most relevant tools to this use case.

Lift Chart¶

Click Performance > Lift Chart to view how effective the model is at separating the SAR and non-SAR alerts. After an alert in the out-of-sample partition gets scored by the model, it is assigned a risk score that measures the likelihood of the alert being a SAR risk or becoming a SAR. In the Lift Chart, alerts are sorted based on the SAR risk, broken down into 10 deciles, and displayed from lowest to the highest.

For each decile, DataRobot computes the average predicted SAR risk (as indicated by the blue plus signs) as well as the average actual SAR event (as indicated by the orange circle). It then connects the respective dots into two distinct line graphs for the predicted and actual SAR risk. The chart shows that the model has slight under-predictions for the alerts that are more likely to be a SAR, but overall, the model performs well.

ROC Curve¶

Now that you know the model is performing well, you can select an explicit threshold to make a binary decision based on the continuous SAR risk predicted by DataRobot. Click Performance > ROC Curve to access a variety of information to help make some of the important decisions in selecting the optimal threshold:

For additional information on the ROC Curve, see ROC Curve.

Next steps¶

Now that you have completed the basic process of building a model to analyze and make predictions on SAR alerts, you can move onto more detailed steps regarding this use case in the Anti-Money Laundering (AML) Alert Scoring guide.