Claim triage notebook¶

Claim payments and claim adjustments are typically an insurance company’s largest expenses. For long-tail lines of business, such as workers’ compensation (which covers medical expenses and lost wages for injured workers), claims can take years to be paid in full; therefore, the true cost of a claim may not be known for many years. However, claim adjustment activities start when a claim is made aware to the insurer.

This notebook aims to outline a workflow for evaluating the severity of an insurance claim in order to triage it effectively.

Download the sample training data here.

Setup¶

Import libraries¶

Connect to DataRobot¶

Read more about different options for connecting to DataRobot from the client.

Import data¶

Check the distribution of the target feature¶

The histogram below shows the distribution of the target, Incurred. You can observe that most claims are closed with a small payment; the median is less than $500, and the 75th percentile is $1820. This right-skewed distribution represents a typical distribution of insurance claims. This is often modeled by Gamma distribution. Autopilot automatically detects the target's distribution and recommends the appropriate optimization metric accordingly.

In this example, DataRobot recommends Gamma Deviance, which is used for modeling in this use case. Besides Gamma Deviance, DataRobot also includes a list of metrics that you can select based on your own needs.

Feature engineering¶

Feature engineering has proven to improve model performance in many use cases, although it is an optional step for this workflow. This provides an opportunity to incorporate domain knowledge into the model. Feature engineering can be as simple as extracting the month from a date so that a seasonal trend can be captured. Alternatively, it can be as complicated as getting aggregated claim counts for the past x years. Therefore depending on the use case, feature engineering can be very time-consuming. For illustration purposes, feature engineering is already performed on the example dataset.

For example, ReportingDelay = ReportDate - AccidentDate. In Python, the following code can be used to achieve this:

df["ReportingDelay"] = df["ReportDate"] - df["AccidentDate"]

The result above assumes that ReportDate and AccidentDate are both recorded in the original dataset.

Modeling¶

Create a project¶

Upload the training data to DataRobot to create a project. In this example, upload the dataframe "df" created in previous steps to DataRobot.

Initiate Autopilot¶

The output for the following snippet displays the features included in the "Informative Features" feature list.

Interpret results¶

Once the data is uploaded to DataRobot, and a project has started, you can start exploring in the GUI. Use the URL below will lead you to the project.

Exploratory Data Analysis¶

Navigate to the Data tab to learn more about your data.

Click each feature to see information such as the summary statistics (min, max, mean, std) of numeric features or a histogram that represents the relationship of a feature with the target.

DataRobot automatically performs data quality checks. In this example, it has detected outliers for the target feature. Click Show Outliers to view them all (outliers are common in insurance claims data). To avoid bias introduced by the outlier, a common practice is to cap the target, such as capping it to the 95th percentile. This cap is especially important for linear models.

Feature Associations¶

Use the Feature Associations tab to visualize the correlations between each pair of the input features. For example, in the Feature Association plot below, the features DaysWorkedPerWeek and PartTimeFullTime (top-left corner) have strong associations and are therefore "clustered" together. Each color block in this matrix is a cluster.

Leaderboard evaluation¶

Click the Models tab to view models as they build.

Autopilot continues building models until it selects the model that best understands how to predict the target feature, SAR. This model is at the top of the Leaderboard, marked with the Recommended for Deployment badge.

Choosing the best model is a balance of accuracy, metric performance, and model simplicity. See the model recommendation process for more information.

Model Insights¶

Select a model on the Leaderboard to view specific model insights.

Lift Chart¶

The Lift Chart shows how effective the model is and how well the model fit is. To create a lift chart, first sort the predictions from low to high. Then, create a number of equal-sized buckets. For each bucket, the average predicted value and average actual value are calculated and plotted. In the example below, the blue curve represents the average predicted claim cost, and the orange curve indicates the average actual claim cost. The upward slope indicates the model has effectively differentiated the claims of low severity (close to 0) on the left and those of high severity (~45K) on the right. The fact that the actual values (orange curve) closely track the predicted values (blue curve) tells you that the model fits the data well.

Note that DataRobot only displays lift charts on validation or holdout partitions.

Feature Impact¶

Once a model is built, it is helpful to understand which features are driving the model. Feature impact ranks features based on feature importance, from the most important to the least important, and also shows the relative importance of those features. In the example below, you can see that InitialCaseEstimate is the most important feature for this model, followed by ClaimDescription, WeeklyRate, and Age.

This example indicates that features after MaritalStatus have little contribution to the model. For example, gender has minimal contribution to the model, indicating that claim severity doesn't vary by the gender of the claimant. If you create a new feature list that does not include gender (and other features less impactful than MaritalStatus) and only includes the most impactful features, the model accuracy should not be significantly impacted. A natural next step is to create a new feature list with only the top features and rerun the model. DataRobot automatically creates a new feature list, "DR Reduced Features", by including features that have a cumulative feature impact of 95%.

See the DataRobot documentation for more information on feature impact.

Feature effects¶

Feature effects indicate how each feature affects predictions. Click on a to see its partial dependence plot. In the example below, notice the partial dependence for the WeeklyRate feature. You can observe that claimants with lower weekly pay have lower claim severity, while claimants with higher weekly pay have higher claim severity.

See the DataRobot documentation for more information about feature effects.

Prediction Explanations¶

When a claims adjuster sees a low prediction for a claim, they are likely to initially ask what the drivers are behind such a low prediction. Insights provided at an individual prediction level can help claim adjusters understand how a prediction is made to increase confidence in the model. By default, DataRobot provides the top 3 Prediction Explanations for each prediction, but you can request up to 10 explanations. Model predictions and explanations can be downloaded in a CSV file. You can control which predictions are populated in the CSV by specifying the thresholds for high and low predictions. The graph below shows the top three explanations for the 3 highest and lowest predictions. The graph shows that, generally, high predictions are associated with older claimants and higher weekly salary, while the low predictions are associated with a lower weekly salary.

See the DataRobot documentation for more information about Prediction Explanations.

Word Cloud¶

ClaimDescription is an unstructured text field.

During Autopilot, DataRobot builds text mining models on textual features. The models' outputs are automatically used as inputs in the subsequent modeling processes. Below is a word cloud for the feature ClaimDescription, showing the keywords parsed out by DataRobot. The font size of the word indicates how frequently the word appears in the data. You can see that "strain" appears very often in the data while "fractured" does not appear as often. Word color indicates severity. Both "strain" and "fractured" (red) are associated with high-severity claims while "finger" and "eye" (blue) are associated with low-severity claims.

DataRobot can transform and model textual data in sophisticated ways. There are hundreds of blueprints with different methods to transform and manipulate text data. TF-IDF and other transformers are applied to raw text data and are sent to sophisticated models. DataRobot is also multilingual, using automatic language identification for text data and supporting different text mining algorithms, depending on the language it detects. The process of feature engineering free text data is notoriously complex and difficult, so data scientists often avoid doing it manually. DataRobot automatically finds, tunes, and interprets the best text mining algorithms for a dataset, saving both time and resources.