SHAP reference¶

SHAP (SHapley Additive exPlanations) is an open-source algorithm used to address the accuracy vs. explainability dilemma. SHAP is based on Shapley Values, the coalitional game theory framework by Lloyd Shapley, Nobel Prize-winning economist. The values are a unified measure of feature importance and are used to interpret predictions from machine learning models. SHAP values not only tell us about the importance of a feature, but also about the direction of the relationship (positive or negative).

The SHAP values of a feature sum up to the difference between the prediction for the instance and the expected prediction (averaged across the dataset). If we consider a model that makes predictions based on several input features, the SHAP value for each feature would represent the average marginal contribution of that feature across all possible feature combinations.

To understand the origination of the project:

Lloyd Shapley asked: How should we divide a payout among a cooperating team whose members made different contributions?

Shapley values answers:

• The Shapley value for member X is the amount of credit they get.
• For every subteam, how much marginal value does member X add when they join the subteam? Shapley value is the weighted mean of this marginal value.
• Total payout is the sum of Shapley values over members.

Scott Lundberg is the primary author of the SHAP Python package, providing a programmatic way to explain predictions:

We can divide credit for model predictions among features!

By assuming that each value of a feature is a “player” in a game, the prediction is the payout. SHAP explains how to fairly distribute the “payout” among features.

SHAP has become increasingly popular due to the SHAP open source package that developed:

• A high-speed exact algorithm for tree ensemble methods (called "TreeExplainer").
• A high-speed approximation algorithm for deep learning models (called "DeepExplainer").
• Several model-agnostic algorithms to estimate Shapley values for any model (including "KernelExplainer" and "PermutationExplainer").

The following key properties of SHAP make it particularly suitable for DataRobot machine learning:

• Local accuracy: The sum of the feature attributions is equal to the output of the model DataRobot is "explaining."
• Missingness: Features that are already missing have no impact.
• Consistency: Changing a model to make a feature more important to the model will never decrease the SHAP attribution assigned to that feature. (For example, model A uses feature X. You then make a new model, B, that uses feature X more heavily (perhaps by doubling the coefficient for that feature and keeping everything else the same). Because of the consistency quality of SHAP, the SHAP importance for feature X in model B is at least as high as it was for feature X in model A.)

Additional readings are listed below.

SHAP contributes to model explainability by:

• Feature Impact: SHAP shows, at a high level, which features are driving model decisions. Without SHAP, results are sensitive to sample size and can change when re-computed unless the sample is quite large. See the deep dive.

• Prediction explanations: There are certain types of data that don't lend themselves to producing results for all columns. This is especially problematic in regulated industries like banking and insurance. SHAP explanations reveal how much each feature is responsible for a given prediction being different from the average. For example, when a real estate record is predicted to sell for $X, SHAP prediction explanations illustrate how much each feature contributes to that price.

• Feature Effects:

  • For Workbench, Feature Effects always uses Permutation Feature Impact.
  • For DataRobot Classic, SHAP does not change the Feature Effects results. The Predicted, Actual, and Partial dependence plots do not use SHAP in any way. However, the bar chart on the left is ordered by SHAP Feature Impact instead of the usual Permutation Feature Impact.

Feature Impact¶

Feature Impact assigns importance to each feature (j) used by a model.

With SHAP¶

Given a model and some observations (up to 5000 rows in the training data), Feature Impact for each feature j is computed as:

     sample average of abs(shap_values for feature j)

Normalize values such that the top feature has an impact of 100%.

With permutation¶

Given a model and some observations (2500 by default and up to 100,000), calculate the metric for the model based on the actual data. For each column j:

• Permute the values of column j.
• Calculate metrics on permuted data.
• Importance = metric_actual - metric_perm

(Optional) Normalize by the largest resulting value.

Prediction explanations¶

SHAP prediction explanations are additive. The sum of SHAP values is exactly equal to:

    [prediction - average(training_predictions)]

DataRobot Classic only (Workbench is SHAP-only) When selecting between XEMP and SHAP, consider your need for accuracy versus interpretability and performance. With XEMP, because all blueprints are included in Autopilot, the results may produce slightly higher accuracy. This is only true in some cases, however, since SHAP supports all key blueprints, meaning that often the accuracy is the same. SHAP does provide higher interpretability and performance:

• Results are intuitive.
• SHAP is computed for all features.
• Results often return 5-20 times faster.
• SHAP is additive.
• The open source nature provides transparency.

Which explainer is used for which model?¶

Within a blueprint, each modeling vertex uses the SHAP explainer that is most optimized to the model type:

• Tree-based models (XGBoost, LightGBM, Random Forest, Decision Tree): TreeExplainer
• Keras deep learning models: DeepExplainer
• Linear models: LinearExplainer

If a blueprint contains more than one modeling task, SHAP values are combined additively to yield the SHAP values for the overall blueprint.

Some modeling tasks are not supported by any type-specific explainers. If a blueprint contains such tasks, the blueprint is explained as a whole unit by the model-agnostic PermutationExplainer.

Additivity in prediction explanations¶

In certain cases, you may notice that SHAP values do not add up to the prediction. The reason may be some or all of the following conditions:

• Some models use a link function to convert the direct output of a modeling task to a different scale. This is very common in binary classification models, which often use the logit link function, and it also happens for regression models with skewed targets, which use the log link function. If the SHAP PermutationExplainer is used, then the SHAP values are in the same units as the prediction—with the link transform already accounted for—and when summed together with the base value, they will reproduce the prediction. But if task-specific explainers are used, such as TreeExplainer or DeepExplainer, then the SHAP values are in the margin units before the link transform. To reproduce the prediction, you will need to sum the SHAP values together with the base value and then apply the inverse link transform as below.

  • For most binary classification problems, the SHAP values correspond to a scale that is different from the probability space [0,1]. This is due to the way that these algorithms map their direct output, y, to something always between 0 and 1, most commonly using a nonlinear function like the logistic function prob = logistic(y). (In technical terms, the model's "link function" is logit(p), which is the inverse of logistic(y). The model direct output, y, is in "log-odds" units and the final prediction prob is in probability units.) In this common situation, the SHAP values are additive in the pre-link "margin space", not in the final probability space. This means sum(shap_values) = logit(prob) - logit(prob_0), where prob_0 is the training average of the model's predictions.

  • Regression problems with a skewed target may use the natural logarithm log() as a link function in a similar way.

• You may have configured an offset, applied before the link, and/or an exposure, applied after the link. (See the Workbench configuration here and the Classic configuration here.) Offsets and exposures are not regular feature columns and do not get SHAP values; instead, they are applied to the model outputs as a post-processing step. SHAP values are computed as if all offsets are 0 and exposures are 1.

• The model may "cap" or "censor" its predictions (for example, enforcing them to be non-negative).

The following pseudocode can be used for verifying additivity in these cases.

# shap_values = output from SHAP prediction explanations
# If you obtained the base_value from the UI prediction distribution chart, first transform it by the link.
base_value = api_shap_base_value or link_function(ui_shap_base_value)
pred = base_value + sum(shap_values)
if offset is not None:
    pred += offset
if link_function == 'log':
    pred = exp(pred)
elif link_function == 'logit:
    pred = exp(pred) / (1 + exp(pred))
if exposure is not None:
    pred *= exposure
pred = predictions_capping(pred)
# at this point, pred matches the prediction output from the model

if link_function == 'log':
   pred = (base_value + sum(shap_values)) * exposure * exp(offset)
elif link_function == 'logit':
    # here expit(x) = exp(x) / (1 + exp(x))
    pred = expit(offset + logit(base_value + sum(shap_values)))
else:
    pred = offset + base_value + sum(shap_values)

How SHAP values are calculated for time-aware¶

In time-aware experiments, SHAP explanations and impact are available when using Workbench (or via the API); calculations are based on input features, not the derived features that result from running time-aware modeling. The calculation method for this data is different than that used for predictive experiments. The sections below provide an overview.

Generally, the way SHAP works is to take one row of the input data at a time and make a prediction, shuffle values, make another prediction, and so on, making a grid of results. Every time DataRobot steps to a row, it performs permutations on it and then moves to the next. Then, it centralizes all the results and uses them to calculate the SHAP values. However, because time-aware experiments do not make row-by-row predictions, this calculation method does not work. Time-aware is not designed to handle permutations of the same date and series, nor can it have duplicate series. With time-aware, you have a flow of dates across series and dates.

To provide SHAP values for time-aware, DataRobot runs SHAP twice. First, it runs the data through to collect a set of prediction permutations SHAP might try. Then, instead of processing them individually, it stores them as parallel prediction caches. Each cache maintains chronological integrity and no single date exists more than once per cache. After predictions, results are reshuffled to present back to SHAP in the expected format. This creates the illusion for SHAP that it processed individual permutations, when in reality the methodology preserved time continuity.

For example, for a dataset with 10 features, SHAP might want to create 10 variations of each row with different feature values included or excluded. If you have 10 features, DataRobot is going to make 10 copies of a column or 10 copies of a row, each with different settings. The results are then stored as 10 parallel caches. DataRobot predicts each cache independently and then reshuffles the data—SHAP "thinks" it has created 10 unique things, predicted them, and they returned with values.

The number of permutations run by SHAP, and therefore the number of parallel caches, is number of features x 2 - 1. These are stored in parallel so that each cache basically has each permutation (e.g., January 2nd, 3rd, etc.,) but does not have a single date existing more than once per cache. This feature of SHAP compatibility prevents duplicate time steps from colliding with one another, as that would break the time series model.

In predictive SHAP, values remain consistent regardless of row ordering—you can shuffle your data, calculate SHAP values, and then rearrange them back to their original order without affecting the results. Each row's SHAP values depend solely on its feature values. With time-aware SHAP, by contrast, values are position-dependent. Even if two data points have identical feature values (excluding the date itself), they will receive different SHAP values when positioned at different points in the time sequence. Moving data points earlier or later in the timeline changes their SHAP values because time series models inherently account for temporal relationships and patterns. This temporal dependency is an important consideration when interpreting time-aware SHAP values. The position of a data point within its chronological context directly impacts its importance and contribution to predictions.

Additional reading¶

The following public information provides additional information on open-source SHAP:

• SHAP for explainable machine learning

• SHAP (SHapley Additive exPlanations)

• Explain Your Model with the SHAP Values

• A Unified Approach to Interpreting Model Predictions

• SHAP GitHub package