XEMP qualitative strength¶
XEMP-based Prediction Explanations provide a visual indicator of the qualitative strength of each explanation presented by the insight. In the API, these values are returned from the qualitativeStrength response parameter of the Prediction Explanation API endpoint.
The distribution is approximated from the validation data; the preview is computed on the validation data.
Score translations¶
The boundaries between indicators (for example, +++, ++, and +) are different when there are different numbers of features in a model. The tables below describe, based on feature count, how the calculations translate to the visual representation. In the tables, q represents the "qualitative" (or "normalized") score.
Some notes:
-
If an explanation’s score is trivial and has little or no qualitative effect, the output displays three grayed out symbols (
+++or---). This indicates, for the represented directionality, that the effect is minor. -
When there are a large number of features, a normalized score greater than 0.2 is represented as
+++, so it is possible for multiple features to display this symbolic score in a single row.
How is the q value calculated?
DataRobot calculates the "qualitative score" or "q" value in the tables below from a normalization process. For the top 50 features, ranked by global Feature Impact score, explanation strengths' raw values are available. To calculate those values, DataRobot does the following on each row:
- Computes explanation strengths (sometimes called “raw scores”) for eligible features. (In the API, this value is returned as
strength.) - Computes a normalization factor. The value used for normalization is the sum of the top 10 largest absolute strengths. This value may differ between rows.
- Generates a normalized (qualitative or “q” score) by dividing all explanation strengths by the computed normalization factor. (This value is not available in the API.)
Use the tables below to convert the normalized scores to qualitative symbols.
Features = 1¶
The following describes the displayed symbolic score based on the calculated qualitative score for models built with a single feature:
| Qualitative Score | Symbolic Score |
|---|---|
| q <= -0.001 | --- |
| -0.001 < q <= 0 | grayed-out --- |
| 0 < q < 0.001 | grayed-out +++ |
| q >= 0.001 | +++ |
Features = 2¶
The following describes the displayed symbolic score based on the calculated qualitative score for models built with two features:
| Qualitative Score | Symbolic Score |
|---|---|
| q < -0.75 | --- |
| -0.75 <= q < -0.25 | -- |
| -0.25 <= q <= -0.001 | - |
| -0.001 < q <= 0 | grayed-out --- |
| 0 < q < 0.001 | grayed-out +++ |
| 0.001 <= q <= 0.25 | + |
| 0.25 < q <= 0.75 | ++ |
| q > 0.75 | +++ |
Features >= 2, < 10¶
The following describes the displayed symbolic score based on the calculated qualitative score for models built with more than two but fewer than 10 features:
| Qualitative Score | Symbolic Score |
|---|---|
| q < -2 / num_features | --- |
| -2 / num_features <= q < -1 / (2 * num_features) | -- |
| -1 / (2 * num_features) <= q <= -0.001 | - |
| -0.001 < q <= 0 | grayed-out --- |
| 0 < q < 0.001 | grayed-out +++ |
| 0.001 <= q <= 1 / (2 * num_features) | + |
| 1 / (2 * num_features) < q <= 2 / num_features | ++ |
| q > 2 / num_features | +++ |
Features >= 10¶
For the top 50 features, ranked by global Feature Impact score, explanation strengths are available. To calculate those values, DataRobot does the following on each row:
- Computes explanation strengths (sometimes called “raw scores”) for eligible features. (In the API, this value is returned as
strength.) - Computes a normalization factor. The value used for normalization is the sum of the top 10 largest absolute strengths. This value may differ between rows.
- Generates a normalized (qualitative or “q” score) by dividing all explanation strengths by the computed normalization factor. (This value is not available in the API.)
Use the table below to convert the normalized scores to qualitative symbols.
The following describes the displayed symbolic score based on the calculated qualitative score for models built with 10 or more features:
| Qualitative Score | Symbolic Score |
|---|---|
| q < -0.2 | --- |
| -0.2 <= q < -0.05 | -- |
| -0.05 <= q <= -0.001 | - |
| -0.001 < q <= 0 | grayed-out --- |
| 0 < q < 0.001 | grayed-out +++ |
| 0.001 <= q <= 0.05 | + |
| 0.05 < q <= 0.2 | ++ |
| q > 0.2 | +++ |