Anomaly Assessment¶
The Anomaly Assessment tab plots data for the selected backtest and provides, below the visualization, SHAP explanations for up to 500 anomalous points. Red points on the chart indicate that explanations are calculated and available. Clicking on an explanation expands and computes the Feature Over Time chart for the selected feature. The chart and explanations together provide some explanation for the source of an anomaly.
Anomaly Assessment chart¶
When you open the tab and click to compute the assessment, the most anomalous point in the validation data is selected by default (a white vertical bar) with corresponding explanations below. Hover on any point to see the prediction for that point; click elsewhere in the chart to move the bar. As the bar moves, the explanations below the chart update.
Note
SHAP explanations are available for up to 500 anomalous points per backtest. When a selected backtest has more than 500 sample points, the display uses red to indicate those points for which SHAP explanations are available and blue to show points without SHAP explanations. In other words, color coding, in this case, represents the availability of SHAP explanations not the value of the anomaly score.
Control the chart display¶
The chart provides several controls that modify the display, allowing you to focus on areas of particular interest.
Backtest / series selector
Use the dropdown to select a specific backtest or the holdout partition. The chart updates to include only data from within that date range. For multiseries projects there is an additional dropdown that allows you to select the series of interest.
Compute for training / Show training data
Initially the Anomaly Assessment chart displays anomalies found in the validation data. Click Compute for training to calculate anomalous points in the training data. Note, however, that training data is not a reliable measure of a model's ability to predict on future data.
Once computed, use the Show training data option to show training and validation (box checked) or only validation data (box unchecked).
Zoom to fit
When Zoom to fit is checked, DataRobot modifies the chart's Y-axis values to match the minimum and maximum of the target values. When unchecked, the chart scales to show the full possible range of target values:
When checked, it scales from the minimum to maximum values, which can change the relative difference in the anomaly score:
See the example in the Accuracy Over Time description for more detail.
Preview handles
Use the handles on preview pane to narrow the display in the chart above. Gradient coloring, in both the preview and the chart, indicates division in partitions, if applicable. Changes to the preview pane also impact the display of the Feature Over Time chart.
Display anomaly information¶
Hover on any point in the chart to see a report of the date and prediction score for that point:
Click a point to move the vertical bar to that point, which in turn updates the displayed SHAP scores. The SHAP score helps to understand how a feature is involved in the prediction.
Note
The SHAP Individual Prediction Explanations insight is not supported for time series projects. However, SHAP scores are run on a subset of points for the Anomaly Assessment.
List SHAP explanations¶
The white vertical bar in the main chart serves as a selector that controls the SHAP explanation display. As you click through the chart, you can notice that the list of explanations (scores) changes. For example, on 11/23/08 the anomaly score was fairly low with a derivation of the feature "Sales" having the most impact:
On 02/07/09, by contrast, a higher score is attributed to the actual precipitation on that day:
If a point is not anomalous, no SHAP scores are listed.
Display the Feature Over Time chart¶
From the list of SHAP scores, click a feature to see its Over Time chart. (Read more about the Over Time chart chart in the time series documentation.) The plot is computed for each backtest and series.
The white bar is based on the location set in the full chart. Note that if the selected anomaly point is in the training data, and Show training data is unchecked, the bar does not display.
Drag the handles in the preview pane to focus the display.
The chart is not available for text or categorical features.