Usage¶
After deploying a model and making predictions in production, monitoring model quality and performance over time is critical to ensure the model remains effective. This monitoring occurs on the Data drift and Accuracy tabs and requires processing large amounts of prediction data. Prediction data processing can be subject to delays or rate limiting. Track prediction processing progress on the Usage tab.
Prediction Tracking chart¶
On the left side of the Monitoring > Usage tab is the Prediction Tracking chart, a bar chart of the prediction processing status over the last 24 hours or 7 days, tracking the number of processed, missing association ID, and rate-limited prediction rows. Depending on the selected view (24-hour or 7-day), the histogram's bins are hour-by-hour or day-by-day.
| Chart element | Description | |
|---|---|---|
| 1 | Select time period | Selects the Last 24 hours or Last 7 days view. |
| 2 | Use log scaling | Applies log scaling to the Prediction Tracking chart for deployments with more than 250,000 rows of predictions. |
| 3 | Time of Receiving Predictions Data (X-axis) |
Displays the time range (by day or hour) represented by a bin, tracking the rows of prediction data received within that range. Predictions are timestamped when a prediction is received by the system for processing. This "time received" value is not equivalent to the timestamp in service health, data drift, and accuracy. For DataRobot prediction environments, this timestamp value can be slightly later than prediction timestamp. For agent deployments, the timestamp represents when the DataRobot API received the prediction data from the agent. |
| 4 | Row Count (Y-axis) |
Displays the number of prediction rows timestamped within a bin's time range (by day or hour). |
| 5 | Prediction processing categories | Displays a bar chart tracking the status of prediction rows:
|
How does prediction rate limiting work?
The Usage tab displays the number of prediction rows subject to the monitoring rate limit of 100MB per hour. However, rate limiting only applies to prediction monitoring, all rows are included in the prediction results even after the rate limit is reached.
Large-scale monitoring prediction tracking
For a monitoring agent deployment, if you implement large-scale monitoring, the prediction rows won't appear in this bar chart; however, the Predictions Processing (Champion) delay will track the pre-aggregated data.
To view additional information on the Prediction Tracking chart, hover over a column to see the time range during which the prediction data was received and the number of rows that were Processed, Rate Limited, or Missing Association ID:
Prediction and actuals processing delay¶
On the right side of the Usage tab are the processing delays for Predictions Processing (Champion) and Actuals Processing (the delay in actuals processing is for all models in the deployment):
The Usage tab recalculates the processing delays without reloading the page. You can check the Updated value to determine when the delays were last updated.
Predictions and actuals upload limits¶
From the Usage tab, you can monitor the hourly, daily, and weekly upload limits configured for your organization's deployments. View charts that visualize the number of predictions and actuals processed and tiles that display the table size limits for returned prediction results.
The Totals tile shows how many predictions and actuals have been processed relative to the configured interval limit (displayed at the bottom of the tile). Additionally, you can view the table size limit for returned prediction results. The table size limits the number of prediction rows stored in DataRobot's database for a deployment. DataRobot stores one row per prediction (or two, for binary classification deployments). For multiclass deployments, information for all classes is stored in one row. Note that the table limit does not change when you change the time interval limit (hourly, daily, weekly). Any request that exceeds the table limit will be rejected, regardless of the time.
If you reach the exact processing limit value (for example, uploading 50,000 actuals in an hour with 50,000 as the limit), and you make an additional request (uploading 10,000 more actuals), then DataRobot processes the additional request and none of the actuals are rate limited. However, DataRobot treats predictions differently because they are processed in smaller chunks. A small chunk is processed, while the remaining predictions are rate limited. For example, if you reached a 50,000 prediction limit and uploaded 50,000 more, a chunk of 1,000 predictions may be processed as part of the small chunk.
You can view the prediction limits configured for a deployment by navigating to the Settings > Service Health tab to know when you can make predictions next if you have already reached the processing limit.
