Once models are deployed, the deployment inventory is the central hub for deployment management activity. It serves as a coordination point for all stakeholders involved in operationalizing models. From the inventory, you can monitor deployed model performance and take action as necessary, as it provides an interface to all actively deployed models.
There are two unique deployment lenses that modify the information displayed in the inventory:
The Prediction Health lens summarizes prediction usage and model status for all active deployments.
The Governance lens reports the operational and social aspects of all active deployments.
To change deployment lenses, click the active lens in the top right corner and select a lens from the dropdown.
Prediction Health lens¶
The Prediction Health lens is the default view of the deployment inventory, detailing prediction activity and model health for each deployment. Across the top of the inventory, the page summarizes the usage and status of all active deployments, with color-coded health indicators.
Beneath the summary is an individual report for each deployment.
The following table describes the information available from the Prediction Health lens:
|Deployment Name||Name assigned to the deployment at creation, the type of prediction server used, and the project name (DataRobot models only).|
|Service||Service health of the individual deployment. The color-coded status indicates the presence or absence of errors in the last 24 hours.|
|Drift||Data Drift occurring in the deployment.|
|Accuracy||Model accuracy evaluated over time.|
|Activity||A bar graph indicating the pattern of predictions over the past seven days. The starting point is the same for each deployment in the inventory. For example, a new deployment will plot that day’s activity and six (blank) days previous.|
|Avg. Predictions/Day||Average number of predictions per day over the last seven days.|
|Last Prediction||Elapsed time since the last prediction was made against the model.|
|Actions||Menu of additional model management activities, including adding data, replacing a model, setting data drift, and sharing and deleting deployments.|
Click on any model entry in the table to view details about that deployment. Each model-specific page provides the above information in a status banner.
Color-coded health indicators¶
The Service Health, Data Drift, and Accuracy summaries in the top part of the display provide an at-a-glance indication of health and accuracy for all deployed models. To view this more detailed information for an individual model, click on the model in the inventory list.
Service Health Summary measures the following error types over the last 24 hours. These are the Data Error Rate and System Error Rate errors recorded for an individual model on the Service Health tab.
- 4xx errors indicate problems with the prediction request submission
- 5xx errors indicate problems with the DataRobot prediction server
Interpret the color indicators as follows:
|Color||Service Health||Data Drift||Accuracy||Action|
|Green / Passing||Zero 4xx or 5xx errors||All attributes' distributions have remained similar since the model was deployed||Accuracy is similar to when the model was deployed.||No action needed.|
|Yellow / At risk||At least one 4xx error and zero 5xx errors||At least one lower-importance attribute's distribution has shifted since the model was deployed.||Accuracy has declined since the model was deployed.||Concerns found but no immediate action needed; monitor.|
|Red / Failing||At least one 5xx error||At least one higher-importance attribute's distribution has shifted since the model was deployed.||Accuracy has severely declined since the model was deployed.||Immediate action needed.|
|Gray / Unknown||No predictions made||Insufficient predictions made (min. 100 required).||Insufficient predictions made (min. 100 required)||Make predictions.|
Live inventory updates¶
The inventory automatically refreshes every 30 seconds and updates the following information:
The Active Deployments tile indicates the number of deployments currently in use. The legend interprets the bar below the active deployment count:
- Deployments owned by you (blue)
- Deployments owned by others (white)
- Remaining allowed deployments (grey)
Inactive deployments do not count toward the allocated limit.
In the example above, the user's organization is allotted 1000 deployments. The user has 54 active deployments, and there are 705 active deployments across the organization. Users within the organization can create 289 more active deployments before reaching the limit. There is one inactive deployment.
The availability information shown on the Active Deployments tile depends on the pricing plan for your organization.
The Predictions tile indicates the number of predictions made since the last refresh.
Individual deployments show the number of predictions made on them during the last 30 seconds.
If a deployment's service health, drift, or accuracy status changes to Failing, the individual deployment will flash red to draw attention to it.
The deployment inventory is sorted by most recent prediction (reported in the Last Prediction column) by default. You can click a column title to sort by that column instead. A blue arrow appears next to the column title indicating whether the order is ascending or descending.
You can sort:
- Deployment Name alphabetically
- Service, Drift, and Accuracy by status (passing to failing)
- Avg. Predictions/Day by most predictions to fewest
- Last Prediction by most recent prediction to oldest
- Build Environment alphabetically
- Model Age by time passed since deployment
The list is sorted secondarily by time of deployment creation. For example, if you sorted by drift status, all deployments whose status is passing would be ordered from most recent creation to oldest, followed by failing deployments most recent to oldest.
To filter the deployment inventory, select Filters at the top of the inventory page.
The filter menu opens, allowing you to select the criteria by which deployments are filtered.
|Ownership||Filters by deployment Owner. Select Owned by me to display only those deployments for which you have the Owner role.|
|Activation Status||Filters by deployment activity status. Active deployments are able to monitor and return new predictions. Inactive deployments can only show insights and statistics about past predictions.|
|Service Status||Filters by deployment service health status. Choose to filter by passing (), at risk (), or failing () status. If a deployment has never had service health enabled, then it will not be included when this filter is applied.|
|Drift Status||Filters by deployment data drift status. Choose to filter by passing (), at risk (), or failing () status. If a deployment previously had data drift enabled and reported a status, then the last-reported status is used for filtering, even if you later disabled data drift for that deployment. If a deployment has never had drift enabled, then it will not be included when this filter is applied.|
|Accuracy Status||Filters by deployment accuracy status. Choose to filter by passing (), at risk (), or failing () status. If a deployment does not have accuracy information available, it is excluded from results when you apply the filter.|
|Importance||Filters by the criticality of deployments, based on prediction volume, exposure to regulatory requirements, and financial impact. Choices include Critical, High, Moderate, and Low.|
|Build environment||Filters by the environment in which the model was built.|
The deployment inventory filtering options depend on the pricing plan for your organization.
After selecting the desired filters, click Apply Filters to update the deployment inventory. The Filters link updates to indicate the number of filters applied.
You are notified if no deployments match your filters. To remove your filters, click the Clear all 3 filters shortcut, or open the filter dialog again and remove them manually.