Skip to content

On-premise users: click in-app to access the full platform documentation for your version of DataRobot.

Set up service health monitoring

On a deployment's Service Health > Settings tab, you can enable segmented analysis for service health; however, to use segmented analysis for data drift and accuracy, you must also enable the following deployment settings:

Once you've enabled the tracking required for your deployment, configure segment analysis to access segmented analysis of service health, data drift, and accuracy statistics by filtering them into unique segment attributes and values.

On a deployment's Service Health > Settings tab, you can configure the Service Health Settings:

Field Description
Segmented Analysis
Track attributes for segmented analysis of training data and predictions Enables DataRobot to monitor deployment predictions by segments, for example by categorical features.
Notifications
Send notification Configures the schedule for service health monitoring notification checks.

After enabling segmented analysis, you must specify the segment attributes to track in training and prediction data before making predictions. Selecting a segment attribute for tracking causes the model's data to be segmented by the attribute, allowing users to closely analyze the segment values that comprise the attributes selected for tracking. Attributes used for segmented analysis must be present in the training dataset for a deployed model, but they don't need to be features of the model. The list of segment attributes available for tracking is limited to categorical features, except the selected series ID used by multiseries deployments. To track an attribute, add it to the Track attributes for segmented analysis of training data and predictions field. The "Consumer" attribute (representing users making prediction requests) is always listed by default.

For time series deployments with segmented analysis enabled, DataRobot automatically adds up to two segmented attributes: Forecast Distance and series id (the ID is only provided for multiseries models). Forecast distance is automatically available as a segment attribute without being explicitly present in the training dataset; it is inferred based on the forecast point and the date being predicted on. These attributes allow you to view accuracy and drift for a specific forecast distance, series, or other defined attribute.

When you have finalized the attributes to track, click Save.

Make predictions and navigate to the tab you want to analyze for your deployment by segment: Service Health, Data Drift, or Accuracy.

Important

Segmented analysis is only available for predictions made after the segmented analysis is enabled.

Schedule notification checks

Service health tracks metrics about a deployment’s ability to respond to prediction requests quickly and reliably. You can view the service health status in the deployment inventory and visualize service health on the Service Health tab.

Note

Only deployment Owners can modify service health monitoring settings; however, Users can configure the conditions under which notifications are sent to them. Consumers cannot modify monitoring or notification settings.

Service health monitoring reports changes to the service health status reported in the deployment inventory, representing the occurrence of 4XX and 5XX errors in your prediction requests or prediction server:

  • 4xx errors indicate problems with the prediction request submission.
  • 5xx errors indicate problems with the DataRobot prediction server.
Color Description Action
Green / Passing Zero 4xx or 5xx errors No action needed.
Yellow / At risk At least one 4xx error and zero 5xx errors Concerns found but no immediate action needed; monitor.
Red / Failing At least one 5xx error Immediate action needed.
Gray / Unknown No predictions made Make predictions.

To schedule recurring checks to determine if service health monitoring email notifications should be sent:

  1. On the Service Health Settings page, in the Notifications section, enable Send notifications.

  2. Configure the settings for service health notifications. The following table lists the scheduling options. All times are displayed in UTC:

    Frequency Description
    Every hour Each hour on the selected minute past the hour.
    Every day Each day at the selected time.
    Every week Each selected day at the selected time.
    Every month Each month, on each selected day, at the selected time. The selected days in a month are provided as numbers (1 to 31) in a comma separated list.
    Every quarter Each month of a quarter, on each selected day, at the selected time. The selected days in each month are provided as numbers (1 to 31) in a comma separated list.
    Every year Each selected month, on each selected day, at the selected time. The selected days in each month are provided as numbers (1 to 31) in a comma separated list.
    Use advanced scheduler
    Minute Each minute defined in a comma-separated list of numbers between 0 and 59, or * for all.
    Hour Each hour defined in a comma-separated list of numbers between 0 and 23, or * for all.
    Day of month Each day defined in a comma-separated list of numbers between 1 and 31, or * for all.
    Month Each month defined in a comma-separated list of numbers between 1 and 12, or * for all.
    Day of week Each weekday defined in a comma-separated list of numbers between 0 and 6, or * for all.
  3. After updating the scheduling settings, click Save.

    Important

    The configured time for the Notifications setting schedules a status check specific to the type of monitoring. DataRobot only sends notifications to subscribers if a monitoring event occurred between checks; for example, if a 4xx or 5xx error is received for service health, or if a data drift status change (from either passing to failing or failing to passing) occurs during the specified window, as shown below:


Updated February 1, 2024
Back to top