Skip to content

On-premise users: click in-app to access the full platform documentation for your version of DataRobot.

Set up service health monitoring

On a deployment's Settings > Service health tab, you can enable segmented analysis for service health; however, to use segmented analysis for data drift and accuracy, you must also enable the following data drift settings:

  • Target monitoring (required to enable data drift and accuracy tracking)

  • Enable feature drift tracking (required to enable data drift tracking)

Once you've enabled the tracking required for your deployment, configure segment analysis to access segmented analysis of service health, data drift, and accuracy statistics by filtering them into unique segment attributes and values.

On a deployment's Service Health Settings, configure the following settings:

Field Description
Segmented Analysis
Track attributes for segmented analysis of training data and predictions Enables DataRobot to monitor deployment predictions by segments, for example by categorical features.
Definition
Range Displays the reference period for service health monitoring notifications; by default, the period is the last 7 days.
Notifications
Send notification Configures the schedule for service health monitoring notification checks.

Select segments for analysis

After enabling segmented analysis, specify the segment attributes to track in training and prediction data before making predictions. Selecting a segment attribute for tracking causes the model's data to be segmented by the attribute, allowing users to closely analyze the segment values that comprise the attributes selected for tracking. Attributes used for segmented analysis must be present in the training dataset for a deployed model, but they don't need to be features of the model. The list of segment attributes available for tracking is limited to categorical features, except the selected series ID used by multiseries deployments. To track an attribute, add it to the Track attributes for segmented analysis of training data and predictions field.

Note

If the training dataset used for the model doesn't contain any features suitable for segmented analysis, a tooltip appears stating: There are no corresponding categorical attributes in the training dataset.

The DataRobot-Consumer attribute (representing users making prediction requests) is always listed by default. For time series deployments with segmented analysis enabled, DataRobot automatically adds up to two segmented attributes: Forecast Distance and series id (the ID is only provided for multiseries models). Forecast distance is automatically available as a segment attribute without being explicitly present in the training dataset; it is inferred based on the forecast point and the date being predicted on. These attributes allow you to view accuracy and drift for a specific forecast distance, series, or other defined attribute. When you have finalized the attributes to track, click Save. Then, make predictions and navigate to the tab you want to analyze for your deployment by segment: Service health, Data drift, or Accuracy.

Important

Segmented analysis is only available for predictions made after segmented analysis is enabled.

Schedule notification checks

Service health tracks metrics about a deployment’s ability to respond to prediction requests quickly and reliably. You can view the service health status in the deployment inventory and visualize service health on the Monitoring > Service health tab.

Note

Only deployment Owners can modify service health monitoring settings; however, Users can configure the conditions under which notifications are sent to them. Consumers cannot modify monitoring or notification settings.

Service health monitoring reports changes to the service health status reported in the deployment inventory, representing the occurrence of 4XX and 5XX errors in your prediction requests or prediction server:

  • 4xx errors indicate problems with the prediction request submission.
  • 5xx errors indicate problems with the DataRobot prediction server.
Color Description Action
/ Green Passing: Zero 4xx or 5xx errors No action needed.
/ Yellow At risk: At least one 4xx error and zero 5xx errors Concerns found but no immediate action needed; monitor.
/ Red Failing: At least one 5xx error Immediate action needed.
/ Gray Unknown: No predictions made Make predictions.

To schedule recurring checks to determine if service health monitoring email notifications should be sent:

  1. On the Service Health Settings page, in the Notifications section, enable Send notifications.

  2. Click Set schedule and configure the settings for service health notifications. The following table lists the scheduling options. All times are displayed in UTC:

    Frequency Description
    Every hour Each hour on the selected minute past the hour.
    Every day Each day at the selected time.
    Every week Each selected day at the selected time.
    Every month Each month, on each selected day, at the selected time. The selected days in a month are provided as numbers (1 to 31) in a comma-separated list.
    Every quarter Each month of a quarter, on each selected day, at the selected time. The selected days in each month are provided as numbers (1 to 31) in a comma-separated list.
    Every year Each selected month, on each selected day, at the selected time. The selected days in each month are provided as numbers (1 to 31) in a comma-separated list.
    Use advanced scheduler
    Minute Each minute defined in a comma-separated list of numbers between 0 and 59, or * for all.
    Hour Each hour defined in a comma-separated list of numbers between 0 and 23, or * for all.
    Day of month Each day defined in a comma-separated list of numbers between 1 and 31, or * for all.
    Month Each month defined in a comma-separated list of numbers between 1 and 12, or * for all.
    Day of week Each weekday defined in a comma-separated list of numbers between 0 and 6, or * for all.
  3. After updating the scheduling settings, click Save.

    Important

    The configured time for the Notifications setting schedules a status check specific to the type of monitoring. DataRobot only sends notifications to subscribers if a monitoring event occurred between checks; for example, if a 4xx or 5xx error is received.


Updated April 3, 2024