Skip to content

On-premise users: click in-app to access the full platform documentation for your version of DataRobot.

Set up custom metrics monitoring

On a deployment's Settings > Custom metrics tab, you can define "at risk" and "failing" thresholds to monitor the status of the custom metrics you created on the Custom metrics tab.

On a deployment's Custom Metrics Settings page, you can configure the following settings:

Field Description
Association ID
Association ID Defines the name of the column that contains the association ID in the prediction dataset for your model. Association IDs function as an identifier for your prediction dataset so you can later match up outcome data (also called "actuals") with those predictions.
Require association ID in prediction requests Requires your prediction dataset to have a column name that matches the name you entered in the Association ID field. When enabled, you will get an error if the column is missing.
Enable automatic actuals feedback for time series models For time series deployments that have indicated an association ID. Enables the automatic submission of actuals, so that you do not need to submit them manually via the UI or API. Once enabled, actuals can be extracted from the data used to generate predictions. As each prediction request is sent, DataRobot can extract an actual value for a given date. This is because when you send prediction rows to forecast, historical data is included. This historical data serves as the actual values for the previous prediction request.
Definition
At Risk / Failing Enables DataRobot to apply logical statements to calculated custom metric values. You can define threshold statements for a custom metric to categorize the deployment as "at risk" or "failing" if either statement evaluates to true.
Notifications
Send notifications Configures the schedule for custom metric monitoring notification checks.

Define custom metric monitoring

Configure thresholds to alert you when a deployed model is "at risk" or "failing" to meet the standards you set for the selected custom metric.

Note

To access the settings in the Definition section, configure and save a metric on the Custom Metrics tab. Only deployment Owners can modify custom metric monitoring settings; however, Users can configure the conditions under which notifications are sent to them. Consumers cannot modify monitoring or notification settings.

You can customize the rules used to calculate the custom metrics status for your deployment on the Custom Metrics Settings page:

  1. In a deployment you want to monitor custom metrics for, click Settings > Custom metrics.

  2. In the Definition section, define logical statements for any of your custom metrics' At Risk and Failing thresholds:

    • For At Risk, click + Add new warning rule.

    • For Failing, click + Add new failing rule.

    Setting Description
    Metric Select the custom metric to add a threshold definition for.
    Condition Select one of the following conidition statements to set the custom metric's threshold for At Risk or Failing:
    • is less than
    • is less than or equal to
    • is greater than
    • is greater than or equal to
    Value Enter a numerical value to set the custom metric's threshold for At Risk or Failing.

    For example, a statement could be: "The ad campaign is At Risk if the Campaign ROI is less than 10000."

  3. After adding custom metrics monitoring definitions, click Save.

Schedule notification checks

To schedule recurring checks to determine if custom metric monitoring email notifications should be sent:

  1. On the Custom Metric Settings page, in the Notifications section, enable Send notifications.

  2. Configure the settings for custom metric notifications. The following table lists the scheduling options. All times are displayed in UTC:

    Frequency Description
    Every hour Each hour on the selected minute past the hour.
    Every day Each day at the selected time.
    Every week Each selected day at the selected time.
    Every month Each month, on each selected day, at the selected time. The selected days in a month are provided as numbers (1 to 31) in a comma-separated list.
    Every quarter Each month of a quarter, on each selected day, at the selected time. The selected days in each month are provided as numbers (1 to 31) in a comma-separated list.
    Every year Each selected month, on each selected day, at the selected time. The selected days in each month are provided as numbers (1 to 31) in a comma-separated list.
    Use advanced scheduler
    Minute Each minute defined in a comma-separated list of numbers between 0 and 59, or * for all.
    Hour Each hour defined in a comma-separated list of numbers between 0 and 23, or * for all.
    Day of month Each day defined in a comma-separated list of numbers between 1 and 31, or * for all.
    Month Each month defined in a comma-separated list of numbers between 1 and 12, or * for all.
    Day of week Each weekday defined in a comma-separated list of numbers between 0 and 6, or * for all.
  3. After updating the scheduling settings, click Save. At the configured time, DataRobot sends emails to subscribers.

    Important

    The configured time for the Notifications setting schedules a status check specific to the type of monitoring. DataRobot only sends notifications to subscribers if a monitoring event occurred between checks; for example, if a custom metric status change (from either passing to failing or failing to passing) occurs during the specified window.


Updated April 3, 2024