Skip to content

On-premise users: click in-app to access the full platform documentation for your version of DataRobot.

Custom metrics

On a deployment's Monitoring > Custom metrics tab, you can use the data you collect from the Data Export tab (or data calculated through other custom metrics) to compute and monitor up to 25 custom business or performance metrics. These metrics are recorded on the configurable Custom Metric Summary dashboard, where you monitor, visualize, and export each metric's change over time. This feature allows you to implement your organization's specialized metrics, expanding on the insights provided by DataRobot's built-in service health, data drift, and accuracy metrics.

Custom metrics limits

You can have up to 25 custom metrics per deployment.

To view and add custom metrics, in the Console, open the deployment for which you want to create custom metrics and click the Monitoring > Custom metrics tab:

Add custom metrics

The Custom metrics tab can track up to 25 metrics per deployment, including metrics hosted externally, and metrics hosted on custom jobs infrastructure. To add a metric, in a predictive or generative modeling deployment, click the Monitoring > Custom metrics tab. Then, on the Custom metrics tab, click + Add custom metric, select one of the following custom metric types, and proceed to the configuration steps linked in the table:

Custom metric type Description
New external metric Add a custom metric where the calculations of the metric are not directly hosted by DataRobot. An external metric is a simple API used to submit a metric value for DataRobot to save and visualize. The metric calculation is handled externally, by the user. External metrics can be combined with other tools in DataRobot like notebooks, jobs, or custom models, or external tools like Airflow or cloud providers to provide the hosting and calculation needed for a particular metric.

External custom metrics provide a simple option to save a value from your AI solution for tracking and visualization in DataRobot. For example, you could track the change in LLM cost, calculated by your LLM provider, over time.
New hosted metric Add a custom metric where the metric calculations are hosted in a custom job within DataRobot. For hosted metrics, DataRobot orchestrates pulling the data, computing the metric values, saving the values to storage, and visualizing the data. No outside tools or infrastructure are required.

Hosted custom metrics provide a complete end-to-end workflow for building business-specific metrics and dashboards in DataRobot.
Create new from template Add a custom metric from a template, or ready-to-use example of a hosted custom metric, where DataRobot provides the code and automates the creation process. With metric templates, the result is a hosted metric, without starting from scratch. Templates are provided by DataRobot and can be used as-is or modified to calculate new metrics.

Hosted custom metric templates provide the simplest way to get started with custom metrics, where DataRobot provides an example implementation and a complete end-to-end workflow. They are ready to use in just a few clicks.

Add external custom metrics

External custom metrics allow you to create metrics with calculations occurring outside of DataRobot. With an external metric, you can submit a metric value for DataRobot to save and visualize. External metrics can be combined with other tools in DataRobot like notebooks, jobs, or custom models, or external tools like Airflow or cloud providers to provide the hosting and calculation needed for a particular metric.

To add an external custom metric, in the Add Custom Metric dialog box, configure the metric settings, and then click Add custom metric:

Field Description
Name (Required) A descriptive name for the metric. This name appears on the Custom Metric Summary dashboard.
Description A description of the custom metric; for example, you could describe the purpose, calculation method, and more.
Name of y-axis (label) (Required) A descriptive name for the dependent variable. This name appears on the custom metric's chart on the Custom Metric Summary dashboard.
Default interval The default interval used by the selected Aggregation type. Only HOUR is supported.
Baseline The value used as a basis for comparison when calculating the x% better or x% worse values.
Aggregation type The type of metric calculation. Select from Sum, Average, or Gauge—a metric with a distinct value measured at single point in time.
Metric direction The directionality of the metric, which controls how changes to the metric are visualized. You can select Higher is better or Lower is better. For example, if you choose Lower is better, a 10% decrease in the calculated value of your custom metric will be considered 10% better, and displayed in green.
Is model-specific When enabled, links the metric to the model with the Model Package ID (the Registered Model Version ID) provided in the dataset. This setting influences when values are aggregated (or uploaded). For example:
  • Model-specific (enabled): Model accuracy metrics are model-specific, so the values are aggregated separately. When you replace a model, the chart for your custom accuracy metric only shows data for the days after the replacement.
  • Not model-specific (disabled): Revenue metrics aren't model-specific, so the values are aggregated together. When you replace a model, the chart for your custom revenue metric doesn't change.
This field can't be edited after you create the metric.
Column name definitions
Timestamp column (Required) The column in the dataset containing a timestamp.
Value column (Required) The column in the dataset containing the values used for custom metric calculation.
Date format The date format used by the timestamp column.

Note

You can override the Column names definition settings when you upload data to a custom metric, as described below.

Add hosted custom metrics

Availability information

Hosted custom metrics are off by default and require custom jobs and environments, which are also off by default. Contact your DataRobot representative or administrator for information on enabling this feature.

Feature flags: Enable Hosted Custom Metrics, Enable Custom Jobs, Enable Notebooks Custom Environments

Hosted custom metrics allow you to implement up to 25 of your organization's specialized metrics in a deployment, uploading the custom metric code using DataRobot Notebooks and hosting the metric calculation on custom jobs infrastructure. After creation, these custom metrics can be reused for other deployments.

To add a hosted custom metric, in the Add Custom Metric dialog box configure the metric settings, and then click Add custom metric from notebook:

Field Description
Name (Required) A descriptive name for the metric. This name appears on the Custom Metric Summary dashboard.
Description A description of the custom metric; for example, you could describe the purpose, calculation method, and more.
Name of y-axis (label) (Required) A descriptive name for the dependent variable. This name appears on the custom metric's chart on the Custom Metric Summary dashboard.
Default interval Determines the default interval used by the selected Aggregation type. Only HOUR is supported.
Baseline Determines the value used as a basis for comparison when calculating the x% better or x% worse values.
Aggregation type Determines if the metric is calculated as a Sum, Average, or Gauge.
Metric direction Determines the directionality of the metric, which controls how changes to the metric are visualized. You can select Higher is better or Lower is better. For example, if you choose Lower is better a 10% decrease in the calculated value of your custom metric will be considered 10% better, displayed in green.
Is Model Specific When enabled, this setting links the metric to the model with the Model Package ID (Registered Model Version ID) provided in the dataset. This setting influences when values are aggregated (or uploaded). For example:
  • Model specific (enabled): Model accuracy metrics are model specific, so the values are aggregated completely separately. When you replace a model, the chart for your custom accuracy metric only shows data for the days after the replacement.
  • Not model specific (disabled): Revenue metrics aren't model specific, so the values are aggregated together. When you replace a model, the chart for your custom revenue metric doesn't change.
This field can't be edited after you create the metric.
Schedule Defines when the custom metrics are populated. Select a frequency (hourly, daily, monthly, etc.) and a time. Select Use advanced scheduler for more precise scheduling options.

After configuring a custom metric, DataRobot loads the notebook that contains the metric's code. The notebook contains one custom metric cell. A custom metric cell is a unique notebook cell, containing Python code defining how the metric is exported and calculated, code for scoring, and code to populate the metric. Modify the code in the custom metric cell as needed. Then, test the code by clicking Test custom metric code at the bottom of the cell. The test creates a custom job. If the test runs successfully, click Deploy custom metric code to add the custom metric to your deployment.

If the code does not run properly, you will receive the Testing custom metric code failed warning after testing completes. Click Open custom metric job to access the job and check the logs to troubleshoot the issue:

To troubleshoot a custom metric's code, navigate to the job's Runs tab, containing a log of the failed test. In the failed run, click View log.

Availability information

The custom metric gallery requires hosted custom metrics and custom jobs and environments, which are off by default. Contact your DataRobot representative or administrator for information on enabling this feature.

Feature flags: Enable Hosted Custom Metrics, Enable Custom Jobs, Enable Notebooks Custom Environments

The custom metrics gallery provides a centralized library containing pre-made, reusable, and shareable code implementing a variety of hosted custom metrics for predictive and generative models. These metrics are recorded on the configurable Custom Metric Summary dashboard, alongside any external custom metrics. From this dashboard, you can monitor, visualize, and export each metric's change over time. This feature allows you to implement your organization's specialized metrics, expanding on the insights provided by DataRobot's built-in service health, data drift, and accuracy metrics.

To add a pre-made custom metric to a deployment:

  1. In the Add custom metric panel, select a custom metric template applicable to your use case.

    The metrics available for your deployment appear at the top of the list with the metric icon highlighted. For example, in a generative modeling deployment:

  2. After you select a metric from the list, in the Custom metric configuration sidebar, configure a metric calculation schedule or run the metric calculation immediately, and, optionally, set a metric baseline value.

  3. Click Create metric. The new metric appears on the Custom metrics dashboard.

  4. After you create a custom metric, you can view the custom job associated with the metric. This job runs on the metric's defined schedule, in the same way as hosted custom metrics (those not from the gallery). To access and manage the associated custom job, click the options icon () and then click Open Custom Job:

    Sidecar metrics

    If you selected a [sidecar metric], when you open the Assemble tab, navigate to the Runtime Parameters section to set the SIDECAR_DEPLOYMENT_ID, associating the sidecar metric with the connected deployment required to calculate that metric. If you haven't deployed a model to calculate the metric, you can find pre-defined models for these metrics as global models.

Upload data to custom metrics

After you create a custom metric, you can provide data to calculate the metric:

  1. On the Custom metrics tab, locate the custom metric for which you want to upload data and click the Upload Data icon.

  2. In the Upload data dialog box, select an upload method and click Next:

    Upload method Description
    Use Data Registry In the Select a dataset panel, upload a dataset or click a dataset from the list, and then click Confirm. The Data Registry includes datasets from the Data Export tab.
    Use API In the Use API Client panel, click Copy to clipboard, and then modify and use the API snippet to upload a dataset. You can upload up to 10,000 values in one API call.
  3. In the Select dataset columns dialog box, configure the following:

    Field Description
    Timestamp column (Required) The column in the dataset containing a timestamp.
    Value column (Required) The column in the dataset containing the values used for custom metric calculation.
    Association ID The row containing the association ID required by the custom metric to link predicted values to actuals.
    Date format The date format used by the timestamp column.
  4. Click Upload data.

Manage custom metrics

On the Custom metrics dashboard, after you've added your custom metrics, you can edit, arrange, or delete them:

To edit or delete a metric, on the Custom metrics tab, locate the custom metric you want to manage, and then click the more options icon:

  • To edit a metric, click Edit, update any configurable settings, and then click Update custom metric.

  • To delete a metric, click Delete.

To arrange or hide metrics on the Custom Metric Summary dashboard, locate the custom metric you want to move or hide:

  • To move a metric, click the grid icon on the left side of the metric tile and then drag the metric to a new location.

  • To hide a metric chart, clear the checkbox next to the metric name.

Configure the custom metric dashboard display settings

Configure the following settings to specify the custom metric calculations you want to view on the dashboard:

Setting Description
1 Model Select the deployment's model, current or previous, to show custom metrics for.
2 Range (UTC) / Date Slider Select the start and end dates of the period from which you want to view custom metrics.
3 Resolution Select the granularity of the date slider. Select from hourly, daily, weekly, and monthly granularity based on the time range selected. If the time range is longer than 7 days, hourly granularity is not available.
4 Refresh Refresh the custom metric dashboard.
5 Reset Reset the custom metric dashboard's display settings to the default.

Updated April 3, 2024