Skip to content

On-premise users: click in-app to access the full platform documentation for your version of DataRobot.

Custom Metrics tab

On a deployment's Custom Metrics tab, you can use the data you collect from the Data Export tab (or data calculated through other custom metrics) to compute and monitor up to 25 custom business or performance metrics. These metrics are recorded on the configurable Custom Metric Summary dashboard, where you monitor, visualize, and export each metric's change over time. This feature allows you to implement your organization's specialized metrics, expanding on the insights provided by DataRobot's built-in Service Health, Data Drift, and Accuracy metrics.

What types of custom metrics are supported?

Three types of custom metrics are available for use. External metrics are generally available in Classic and NextGen, hosted custom metrics are available for public preview in Classic and NextGen, and hosted custom metric templates are available for public preview in NextGen:

Custom metric type Maturity Description
External custom metrics GA
  • What: A custom metric where the calculations of the metric are not directly hosted by DataRobot. An external metric is a simple API used to submit a metric value for DataRobot to save and visualize. The metric calculation is handled externally, by the user. External metrics can be combined with other tools in DataRobot like Notebooks, Jobs, or Custom Models, or external tools like Airflow or cloud providers to provide the hosting and calculation needed for a particular metric.
  • Where: On the Custom Metrics tab of any deployment, click Add custom metric > New external metric (or Create new external metric in Classic).
  • Why: Provides a simple option to save a number from your AI solution that you want for tracking and visualization in DataRobot. For example, you could track the change in LLM cost, calculated by your LLM provider, over time.
Hosted custom metrics Public Preview
  • What: A custom metric where the metric calculations are hosted in a custom job within DataRobot. For hosted metrics, DataRobot orchestrates pulling the data, computing the metric values, saving the values to storage, and visualizing the data. No outside tools or infrastructure are required.
  • Where: On the Custom Metrics tab of any deployment, click Add custom metric > New hosted metric (or Create new hosted metric in Classic).
  • Why: Provides a complete end-to-end workflow for building business-specific metrics and dashboards in DataRobot.
Hosted custom metric templates Public Preview
  • What: A template or ready-to-use example of a hosted custom metric, where DataRobot provides the user the code and automates the creation process for a hosted custom metric. For metric templates, the result is a hosted metric, without starting from scratch. Templates are provided by DataRobot and can be used as-is or modified to calculate new metrics.
  • Where: On the Custom Metrics tab of any deployment in NextGen only, click Add custom metric > Create new from template.
  • Why: Provides the simplest way to get started with custom metrics, where DataRobot provides an example implementation and a complete end-to-end workflow. They are ready to use in just a few clicks.

To access custom metrics, in the top navigation bar, click Deployments and, on the Deployments tab, click on the deployment for which you want to create custom metrics. Then, in the deployment, click the Custom Metrics tab. The Summary tab opens:

Add custom metrics

The Custom Metrics tab can track up to 25 metrics. To add custom metrics:

  1. On the Custom Metrics > Summary tab, click + Add Custom Metric.

  2. In the Add Custom Metric dialog box, click Create new external metric, click Next, and then configure the metric settings:

    Field Description
    Name A descriptive name for the metric. This name appears on the Custom Metric Summary dashboard.
    Description A description of the custom metric; for example, you could describe the purpose, calculation method, and more.
    Name of y-axis A descriptive name for the dependent variable. This name appears on the custom metric's chart on the Custom Metric Summary dashboard.
    Default interval The default interval used by the selected Aggregation type. Only HOUR is supported.
    Baseline The value used as a basis for comparison when calculating the x% better or x% worse values.
    Aggregation type If the metric is calculated as a Sum, Average, or Gauge.
    Metric direction The directionality of the metric and changes how changes the metric are visualized. You can select Higher is better or Lower is better. For example, if you choose Lower is better a 10% decrease in the calculated value of your custom metric will be considered 10% better, displayed in green.
    Is Model Specific When enabled, links the metric to the model with the Model Package ID (Registered Model Version ID) provided in the dataset. This setting influences when values are aggregated (or uploaded). For example:
    • Model specific (enabled): Model accuracy metrics are model specific, so the values are aggregated completely separately. When you replace a model, the chart for your custom accuracy metric onlys show data for the days after the replacement.
    • Not model specific (disabled): Revenue metrics aren't model specific, so the values are aggregated together. When you replace a model, the chart for your custom revenue metric doesn't change.
    This field can't be edited after you create the metric.
    Column names definition
    Timestamp column The column in the dataset containing a timestamp.
    Value column The column in the dataset containing the values used for custom metric calculation.
    Date format The date format used by the timestamp column.

    Note

    You can override the Column names definition settings when you upload data to a custom metric.

  3. Click Add custom metric.

Upload data to custom metrics

After you create a custom metric, you can provide data to calculate the metric:

  1. On the Custom Metrics tab, locate the custom metric for which you want to upload data, and then click the Upload Data icon.

  2. In the Upload Data dialog box, select an upload method, and then click Next:

    Upload method Description
    Upload data as file In the Choose file dialog box, drag and drop file(s) to upload, or click Choose file > Local file to browse your local filesystem, and then click Submit data. You can upload up to 10GB uploaded in one file.
    Use AI Catalog In the Select a dataset from the AI Catalog dialog box, click a dataset from the list, and then click Select a dataset. The AI Catalog includes datasets from the Data Export tab.
    Use API In the Use API Client dialog box, click Copy to clipboard, and then modify and use the API snippet to upload a dataset. You can upload up to 10,000 values in one API call.
  3. In the Select dataset columns dialog box, configure the following:

    Field Description
    Timestamp column The column in the dataset containing a timestamp.
    Value column The column in the dataset containing the values used for custom metric calculation.
    Association ID The row containing the association ID required by the custom metric to link predicted values to actuals.
    Date format The date format used by the timestamp column.
  4. Click Upload data.

Manage custom metrics

On the Custom Metrics dashboard, after you've added your custom metrics, you can edit, arrange, or delete them.

To edit or delete a metric, on the Custom Metrics tab, locate the custom metric you want to manage, and then click the more options icon:

  • To edit a metric, click Edit, update any configurable settings, and then click Update custom metric.

  • To delete a metric, click Delete.

To arrange or hide metrics on the Custom Metric Summary dashboard, locate the custom metric you want to move or hide:

  • To move a metric, click the grid icon () on the left side of the metric tile and then drag the metric to a new location.

  • To hide a metric's chart, clear the checkbox next to the metric name.

Configure the custom metric dashboard display settings

Configure the following settings to specify the custom metric calculations you want to view on the dashboard:

Setting Description
1 Model Select the deployment's model, current or previous, to show custom metrics for.
2 Range (UTC) Select the start and end dates of the period from which you want to view custom metrics.
3 Resolution Select the granularity of the date slider. Select from hourly, daily, weekly, and monthly granularity based on the time range selected. If the time range is longer than 7 days, hourly granularity is not available.
4 Refresh Refresh the custom metric dashboard.
5 Reset Reset the custom metric dashboard's display settings to the default.

Updated March 1, 2024