Custom metrics¶

On a deployment's Monitoring > Custom metrics tab, you can use the data you collect from the Data exploration tab (or data calculated through other custom metrics) to compute and monitor custom business or performance metrics. These metrics are recorded on the configurable Custom Metric Summary dashboard, where you monitor, visualize, and export each metric's change over time. This feature allows you to implement your organization's specialized metrics, expanding on the insights provided by DataRobot's built-in service health, data drift, and accuracy metrics.

To view and add custom metrics, in the Console, open the deployment for which you want to create custom metrics and click the Monitoring > Custom metrics tab:

Add custom metrics¶

The Custom metrics tab can track up to 25 metrics per deployment, including metrics hosted externally, and metrics hosted on custom jobs infrastructure. To add a metric, in a predictive or generative modeling deployment, click the Monitoring > Custom metrics tab. Then, on the Custom metrics tab, click + Add custom metric, select one of the following custom metric types, and proceed to the configuration steps linked in the table:

Add external custom metrics¶

External custom metrics allow you to create metrics with calculations occurring outside of DataRobot. With an external metric, you can submit a metric value for DataRobot to save and visualize. External metrics can be combined with other tools in DataRobot like notebooks, jobs, or custom models, or external tools like Airflow or cloud providers to provide the hosting and calculation needed for a particular metric.

To add an external custom metric, in the Add Custom Metric dialog box, configure the metric settings, and then click Add custom metric:

Add hosted custom metrics¶

Hosted custom metrics allow you to implement up to 25 of your organization's specialized metrics in a deployment, uploading the custom metric code using DataRobot Notebooks and hosting the metric calculation on custom jobs infrastructure. After creation, these custom metrics can be reused for other deployments.

To add a hosted custom metric, in the Add Custom Metric dialog box configure the metric settings, and then click Add custom metric from notebook:

After configuring a custom metric, DataRobot loads the notebook that contains the metric's code. The notebook contains one custom metric cell. A custom metric cell is a unique notebook cell, containing Python code defining how the metric is exported and calculated, code for scoring, and code to populate the metric. Modify the code in the custom metric cell as needed. Then, test the code by clicking Test custom metric code at the bottom of the cell. The test creates a custom job. If the test runs successfully, click Deploy custom metric code to add the custom metric to your deployment.

If the code does not run properly, you will receive the Testing custom metric code failed warning after testing completes. Click Open custom metric job to access the job and check the logs to troubleshoot the issue:

To troubleshoot a custom metric's code, navigate to the job's Runs tab, containing a log of the failed test. In the failed run, click View log.

Add hosted custom metrics from the gallery¶

The custom metrics gallery provides a centralized library containing pre-made, reusable, and shareable code implementing a variety of hosted custom metrics for predictive and generative models. These metrics are recorded on the configurable Custom Metric Summary dashboard, alongside any external custom metrics. From this dashboard, you can monitor, visualize, and export each metric's change over time. This feature allows you to implement your organization's specialized metrics, expanding on the insights provided by DataRobot's built-in service health, data drift, and accuracy metrics.

To add a pre-made custom metric to a deployment:

1. In the Add custom metric panel, select a custom metric template applicable to your use case.

   

   The metrics available for your deployment appear at the top of the list with the metric icon highlighted. For example, in a generative modeling deployment:

   

2. After you select a metric from the list, in the Custom metric configuration sidebar, configure a metric calculation schedule or run the metric calculation immediately, and, optionally, set a metric baseline value.

   

3. Click Create metric. The new metric appears on the Custom metrics dashboard.

   

4. After you create a custom metric, you can view the custom job associated with the metric. This job runs on the metric's defined schedule, in the same way as hosted custom metrics (those not from the gallery). To access and manage the associated custom job, click the options icon () and then click Open Custom Job:

   

   Sidecar metrics

   If you selected a [sidecar metric], when you open the Assemble tab, navigate to the Runtime Parameters section to set the SIDECAR_DEPLOYMENT_ID, associating the sidecar metric with the connected deployment required to calculate that metric. If you haven't deployed a model to calculate the metric, you can find pre-defined models for these metrics as global models.

Upload data to custom metrics¶

After you create a custom metric, you can provide data to calculate the metric:

1. On the Custom metrics tab, locate the custom metric for which you want to upload data and click the Upload Data icon.

   

2. In the Upload data dialog box, select an upload method and click Next:

   

   Upload method	Description
   Use Data Registry	In the Select a dataset panel, upload a dataset or click a dataset from the list, and then click Confirm. The Data Registry includes datasets from the Data exploration tab.
   Use API	In the Use API Client panel, click Copy to clipboard, and then modify and use the API snippet to upload a dataset. You can upload up to 10,000 values in one API call.

3. In the Select dataset columns dialog box, configure the following:

   

   Field	Description
   Timestamp column	(Required) The column in the dataset containing a timestamp.
   Value column	(Required) The column in the dataset containing the values used for custom metric calculation.
   Association ID	The row containing the association ID required by the custom metric to link predicted values to actuals.
   Date format	The date format used by the timestamp column.

4. Click Upload data.

Manage custom metrics¶

On the Custom metrics dashboard, after you've added your custom metrics, you can edit, arrange, or delete them:

Configure the custom metric dashboard display settings¶

Configure the following settings to specify the custom metric calculations you want to view on the dashboard:

Explore metric data¶

On the Custom metrics tab of a Generative AI deployment, in a custom metric chart's header, click Explore data to view prompts, responses, user ratings, and other custom metric scores per row, matched by association ID. This view provides insight into the quality of the Generative AI model's responses, as rated by users and based on any Generative AI custom metrics you implement. You can also click to download the data for the custom metric:

After you open the data quality table for data exploration, you can view prompts, responses, user ratings, and other custom metric scores per row, matched by association ID. The custom metric currently presented in the table is highlighted. Locate specific rows by filtering the table to only display rows containing Prompt values, Response values, or Actual values. Click Search to locate a specific row in the Data quality table by prompt. Click the open icon to expand the details panel, displaying a row's full Prompt and the Response matched with the prompt by association ID: