Custom Metrics tab¶
On a deployment's Custom Metrics tab, you can use the data you collect from the Data Export tab (or data calculated through other custom metrics) to compute and monitor up to 25 custom business or performance metrics. These metrics are recorded on the configurable Custom Metric Summary dashboard, where you monitor, visualize, and export each metric's change over time. This feature allows you to implement your organization's specialized metrics, expanding on the insights provided by DataRobot's built-in Service Health, Data Drift, and Accuracy metrics.
What types of custom metrics are supported?
Three types of custom metrics are available for use. External metrics are generally available in Classic and NextGen, hosted custom metrics are available for public preview in Classic and NextGen, and hosted custom metric templates are available for public preview in NextGen:
Custom metric type | Maturity | Description |
---|---|---|
External custom metrics | GA |
|
Hosted custom metrics | Public Preview |
|
Hosted custom metric templates | Public Preview |
|
To access custom metrics, in the top navigation bar, click Deployments and, on the Deployments tab, click on the deployment for which you want to create custom metrics. Then, in the deployment, click the Custom Metrics tab. The Summary tab opens:
Add custom metrics¶
The Custom Metrics tab can track up to 25 metrics. To add custom metrics:
-
On the Custom Metrics > Summary tab, click + Add Custom Metric.
-
In the Add Custom Metric dialog box, click Create new external metric, click Next, and then configure the metric settings:
Field Description Name A descriptive name for the metric. This name appears on the Custom Metric Summary dashboard. Description A description of the custom metric; for example, you could describe the purpose, calculation method, and more. Name of y-axis A descriptive name for the dependent variable. This name appears on the custom metric's chart on the Custom Metric Summary dashboard. Default interval The default interval used by the selected Aggregation type. Only HOUR is supported. Baseline The value used as a basis for comparison when calculating the x% better or x% worse values. Aggregation type If the metric is calculated as a Sum, Average, or Gauge. Metric direction The directionality of the metric and changes how changes the metric are visualized. You can select Higher is better or Lower is better. For example, if you choose Lower is better a 10% decrease in the calculated value of your custom metric will be considered 10% better, displayed in green. Is Model Specific When enabled, links the metric to the model with the Model Package ID (Registered Model Version ID) provided in the dataset. This setting influences when values are aggregated (or uploaded). For example: - Model specific (enabled): Model accuracy metrics are model specific, so the values are aggregated completely separately. When you replace a model, the chart for your custom accuracy metric onlys show data for the days after the replacement.
- Not model specific (disabled): Revenue metrics aren't model specific, so the values are aggregated together. When you replace a model, the chart for your custom revenue metric doesn't change.
Column names definition Timestamp column The column in the dataset containing a timestamp. Value column The column in the dataset containing the values used for custom metric calculation. Date format The date format used by the timestamp column. Note
You can override the Column names definition settings when you upload data to a custom metric.
-
Click Add custom metric.
Upload data to custom metrics¶
After you create a custom metric, you can provide data to calculate the metric:
-
On the Custom Metrics tab, locate the custom metric for which you want to upload data, and then click the Upload Data icon.
-
In the Upload Data dialog box, select an upload method, and then click Next:
Upload method Description Upload data as file In the Choose file dialog box, drag and drop file(s) to upload, or click Choose file > Local file to browse your local filesystem, and then click Submit data. You can upload up to 10GB uploaded in one file. Use AI Catalog In the Select a dataset from the AI Catalog dialog box, click a dataset from the list, and then click Select a dataset. The AI Catalog includes datasets from the Data Export tab. Use API In the Use API Client dialog box, click Copy to clipboard, and then modify and use the API snippet to upload a dataset. You can upload up to 10,000 values in one API call. -
In the Select dataset columns dialog box, configure the following:
Field Description Timestamp column The column in the dataset containing a timestamp. Value column The column in the dataset containing the values used for custom metric calculation. Association ID The row containing the association ID required by the custom metric to link predicted values to actuals. Date format The date format used by the timestamp column. -
Click Upload data.
Manage custom metrics¶
On the Custom Metrics dashboard, after you've added your custom metrics, you can edit, arrange, or delete them.
To edit or delete a metric, on the Custom Metrics tab, locate the custom metric you want to manage, and then click the more options icon:
-
To edit a metric, click Edit, update any configurable settings, and then click Update custom metric.
-
To delete a metric, click Delete.
To arrange or hide metrics on the Custom Metric Summary dashboard, locate the custom metric you want to move or hide:
-
To move a metric, click the grid icon () on the left side of the metric tile and then drag the metric to a new location.
-
To hide a metric's chart, clear the checkbox next to the metric name.
Configure the custom metric dashboard display settings¶
Configure the following settings to specify the custom metric calculations you want to view on the dashboard:
Setting | Description | |
---|---|---|
1 | Model | Select the deployment's model, current or previous, to show custom metrics for. |
2 | Range (UTC) | Select the start and end dates of the period from which you want to view custom metrics. |
3 | Resolution | Select the granularity of the date slider. Select from hourly, daily, weekly, and monthly granularity based on the time range selected. If the time range is longer than 7 days, hourly granularity is not available. |
4 | Refresh | Refresh the custom metric dashboard. |
5 | Reset | Reset the custom metric dashboard's display settings to the default. |