# Custom metrics
URL: https://docs.datarobot.com/en/docs/api/reference/sdk/custom-metrics.html

# Custom metrics

### class datarobot.models.deployment.custom_metrics.CustomMetric

A DataRobot custom metric.

Added in version v3.4.

- Variables:

#### classmethod create(name, deployment_id, units, is_model_specific, aggregation_type, time_step='hour', directionality=None, description=None, baseline_value=None, value_column_name=None, sample_count_column_name=None, timestamp_column_name=None, timestamp_format=None, batch_column_name=None, categories=None, is_geospatial=None, geospatial_segment_attribute=None)

Create a custom metric for a deployment

- Parameters:
- Returns: The custom metric object.
- Return type: CustomMetric

> [!NOTE] Examples
> ```
> from datarobot.models.deployment import CustomMetric
> from datarobot.enums import CustomMetricAggregationType, CustomMetricDirectionality
> 
> custom_metric = CustomMetric.create(
>     deployment_id="5c939e08962d741e34f609f0",
>     name="Sample metric",
>     units="Y",
>     baseline_value=12,
>     is_model_specific=True,
>     aggregation_type=CustomMetricAggregationType.AVERAGE,
>     directionality=CustomMetricDirectionality.HIGHER_IS_BETTER
>     )
> ```

#### classmethod get(deployment_id, custom_metric_id)

Get a custom metric for a deployment

- Parameters:
- Returns: The custom metric object.
- Return type: CustomMetric

> [!NOTE] Examples
> ```
> from datarobot.models.deployment import CustomMetric
> 
> custom_metric = CustomMetric.get(
>     deployment_id="5c939e08962d741e34f609f0",
>     custom_metric_id="65f17bdcd2d66683cdfc1113"
> )
> 
> custom_metric.id
> >>>'65f17bdcd2d66683cdfc1113'
> ```

#### classmethod list(deployment_id)

List all custom metrics for a deployment

- Parameters: deployment_id ( str ) – The ID of the deployment.
- Returns: custom_metrics – A list of custom metrics objects.
- Return type: list

> [!NOTE] Examples
> ```
> from datarobot.models.deployment import CustomMetric
> 
> custom_metrics = CustomMetric.list(deployment_id="5c939e08962d741e34f609f0")
> custom_metrics[0].id
> >>>'65f17bdcd2d66683cdfc1113'
> ```

#### classmethod delete(deployment_id, custom_metric_id)

Delete a custom metric associated with a deployment.

- Parameters:
- Return type: None

> [!NOTE] Examples
> ```
> from datarobot.models.deployment import CustomMetric
> 
> CustomMetric.delete(
>     deployment_id="5c939e08962d741e34f609f0",
>     custom_metric_id="65f17bdcd2d66683cdfc1113"
> )
> ```

#### update(name=None, units=None, aggregation_type=None, directionality=None, time_step=None, description=None, baseline_value=None, value_column_name=None, sample_count_column_name=None, timestamp_column_name=None, timestamp_format=None, batch_column_name=None)

Update metadata of a custom metric

- Parameters:
- Returns: The custom metric object.
- Return type: CustomMetric

> [!NOTE] Examples
> ```
> from datarobot.models.deployment import CustomMetric
> from datarobot.enums import CustomMetricAggregationType, CustomMetricDirectionality
> 
> custom_metric = CustomMetric.get(
>     deployment_id="5c939e08962d741e34f609f0",
>     custom_metric_id="65f17bdcd2d66683cdfc1113"
> )
> custom_metric = custom_metric.update(
>     deployment_id="5c939e08962d741e34f609f0",
>     name="Sample metric",
>     units="Y",
>     baseline_value=12,
>     is_model_specific=True,
>     aggregation_type=CustomMetricAggregationType.AVERAGE,
>     directionality=CustomMetricDirectionality.HIGHER_IS_BETTER
>     )
> ```

#### unset_baseline()

Unset the baseline value of a custom metric

- Return type: None

> [!NOTE] Examples
> ```
> from datarobot.models.deployment import CustomMetric
> from datarobot.enums import CustomMetricAggregationType, CustomMetricDirectionality
> 
> custom_metric = CustomMetric.get(
>     deployment_id="5c939e08962d741e34f609f0",
>     custom_metric_id="65f17bdcd2d66683cdfc1113"
> )
> custom_metric.baseline_values
> >>> [{'value': 12.0}]
> custom_metric.unset_baseline()
> custom_metric.baseline_values
> >>> []
> ```

#### submit_values(data, model_id=None, model_package_id=None, dry_run=False)

Submit aggregated custom metrics values from JSON.

- Parameters:
- Return type: None

> [!NOTE] Examples
> ```
> from datarobot.models.deployment import CustomMetric
> 
> custom_metric = CustomMetric.get(
>     deployment_id="5c939e08962d741e34f609f0",
>     custom_metric_id="65f17bdcd2d66683cdfc1113"
> )
> 
> # data for values over time
> data = [{
>     'value': 12,
>     'sample_size': 3,
>     'timestamp': '2024-03-15T14:00:00'
> }]
> 
> # data witch association ID
> data = [{
>     'value': 12,
>     'sample_size': 3,
>     'timestamp': '2024-03-15T14:00:00',
>     'association_id': '65f44d04dbe192b552e752ed'
> }]
> 
> # data for batches
> data = [{
>     'value': 12,
>     'sample_size': 3,
>     'batch': '65f44c93fedc5de16b673a0d'
> }]
> 
> # for deployment specific metrics
> custom_metric.submit_values(data=data)
> 
> # for model specific metrics pass model_package_id or model_id
> custom_metric.submit_values(data=data, model_package_id="6421df32525c58cc6f991f25")
> 
> # dry run
> custom_metric.submit_values(data=data, model_package_id="6421df32525c58cc6f991f25", dry_run=True)
> ```

#### submit_single_value(value, model_id=None, model_package_id=None, dry_run=False, segments=None)

Submit a single custom metric value at the current moment.

- Parameters:
- Return type: None

> [!NOTE] Examples
> ```
> from datarobot.models.deployment import CustomMetric
> 
> custom_metric = CustomMetric.get(
>     deployment_id="5c939e08962d741e34f609f0",
>     custom_metric_id="65f17bdcd2d66683cdfc1113"
> )
> 
> # for deployment specific metrics
> custom_metric.submit_single_value(value=121)
> 
> # for model specific metrics pass model_package_id or model_id
> custom_metric.submit_single_value(value=121, model_package_id="6421df32525c58cc6f991f25")
> 
> # dry run
> custom_metric.submit_single_value(value=121, model_package_id="6421df32525c58cc6f991f25", dry_run=True)
> 
> # for segmented analysis
> segments = [{"name": "custom_seg", "value": "val_1"}]
> custom_metric.submit_single_value(value=121, model_package_id="6421df32525c58cc6f991f25", segments=segments)
> ```

#### submit_values_from_catalog(dataset_id, model_id=None, model_package_id=None, batch_id=None, segments=None, geospatial=None)

Submit aggregated custom metrics values from dataset (AI catalog).
The names of the columns in the dataset should correspond to the names of the columns that were defined in
the custom metric. In addition, the format of the timestamps should also be the same as defined in the metric.

- Parameters:
- Return type: None

> [!NOTE] Examples
> ```
> from datarobot.models.deployment import CustomMetric
> 
> custom_metric = CustomMetric.get(
>     deployment_id="5c939e08962d741e34f609f0",
>     custom_metric_id="65f17bdcd2d66683cdfc1113"
> )
> 
> # for deployment specific metrics
> custom_metric.submit_values_from_catalog(dataset_id="61093144cabd630828bca321")
> 
> # for model specific metrics pass model_package_id or model_id
> custom_metric.submit_values_from_catalog(
>     dataset_id="61093144cabd630828bca321",
>     model_package_id="6421df32525c58cc6f991f25"
> )
> 
> # for segmented analysis
> segments = [{"name": "custom_seg", "column": "column_with_segment_values"}]
> custom_metric.submit_values_from_catalog(
>     dataset_id="61093144cabd630828bca321",
>     model_package_id="6421df32525c58cc6f991f25",
>     segments=segments
> )
> ```

#### get_values_over_time(start, end, model_package_id=None, model_id=None, segment_attribute=None, segment_value=None, bucket_size='P7D')

Retrieve values of a single custom metric over a time period.

- Parameters:
- Returns: custom_metric_over_time – The queried custom metric values over time information.
- Return type: CustomMetricValuesOverTime

> [!NOTE] Examples
> ```
> from datarobot.models.deployment import CustomMetric
> from datetime import datetime, timedelta
> 
> now=datetime.now()
> custom_metric = CustomMetric.get(
>     deployment_id="5c939e08962d741e34f609f0",
>     custom_metric_id="65f17bdcd2d66683cdfc1113"
> )
> values_over_time = custom_metric.get_values_over_time(start=now - timedelta(days=7), end=now)
> 
> values_over_time.bucket_values
> >>> {datetime.datetime(2024, 3, 22, 14, 0, tzinfo=tzutc()): 1.0,
> >>> datetime.datetime(2024, 3, 22, 15, 0, tzinfo=tzutc()): 123.0}}
> 
> values_over_time.bucket_sample_sizes
> >>> {datetime.datetime(2024, 3, 22, 14, 0, tzinfo=tzutc()): 1,
> >>>  datetime.datetime(2024, 3, 22, 15, 0, tzinfo=tzutc()): 1}}
> 
> values_over_time.get_buckets_as_dataframe()
> >>>                        start                       end  value  sample_size
> >>> 0  2024-03-21 16:00:00+00:00 2024-03-21 17:00:00+00:00    NaN          NaN
> >>> 1  2024-03-21 17:00:00+00:00 2024-03-21 18:00:00+00:00    NaN          NaN
> ```

#### get_values_over_space(start=None, end=None, model_id=None, model_package_id=None)

Retrieve values of a custom metric over space.

- Parameters:
- Returns: custom_metric_over_space – The queried custom metric values over space information.
- Return type: CustomMetricValuesOverSpace

> [!NOTE] Examples
> ```
> from datarobot.models.deployment import CustomMetric
> 
> custom_metric = CustomMetric.get(
>     deployment_id="5c939e08962d741e34f609f0",
>     custom_metric_id="65f17bdcd2d66683cdfc1113"
> )
> values_over_space = custom_metric.get_values_over_space(model_package_id='6421df32525c58cc6f991f25')
> ```

#### get_summary(start, end, model_package_id=None, model_id=None, segment_attribute=None, segment_value=None)

Retrieve the summary of a custom metric over a time period.

- Parameters:
- Returns: custom_metric_summary – The summary of the custom metric.
- Return type: CustomMetricSummary

> [!NOTE] Examples
> ```
> from datarobot.models.deployment import CustomMetric
> from datetime import datetime, timedelta
> 
> now=datetime.now()
> custom_metric = CustomMetric.get(
>     deployment_id="5c939e08962d741e34f609f0",
>     custom_metric_id="65f17bdcd2d66683cdfc1113"
> )
> summary = custom_metric.get_summary(start=now - timedelta(days=7), end=now)
> 
> print(summary)
> >> "CustomMetricSummary(2024-03-21 15:52:13.392178+00:00 - 2024-03-22 15:52:13.392168+00:00:
> {'id': '65fd9b1c0c1a840bc6751ce0', 'name': 'Test METRIC', 'value': 215.0, 'sample_count': 13,
> 'baseline_value': 12.0, 'percent_change': 24.02})"
> ```

#### get_values_over_batch(batch_ids=None, model_package_id=None, model_id=None, segment_attribute=None, segment_value=None)

Retrieve values of a single custom metric over batches.

- Parameters:
- Returns: custom_metric_over_batch – The queried custom metric values over batch information.
- Return type: CustomMetricValuesOverBatch

> [!NOTE] Examples
> ```
>    from datarobot.models.deployment import CustomMetric
> 
>    custom_metric = CustomMetric.get(
>        deployment_id="5c939e08962d741e34f609f0",
>        custom_metric_id="65f17bdcd2d66683cdfc1113"
>    )
>    # all batch metrics all model specific
>    values_over_batch = custom_metric.get_values_over_batch(model_package_id='6421df32525c58cc6f991f25')
> 
>    values_over_batch.bucket_values
>    >>> {'6572db2c9f9d4ad3b9de33d0': 35.0, '6572db2c9f9d4ad3b9de44e1': 105.0}
> 
>    values_over_batch.bucket_sample_sizes
>    >>> {'6572db2c9f9d4ad3b9de33d0': 6, '6572db2c9f9d4ad3b9de44e1': 8}
> 
> values_over_batch.get_buckets_as_dataframe()
> >>>                    batch_id                     batch_name  value  sample_size
> >>> 0  6572db2c9f9d4ad3b9de33d0  Batch 1 - 03/26/2024 13:04:46   35.0            6
> >>> 1  6572db2c9f9d4ad3b9de44e1  Batch 2 - 03/26/2024 13:06:04  105.0            8
> ```

#### get_batch_summary(batch_ids=None, model_package_id=None, model_id=None, segment_attribute=None, segment_value=None)

Retrieve the summary of a custom metric over a batch.

- Parameters:
- Returns: custom_metric_summary – The batch summary of the custom metric.
- Return type: CustomMetricBatchSummary

> [!NOTE] Examples
> ```
> from datarobot.models.deployment import CustomMetric
> 
> custom_metric = CustomMetric.get(
>     deployment_id="5c939e08962d741e34f609f0",
>     custom_metric_id="65f17bdcd2d66683cdfc1113"
> )
> # all batch metrics all model specific
> batch_summary = custom_metric.get_batch_summary(model_package_id='6421df32525c58cc6f991f25')
> 
> print(batch_summary)
> >> CustomMetricBatchSummary({'id': '6605396413434b3a7b74342c', 'name': 'batch metric', 'value': 41.25,
> 'sample_count': 28, 'baseline_value': 123.0, 'percent_change': -66.46})
> ```

### class datarobot.models.deployment.custom_metrics.CustomMetricValuesOverTime

Custom metric over time information.

Added in version v3.4.

- Variables:

#### classmethod get(deployment_id, custom_metric_id, start, end, model_id=None, model_package_id=None, segment_attribute=None, segment_value=None, bucket_size='P7D')

Retrieve values of a single custom metric over a time period.

- Parameters:
- Returns: custom_metric_over_time – The queried custom metric values over time information.
- Return type: CustomMetricValuesOverTime

#### property bucket_values : Dict[datetime, int]

The metric value for all time buckets, keyed by start time of the bucket.

- Returns: bucket_values
- Return type: Dict

#### property bucket_sample_sizes : Dict[datetime, int]

The sample size for all time buckets, keyed by start time of the bucket.

- Returns: bucket_sample_sizes
- Return type: Dict

#### get_buckets_as_dataframe()

Retrieves all custom metrics buckets in a pandas DataFrame.

- Returns: buckets
- Return type: pd.DataFrame

### class datarobot.models.deployment.custom_metrics.CustomMetricSummary

The summary of a custom metric.

Added in version v3.4.

- Variables:

#### classmethod get(deployment_id, custom_metric_id, start, end, model_id=None, model_package_id=None, segment_attribute=None, segment_value=None)

Retrieve the summary of a custom metric over a time period.

- Parameters:
- Returns: custom_metric_summary – The summary of the custom metric.
- Return type: CustomMetricSummary

### class datarobot.models.deployment.custom_metrics.CustomMetricValuesOverBatch

Custom metric over batch information.

Added in version v3.4.

- Variables:

#### classmethod get(deployment_id, custom_metric_id, batch_ids=None, model_id=None, model_package_id=None, segment_attribute=None, segment_value=None)

Retrieve values of a single custom metric over batches.

- Parameters:
- Returns: custom_metric_over_batch – The queried custom metric values over batch information.
- Return type: CustomMetricValuesOverBatch

#### property bucket_values : Dict[str, int]

The metric value for all batch buckets, keyed by batch ID

- Returns: bucket_values
- Return type: Dict

#### property bucket_sample_sizes : Dict[str, int]

The sample size for all batch buckets, keyed by batch ID.

- Returns: bucket_sample_sizes
- Return type: Dict

#### get_buckets_as_dataframe()

Retrieves all custom metrics buckets in a pandas DataFrame.

- Returns: buckets
- Return type: pd.DataFrame

### class datarobot.models.deployment.custom_metrics.CustomMetricBatchSummary

The batch summary of a custom metric.

Added in version v3.4.

- Variables: metric ( Dict ) – The summary of the batch custom metric.

#### classmethod get(deployment_id, custom_metric_id, batch_ids=None, model_id=None, model_package_id=None, segment_attribute=None, segment_value=None)

Retrieve the summary of a custom metric over a batch.

- Parameters:
- Returns: custom_metric_summary – The batch summary of the custom metric.
- Return type: CustomMetricBatchSummary

### class datarobot.models.deployment.custom_metrics.HostedCustomMetricTemplate

Template for hosted custom metric.

#### classmethod list(search=None, order_by=None, metric_type=None, offset=None, limit=None)

List all hosted custom metric templates.

- Parameters:
- Returns: templates
- Return type: List[HostedCustomMetricTemplate]

#### classmethod get(template_id)

Get a hosted custom metric template by ID.

- Parameters: template_id ( str ) – ID of the template.
- Returns: template
- Return type: HostedCustomMetricTemplate

### class datarobot.models.deployment.custom_metrics.HostedCustomMetric

Hosted custom metric.

#### classmethod list(job_id, skip=None, limit=None)

List all hosted custom metrics for a job.

- Parameters: job_id ( str ) – ID of the job.
- Returns: metrics
- Return type: List[HostedCustomMetric]

#### classmethod create_from_template(template_id, deployment_id, job_name, custom_metric_name, job_description=None, custom_metric_description=None, sidecar_deployment_id=None, baseline_value=None, timestamp=None, value=None, sample_count=None, batch=None, schedule=None, parameter_overrides=None)

Create a hosted custom metric from a template.
A shortcut for 2 calls:
Job.from_custom_metric_template(template_id)
HostedCustomMetrics.create_from_custom_job()

- Parameters:
- Returns: metric
- Return type: HostedCustomMetric

#### classmethod create_from_custom_job(custom_job_id, deployment_id, name, description=None, baseline_value=None, timestamp=None, value=None, sample_count=None, batch=None, schedule=None, parameter_overrides=None, geospatial_segment_attribute=None)

Create a hosted custom metric from existing custom job.

- Parameters:
- Returns: metric
- Return type: HostedCustomMetric

#### update(name=None, description=None, units=None, directionality=None, aggregation_type=None, baseline_value=None, timestamp=None, value=None, sample_count=None, batch=None, schedule=None, parameter_overrides=None)

Update the hosted custom metric.

- Parameters:
- Returns: metric
- Return type: HostedCustomMetric

#### delete()

Delete the hosted custom metric.

- Return type: None

### class datarobot.models.deployment.custom_metrics.DeploymentDetails

Information about a hosted custom metric deployment.

### class datarobot.models.deployment.custom_metrics.MetricBaselineValue

The baseline values for a custom metric.

### class datarobot.models.deployment.custom_metrics.SampleCountField

A weight column used with columnar datasets if pre-aggregated metric values are provided.

### class datarobot.models.deployment.custom_metrics.ValueField

A custom metric value source for when reading values from a columnar dataset like a file.

### class datarobot.models.deployment.custom_metrics.MetricTimestampSpoofing

Custom metric timestamp spoofing. Occurs when reading values from a file, like a dataset.
By default, replicates pd.to_datetime formatting behavior.

### class datarobot.models.deployment.custom_metrics.BatchField

A custom metric batch ID source for when reading values from a columnar dataset like a file.

### class datarobot.models.deployment.custom_metrics.HostedCustomMetricBlueprint

Hosted custom metric blueprints provide an option to share custom metric settings between multiple
custom metrics sharing the same custom jobs. When a custom job of a hosted custom metric type is connected to the
deployment, all the custom metric parameters from the blueprint are automatically copied.

#### classmethod get(custom_job_id)

Get a hosted custom metric blueprint.

- Parameters: custom_job_id ( str ) – ID of the custom job.
- Returns: blueprint
- Return type: HostedCustomMetricBlueprint

#### classmethod create(custom_job_id, directionality, units, type, time_step, is_model_specific, is_geospatial=None)

Create a hosted custom metric blueprint.

- Parameters:
- Returns: blueprint
- Return type: HostedCustomMetricBlueprint

#### update(directionality=None, units=None, type=None, time_step=None, is_model_specific=None, is_geospatial=None)

Update a hosted custom metric blueprint.

- Parameters:
- Returns: updated_blueprint
- Return type: HostedCustomMetricBlueprint

### class datarobot.models.deployment.custom_metrics.CustomMetricValuesOverSpace

Custom metric values over space.

Added in version v3.7.

- Variables:

#### classmethod get(deployment_id, custom_metric_id, start=None, end=None, model_package_id=None, model_id=None)

Retrieve custom metric values over space.

- Parameters:
- Returns: values_over_space – Custom metric values over geospatial hexagons.
- Return type: CustomMetricValuesOverSpace

---
