Developer documentation > API reference > Python API client > Monitoring > Custom metrics

Custom metrics¶

CustomMetric¶

class datarobot.models.deployment.custom_metrics.CustomMetric

A DataRobot custom metric.

Added in version v3.4.

Variables

Attribute	Type	Description
`id`	`str`	The ID of the custom metric.
`deployment_id`	`str`	The ID of the deployment.
`name`	`str`	The name of the custom metric.
`units`	`str`	The units, or the y-axis label, of the given custom metric.
`baseline_values`	`BaselinesValues`	The baseline value used to add “reference dots” to the values over time chart.
`is_model_specific`	`bool`	Determines whether the metric is related to the model or deployment.
`type`	`CustomMetricAggregationType`	The aggregation type of the custom metric.
`directionality`	`CustomMetricDirectionality`	The directionality of the custom metric.
`time_step`	`CustomMetricBucketTimeStep`	Custom metric time bucket size.
`description`	`str`	A description of the custom metric.
`association_id`	`DatasetColumn`	A custom metric association_id column source when reading values from columnar dataset.
`timestamp`	`DatasetColumn`	A custom metric timestamp column source when reading values from columnar dataset.
`value`	`DatasetColumn`	A custom metric value source when reading values from columnar dataset.
`sample_count`	`DatasetColumn`	A custom metric sample source when reading values from columnar dataset.
`batch`	`str`	A custom metric batch ID source when reading values from columnar dataset.

create()¶

classmethod create()

Create a custom metric for a deployment

Parameters

Parameter	Type	Description
`name`	`str`	The name of the custom metric.
`deployment_id`	`str`	The id of the deployment.
`units`	`str`	The units, or the y-axis label, of the given custom metric.
`baseline_value`	`float`	The baseline value used to add “reference dots” to the values over time chart.
`is_model_specific`	`bool`	Determines whether the metric is related to the model or deployment.
`aggregation_type`	`CustomMetricAggregationType`	The aggregation type of the custom metric.
`directionality`	`CustomMetricDirectionality`	The directionality of the custom metric.
`time_step`	`CustomMetricBucketTimeStep`	Custom metric time bucket size.
`description`	`Optional[str]`	A description of the custom metric.
`value_column_name`	`Optional[str]`	A custom metric value column name when reading values from columnar dataset.
`sample_count_column_name`	`Optional[str]`	Points to a weight column name if users provide pre-aggregated metric values from columnar dataset.
`timestamp_column_name`	`Optional[str]`	A custom metric timestamp column name when reading values from columnar dataset.
`timestamp_format`	`Optional[str]`	A custom metric timestamp format when reading values from columnar dataset.
`batch_column_name`	`Optional[str]`	A custom metric batch ID column name when reading values from columnar dataset.
`is_geospatial`	`Optional[bool]`	Determines whether the metric is geospatial or not.
`geospatial_segment_attribute`	`Optional[str]`	The name of the geospatial segment attribute.

Returns

Returns	Description
The custom metric object.

Return type: CustomMetric

Examples

from datarobot.models.deployment import CustomMetric
from datarobot.enums import CustomMetricAggregationType, CustomMetricDirectionality

custom_metric = CustomMetric.create(
    deployment_id="5c939e08962d741e34f609f0",
    name="Sample metric",
    units="Y",
    baseline_value=12,
    is_model_specific=True,
    aggregation_type=CustomMetricAggregationType.AVERAGE,
    directionality=CustomMetricDirectionality.HIGHER_IS_BETTER
    )

get()¶

classmethod get()

Get a custom metric for a deployment

Parameters

Parameter	Type	Description
`deployment_id`	`str`	The ID of the deployment.
`custom_metric_id`	`str`	The ID of the custom metric.

Returns

Returns	Description
The custom metric object.

Return type: CustomMetric

Examples

from datarobot.models.deployment import CustomMetric

custom_metric = CustomMetric.get(
    deployment_id="5c939e08962d741e34f609f0",
    custom_metric_id="65f17bdcd2d66683cdfc1113"
)

custom_metric.id
>>>'65f17bdcd2d66683cdfc1113'

list()¶

classmethod list()

List all custom metrics for a deployment

Parameters

Parameter	Type	Description
`deployment_id`	`str`	The ID of the deployment.

Returns

Returns	Description
custom_metrics	A list of custom metrics objects.

Return type: list

Examples

from datarobot.models.deployment import CustomMetric

custom_metrics = CustomMetric.list(deployment_id="5c939e08962d741e34f609f0")
custom_metrics[0].id
>>>'65f17bdcd2d66683cdfc1113'

delete()¶

classmethod delete()

Delete a custom metric associated with a deployment.

Parameters

Parameter	Type	Description
`deployment_id`	`str`	The ID of the deployment.
`custom_metric_id`	`str`	The ID of the custom metric.

Return type: None

Examples

from datarobot.models.deployment import CustomMetric

CustomMetric.delete(
    deployment_id="5c939e08962d741e34f609f0",
    custom_metric_id="65f17bdcd2d66683cdfc1113"
)

update()¶

method update()

Update metadata of a custom metric

Parameters

Parameter	Type	Description
`name`	`Optional[str]`	The name of the custom metric.
`units`	`Optional[str]`	The units, or the y-axis label, of the given custom metric.
`baseline_value`	`Optional[float]`	The baseline value used to add “reference dots” to the values over time chart.
`aggregation_type`	`Optional[CustomMetricAggregationType]`	The aggregation type of the custom metric.
`directionality`	`Optional[CustomMetricDirectionality]`	The directionality of the custom metric.
`time_step`	`Optional[CustomMetricBucketTimeStep]`	Custom metric time bucket size.
`description`	`Optional[str]`	A description of the custom metric.
`value_column_name`	`Optional[str]`	A custom metric value column name when reading values from columnar dataset.
`sample_count_column_name`	`Optional[str]`	Points to a weight column name if users provide pre-aggregated metric values from columnar dataset.
`timestamp_column_name`	`Optional[str]`	A custom metric timestamp column name when reading values from columnar dataset.
`timestamp_format`	`Optional[str]`	A custom metric timestamp format when reading values from columnar dataset.
`batch_column_name`	`Optional[str]`	A custom metric batch ID column name when reading values from columnar dataset.

Returns

Returns	Description
The custom metric object.

Return type: CustomMetric

Examples

from datarobot.models.deployment import CustomMetric
from datarobot.enums import CustomMetricAggregationType, CustomMetricDirectionality

custom_metric = CustomMetric.get(
    deployment_id="5c939e08962d741e34f609f0",
    custom_metric_id="65f17bdcd2d66683cdfc1113"
)
custom_metric = custom_metric.update(
    deployment_id="5c939e08962d741e34f609f0",
    name="Sample metric",
    units="Y",
    baseline_value=12,
    is_model_specific=True,
    aggregation_type=CustomMetricAggregationType.AVERAGE,
    directionality=CustomMetricDirectionality.HIGHER_IS_BETTER
    )

unset_baseline()¶

method unset_baseline()

Unset the baseline value of a custom metric

Return type: None

Examples

from datarobot.models.deployment import CustomMetric
from datarobot.enums import CustomMetricAggregationType, CustomMetricDirectionality

custom_metric = CustomMetric.get(
    deployment_id="5c939e08962d741e34f609f0",
    custom_metric_id="65f17bdcd2d66683cdfc1113"
)
custom_metric.baseline_values
>>> [{'value': 12.0}]
custom_metric.unset_baseline()
custom_metric.baseline_values
>>> []

submit_values()¶

method submit_values()

Submit aggregated custom metrics values from JSON.

Parameters

Parameter	Type	Description
`data`	`pd.DataFrame` or `List[CustomMetricBucket]`	The data containing aggregated custom metric values.
`model_id`	`Optional[str]`	For a model metric: the ID of the associated champion/challenger model, used to update the metric values. For a deployment metric: the ID of the model is not needed.
`model_package_id`	`Optional[str]`	For a model metric: the ID of the associated champion/challenger model, used to update the metric values. For a deployment metric: the ID of the model package is not needed.
`dry_run`	`Optional[bool]`	Specifies whether or not metric data is submitted in production mode (where data is saved).

Return type: None

Examples

from datarobot.models.deployment import CustomMetric

custom_metric = CustomMetric.get(
    deployment_id="5c939e08962d741e34f609f0",
    custom_metric_id="65f17bdcd2d66683cdfc1113"
)

# data for values over time
data = [{
    'value': 12,
    'sample_size': 3,
    'timestamp': '2024-03-15T14:00:00'
}]

# data witch association ID
data = [{
    'value': 12,
    'sample_size': 3,
    'timestamp': '2024-03-15T14:00:00',
    'association_id': '65f44d04dbe192b552e752ed'
}]

# data for batches
data = [{
    'value': 12,
    'sample_size': 3,
    'batch': '65f44c93fedc5de16b673a0d'
}]

# for deployment specific metrics
custom_metric.submit_values(data=data)

# for model specific metrics pass model_package_id or model_id
custom_metric.submit_values(data=data, model_package_id="6421df32525c58cc6f991f25")

# dry run
custom_metric.submit_values(data=data, model_package_id="6421df32525c58cc6f991f25", dry_run=True)

submit_single_value()¶

method submit_single_value()

Submit a single custom metric value at the current moment.

Parameters

Parameter	Type	Description
`value`	`float`	Single numeric custom metric value.
`model_id`	`Optional[str]`	For a model metric: the ID of the associated champion/challenger model, used to update the metric values. For a deployment metric: the ID of the model is not needed.
`model_package_id`	`Optional[str]`	For a model metric: the ID of the associated champion/challenger model, used to update the metric values. For a deployment metric: the ID of the model package is not needed.
`dry_run`	`Optional[bool]`	Specifies whether or not metric data is submitted in production mode (where data is saved).
`segments`	`Optional[CustomMetricSegmentFromJSON]`	A list of segments for a custom metric used in segmented analysis.

Return type: None

Examples

from datarobot.models.deployment import CustomMetric

custom_metric = CustomMetric.get(
    deployment_id="5c939e08962d741e34f609f0",
    custom_metric_id="65f17bdcd2d66683cdfc1113"
)

# for deployment specific metrics
custom_metric.submit_single_value(value=121)

# for model specific metrics pass model_package_id or model_id
custom_metric.submit_single_value(value=121, model_package_id="6421df32525c58cc6f991f25")

# dry run
custom_metric.submit_single_value(value=121, model_package_id="6421df32525c58cc6f991f25", dry_run=True)

# for segmented analysis
segments = [{"name": "custom_seg", "value": "val_1"}]
custom_metric.submit_single_value(value=121, model_package_id="6421df32525c58cc6f991f25", segments=segments)

submit_values_from_catalog()¶

method submit_values_from_catalog()

Submit aggregated custom metrics values from dataset (AI catalog). The names of the columns in the dataset should correspond to the names of the columns that were defined in the custom metric. In addition, the format of the timestamps should also be the same as defined in the metric.

Parameters

Parameter	Type	Description
`dataset_id`	`str`	The ID of the source dataset.
`model_id`	`Optional[str]`	For a model metric: the ID of the associated champion/challenger model, used to update the metric values. For a deployment metric: the ID of the model is not needed.
`model_package_id`	`Optional[str]`	For a model metric: the ID of the associated champion/challenger model, used to update the metric values. For a deployment metric: the ID of the model package is not needed.
`batch_id`	`Optional[str]`	Specifies a batch ID associated with all values provided by this dataset, an alternative to providing batch IDs as a column within a dataset (at the record level).
`segments`	`Optional[CustomMetricSegmentFromDataset]`	A list of segments for a custom metric used in segmented analysis.
`geospatial`	`Optional[Geospatial]`	A geospatial column source when reading values from columnar dataset.

Return type: None

Examples

from datarobot.models.deployment import CustomMetric

custom_metric = CustomMetric.get(
    deployment_id="5c939e08962d741e34f609f0",
    custom_metric_id="65f17bdcd2d66683cdfc1113"
)

# for deployment specific metrics
custom_metric.submit_values_from_catalog(dataset_id="61093144cabd630828bca321")

# for model specific metrics pass model_package_id or model_id
custom_metric.submit_values_from_catalog(
    dataset_id="61093144cabd630828bca321",
    model_package_id="6421df32525c58cc6f991f25"
)

# for segmented analysis
segments = [{"name": "custom_seg", "column": "column_with_segment_values"}]
custom_metric.submit_values_from_catalog(
    dataset_id="61093144cabd630828bca321",
    model_package_id="6421df32525c58cc6f991f25",
    segments=segments
)

get_values_over_time()¶

method get_values_over_time()

Retrieve values of a single custom metric over a time period.

Parameters

Parameter	Type	Description
`start`	`datetime` or `str`	Start of the time period.
`end`	`datetime` or `str`	End of the time period.
`model_id`	`Optional[str]`	The ID of the model.
`model_package_id`	`Optional[str]`	The ID of the model package.
`bucket_size`	`Optional[str]`	Time duration of a bucket, in ISO 8601 time duration format.
`segment_attribute`	`Optional[str]`	The name of the segment on which segment analysis is being performed.
`segment_value`	`Optional[str]`	The value of the segment_attribute to segment on.

Returns

Returns	Description
custom_metric_over_time	The queried custom metric values over time information.

Return type: CustomMetricValuesOverTime

Examples

from datarobot.models.deployment import CustomMetric
from datetime import datetime, timedelta

now=datetime.now()
custom_metric = CustomMetric.get(
    deployment_id="5c939e08962d741e34f609f0",
    custom_metric_id="65f17bdcd2d66683cdfc1113"
)
values_over_time = custom_metric.get_values_over_time(start=now - timedelta(days=7), end=now)

values_over_time.bucket_values
>>> {datetime.datetime(2024, 3, 22, 14, 0, tzinfo=tzutc()): 1.0,
>>> datetime.datetime(2024, 3, 22, 15, 0, tzinfo=tzutc()): 123.0}}

values_over_time.bucket_sample_sizes
>>> {datetime.datetime(2024, 3, 22, 14, 0, tzinfo=tzutc()): 1,
>>>  datetime.datetime(2024, 3, 22, 15, 0, tzinfo=tzutc()): 1}}

values_over_time.get_buckets_as_dataframe()
>>>                        start                       end  value  sample_size
>>> 0  2024-03-21 16:00:00+00:00 2024-03-21 17:00:00+00:00    NaN          NaN
>>> 1  2024-03-21 17:00:00+00:00 2024-03-21 18:00:00+00:00    NaN          NaN

get_values_over_space()¶

method get_values_over_space()

Retrieve values of a custom metric over space.

Parameters

Parameter	Type	Description
`start`	`Optional[datetime]`	Start of the time period.
`end`	`Optional[datetime]`	End of the time period.
`model_id`	`Optional[str]`	The ID of the model.
`model_package_id`	`Optional[str]`	The ID of the model package.

Returns

Returns	Description
custom_metric_over_space	The queried custom metric values over space information.

Return type: CustomMetricValuesOverSpace

Examples

from datarobot.models.deployment import CustomMetric

custom_metric = CustomMetric.get(
    deployment_id="5c939e08962d741e34f609f0",
    custom_metric_id="65f17bdcd2d66683cdfc1113"
)
values_over_space = custom_metric.get_values_over_space(model_package_id='6421df32525c58cc6f991f25')

get_summary()¶

method get_summary()

Retrieve the summary of a custom metric over a time period.

Parameters

Parameter	Type	Description
`start`	`datetime` or `str`	Start of the time period.
`end`	`datetime` or `str`	End of the time period.
`model_id`	`Optional[str]`	The ID of the model.
`model_package_id`	`Optional[str]`	The ID of the model package.
`segment_attribute`	`Optional[str]`	The name of the segment on which segment analysis is being performed.
`segment_value`	`Optional[str]`	The value of the segment_attribute to segment on.

Returns

Returns	Description
custom_metric_summary	The summary of the custom metric.

Return type: CustomMetricSummary

Examples

from datarobot.models.deployment import CustomMetric
from datetime import datetime, timedelta

now=datetime.now()
custom_metric = CustomMetric.get(
    deployment_id="5c939e08962d741e34f609f0",
    custom_metric_id="65f17bdcd2d66683cdfc1113"
)
summary = custom_metric.get_summary(start=now - timedelta(days=7), end=now)

print(summary)
>> "CustomMetricSummary(2024-03-21 15:52:13.392178+00:00 - 2024-03-22 15:52:13.392168+00:00:
{'id': '65fd9b1c0c1a840bc6751ce0', 'name': 'Test METRIC', 'value': 215.0, 'sample_count': 13,
'baseline_value': 12.0, 'percent_change': 24.02})"

get_values_over_batch()¶

method get_values_over_batch()

Retrieve values of a single custom metric over batches.

Parameters

Parameter	Type	Description
`batch_ids`	`Optional[List[str]]`	Specify a list of batch IDs to pull the data for.
`model_id`	`Optional[str]`	The ID of the model.
`model_package_id`	`Optional[str]`	The ID of the model package.
`segment_attribute`	`Optional[str]`	The name of the segment on which segment analysis is being performed.
`segment_value`	`Optional[str]`	The value of the segment_attribute to segment on.

Returns

Returns	Description
custom_metric_over_batch	The queried custom metric values over batch information.

Return type: CustomMetricValuesOverBatch

Examples

   from datarobot.models.deployment import CustomMetric

   custom_metric = CustomMetric.get(
       deployment_id="5c939e08962d741e34f609f0",
       custom_metric_id="65f17bdcd2d66683cdfc1113"
   )
   # all batch metrics all model specific
   values_over_batch = custom_metric.get_values_over_batch(model_package_id='6421df32525c58cc6f991f25')

   values_over_batch.bucket_values
   >>> {'6572db2c9f9d4ad3b9de33d0': 35.0, '6572db2c9f9d4ad3b9de44e1': 105.0}

   values_over_batch.bucket_sample_sizes
   >>> {'6572db2c9f9d4ad3b9de33d0': 6, '6572db2c9f9d4ad3b9de44e1': 8}

values_over_batch.get_buckets_as_dataframe()
>>>                    batch_id                     batch_name  value  sample_size
>>> 0  6572db2c9f9d4ad3b9de33d0  Batch 1 - 03/26/2024 13:04:46   35.0            6
>>> 1  6572db2c9f9d4ad3b9de44e1  Batch 2 - 03/26/2024 13:06:04  105.0            8

get_batch_summary()¶

method get_batch_summary()

Retrieve the summary of a custom metric over a batch.

Parameters

Parameter	Type	Description
`batch_ids`	`Optional[List[str]]`	Specify a list of batch IDs to pull the data for.
`model_id`	`Optional[str]`	The ID of the model.
`model_package_id`	`Optional[str]`	The ID of the model package.
`segment_attribute`	`Optional[str]`	The name of the segment on which segment analysis is being performed.
`segment_value`	`Optional[str]`	The value of the segment_attribute to segment on.

Returns

Returns	Description
custom_metric_summary	The batch summary of the custom metric.

Return type: CustomMetricBatchSummary

Examples

from datarobot.models.deployment import CustomMetric

custom_metric = CustomMetric.get(
    deployment_id="5c939e08962d741e34f609f0",
    custom_metric_id="65f17bdcd2d66683cdfc1113"
)
# all batch metrics all model specific
batch_summary = custom_metric.get_batch_summary(model_package_id='6421df32525c58cc6f991f25')

print(batch_summary)
>> CustomMetricBatchSummary({'id': '6605396413434b3a7b74342c', 'name': 'batch metric', 'value': 41.25,
'sample_count': 28, 'baseline_value': 123.0, 'percent_change': -66.46})

CustomMetricValuesOverTime¶

class datarobot.models.deployment.custom_metrics.CustomMetricValuesOverTime

Custom metric over time information.

Added in version v3.4.

Variables

Attribute	Type	Description
`buckets`	`List[Bucket]`	A list of bucketed time periods and the custom metric values aggregated over that period.
`summary`	`Summary`	The summary of values over time retrieval.
`metric`	`Dict`	A custom metric definition.
`deployment_id`	`str`	The ID of the deployment.
`segment_attribute`	`str`	The name of the segment on which segment analysis is being performed.
`segment_value`	`str`	The value of the segment_attribute to segment on.

get()¶

classmethod get()

Retrieve values of a single custom metric over a time period.

Parameters

Parameter	Type	Description
`custom_metric_id`	`str`	The ID of the custom metric.
`deployment_id`	`str`	The ID of the deployment.
`start`	`datetime` or `str`	Start of the time period.
`end`	`datetime` or `str`	End of the time period.
`model_id`	`Optional[str]`	The ID of the model.
`model_package_id`	`Optional[str]`	The ID of the model package.
`bucket_size`	`Optional[str]`	Time duration of a bucket, in ISO 8601 time duration format.
`segment_attribute`	`Optional[str]`	The name of the segment on which segment analysis is being performed.
`segment_value`	`Optional[str]`	The value of the segment_attribute to segment on.

Returns

Returns	Description
custom_metric_over_time	The queried custom metric values over time information.

Return type: CustomMetricValuesOverTime

bucket_values¶

property bucket_values

The metric value for all time buckets, keyed by start time of the bucket.

Returns

Returns	Description
bucket_values

Return type: Dict

bucket_sample_sizes¶

property bucket_sample_sizes

The sample size for all time buckets, keyed by start time of the bucket.

Returns

Returns	Description
bucket_sample_sizes

Return type: Dict

get_buckets_as_dataframe()¶

method get_buckets_as_dataframe()

Retrieves all custom metrics buckets in a pandas DataFrame.

Returns

Returns	Description
buckets

Return type: pd.DataFrame

CustomMetricSummary¶

class datarobot.models.deployment.custom_metrics.CustomMetricSummary

The summary of a custom metric.

Added in version v3.4.

Variables

Attribute	Type	Description
`period`	`Period`	A time period defined by a start and end tie
`metric`	`Dict`	The summary of the custom metric.

get()¶

classmethod get()

Retrieve the summary of a custom metric over a time period.

Parameters

Parameter	Type	Description
`custom_metric_id`	`str`	The ID of the custom metric.
`deployment_id`	`str`	The ID of the deployment.
`start`	`datetime` or `str`	Start of the time period.
`end`	`datetime` or `str`	End of the time period.
`model_id`	`Optional[str]`	The ID of the model.
`model_package_id`	`Optional[str]`	The ID of the model package.
`segment_attribute`	`Optional[str]`	The name of the segment on which segment analysis is being performed.
`segment_value`	`Optional[str]`	The value of the segment_attribute to segment on.

Returns

Returns	Description
custom_metric_summary	The summary of the custom metric.

Return type: CustomMetricSummary

CustomMetricValuesOverBatch¶

class datarobot.models.deployment.custom_metrics.CustomMetricValuesOverBatch

Custom metric over batch information.

Added in version v3.4.

Variables

Attribute	Type	Description
`buckets`	`List[BatchBucket]`	A list of buckets with custom metric values aggregated over batches.
`metric`	`Dict`	A custom metric definition.
`deployment_id`	`str`	The ID of the deployment.
`segment_attribute`	`str`	The name of the segment on which segment analysis is being performed.
`segment_value`	`str`	The value of the segment_attribute to segment on.

get()¶

classmethod get()

Retrieve values of a single custom metric over batches.

Parameters

Parameter	Type	Description
`custom_metric_id`	`str`	The ID of the custom metric.
`deployment_id`	`str`	The ID of the deployment.
`batch_ids`	`Optional[List[str]]`	Specify a list of batch IDs to pull the data for.
`model_id`	`Optional[str]`	The ID of the model.
`model_package_id`	`Optional[str]`	The ID of the model package.
`segment_attribute`	`Optional[str]`	The name of the segment on which segment analysis is being performed.
`segment_value`	`Optional[str]`	The value of the segment_attribute to segment on.

Returns

Returns	Description
custom_metric_over_batch	The queried custom metric values over batch information.

Return type: CustomMetricValuesOverBatch

bucket_values¶

property bucket_values

The metric value for all batch buckets, keyed by batch ID

Returns

Returns	Description
bucket_values

Return type: Dict

bucket_sample_sizes¶

property bucket_sample_sizes

The sample size for all batch buckets, keyed by batch ID.

Returns

Returns	Description
bucket_sample_sizes

Return type: Dict

get_buckets_as_dataframe()¶

method get_buckets_as_dataframe()

Retrieves all custom metrics buckets in a pandas DataFrame.

Returns

Returns	Description
buckets

Return type: pd.DataFrame

CustomMetricBatchSummary¶

class datarobot.models.deployment.custom_metrics.CustomMetricBatchSummary

The batch summary of a custom metric.

Added in version v3.4.

Variables

Attribute	Type	Description
`metric`	`Dict`	The summary of the batch custom metric.

get()¶

classmethod get()

Retrieve the summary of a custom metric over a batch.

Parameters

Parameter	Type	Description
`custom_metric_id`	`str`	The ID of the custom metric.
`deployment_id`	`str`	The ID of the deployment.
`batch_ids`	`Optional[List[str]]`	Specify a list of batch IDs to pull the data for.
`model_id`	`Optional[str]`	The ID of the model.
`model_package_id`	`Optional[str]`	The ID of the model package.
`segment_attribute`	`Optional[str]`	The name of the segment on which segment analysis is being performed.
`segment_value`	`Optional[str]`	The value of the segment_attribute to segment on.

Returns

Returns	Description
custom_metric_summary	The batch summary of the custom metric.

Return type: CustomMetricBatchSummary

HostedCustomMetricTemplate¶

class datarobot.models.deployment.custom_metrics.HostedCustomMetricTemplate

Template for hosted custom metric.

list()¶

classmethod list()

List all hosted custom metric templates.

Parameters

Parameter	Type	Description
`search`	`Optional[str]`	Search string.
`order_by`	`Optional[ListHostedCustomMetricTemplatesSortQueryParams]`	Ordering field.
`metric_type`	`Optional[HostedCustomMetricsTemplateMetricTypeQueryParams]`	Type of the metric.
`offset`	`Optional[int]`	Offset for pagination.
`limit`	`Optional[int]`	Limit for pagination.

Returns

Returns	Description
templates

Return type: List[HostedCustomMetricTemplate]

get()¶

classmethod get()

Get a hosted custom metric template by ID.

Parameters

Parameter	Type	Description
`template_id`	`str`	ID of the template.

Returns

Returns	Description
template

Return type: HostedCustomMetricTemplate

HostedCustomMetric¶

class datarobot.models.deployment.custom_metrics.HostedCustomMetric

Hosted custom metric.

list()¶

classmethod list()

List all hosted custom metrics for a job.

Parameters

Parameter	Type	Description
`job_id`	`str`	ID of the job.

Returns

Returns	Description
metrics

Return type: List[HostedCustomMetric]

create_from_template()¶

classmethod create_from_template()

Create a hosted custom metric from a template. A shortcut for 2 calls: Job.from_custom_metric_template(template_id) HostedCustomMetrics.create_from_custom_job()

Parameters

Parameter	Type	Description
`template_id`	`str`	ID of the template.
`deployment_id`	`str`	ID of the deployment.
`job_name`	`str`	Name of the job.
`custom_metric_name`	`str`	Name of the metric.
`job_description`	`Optional[str]`	Description of the job.
`custom_metric_description`	`Optional[str]`	Description of the metric.
`sidecar_deployment_id`	`Optional[str]`	ID of the sidecar deployment.
`baseline_value`	`Optional[float]`	Baseline value.
`timestamp`	`Optional[MetricTimestampSpoofing]`	Timestamp details.
`value`	`Optional[ValueField]`	Value details.
`sample_count`	`Optional[SampleCountField]`	Sample count details.
`batch`	`Optional[BatchField]`	Batch details.
`schedule`	`Optional[Schedule]`	Schedule details.
`parameter_overrides`	`Optional[List[RuntimeParameterValue]]`	Parameter overrides.

Returns

Returns	Description
metric

Return type: HostedCustomMetric

create_from_custom_job()¶

classmethod create_from_custom_job()

Create a hosted custom metric from existing custom job.

Parameters

Parameter	Type	Description
`custom_job_id`	`str`	ID of the custom job.
`deployment_id`	`str`	ID of the deployment.
`name`	`str`	Name of the metric.
`description`	`Optional[str]`	Description of the metric.
`baseline_value`	`Optional[float]`	Baseline value.
`timestamp`	`Optional[MetricTimestampSpoofing]`	Timestamp details.
`value`	`Optional[ValueField]`	Value details.
`sample_count`	`Optional[SampleCountField]`	Sample count details.
`batch`	`Optional[BatchField]`	Batch details.
`schedule`	`Optional[Schedule]`	Schedule details.
`parameter_overrides`	`Optional[List[RuntimeParameterValue]]`	Parameter overrides.
`geospatial_segment_attribute`	`Optional[str]`	The name of the geospatial segment attribute. Only applicable for geospatial custom metrics.

Returns

Returns	Description
metric

Return type: HostedCustomMetric

update()¶

method update()

Update the hosted custom metric.

Parameters

Parameter	Type	Description
`name`	`Optional[str]`	Name of the metric.
`description`	`Optional[str]`	Description of the metric.
`units`	`Optional[str]`	Units of the metric.
`directionality`	`Optional[str]`	Directionality of the metric.
`aggregation_type`	`Optional[CustomMetricAggregationType]`	Aggregation type of the metric.
`baseline_value`	`Optional[float]`	Baseline values.
`timestamp`	`Optional[MetricTimestampSpoofing]`	Timestamp details.
`value`	`Optional[ValueField]`	Value details.
`sample_count`	`Optional[SampleCountField]`	Sample count details.
`batch`	`Optional[BatchField]`	Batch details.
`schedule`	`Optional[Schedule]`	Schedule details.
`parameter_overrides`	`Optional[List[RuntimeParameterValue]]`	Parameter overrides.

Returns

Returns	Description
metric

Return type: HostedCustomMetric

delete()¶

method delete()

Delete the hosted custom metric.

Return type: None

DeploymentDetails¶

class datarobot.models.deployment.custom_metrics.DeploymentDetails

Information about a hosted custom metric deployment.

MetricBaselineValue¶

class datarobot.models.deployment.custom_metrics.MetricBaselineValue

The baseline values for a custom metric.

SampleCountField¶

class datarobot.models.deployment.custom_metrics.SampleCountField

A weight column used with columnar datasets if pre-aggregated metric values are provided.

ValueField¶

class datarobot.models.deployment.custom_metrics.ValueField

A custom metric value source for when reading values from a columnar dataset like a file.

MetricTimestampSpoofing¶

class datarobot.models.deployment.custom_metrics.MetricTimestampSpoofing

Custom metric timestamp spoofing. Occurs when reading values from a file, like a dataset. By default, replicates pd.to_datetime formatting behavior.

BatchField¶

class datarobot.models.deployment.custom_metrics.BatchField

A custom metric batch ID source for when reading values from a columnar dataset like a file.

HostedCustomMetricBlueprint¶

class datarobot.models.deployment.custom_metrics.HostedCustomMetricBlueprint

Hosted custom metric blueprints provide an option to share custom metric settings between multiple custom metrics sharing the same custom jobs. When a custom job of a hosted custom metric type is connected to the deployment, all the custom metric parameters from the blueprint are automatically copied.

get()¶

classmethod get()

Get a hosted custom metric blueprint.

Parameters

Parameter	Type	Description
`custom_job_id`	`str`	ID of the custom job.

Returns

Returns	Description
blueprint

Return type: HostedCustomMetricBlueprint

create()¶

classmethod create()

Create a hosted custom metric blueprint.

Parameters

Parameter	Type	Description
`custom_job_id`	`str`	ID of the custom job.
`directionality`	`str`	Directionality of the metric.
`units`	`str`	Units of the metric.
`type`	`str`	Type of the metric.
`time_step`	`str`	Time step of the metric.
`is_model_specific`	`bool`	Whether the metric is model specific.
`is_geospatial`	`Optional[bool]`	Determines whether the metric is geospatial.

Returns

Returns	Description
blueprint

Return type: HostedCustomMetricBlueprint

update()¶

method update()

Update a hosted custom metric blueprint.

Parameters

Parameter	Type	Description
`directionality`	`Optional[str]`	Directionality of the metric.
`units`	`Optional[str]`	Units of the metric.
`type`	`Optional[str]`	Type of the metric.
`time_step`	`Optional[str]`	Time step of the metric.
`is_model_specific`	`Optional[bool]`	Determines whether the metric is model specific.
`is_geospatial`	`Optional[bool]`	Determines whether the metric is geospatial.

Returns

Returns	Description
updated_blueprint

Return type: HostedCustomMetricBlueprint

CustomMetricValuesOverSpace¶

class datarobot.models.deployment.custom_metrics.CustomMetricValuesOverSpace

Custom metric values over space.

Added in version v3.7.

Variables

Attribute	Type	Description
`buckets`	`List[BatchBucket]`	A list of buckets with custom metric values aggregated over geospatial hexagons.
`metric`	`Dict`	A custom metric definition.
`model_id`	`str`	The ID of the model.
`model_package_id`	`str`	The ID of the model package (also known as registered model version id).
`summary`	`Dict`	Start-end interval over which data is retrieved.

get()¶

classmethod get()

Retrieve custom metric values over space.

Parameters

Parameter	Type	Description
`deployment_id`	`str`	The id of the deployment.
`custom_metric_id`	`str`	The id of the custom metric.
`start`	`datetime`	The start time of the interval.
`end`	`datetime`	The end time of the interval.
`model_package_id`	`str`	The id of the model package.
`model_id`	`str`	The id of the model.

Returns

Returns	Description
values_over_space	Custom metric values over geospatial hexagons.

Return type: CustomMetricValuesOverSpace