Create a hosted custom metric from a template with code¶
Hosted custom metrics run user-provided code on DataRobot infrastructure to calculate your organization's customized metrics. DataRobot provides a variety of templates for common metrics. These metrics can be used as is, or as a starting point for user-provided metrics. In this tutorial, you create a hosted custom metric using the Python SDK.
前提条件¶
Before you start, import all objects used in this tutorial and initialize the DataRobot client:
import datarobot as dr
from datarobot.enums import HostedCustomMetricsTemplateMetricTypeQueryParams
from datarobot.models.deployment.custom_metrics import HostedCustomMetricTemplate, HostedCustomMetric, \
HostedCustomMetricBlueprint, CustomMetric, MetricTimestampSpoofing, ValueField, SampleCountField, BatchField
from datarobot.models.registry import JobRun
from datarobot.models.registry.job import Job
from datarobot import Deployment
from datarobot.models.runtime_parameters import RuntimeParameterValue
from datarobot.models.types import Schedule
dr.Client(token="<DataRobot API Token>", endpoint="<DataRobot URL>")
gen_ai_deployment_1 = Deployment.get('<Deployment Id>')
List hosted custom metrics templates¶
Before creating a hosted custom metric from a template, retrieve the LLM metric template to use as the basis of the new metric. To do this, specify the metric_type and, because the deployments are LLM models handling Japanese text, search for the specific metric by name, limiting the search to 1 result. Store the result in templates for the next step.
templates = HostedCustomMetricTemplate.list(
search="[JP] Character Count",
metric_type=HostedCustomMetricsTemplateMetricTypeQueryParams.LLM,
limit=1,
offset=0,
)
Create a hosted custom metric in one step¶
Locate the custom metric template to create a hosted custom metric. This method is a shortcut, combining two steps to create a new custom metric from the retrieved template:
- Create a custom job for a hosted custom metric from the template retrieved in the previous step (stored in
templates). - Connect the hosted custom metric job to the deployment defined during the prerequisites step (stored in
gen_ai_deployment_1).
Because we are creating two objects, specify both the job name and custom metric name in addition to the template and deployment IDs. Additionally, define the job schedule and the runtime parameter overrides for the deployment.
hosted_custom_metric = HostedCustomMetric.create_from_template(
template_id=templates[0].id,
deployment_id=gen_ai_deployment_1.id,
job_name="Hosted Custom Metric Character Count",
custom_metric_name="Character Count",
job_description="Hosted Custom Metric",
custom_metric_description="LLM Character Count",
baseline_value=10,
timestamp=MetricTimestampSpoofing(
column_name="timestamp",
time_format="%Y-%m-%d %H:%M:%S",
),
value = ValueField(column_name="value"),
sample_count=SampleCountField(column_name='Sample Count'),
batch=BatchField(column_name='Batch'),
schedule=Schedule(
day_of_week=[0],
hour=['*'],
minute=['*'],
day_of_month=[12],
month=[1],
),
parameter_overrides=[RuntimeParameterValue(field_name='DRY_RUN', value="0", type="string")]
)
Once you create the hosted custom metric, initiate the manual run.
job_run = JobRun.create(
job_id=hosted_custom_metric.custom_job_id
runtime_parameter_values=[
RuntimeParameterValue(field_name='DRY_RUN', value="1", type="string"),
RuntimeParameterValue(field_name='DEPLOYMENT_ID', value=gen_ai_deployment_1.id, type="deployment"),
RuntimeParameterValue(field_name='CUSTOM_METRIC_ID', value=hosted_custom_metric.id, type="customMetric"),
]
)
print(job_run.status)
Create a hosted custom metric in two steps¶
You can also perform these steps manually, in sequence. This is useful if you want to edit the custom metric blueprint before attaching the custom job to the deployment. When you attach the job to the deployment, most settings are copied from the blueprint (unless you provide an override). To create the hosted custom metric manually, first, create a custom job from the template retrieved earlier (stored in templates).
job = Job.create_from_custom_metric_gallery_template(
template_id=templates[0].id,
name="Job created from template",
description="Job created from template"
)
Next, retrieve the default blueprint, provided by the template, and edit it.
blueprint = HostedCustomMetricBlueprint.get(job.id)
print(f"Original directionality: {blueprint.directionality}")
Then, update the parameters of the custom metric in the blueprint.
updated_blueprint = blueprint.update(
directionality='lowerIsBetter',
units='characters',
type='gauge',
time_step='hour',
is_model_specific=False
)
print(f"Updated directionality: {updated_blueprint.directionality}")
Now create the hosted custom metric. As in the shortcut method, you can provide the job schedule, runtime parameter overrides, and custom metric parameters specific to this deployment.
another_hosted_custom_metric = HostedCustomMetric.create_from_custom_job(
custom_job_id=job.id,
deployment_id=gen_ai_deployment_1.id,
name="Custom metric created in 2 steps",
)
After creating and configuring the metric, verify that the changes to the blueprint are reflected in the custom metric.
another_custom_metric = CustomMetric.get(custom_metric_id=another_hosted_custom_metric.id, deployment_id=gen_ai_deployment_1.id)
print(f"Directionality of another custom metric: {another_custom_metric.directionality}")
Finally, create a manual job run for the custom metric job.
job_run = JobRun.create(
job_id=job.id,
runtime_parameter_values=[
RuntimeParameterValue(field_name='DRY_RUN', value="1", type="string"),
RuntimeParameterValue(field_name='DEPLOYMENT_ID', value=gen_ai_deployment_1.id, type="deployment"),
RuntimeParameterValue(field_name='CUSTOM_METRIC_ID', value=another_hosted_custom_metric.id, type="customMetric"),
]
)
print(job_run.status)
List hosted custom metrics associated with a job¶
To list all hosted custom metrics associated with a custom job, use the following code:
hosted_custom_metrics = HostedCustomMetric.list(deployment_id=hosted_custom_metric.custom_job_id)
for metric in hosted_custom_metrics:
print(metric.name)
Delete hosted custom metrics¶
You can delete the hosted custom metric, removing the custom metric from the deployment but keeping the job, allowing you to create the metric for another deployment.
hosted_custom_metric.delete()
another_hosted_custom_metric.delete()
If necessary, you can delete the entire custom job. If there are any custom metrics associated with that job, they are also deleted.
job.delete()