Use the DR Custom Metrics module¶

The DR Custom Metrics module facilitates better synchronization with existing metrics in DataRobot. The logic of this module is based on unique names for custom metrics, allowing you to operate on metrics without knowing their IDs. Using this solution, you can define the metric earlier (e.g., before creating the deployment) and synchronize it with DataRobot at the appropriate time.

The DRCustomMetric class allows you to create new or fetch existing metrics from DataRobot. You can provide custom metrics configuration through YAML, JSON, or a dict. The configuration contains metadata describing the custom metric.

The DRCustomMetric class contains two methods:

• DRCustomMetric.sync(): Retrieves information about existing custom metrics in DataRobot. If a metric is defined locally but not in DataRobot, it is created in DataRobot.

• DRCustomMetric.report(): Reports a single value based on a unique name.

dr_cm = DRCustomMetric(
    dr_client=client, deployment_id=deployment_id, model_package_id=model_package_id
)

metric_config_yaml = f"""
     customMetrics:
       - name: new metric
         description: metric description
         type: average
         timeStep: hour
         units: count
         directionality: lowerIsBetter
         isModelSpecific: yes
         baselineValue: 0
     """

dr_cm.set_config(config_yaml=metric_config_yaml)
dr_cm.sync()
dr_cm.get_dr_custom_metrics()
> [{"name": "existing metric", "id": "65ef19410239ff8015f05a94", ...}, 
>  {"name": "new metric", "id": "65ef197ce5d7b2176ceecf3a", ...}]

dr_cm.report_value("existing metric", 1)
dr_cm.report_value("new metric", 9)