NextGen experience > Registry > Jobs workshop > Create custom jobs > Create a hosted metric job

Create a hosted metric job¶

Add a custom job, manually or from a template, for a hosted metric, defining the metric settings and associating the metric with a deployment. To view and add hosted metric jobs, navigate to the Jobs > Metrics tab, and then:

To add a new hosted metric job manually, click + Add new hosted metric job (or the minimized add button when the job panel is open).
To create a hosted metric job from a template, next to the add button, click , and then, under Custom Metric, click Create new from template.

The new job opens to the Assemble tab. Depending on the creation option you selected, proceed to the configuration steps linked in the table below.

Hosted metric job type	Description
Add new hosted metric job	Manually add a job for a new hosted metric, defining the metric settings and associating the metric with a deployment.
Create new from template	Add a job for a hosted metric from a template provided by DataRobot, associating the metric with a deployment and setting a baseline.

Add a new hosted metric job¶

To manually add a hosted metric job:

On the Assemble tab for the new hosted metric job, click the job name (or the edit icon ) to enter a new job name, and then click confirm .
In the Environment section, select a Base environment for the job.

In the Metadata section, configure the following custom metric job fields:

Field	Description
Name of y-axis (label)	A descriptive name for the dependent variable. This name appears on the custom metric's chart on the Custom Metric Summary dashboard.
Default interval	Determines the default interval used by the selected Aggregation type. Only HOUR is supported.
Aggregation type	Determines if the metric is calculated as a Sum, Average, or Gauge—a metric with a distinct value measured at single point in time.
Metric direction	Determines the directionality of the metric, which controls how changes to the metric are visualized. You can select Higher is better or Lower is better. For example, if you choose Lower is better a 10% decrease in the calculated value of your custom metric will be considered 10% better, displayed in green.
Is Model Specific	When enabled, this setting links the metric to the model with the Model Package ID (Registered Model Version ID) provided in the dataset. This setting influences when values are aggregated (or uploaded). For example: Model specific (enabled): Model accuracy metrics are model specific, so the values are aggregated completely separately. When you replace a model, the chart for your custom accuracy metric onlys show data for the days after the replacement. Not model specific (disabled): Revenue metrics aren't model specific, so the values are aggregated together. When you replace a model, the chart for your custom revenue metric doesn't change. This field can't be edited after you create the metric.

In the Files section, assemble the custom job. Drag files into the box, or use the options in this section to create or upload the files required to assemble a custom job:

Option Description

Choose from source / Upload Upload existing custom job files (run.sh, metadata.yaml, etc.) as Local Files or a Local Folder.

Create

Create a new file, empty or containing a template, and save it to the custom job:

Create run.sh: Creates a basic, editable example of an entry point file.
Create metadata.yaml: Creates a basic, editable example of a runtime parameters file.
Create README.md: Creates a basic, editable README file.
Create job.py: Creates a basic, editable Python job file to print runtime parameters and deployments.
Create example job: Combines all template files to create a basic, editable custom job. You can quickly configure the runtime parameters and run this example job.
Create blank file: Creates an empty file. Click the edit icon next to Untitled to provide a file name and extension, then add your custom contents. In the next step, it is possible to identify files created this way, with a custom name and content, as the entry point. After you configure the new file, click Save.

File replacement

If you add a new file with the same name as an existing file, when you click Save, the old file is replaced in the Files section.

In the Settings section, configure the Entry point shell (.sh) file for the job. If you've added a run.sh file, that file is the entry point; otherwise, you must select the entry point shell file from the drop-down list. The entry point file allows you to orchestrate multiple job files:

In the Resources section, next to the section header, click Edit and configure the following:

Preview

Custom job resource bundles are off by default. Contact your DataRobot representative or administrator for information on enabling this feature.

Feature flag: Enable Resource Bundles

Setting	Description
Resource bundle (preview)	Configure the resources the custom job uses to run.
Network access	Configure the egress traffic of the custom job. Under Network access, select one of the following: Public: The default setting. The custom job can access any fully qualified domain name (FQDN) in a public network to leverage third-party services. None: The custom job is isolated from the public network and cannot access third party services.

Default network access

For the Managed AI Platform, the Network access setting is set to Public by default and the setting is configurable. For the Self-Managed AI Platform, the Network access setting is set to None by default and the setting is restricted; however, an administrator can change this behavior during DataRobot platform configuration. Contact your DataRobot representative or administrator for more information.

(Optional) If you uploaded a metadata.yaml file, define the Runtime parameters. Click the edit icon for each key value row you want to configure.
(Optional) Configure additional Key values for Tags, Metrics, Training parameters, and Artifacts.
In the Connected deployments panel, click + Connect to deployment, define a Custom metric name, and select a Deployment ID to connect it to that deployment.
Edit the Custom metric name and select a Deployment ID, then, set a Baseline—the value used as a basis for comparison when calculating the x% better or x% worse values—and click Connect.

How many Deployments can I connect to a hosted custom metric job?

You can connect up to 10 deployments to a hosted custom metric job.

Connected deployments and runtime parameters

After you connect a deployment to a hosted custom metric job and schedule a run, you can't modify the metadata.yaml file for runtime parameters. You must disconnect all connected deployments to make any modifications to the metadata.yaml file.