# Create a hosted metric job

> Create a hosted metric job - How to add a custom job, manually or from a template, for a hosted
> metric, defining the metric settings and associating the metric with a deployment.

This Markdown file sits beside the HTML page at the same path (with a `.md` suffix). It summarizes the topic and lists links for tools and LLM context.

Companion generated at `2026-05-06T18:17:10.042250+00:00` (UTC).

## Primary page

- [Create a hosted metric job](https://docs.datarobot.com/en/docs/workbench/nxt-registry/nxt-jobs-workshop/nxt-create-jobs/nxt-create-hosted-metric-job.html): Full documentation for this topic (HTML).

## Sections on this page

- [Add a new hosted metric job](https://docs.datarobot.com/en/docs/workbench/nxt-registry/nxt-jobs-workshop/nxt-create-jobs/nxt-create-hosted-metric-job.html#add-a-new-hosted-metric-job): In-page section heading.
- [Create a hosted metric job from a template](https://docs.datarobot.com/en/docs/workbench/nxt-registry/nxt-jobs-workshop/nxt-create-jobs/nxt-create-hosted-metric-job.html#create-a-hosted-metric-job-from-a-template): In-page section heading.

## Related documentation

- [NextGen UI documentation](https://docs.datarobot.com/en/docs/workbench/index.html): Linked from this page.
- [Registry](https://docs.datarobot.com/en/docs/workbench/nxt-registry/index.html): Linked from this page.
- [Jobs](https://docs.datarobot.com/en/docs/workbench/nxt-registry/nxt-jobs-workshop/index.html): Linked from this page.
- [Create custom jobs](https://docs.datarobot.com/en/docs/workbench/nxt-registry/nxt-jobs-workshop/nxt-create-jobs/index.html): Linked from this page.
- [define runtime parameters in ametadata.yamlfile](https://docs.datarobot.com/en/docs/workbench/nxt-registry/nxt-jobs-workshop/nxt-runtime-parameters-custom-jobs.html): Linked from this page.
- [Key values](https://docs.datarobot.com/en/docs/workbench/nxt-registry/nxt-jobs-workshop/nxt-key-values-custom-jobs.html): Linked from this page.
- [geospatial/location feature](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-data-drift.html#enable-geospatial-monitoring-for-a-deployment): Linked from this page.
- [global model](https://docs.datarobot.com/en/docs/workbench/nxt-registry/nxt-model-directory/nxt-global-models.html): Linked from this page.

## Documentation content

Add a custom job, manually or from a template, for a hosted metric, defining the metric settings and associating the metric with a deployment. To view and add hosted metric jobs, navigate to the Jobs > Metrics tab, and then:

- To add a new hosted metric job manually, click+ Add new hosted metric job(or the minimized add buttonwhen the job panel is open).
- To create a hosted metric job from a template, next to the add button, click, and then, underCustom Metric, clickCreate new from template.

The new job opens to the Assemble tab. Depending on the creation option you selected, proceed to the configuration steps linked in the table below.

| Hosted metric job type | Description |
| --- | --- |
| Add new hosted metric job | Manually add a job for a new hosted metric, defining the metric settings and associating the metric with a deployment. |
| Create new from template | Add a job for a hosted metric from a template provided by DataRobot, associating the metric with a deployment and setting a baseline. |

## Add a new hosted metric job

To manually add a hosted metric job:

1. On theAssembletab for the new hosted metric job, click the job name (or the edit icon) to enter a new job name, and then click confirm.
2. In theEnvironmentsection, select aBase environmentfor the job. The available drop-in environments depend on your DataRobot installation; however, the table below lists commonly available public drop-in environments withtemplates in the DRUM repository. Depending on your DataRobot installation, the Python version of these environments may vary, and additional non-public environments may be available for use. Managed AI Platform (SaaS)Self-Managed AI PlatformDrop-in environment securityStarting with the March 2025 Managed AI Platform release, most general purpose DataRobot custom model drop-in environments are security-hardened container images. When you require a security-hardened environment for running custom jobs, only shell code following thePOSIX-shell standardis supported. Security-hardened environments following the POSIX-shell standard support a limited set of shell utilities.Drop-in environment securityStarting with the 11.0 Self-Managed AI Platform release, most general purpose DataRobot custom model drop-in environments are security-hardened container images. When you require a security-hardened environment for running custom jobs, only shell code following thePOSIX-shell standardis supported. Security-hardened environments following the POSIX-shell standard support a limited set of shell utilities. Environment name & exampleCompatibility & artifact file extensionPython 3.XPython-based custom models and jobs. You are responsible for installing all required dependencies through the inclusion of arequirements.txtfile in your model files.Python 3.X GenAI AgentsGenerative AI models (Text GenerationorVector Databasetarget type)Python 3.X ONNX Drop-InONNX models and jobs (.onnx)Python 3.X PMML Drop-InPMML models and jobs (.pmml)Python 3.X PyTorch Drop-InPyTorch models and jobs (.pth)Python 3.X Scikit-Learn Drop-InScikit-Learn models and jobs (.pkl)Python 3.X XGBoost Drop-InNative XGBoost models and jobs (.pkl)Python 3.X Keras Drop-InKeras models and jobs backed by tensorflow (.h5)Java Drop-InDataRobot Scoring Code models (.jar)R Drop-in EnvironmentR models trained using CARET (.rds)Due to the time required to install all libraries recommended by CARET, only model types that are also package names are installed (e.g.,brnn,glmnet). Make a copy of this environment and modify the Dockerfile to install the additional, required packages. To decrease build times when you customize this environment, you can also remove unnecessary lines in the# Install caret modelssection, installing only what you need. Review theCARET documentationto check if your model's method matches its package name. (Log in to GitHub before clicking this link.) scikit-learnAll Python environments contain scikit-learn to help with preprocessing (if necessary), but only scikit-learn can make predictions onsklearnmodels.
3. In theMetadatasection, configure the following custom metric job fields: FieldDescriptionName of y-axis (label)A descriptive name for the dependent variable. This name appears on the custom metric's chart on theCustom Metric Summarydashboard.Default intervalDetermines the default interval used by the selectedAggregation type. OnlyHOURis supported.Aggregation typeDetermines if the metric is calculated as aSum,Average, orGauge—a metric with a distinct value measured at single point in time.Metric directionDetermines the directionality of the metric, which controls how changes to the metric are visualized. You can selectHigher is betterorLower is better. For example, if you chooseLower is bettera 10% decrease in the calculated value of your custom metric will be considered10% better, displayed in green.Is Model SpecificWhen enabled, this setting links the metric to the model with theModel Package ID(Registered Model Version ID) provided in the dataset. This setting influences when values are aggregated (or uploaded). For example:Model specific (enabled): Model accuracy metrics are model specific, so the values are aggregated completely separately. When you replace a model, the chart for your custom accuracy metric only shows data for the days after the replacement.Not model specific (disabled): Revenue metrics aren't model specific, so the values are aggregated together. When you replace a model, the chart for your custom revenue metric doesn't change.This field can't be edited after you create the metric.Is GeospatialDetermines if the custom metric will use geospatial data. PremiumGeospatial monitoring is a premium feature. Contact your DataRobot representative or administrator for information on enabling the feature. Geospatial feature monitoring supportGeospatial feature monitoring is supported for binary classification, multiclass, regression, and location target types.
4. In theFilessection, assemble the custom job. Drag files into the box, or use the options in this section to create or upload the files required to assemble a custom job: OptionDescriptionChoose from source / UploadUpload existing custom job files (run.sh,metadata.yaml, etc.) asLocal Filesor aLocal Folder.CreateCreate a new file, empty or containing a template, and save it to the custom job:Create run.sh: Creates a basic, editable example of an entry point file.Create metadata.yaml: Creates a basic, editable example of a runtime parameters file.Create README.md: Creates a basic, editable README file.Create job.py: Creates a basic, editable Python job file to print runtime parameters and deployments.Create example job: Combines all template files to create a basic, editable custom job. You can quickly configure the runtime parameters and run this example job.Create blank file: Creates an empty file. Click the edit iconnext toUntitledto provide a file name and extension, then add your custom contents. In the next step, it is possible to identify files created this way, with a custom name and content, as the entry point. After you configure the new file, clickSave. File replacementIf you add a new file with the same name as an existing file, when you clickSave, the old file is replaced in theFilessection.
5. In theSettingssection, configure theEntry pointshell (.sh) file for the job. If you've added arun.shfile, that file is the entry point; otherwise, you must select the entry point shell file from the dropdown list. The entry point file allows you to orchestrate multiple job files:
6. In theResourcessection, next to the section header, clickEditand configure the following: PreviewCustom job resource bundles are off by default. Contact your DataRobot representative or administrator for information on enabling this feature.Feature flag:Enable Resource Bundles SettingDescriptionResource bundle (preview)Configure the resources the custom job uses to run.Network accessConfigure the egress traffic of the custom job. UnderNetwork access, select one of the following:Public: The default setting. The custom job can access any fully qualified domain name (FQDN) in a public network to leverage third-party services.None: The custom job is isolated from the public network and cannot access third party services. Default network accessFor theManaged AI Platform, theNetwork accesssetting is set toPublicby default and the setting is configurable. For theSelf-Managed AI Platform, theNetwork accesssetting is set toNoneby default and the setting is restricted; however, an administrator can change this behavior during DataRobot platform configuration. Contact your DataRobot representative or administrator for more information.
7. (Optional) DefineRuntime parameters. Click+ Add runtime parameterto define a new runtime parameter by providing aName,Type,Value, and, optionally, aDescription. Alternativelydefine runtime parameters in ametadata.yamlfile. A template for this file is available from theFiles >Createdropdown. For existing runtime parameters, clickEditto edit parameter values, remove parameters, or reset parameter values.
8. (Optional) Configure additionalKey valuesforTags,Metrics,Training parameters, andArtifacts.
9. In theConnected deploymentspanel, click+ Connect to deployment, define aCustom metric name, and select aDeployment IDto connect it to that deployment.
10. Edit theCustom metric nameand select aDeployment ID, then, set aBaseline—the value used as a basis for comparison when calculating thex% betterorx% worsevalues. If you selectedIs Geospatialin theMetadatasection, select a deployment with at least onegeospatial/location featureand define theGeospatial segment attribute. Then, clickConnect. Standard custom metricGeospatial custom metricGeospatial feature monitoring supportGeospatial feature monitoring is supported for binary classification, multiclass, and regression target types. How many deployments can I connect to a hosted custom metric job?You can connect up to 10 deployments to a hosted custom metric job. Connected deployments and runtime parametersAfter you connect a deployment to a hosted custom metric job and schedule a run, you can't modify themetadata.yamlfile for runtime parameters. You must disconnect all connected deployments to make any modifications to themetadata.yamlfile.

## Create a hosted metric job from a template

To add a pre-made metric from a template:

> [!NOTE] Preview
> The jobs template gallery is on by default.
> 
> Feature flag: Enable Custom Jobs Template Gallery

1. In theAdd custom job from gallerypanel, select a custom metric template applicable to your intended use case and clickCreate metric. Binary ClassificationRegressionLLM (Generative)Agentic workflow / LLMCustom metric templateDescriptionRecall for top x%Measures model performance limited to a certain top fraction of the sorted predicted probabilities. Recall is a measure of a model's performance that calculates the proportion of actual positives that are correctly identified by the model.Precision for top x%Measures model performance limited to a certain top fraction of the sorted predicted probabilities. Precision is a measure of a model's performance that calculates the proportion of correctly predicted positive observations from the total predicted positive.F1 for top x%Measures model performance limited to a certain top fraction of the sorted predicted probabilities. F1 score is a measure of a model's performance which considers both precision and recall.AUC (Area Under the ROC Curve) for top x%Measures model performance limited to a certain top fraction of the sorted predicted probabilities.Custom metric templateDescriptionMean Squared Logarithmic Error (MSLE)Calculates the mean of the squared differences between logarithms of the predicted and actual values. It is a loss function used in regression problems when the target values are expected to have exponential growth, like population counts, average sales of a commodity over a time period, and so on.Median Absolute Error (MedAE)Calculates the median of the absolute differences between the target and the predicted values. It is a robust metric used in regression problems to measure the accuracy of predictions.Custom metric templateDescriptionCompletion Reading TimeEstimates the average time it takes a person to read text generated by the LLM.Completion Tokens MeanCalculates the mean number of tokens in completions for the time period requested. The cl100k_base encoding used only supports OpenAI models: gpt-4, gpt-3.5-turbo, and text-embedding-ada-002. If you use a different model, change the encoding.Cosine Similarity AverageCalculates the mean cosine similarity between each prompt vector and corresponding context vectors.Cosine Similarity MaximumCalculates the maximum cosine similarity between each prompt vector and corresponding context vectors.Cosine Similarity MinimumCalculates the minimum cosine similarity between each prompt vector and corresponding context vectors.CostEstimates the financial cost of using the LLM by calculating the number of tokens in the input, output, and retrieved text, and then applying token pricing. The cl100k_base encoding used only supports OpenAI models: gpt-4, gpt-3.5-turbo, and text-embedding-ada-002. If you use a different model, change the encoding.Dale Chall ReadabilityMeasures the U.S. grade level required to understand a text based on the percentage of difficult words and average sentence length.Euclidean AverageCalculates the mean Euclidean distance between each prompt vector and corresponding context vectors.Euclidean MaximumCalculates the maximum Euclidean distance between each prompt vector and corresponding context vectors.Euclidean MinimumCalculates the minimum Euclidean distance between each prompt vector and corresponding context vectors.Flesch Reading EaseMeasures the readability of text based on the average sentence length and average number of syllables per word.Prompt Injection [sidecar metric]Detects input manipulations, such as overwriting or altering system prompts, that are intended to modify the model's output. This metric requires an additional deployment of the Prompt Injection Classifierglobal model.Prompt Tokens MeanCalculates the mean number of tokens in prompts for the time period requested. The cl100k_base encoding used only supports OpenAI models: gpt-4, gpt-3.5-turbo, and text-embedding-ada-002. If you use a different model, change the encoding.Sentence CountCalculates the total number of sentences in user prompts and text generated by the LLM.SentimentClassifies text sentiment as positive or negativeSentiment [sidecar metric]Classifies text sentiment as positive or negative using a pre-trained sentiment classification model. This metric requires an additional deployment of the Sentiment Classifierglobal model.Syllable CountCalculates the total number of syllables in the words in user prompts and text generated by the LLM.Tokens MeanCalculates the mean of tokens in prompts and completions. The cl100k_base encoding used only supports OpenAI models: gpt-4, gpt-3.5-turbo, and text-embedding-ada-002. If you use a different model, change the encoding.Toxicity [sidecar metric]Measures the toxicity of text using a pre-trained hate speech classification model to safeguard against harmful content. This metric requires an additional deployment of the Toxicity Classifierglobal model.Word CountCalculates the total number of words in user prompts and text generated by the LLM.Japanese text metrics[JP] Character CountCalculates the total number of characters generated while working with the LLM.[JP] PII occurrence countCalculates the total number of PII occurrences while working with the LLM.Custom metric templateDescriptionAgentic completion tokensCalculates the total completion tokens of agent-based LLM calls.Agentic costCalculates the total cost of agent-based LLM calls. Requires that each LLM span reports token usage so the metric can compute cost from the trace.Agentic prompt tokensCalculates the total prompt tokens of agent-based LLM calls.
2. Review the job description,Execution environment,Metadata, andFiles. If necessary, set theCustom metric configurationsetting to select aSidecar deployment, then, clickCreate custom job: The hosted metric job opens to theAssembletab. Sidecar metricsIf you selected a[sidecar metric], ensure that you set theSidecar deploymentsetting in theCustom metric configuration. To verify the connection, when you open on theAssembletab, navigate to theRuntime Parametersand confirm theSIDECAR_DEPLOYMENT_IDparameter is set, associating the sidecar metric with the connected deployment required to calculate that metric. If you haven't deployed a model to calculate the metric, you can find pre-defined models for these metrics asglobal models.
3. On theAssembletab, you can optionally modify the template's default name,Environment,Files,Settings,Resources,Runtime Parameters, orKey values, just as with astandard custom metric job.
4. In theConnected deploymentspanel, click+ Connect to deployment. Connected deployments and runtime parametersAfter you connect a deployment to a hosted custom metric job and schedule a run, you can't modify themetadata.yamlfile for runtime parameters. You must disconnect all connected deployments to make any modifications to themetadata.yamlfile.
5. Edit theCustom metric nameand select aDeployment ID, then, set aBaseline—the value used as a basis for comparison when calculating thex% betterorx% worsevalues—and clickConnect. How many Deployments can I connect to a hosted custom metric job?You can connect up to 10 deployments to a hosted custom metric job.