# Generative model monitoring

> Generative model monitoring - The text generation target type for DataRobot custom and external
> models is compatible with generative Large Language Models (LLMs), allowing you to deploy generative
> models, make predictions, monitor model performance statistics, explore data, and create custom
> metrics.

This Markdown file sits beside the HTML page at the same path (with a `.md` suffix). It summarizes the topic and lists links for tools and LLM context.

Companion generated at `2026-05-06T18:17:10.028792+00:00` (UTC).

## Primary page

- [Generative model monitoring](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-genai-monitoring.html): Full documentation for this topic (HTML).

## Sections on this page

- [Create and deploy a generative custom model](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-genai-monitoring.html#create-and-deploy-a-generative-custom-model): In-page section heading.
- [Add a generative custom model](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-genai-monitoring.html#add-a-generative-custom-model): In-page section heading.
- [Assemble and deploy a generative custom model](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-genai-monitoring.html#assemble-and-deploy-a-generative-custom-model): In-page section heading.
- [Create and deploy an external generative model](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-genai-monitoring.html#create-and-deploy-an-external-generative-model): In-page section heading.
- [Monitor a deployed generative model](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-genai-monitoring.html#monitor-a-deployed-generative-model): In-page section heading.
- [Data drift for generative models](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-genai-monitoring.html#feature-details-for-generative-models): In-page section heading.

## Related documentation

- [NextGen UI documentation](https://docs.datarobot.com/en/docs/workbench/index.html): Linked from this page.
- [Console](https://docs.datarobot.com/en/docs/workbench/nxt-console/index.html): Linked from this page.
- [Monitoring](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/index.html): Linked from this page.
- [Custom metrics](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-custom-metrics.html): Linked from this page.
- [during](https://docs.datarobot.com/en/docs/workbench/nxt-registry/nxt-model-directory/nxt-deploy-models.html#custom-metrics): Linked from this page.
- [after](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-settings/nxt-custom-metrics-settings.html): Linked from this page.
- [workshop](https://docs.datarobot.com/en/docs/workbench/nxt-registry/nxt-model-workshop/index.html): Linked from this page.
- [testing](https://docs.datarobot.com/en/docs/workbench/nxt-registry/nxt-model-workshop/nxt-test-custom-model.html): Linked from this page.
- [registered custom model](https://docs.datarobot.com/en/docs/workbench/nxt-registry/nxt-model-directory/nxt-register-cus-models.html): Linked from this page.
- [drop-in environment](https://docs.datarobot.com/en/docs/workbench/nxt-registry/nxt-model-workshop/nxt-create-custom-model.html#drop-in-environments): Linked from this page.
- [custom model code](https://docs.datarobot.com/en/docs/api/code-first-tools/drum/structured-custom-models.html): Linked from this page.
- [runtime parameters](https://docs.datarobot.com/en/docs/api/code-first-tools/drum/custom-model-runtime-parameters.html): Linked from this page.
- [libraries (and versions)](https://docs.datarobot.com/en/docs/classic-ui/mlops/deployment/custom-models/custom-model-workshop/custom-model-dependencies.html): Linked from this page.
- [make predictions](https://docs.datarobot.com/en/docs/api/dev-learning/python/predictions/index.html): Linked from this page.
- [monitoring agent](https://docs.datarobot.com/en/docs/classic-ui/mlops/deployment/mlops-agent/index.html): Linked from this page.
- [add the required information](https://docs.datarobot.com/en/docs/workbench/nxt-registry/nxt-model-directory/nxt-register-ext-models.html): Linked from this page.
- [service health](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-service-health.html): Linked from this page.
- [usage](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-usage.html): Linked from this page.
- [deployment data](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-data-exploration.html): Linked from this page.
- [data drift](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-data-drift.html): Linked from this page.
- [Overview](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-overview/nxt-overview.html): Linked from this page.

## Documentation content

> [!NOTE] Availability information
> Monitoring support for generative models is a premium feature. Contact your DataRobot representative or administrator for information on enabling this feature.

Using the text generation target type for custom and external models, a premium LLMOps feature, deploy generative Large Language Models (LLMs) to make predictions, monitor service, usage, and data drift statistics, and create custom metrics. DataRobot supports LLMs through two deployment methods:

| Method | Description |
| --- | --- |
| Create a text generation model as a custom model in DataRobot | Create and deploy a text generation model using the workshop, calling the LLM's API to generate text and allowing MLOps to access the LLM's input and output for monitoring. To call the LLM's API, you should enable public network access for custom models. |
| Monitor a text generation model running externally | Create and deploy a text generation model on your infrastructure (local or cloud), using the monitoring agent to communicate the input and output of your LLM to DataRobot for monitoring. |

> [!TIP] Custom metrics for evaluation and moderation require an association ID
> For the metrics added when you configure evaluations and moderations, to view data on the Custom metrics tab, ensure that you set an association ID and enable prediction storage before you start making predictions through the deployed LLM.If you don't set an association ID and provide association IDs alongside the LLM's predictions, the metrics for the moderations won't be calculated on the [Custom metrics](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-custom-metrics.html) tab.After you define the association ID, you can enable automatic association ID generation to ensure these metrics appear on the Custom metrics tab. You can enable this setting [during](https://docs.datarobot.com/en/docs/workbench/nxt-registry/nxt-model-directory/nxt-deploy-models.html#custom-metrics) or [after](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-settings/nxt-custom-metrics-settings.html) deployment.

## Create and deploy a generative custom model

Custom inference models are user-created, pretrained models that you can upload to DataRobot (as a collection of files) via the [workshop](https://docs.datarobot.com/en/docs/workbench/nxt-registry/nxt-model-workshop/index.html). You can then upload a model artifact to create, test, and deploy custom inference models to DataRobot's centralized deployment hub.

### Add a generative custom model

To add a generative model to the Workshop:

1. ClickRegistry > Workshop. This tab lists the models you have created:
2. Click+ Add model(or thebutton when the custom model panel is open):
3. On theAdd a modelpage, define the following fields underConfigure the model: FieldDescriptionModel nameEnter a descriptive name for the custom model.Target typeSelectText Generation.Target nameEnter the name of the dataset column that contains the generative AI model's output, for exampleresultText.Advanced configurationLanguageEnter the programming language used to build the generative AI model.DescriptionEnter a description of the model's contents and purpose.
4. After completing the fields, clickAdd model. The custom model opens to theAssembletab.

### Assemble and deploy a generative custom model

To assemble, test, and deploy a generative model from the Workshop:

1. At the top of theAssembletab, underEnvironment, select a GenAI model environment from theBase environmentlist. The model environment is used fortestingthe custom model anddeployingtheregistered custom model.
2. To populate theDependenciessection, you can upload arequirements.txtfile in theFilessection, allowing DataRobot to build the optimal image.
3. In theFilessection, add the required custom model files. If you aren't pairing the model with adrop-in environment, this includes the custom model environment requirements and astart_server.shfile. You can add files in several ways: ElementDescription1FilesDrag files into the group box for upload.2Choose from sourceClick to browse forLocal Filesor aLocal Folder.3UploadClick to browse forLocal Filesor aLocal Folderor to pull files from a remote repository.4CreateCreate a new file, empty or as a template, and save it to the custom model:Create model-metadata.yaml: Creates a basic, editable example of a runtime parameters file.Create blank file: Creates an empty file. Click the edit icon () next toUntitledto provide a file name and extension, then add your custom contents. AbasicLLM assembled in the workshop should, at minimum, include the following files: FileContentscustom.pyThecustom model code, calling the LLM service's API throughpublic network access for custom models.model-metadata.yamlThe custom model metadata andruntime parametersrequired by the generative model.requirements.txtThelibraries (and versions)required by the generative model.
4. After you add the required model files,add training data. To provide a training baseline for drift monitoring, upload a dataset containingat least20 rows of prompts and responses relevant to the topic your generative model is intended to answer questions about. These prompts and responses can be taken from documentation, manually created, or generated.
5. Next, click theTesttab, clickRun new test, and then clickRunto start theStartupandPrediction errortests—the only tests supported for theText Generationtarget type.
6. ClickRegister a model,provide the model information, and clickRegister model. The registered model opens in theModels directorytab.
7. In the registered model version header, clickDeploy, and thenconfigure the deployment settings. You can nowmake predictionsas you would with any other DataRobot model.

## Create and deploy an external generative model

External model packages allow you to register and deploy external generative models. You can use the [monitoring agent](https://docs.datarobot.com/en/docs/classic-ui/mlops/deployment/mlops-agent/index.html) to access MLOps monitoring capabilities with these model types.

To create and deploy an external generative model monitored by the monitoring agent, add an external model as a registered model or version through Registry:

1. On theRegistry > Modelstab, click+ Register a model(or thebutton when the registered model or version info panel is open): TheRegister a modelpanel opens to theExternal modeltab.
2. On theExternal modeltab, underConfigure the model, selectAdd a version to an existing registered modelorCreate a new registered model.
3. From theTarget typelist, clickText generationandadd the required informationabout the agent-monitored generative model.
4. In theOptional settings, provide a training baseline for drift monitoring. To do this, underTraining data, click+ Add dataand upload a dataset containingat least20 rows of prompts and responses relevant to the topic your generative model is intended to answer questions about. These prompts and responses can be taken from documentation, manually created, or generated.
5. Once you've configured all required fields, clickRegister model. The model version opens on theRegistry > Modelstab.
6. In the registered model version header, clickDeploy, and thenconfigure the deployment settings.

## Monitor a deployed generative model

To monitor a generative model in production, you can view [service health](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-service-health.html) and [usage](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-usage.html) statistics, explore [deployment data](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-data-exploration.html), create [custom metrics](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-custom-metrics.html), and identify [data drift](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-data-drift.html).

**Service Health:**
[https://docs.datarobot.com/en/docs/images/nxt-text-generation-service-health.png](https://docs.datarobot.com/en/docs/images/nxt-text-generation-service-health.png)

**Usage:**
[https://docs.datarobot.com/en/docs/images/nxt-text-generation-usage.png](https://docs.datarobot.com/en/docs/images/nxt-text-generation-usage.png)

**Data Exploration:**
[https://docs.datarobot.com/en/docs/images/nxt-text-generation-data-exploration.png](https://docs.datarobot.com/en/docs/images/nxt-text-generation-data-exploration.png)

**Custom Metrics:**
[https://docs.datarobot.com/en/docs/images/nxt-text-generation-custom-metrics.png](https://docs.datarobot.com/en/docs/images/nxt-text-generation-custom-metrics.png)

**Data Drift:**
[https://docs.datarobot.com/en/docs/images/nxt-text-generation-data-drift.png](https://docs.datarobot.com/en/docs/images/nxt-text-generation-data-drift.png)


### Data drift for generative models

To monitor drift in a generative model's prediction data, DataRobot compares new prompts and responses to the prompts and responses in the training data you uploaded during model creation. To provide an adequate training baseline for comparison, the uploaded training dataset should contain at least 20 rows of prompts and responses relevant to the topic your model is intended to answer questions about. These prompts and responses can be taken from documentation, manually created, or generated.

On the Monitoring > Data drift tab for a generative model, you can view the [Feature Drift vs. Feature Importance](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-data-drift.html#feature-drift-vs-feature-importance-chart), [Feature Details](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-genai-monitoring.html#feature-details-for-generative-models), and [Drift Over Time](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-data-drift.html#drift-over-time-chart) charts. In addition, the [Drill down](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-data-drift.html#drill-down-on-the-data-drift-tab) tab is available for generative models. To learn how to adjust the Data drift dashboard to focus on a specific model, time period, or feature, see the [Configure the Data Drift dashboard](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-data-drift.html#configure-the-data-drift-dashboard) documentation.

#### Feature details for generative models

The Feature Details chart includes new functionality for text generation models, providing a word cloud visualizing differences in the data distribution for each token in the dataset between the training and scoring periods. By default, the Feature Details chart includes information about the question (or prompt) and answer (or target, model completion, output, or response). These are Text features, and in the example below, the question feature is prompt and the answer feature is target:

| Feature | Description |
| --- | --- |
| prompt | A word cloud visualizing the difference in data distribution for each user prompt or question token between the training and scoring periods and revealing how much each token contributes to data drift in the user prompt data. |
| target | A word cloud visualizing the difference in data distribution for each model output or answer token between the training and scoring periods and revealing how much each token contributes to data drift in the model output data. |

> [!NOTE] Features in the Feature Details chart
> The feature names for the generative model's input and output depend on the feature names in your model's data; therefore, the prompt and target features in the example above will be replaced by the names of the input and output columns in your model's data. You can view these feature names in the Target and Prompt column name fields on the [Overview](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-overview/nxt-overview.html) tab for a generative model.

You can also designate other features for data drift tracking; for example, you could decide to track the model's temperature, monitoring the level of creativity in the generative model's responses from high creativity (1) to low (0).

To interpret the feature drift word cloud for a text feature like prompt or target, hover over a user prompt or model output token to view the following details:

| Chart element | Description |
| --- | --- |
| Token | The tokenized text represented by the word in the word cloud. Text size represents the token's drift contribution and text color represents the dataset prevalence. Stop words are hidden from this chart. |
| Drift contribution | How much this particular token contributes to the feature's drift value, as reported in the Feature Drift vs. Feature Importance chart. |
| Data distribution | How much more often this particular token appears in the training data or the predictions data. Blue: This token appears X% more often in training data.Red: This token appearsX% more often in predictions data. |

> [!TIP] Tip
> When your pointer is over the word cloud, you can scroll up to zoom in and view the text of smaller tokens.