# Configure evaluation and moderation
URL: https://docs.datarobot.com/en/docs/agentic-ai/agentic-deploy/agentic-configure-evaluation-moderation.html

> How to configure evaluation and moderation guardrails for a custom text generation model and agentic workflows in the workshop.

# Configure evaluation and moderation

> [!NOTE] Premium
> Evaluation and moderation guardrails are a premium feature. Contact your DataRobot representative or administrator for information on enabling this feature.
> 
> Feature flag: Enable Moderation Guardrails ( Premium), Enable Global Models in the Model Registry ( Premium), Enable Additional Custom Model Output in Prediction Responses

Evaluation and moderation guardrails help your organization block prompt injection and hateful, toxic, or inappropriate prompts and responses. It can also prevent hallucinations or low-confidence responses and, more generally, keep the model on topic. In addition, these guardrails can safeguard against the sharing of personally identifiable information (PII). Many evaluation and moderation guardrails connect a deployed text generation model (LLM) or agentic workflow to a deployed guard model. These guard models make predictions on LLM prompts and responses and then report these predictions and statistics to the central LLM or agentic workflow deployment.

To use evaluation and moderation guardrails, first create and deploy guard models to make predictions on an LLM's prompts or responses; for example, a guard model could identify prompt injection or toxic responses. Then, when you create a custom model with the Text Generation or Agentic Workflow target type, define one or more evaluation and moderation guardrails.

## Select evaluation and moderation guardrails

When you create a custom model with the Text Generation or Agentic Workflow target type, define one or more evaluation and moderation guardrails.

To select and configure evaluation and moderation guardrails:

1. In theWorkshop, open theAssembletab of a custom model with theText GenerationorAgentic Workflowtarget type andassemble a model, eithermanually from a custom model you created outside of DataRobotorautomatically from a model built in a Use Case's LLM playground: When you assemble a text generation model with moderations, ensure you configure any requiredruntime parameters(for example, credentials) orresource settings(for example, public network access). Finally, set theBase environmentto a moderation-compatible environment; for example,[GenAI] Python 3.12 with Moderations: Resource settingsDataRobot recommends creating the LLM custom model using larger resource bundles with more memory and CPU resources.
2. After you've configured the custom model's required settings, navigate to theEvaluation and moderationsection and clickConfigure:
3. On theConfigure evaluation and moderationpanel, in theConfiguration summary, access the following settings: SettingDescriptionShow workflowReview how evaluations are executed in DataRobot. All evaluations and their respective moderations run in parallel.Moderation settingsSet the following:Set moderation timeout: Configure the maximum wait time (in seconds) for moderations before the system automatically times out.Timeout action: Define what happens if the moderation system times out:Score prompt / responseorBlock prompt / response.NeMo evaluator settingsSet theNeMo evaluator deploymentused by the NeMo Evaluator metrics. The dropdown shows "No options available" until you have created a NeMo evaluator workload and workload deployment via the Workload API. You must complete that step before you can configure the NeMo Evaluator metrics.
4. In theConfigure evaluation and moderationpanel, click one of the following metric cards to configure the required properties. The panel has two sections:All MetricsandNeMo metrics. From theConfiguration summarysidebar you can openShow workflow,Moderation settings, orNeMo evaluator settingsto configure the evaluator deployment used by all NeMo evaluator metrics. All MetricsNeMo metricsEvaluation metricRequiresDescriptionContent SafetyA deployed NIM modelllama-3.1-nemoguard-8b-content-safetyimported fromNVIDIA GPU Cloud (NGC) Catalog.Classify prompts and responses as safe or unsafe; return a list of any unsafe categories detected.CostLLM cost settingsCalculate the cost of generating the LLM response using the provided input cost-per-token, and output cost-per-token values. The cost calculation also includes the cost of citations. For more information, seeCost metric settings.Custom DeploymentCustom deploymentUse any deployment to evaluate and moderate your LLM (supported target types: regression, binary classification,multiclass, text generation).Emotions ClassifierEmotions Classifier deploymentClassify prompt or response text by emotion.FaithfulnessLLM, vector databaseMeasure if the LLM response matches the source to identify possible hallucinations.JailbreakA deployed NIM modelnemoguard-jailbreak-detectimported fromNVIDIA GPU Cloud (NGC) Catalog.Classify jailbreak attempts using NemoGuard JailbreakDetect.PII DetectionPresidio PII DetectionDetect Personally Identifiable Information (PII) in text using the Microsoft Presidio library.Prompt InjectionPrompt Injection ClassifierDetect input manipulations, such as overwriting or altering system prompts, intended to modify the model's output.Prompt tokensN/ATrack the number of tokens associated with the input to the LLM and/or retrieved text from the vector database.Response tokensN/ATrack the number of tokens associated with the output from the LLM and/or retrieved text from the vector database.ROUGE-1Vector databaseCalculate the similarity between the response generated from an LLM blueprint and the documents retrieved from the vector database.ToxicityToxicity ClassifierClassify content toxicity to apply moderation techniques, safeguarding against dissemination of harmful content.Agentic workflow metricsAgent Goal AccuracyLLMEvaluate agentic workflow performance in achieving specified objectives in scenarios without a known benchmark. (This agentic workflow metric is distinct from the NeMo Evaluator metric of the same name underNeMo metrics.)Task AdherenceLLMMeasure whether the agentic workflow response is relevant, complete, and aligned with user expectations.Guideline AdherenceLLM, guideline settingEvaluate how well the response follows the defined guideline using a judge LLM. Returnstruewhen the guideline is followed,falseotherwise. You must supply the guideline and select an LLM (from the gateway or a deployment) when configuring.Global models for evaluation metric deploymentsThe deployments required for PII detection, prompt injection detection, emotion classification, and toxicity classification are available asglobal models in RegistryMulticlass custom deployment metric limitsMulticlasscustom deployment metrics can have:Up to10classes defined in theMatcheslist for moderation criteria.Up to100class names in the guard model.TheNeMo Evaluator metrics(Agent Goal Accuracy, Context Relevance, Faithfulness, LLM Judge, Response Groundedness, Response Relevancy, Topic Adherence) require aNeMo evaluator workload deployment, set inNeMo evaluator settingsin the Configuration summary sidebar. Create the workload and workload deployment via the Workload API before you can select it; theSelect a workload deploymentdropdown shows "No options available" until a deployment exists. Each of these metrics also uses an LLM judge (DataRobot deployment or LLM gateway). Response Relevancy additionally requires an embedding deployment. Topic Adherence and LLM Judge have additional configuration.Stay on topic for inputsandStay on topic for outputdo notuse the NeMo evaluator deployment. They use aNIM deploymentof thellama-3.1-nemoguard-8b-topic-controlmodel (like Content safety and Jailbreak use NIM models). Configure them with LLM typeNIM, select the topic-control NIM deployment, and optionally edit the NeMo guardrails configuration files.Evaluator metricRequiresDescriptionAgent Goal AccuracyEvaluator deployment, LLMEvaluate how well the agent fulfills the user's query. This is distinct from the Agent Goal Accuracy metric underAll metrics(agentic workflow).Context RelevanceEvaluator deployment, LLMMeasure how relevant the provided context is to the response.FaithfulnessEvaluator deployment, LLMEvaluate whether the response stays faithful to the provided context using the NeMo Evaluator. This is distinct from the non-NeMo Faithfulness metric listed underAll metrics.LLM JudgeEvaluator deployment, LLMUse a judge LLM to evaluate a user defined metric.Response GroundednessEvaluator deployment, LLMEvaluate whether the response is grounded in the provided context.Response RelevancyEvaluator deployment, LLM, Embedding deploymentMeasure how relevant the response is to the user's query.Topic AdherenceEvaluator deployment, LLM, Metric mode, Reference topicsAssess whether the response adheres to the expected topics.Topic control metricsStay on topic for inputsNIM deployment ofllama-3.1-nemoguard-8b-topic-control, NVIDIA NeMo guardrails configurationUse NVIDIA NeMo Guardrails to provide topic boundaries, ensuring prompts are topic-relevant and do not use blocked terms.Stay on topic for outputNIM deployment ofllama-3.1-nemoguard-8b-topic-control, NVIDIA NeMo guardrails configurationUse NVIDIA NeMo Guardrails to provide topic boundaries, ensuring responses are topic-relevant and do not use blocked terms.To set theNeMo evaluator deploymentused by the NeMo Evaluator metrics, openNeMo evaluator settingsfrom the Configuration summary sidebar. The evaluator deployment will be applied to all NeMo Evaluator metrics. From theSelect a workload deploymentdropdown list, choose the workload deployment for the NeMo evaluator.NeMo evaluator settings panelThe dropdown shows "No options available" until you have created a NeMo evaluator workload and workload deployment via the Workload API. You must complete that step before you can configure the NeMo Evaluator metrics.
5. Depending on the metric selected above, configure the following fields: FieldDescriptionGeneral settingsNameEnter a unique name if adding multiple instances of the evaluation metric.Apply toSelect one or both ofPromptandResponse, depending on the evaluation metric. Note that when you selectPrompt, it's the user prompt, not the final LLM prompt, that is used for metric calculation. This field is only configurable for metrics that apply to both the prompt and the response.Custom Deployment, PII Detection, Prompt Injection, Emotions Classifier, and Toxicity settingsDeployment nameFor evaluation metrics calculated by a guard model, select the custom model deployment.Custom Deployment settingsInput column nameThis name is defined by the custom model creator. Forglobal models created by DataRobot, the default input column name istext. If the guard model for the custom deployment has themoderations.input_column_namekey valuedefined, this field is populated automatically.Output column nameThis name is defined by the custom model creator, and needs to refer to the target column for the model. The target name is listed on the deployment'sOverviewtab (and often has_PREDICTIONappended to it). You can confirm the column names byexporting and viewing the CSV data from the custom deployment. If the guard model for the custom deployment has themoderations.output_column_namekey valuedefined, this field is populated automatically.Guideline Adherence settingGuidelineThe rule or criteria the agent's response should follow. The selected LLM acts as a judge to evaluate whether the response adheres to this guideline and returnstrue(guideline followed) orfalse(guideline not followed). You must supply the guideline and select an LLM (from the gateway or a deployment) when configuring this metric.Faithfulness, Task Adherence, and Guideline Adherence settingsLLMSelect an LLM to evaluate the selected metric. For Faithfulness, once you select an LLM, you have the option of using your ownuser-providedcredentials instead of DataRobot-provided.NeMo Evaluator metric settingsSelect LLM as a judgeSelect an LLM to evaluate the selected metric.Evaluator deploymentFor the NeMo Evaluator metrics only: set in theNeMo evaluator settingssidebar panel (Select a workload deployment). The NeMo evaluator workload deployment is shared by those metrics. Create the workload and workload deployment via the Workload API before configuring; see the prerequisites above.Topic control settingsLLM TypeSelectAzure OpenAI,OpenAI, orNIM. For theAzure OpenAILLM type, additionally enter anOpenAI API deployment; forNIMenter aNIM deployment. If you use the LLM gateway, the default experience, DataRobot-supplied credentials are provided. When LLM type isAzure OpenAIorOpenAI, clickChange credentialsto provide your own authentication.FilesFor theStay on topicevaluations, next to a file, clickto modify the NeMo guardrails configuration files. In particular, updateprompts.ymlwith allowed and blocked topics andblocked_terms.txtwith the blocked terms, providing rules for NeMo guardrails to enforce. Theblocked_terms.txtfile is shared between the input and output topic control metrics; therefore, modifyingblocked_terms.txtin the input metric modifies it for the output metric and vice versa. Only two topic control metrics can exist in a custom model, one for input and one for output.Moderation settingsConfigure and apply moderationEnable this setting to expand theModerationsection and define the criteria that determine when moderation logic is applied. Cost metric settingsFor theCostmetric, define theInputandOutputcost incurrency amount / tokens amountformat, then clickAdd:TheCostmetric doesn't include theModerationsection toConfigure and apply moderation.
6. In theModerationsection, withConfigure and apply moderationenabled, for each evaluation metric, set the following: SettingDescriptionModeration criteriaIf applicable, set the threshold settings evaluated to trigger moderation logic. For numeric metrics (int or float), you can useless than,greater than, orequals towith a value of your choice. For binary metrics (for example, Agent Goal Accuracy), useequals to0 or 1. For the Emotions Classifier, selectMatchesorDoes not matchand define a list of classes (emotions) to trigger moderation logic.Moderation methodSelectReport,Report and block, orReplace(if applicable).Moderation messageIf you selectReport and block, you can optionally modify the default message.
7. After configuring the required fields, clickAddto save the evaluation and return to the evaluation selection page. Then, select and configure another metric, or clickSave configuration. The guardrails you selected appear in theEvaluation and moderationsection of theAssembletab.

After you add guardrails to a text generation custom model, you can [test](https://docs.datarobot.com/en/docs/workbench/nxt-registry/nxt-model-workshop/nxt-test-custom-model.html), [register](https://docs.datarobot.com/en/docs/workbench/nxt-registry/nxt-model-directory/nxt-register-cus-models.html), and [deploy](https://docs.datarobot.com/en/docs/workbench/nxt-registry/nxt-model-directory/nxt-deploy-models.html) the model to make predictions in production. After making [predictions](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-predictions/index.html), you can view the evaluation metrics on the [Custom metrics](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-custom-metrics.html) tab and prompts, responses, and feedback (if configured) on the [Data exploration](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-data-exploration.html) tab.

> [!NOTE] Tracing tab
> When you add moderations to an LLM deployment, you can't view custom metric data by row on the [Data exploration > Tracing](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-data-exploration.html) tab.

### Change credentials

DataRobot provides credentials for [available LLMs](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/llm-availability.html) using the LLM gateway. With certain metrics and LLMs or LLM types, you can instead use your own credentials for authentication. Before proceeding, define user-specified credentials on the [credentials management](https://docs.datarobot.com/en/docs/platform/acct-settings/stored-creds.html#credentials-management) page.

#### Topic control metrics

To change credentials for either Stay on topic for inputs or Stay on topic for output, choose the LLM type and click Change credentials.

**LLM type: Azure OpenAI:**
Provide the Azure OpenAI API deployment and the OpenAI API base URL. Then, from the dropdown, select the set of credentials to apply.

[https://docs.datarobot.com/en/docs/images/change-metric-creds-azure.png](https://docs.datarobot.com/en/docs/images/change-metric-creds-azure.png)

**LLM type: OpenAI:**
From the dropdown, select the set of credentials to apply.

[https://docs.datarobot.com/en/docs/images/change-metric-creds-openai.png](https://docs.datarobot.com/en/docs/images/change-metric-creds-openai.png)

**LLM type: NIM:**
Select the NIM deployment (for example, the topic-control model). Credentials are typically provided via the deployment configuration.


To revert to DataRobot-provided credentials, click Revert credentials.

#### Faithfulness metric

To change credentials for Faithfulness, select the LLM and click Change credentials.

The following table lists the required fields:

| Provider | Fields |
| --- | --- |
| Amazon | AWS account (credentials)AWS region |
| Azure OpenAI | OpenAI API deploymentOpenAI API base URLCredentials |
| Google | Service account (credentials)Google region |
| OpenAI | Credentials |

To revert to DataRobot-provided credentials, click Revert credentials.

### Considerations for NeMo Evaluator metrics

When using NeMo Evaluator metrics, consider the following:

- LLM judge output: The NeMo evaluator expects the LLM judge to return data in the correct JSON schema. Some models (for example, certain Llama versions) may return Python code or other formats instead, which can cause the evaluator to fail. Choose an LLM judge that reliably returns the expected format; newer models are often better at following JSON output instructions.
- Rate and token limits: Be aware of rate limits and token limits when using NeMo Evaluator guards; hitting these limits can cause evaluation failures.

You can use the [Activity log > Moderation](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-activity-log/nxt-moderation.html) tab and evaluator logs to debug why a request was blocked or why a guard failed.

### Global models for evaluation metric deployments

The deployments required for PII detection, prompt injection detection, emotion classification, and toxicity classification are available as [global models in Registry](https://docs.datarobot.com/en/docs/workbench/nxt-registry/nxt-model-directory/nxt-global-models.html). The following global models are available:

| Model | Type | Target | Description |
| --- | --- | --- | --- |
| Prompt Injection Classifier | Binary | injection | Classifies text as prompt injection or legitimate. This guard model requires one column named text, containing the text to classify. For more information, see the deberta-v3-base-injection model details. |
| Toxicity Classifier | Binary | toxicity | Classifies text as toxic or non-toxic. This guard model requires one column named text, containing the text to classify. For more information, see the toxic-comment-model details. |
| Sentiment Classifier | Binary | sentiment | Classifies text sentiment as positive or negative. This model requires one column named text, containing the text to classify. For more information, see the distilbert-base-uncased-finetuned-sst-2-english model details. |
| Emotions Classifier | Multiclass | target | Classifies text by emotion. This is a multilabel model, meaning that multiple emotions can be applied to the text. This model requires one column named text, containing the text to classify. For more information, see the roberta-base-go_emotions-onnx model details. |
| Refusal Score | Regression | target | Outputs a maximum similarity score, comparing the input to a list of cases where an LLM has refused to answer a query because the prompt is outside the limits of what the model is configured to answer. |
| Presidio PII Detection | Binary | contains_pii | Detects and replaces Personally Identifiable Information (PII) in text. This guard model requires one column named text, containing the text to be classified. The types of PII to detect can optionally be specified in a column, 'entities', as a comma-separated string. If this column is not specified, all supported entities will be detected. Entity types can be found in the PII entities supported by Presidio documentation. In addition to the detection result, the model returns an anonymized_text column, containing an updated version of the input with detected PII replaced with placeholders. For more information, see the Presidio: Data Protection and De-identification SDK documentation. |
| Zero-shot Classifier | Binary | target | Performs zero-shot classification on text with user-specified labels. This model requires classified text in a column named text and class labels as a comma-separated string in a column named labels. It expects the same set of labels for all rows; therefore, the labels provided in the first row are used. For more information, see the deberta-v3-large-zeroshot-v1 model details. |
| Python Dummy Binary Classification | Binary | target | Always yields 0.75 for the positive class. For more information, see the python3_dummy_binary model template. |

## View evaluation and moderation guardrails

When a text generation model with guardrails is registered and deployed, you can view the configured guardrails on the [registered model'sOverview](https://docs.datarobot.com/en/docs/workbench/nxt-registry/nxt-model-directory/nxt-view-manage-reg-models.html#view-version-details) tab and the [deployment'sOverview](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-overview/nxt-overview.html) tab:

**Registry:**
[https://docs.datarobot.com/en/docs/images/nxt-evaluation-moderation-reg-overview.png](https://docs.datarobot.com/en/docs/images/nxt-evaluation-moderation-reg-overview.png)

**Console:**
[https://docs.datarobot.com/en/docs/images/nxt-evaluation-moderation-deploy-overview.png](https://docs.datarobot.com/en/docs/images/nxt-evaluation-moderation-deploy-overview.png)


> [!NOTE] Evaluation and moderation logs
> On the [Activity log > Moderation](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-activity-log/nxt-moderation.html) tab of a deployed LLM with evaluation and moderation configured, you can view a history of evaluation and moderation-related events for the deployment to diagnose issues with a deployment's configured evaluations and moderations.

---
