# AI robustness testing

> AI robustness testing - Configure generative AI robustness insights and tests with the Python client
> to measure quality, toxicity, relevance, and similar metrics on prompts and responses.

This Markdown file sits beside the HTML page at the same path (with a `.md` suffix). It summarizes the topic and lists links for tools and LLM context.

Companion generated at `2026-05-06T18:17:09.592255+00:00` (UTC).

## Primary page

- [AI robustness testing](https://docs.datarobot.com/en/docs/api/dev-learning/python/genai/robustness-testing.html): Full documentation for this topic (HTML).

## Sections on this page

- [Configure insights for robustness testing](https://docs.datarobot.com/en/docs/api/dev-learning/python/genai/robustness-testing.html#configure-insights-for-robustness-testing): In-page section heading.
- [Set up test configurations](https://docs.datarobot.com/en/docs/api/dev-learning/python/genai/robustness-testing.html#set-up-test-configurations): In-page section heading.
- [Run robustness tests](https://docs.datarobot.com/en/docs/api/dev-learning/python/genai/robustness-testing.html#run-robustness-tests): In-page section heading.
- [Configure multiple insights](https://docs.datarobot.com/en/docs/api/dev-learning/python/genai/robustness-testing.html#configure-multiple-insights): In-page section heading.
- [Get an insight configuration](https://docs.datarobot.com/en/docs/api/dev-learning/python/genai/robustness-testing.html#get-an-insight-configuration): In-page section heading.
- [List insight configurations](https://docs.datarobot.com/en/docs/api/dev-learning/python/genai/robustness-testing.html#list-insight-configurations): In-page section heading.

## Related documentation

- [Developer documentation](https://docs.datarobot.com/en/docs/api/index.html): Linked from this page.
- [Developer learning](https://docs.datarobot.com/en/docs/api/dev-learning/index.html): Linked from this page.
- [Python API client user guide](https://docs.datarobot.com/en/docs/api/dev-learning/python/index.html): Linked from this page.
- [AI robustness tests](https://docs.datarobot.com/en/docs/api/reference/sdk/gen-testing.html#datarobot.models.genai.insights_configuration.InsightsConfiguration): Linked from this page.

## Documentation content

[AI robustness tests](https://docs.datarobot.com/en/docs/api/reference/sdk/gen-testing.html#datarobot.models.genai.insights_configuration.InsightsConfiguration) help validate that your generative AI models perform well under various conditions and meet quality standards. Robustness testing allows you to configure insights that measure different aspects of model performance, such as toxicity, relevance, and coherence.

## Configure insights for robustness testing

Set up insights to test AI model robustness. When creating an insight configuration, you should specify the following:

- custom_model_version_id : The ID of the custom model version to test.
- insight_name : A user-friendly name for the insight.
- insight_type : The type of insight (e.g., OOTB_METRIC ).
- ootb_metric_name : The name of the DataRobot-provided metric (for the OOTB_METRIC type).
- stage : The stage when the metric is calculated ( PROMPT or RESPONSE ).

```
import datarobot as dr
custom_model_version = dr.CustomModelVersion.get(version_id)
insight_config = dr.InsightsConfiguration.create(
    custom_model_version_id=custom_model_version.id,
    insight_name="Toxicity Check",
    insight_type=dr.InsightTypes.OOTB_METRIC,
    ootb_metric_name="Toxicity",
    stage=dr.InsightStage.RESPONSE
)
insight_config
```

## Set up test configurations

To configure test parameters:

```
eval_dataset = dr.EvaluationDatasetConfiguration.create(
    name="Robustness Test Dataset",
    dataset_id=dataset.id
)
cost_config = dr.CostConfiguration.create(
    name="Test Cost Config",
    cost_per_token=0.0001
)
insight_config.update(
    evaluation_dataset_configuration_id=eval_dataset.id,
    cost_configuration_id=cost_config.id
)
```

## Run robustness tests

To execute tests and review results:

```
insight_config = dr.InsightsConfiguration.get(insight_config_id)
if insight_config.execution_status == "COMPLETED":
    results = insight_config.get_results()
    print(f"Test results: {results}")
elif insight_config.execution_status == "ERROR":
    print(f"Test failed: {insight_config.error_message}")
    print(f"Resolution: {insight_config.error_resolution}")
```

## Configure multiple insights

To set up multiple tests for comprehensive validation:

```
insights = [
    {
        "insight_name": "Toxicity Check",
        "ootb_metric_name": "Toxicity",
        "stage": dr.InsightStage.RESPONSE
    },
    {
        "insight_name": "Relevance Check",
        "ootb_metric_name": "Relevance",
        "stage": dr.InsightStage.RESPONSE
    },
    {
        "insight_name": "Coherence Check",
        "ootb_metric_name": "Coherence",
        "stage": dr.InsightStage.RESPONSE
    }
]
for insight_config in insights:
    dr.InsightsConfiguration.create(
        custom_model_version_id=custom_model_version.id,
        **insight_config
    )
```

## Get an insight configuration

To retrieve a specific insight configuration:

```
insight_config = dr.InsightsConfiguration.get(insight_config_id)
print(f"Insight name: {insight_config.insight_name}")
print(f"Insight type: {insight_config.insight_type}")
print(f"Execution status: {insight_config.execution_status}")
```

## List insight configurations

Get all insights for a custom model version:

```
custom_model_version = dr.CustomModelVersion.get(version_id)
insights = dr.InsightsConfiguration.list(custom_model_version_id=custom_model_version.id)
for insight in insights:
    print(f"{insight.insight_name}: {insight.execution_status}")
```