AI Robustness Tests (GenAI)¶
This page outlines the operations, endpoints, parameters, and example requests and responses for the AI Robustness Tests (GenAI).
POST /api/v2/genai/costMetricConfigurations/¶
Create a new cost metric configuration.
Code samples¶
curl -X POST https://app.datarobot.com/api/v2/genai/costMetricConfigurations/ \
-H "Content-Type: application/json" \
-H "Accept: application/json" \
-H "Authorization: Bearer {access-token}" \
-d '{CreateCostMetricConfigurationRequest}'
Body parameter¶
{
"costMetricConfigurations": [
{
"currencyCode": "USD",
"customModelLLMValidationId": "string",
"inputTokenPrice": 0.01,
"llmId": "azure-openai-gpt-3.5-turbo",
"outputTokenPrice": 0.01,
"referenceInputTokenCount": 1000,
"referenceOutputTokenCount": 1000
}
],
"name": "string",
"playgroundId": "string",
"useCaseId": "string"
}
Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
| body | body | CreateCostMetricConfigurationRequest | true | none |
Example responses¶
201 Response
{
"costConfigurationId": "string",
"costMetricConfigurations": [
{
"currencyCode": "USD",
"customModelLLMValidationId": "string",
"inputTokenPrice": 0.01,
"llmId": "azure-openai-gpt-3.5-turbo",
"outputTokenPrice": 0.01,
"referenceInputTokenCount": 1000,
"referenceOutputTokenCount": 1000
}
],
"name": "string",
"playgroundId": "string",
"useCaseId": "string"
}
Responses¶
| Status | Meaning | Description | Schema |
|---|---|---|---|
| 201 | Created | Cost configuration created successfully | CostMetricConfigurationResponse |
| 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse |
To perform this operation, you must be authenticated by means of one of the following methods:
BearerAuth
DELETE /api/v2/genai/costMetricConfigurations/{costMetricConfigurationId}/¶
Delete an existing cost metric configuration.
Code samples¶
curl -X DELETE https://app.datarobot.com/api/v2/genai/costMetricConfigurations/{costMetricConfigurationId}/ \
-H "Accept: application/json" \
-H "Authorization: Bearer {access-token}"
Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
| costMetricConfigurationId | path | string | true | The ID of the cost metric configuration to delete. |
Example responses¶
422 Response
{
"detail": [
{
"loc": [
"string"
],
"msg": "string",
"type": "string"
}
]
}
Responses¶
| Status | Meaning | Description | Schema |
|---|---|---|---|
| 204 | No Content | Cost metric configuration successfully deleted. | None |
| 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse |
To perform this operation, you must be authenticated by means of one of the following methods:
BearerAuth
GET /api/v2/genai/costMetricConfigurations/{costMetricConfigurationId}/¶
Retrieve an existing cost metric configuration.
Code samples¶
curl -X GET https://app.datarobot.com/api/v2/genai/costMetricConfigurations/{costMetricConfigurationId}/ \
-H "Accept: application/json" \
-H "Authorization: Bearer {access-token}"
Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
| costMetricConfigurationId | path | string | true | The ID of the cost metric configuration to retrieve. |
Example responses¶
200 Response
{
"costConfigurationId": "string",
"costMetricConfigurations": [
{
"currencyCode": "USD",
"customModelLLMValidationId": "string",
"inputTokenPrice": 0.01,
"llmId": "azure-openai-gpt-3.5-turbo",
"outputTokenPrice": 0.01,
"referenceInputTokenCount": 1000,
"referenceOutputTokenCount": 1000
}
],
"name": "string",
"playgroundId": "string",
"useCaseId": "string"
}
Responses¶
| Status | Meaning | Description | Schema |
|---|---|---|---|
| 200 | OK | Cost metric configuration successfully retrieved. | CostMetricConfigurationResponse |
| 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse |
To perform this operation, you must be authenticated by means of one of the following methods:
BearerAuth
PATCH /api/v2/genai/costMetricConfigurations/{costMetricConfigurationId}/¶
Edit an existing cost metric configuration.
Code samples¶
curl -X PATCH https://app.datarobot.com/api/v2/genai/costMetricConfigurations/{costMetricConfigurationId}/ \
-H "Content-Type: application/json" \
-H "Accept: application/json" \
-H "Authorization: Bearer {access-token}" \
-d '{undefined}'
Body parameter¶
{
"costMetricConfigurations": [
{
"currencyCode": "USD",
"customModelLLMValidationId": "string",
"inputTokenPrice": 0.01,
"llmId": "azure-openai-gpt-3.5-turbo",
"outputTokenPrice": 0.01,
"referenceInputTokenCount": 1000,
"referenceOutputTokenCount": 1000
}
],
"name": "string"
}
Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
| costMetricConfigurationId | path | string | true | The ID of the cost metric configuration to edit. |
| body | body | EditCostMetricConfigurationRequest | true | none |
Example responses¶
200 Response
{
"costConfigurationId": "string",
"costMetricConfigurations": [
{
"currencyCode": "USD",
"customModelLLMValidationId": "string",
"inputTokenPrice": 0.01,
"llmId": "azure-openai-gpt-3.5-turbo",
"outputTokenPrice": 0.01,
"referenceInputTokenCount": 1000,
"referenceOutputTokenCount": 1000
}
],
"name": "string",
"playgroundId": "string",
"useCaseId": "string"
}
Responses¶
| Status | Meaning | Description | Schema |
|---|---|---|---|
| 200 | OK | Cost metric configuration successfully updated. | CostMetricConfigurationResponse |
| 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse |
To perform this operation, you must be authenticated by means of one of the following methods:
BearerAuth
GET /api/v2/genai/evaluationDatasetConfigurations/¶
List evaluation dataset configurations.
Code samples¶
curl -X GET https://app.datarobot.com/api/v2/genai/evaluationDatasetConfigurations/?useCaseId=string&playgroundId=string \
-H "Accept: application/json" \
-H "Authorization: Bearer {access-token}"
Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
| useCaseId | query | string | true | Only retrieve the evaluation dataset configurations associated with this use case ID. |
| playgroundId | query | string | true | Only retrieve the evaluation dataset configuration associated with this playground ID. |
| evaluationDatasetConfigurationId | query | any | false | Only retrieve the evaluation dataset configuration with this ID. |
| offset | query | integer | false | Skip the specified number of values. |
| limit | query | integer | false | Retrieve only the specified number of values. |
| search | query | any | false | Only retrieve the evaluation dataset configurations matching the search query. |
| sort | query | any | false | Apply this sort order to the results. Valid options are "name", "creationUserId", "creationDate", "datasetId", "userName", "datasetName", "promptColumnName", "responseColumnName". Prefix the attribute name with a dash to sort in descending order, e.g., sort=-creationDate. |
| correctnessEnabledOnly | query | boolean | false | If true, only retrieve the evaluation dataset configurations with correctness enabled. The default is false. |
| completedOnly | query | boolean | false | If true, only retrieve the evaluation dataset configurations where the evaluation dataset is in the completed status. The default is false. |
Example responses¶
200 Response
{
"count": 0,
"data": [
{
"correctnessEnabled": true,
"creationDate": "2019-08-24T14:15:22Z",
"creationUserId": "string",
"datasetId": "string",
"datasetName": "string",
"errorMessage": "string",
"executionStatus": "NEW",
"id": "string",
"name": "string",
"playgroundId": "string",
"promptColumnName": "string",
"responseColumnName": "string",
"rowsCount": 0,
"size": 0,
"tenantId": "string",
"useCaseId": "string",
"userName": "string"
}
],
"next": "string",
"previous": "string",
"totalCount": 0
}
Responses¶
| Status | Meaning | Description | Schema |
|---|---|---|---|
| 200 | OK | Evaluation dataset configurations successfully retrieved. | ListEvaluationDatasetConfigurationResponse |
| 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse |
To perform this operation, you must be authenticated by means of one of the following methods:
BearerAuth
POST /api/v2/genai/evaluationDatasetConfigurations/¶
Create a new evaluation dataset configuration.
Code samples¶
curl -X POST https://app.datarobot.com/api/v2/genai/evaluationDatasetConfigurations/ \
-H "Content-Type: application/json" \
-H "Accept: application/json" \
-H "Authorization: Bearer {access-token}" \
-d '{CreateEvaluationDatasetConfigurationRequest}'
Body parameter¶
{
"correctnessEnabled": true,
"datasetId": "string",
"isSyntheticDataset": false,
"name": "string",
"playgroundId": "string",
"promptColumnName": "string",
"responseColumnName": "string",
"useCaseId": "string"
}
Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
| body | body | CreateEvaluationDatasetConfigurationRequest | true | none |
Example responses¶
201 Response
{
"correctnessEnabled": true,
"creationDate": "2019-08-24T14:15:22Z",
"creationUserId": "string",
"datasetId": "string",
"datasetName": "string",
"errorMessage": "string",
"executionStatus": "NEW",
"id": "string",
"name": "string",
"playgroundId": "string",
"promptColumnName": "string",
"responseColumnName": "string",
"rowsCount": 0,
"size": 0,
"tenantId": "string",
"useCaseId": "string",
"userName": "string"
}
Responses¶
| Status | Meaning | Description | Schema |
|---|---|---|---|
| 201 | Created | Evaluation dataset configuration successfully created | EvaluationDatasetConfigurationResponse |
| 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse |
To perform this operation, you must be authenticated by means of one of the following methods:
BearerAuth
DELETE /api/v2/genai/evaluationDatasetConfigurations/{evaluationDatasetConfigurationId}/¶
Delete an existing evaluation dataset configuration.
Code samples¶
curl -X DELETE https://app.datarobot.com/api/v2/genai/evaluationDatasetConfigurations/{evaluationDatasetConfigurationId}/ \
-H "Accept: application/json" \
-H "Authorization: Bearer {access-token}"
Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
| evaluationDatasetConfigurationId | path | string | true | The ID of the evaluation dataset configuration to delete. |
Example responses¶
422 Response
{
"detail": [
{
"loc": [
"string"
],
"msg": "string",
"type": "string"
}
]
}
Responses¶
| Status | Meaning | Description | Schema |
|---|---|---|---|
| 204 | No Content | Evaluation dataset configuration successfully deleted. | None |
| 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse |
To perform this operation, you must be authenticated by means of one of the following methods:
BearerAuth
GET /api/v2/genai/evaluationDatasetConfigurations/{evaluationDatasetConfigurationId}/¶
Retrieve an existing evaluation dataset configuration.
Code samples¶
curl -X GET https://app.datarobot.com/api/v2/genai/evaluationDatasetConfigurations/{evaluationDatasetConfigurationId}/ \
-H "Accept: application/json" \
-H "Authorization: Bearer {access-token}"
Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
| evaluationDatasetConfigurationId | path | string | true | The ID of the evaluation dataset configuration to retrieve. |
Example responses¶
200 Response
{
"correctnessEnabled": true,
"creationDate": "2019-08-24T14:15:22Z",
"creationUserId": "string",
"datasetId": "string",
"datasetName": "string",
"errorMessage": "string",
"executionStatus": "NEW",
"id": "string",
"name": "string",
"playgroundId": "string",
"promptColumnName": "string",
"responseColumnName": "string",
"rowsCount": 0,
"size": 0,
"tenantId": "string",
"useCaseId": "string",
"userName": "string"
}
Responses¶
| Status | Meaning | Description | Schema |
|---|---|---|---|
| 200 | OK | Evaluation dataset configuration successfully retrieved. | EvaluationDatasetConfigurationResponse |
| 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse |
To perform this operation, you must be authenticated by means of one of the following methods:
BearerAuth
PATCH /api/v2/genai/evaluationDatasetConfigurations/{evaluationDatasetConfigurationId}/¶
Edit an existing evaluation dataset configuration.
Code samples¶
curl -X PATCH https://app.datarobot.com/api/v2/genai/evaluationDatasetConfigurations/{evaluationDatasetConfigurationId}/ \
-H "Content-Type: application/json" \
-H "Accept: application/json" \
-H "Authorization: Bearer {access-token}" \
-d '{undefined}'
Body parameter¶
{
"correctnessEnabled": true,
"datasetId": "string",
"name": "string",
"promptColumnName": "string",
"responseColumnName": "string"
}
Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
| evaluationDatasetConfigurationId | path | string | true | The ID of the evaluation dataset configuration to edit. |
| body | body | EditEvaluationDatasetConfigurationRequest | true | none |
Example responses¶
200 Response
{
"correctnessEnabled": true,
"creationDate": "2019-08-24T14:15:22Z",
"creationUserId": "string",
"datasetId": "string",
"datasetName": "string",
"errorMessage": "string",
"executionStatus": "NEW",
"id": "string",
"name": "string",
"playgroundId": "string",
"promptColumnName": "string",
"responseColumnName": "string",
"rowsCount": 0,
"size": 0,
"tenantId": "string",
"useCaseId": "string",
"userName": "string"
}
Responses¶
| Status | Meaning | Description | Schema |
|---|---|---|---|
| 200 | OK | Evaluation dataset configuration successfully updated. | EvaluationDatasetConfigurationResponse |
| 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse |
To perform this operation, you must be authenticated by means of one of the following methods:
BearerAuth
DELETE /api/v2/genai/evaluationDatasetMetricAggregations/¶
Delete the evaluation dataset metric aggregation associated with the specified LLM blueprint IDs and/or chat IDs.
Code samples¶
curl -X DELETE https://app.datarobot.com/api/v2/genai/evaluationDatasetMetricAggregations/ \
-H "Accept: application/json" \
-H "Authorization: Bearer {access-token}"
Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
| llmBlueprintIds | query | any | false | The IDs of the LLM blueprints to delete the associated evaluation dataset metric aggregation for. If both llmBlueprintIds and chatIds are specified, will delete the aggregation record only if it matches both criteria. |
| chatIds | query | any | false | The IDs of the chats to delete the associated evaluation dataset metric aggregation for. If both llmBlueprintIds and chatIds are specified, will delete the aggregation record only if it matches both criteria. |
Example responses¶
422 Response
{
"detail": [
{
"loc": [
"string"
],
"msg": "string",
"type": "string"
}
]
}
Responses¶
| Status | Meaning | Description | Schema |
|---|---|---|---|
| 204 | No Content | Evaluation dataset metric aggregation successfully deleted. | None |
| 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse |
To perform this operation, you must be authenticated by means of one of the following methods:
BearerAuth
GET /api/v2/genai/evaluationDatasetMetricAggregations/¶
List evaluation dataset metric aggregations.
Code samples¶
curl -X GET https://app.datarobot.com/api/v2/genai/evaluationDatasetMetricAggregations/ \
-H "Accept: application/json" \
-H "Authorization: Bearer {access-token}"
Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
| llmBlueprintIds | query | any | false | Only retrieve the evaluation dataset metric aggregations associated with these LLM blueprint IDs. |
| chatIds | query | any | false | Only retrieve the evaluation dataset metric aggregations associated with these chat IDs. |
| evaluationDatasetConfigurationIds | query | any | false | Only retrieve the evaluation dataset metric aggregations associated with these evaluation dataset configuration IDs. |
| metricNames | query | any | false | Only retrieve the evaluation dataset metric aggregations associated with these metric names. |
| aggregationTypes | query | any | false | Only retrieve the evaluation dataset metric aggregations associated with these aggregation types. |
| currentConfigurationOnly | query | boolean | false | Only retrieve the evaluation dataset metric aggregations associated with the current configuration of the llmblueprints. |
| sort | query | any | false | Apply this sort order to the results. Valid options are "name", "creationUserId", "creationDate", "datasetId", "userName", "datasetName", "promptColumnName", "responseColumnName". Prefix the attribute name with a dash to sort in descending order, e.g., sort=-creationDate. |
| offset | query | integer | false | Skip the specified number of values. |
| limit | query | integer | false | Retrieve only the specified number of values. |
| nonErroredOnly | query | boolean | false | If true, only retrieve the evaluation dataset metric aggregations that are in a non-errored status. The default is false. |
Example responses¶
200 Response
{
"count": 0,
"data": [
{
"aggregationType": "average",
"aggregationValue": 0,
"chatId": "string",
"chatLink": "string",
"chatName": "string",
"creationDate": "2019-08-24T14:15:22Z",
"creationUserId": "string",
"creationUserName": "string",
"datasetId": "string",
"datasetName": "string",
"evaluationDatasetConfigurationId": "string",
"llmBlueprintId": "string",
"metricName": "string",
"ootbDatasetName": "jailbreak-v1.csv",
"tenantId": "string"
}
],
"next": "string",
"previous": "string",
"totalCount": 0
}
Responses¶
| Status | Meaning | Description | Schema |
|---|---|---|---|
| 200 | OK | Evaluation dataset metric aggregations successfully retrieved. | ListEvaluationDatasetMetricAggregationResponse |
| 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse |
To perform this operation, you must be authenticated by means of one of the following methods:
BearerAuth
POST /api/v2/genai/evaluationDatasetMetricAggregations/¶
Create a new evaluation dataset metric aggregation.
Code samples¶
curl -X POST https://app.datarobot.com/api/v2/genai/evaluationDatasetMetricAggregations/ \
-H "Content-Type: application/json" \
-H "Accept: application/json" \
-H "Authorization: Bearer {access-token}" \
-d '{CreateEvaluationDatasetMetricAggregationRequest}'
Body parameter¶
{
"chatName": "Aggregated chat",
"evaluationDatasetConfigurationId": "string",
"insightsConfiguration": [
{
"aggregationTypes": [
"average"
],
"costConfigurationId": "string",
"customMetricId": "string",
"customModelLLMValidationId": "string",
"deploymentId": "string",
"errorMessage": "string",
"errorResolution": [
"string"
],
"evaluationDatasetConfigurationId": "string",
"executionStatus": "NEW",
"insightName": "string",
"insightType": "Reference",
"isTransferable": false,
"llmId": "azure-openai-gpt-3.5-turbo",
"llmIsActive": true,
"llmIsDeprecated": true,
"modelId": "string",
"modelPackageRegisteredModelId": "string",
"moderationConfiguration": {
"guardConditions": [
{
"comparand": 0,
"comparator": "greaterThan"
}
],
"intervention": {
"action": "block",
"message": "string"
}
},
"nemoMetricId": "string",
"ootbMetricId": "string",
"ootbMetricName": "latency",
"resultUnit": "s",
"sidecarModelMetricMetadata": {
"expectedResponseColumnName": "string",
"promptColumnName": "string",
"responseColumnName": "string",
"targetColumnName": "string"
},
"sidecarModelMetricValidationId": "string",
"stage": "prompt_pipeline"
}
],
"llmBlueprintIds": [
"string"
]
}
Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
| body | body | CreateEvaluationDatasetMetricAggregationRequest | true | none |
Example responses¶
202 Response
{
"chatIds": [
"string"
],
"jobId": "string"
}
Responses¶
| Status | Meaning | Description | Schema |
|---|---|---|---|
| 202 | Accepted | Evaluation dataset metric aggregation job successfully accepted. Follow the Location header to poll for job execution status. |
CreateEvaluationDatasetMetricAggregationResponse |
| 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse |
To perform this operation, you must be authenticated by means of one of the following methods:
BearerAuth
GET /api/v2/genai/evaluationDatasetMetricAggregations/aggregateByLLMBlueprint/¶
List evaluation dataset metric aggregations aggregated by llm blueprint.
Code samples¶
curl -X GET https://app.datarobot.com/api/v2/genai/evaluationDatasetMetricAggregations/aggregateByLLMBlueprint/ \
-H "Accept: application/json" \
-H "Authorization: Bearer {access-token}"
Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
| llmBlueprintIds | query | any | false | Only retrieve the evaluation dataset metric aggregations associated with these LLM blueprint IDs. |
| chatIds | query | any | false | Only retrieve the evaluation dataset metric aggregations associated with these chat IDs. |
| evaluationDatasetConfigurationIds | query | any | false | Only retrieve the evaluation dataset metric aggregations associated with these evaluation dataset configuration IDs. |
| metricNames | query | any | false | Only retrieve the evaluation dataset metric aggregations associated with these metric names. |
| aggregationTypes | query | any | false | Only retrieve the evaluation dataset metric aggregations associated with these aggregation types. |
| currentConfigurationOnly | query | boolean | false | Only retrieve the evaluation dataset metric aggregations associated with the current configuration of the llmblueprints. |
| offset | query | integer | false | Skip the specified number of values. |
| limit | query | integer | false | Retrieve only the specified number of values. |
| nonErroredOnly | query | boolean | false | If true, only retrieve the evaluation dataset metric aggregations that are in a non-errored status. The default is false. |
Example responses¶
200 Response
{
"count": 0,
"data": [
{
"aggregatedItemCount": 0,
"aggregatedItemDetails": [
{
"chatId": "string",
"chatLink": "string",
"chatName": "string",
"creationDate": "2019-08-24T14:15:22Z",
"creationUserId": "string",
"creationUserName": "string"
}
],
"aggregationType": "average",
"aggregationValue": 0,
"datasetId": "string",
"datasetName": "string",
"evaluationDatasetConfigurationId": "string",
"llmBlueprintId": "string",
"metricName": "string",
"ootbDatasetName": "jailbreak-v1.csv",
"tenantId": "string"
}
],
"next": "string",
"previous": "string",
"totalCount": 0
}
Responses¶
| Status | Meaning | Description | Schema |
|---|---|---|---|
| 200 | OK | Evaluation dataset metric aggregations aggregated by llm blueprint successfully retrieved. | ListEvaluationDatasetMetricAggregationAggregatedByLLMBlueprintResponse |
| 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse |
To perform this operation, you must be authenticated by means of one of the following methods:
BearerAuth
GET /api/v2/genai/evaluationDatasetMetricAggregations/uniqueFieldValues/{uniqueField}/¶
List evaluation dataset metric aggregations unique computed metrics.
Code samples¶
curl -X GET https://app.datarobot.com/api/v2/genai/evaluationDatasetMetricAggregations/uniqueFieldValues/{uniqueField}/ \
-H "Accept: application/json" \
-H "Authorization: Bearer {access-token}"
Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
| uniqueField | path | EvaluationDatasetMetricAggregationFieldQueryParam | true | Retrieve the list of this unique field. |
| llmBlueprintIds | query | any | false | Only retrieve the list of the unique field associated with these LLM blueprint IDs. |
| metricNames | query | any | false | Only retrieve the list of the unique field associated with these metric names. |
| chatIds | query | any | false | Only retrieve the list of the unique field associated with these chat IDs. |
| evaluationDatasetConfigurationIds | query | any | false | Only retrieve the list of the unique field associated with these evaluation dataset configuration IDs. |
| aggregationTypes | query | any | false | Only retrieve the list of the unique field associated with these aggregation types. |
| currentConfigurationOnly | query | boolean | false | Only retrieve the evaluation dataset metric aggregations associated with the current configuration of the llmblueprints. |
| offset | query | integer | false | Skip the specified number of values. |
| limit | query | integer | false | Retrieve only the specified number of values. |
| nonErroredOnly | query | boolean | false | If true, only retrieve the list of the unique field for aggregation records that are in a non-errored status. The default is false. |
Enumerated Values¶
| Parameter | Value |
|---|---|
| uniqueField | [metricName, llmBlueprintId, aggregationType, evaluationDatasetConfigurationId] |
Example responses¶
200 Response
{
"count": 0,
"data": [
{
"uniqueFieldValue": "string"
}
],
"next": "string",
"previous": "string",
"totalCount": 0
}
Responses¶
| Status | Meaning | Description | Schema |
|---|---|---|---|
| 200 | OK | Evaluation dataset metric aggregations unique computed metrics successfully retrieved. | ListEvaluationDatasetMetricAggregationUniqueFieldValuesResponse |
| 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse |
To perform this operation, you must be authenticated by means of one of the following methods:
BearerAuth
GET /api/v2/genai/llmTestConfigurations/¶
List LLM test configuration.
Code samples¶
curl -X GET https://app.datarobot.com/api/v2/genai/llmTestConfigurations/ \
-H "Accept: application/json" \
-H "Authorization: Bearer {access-token}"
Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
| useCaseId | query | any | false | Use Case ID. |
| offset | query | integer | false | Skip the specified number of values. |
| limit | query | integer | false | Retrieve only the specified number of values. |
| testConfigType | query | LLMTestConfigurationType | false | Whether to return out-of-the-box (ootb) or custom LLM test configurations in the response. |
Enumerated Values¶
| Parameter | Value |
|---|---|
| testConfigType | [ootb, custom] |
Example responses¶
200 Response
{
"count": 0,
"data": [
{
"creationDate": "2019-08-24T14:15:22Z",
"creationUserId": "string",
"datasetEvaluations": [
{
"errorMessage": "string",
"evaluationDatasetConfigurationId": "string",
"evaluationDatasetName": "string",
"evaluationName": "string",
"insightConfiguration": {
"aggregationTypes": [
"average"
],
"costConfigurationId": "string",
"customMetricId": "string",
"customModelLLMValidationId": "string",
"deploymentId": "string",
"errorMessage": "string",
"errorResolution": [
"string"
],
"evaluationDatasetConfigurationId": "string",
"executionStatus": "NEW",
"insightName": "string",
"insightType": "Reference",
"isTransferable": false,
"llmId": "azure-openai-gpt-3.5-turbo",
"llmIsActive": true,
"llmIsDeprecated": true,
"modelId": "string",
"modelPackageRegisteredModelId": "string",
"moderationConfiguration": {
"guardConditions": [
{
"comparand": 0,
"comparator": "greaterThan"
}
],
"intervention": {
"action": "block",
"message": "string"
}
},
"nemoMetricId": "string",
"ootbMetricId": "string",
"ootbMetricName": "latency",
"resultUnit": "s",
"sidecarModelMetricMetadata": {
"expectedResponseColumnName": "string",
"promptColumnName": "string",
"responseColumnName": "string",
"targetColumnName": "string"
},
"sidecarModelMetricValidationId": "string",
"stage": "prompt_pipeline"
},
"insightGradingCriteria": {
"passThreshold": 100
},
"maxNumPrompts": 100,
"ootbDataset": {
"datasetName": "jailbreak-v1.csv",
"datasetUrl": "https://s3.amazonaws.com/datarobot_public_datasets/genai/jailbreak-v1.csv",
"promptColumnName": "string",
"responseColumnName": "string",
"rowsCount": 0,
"warning": "string"
},
"promptSamplingStrategy": "random_without_replacement"
}
],
"description": "string",
"errorMessage": "string",
"id": "string",
"isOutOfTheBoxTestConfiguration": true,
"lastUpdateDate": "2019-08-24T14:15:22Z",
"lastUpdateUserId": "string",
"llmTestGradingCriteria": {
"passThreshold": 100
},
"name": "string",
"useCaseId": "string",
"warnings": [
{
"property1": "string",
"property2": "string"
}
]
}
],
"next": "string",
"previous": "string",
"totalCount": 0
}
Responses¶
| Status | Meaning | Description | Schema |
|---|---|---|---|
| 200 | OK | Successful Response | ListLLMTestConfigurationsResponse |
| 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse |
To perform this operation, you must be authenticated by means of one of the following methods:
BearerAuth
POST /api/v2/genai/llmTestConfigurations/¶
Create a new LLM test configuration.
Code samples¶
curl -X POST https://app.datarobot.com/api/v2/genai/llmTestConfigurations/ \
-H "Content-Type: application/json" \
-H "Accept: application/json" \
-H "Authorization: Bearer {access-token}" \
-d '{CreateLLMTestConfigurationRequest}'
Body parameter¶
{
"datasetEvaluations": [
{
"evaluationDatasetConfigurationId": "string",
"evaluationName": "string",
"insightConfiguration": {
"aggregationTypes": [
"average"
],
"costConfigurationId": "string",
"customMetricId": "string",
"customModelLLMValidationId": "string",
"deploymentId": "string",
"errorMessage": "string",
"errorResolution": [
"string"
],
"evaluationDatasetConfigurationId": "string",
"executionStatus": "NEW",
"insightName": "string",
"insightType": "Reference",
"isTransferable": false,
"llmId": "azure-openai-gpt-3.5-turbo",
"llmIsActive": true,
"llmIsDeprecated": true,
"modelId": "string",
"modelPackageRegisteredModelId": "string",
"moderationConfiguration": {
"guardConditions": [
{
"comparand": 0,
"comparator": "greaterThan"
}
],
"intervention": {
"action": "block",
"message": "string"
}
},
"nemoMetricId": "string",
"ootbMetricId": "string",
"ootbMetricName": "latency",
"resultUnit": "s",
"sidecarModelMetricMetadata": {
"expectedResponseColumnName": "string",
"promptColumnName": "string",
"responseColumnName": "string",
"targetColumnName": "string"
},
"sidecarModelMetricValidationId": "string",
"stage": "prompt_pipeline"
},
"insightGradingCriteria": {
"passThreshold": 100
},
"maxNumPrompts": 0,
"ootbDatasetName": "jailbreak-v1.csv",
"promptSamplingStrategy": "random_without_replacement"
}
],
"description": "",
"llmTestGradingCriteria": {
"passThreshold": 100
},
"name": "string",
"useCaseId": "string"
}
Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
| body | body | CreateLLMTestConfigurationRequest | true | none |
Example responses¶
201 Response
{
"creationDate": "2019-08-24T14:15:22Z",
"creationUserId": "string",
"datasetEvaluations": [
{
"errorMessage": "string",
"evaluationDatasetConfigurationId": "string",
"evaluationDatasetName": "string",
"evaluationName": "string",
"insightConfiguration": {
"aggregationTypes": [
"average"
],
"costConfigurationId": "string",
"customMetricId": "string",
"customModelLLMValidationId": "string",
"deploymentId": "string",
"errorMessage": "string",
"errorResolution": [
"string"
],
"evaluationDatasetConfigurationId": "string",
"executionStatus": "NEW",
"insightName": "string",
"insightType": "Reference",
"isTransferable": false,
"llmId": "azure-openai-gpt-3.5-turbo",
"llmIsActive": true,
"llmIsDeprecated": true,
"modelId": "string",
"modelPackageRegisteredModelId": "string",
"moderationConfiguration": {
"guardConditions": [
{
"comparand": 0,
"comparator": "greaterThan"
}
],
"intervention": {
"action": "block",
"message": "string"
}
},
"nemoMetricId": "string",
"ootbMetricId": "string",
"ootbMetricName": "latency",
"resultUnit": "s",
"sidecarModelMetricMetadata": {
"expectedResponseColumnName": "string",
"promptColumnName": "string",
"responseColumnName": "string",
"targetColumnName": "string"
},
"sidecarModelMetricValidationId": "string",
"stage": "prompt_pipeline"
},
"insightGradingCriteria": {
"passThreshold": 100
},
"maxNumPrompts": 100,
"ootbDataset": {
"datasetName": "jailbreak-v1.csv",
"datasetUrl": "https://s3.amazonaws.com/datarobot_public_datasets/genai/jailbreak-v1.csv",
"promptColumnName": "string",
"responseColumnName": "string",
"rowsCount": 0,
"warning": "string"
},
"promptSamplingStrategy": "random_without_replacement"
}
],
"description": "string",
"errorMessage": "string",
"id": "string",
"isOutOfTheBoxTestConfiguration": true,
"lastUpdateDate": "2019-08-24T14:15:22Z",
"lastUpdateUserId": "string",
"llmTestGradingCriteria": {
"passThreshold": 100
},
"name": "string",
"useCaseId": "string",
"warnings": [
{
"property1": "string",
"property2": "string"
}
]
}
Responses¶
| Status | Meaning | Description | Schema |
|---|---|---|---|
| 201 | Created | Successful Response | LLMTestConfigurationResponse |
| 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse |
To perform this operation, you must be authenticated by means of one of the following methods:
BearerAuth
GET /api/v2/genai/llmTestConfigurations/nonOotbDatasets/¶
List the supported non out-of-the-box datasets that can be used with an LLM test configuration.
Code samples¶
curl -X GET https://app.datarobot.com/api/v2/genai/llmTestConfigurations/nonOotbDatasets/?useCaseId=string \
-H "Accept: application/json" \
-H "Authorization: Bearer {access-token}"
Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
| useCaseId | query | string | true | Use Case ID. |
| offset | query | integer | false | Skip the specified number of values. |
| limit | query | integer | false | Retrieve only the specified number of values. |
| search | query | any | false | Only retrieve the datasets with names matching the search query. |
Example responses¶
200 Response
{
"count": 0,
"data": [
{
"correctnessEnabled": true,
"creationDate": "2019-08-24T14:15:22Z",
"creationUserId": "string",
"datasetId": "string",
"datasetName": "string",
"errorMessage": "string",
"executionStatus": "NEW",
"id": "string",
"name": "string",
"playgroundId": "string",
"promptColumnName": "string",
"responseColumnName": "string",
"rowsCount": 0,
"size": 0,
"tenantId": "string",
"useCaseId": "string",
"userName": "string"
}
],
"next": "string",
"previous": "string",
"totalCount": 0
}
Responses¶
| Status | Meaning | Description | Schema |
|---|---|---|---|
| 200 | OK | Non out-of-the-box datasets successfully retrieved. | ListLLMTestConfigurationNonOOTBDatasetsResponse |
| 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse |
To perform this operation, you must be authenticated by means of one of the following methods:
BearerAuth
GET /api/v2/genai/llmTestConfigurations/ootbDatasets/¶
List the supported out-of-the-box datasets that can be used with an LLM test configuration.
Code samples¶
curl -X GET https://app.datarobot.com/api/v2/genai/llmTestConfigurations/ootbDatasets/ \
-H "Accept: application/json" \
-H "Authorization: Bearer {access-token}"
Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
| offset | query | integer | false | Skip the specified number of values. |
| limit | query | integer | false | Retrieve only the specified number of values. |
| search | query | any | false | Only retrieve the datasets with names matching the search query. |
Example responses¶
200 Response
{
"count": 0,
"data": [
{
"datasetName": "jailbreak-v1.csv",
"datasetUrl": "https://s3.amazonaws.com/datarobot_public_datasets/genai/jailbreak-v1.csv",
"promptColumnName": "string",
"responseColumnName": "string",
"rowsCount": 0,
"warning": "string"
}
],
"next": "string",
"previous": "string",
"totalCount": 0
}
Responses¶
| Status | Meaning | Description | Schema |
|---|---|---|---|
| 200 | OK | Out-of-the-box datasets successfully retrieved. | ListLLMTestConfigurationOOTBDatasetsResponse |
| 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse |
To perform this operation, you must be authenticated by means of one of the following methods:
BearerAuth
GET /api/v2/genai/llmTestConfigurations/supportedInsights/¶
List the supported LLM test insight configurations for the specified use case.
Code samples¶
curl -X GET https://app.datarobot.com/api/v2/genai/llmTestConfigurations/supportedInsights/ \
-H "Accept: application/json" \
-H "Authorization: Bearer {access-token}"
Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
| useCaseId | query | any | false | If specified, only retrieve the insights supported by this use case ID. |
| playgroundId | query | any | false | If specified, only retrieve the insights supported by the use case for which the playgroundId belongs. |
Example responses¶
200 Response
{
"datasetsCompatibility": [
{
"incompatibleDatasets": [
{
"datasetId": "string",
"datasetName": "string"
}
],
"insightName": "string"
}
],
"supportedInsightConfigurations": [
{
"aggregationTypes": [
"average"
],
"costConfigurationId": "string",
"customMetricId": "string",
"customModelLLMValidationId": "string",
"deploymentId": "string",
"errorMessage": "string",
"errorResolution": [
"string"
],
"evaluationDatasetConfigurationId": "string",
"executionStatus": "NEW",
"insightName": "string",
"insightType": "Reference",
"isTransferable": false,
"llmId": "azure-openai-gpt-3.5-turbo",
"llmIsActive": true,
"llmIsDeprecated": true,
"modelId": "string",
"modelPackageRegisteredModelId": "string",
"moderationConfiguration": {
"guardConditions": [
{
"comparand": 0,
"comparator": "greaterThan"
}
],
"intervention": {
"action": "block",
"message": "string"
}
},
"nemoMetricId": "string",
"ootbMetricId": "string",
"ootbMetricName": "latency",
"resultUnit": "s",
"sidecarModelMetricMetadata": {
"expectedResponseColumnName": "string",
"promptColumnName": "string",
"responseColumnName": "string",
"targetColumnName": "string"
},
"sidecarModelMetricValidationId": "string",
"stage": "prompt_pipeline"
}
]
}
Responses¶
| Status | Meaning | Description | Schema |
|---|---|---|---|
| 200 | OK | LLM test supported insight configurations successfully retrieved. | LLMTestConfigurationSupportedInsightsResponse |
| 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse |
To perform this operation, you must be authenticated by means of one of the following methods:
BearerAuth
DELETE /api/v2/genai/llmTestConfigurations/{llmTestConfigurationId}/¶
Delete an existing LLM test configuration.
Code samples¶
curl -X DELETE https://app.datarobot.com/api/v2/genai/llmTestConfigurations/{llmTestConfigurationId}/ \
-H "Accept: application/json" \
-H "Authorization: Bearer {access-token}"
Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
| llmTestConfigurationId | path | string | true | The ID of the LLM Test Configuration to delete. |
Example responses¶
422 Response
{
"detail": [
{
"loc": [
"string"
],
"msg": "string",
"type": "string"
}
]
}
Responses¶
| Status | Meaning | Description | Schema |
|---|---|---|---|
| 204 | No Content | Successful Response | None |
| 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse |
To perform this operation, you must be authenticated by means of one of the following methods:
BearerAuth
GET /api/v2/genai/llmTestConfigurations/{llmTestConfigurationId}/¶
Retrieve an existing LLM test configuration.
Code samples¶
curl -X GET https://app.datarobot.com/api/v2/genai/llmTestConfigurations/{llmTestConfigurationId}/ \
-H "Accept: application/json" \
-H "Authorization: Bearer {access-token}"
Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
| llmTestConfigurationId | path | string | true | The ID of the LLM Test Configuration to retrieve. |
Example responses¶
200 Response
{
"creationDate": "2019-08-24T14:15:22Z",
"creationUserId": "string",
"datasetEvaluations": [
{
"errorMessage": "string",
"evaluationDatasetConfigurationId": "string",
"evaluationDatasetName": "string",
"evaluationName": "string",
"insightConfiguration": {
"aggregationTypes": [
"average"
],
"costConfigurationId": "string",
"customMetricId": "string",
"customModelLLMValidationId": "string",
"deploymentId": "string",
"errorMessage": "string",
"errorResolution": [
"string"
],
"evaluationDatasetConfigurationId": "string",
"executionStatus": "NEW",
"insightName": "string",
"insightType": "Reference",
"isTransferable": false,
"llmId": "azure-openai-gpt-3.5-turbo",
"llmIsActive": true,
"llmIsDeprecated": true,
"modelId": "string",
"modelPackageRegisteredModelId": "string",
"moderationConfiguration": {
"guardConditions": [
{
"comparand": 0,
"comparator": "greaterThan"
}
],
"intervention": {
"action": "block",
"message": "string"
}
},
"nemoMetricId": "string",
"ootbMetricId": "string",
"ootbMetricName": "latency",
"resultUnit": "s",
"sidecarModelMetricMetadata": {
"expectedResponseColumnName": "string",
"promptColumnName": "string",
"responseColumnName": "string",
"targetColumnName": "string"
},
"sidecarModelMetricValidationId": "string",
"stage": "prompt_pipeline"
},
"insightGradingCriteria": {
"passThreshold": 100
},
"maxNumPrompts": 100,
"ootbDataset": {
"datasetName": "jailbreak-v1.csv",
"datasetUrl": "https://s3.amazonaws.com/datarobot_public_datasets/genai/jailbreak-v1.csv",
"promptColumnName": "string",
"responseColumnName": "string",
"rowsCount": 0,
"warning": "string"
},
"promptSamplingStrategy": "random_without_replacement"
}
],
"description": "string",
"errorMessage": "string",
"id": "string",
"isOutOfTheBoxTestConfiguration": true,
"lastUpdateDate": "2019-08-24T14:15:22Z",
"lastUpdateUserId": "string",
"llmTestGradingCriteria": {
"passThreshold": 100
},
"name": "string",
"useCaseId": "string",
"warnings": [
{
"property1": "string",
"property2": "string"
}
]
}
Responses¶
| Status | Meaning | Description | Schema |
|---|---|---|---|
| 200 | OK | Successful Response | LLMTestConfigurationResponse |
| 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse |
To perform this operation, you must be authenticated by means of one of the following methods:
BearerAuth
PATCH /api/v2/genai/llmTestConfigurations/{llmTestConfigurationId}/¶
Edit an existing LLM test configuration.
Code samples¶
curl -X PATCH https://app.datarobot.com/api/v2/genai/llmTestConfigurations/{llmTestConfigurationId}/ \
-H "Content-Type: application/json" \
-H "Accept: application/json" \
-H "Authorization: Bearer {access-token}" \
-d '{undefined}'
Body parameter¶
{
"datasetEvaluations": [
{
"evaluationDatasetConfigurationId": "string",
"evaluationName": "string",
"insightConfiguration": {
"aggregationTypes": [
"average"
],
"costConfigurationId": "string",
"customMetricId": "string",
"customModelLLMValidationId": "string",
"deploymentId": "string",
"errorMessage": "string",
"errorResolution": [
"string"
],
"evaluationDatasetConfigurationId": "string",
"executionStatus": "NEW",
"insightName": "string",
"insightType": "Reference",
"isTransferable": false,
"llmId": "azure-openai-gpt-3.5-turbo",
"llmIsActive": true,
"llmIsDeprecated": true,
"modelId": "string",
"modelPackageRegisteredModelId": "string",
"moderationConfiguration": {
"guardConditions": [
{
"comparand": 0,
"comparator": "greaterThan"
}
],
"intervention": {
"action": "block",
"message": "string"
}
},
"nemoMetricId": "string",
"ootbMetricId": "string",
"ootbMetricName": "latency",
"resultUnit": "s",
"sidecarModelMetricMetadata": {
"expectedResponseColumnName": "string",
"promptColumnName": "string",
"responseColumnName": "string",
"targetColumnName": "string"
},
"sidecarModelMetricValidationId": "string",
"stage": "prompt_pipeline"
},
"insightGradingCriteria": {
"passThreshold": 100
},
"maxNumPrompts": 0,
"ootbDatasetName": "jailbreak-v1.csv",
"promptSamplingStrategy": "random_without_replacement"
}
],
"description": "string",
"llmTestGradingCriteria": {
"passThreshold": 100
},
"name": "string"
}
Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
| llmTestConfigurationId | path | string | true | The ID of the LLM Test Configuration to update. |
| body | body | EditLLMTestConfigurationRequest | true | none |
Example responses¶
200 Response
{
"creationDate": "2019-08-24T14:15:22Z",
"creationUserId": "string",
"datasetEvaluations": [
{
"errorMessage": "string",
"evaluationDatasetConfigurationId": "string",
"evaluationDatasetName": "string",
"evaluationName": "string",
"insightConfiguration": {
"aggregationTypes": [
"average"
],
"costConfigurationId": "string",
"customMetricId": "string",
"customModelLLMValidationId": "string",
"deploymentId": "string",
"errorMessage": "string",
"errorResolution": [
"string"
],
"evaluationDatasetConfigurationId": "string",
"executionStatus": "NEW",
"insightName": "string",
"insightType": "Reference",
"isTransferable": false,
"llmId": "azure-openai-gpt-3.5-turbo",
"llmIsActive": true,
"llmIsDeprecated": true,
"modelId": "string",
"modelPackageRegisteredModelId": "string",
"moderationConfiguration": {
"guardConditions": [
{
"comparand": 0,
"comparator": "greaterThan"
}
],
"intervention": {
"action": "block",
"message": "string"
}
},
"nemoMetricId": "string",
"ootbMetricId": "string",
"ootbMetricName": "latency",
"resultUnit": "s",
"sidecarModelMetricMetadata": {
"expectedResponseColumnName": "string",
"promptColumnName": "string",
"responseColumnName": "string",
"targetColumnName": "string"
},
"sidecarModelMetricValidationId": "string",
"stage": "prompt_pipeline"
},
"insightGradingCriteria": {
"passThreshold": 100
},
"maxNumPrompts": 100,
"ootbDataset": {
"datasetName": "jailbreak-v1.csv",
"datasetUrl": "https://s3.amazonaws.com/datarobot_public_datasets/genai/jailbreak-v1.csv",
"promptColumnName": "string",
"responseColumnName": "string",
"rowsCount": 0,
"warning": "string"
},
"promptSamplingStrategy": "random_without_replacement"
}
],
"description": "string",
"errorMessage": "string",
"id": "string",
"isOutOfTheBoxTestConfiguration": true,
"lastUpdateDate": "2019-08-24T14:15:22Z",
"lastUpdateUserId": "string",
"llmTestGradingCriteria": {
"passThreshold": 100
},
"name": "string",
"useCaseId": "string",
"warnings": [
{
"property1": "string",
"property2": "string"
}
]
}
Responses¶
| Status | Meaning | Description | Schema |
|---|---|---|---|
| 200 | OK | Successful Response | LLMTestConfigurationResponse |
| 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse |
To perform this operation, you must be authenticated by means of one of the following methods:
BearerAuth
GET /api/v2/genai/llmTestResults/¶
List LLM test results.
Code samples¶
curl -X GET https://app.datarobot.com/api/v2/genai/llmTestResults/ \
-H "Accept: application/json" \
-H "Authorization: Bearer {access-token}"
Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
| llmTestConfigurationId | query | any | false | LLM Test Configuration ID. |
| llmBlueprintId | query | any | false | LLM Blueprint ID. |
| llmTestSuiteId | query | any | false | LLM Test Suite ID. |
| offset | query | integer | false | Skip the specified number of values. |
| limit | query | integer | false | Retrieve only the specified number of values. |
Example responses¶
200 Response
{
"count": 0,
"data": [
{
"creationDate": "2019-08-24T14:15:22Z",
"creationUserId": "string",
"creationUserName": "string",
"errorMessage": "string",
"errorResolution": "string",
"executionStatus": "NEW",
"gradingResult": "PASS",
"id": "string",
"insightEvaluationResults": [
{
"aggregationType": "average",
"aggregationValue": 0,
"chatId": "string",
"chatName": "string",
"evaluationDatasetConfigurationId": "string",
"evaluationDatasetName": "string",
"evaluationName": "string",
"executionStatus": "NEW",
"gradingResult": "PASS",
"id": "string",
"insightGradingCriteria": {
"passThreshold": 100
},
"lastUpdateDate": "2019-08-24T14:15:22Z",
"llmTestResultId": "string",
"maxNumPrompts": 0,
"metricName": "string",
"promptSamplingStrategy": "random_without_replacement"
}
],
"isOutOfTheBoxTestConfiguration": true,
"llmBlueprintId": "string",
"llmBlueprintSnapshot": {
"description": "string",
"id": "string",
"llmId": "azure-openai-gpt-3.5-turbo",
"llmSettings": {
"maxCompletionLength": 0,
"systemPrompt": "string",
"temperature": 0,
"topP": 0
},
"name": "string",
"playgroundId": "string",
"promptType": "CHAT_HISTORY_AWARE",
"snapshotDate": "2019-08-24T14:15:22Z",
"vectorDatabaseId": "string",
"vectorDatabaseSettings": {
"addNeighborChunks": false,
"maxDocumentsRetrievedPerPrompt": 1,
"maxTokens": 1,
"retriever": "SINGLE_LOOKUP_RETRIEVER"
}
},
"llmTestConfigurationId": "string",
"llmTestConfigurationName": "string",
"llmTestGradingCriteria": {
"passThreshold": 100
},
"llmTestSuiteId": "string",
"passPercentage": 0,
"useCaseId": "string"
}
],
"next": "string",
"previous": "string",
"totalCount": 0
}
Responses¶
| Status | Meaning | Description | Schema |
|---|---|---|---|
| 200 | OK | Successful Response | ListLLMTestResultResponse |
| 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse |
To perform this operation, you must be authenticated by means of one of the following methods:
BearerAuth
POST /api/v2/genai/llmTestResults/¶
Create a new LLM test result.
Code samples¶
curl -X POST https://app.datarobot.com/api/v2/genai/llmTestResults/ \
-H "Content-Type: application/json" \
-H "Accept: application/json" \
-H "Authorization: Bearer {access-token}" \
-d '{CreateLLMTestResultRequest}'
Body parameter¶
{
"llmBlueprintId": "string",
"llmTestConfigurationId": "string"
}
Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
| body | body | CreateLLMTestResultRequest | true | none |
Example responses¶
202 Response
{
"creationDate": "2019-08-24T14:15:22Z",
"creationUserId": "string",
"creationUserName": "string",
"errorMessage": "string",
"errorResolution": "string",
"executionStatus": "NEW",
"gradingResult": "PASS",
"id": "string",
"insightEvaluationResults": [
{
"aggregationType": "average",
"aggregationValue": 0,
"chatId": "string",
"chatName": "string",
"evaluationDatasetConfigurationId": "string",
"evaluationDatasetName": "string",
"evaluationName": "string",
"executionStatus": "NEW",
"gradingResult": "PASS",
"id": "string",
"insightGradingCriteria": {
"passThreshold": 100
},
"lastUpdateDate": "2019-08-24T14:15:22Z",
"llmTestResultId": "string",
"maxNumPrompts": 0,
"metricName": "string",
"promptSamplingStrategy": "random_without_replacement"
}
],
"isOutOfTheBoxTestConfiguration": true,
"llmBlueprintId": "string",
"llmBlueprintSnapshot": {
"description": "string",
"id": "string",
"llmId": "azure-openai-gpt-3.5-turbo",
"llmSettings": {
"maxCompletionLength": 0,
"systemPrompt": "string",
"temperature": 0,
"topP": 0
},
"name": "string",
"playgroundId": "string",
"promptType": "CHAT_HISTORY_AWARE",
"snapshotDate": "2019-08-24T14:15:22Z",
"vectorDatabaseId": "string",
"vectorDatabaseSettings": {
"addNeighborChunks": false,
"maxDocumentsRetrievedPerPrompt": 1,
"maxTokens": 1,
"retriever": "SINGLE_LOOKUP_RETRIEVER"
}
},
"llmTestConfigurationId": "string",
"llmTestConfigurationName": "string",
"llmTestGradingCriteria": {
"passThreshold": 100
},
"llmTestSuiteId": "string",
"passPercentage": 0,
"useCaseId": "string"
}
Responses¶
| Status | Meaning | Description | Schema |
|---|---|---|---|
| 202 | Accepted | Successful Response | LLMTestResultResponse |
| 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse |
To perform this operation, you must be authenticated by means of one of the following methods:
BearerAuth
DELETE /api/v2/genai/llmTestResults/{llmTestResultId}/¶
Delete an existing LLM test result.
Code samples¶
curl -X DELETE https://app.datarobot.com/api/v2/genai/llmTestResults/{llmTestResultId}/ \
-H "Accept: application/json" \
-H "Authorization: Bearer {access-token}"
Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
| llmTestResultId | path | string | true | The ID of the LLM Test Result to delete. |
Example responses¶
422 Response
{
"detail": [
{
"loc": [
"string"
],
"msg": "string",
"type": "string"
}
]
}
Responses¶
| Status | Meaning | Description | Schema |
|---|---|---|---|
| 204 | No Content | Successful Response | None |
| 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse |
To perform this operation, you must be authenticated by means of one of the following methods:
BearerAuth
GET /api/v2/genai/llmTestResults/{llmTestResultId}/¶
Retrieve an existing LLM test result.
Code samples¶
curl -X GET https://app.datarobot.com/api/v2/genai/llmTestResults/{llmTestResultId}/ \
-H "Accept: application/json" \
-H "Authorization: Bearer {access-token}"
Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
| llmTestResultId | path | string | true | The ID of the LLM Test Result to retrieve. |
Example responses¶
200 Response
{
"creationDate": "2019-08-24T14:15:22Z",
"creationUserId": "string",
"creationUserName": "string",
"errorMessage": "string",
"errorResolution": "string",
"executionStatus": "NEW",
"gradingResult": "PASS",
"id": "string",
"insightEvaluationResults": [
{
"aggregationType": "average",
"aggregationValue": 0,
"chatId": "string",
"chatName": "string",
"evaluationDatasetConfigurationId": "string",
"evaluationDatasetName": "string",
"evaluationName": "string",
"executionStatus": "NEW",
"gradingResult": "PASS",
"id": "string",
"insightGradingCriteria": {
"passThreshold": 100
},
"lastUpdateDate": "2019-08-24T14:15:22Z",
"llmTestResultId": "string",
"maxNumPrompts": 0,
"metricName": "string",
"promptSamplingStrategy": "random_without_replacement"
}
],
"isOutOfTheBoxTestConfiguration": true,
"llmBlueprintId": "string",
"llmBlueprintSnapshot": {
"description": "string",
"id": "string",
"llmId": "azure-openai-gpt-3.5-turbo",
"llmSettings": {
"maxCompletionLength": 0,
"systemPrompt": "string",
"temperature": 0,
"topP": 0
},
"name": "string",
"playgroundId": "string",
"promptType": "CHAT_HISTORY_AWARE",
"snapshotDate": "2019-08-24T14:15:22Z",
"vectorDatabaseId": "string",
"vectorDatabaseSettings": {
"addNeighborChunks": false,
"maxDocumentsRetrievedPerPrompt": 1,
"maxTokens": 1,
"retriever": "SINGLE_LOOKUP_RETRIEVER"
}
},
"llmTestConfigurationId": "string",
"llmTestConfigurationName": "string",
"llmTestGradingCriteria": {
"passThreshold": 100
},
"llmTestSuiteId": "string",
"passPercentage": 0,
"useCaseId": "string"
}
Responses¶
| Status | Meaning | Description | Schema |
|---|---|---|---|
| 200 | OK | Successful Response | LLMTestResultResponse |
| 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse |
To perform this operation, you must be authenticated by means of one of the following methods:
BearerAuth
GET /api/v2/genai/llmTestSuites/¶
List LLM test suites.
Code samples¶
curl -X GET https://app.datarobot.com/api/v2/genai/llmTestSuites/ \
-H "Accept: application/json" \
-H "Authorization: Bearer {access-token}"
Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
| useCaseId | query | any | false | Only retrieve the LLM test suites associated with this use case ID. |
| offset | query | integer | false | Skip the specified number of values. |
| limit | query | integer | false | Retrieve only the specified number of values. |
| sort | query | any | false | Apply this sort order to the results. Valid options are "name" and "creationDate". Prefix the attribute name with a dash to sort in descending order, e.g., sort=-creationDate. |
Example responses¶
200 Response
{
"count": 0,
"data": [
{
"creationDate": "2019-08-24T14:15:22Z",
"creationUserId": "string",
"description": "string",
"id": "string",
"llmTestConfigurationIds": [
"string"
],
"name": "string",
"useCaseId": "string"
}
],
"next": "string",
"previous": "string",
"totalCount": 0
}
Responses¶
| Status | Meaning | Description | Schema |
|---|---|---|---|
| 200 | OK | Successful Response | ListLLMTestSuitesResponse |
| 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse |
To perform this operation, you must be authenticated by means of one of the following methods:
BearerAuth
POST /api/v2/genai/llmTestSuites/¶
Create a new LLM test suite.
Code samples¶
curl -X POST https://app.datarobot.com/api/v2/genai/llmTestSuites/ \
-H "Content-Type: application/json" \
-H "Accept: application/json" \
-H "Authorization: Bearer {access-token}" \
-d '{CreateLLMTestSuiteRequest}'
Body parameter¶
{
"description": "",
"llmTestConfigurationIds": [],
"name": "string",
"useCaseId": "string"
}
Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
| body | body | CreateLLMTestSuiteRequest | true | none |
Example responses¶
201 Response
{
"creationDate": "2019-08-24T14:15:22Z",
"creationUserId": "string",
"description": "string",
"id": "string",
"llmTestConfigurationIds": [
"string"
],
"name": "string",
"useCaseId": "string"
}
Responses¶
| Status | Meaning | Description | Schema |
|---|---|---|---|
| 201 | Created | Successful Response | LLMTestSuiteResponse |
| 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse |
To perform this operation, you must be authenticated by means of one of the following methods:
BearerAuth
DELETE /api/v2/genai/llmTestSuites/{llmTestSuiteId}/¶
Delete an existing LLM test suite.
Code samples¶
curl -X DELETE https://app.datarobot.com/api/v2/genai/llmTestSuites/{llmTestSuiteId}/ \
-H "Accept: application/json" \
-H "Authorization: Bearer {access-token}"
Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
| llmTestSuiteId | path | string | true | The ID of the LLM test suite to delete. |
Example responses¶
422 Response
{
"detail": [
{
"loc": [
"string"
],
"msg": "string",
"type": "string"
}
]
}
Responses¶
| Status | Meaning | Description | Schema |
|---|---|---|---|
| 204 | No Content | Successful Response | None |
| 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse |
To perform this operation, you must be authenticated by means of one of the following methods:
BearerAuth
GET /api/v2/genai/llmTestSuites/{llmTestSuiteId}/¶
Retrieve an existing LLM test suite.
Code samples¶
curl -X GET https://app.datarobot.com/api/v2/genai/llmTestSuites/{llmTestSuiteId}/ \
-H "Accept: application/json" \
-H "Authorization: Bearer {access-token}"
Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
| llmTestSuiteId | path | string | true | The ID of the LLM test suite to retrieve. |
Example responses¶
200 Response
{
"creationDate": "2019-08-24T14:15:22Z",
"creationUserId": "string",
"description": "string",
"id": "string",
"llmTestConfigurationIds": [
"string"
],
"name": "string",
"useCaseId": "string"
}
Responses¶
| Status | Meaning | Description | Schema |
|---|---|---|---|
| 200 | OK | Successful Response | LLMTestSuiteResponse |
| 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse |
To perform this operation, you must be authenticated by means of one of the following methods:
BearerAuth
PATCH /api/v2/genai/llmTestSuites/{llmTestSuiteId}/¶
Edit an existing LLM test suite.
Code samples¶
curl -X PATCH https://app.datarobot.com/api/v2/genai/llmTestSuites/{llmTestSuiteId}/ \
-H "Content-Type: application/json" \
-H "Accept: application/json" \
-H "Authorization: Bearer {access-token}" \
-d '{undefined}'
Body parameter¶
{
"description": "string",
"llmTestConfigurationIds": [
"string"
],
"name": "string"
}
Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
| llmTestSuiteId | path | string | true | The ID of the LLM test suite to edit. |
| body | body | EditLLMTestSuiteRequest | true | none |
Example responses¶
200 Response
{
"creationDate": "2019-08-24T14:15:22Z",
"creationUserId": "string",
"description": "string",
"id": "string",
"llmTestConfigurationIds": [
"string"
],
"name": "string",
"useCaseId": "string"
}
Responses¶
| Status | Meaning | Description | Schema |
|---|---|---|---|
| 200 | OK | Successful Response | LLMTestSuiteResponse |
| 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse |
To perform this operation, you must be authenticated by means of one of the following methods:
BearerAuth
DELETE /api/v2/genai/ootbMetricConfigurations/{ootbMetricConfigurationId}/¶
Delete single OOTB metric configuration.
Code samples¶
curl -X DELETE https://app.datarobot.com/api/v2/genai/ootbMetricConfigurations/{ootbMetricConfigurationId}/ \
-H "Accept: application/json" \
-H "Authorization: Bearer {access-token}"
Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
| ootbMetricConfigurationId | path | string | true | The ID of the metric configuration. |
Example responses¶
422 Response
{
"detail": [
{
"loc": [
"string"
],
"msg": "string",
"type": "string"
}
]
}
Responses¶
| Status | Meaning | Description | Schema |
|---|---|---|---|
| 204 | No Content | OOTB metric configuration successfully deleted. | None |
| 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse |
To perform this operation, you must be authenticated by means of one of the following methods:
BearerAuth
GET /api/v2/genai/ootbMetricConfigurations/{ootbMetricConfigurationId}/¶
Get OOTB metric configuration from the configuration.
Code samples¶
curl -X GET https://app.datarobot.com/api/v2/genai/ootbMetricConfigurations/{ootbMetricConfigurationId}/ \
-H "Accept: application/json" \
-H "Authorization: Bearer {access-token}"
Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
| ootbMetricConfigurationId | path | string | true | The ID of the metric configuration. |
Example responses¶
200 Response
{
"customModelLLMValidationId": "string",
"customOotbMetricName": "string",
"errorMessage": "string",
"errorResolution": [
"ootbMetricName"
],
"executionStatus": "NEW",
"llmId": "azure-openai-gpt-3.5-turbo",
"moderationConfiguration": {
"guardConditions": [
{
"comparand": 0,
"comparator": "greaterThan"
}
],
"intervention": {
"action": "block",
"message": "string"
}
},
"ootbMetricConfigurationId": "string",
"ootbMetricName": "latency"
}
Responses¶
| Status | Meaning | Description | Schema |
|---|---|---|---|
| 200 | OK | OOTB metric configuration | OOTBMetricConfigurationResponse |
| 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse |
To perform this operation, you must be authenticated by means of one of the following methods:
BearerAuth
GET /api/v2/genai/sidecarModelMetricValidations/¶
List sidecar model metric validations.
Code samples¶
curl -X GET https://app.datarobot.com/api/v2/genai/sidecarModelMetricValidations/ \
-H "Accept: application/json" \
-H "Authorization: Bearer {access-token}"
Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
| useCaseId | query | any | false | Only retrieve the sidecar model metric validations associated with these use case IDs. |
| offset | query | integer | false | Skip the specified number of values. |
| limit | query | integer | false | Retrieve only the specified number of values. |
| search | query | any | false | Only retrieve the sidecar model metric validations matching the search query. |
| sort | query | any | false | Apply this sort order to the results. Valid options are "name", "deploymentName", "userName", "creationDate". Prefix the attribute name with a dash to sort in descending order, e.g., sort=-creationDate. |
| completedOnly | query | boolean | false | If true, only retrieve the completed sidecar model metric validations. The default is false. |
| deploymentId | query | any | false | Only retrieve the sidecar model metric validations associated with this deployment ID. |
| modelId | query | any | false | Only retrieve the sidecar model metric validations associated with this model ID. |
| promptColumnName | query | any | false | Only retrieve the sidecar model metric validations where the custom model uses this column name for prompt input. |
| targetColumnName | query | any | false | Only retrieve the sidecar model metric validations where the custom model uses this column name for prediction output. |
| citationsPrefixColumnName | query | any | false | Only retrieve the sidecar model metric validations where the custom model uses this column name prefix for citation inputs. |
Example responses¶
200 Response
{
"count": 0,
"data": [
{
"citationsPrefixColumnName": "string",
"creationDate": "2019-08-24T14:15:22Z",
"deploymentAccessData": {
"authorizationHeader": "[REDACTED]",
"chatApiUrl": "string",
"datarobotKey": "string",
"inputType": "CSV",
"modelType": "TEXT_GENERATION",
"predictionApiUrl": "string"
},
"deploymentId": "string",
"deploymentName": "string",
"errorMessage": "string",
"errorResolution": [
"ootbMetricName"
],
"expectedResponseColumnName": "string",
"id": "string",
"modelId": "string",
"moderationConfiguration": {
"guardConditions": [
{
"comparand": 0,
"comparator": "greaterThan"
}
],
"intervention": {
"action": "block",
"message": "string"
}
},
"name": "string",
"playgroundId": "string",
"predictionTimeout": 0,
"promptColumnName": "string",
"responseColumnName": "string",
"targetColumnName": "string",
"tenantId": "string",
"useCaseId": "string",
"userId": "string",
"userName": "string",
"validationStatus": "TESTING"
}
],
"next": "string",
"previous": "string",
"totalCount": 0
}
Responses¶
| Status | Meaning | Description | Schema |
|---|---|---|---|
| 200 | OK | Sidecar model metric validations successfully retrieved. | ListSidecarModelMetricValidationnResponse |
| 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse |
To perform this operation, you must be authenticated by means of one of the following methods:
BearerAuth
POST /api/v2/genai/sidecarModelMetricValidations/¶
Validate a metric hosted in a custom model deployment (also known as a sidecar model metric) for use in the playground.
Code samples¶
curl -X POST https://app.datarobot.com/api/v2/genai/sidecarModelMetricValidations/ \
-H "Content-Type: application/json" \
-H "Accept: application/json" \
-H "Authorization: Bearer {access-token}" \
-d '{CreateSidecarModelMetricValidationRequest}'
Body parameter¶
{
"citationsPrefixColumnName": "string",
"deploymentId": "string",
"expectedResponseColumnName": "string",
"modelId": "string",
"moderationConfiguration": {
"guardConditions": [
{
"comparand": 0,
"comparator": "greaterThan"
}
],
"intervention": {
"action": "block",
"message": "string"
}
},
"name": "Untitled",
"playgroundId": "string",
"predictionTimeout": 300,
"promptColumnName": "string",
"responseColumnName": "string",
"targetColumnName": "string",
"useCaseId": "string"
}
Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
| body | body | CreateSidecarModelMetricValidationRequest | true | none |
Example responses¶
202 Response
{
"citationsPrefixColumnName": "string",
"creationDate": "2019-08-24T14:15:22Z",
"deploymentAccessData": {
"authorizationHeader": "[REDACTED]",
"chatApiUrl": "string",
"datarobotKey": "string",
"inputType": "CSV",
"modelType": "TEXT_GENERATION",
"predictionApiUrl": "string"
},
"deploymentId": "string",
"deploymentName": "string",
"errorMessage": "string",
"errorResolution": [
"ootbMetricName"
],
"expectedResponseColumnName": "string",
"id": "string",
"modelId": "string",
"moderationConfiguration": {
"guardConditions": [
{
"comparand": 0,
"comparator": "greaterThan"
}
],
"intervention": {
"action": "block",
"message": "string"
}
},
"name": "string",
"playgroundId": "string",
"predictionTimeout": 0,
"promptColumnName": "string",
"responseColumnName": "string",
"targetColumnName": "string",
"tenantId": "string",
"useCaseId": "string",
"userId": "string",
"userName": "string",
"validationStatus": "TESTING"
}
Responses¶
| Status | Meaning | Description | Schema |
|---|---|---|---|
| 202 | Accepted | Sidecar model metric validation job successfully accepted. Follow the Location header to poll for job execution status. |
SidecarModelMetricValidationResponse |
| 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse |
To perform this operation, you must be authenticated by means of one of the following methods:
BearerAuth
DELETE /api/v2/genai/sidecarModelMetricValidations/{validationId}/¶
Delete an existing sidecar model metric validation.
Code samples¶
curl -X DELETE https://app.datarobot.com/api/v2/genai/sidecarModelMetricValidations/{validationId}/ \
-H "Accept: application/json" \
-H "Authorization: Bearer {access-token}"
Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
| validationId | path | string | true | The ID of the sidecar model metric validation to delete. |
Example responses¶
422 Response
{
"detail": [
{
"loc": [
"string"
],
"msg": "string",
"type": "string"
}
]
}
Responses¶
| Status | Meaning | Description | Schema |
|---|---|---|---|
| 204 | No Content | Sidecar model metric validation successfully deleted. | None |
| 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse |
To perform this operation, you must be authenticated by means of one of the following methods:
BearerAuth
GET /api/v2/genai/sidecarModelMetricValidations/{validationId}/¶
Retrieve the status of validating a sidecar model metric.
Code samples¶
curl -X GET https://app.datarobot.com/api/v2/genai/sidecarModelMetricValidations/{validationId}/ \
-H "Accept: application/json" \
-H "Authorization: Bearer {access-token}"
Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
| validationId | path | string | true | The ID of the sidecar model metric validation to retrieve. |
Example responses¶
200 Response
{
"citationsPrefixColumnName": "string",
"creationDate": "2019-08-24T14:15:22Z",
"deploymentAccessData": {
"authorizationHeader": "[REDACTED]",
"chatApiUrl": "string",
"datarobotKey": "string",
"inputType": "CSV",
"modelType": "TEXT_GENERATION",
"predictionApiUrl": "string"
},
"deploymentId": "string",
"deploymentName": "string",
"errorMessage": "string",
"errorResolution": [
"ootbMetricName"
],
"expectedResponseColumnName": "string",
"id": "string",
"modelId": "string",
"moderationConfiguration": {
"guardConditions": [
{
"comparand": 0,
"comparator": "greaterThan"
}
],
"intervention": {
"action": "block",
"message": "string"
}
},
"name": "string",
"playgroundId": "string",
"predictionTimeout": 0,
"promptColumnName": "string",
"responseColumnName": "string",
"targetColumnName": "string",
"tenantId": "string",
"useCaseId": "string",
"userId": "string",
"userName": "string",
"validationStatus": "TESTING"
}
Responses¶
| Status | Meaning | Description | Schema |
|---|---|---|---|
| 200 | OK | Sidecar model metric validation status successfully retrieved. | SidecarModelMetricValidationResponse |
| 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse |
To perform this operation, you must be authenticated by means of one of the following methods:
BearerAuth
PATCH /api/v2/genai/sidecarModelMetricValidations/{validationId}/¶
Edit an existing sidecar model metric validation.
Code samples¶
curl -X PATCH https://app.datarobot.com/api/v2/genai/sidecarModelMetricValidations/{validationId}/ \
-H "Content-Type: application/json" \
-H "Accept: application/json" \
-H "Authorization: Bearer {access-token}" \
-d '{undefined}'
Body parameter¶
{
"chatModelId": "string",
"citationsPrefixColumnName": "string",
"deploymentId": "string",
"expectedResponseColumnName": "string",
"modelId": "string",
"moderationConfiguration": {
"guardConditions": [
{
"comparand": 0,
"comparator": "greaterThan"
}
],
"intervention": {
"action": "block",
"message": "string"
}
},
"name": "string",
"predictionTimeout": 1,
"promptColumnName": "string",
"responseColumnName": "string",
"targetColumnName": "string"
}
Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
| validationId | path | string | true | The ID of the sidecar model metric validation to edit. |
| body | body | EditSidecarModelMetricValidationRequest | true | none |
Example responses¶
200 Response
{
"citationsPrefixColumnName": "string",
"creationDate": "2019-08-24T14:15:22Z",
"deploymentAccessData": {
"authorizationHeader": "[REDACTED]",
"chatApiUrl": "string",
"datarobotKey": "string",
"inputType": "CSV",
"modelType": "TEXT_GENERATION",
"predictionApiUrl": "string"
},
"deploymentId": "string",
"deploymentName": "string",
"errorMessage": "string",
"errorResolution": [
"ootbMetricName"
],
"expectedResponseColumnName": "string",
"id": "string",
"modelId": "string",
"moderationConfiguration": {
"guardConditions": [
{
"comparand": 0,
"comparator": "greaterThan"
}
],
"intervention": {
"action": "block",
"message": "string"
}
},
"name": "string",
"playgroundId": "string",
"predictionTimeout": 0,
"promptColumnName": "string",
"responseColumnName": "string",
"targetColumnName": "string",
"tenantId": "string",
"useCaseId": "string",
"userId": "string",
"userName": "string",
"validationStatus": "TESTING"
}
Responses¶
| Status | Meaning | Description | Schema |
|---|---|---|---|
| 200 | OK | Sidecar model metric validation successfully updated. | SidecarModelMetricValidationResponse |
| 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse |
To perform this operation, you must be authenticated by means of one of the following methods:
BearerAuth
POST /api/v2/genai/sidecarModelMetricValidations/{validationId}/revalidate/¶
Revalidate an existing sidecar model metric validation.
Code samples¶
curl -X POST https://app.datarobot.com/api/v2/genai/sidecarModelMetricValidations/{validationId}/revalidate/ \
-H "Accept: application/json" \
-H "Authorization: Bearer {access-token}"
Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
| validationId | path | string | true | The ID of the sidecar model metric validation to revalidate. |
Example responses¶
200 Response
{
"citationsPrefixColumnName": "string",
"creationDate": "2019-08-24T14:15:22Z",
"deploymentAccessData": {
"authorizationHeader": "[REDACTED]",
"chatApiUrl": "string",
"datarobotKey": "string",
"inputType": "CSV",
"modelType": "TEXT_GENERATION",
"predictionApiUrl": "string"
},
"deploymentId": "string",
"deploymentName": "string",
"errorMessage": "string",
"errorResolution": [
"ootbMetricName"
],
"expectedResponseColumnName": "string",
"id": "string",
"modelId": "string",
"moderationConfiguration": {
"guardConditions": [
{
"comparand": 0,
"comparator": "greaterThan"
}
],
"intervention": {
"action": "block",
"message": "string"
}
},
"name": "string",
"playgroundId": "string",
"predictionTimeout": 0,
"promptColumnName": "string",
"responseColumnName": "string",
"targetColumnName": "string",
"tenantId": "string",
"useCaseId": "string",
"userId": "string",
"userName": "string",
"validationStatus": "TESTING"
}
Responses¶
| Status | Meaning | Description | Schema |
|---|---|---|---|
| 200 | OK | Sidecar model metric successfully revalidated. | SidecarModelMetricValidationResponse |
| 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse |
To perform this operation, you must be authenticated by means of one of the following methods:
BearerAuth
POST /api/v2/genai/syntheticEvaluationDatasetGenerations/¶
Generate a synthetic evaluation dataset.
Code samples¶
curl -X POST https://app.datarobot.com/api/v2/genai/syntheticEvaluationDatasetGenerations/ \
-H "Content-Type: application/json" \
-H "Accept: application/json" \
-H "Authorization: Bearer {access-token}" \
-d '{SyntheticEvaluationDatasetGenerationRequest}'
Body parameter¶
{
"datasetName": "string",
"language": "string",
"llmId": "azure-openai-gpt-3.5-turbo",
"llmSettings": {
"maxCompletionLength": 0,
"systemPrompt": "string",
"temperature": 0,
"topP": 0
},
"vectorDatabaseId": "string"
}
Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
| body | body | SyntheticEvaluationDatasetGenerationRequest | true | none |
Example responses¶
202 Response
{
"datasetId": "string",
"promptColumnName": "string",
"responseColumnName": "string"
}
Responses¶
| Status | Meaning | Description | Schema |
|---|---|---|---|
| 202 | Accepted | Synthetic evaluation data generation job successfully accepted. Follow the Location header to poll for job execution status. |
SyntheticEvaluationDatasetGenerationResponse |
| 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse |
To perform this operation, you must be authenticated by means of one of the following methods:
BearerAuth
Schemas¶
AggregatedAggregationValue
{
"count": 0,
"item": "string",
"value": 0
}
AggregatedAggregationValue
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| count | integer | true | The number of metric aggregation items aggregated. | |
| item | string | true | The name of the item. | |
| value | number | true | The value associated with the item. |
AggregationType
"average"
AggregationType
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| AggregationType | string | false | The type of the metric aggregation. |
Enumerated Values¶
| Property | Value |
|---|---|
| AggregationType | [average, percentYes, classPercentCoverage, ngramImportance, guardConditionPercentYes] |
AggregationValue
{
"item": "string",
"value": 0
}
AggregationValue
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| item | string | true | The name of the item. | |
| value | number | true | The value associated with the item. |
CommonLLMSettings
{
"maxCompletionLength": 0,
"systemPrompt": "string",
"temperature": 0,
"topP": 0
}
CommonLLMSettings
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| maxCompletionLength | any | false | Maximum number of tokens allowed in the completion. The combined count of this value and prompt tokens must be below the model's maximum context size, where prompt token count is comprised of system prompt, user prompt, recent chat history, and vector database citations. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | integer | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| systemPrompt | any | false | System prompt guides the style of the LLM response. It is a "universal" prompt, prepended to all individual prompts. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | maxLength: 500000 |
none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| temperature | any | false | Temperature controls the randomness of model output, where higher values return more diverse output and lower values return more deterministic results. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | number | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| topP | any | false | Top P sets a threshold that controls the selection of words included in the response, based on a cumulative probability cutoff for token selection. For example, 0.2 considers only the top 20% probability mass. Higher numbers return more diverse options for outputs. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | number | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
CostMetricConfigurationResponse
{
"costConfigurationId": "string",
"costMetricConfigurations": [
{
"currencyCode": "USD",
"customModelLLMValidationId": "string",
"inputTokenPrice": 0.01,
"llmId": "azure-openai-gpt-3.5-turbo",
"outputTokenPrice": 0.01,
"referenceInputTokenCount": 1000,
"referenceOutputTokenCount": 1000
}
],
"name": "string",
"playgroundId": "string",
"useCaseId": "string"
}
CostMetricConfigurationResponse
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| costConfigurationId | string | true | The ID of the cost metric configuration. | |
| costMetricConfigurations | [LLMCostConfigurationResponse] | true | The list of individual LLM cost configurations that constitute this cost metric configuration. | |
| name | any | false | The name to use for the cost configuration. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | maxLength: 5000 |
none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| playgroundId | any | false | The ID of the playground associated with the cost metric configuration. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| useCaseId | string | true | The ID of the use case associated with the cost metric configuration. |
CreateCostMetricConfigurationRequest
{
"costMetricConfigurations": [
{
"currencyCode": "USD",
"customModelLLMValidationId": "string",
"inputTokenPrice": 0.01,
"llmId": "azure-openai-gpt-3.5-turbo",
"outputTokenPrice": 0.01,
"referenceInputTokenCount": 1000,
"referenceOutputTokenCount": 1000
}
],
"name": "string",
"playgroundId": "string",
"useCaseId": "string"
}
CreateCostMetricConfigurationRequest
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| costMetricConfigurations | [LLMCostConfigurationResponse] | true | The list of cost metric configurations to use. | |
| name | any | false | The name to use for the cost configuration. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | maxLength: 5000 |
none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| playgroundId | string | true | The ID of the playground to associate with the cost metric configuration. | |
| useCaseId | string | true | The ID of the use case to associate with the cost metric configuration. |
CreateEvaluationDatasetConfigurationRequest
{
"correctnessEnabled": true,
"datasetId": "string",
"isSyntheticDataset": false,
"name": "string",
"playgroundId": "string",
"promptColumnName": "string",
"responseColumnName": "string",
"useCaseId": "string"
}
CreateEvaluationDatasetConfigurationRequest
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| correctnessEnabled | any | false | Whether correctness is enabled for the evaluation dataset configuration. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | boolean | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| datasetId | string | true | The ID of the evaluation dataset. | |
| isSyntheticDataset | boolean | false | Whether the evaluation dataset is synthetic. | |
| name | any | false | The name of the evaluation dataset configuration. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | maxLength: 5000 |
none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| playgroundId | string | true | The ID of the playground to associate with the evaluation dataset configuration. | |
| promptColumnName | string | true | The name of the dataset column containing the prompt text. | |
| responseColumnName | any | false | The name of the dataset column containing the response text. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| useCaseId | string | true | The ID of the use case to associate with the evaluation dataset configuration. |
CreateEvaluationDatasetMetricAggregationRequest
{
"chatName": "Aggregated chat",
"evaluationDatasetConfigurationId": "string",
"insightsConfiguration": [
{
"aggregationTypes": [
"average"
],
"costConfigurationId": "string",
"customMetricId": "string",
"customModelLLMValidationId": "string",
"deploymentId": "string",
"errorMessage": "string",
"errorResolution": [
"string"
],
"evaluationDatasetConfigurationId": "string",
"executionStatus": "NEW",
"insightName": "string",
"insightType": "Reference",
"isTransferable": false,
"llmId": "azure-openai-gpt-3.5-turbo",
"llmIsActive": true,
"llmIsDeprecated": true,
"modelId": "string",
"modelPackageRegisteredModelId": "string",
"moderationConfiguration": {
"guardConditions": [
{
"comparand": 0,
"comparator": "greaterThan"
}
],
"intervention": {
"action": "block",
"message": "string"
}
},
"nemoMetricId": "string",
"ootbMetricId": "string",
"ootbMetricName": "latency",
"resultUnit": "s",
"sidecarModelMetricMetadata": {
"expectedResponseColumnName": "string",
"promptColumnName": "string",
"responseColumnName": "string",
"targetColumnName": "string"
},
"sidecarModelMetricValidationId": "string",
"stage": "prompt_pipeline"
}
],
"llmBlueprintIds": [
"string"
]
}
CreateEvaluationDatasetMetricAggregationRequest
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| chatName | string | false | maxLength: 5000 |
The name for the new chat that will contain the associated prompts and responses. |
| evaluationDatasetConfigurationId | string | true | The ID of the evaluation dataset configuration. | |
| insightsConfiguration | [InsightsConfigurationWithAdditionalData] | true | minItems: 1 |
The configuration of insights for the metric aggregation. |
| llmBlueprintIds | [string] | true | maxItems: 3 minItems: 1 |
The IDs of the LLM blueprints to use for the metric aggregation. |
CreateEvaluationDatasetMetricAggregationResponse
{
"chatIds": [
"string"
],
"jobId": "string"
}
CreateEvaluationDatasetMetricAggregationResponse
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| chatIds | [string] | true | The IDs of the chats associated with the metric aggregation. | |
| jobId | string(uuid4) | true | The ID of the evaluation dataset metric aggregation job. |
CreateLLMTestConfigurationRequest
{
"datasetEvaluations": [
{
"evaluationDatasetConfigurationId": "string",
"evaluationName": "string",
"insightConfiguration": {
"aggregationTypes": [
"average"
],
"costConfigurationId": "string",
"customMetricId": "string",
"customModelLLMValidationId": "string",
"deploymentId": "string",
"errorMessage": "string",
"errorResolution": [
"string"
],
"evaluationDatasetConfigurationId": "string",
"executionStatus": "NEW",
"insightName": "string",
"insightType": "Reference",
"isTransferable": false,
"llmId": "azure-openai-gpt-3.5-turbo",
"llmIsActive": true,
"llmIsDeprecated": true,
"modelId": "string",
"modelPackageRegisteredModelId": "string",
"moderationConfiguration": {
"guardConditions": [
{
"comparand": 0,
"comparator": "greaterThan"
}
],
"intervention": {
"action": "block",
"message": "string"
}
},
"nemoMetricId": "string",
"ootbMetricId": "string",
"ootbMetricName": "latency",
"resultUnit": "s",
"sidecarModelMetricMetadata": {
"expectedResponseColumnName": "string",
"promptColumnName": "string",
"responseColumnName": "string",
"targetColumnName": "string"
},
"sidecarModelMetricValidationId": "string",
"stage": "prompt_pipeline"
},
"insightGradingCriteria": {
"passThreshold": 100
},
"maxNumPrompts": 0,
"ootbDatasetName": "jailbreak-v1.csv",
"promptSamplingStrategy": "random_without_replacement"
}
],
"description": "",
"llmTestGradingCriteria": {
"passThreshold": 100
},
"name": "string",
"useCaseId": "string"
}
CreateLLMTestConfigurationRequest
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| datasetEvaluations | [DatasetEvaluationRequest] | true | maxItems: 10 minItems: 1 |
Dataset evaluations. |
| description | string | false | maxLength: 5000 |
LLM test configuration description. |
| llmTestGradingCriteria | LLMTestGradingCriteria | true | LLM test grading criteria. | |
| name | string | true | maxLength: 5000 minLength: 1 minLength: 1 |
LLM test configuration name. |
| useCaseId | string | true | The use case ID associated with the LLM Test configuration. |
CreateLLMTestResultRequest
{
"llmBlueprintId": "string",
"llmTestConfigurationId": "string"
}
CreateLLMTestResultRequest
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| llmBlueprintId | string | true | The LLM Blueprint ID associated with the LLM Test result. | |
| llmTestConfigurationId | string | true | The use case ID associated with the LLM Test result. |
CreateLLMTestSuiteRequest
{
"description": "",
"llmTestConfigurationIds": [],
"name": "string",
"useCaseId": "string"
}
CreateLLMTestSuiteRequest
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| description | string | false | maxLength: 5000 |
The description of the LLM test suite. |
| llmTestConfigurationIds | [string] | false | maxItems: 100 |
The IDs of the LLM test configurations in the LLM test suite. |
| name | string | true | maxLength: 5000 minLength: 1 minLength: 1 |
The name of the LLM test suite. |
| useCaseId | string | true | The ID of the use case to associate with the LLM test suite. |
CreateSidecarModelMetricValidationRequest
{
"citationsPrefixColumnName": "string",
"deploymentId": "string",
"expectedResponseColumnName": "string",
"modelId": "string",
"moderationConfiguration": {
"guardConditions": [
{
"comparand": 0,
"comparator": "greaterThan"
}
],
"intervention": {
"action": "block",
"message": "string"
}
},
"name": "Untitled",
"playgroundId": "string",
"predictionTimeout": 300,
"promptColumnName": "string",
"responseColumnName": "string",
"targetColumnName": "string",
"useCaseId": "string"
}
CreateSidecarModelMetricValidationRequest
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| citationsPrefixColumnName | any | false | The column name prefix the custom model uses for citation inputs. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | maxLength: 5000 |
none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| deploymentId | string | true | The ID of the custom model deployment. | |
| expectedResponseColumnName | any | false | The name of the column the custom model uses for the expected response text input. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | maxLength: 5000 |
none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| modelId | any | false | The ID of the model used in the deployment. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| moderationConfiguration | any | false | The moderation configuration to be associated with the sidecar model metric. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | ModerationConfigurationWithoutID | false | Moderation Configuration associated with an insight. |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| name | string | false | maxLength: 5000 |
The name to use for the validated custom model. |
| playgroundId | string | true | The ID of the playground to associate with the validated custom model. | |
| predictionTimeout | integer | false | maximum: 600 minimum: 1 |
The timeout in seconds for the prediction when validating a custom model. Defaults to 300. |
| promptColumnName | any | false | The name of the column the custom model uses for prompt text input. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | maxLength: 5000 |
none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| responseColumnName | any | false | The name of the column the custom model uses for response text input. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | maxLength: 5000 |
none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| targetColumnName | string | true | maxLength: 5000 |
The name of the column the custom model uses for prediction output. |
| useCaseId | string | true | The ID of the use case to associate with the validated custom model. |
CustomModelLLMSettings
{
"externalLlmContextSize": 128,
"systemPrompt": "string",
"validationId": "string"
}
CustomModelLLMSettings
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| externalLlmContextSize | any | false | The external LLM's context size, in tokens. This value is only used for pruning documents supplied to the LLM when a vector database is associated with the LLM blueprint. It does not affect the external LLM's actual context size in any way and is not supplied to the LLM. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | integer | false | maximum: 128000 minimum: 128 |
none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| systemPrompt | any | false | System prompt guides the style of the LLM response. It is a "universal" prompt, prepended to all individual prompts. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | maxLength: 500000 |
none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| validationId | any | false | The validation ID of the custom model LLM. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
CustomModelValidationStatus
"TESTING"
CustomModelValidationStatus
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| CustomModelValidationStatus | string | false | Status of custom model validation. |
Enumerated Values¶
| Property | Value |
|---|---|
| CustomModelValidationStatus | [TESTING, PASSED, FAILED] |
DatasetEvaluationRequest
{
"evaluationDatasetConfigurationId": "string",
"evaluationName": "string",
"insightConfiguration": {
"aggregationTypes": [
"average"
],
"costConfigurationId": "string",
"customMetricId": "string",
"customModelLLMValidationId": "string",
"deploymentId": "string",
"errorMessage": "string",
"errorResolution": [
"string"
],
"evaluationDatasetConfigurationId": "string",
"executionStatus": "NEW",
"insightName": "string",
"insightType": "Reference",
"isTransferable": false,
"llmId": "azure-openai-gpt-3.5-turbo",
"llmIsActive": true,
"llmIsDeprecated": true,
"modelId": "string",
"modelPackageRegisteredModelId": "string",
"moderationConfiguration": {
"guardConditions": [
{
"comparand": 0,
"comparator": "greaterThan"
}
],
"intervention": {
"action": "block",
"message": "string"
}
},
"nemoMetricId": "string",
"ootbMetricId": "string",
"ootbMetricName": "latency",
"resultUnit": "s",
"sidecarModelMetricMetadata": {
"expectedResponseColumnName": "string",
"promptColumnName": "string",
"responseColumnName": "string",
"targetColumnName": "string"
},
"sidecarModelMetricValidationId": "string",
"stage": "prompt_pipeline"
},
"insightGradingCriteria": {
"passThreshold": 100
},
"maxNumPrompts": 0,
"ootbDatasetName": "jailbreak-v1.csv",
"promptSamplingStrategy": "random_without_replacement"
}
DatasetEvaluationRequest
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| evaluationDatasetConfigurationId | any | false | The ID of the evaluation dataset configuration for this dataset evaluation. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| evaluationName | string | true | maxLength: 5000 minLength: 1 minLength: 1 |
The name of the evaluation. This name should provide context regarding what is being evaluated. |
| insightConfiguration | InsightsConfigurationWithAdditionalData | true | The configuration of insights with extra data. | |
| insightGradingCriteria | InsightGradingCriteria | true | Grading criteria for an insight. | |
| maxNumPrompts | integer | false | maximum: 5000 minimum: 0 |
The max number of prompts to evaluate. |
| ootbDatasetName | any | false | Out-of-the-box evaluation dataset name. This applies only to our predefined public evaluation datasets. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | OOTBDatasetName | false | Out-of-the-box dataset name. |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| promptSamplingStrategy | PromptSamplingStrategy | false | The prompt sampling strategy. Controls how max_num_prompts are sampled. |
DatasetEvaluationResponse
{
"errorMessage": "string",
"evaluationDatasetConfigurationId": "string",
"evaluationDatasetName": "string",
"evaluationName": "string",
"insightConfiguration": {
"aggregationTypes": [
"average"
],
"costConfigurationId": "string",
"customMetricId": "string",
"customModelLLMValidationId": "string",
"deploymentId": "string",
"errorMessage": "string",
"errorResolution": [
"string"
],
"evaluationDatasetConfigurationId": "string",
"executionStatus": "NEW",
"insightName": "string",
"insightType": "Reference",
"isTransferable": false,
"llmId": "azure-openai-gpt-3.5-turbo",
"llmIsActive": true,
"llmIsDeprecated": true,
"modelId": "string",
"modelPackageRegisteredModelId": "string",
"moderationConfiguration": {
"guardConditions": [
{
"comparand": 0,
"comparator": "greaterThan"
}
],
"intervention": {
"action": "block",
"message": "string"
}
},
"nemoMetricId": "string",
"ootbMetricId": "string",
"ootbMetricName": "latency",
"resultUnit": "s",
"sidecarModelMetricMetadata": {
"expectedResponseColumnName": "string",
"promptColumnName": "string",
"responseColumnName": "string",
"targetColumnName": "string"
},
"sidecarModelMetricValidationId": "string",
"stage": "prompt_pipeline"
},
"insightGradingCriteria": {
"passThreshold": 100
},
"maxNumPrompts": 100,
"ootbDataset": {
"datasetName": "jailbreak-v1.csv",
"datasetUrl": "https://s3.amazonaws.com/datarobot_public_datasets/genai/jailbreak-v1.csv",
"promptColumnName": "string",
"responseColumnName": "string",
"rowsCount": 0,
"warning": "string"
},
"promptSamplingStrategy": "random_without_replacement"
}
DatasetEvaluationResponse
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| errorMessage | any | false | The error message associated with the dataset evaluation. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| evaluationDatasetConfigurationId | any | false | The ID of the evaluation dataset configuration for this dataset evaluation. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| evaluationDatasetName | any | true | Evaluation dataset name. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | maxLength: 5000 |
none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| evaluationName | string | true | maxLength: 5000 minLength: 1 minLength: 1 |
The name of the evaluation. This name should provide context regarding what is being evaluated. |
| insightConfiguration | InsightsConfigurationWithAdditionalData | true | The configuration of insights with extra data. | |
| insightGradingCriteria | InsightGradingCriteria | true | Grading criteria for an insight. | |
| maxNumPrompts | integer | false | maximum: 5000 |
The max number of prompts to evaluate. |
| ootbDataset | any | false | Out-of-the-box evaluation dataset. This applies only to our predefined public evaluation datasets. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | OOTBDataset | false | Out-of-the-box dataset. |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| promptSamplingStrategy | PromptSamplingStrategy | false | The prompt sampling strategy. Controls how max_num_prompts are sampled. |
DatasetIdentifier
{
"datasetId": "string",
"datasetName": "string"
}
DatasetIdentifier
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| datasetId | any | true | The ID of the dataset, if any. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| datasetName | string | true | The name of the dataset. |
DeploymentAccessData
{
"authorizationHeader": "[REDACTED]",
"chatApiUrl": "string",
"datarobotKey": "string",
"inputType": "CSV",
"modelType": "TEXT_GENERATION",
"predictionApiUrl": "string"
}
DeploymentAccessData
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| authorizationHeader | string | false | The Authorization header to use for the deployment. |
|
| chatApiUrl | any | false | The URL of the deployment's chat API. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| datarobotKey | any | true | The server key associated with the prediction API. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| inputType | DeploymentInputType | true | The format of the input data. | |
| modelType | SupportedDeploymentType | true | The type of the target output the deployment produces. | |
| predictionApiUrl | string | true | The URL of the deployment's prediction API. |
DeploymentInputType
"CSV"
DeploymentInputType
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| DeploymentInputType | string | false | The format of the input data submitted to a DataRobot deployment. |
Enumerated Values¶
| Property | Value |
|---|---|
| DeploymentInputType | [CSV, JSON] |
EditCostMetricConfigurationRequest
{
"costMetricConfigurations": [
{
"currencyCode": "USD",
"customModelLLMValidationId": "string",
"inputTokenPrice": 0.01,
"llmId": "azure-openai-gpt-3.5-turbo",
"outputTokenPrice": 0.01,
"referenceInputTokenCount": 1000,
"referenceOutputTokenCount": 1000
}
],
"name": "string"
}
EditCostMetricConfigurationRequest
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| costMetricConfigurations | [LLMCostConfigurationResponse] | true | minItems: 1 |
The list of LLM cost configurations to apply to this cost metric configuration. |
| name | any | false | The name to use for the cost configuration. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | maxLength: 5000 |
none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
EditEvaluationDatasetConfigurationRequest
{
"correctnessEnabled": true,
"datasetId": "string",
"name": "string",
"promptColumnName": "string",
"responseColumnName": "string"
}
EditEvaluationDatasetConfigurationRequest
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| correctnessEnabled | any | false | If specified, enables or disables correctness for the evaluation dataset configuration. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | boolean | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| datasetId | any | false | If specified, updates the ID of the evaluation dataset. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| name | any | false | If specified, renames the evaluation dataset configuration to this value. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | maxLength: 5000 |
none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| promptColumnName | any | false | If specified, changes the expected name of the dataset column containing the prompt text. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| responseColumnName | any | false | If specified, changes the expected name of the dataset column containing the response text. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
EditLLMTestConfigurationRequest
{
"datasetEvaluations": [
{
"evaluationDatasetConfigurationId": "string",
"evaluationName": "string",
"insightConfiguration": {
"aggregationTypes": [
"average"
],
"costConfigurationId": "string",
"customMetricId": "string",
"customModelLLMValidationId": "string",
"deploymentId": "string",
"errorMessage": "string",
"errorResolution": [
"string"
],
"evaluationDatasetConfigurationId": "string",
"executionStatus": "NEW",
"insightName": "string",
"insightType": "Reference",
"isTransferable": false,
"llmId": "azure-openai-gpt-3.5-turbo",
"llmIsActive": true,
"llmIsDeprecated": true,
"modelId": "string",
"modelPackageRegisteredModelId": "string",
"moderationConfiguration": {
"guardConditions": [
{
"comparand": 0,
"comparator": "greaterThan"
}
],
"intervention": {
"action": "block",
"message": "string"
}
},
"nemoMetricId": "string",
"ootbMetricId": "string",
"ootbMetricName": "latency",
"resultUnit": "s",
"sidecarModelMetricMetadata": {
"expectedResponseColumnName": "string",
"promptColumnName": "string",
"responseColumnName": "string",
"targetColumnName": "string"
},
"sidecarModelMetricValidationId": "string",
"stage": "prompt_pipeline"
},
"insightGradingCriteria": {
"passThreshold": 100
},
"maxNumPrompts": 0,
"ootbDatasetName": "jailbreak-v1.csv",
"promptSamplingStrategy": "random_without_replacement"
}
],
"description": "string",
"llmTestGradingCriteria": {
"passThreshold": 100
},
"name": "string"
}
EditLLMTestConfigurationRequest
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| datasetEvaluations | any | false | New Dataset evaluations. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | [DatasetEvaluationRequest] | false | maxItems: 10 minItems: 1 |
[Dataset evaluation.] |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| description | any | false | New LLM test configuration description. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | maxLength: 5000 |
none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| llmTestGradingCriteria | any | false | New LLM test grading criteria. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | LLMTestGradingCriteria | false | Grading criteria for the LLM Test configuration. |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| name | any | false | New LLM test configuration name. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | maxLength: 5000 minLength: 1 minLength: 1 |
none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
EditLLMTestSuiteRequest
{
"description": "string",
"llmTestConfigurationIds": [
"string"
],
"name": "string"
}
EditLLMTestSuiteRequest
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| description | any | false | The description of the LLM test suite. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | maxLength: 5000 |
none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| llmTestConfigurationIds | any | false | The IDs of the LLM test configurations in the LLM test suite. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | [string] | false | maxItems: 100 |
none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| name | any | false | The name of the LLM test suite. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | maxLength: 5000 minLength: 1 minLength: 1 |
none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
EditSidecarModelMetricValidationRequest
{
"chatModelId": "string",
"citationsPrefixColumnName": "string",
"deploymentId": "string",
"expectedResponseColumnName": "string",
"modelId": "string",
"moderationConfiguration": {
"guardConditions": [
{
"comparand": 0,
"comparator": "greaterThan"
}
],
"intervention": {
"action": "block",
"message": "string"
}
},
"name": "string",
"predictionTimeout": 1,
"promptColumnName": "string",
"responseColumnName": "string",
"targetColumnName": "string"
}
EditSidecarModelMetricValidationRequest
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| chatModelId | any | false | The model ID to specify when calling the OpenAI chat completion API of the deployment. If this parameter is specified, the deployment must support the OpenAI chat completion API. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | maxLength: 5000 |
none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| citationsPrefixColumnName | any | false | If specified, changes the column name prefix that will be used to submit the citation inputs to the sidecar model. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | maxLength: 5000 |
none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| deploymentId | any | false | If specified, changes the ID of the deployment associated with this custom model validation. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| expectedResponseColumnName | any | false | If specified, changes the name of the column that will be used to submit the expected response text input to the sidecar model. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | maxLength: 5000 |
none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| modelId | any | false | If specified, changes the ID of the model associated with this custom model validation. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| moderationConfiguration | any | false | The moderation configuration to be associated with the sidecar model metric. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | ModerationConfigurationWithoutID | false | Moderation Configuration associated with an insight. |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| name | any | false | If specified, renames the custom model validation to this value. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | maxLength: 5000 minLength: 1 minLength: 1 |
none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| predictionTimeout | any | false | If specified, sets the timeout in seconds for the prediction when validating a custom model. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | integer | false | maximum: 600 minimum: 1 |
none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| promptColumnName | any | false | If specified, changes the name of the column that will be used to format the prompt text input for the custom model deployment. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | maxLength: 5000 |
none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| responseColumnName | any | false | If specified, changes the name of the column that will be used to submit the response text input to the sidecar model. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | maxLength: 5000 |
none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| targetColumnName | any | false | If specified, changes the name of the column that will be used to extract the prediction response from the custom model deployment. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | maxLength: 5000 |
none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
EvaluationDatasetConfigurationResponse
{
"correctnessEnabled": true,
"creationDate": "2019-08-24T14:15:22Z",
"creationUserId": "string",
"datasetId": "string",
"datasetName": "string",
"errorMessage": "string",
"executionStatus": "NEW",
"id": "string",
"name": "string",
"playgroundId": "string",
"promptColumnName": "string",
"responseColumnName": "string",
"rowsCount": 0,
"size": 0,
"tenantId": "string",
"useCaseId": "string",
"userName": "string"
}
EvaluationDatasetConfigurationResponse
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| correctnessEnabled | any | true | Whether correctness is enabled for the evaluation dataset configuration. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | boolean | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| creationDate | string(date-time) | true | The creation date of the evaluation dataset configuration (ISO 8601 formatted). | |
| creationUserId | string | true | The ID of the user that created the evaluation dataset configuration. | |
| datasetId | string | true | The ID of the evaluation dataset. | |
| datasetName | string | true | The name of the evaluation dataset. | |
| errorMessage | any | false | The error message associated with the evaluation dataset configuration. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| executionStatus | ExecutionStatus | true | The execution status of the evaluation dataset. | |
| id | string | true | The ID of the evaluation dataset configuration. | |
| name | string | true | The name of the evaluation dataset configuration. | |
| playgroundId | any | true | The ID of the playground associated with the evaluation dataset configuration. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| promptColumnName | string | true | The name of the dataset column containing the prompt text. | |
| responseColumnName | any | true | The name of the dataset column containing the response text. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| rowsCount | integer | true | The rows count of the evaluation dataset. | |
| size | integer | true | The size of the evaluation dataset (in bytes). | |
| tenantId | string(uuid4) | true | The ID of the DataRobot tenant this evaluation dataset configuration belongs to. | |
| useCaseId | string | true | The ID of the use case associated with the evaluation dataset configuration. | |
| userName | string | true | The name of the user that created the evaluation dataset configuration. |
EvaluationDatasetMetricAggregationAggregatedByLLMBlueprintResponse
{
"aggregatedItemCount": 0,
"aggregatedItemDetails": [
{
"chatId": "string",
"chatLink": "string",
"chatName": "string",
"creationDate": "2019-08-24T14:15:22Z",
"creationUserId": "string",
"creationUserName": "string"
}
],
"aggregationType": "average",
"aggregationValue": 0,
"datasetId": "string",
"datasetName": "string",
"evaluationDatasetConfigurationId": "string",
"llmBlueprintId": "string",
"metricName": "string",
"ootbDatasetName": "jailbreak-v1.csv",
"tenantId": "string"
}
EvaluationDatasetMetricAggregationAggregatedByLLMBlueprintResponse
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| aggregatedItemCount | integer | true | Number of items aggregated. | |
| aggregatedItemDetails | [EvaluationDatasetMetricAggregationChatDetails] | true | List of details for aggregated items. | |
| aggregationType | AggregationType | true | The type of metric aggregation. | |
| aggregationValue | any | true | The aggregated value of the metric. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | number | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | [AggregationValue] | false | [An individual record in an itemized metric aggregation.] |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | [AggregatedAggregationValue] | false | [Aggregated record of multiple of the same item across different metric aggregation runs.] |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| datasetId | any | true | The dataset ID of the evaluation dataset configuration. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| datasetName | any | true | The Data Registry dataset name of the evaluation dataset configuration. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| evaluationDatasetConfigurationId | any | true | The ID of the evaluation dataset configuration associated with the metric aggregation. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| llmBlueprintId | string | true | The ID of the LLM blueprint associated with the metric aggregation. | |
| metricName | string | true | The name of the metric associated with the metric aggregation. | |
| ootbDatasetName | any | true | The name of the out-of-the-box dataset. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | OOTBDatasetName | false | Out-of-the-box dataset name. |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| tenantId | string(uuid4) | true | The ID of the tenant the metric aggregation belongs to. |
EvaluationDatasetMetricAggregationChatDetails
{
"chatId": "string",
"chatLink": "string",
"chatName": "string",
"creationDate": "2019-08-24T14:15:22Z",
"creationUserId": "string",
"creationUserName": "string"
}
EvaluationDatasetMetricAggregationChatDetails
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| chatId | string | true | The ID of the chat associated with the metric aggregation. | |
| chatLink | string | true | The link to the chat associated with the metric aggregation. | |
| chatName | string | true | The name of the chat associated with the metric aggregation. | |
| creationDate | string(date-time) | true | The creation date of the metric aggregation (ISO 8601 formatted). | |
| creationUserId | string | true | The ID of the user that created the metric aggregation. | |
| creationUserName | string | true | The name of the user that created the metric aggregation. |
EvaluationDatasetMetricAggregationFieldQueryParam
"metricName"
EvaluationDatasetMetricAggregationFieldQueryParam
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| EvaluationDatasetMetricAggregationFieldQueryParam | string | false | Field used for aggregation when listing evaluation dataset metric aggregations. |
Enumerated Values¶
| Property | Value |
|---|---|
| EvaluationDatasetMetricAggregationFieldQueryParam | [metricName, llmBlueprintId, aggregationType, evaluationDatasetConfigurationId] |
EvaluationDatasetMetricAggregationResponse
{
"aggregationType": "average",
"aggregationValue": 0,
"chatId": "string",
"chatLink": "string",
"chatName": "string",
"creationDate": "2019-08-24T14:15:22Z",
"creationUserId": "string",
"creationUserName": "string",
"datasetId": "string",
"datasetName": "string",
"evaluationDatasetConfigurationId": "string",
"llmBlueprintId": "string",
"metricName": "string",
"ootbDatasetName": "jailbreak-v1.csv",
"tenantId": "string"
}
EvaluationDatasetMetricAggregationResponse
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| aggregationType | AggregationType | true | The type of metric aggregation. | |
| aggregationValue | any | true | The aggregated value of the metric. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | number | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | [AggregationValue] | false | [An individual record in an itemized metric aggregation.] |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | [AggregatedAggregationValue] | false | [Aggregated record of multiple of the same item across different metric aggregation runs.] |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| chatId | string | true | The ID of the chat associated with the metric aggregation. | |
| chatLink | string | true | The link to the chat associated with the metric aggregation. | |
| chatName | string | true | The name of the chat associated with the metric aggregation. | |
| creationDate | string(date-time) | true | The creation date of the metric aggregation (ISO 8601 formatted). | |
| creationUserId | string | true | The ID of the user that created the metric aggregation. | |
| creationUserName | string | true | The name of the user that created the metric aggregation. | |
| datasetId | any | true | The dataset ID of the evaluation dataset configuration. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| datasetName | any | true | The Data Registry dataset name of the evaluation dataset configuration. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| evaluationDatasetConfigurationId | any | true | The ID of the evaluation dataset configuration associated with the metric aggregation. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| llmBlueprintId | string | true | The ID of the LLM blueprint associated with the metric aggregation. | |
| metricName | string | true | The name of the metric associated with the metric aggregation. | |
| ootbDatasetName | any | true | The name of the out-of-the-box dataset. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | OOTBDatasetName | false | Out-of-the-box dataset name. |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| tenantId | string(uuid4) | true | The ID of the tenant the metric aggregation belongs to. |
EvaluationDatasetMetricAggregationUniqueFieldValuesResponse
{
"uniqueFieldValue": "string"
}
EvaluationDatasetMetricAggregationUniqueFieldValuesResponse
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| uniqueFieldValue | string | true | The unique value associated with the metric aggregation. |
ExecutionStatus
"NEW"
ExecutionStatus
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| ExecutionStatus | string | false | Job and entity execution status. |
Enumerated Values¶
| Property | Value |
|---|---|
| ExecutionStatus | [NEW, RUNNING, COMPLETED, REQUIRES_USER_INPUT, SKIPPED, ERROR] |
GradingResult
"PASS"
GradingResult
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| GradingResult | string | false | Grading result. |
Enumerated Values¶
| Property | Value |
|---|---|
| GradingResult | [PASS, FAIL] |
GuardCondition
{
"comparand": 0,
"comparator": "greaterThan"
}
GuardCondition
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| comparand | any | true | The comparand(s) used in the guard condition. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | number | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | boolean | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | [string] | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| comparator | GuardConditionComparator | true | The comparator used in the guard condition. |
GuardConditionComparator
"greaterThan"
GuardConditionComparator
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| GuardConditionComparator | string | false | The comparator used in a guard condition. |
Enumerated Values¶
| Property | Value |
|---|---|
| GuardConditionComparator | [greaterThan, lessThan, equals, notEquals, is, isNot, matches, doesNotMatch, contains, doesNotContain] |
HTTPValidationErrorResponse
{
"detail": [
{
"loc": [
"string"
],
"msg": "string",
"type": "string"
}
]
}
HTTPValidationErrorResponse
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| detail | [ValidationError] | false | none |
InsightErrorResolution
"ootbMetricName"
InsightErrorResolution
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| InsightErrorResolution | string | false | Error type linking directly to the field name that is related to the error. |
Enumerated Values¶
| Property | Value |
|---|---|
| InsightErrorResolution | [ootbMetricName, intervention, guardCondition, sidecarOverall, sidecarRevalidate, sidecarDeploymentId, sidecarInputColumnName, sidecarOutputColumnName, promptPipelineFiles, promptPipelineTemplateId, responsePipelineFiles, responsePipelineTemplateId] |
InsightEvaluationResultResponse
{
"aggregationType": "average",
"aggregationValue": 0,
"chatId": "string",
"chatName": "string",
"evaluationDatasetConfigurationId": "string",
"evaluationDatasetName": "string",
"evaluationName": "string",
"executionStatus": "NEW",
"gradingResult": "PASS",
"id": "string",
"insightGradingCriteria": {
"passThreshold": 100
},
"lastUpdateDate": "2019-08-24T14:15:22Z",
"llmTestResultId": "string",
"maxNumPrompts": 0,
"metricName": "string",
"promptSamplingStrategy": "random_without_replacement"
}
InsightEvaluationResultResponse
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| aggregationType | any | false | Aggregation type. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | AggregationType | false | The type of the metric aggregation. |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| aggregationValue | any | false | Aggregation value. None indicates that the aggregation failed. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | number | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | [AggregationValue] | false | [An individual record in an itemized metric aggregation.] |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| chatId | string | true | Chat ID. | |
| chatName | any | true | Chat name. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | maxLength: 5000 |
none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| evaluationDatasetConfigurationId | any | false | Evaluation dataset configuration ID. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| evaluationDatasetName | any | false | Evaluation dataset name. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | maxLength: 5000 |
none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| evaluationName | string | true | maxLength: 5000 |
Evaluation name. |
| executionStatus | ExecutionStatus | false | The execution status of the insight evaluation result. | |
| gradingResult | any | false | The grading result for this insight evaluation result. If not specified, execution status is not COMPLETED. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | GradingResult | false | Grading result. |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| id | string | true | Insight evaluation result ID. | |
| insightGradingCriteria | InsightGradingCriteria | true | Insight grading criteria. | |
| lastUpdateDate | string(date-time) | true | Last update date of the insight evaluation result (ISO 8601 formatted). | |
| llmTestResultId | string | true | LLM test result ID this insight evaluation result is associated to. | |
| maxNumPrompts | integer | true | Number of prompts used in evaluation. | |
| metricName | any | false | Name of the metric. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| promptSamplingStrategy | PromptSamplingStrategy | true | Prompt sampling strategy for maxNumPrompts. |
InsightGradingCriteria
{
"passThreshold": 100
}
InsightGradingCriteria
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| passThreshold | integer | true | maximum: 100 minimum: 0 |
The percentage threshold for Pass result. Greater than or equal to this threshold indicates a Pass. |
InsightToEvalDatasetsCompatibility
{
"incompatibleDatasets": [
{
"datasetId": "string",
"datasetName": "string"
}
],
"insightName": "string"
}
InsightToEvalDatasetsCompatibility
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| incompatibleDatasets | [DatasetIdentifier] | true | The list of incompatible datasets. | |
| insightName | string | true | The name of the insight. |
InsightTypes
"Reference"
InsightTypes
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| InsightTypes | string | false | The type of insight. |
Enumerated Values¶
| Property | Value |
|---|---|
| InsightTypes | [Reference, Quality metric, Operational metric, Evaluation deployment, Custom metric, Nemo] |
InsightsConfigurationWithAdditionalData
{
"aggregationTypes": [
"average"
],
"costConfigurationId": "string",
"customMetricId": "string",
"customModelLLMValidationId": "string",
"deploymentId": "string",
"errorMessage": "string",
"errorResolution": [
"string"
],
"evaluationDatasetConfigurationId": "string",
"executionStatus": "NEW",
"insightName": "string",
"insightType": "Reference",
"isTransferable": false,
"llmId": "azure-openai-gpt-3.5-turbo",
"llmIsActive": true,
"llmIsDeprecated": true,
"modelId": "string",
"modelPackageRegisteredModelId": "string",
"moderationConfiguration": {
"guardConditions": [
{
"comparand": 0,
"comparator": "greaterThan"
}
],
"intervention": {
"action": "block",
"message": "string"
}
},
"nemoMetricId": "string",
"ootbMetricId": "string",
"ootbMetricName": "latency",
"resultUnit": "s",
"sidecarModelMetricMetadata": {
"expectedResponseColumnName": "string",
"promptColumnName": "string",
"responseColumnName": "string",
"targetColumnName": "string"
},
"sidecarModelMetricValidationId": "string",
"stage": "prompt_pipeline"
}
InsightsConfigurationWithAdditionalData
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| aggregationTypes | any | true | The aggregation types used in the insights configuration. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | [AggregationType] | false | [The type of the metric aggregation.] |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| costConfigurationId | any | false | The ID of the cost configuration. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| customMetricId | any | false | The ID of the custom metric (if using a custom metric). |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| customModelLLMValidationId | any | false | The ID of the custom model LLM validation if using a custom model LLM for OOTB metrics. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| deploymentId | any | false | The ID of the custom model deployment associated with the insight. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| errorMessage | any | false | The error message associated with the evaluation dataset configuration or sidecar model metric validation or OOTB metric. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| errorResolution | any | false | The error type associated with the insight error status and error message as an indicator of what fields needs to be edited if any. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | [string] | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| evaluationDatasetConfigurationId | any | false | The ID of the evaluation dataset configuration. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| executionStatus | any | false | The execution status of the evaluation dataset configuration. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | ExecutionStatus | false | Job and entity execution status. |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| insightName | string | true | maxLength: 5000 minLength: 1 minLength: 1 |
The name of the insight. |
| insightType | any | false | The type of the insight. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | InsightTypes | false | The type of insight. |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| isTransferable | boolean | false | Indicates if insight can be transferred to production. | |
| llmId | any | false | The LLM ID for OOTB metrics that use LLMs. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | LanguageModelTypeId | false | The ID that defines the type of the LLM. |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| llmIsActive | any | false | Whether the LLM is active. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | boolean | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| llmIsDeprecated | any | false | Whether the LLM is deprecated and will be removed in a future release. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | boolean | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| modelId | any | false | The ID of the model associated with deploymentId. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| modelPackageRegisteredModelId | any | false | The ID of the registered model package associated with deploymentId. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| moderationConfiguration | any | false | The moderation configuration associated with the insight configuration. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | ModerationConfigurationWithID | false | Moderation Configuration associated with an insight. |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | ModerationConfigurationWithoutID | false | Moderation Configuration associated with an insight. |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| nemoMetricId | any | false | The ID of the Nemo configuration. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| ootbMetricId | any | false | The ID of the ootb metric (if using an ootb metric). |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| ootbMetricName | any | false | The OOTB metric name. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | OOTBMetricInsightNames | false | The Out-Of-The-Box metric name that can be used in the playground. |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| resultUnit | any | false | The unit of measurement associated with the insight result. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | MetricUnit | false | The unit of measurement associated with a metric. |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| sidecarModelMetricMetadata | any | false | The metadata of the sidecar model metric (if using a sidecar model metric). |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | SidecarModelMetricMetadata | false | The metadata of a sidecar model metric. |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| sidecarModelMetricValidationId | any | false | The ID of the sidecar model metric validation (if using a sidecar model metric). |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| stage | any | false | The stage (prompt or response) where insight is calculated at. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | PipelineStage | false | Enum that describes at which stage the metric may be calculated. |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
Intervention
{
"action": "block",
"message": "string"
}
Intervention
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| action | ModerationAction | true | The intervention strategy. | |
| message | string | true | minLength: 1 minLength: 1 |
The intervention message to replace the prediction when a guard condition is satisfied. |
LLMBlueprintSnapshot
{
"description": "string",
"id": "string",
"llmId": "azure-openai-gpt-3.5-turbo",
"llmSettings": {
"maxCompletionLength": 0,
"systemPrompt": "string",
"temperature": 0,
"topP": 0
},
"name": "string",
"playgroundId": "string",
"promptType": "CHAT_HISTORY_AWARE",
"snapshotDate": "2019-08-24T14:15:22Z",
"vectorDatabaseId": "string",
"vectorDatabaseSettings": {
"addNeighborChunks": false,
"maxDocumentsRetrievedPerPrompt": 1,
"maxTokens": 1,
"retriever": "SINGLE_LOOKUP_RETRIEVER"
}
}
LLMBlueprintSnapshot
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| description | string | true | The description of the LLMBlueprint at the time of snapshotting. | |
| id | string | true | The ID of the LLMBlueprint for which the snapshot was produced. | |
| llmId | any | false | The ID of the LLM selected for this LLM blueprint. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | LanguageModelTypeId | false | The ID that defines the type of the LLM. |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| llmSettings | any | false | A key/value dictionary of LLM settings. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | CommonLLMSettings | false | The settings that are available for all non-custom LLMs. |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | CustomModelLLMSettings | false | The settings that are available for custom model LLMs. |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| name | string | true | The name of the LLMBlueprint at the time of snapshotting. | |
| playgroundId | string | true | The playground id of the LLMBlueprint. | |
| promptType | PromptType | true | The prompting type of the LLMBlueprint at the time of snapshotting. | |
| snapshotDate | string(date-time) | false | The date when the snapshot was produced. | |
| vectorDatabaseId | any | false | The ID of the vector database linked to this LLM blueprint. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| vectorDatabaseSettings | any | false | A key/value dictionary of vector database settings. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | VectorDatabaseSettings | false | Vector database retrieval settings. |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
LLMCostConfigurationResponse
{
"currencyCode": "USD",
"customModelLLMValidationId": "string",
"inputTokenPrice": 0.01,
"llmId": "azure-openai-gpt-3.5-turbo",
"outputTokenPrice": 0.01,
"referenceInputTokenCount": 1000,
"referenceOutputTokenCount": 1000
}
LLMCostConfigurationResponse
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| currencyCode | string | false | maxLength: 7 |
The arbitrary code code of the currency of inputTokenPrice and outputTokenPrice. |
| customModelLLMValidationId | any | false | The ID of the custom model LLM validation (if using a custom model LLM). |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| inputTokenPrice | number | false | minimum: 0 |
The price of processing referenceInputTokenCount input tokens. |
| llmId | LanguageModelTypeId | true | The ID of the LLM associated with this cost configuration. | |
| outputTokenPrice | number | false | minimum: 0 |
The price of processing referenceOutputTokenCount output tokens. |
| referenceInputTokenCount | integer | false | minimum: 0 |
The number of input tokens corresponding to inputTokenPrice. |
| referenceOutputTokenCount | integer | false | minimum: 0 |
The number of output tokens corresponding to outputTokenPrice. |
LLMTestConfigurationNonOOTBDatasetResponse
{
"correctnessEnabled": true,
"creationDate": "2019-08-24T14:15:22Z",
"creationUserId": "string",
"datasetId": "string",
"datasetName": "string",
"errorMessage": "string",
"executionStatus": "NEW",
"id": "string",
"name": "string",
"playgroundId": "string",
"promptColumnName": "string",
"responseColumnName": "string",
"rowsCount": 0,
"size": 0,
"tenantId": "string",
"useCaseId": "string",
"userName": "string"
}
LLMTestConfigurationNonOOTBDatasetResponse
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| correctnessEnabled | any | true | Whether correctness is enabled for the evaluation dataset configuration. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | boolean | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| creationDate | string(date-time) | true | The creation date of the evaluation dataset configuration (ISO 8601 formatted). | |
| creationUserId | string | true | The ID of the user that created the evaluation dataset configuration. | |
| datasetId | string | true | The ID of the evaluation dataset. | |
| datasetName | string | true | The name of the evaluation dataset. | |
| errorMessage | any | false | The error message associated with the evaluation dataset configuration. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| executionStatus | ExecutionStatus | true | The execution status of the evaluation dataset. | |
| id | string | true | The ID of the evaluation dataset configuration. | |
| name | string | true | The name of the evaluation dataset configuration. | |
| playgroundId | any | true | The ID of the playground associated with the evaluation dataset configuration. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| promptColumnName | string | true | The name of the dataset column containing the prompt text. | |
| responseColumnName | any | true | The name of the dataset column containing the response text. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| rowsCount | integer | true | The rows count of the evaluation dataset. | |
| size | integer | true | The size of the evaluation dataset (in bytes). | |
| tenantId | string(uuid4) | true | The ID of the DataRobot tenant this evaluation dataset configuration belongs to. | |
| useCaseId | string | true | The ID of the use case associated with the evaluation dataset configuration. | |
| userName | string | true | The name of the user that created the evaluation dataset configuration. |
LLMTestConfigurationOOTBDatasetResponse
{
"datasetName": "jailbreak-v1.csv",
"datasetUrl": "https://s3.amazonaws.com/datarobot_public_datasets/genai/jailbreak-v1.csv",
"promptColumnName": "string",
"responseColumnName": "string",
"rowsCount": 0,
"warning": "string"
}
LLMTestConfigurationOOTBDatasetResponse
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| datasetName | OOTBDatasetName | true | The name of the evaluation dataset. | |
| datasetUrl | any | true | The public URL of the evaluation dataset. This applies only to our predefined public evaluation datasets. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | OOTBDatasetUrl | false | Out-of-the-box dataset URL. |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| promptColumnName | string | true | maxLength: 5000 minLength: 1 minLength: 1 |
The name of the prompt column. |
| responseColumnName | any | true | The name of the response column, if present. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | maxLength: 5000 minLength: 1 minLength: 1 |
none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| rowsCount | integer | true | The number rows in the dataset. | |
| warning | any | false | Warning about the content of the dataset. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
LLMTestConfigurationResponse
{
"creationDate": "2019-08-24T14:15:22Z",
"creationUserId": "string",
"datasetEvaluations": [
{
"errorMessage": "string",
"evaluationDatasetConfigurationId": "string",
"evaluationDatasetName": "string",
"evaluationName": "string",
"insightConfiguration": {
"aggregationTypes": [
"average"
],
"costConfigurationId": "string",
"customMetricId": "string",
"customModelLLMValidationId": "string",
"deploymentId": "string",
"errorMessage": "string",
"errorResolution": [
"string"
],
"evaluationDatasetConfigurationId": "string",
"executionStatus": "NEW",
"insightName": "string",
"insightType": "Reference",
"isTransferable": false,
"llmId": "azure-openai-gpt-3.5-turbo",
"llmIsActive": true,
"llmIsDeprecated": true,
"modelId": "string",
"modelPackageRegisteredModelId": "string",
"moderationConfiguration": {
"guardConditions": [
{
"comparand": 0,
"comparator": "greaterThan"
}
],
"intervention": {
"action": "block",
"message": "string"
}
},
"nemoMetricId": "string",
"ootbMetricId": "string",
"ootbMetricName": "latency",
"resultUnit": "s",
"sidecarModelMetricMetadata": {
"expectedResponseColumnName": "string",
"promptColumnName": "string",
"responseColumnName": "string",
"targetColumnName": "string"
},
"sidecarModelMetricValidationId": "string",
"stage": "prompt_pipeline"
},
"insightGradingCriteria": {
"passThreshold": 100
},
"maxNumPrompts": 100,
"ootbDataset": {
"datasetName": "jailbreak-v1.csv",
"datasetUrl": "https://s3.amazonaws.com/datarobot_public_datasets/genai/jailbreak-v1.csv",
"promptColumnName": "string",
"responseColumnName": "string",
"rowsCount": 0,
"warning": "string"
},
"promptSamplingStrategy": "random_without_replacement"
}
],
"description": "string",
"errorMessage": "string",
"id": "string",
"isOutOfTheBoxTestConfiguration": true,
"lastUpdateDate": "2019-08-24T14:15:22Z",
"lastUpdateUserId": "string",
"llmTestGradingCriteria": {
"passThreshold": 100
},
"name": "string",
"useCaseId": "string",
"warnings": [
{
"property1": "string",
"property2": "string"
}
]
}
LLMTestConfigurationResponse
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| creationDate | any | false | The creation date of the LLM Test configuration. For OOTB LLM Test configurations this is null. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string(date-time) | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| creationUserId | any | false | The ID of the user who created the LLM Test configuration. For OOTB LLM Test configurations this is null. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| datasetEvaluations | [DatasetEvaluationResponse] | true | The LLM test dataset evaluations. | |
| description | string | true | The description of the LLM Test configuration. | |
| errorMessage | any | false | The error message associated with the LLM test configuration. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| id | string | true | The ID of the LLM Test configuration. | |
| isOutOfTheBoxTestConfiguration | boolean | true | Identifies the LLM Test configuration as an out-of-the-box (OOTB) test configuration. | |
| lastUpdateDate | any | false | The last update date of the LLM Test configuration. For OOTB LLM Test configurations this is null. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string(date-time) | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| lastUpdateUserId | any | false | The ID of the user who last updated the LLM Test configuration. For OOTB LLM Test configurations this is null. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| llmTestGradingCriteria | LLMTestGradingCriteria | true | The LLM test grading criteria. | |
| name | string | true | The name of the LLM Test configuration. | |
| useCaseId | any | false | If specified, the use case ID associated with the LLM Test configuration. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| warnings | [object] | true | Warnings for this LLM test configuration. | |
| » additionalProperties | string | false | none |
LLMTestConfigurationSupportedInsightsResponse
{
"datasetsCompatibility": [
{
"incompatibleDatasets": [
{
"datasetId": "string",
"datasetName": "string"
}
],
"insightName": "string"
}
],
"supportedInsightConfigurations": [
{
"aggregationTypes": [
"average"
],
"costConfigurationId": "string",
"customMetricId": "string",
"customModelLLMValidationId": "string",
"deploymentId": "string",
"errorMessage": "string",
"errorResolution": [
"string"
],
"evaluationDatasetConfigurationId": "string",
"executionStatus": "NEW",
"insightName": "string",
"insightType": "Reference",
"isTransferable": false,
"llmId": "azure-openai-gpt-3.5-turbo",
"llmIsActive": true,
"llmIsDeprecated": true,
"modelId": "string",
"modelPackageRegisteredModelId": "string",
"moderationConfiguration": {
"guardConditions": [
{
"comparand": 0,
"comparator": "greaterThan"
}
],
"intervention": {
"action": "block",
"message": "string"
}
},
"nemoMetricId": "string",
"ootbMetricId": "string",
"ootbMetricName": "latency",
"resultUnit": "s",
"sidecarModelMetricMetadata": {
"expectedResponseColumnName": "string",
"promptColumnName": "string",
"responseColumnName": "string",
"targetColumnName": "string"
},
"sidecarModelMetricValidationId": "string",
"stage": "prompt_pipeline"
}
]
}
LLMTestConfigurationSupportedInsightsResponse
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| datasetsCompatibility | [InsightToEvalDatasetsCompatibility] | true | The list of insight to evaluation datasets compatibility. | |
| supportedInsightConfigurations | [InsightsConfigurationWithAdditionalData] | true | The list of supported insight configurations for the LLM Tests. |
LLMTestConfigurationType
"ootb"
LLMTestConfigurationType
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| LLMTestConfigurationType | string | false | Type of LLMTestConfiguration. |
Enumerated Values¶
| Property | Value |
|---|---|
| LLMTestConfigurationType | [ootb, custom] |
LLMTestGradingCriteria
{
"passThreshold": 100
}
LLMTestGradingCriteria
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| passThreshold | integer | true | maximum: 100 minimum: 0 |
The percentage threshold for Pass results across dataset-insight pairs. |
LLMTestResultResponse
{
"creationDate": "2019-08-24T14:15:22Z",
"creationUserId": "string",
"creationUserName": "string",
"errorMessage": "string",
"errorResolution": "string",
"executionStatus": "NEW",
"gradingResult": "PASS",
"id": "string",
"insightEvaluationResults": [
{
"aggregationType": "average",
"aggregationValue": 0,
"chatId": "string",
"chatName": "string",
"evaluationDatasetConfigurationId": "string",
"evaluationDatasetName": "string",
"evaluationName": "string",
"executionStatus": "NEW",
"gradingResult": "PASS",
"id": "string",
"insightGradingCriteria": {
"passThreshold": 100
},
"lastUpdateDate": "2019-08-24T14:15:22Z",
"llmTestResultId": "string",
"maxNumPrompts": 0,
"metricName": "string",
"promptSamplingStrategy": "random_without_replacement"
}
],
"isOutOfTheBoxTestConfiguration": true,
"llmBlueprintId": "string",
"llmBlueprintSnapshot": {
"description": "string",
"id": "string",
"llmId": "azure-openai-gpt-3.5-turbo",
"llmSettings": {
"maxCompletionLength": 0,
"systemPrompt": "string",
"temperature": 0,
"topP": 0
},
"name": "string",
"playgroundId": "string",
"promptType": "CHAT_HISTORY_AWARE",
"snapshotDate": "2019-08-24T14:15:22Z",
"vectorDatabaseId": "string",
"vectorDatabaseSettings": {
"addNeighborChunks": false,
"maxDocumentsRetrievedPerPrompt": 1,
"maxTokens": 1,
"retriever": "SINGLE_LOOKUP_RETRIEVER"
}
},
"llmTestConfigurationId": "string",
"llmTestConfigurationName": "string",
"llmTestGradingCriteria": {
"passThreshold": 100
},
"llmTestSuiteId": "string",
"passPercentage": 0,
"useCaseId": "string"
}
LLMTestResultResponse
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| creationDate | string(date-time) | true | LLM test result creation date (ISO 8601 formatted). | |
| creationUserId | string | true | ID of the user that created this LLM test result. | |
| creationUserName | string | true | The name of the user who created this LLM result. | |
| errorMessage | any | false | The error message if the LLM Test Result failed. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| errorResolution | any | false | The error resolution message if the LLM Test Result failed. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| executionStatus | ExecutionStatus | true | The LLM Test execution status. | |
| gradingResult | any | false | The grading result based on the llm test grading criteria. If not specified, execution status is not COMPLETED. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | GradingResult | false | Grading result. |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| id | string | true | LLM test result ID. | |
| insightEvaluationResults | [InsightEvaluationResultResponse] | true | The Insight evaluation results. | |
| isOutOfTheBoxTestConfiguration | boolean | true | Identifies the LLM Test configuration as an out-of-the-box (OOTB) test configuration. | |
| llmBlueprintId | string | true | LLM Blueprint ID. | |
| llmBlueprintSnapshot | LLMBlueprintSnapshot | true | A snapshot of the llm blueprint entity at the time of LLM Test execution. | |
| llmTestConfigurationId | string | true | LLM test configuration ID this LLM result is associated to. | |
| llmTestConfigurationName | any | true | Name of the LLM test configuration this LLM result is associated to. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | maxLength: 5000 minLength: 1 minLength: 1 |
none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| llmTestGradingCriteria | LLMTestGradingCriteria | true | LLM test grading criteria. | |
| llmTestSuiteId | any | false | LLM test suite ID to which the LLM test configuration is associated to. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| passPercentage | any | false | The percentage of underlying insight evaluation results that have a PASS grading result. If not specified, execution status is not COMPLETED. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | number | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| useCaseId | string | true | Use case ID this LLM test result belongs to. |
LLMTestSuiteResponse
{
"creationDate": "2019-08-24T14:15:22Z",
"creationUserId": "string",
"description": "string",
"id": "string",
"llmTestConfigurationIds": [
"string"
],
"name": "string",
"useCaseId": "string"
}
LLMTestSuiteResponse
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| creationDate | string(date-time) | true | The creation date of the chat (ISO 8601 formatted). | |
| creationUserId | string | true | The ID of the user that created the chat. | |
| description | string | true | The description of the LLM test suite. | |
| id | string | true | The ID of the LLM test suite. | |
| llmTestConfigurationIds | [string] | true | The IDs of the LLM test configurations in this LLM test suite. | |
| name | string | true | The name of the LLM test suite. | |
| useCaseId | string | true | The ID of the use case associated with the LLM test suite. |
LanguageModelTypeId
"azure-openai-gpt-3.5-turbo"
LanguageModelTypeId
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| LanguageModelTypeId | string | false | The ID that defines the type of the LLM. |
Enumerated Values¶
| Property | Value |
|---|---|
| LanguageModelTypeId | [azure-openai-gpt-3.5-turbo, azure-openai-gpt-3.5-turbo-16k, azure-openai-gpt-4, azure-openai-gpt-4-32k, azure-openai-gpt-4-turbo, azure-openai-gpt-4-o, azure-openai-gpt-4-o-mini, amazon-titan, amazon-nova-micro, amazon-nova-lite, amazon-nova-pro, anthropic-claude-2, anthropic-claude-3-haiku, anthropic-claude-3-sonnet, anthropic-claude-3-opus, anthropic-claude-3.5-sonnet-v1, amazon-anthropic-claude-3.5-sonnet-v2, google-bison, google-gemini-1.5-flash, google-gemini-1.5-pro, custom-model] |
ListCustomModelValidationSortQueryParam
"name"
ListCustomModelValidationSortQueryParam
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| ListCustomModelValidationSortQueryParam | string | false | Sort order values for listing custom model validations. |
Enumerated Values¶
| Property | Value |
|---|---|
| ListCustomModelValidationSortQueryParam | [name, -name, deploymentName, -deploymentName, userName, -userName, creationDate, -creationDate] |
ListEvaluationDatasetConfigurationResponse
{
"count": 0,
"data": [
{
"correctnessEnabled": true,
"creationDate": "2019-08-24T14:15:22Z",
"creationUserId": "string",
"datasetId": "string",
"datasetName": "string",
"errorMessage": "string",
"executionStatus": "NEW",
"id": "string",
"name": "string",
"playgroundId": "string",
"promptColumnName": "string",
"responseColumnName": "string",
"rowsCount": 0,
"size": 0,
"tenantId": "string",
"useCaseId": "string",
"userName": "string"
}
],
"next": "string",
"previous": "string",
"totalCount": 0
}
ListEvaluationDatasetConfigurationResponse
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| count | integer | true | The number of records on this page. | |
| data | [EvaluationDatasetConfigurationResponse] | true | The list of records. | |
| next | any | true | The URL to the next page, or null if there is no such page. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| previous | any | true | The URL to the previous page, or null if there is no such page. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| totalCount | integer | true | The total number of records. |
ListEvaluationDatasetConfigurationsQueryParam
"name"
ListEvaluationDatasetConfigurationsQueryParam
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| ListEvaluationDatasetConfigurationsQueryParam | string | false | Sort order values for listing evaluation dataset configurations. |
Enumerated Values¶
| Property | Value |
|---|---|
| ListEvaluationDatasetConfigurationsQueryParam | [name, -name, creationUserId, -creationUserId, creationDate, -creationDate, datasetId, -datasetId, userName, -userName, datasetName, -datasetName, promptColumnName, -promptColumnName, responseColumnName, -responseColumnName] |
ListEvaluationDatasetMetricAggregationAggregatedByLLMBlueprintResponse
{
"count": 0,
"data": [
{
"aggregatedItemCount": 0,
"aggregatedItemDetails": [
{
"chatId": "string",
"chatLink": "string",
"chatName": "string",
"creationDate": "2019-08-24T14:15:22Z",
"creationUserId": "string",
"creationUserName": "string"
}
],
"aggregationType": "average",
"aggregationValue": 0,
"datasetId": "string",
"datasetName": "string",
"evaluationDatasetConfigurationId": "string",
"llmBlueprintId": "string",
"metricName": "string",
"ootbDatasetName": "jailbreak-v1.csv",
"tenantId": "string"
}
],
"next": "string",
"previous": "string",
"totalCount": 0
}
ListEvaluationDatasetMetricAggregationAggregatedByLLMBlueprintResponse
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| count | integer | true | The number of records on this page. | |
| data | [EvaluationDatasetMetricAggregationAggregatedByLLMBlueprintResponse] | true | The list of records. | |
| next | any | true | The URL to the next page, or null if there is no such page. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| previous | any | true | The URL to the previous page, or null if there is no such page. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| totalCount | integer | true | The total number of records. |
ListEvaluationDatasetMetricAggregationResponse
{
"count": 0,
"data": [
{
"aggregationType": "average",
"aggregationValue": 0,
"chatId": "string",
"chatLink": "string",
"chatName": "string",
"creationDate": "2019-08-24T14:15:22Z",
"creationUserId": "string",
"creationUserName": "string",
"datasetId": "string",
"datasetName": "string",
"evaluationDatasetConfigurationId": "string",
"llmBlueprintId": "string",
"metricName": "string",
"ootbDatasetName": "jailbreak-v1.csv",
"tenantId": "string"
}
],
"next": "string",
"previous": "string",
"totalCount": 0
}
ListEvaluationDatasetMetricAggregationResponse
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| count | integer | true | The number of records on this page. | |
| data | [EvaluationDatasetMetricAggregationResponse] | true | The list of records. | |
| next | any | true | The URL to the next page, or null if there is no such page. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| previous | any | true | The URL to the previous page, or null if there is no such page. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| totalCount | integer | true | The total number of records. |
ListEvaluationDatasetMetricAggregationSortQueryParam
"metricName"
ListEvaluationDatasetMetricAggregationSortQueryParam
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| ListEvaluationDatasetMetricAggregationSortQueryParam | string | false | Sort order values for listing evaluation dataset metric aggregations. |
Enumerated Values¶
| Property | Value |
|---|---|
| ListEvaluationDatasetMetricAggregationSortQueryParam | [metricName, -metricName, aggregationValue, -aggregationValue, datasetName, -datasetName, datasetId, -datasetId, creationUserId, -creationUserId, creationUserName, -creationUserName, creationDate, -creationDate, evaluationDatasetConfigurationId, -evaluationDatasetConfigurationId] |
ListEvaluationDatasetMetricAggregationUniqueFieldValuesResponse
{
"count": 0,
"data": [
{
"uniqueFieldValue": "string"
}
],
"next": "string",
"previous": "string",
"totalCount": 0
}
ListEvaluationDatasetMetricAggregationUniqueFieldValuesResponse
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| count | integer | true | The number of records on this page. | |
| data | [EvaluationDatasetMetricAggregationUniqueFieldValuesResponse] | true | The list of records. | |
| next | any | true | The URL to the next page, or null if there is no such page. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| previous | any | true | The URL to the previous page, or null if there is no such page. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| totalCount | integer | true | The total number of records. |
ListLLMTestConfigurationNonOOTBDatasetsResponse
{
"count": 0,
"data": [
{
"correctnessEnabled": true,
"creationDate": "2019-08-24T14:15:22Z",
"creationUserId": "string",
"datasetId": "string",
"datasetName": "string",
"errorMessage": "string",
"executionStatus": "NEW",
"id": "string",
"name": "string",
"playgroundId": "string",
"promptColumnName": "string",
"responseColumnName": "string",
"rowsCount": 0,
"size": 0,
"tenantId": "string",
"useCaseId": "string",
"userName": "string"
}
],
"next": "string",
"previous": "string",
"totalCount": 0
}
ListLLMTestConfigurationNonOOTBDatasetsResponse
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| count | integer | true | The number of records on this page. | |
| data | [LLMTestConfigurationNonOOTBDatasetResponse] | true | The list of records. | |
| next | any | true | The URL to the next page, or null if there is no such page. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| previous | any | true | The URL to the previous page, or null if there is no such page. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| totalCount | integer | true | The total number of records. |
ListLLMTestConfigurationOOTBDatasetsResponse
{
"count": 0,
"data": [
{
"datasetName": "jailbreak-v1.csv",
"datasetUrl": "https://s3.amazonaws.com/datarobot_public_datasets/genai/jailbreak-v1.csv",
"promptColumnName": "string",
"responseColumnName": "string",
"rowsCount": 0,
"warning": "string"
}
],
"next": "string",
"previous": "string",
"totalCount": 0
}
ListLLMTestConfigurationOOTBDatasetsResponse
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| count | integer | true | The number of records on this page. | |
| data | [LLMTestConfigurationOOTBDatasetResponse] | true | The list of records. | |
| next | any | true | The URL to the next page, or null if there is no such page. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| previous | any | true | The URL to the previous page, or null if there is no such page. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| totalCount | integer | true | The total number of records. |
ListLLMTestConfigurationsResponse
{
"count": 0,
"data": [
{
"creationDate": "2019-08-24T14:15:22Z",
"creationUserId": "string",
"datasetEvaluations": [
{
"errorMessage": "string",
"evaluationDatasetConfigurationId": "string",
"evaluationDatasetName": "string",
"evaluationName": "string",
"insightConfiguration": {
"aggregationTypes": [
"average"
],
"costConfigurationId": "string",
"customMetricId": "string",
"customModelLLMValidationId": "string",
"deploymentId": "string",
"errorMessage": "string",
"errorResolution": [
"string"
],
"evaluationDatasetConfigurationId": "string",
"executionStatus": "NEW",
"insightName": "string",
"insightType": "Reference",
"isTransferable": false,
"llmId": "azure-openai-gpt-3.5-turbo",
"llmIsActive": true,
"llmIsDeprecated": true,
"modelId": "string",
"modelPackageRegisteredModelId": "string",
"moderationConfiguration": {
"guardConditions": [
{
"comparand": 0,
"comparator": "greaterThan"
}
],
"intervention": {
"action": "block",
"message": "string"
}
},
"nemoMetricId": "string",
"ootbMetricId": "string",
"ootbMetricName": "latency",
"resultUnit": "s",
"sidecarModelMetricMetadata": {
"expectedResponseColumnName": "string",
"promptColumnName": "string",
"responseColumnName": "string",
"targetColumnName": "string"
},
"sidecarModelMetricValidationId": "string",
"stage": "prompt_pipeline"
},
"insightGradingCriteria": {
"passThreshold": 100
},
"maxNumPrompts": 100,
"ootbDataset": {
"datasetName": "jailbreak-v1.csv",
"datasetUrl": "https://s3.amazonaws.com/datarobot_public_datasets/genai/jailbreak-v1.csv",
"promptColumnName": "string",
"responseColumnName": "string",
"rowsCount": 0,
"warning": "string"
},
"promptSamplingStrategy": "random_without_replacement"
}
],
"description": "string",
"errorMessage": "string",
"id": "string",
"isOutOfTheBoxTestConfiguration": true,
"lastUpdateDate": "2019-08-24T14:15:22Z",
"lastUpdateUserId": "string",
"llmTestGradingCriteria": {
"passThreshold": 100
},
"name": "string",
"useCaseId": "string",
"warnings": [
{
"property1": "string",
"property2": "string"
}
]
}
],
"next": "string",
"previous": "string",
"totalCount": 0
}
ListLLMTestConfigurationsResponse
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| count | integer | true | The number of records on this page. | |
| data | [LLMTestConfigurationResponse] | true | The list of records. | |
| next | any | true | The URL to the next page, or null if there is no such page. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| previous | any | true | The URL to the previous page, or null if there is no such page. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| totalCount | integer | true | The total number of records. |
ListLLMTestResultResponse
{
"count": 0,
"data": [
{
"creationDate": "2019-08-24T14:15:22Z",
"creationUserId": "string",
"creationUserName": "string",
"errorMessage": "string",
"errorResolution": "string",
"executionStatus": "NEW",
"gradingResult": "PASS",
"id": "string",
"insightEvaluationResults": [
{
"aggregationType": "average",
"aggregationValue": 0,
"chatId": "string",
"chatName": "string",
"evaluationDatasetConfigurationId": "string",
"evaluationDatasetName": "string",
"evaluationName": "string",
"executionStatus": "NEW",
"gradingResult": "PASS",
"id": "string",
"insightGradingCriteria": {
"passThreshold": 100
},
"lastUpdateDate": "2019-08-24T14:15:22Z",
"llmTestResultId": "string",
"maxNumPrompts": 0,
"metricName": "string",
"promptSamplingStrategy": "random_without_replacement"
}
],
"isOutOfTheBoxTestConfiguration": true,
"llmBlueprintId": "string",
"llmBlueprintSnapshot": {
"description": "string",
"id": "string",
"llmId": "azure-openai-gpt-3.5-turbo",
"llmSettings": {
"maxCompletionLength": 0,
"systemPrompt": "string",
"temperature": 0,
"topP": 0
},
"name": "string",
"playgroundId": "string",
"promptType": "CHAT_HISTORY_AWARE",
"snapshotDate": "2019-08-24T14:15:22Z",
"vectorDatabaseId": "string",
"vectorDatabaseSettings": {
"addNeighborChunks": false,
"maxDocumentsRetrievedPerPrompt": 1,
"maxTokens": 1,
"retriever": "SINGLE_LOOKUP_RETRIEVER"
}
},
"llmTestConfigurationId": "string",
"llmTestConfigurationName": "string",
"llmTestGradingCriteria": {
"passThreshold": 100
},
"llmTestSuiteId": "string",
"passPercentage": 0,
"useCaseId": "string"
}
],
"next": "string",
"previous": "string",
"totalCount": 0
}
ListLLMTestResultResponse
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| count | integer | true | The number of records on this page. | |
| data | [LLMTestResultResponse] | true | The list of records. | |
| next | any | true | The URL to the next page, or null if there is no such page. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| previous | any | true | The URL to the previous page, or null if there is no such page. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| totalCount | integer | true | The total number of records. |
ListLLMTestSuiteSortQueryParam
"name"
ListLLMTestSuiteSortQueryParam
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| ListLLMTestSuiteSortQueryParam | string | false | Sort order values for listing chats. |
Enumerated Values¶
| Property | Value |
|---|---|
| ListLLMTestSuiteSortQueryParam | [name, -name, creationDate, -creationDate] |
ListLLMTestSuitesResponse
{
"count": 0,
"data": [
{
"creationDate": "2019-08-24T14:15:22Z",
"creationUserId": "string",
"description": "string",
"id": "string",
"llmTestConfigurationIds": [
"string"
],
"name": "string",
"useCaseId": "string"
}
],
"next": "string",
"previous": "string",
"totalCount": 0
}
ListLLMTestSuitesResponse
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| count | integer | true | The number of records on this page. | |
| data | [LLMTestSuiteResponse] | true | The list of records. | |
| next | any | true | The URL to the next page, or null if there is no such page. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| previous | any | true | The URL to the previous page, or null if there is no such page. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| totalCount | integer | true | The total number of records. |
ListSidecarModelMetricValidationnResponse
{
"count": 0,
"data": [
{
"citationsPrefixColumnName": "string",
"creationDate": "2019-08-24T14:15:22Z",
"deploymentAccessData": {
"authorizationHeader": "[REDACTED]",
"chatApiUrl": "string",
"datarobotKey": "string",
"inputType": "CSV",
"modelType": "TEXT_GENERATION",
"predictionApiUrl": "string"
},
"deploymentId": "string",
"deploymentName": "string",
"errorMessage": "string",
"errorResolution": [
"ootbMetricName"
],
"expectedResponseColumnName": "string",
"id": "string",
"modelId": "string",
"moderationConfiguration": {
"guardConditions": [
{
"comparand": 0,
"comparator": "greaterThan"
}
],
"intervention": {
"action": "block",
"message": "string"
}
},
"name": "string",
"playgroundId": "string",
"predictionTimeout": 0,
"promptColumnName": "string",
"responseColumnName": "string",
"targetColumnName": "string",
"tenantId": "string",
"useCaseId": "string",
"userId": "string",
"userName": "string",
"validationStatus": "TESTING"
}
],
"next": "string",
"previous": "string",
"totalCount": 0
}
ListSidecarModelMetricValidationnResponse
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| count | integer | true | The number of records on this page. | |
| data | [SidecarModelMetricValidationResponse] | true | The list of records. | |
| next | any | true | The URL to the next page, or null if there is no such page. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| previous | any | true | The URL to the previous page, or null if there is no such page. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| totalCount | integer | true | The total number of records. |
MetricUnit
"s"
MetricUnit
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| MetricUnit | string | false | The unit of measurement associated with a metric. |
Enumerated Values¶
| Property | Value |
|---|---|
| MetricUnit | [s, ms, %] |
ModerationAction
"block"
ModerationAction
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| ModerationAction | string | false | The moderation strategy. |
Enumerated Values¶
| Property | Value |
|---|---|
| ModerationAction | [block, report, reportAndBlock] |
ModerationConfigurationWithID
{
"guardConditions": [
{
"comparand": 0,
"comparator": "greaterThan"
}
],
"intervention": {
"action": "block",
"message": "string"
}
}
ModerationConfigurationWithID
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| guardConditions | [GuardCondition] | true | maxItems: 1 minItems: 1 |
The guard conditions associated with a metric. |
| intervention | Intervention | true | The intervention specific moderation configuration. |
ModerationConfigurationWithoutID
{
"guardConditions": [
{
"comparand": 0,
"comparator": "greaterThan"
}
],
"intervention": {
"action": "block",
"message": "string"
}
}
ModerationConfigurationWithoutID
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| guardConditions | [GuardCondition] | true | maxItems: 1 minItems: 1 |
The guard conditions associated with a metric. |
| intervention | Intervention | true | The intervention specific moderation configuration. |
OOTBDataset
{
"datasetName": "jailbreak-v1.csv",
"datasetUrl": "https://s3.amazonaws.com/datarobot_public_datasets/genai/jailbreak-v1.csv",
"promptColumnName": "string",
"responseColumnName": "string",
"rowsCount": 0,
"warning": "string"
}
OOTBDataset
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| datasetName | OOTBDatasetName | true | The name of the evaluation dataset. | |
| datasetUrl | any | true | The public URL of the evaluation dataset. This applies only to our predefined public evaluation datasets. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | OOTBDatasetUrl | false | Out-of-the-box dataset URL. |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| promptColumnName | string | true | maxLength: 5000 minLength: 1 minLength: 1 |
The name of the prompt column. |
| responseColumnName | any | true | The name of the response column, if present. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | maxLength: 5000 minLength: 1 minLength: 1 |
none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| rowsCount | integer | true | The number rows in the dataset. | |
| warning | any | false | Warning about the content of the dataset. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
OOTBDatasetName
"jailbreak-v1.csv"
OOTBDatasetName
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| OOTBDatasetName | string | false | Out-of-the-box dataset name. |
Enumerated Values¶
| Property | Value |
|---|---|
| OOTBDatasetName | [jailbreak-v1.csv, bbq-lite-age-v1.csv, bbq-lite-gender-v1.csv, bbq-lite-race-ethnicity-v1.csv, bbq-lite-religion-v1.csv, bbq-lite-disability-status-v1.csv, bbq-lite-sexual-orientation-v1.csv, bbq-lite-nationality-v1.csv, bbq-lite-ses-v1.csv, completeness-parent-v1.csv, completeness-grandparent-v1.csv, completeness-great-grandparent-v1.csv, pii-v1.csv, toxicity-v2.csv, jbbq-age-v1.csv, jbbq-gender-identity-v1.csv, jbbq-physical-appearance-v1.csv, jbbq-disability-status-v1.csv, jbbq-sexual-orientation-v1.csv] |
OOTBDatasetUrl
"https://s3.amazonaws.com/datarobot_public_datasets/genai/jailbreak-v1.csv"
OOTBDatasetUrl
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| OOTBDatasetUrl | string | false | Out-of-the-box dataset URL. |
Enumerated Values¶
| Property | Value |
|---|---|
| OOTBDatasetUrl | [https://s3.amazonaws.com/datarobot_public_datasets/genai/jailbreak-v1.csv, https://s3.amazonaws.com/datarobot_public_datasets/genai/bbq-lite-age-v1.csv, https://s3.amazonaws.com/datarobot_public_datasets/genai/bbq-lite-gender-v1.csv, https://s3.amazonaws.com/datarobot_public_datasets/genai/bbq-lite-race-ethnicity-v1.csv, https://s3.amazonaws.com/datarobot_public_datasets/genai/bbq-lite-religion-v1.csv, https://s3.amazonaws.com/datarobot_public_datasets/genai/bbq-lite-disability-status-v1.csv, https://s3.amazonaws.com/datarobot_public_datasets/genai/bbq-lite-sexual-orientation-v1.csv, https://s3.amazonaws.com/datarobot_public_datasets/genai/bbq-lite-nationality-v1.csv, https://s3.amazonaws.com/datarobot_public_datasets/genai/bbq-lite-ses-v1.csv, https://s3.amazonaws.com/datarobot_public_datasets/genai/completeness-parent-v1.csv, https://s3.amazonaws.com/datarobot_public_datasets/genai/completeness-grandparent-v1.csv, https://s3.amazonaws.com/datarobot_public_datasets/genai/completeness-great-grandparent-v1.csv, https://s3.amazonaws.com/datarobot_public_datasets/genai/pii-v1.csv] |
OOTBMetricConfigurationResponse
{
"customModelLLMValidationId": "string",
"customOotbMetricName": "string",
"errorMessage": "string",
"errorResolution": [
"ootbMetricName"
],
"executionStatus": "NEW",
"llmId": "azure-openai-gpt-3.5-turbo",
"moderationConfiguration": {
"guardConditions": [
{
"comparand": 0,
"comparator": "greaterThan"
}
],
"intervention": {
"action": "block",
"message": "string"
}
},
"ootbMetricConfigurationId": "string",
"ootbMetricName": "latency"
}
OOTBMetricConfigurationResponse
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| customModelLLMValidationId | any | false | The ID of the custom model LLM validation (if using a custom model LLM). |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| customOotbMetricName | any | false | The custom OOTB metric name to be associated with the OOTB metric. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| errorMessage | any | false | The error message associated with the OOTB metric configuration. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| errorResolution | any | false | The error type associated with the insight error status and error message as an indicator of what fields needs to be edited if any. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | [InsightErrorResolution] | false | [Error type linking directly to the field name that is related to the error.] |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| executionStatus | ExecutionStatus | true | The execution status of the OOTB metric configuration. | |
| llmId | any | false | The ID of the LLM to use for correctness and faithfulness metrics. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | LanguageModelTypeId | false | The ID that defines the type of the LLM. |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| moderationConfiguration | any | false | The moderation configuration to be associated with the OOTB metric. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | ModerationConfigurationWithoutID | false | Moderation Configuration associated with an insight. |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| ootbMetricConfigurationId | string | true | The ID of OOTB metric. | |
| ootbMetricName | OOTBMetricInsightNames | true | The Out-Of-The-Box metric name that can be used in the playground. |
OOTBMetricInsightNames
"latency"
OOTBMetricInsightNames
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| OOTBMetricInsightNames | string | false | The Out-Of-The-Box metric name that can be used in the playground. |
Enumerated Values¶
| Property | Value |
|---|---|
| OOTBMetricInsightNames | [latency, citations, rouge_1, faithfulness, correctness, prompt_tokens, response_tokens, document_tokens, all_tokens, jailbreak_violation, toxicity_violation, pii_violation, exact_match, starts_with, contains] |
PipelineStage
"prompt_pipeline"
PipelineStage
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| PipelineStage | string | false | Enum that describes at which stage the metric may be calculated. |
Enumerated Values¶
| Property | Value |
|---|---|
| PipelineStage | [prompt_pipeline, response_pipeline] |
PromptSamplingStrategy
"random_without_replacement"
PromptSamplingStrategy
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| PromptSamplingStrategy | string | false | The prompt sampling strategy for the evaluation dataset configuration. |
Enumerated Values¶
| Property | Value |
|---|---|
| PromptSamplingStrategy | [random_without_replacement, first_n_rows] |
PromptType
"CHAT_HISTORY_AWARE"
PromptType
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| PromptType | string | false | Determines whether chat history is submitted as context to the user prompt. |
Enumerated Values¶
| Property | Value |
|---|---|
| PromptType | [CHAT_HISTORY_AWARE, ONE_TIME_PROMPT] |
SidecarModelMetricMetadata
{
"expectedResponseColumnName": "string",
"promptColumnName": "string",
"responseColumnName": "string",
"targetColumnName": "string"
}
SidecarModelMetricMetadata
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| expectedResponseColumnName | any | false | The name of the column the custom model uses for expected response text input. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| promptColumnName | any | false | The name of the column the custom model uses for prompt text input. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| responseColumnName | any | false | The name of the column the custom model uses for response text input. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| targetColumnName | any | true | The name of the column the custom model uses for prediction output. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
SidecarModelMetricValidationResponse
{
"citationsPrefixColumnName": "string",
"creationDate": "2019-08-24T14:15:22Z",
"deploymentAccessData": {
"authorizationHeader": "[REDACTED]",
"chatApiUrl": "string",
"datarobotKey": "string",
"inputType": "CSV",
"modelType": "TEXT_GENERATION",
"predictionApiUrl": "string"
},
"deploymentId": "string",
"deploymentName": "string",
"errorMessage": "string",
"errorResolution": [
"ootbMetricName"
],
"expectedResponseColumnName": "string",
"id": "string",
"modelId": "string",
"moderationConfiguration": {
"guardConditions": [
{
"comparand": 0,
"comparator": "greaterThan"
}
],
"intervention": {
"action": "block",
"message": "string"
}
},
"name": "string",
"playgroundId": "string",
"predictionTimeout": 0,
"promptColumnName": "string",
"responseColumnName": "string",
"targetColumnName": "string",
"tenantId": "string",
"useCaseId": "string",
"userId": "string",
"userName": "string",
"validationStatus": "TESTING"
}
SidecarModelMetricValidationResponse
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| citationsPrefixColumnName | any | true | The column name prefix the custom model uses for citation inputs. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| creationDate | string(date-time) | true | The creation date of the custom model validation (ISO 8601 formatted). | |
| deploymentAccessData | any | true | The parameters used for accessing the deployment. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | DeploymentAccessData | false | Add authorization_header to avoid breaking change to API. |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| deploymentId | string | true | The ID of the custom model deployment. | |
| deploymentName | any | false | The name of the custom model deployment. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| errorMessage | any | false | The error message associated with the validation error (if the validation failed). |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| errorResolution | any | false | The error type associated with the insight error status and error message as an indicator of what fields needs to be edited if any. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | [InsightErrorResolution] | false | [Error type linking directly to the field name that is related to the error.] |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| expectedResponseColumnName | any | true | The name of the column the custom model uses for expected response text input. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| id | string | true | The ID of the custom model validation. | |
| modelId | string | true | The ID of the model used in the deployment. | |
| moderationConfiguration | any | false | The moderation configuration associated with the sidecar model metric. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | ModerationConfigurationWithoutID | false | Moderation Configuration associated with an insight. |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| name | string | true | The name of the validated custom model. | |
| playgroundId | any | true | The ID of the playground associated with the sidecar model metric validation. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| predictionTimeout | integer | true | The timeout in seconds for the prediction API used in this custom model validation. | |
| promptColumnName | any | true | The name of the column the custom model uses for prompt text input. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| responseColumnName | any | true | The name of the column the custom model uses for response text input. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| targetColumnName | string | true | The name of the column the custom model uses for prediction output. | |
| tenantId | string(uuid4) | true | The ID of the tenant the custom model validation belongs to. | |
| useCaseId | any | true | The ID of the use case associated with the validated custom model. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| userId | string | true | The ID of the user that created this custom model validation. | |
| userName | any | false | The name of the user that created this custom model validation. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| validationStatus | CustomModelValidationStatus | true | The status of the custom model validation. |
SupportedDeploymentType
"TEXT_GENERATION"
SupportedDeploymentType
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| SupportedDeploymentType | string | false | The type of the target output a DataRobot deployment produces. |
Enumerated Values¶
| Property | Value |
|---|---|
| SupportedDeploymentType | [TEXT_GENERATION, VECTOR_DATABASE, UNSTRUCTURED, REGRESSION, MULTICLASS, BINARY, NOT_SUPPORTED] |
SyntheticEvaluationDatasetGenerationRequest
{
"datasetName": "string",
"language": "string",
"llmId": "azure-openai-gpt-3.5-turbo",
"llmSettings": {
"maxCompletionLength": 0,
"systemPrompt": "string",
"temperature": 0,
"topP": 0
},
"vectorDatabaseId": "string"
}
SyntheticEvaluationDatasetGenerationRequest
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| datasetName | any | false | The name to use for the generated dataset. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | maxLength: 255 |
none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| language | any | false | The language to use for the generated dataset. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| llmId | LanguageModelTypeId | true | The ID of the LLM to use for synthetic dataset generation. | |
| llmSettings | any | false | If specified, uses these LLM settings for the prompt and updates the settings of the corresponding chat or LLM blueprint to use these LLM settings. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | CommonLLMSettings | false | The settings that are available for all non-custom LLMs. |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | CustomModelLLMSettings | false | The settings that are available for custom model LLMs. |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| vectorDatabaseId | any | false | The ID of the vector database to use for synthetic dataset generation. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
SyntheticEvaluationDatasetGenerationResponse
{
"datasetId": "string",
"promptColumnName": "string",
"responseColumnName": "string"
}
SyntheticEvaluationDatasetGenerationResponse
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| datasetId | string | true | The ID of the created dataset. | |
| promptColumnName | string | true | The name of the dataset column containing the prompt text. | |
| responseColumnName | string | true | The name of the dataset column containing the response text. |
ValidationError
{
"loc": [
"string"
],
"msg": "string",
"type": "string"
}
ValidationError
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| loc | [anyOf] | true | none |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | string | false | none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | integer | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| msg | string | true | none | |
| type | string | true | none |
VectorDatabaseRetrievers
"SINGLE_LOOKUP_RETRIEVER"
VectorDatabaseRetrievers
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| VectorDatabaseRetrievers | string | false | The method used to retrieve relevant chunks from the vector database. |
Enumerated Values¶
| Property | Value |
|---|---|
| VectorDatabaseRetrievers | [SINGLE_LOOKUP_RETRIEVER, CONVERSATIONAL_RETRIEVER, MULTI_STEP_RETRIEVER] |
VectorDatabaseSettings
{
"addNeighborChunks": false,
"maxDocumentsRetrievedPerPrompt": 1,
"maxTokens": 1,
"retriever": "SINGLE_LOOKUP_RETRIEVER"
}
VectorDatabaseSettings
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| addNeighborChunks | boolean | false | Add neighboring chunks to those that the similarity search retrieves, such that when selected, search returns i, i-1, and i+1. | |
| maxDocumentsRetrievedPerPrompt | any | false | The maximum number of chunks to retrieve from the vector database. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | integer | false | maximum: 10 minimum: 1 |
none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| maxTokens | any | false | The maximum number of tokens to retrieve from the vector database. |
anyOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | integer | false | maximum: 51200 minimum: 1 |
none |
or
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| » anonymous | null | false | none |
continued
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| retriever | VectorDatabaseRetrievers | false | The method used to retrieve relevant chunks from the vector database. |
Updated March 25, 2025
Was this page helpful?
Great! Let us know what you found helpful.
What can we do to improve the content?
Thanks for your feedback!