DataRobot APIのリソース > APIリファレンスドキュメント > DataRobot REST API > Develop > Generative AI > Custom Model LLMs (GenAI) > LLM Test Results (GenAI)

LLM Test Results (GenAI)¶

This page outlines the operations, endpoints, parameters, and example requests and responses for the LLM Test Results (GenAI).

GET /api/v2/genai/llmTestResults/¶

List LLM test results.

Code samples¶

# You can also use wget
curl -X GET https://app.datarobot.com/api/v2/genai/llmTestResults/ \
  -H "Accept: application/json" \
  -H "Authorization: Bearer {access-token}"

Parameters

Name	In	Type	Required	Description
llmTestConfigurationId	query	string	false	LLM Test Configuration ID.
llmBlueprintId	query	string	false	LLM Blueprint ID.
llmTestSuiteId	query	string	false	LLM Test Suite ID.
offset	query	integer	false	Skip the specified number of values.
limit	query	integer	false	Retrieve only the specified number of values.

Example responses¶

200 Response

{
  "count": 0,
  "data": [
    {
      "creationDate": "2019-08-24T14:15:22Z",
      "creationUserId": "string",
      "creationUserName": "string",
      "errorMessage": "An unknown error occurred while generating a LLM test result.",
      "errorResolution": "Please validate that the submitted LLM Test Configuration is valid. If so, try submitting a new LLM Test Result. Otherwise contact the DataRobot team for assistance.",
      "executionStatus": "NEW",
      "gradingResult": "PASS",
      "id": "string",
      "insightEvaluationResults": [
        {
          "aggregationType": "average",
          "aggregationValue": 0,
          "chatId": "string",
          "chatName": "string",
          "evaluationDatasetConfigurationId": "string",
          "evaluationDatasetName": "string",
          "evaluationName": "string",
          "executionStatus": "NEW",
          "gradingResult": "PASS",
          "id": "string",
          "insightGradingCriteria": {
            "passThreshold": 100
          },
          "lastUpdateDate": "2019-08-24T14:15:22Z",
          "llmTestResultId": "string",
          "maxNumPrompts": 0,
          "metricName": "string",
          "promptSamplingStrategy": "random_without_replacement"
        }
      ],
      "llmBlueprintId": "string",
      "llmBlueprintSnapshot": {
        "description": "string",
        "id": "string",
        "llmId": "azure-openai-gpt-3.5-turbo",
        "llmSettings": {
          "maxCompletionLength": 0,
          "systemPrompt": "string",
          "temperature": 0,
          "topP": 0
        },
        "name": "string",
        "playgroundId": "string",
        "promptType": "CHAT_HISTORY_AWARE",
        "snapshotDate": "2019-08-24T14:15:22Z",
        "vectorDatabaseId": "string",
        "vectorDatabaseSettings": {
          "addNeighborChunks": false,
          "maxDocumentsRetrievedPerPrompt": 1,
          "maxTokens": 1
        }
      },
      "llmTestConfigurationId": "string",
      "llmTestConfigurationName": "string",
      "llmTestGradingCriteria": {
        "passThreshold": 100
      },
      "llmTestSuiteId": "string",
      "passPercentage": 0,
      "useCaseId": "string"
    }
  ],
  "next": "string",
  "previous": "string",
  "totalCount": 0
}

Responses¶

Status	Meaning	Description	Schema
200	OK	Successful Response	ListLLMTestResultResponse
422	Unprocessable Entity	Validation Error	HTTPValidationErrorResponse

To perform this operation, you must be authenticated by means of one of the following methods:

BearerAuth

POST /api/v2/genai/llmTestResults/¶

Create a new LLM test result.

Code samples¶

# You can also use wget
curl -X POST https://app.datarobot.com/api/v2/genai/llmTestResults/ \
  -H "Content-Type: application/json" \
  -H "Accept: application/json" \
  -H "Authorization: Bearer {access-token}" \
  -d '{CreateLLMTestResultRequest}'

Body parameter¶

{
  "llmBlueprintId": "string",
  "llmTestConfigurationId": "string"
}

Parameters

Name	In	Type	Required	Description
body	body	CreateLLMTestResultRequest	true	none

Example responses¶

202 Response

{
  "creationDate": "2019-08-24T14:15:22Z",
  "creationUserId": "string",
  "creationUserName": "string",
  "errorMessage": "An unknown error occurred while generating a LLM test result.",
  "errorResolution": "Please validate that the submitted LLM Test Configuration is valid. If so, try submitting a new LLM Test Result. Otherwise contact the DataRobot team for assistance.",
  "executionStatus": "NEW",
  "gradingResult": "PASS",
  "id": "string",
  "insightEvaluationResults": [
    {
      "aggregationType": "average",
      "aggregationValue": 0,
      "chatId": "string",
      "chatName": "string",
      "evaluationDatasetConfigurationId": "string",
      "evaluationDatasetName": "string",
      "evaluationName": "string",
      "executionStatus": "NEW",
      "gradingResult": "PASS",
      "id": "string",
      "insightGradingCriteria": {
        "passThreshold": 100
      },
      "lastUpdateDate": "2019-08-24T14:15:22Z",
      "llmTestResultId": "string",
      "maxNumPrompts": 0,
      "metricName": "string",
      "promptSamplingStrategy": "random_without_replacement"
    }
  ],
  "llmBlueprintId": "string",
  "llmBlueprintSnapshot": {
    "description": "string",
    "id": "string",
    "llmId": "azure-openai-gpt-3.5-turbo",
    "llmSettings": {
      "maxCompletionLength": 0,
      "systemPrompt": "string",
      "temperature": 0,
      "topP": 0
    },
    "name": "string",
    "playgroundId": "string",
    "promptType": "CHAT_HISTORY_AWARE",
    "snapshotDate": "2019-08-24T14:15:22Z",
    "vectorDatabaseId": "string",
    "vectorDatabaseSettings": {
      "addNeighborChunks": false,
      "maxDocumentsRetrievedPerPrompt": 1,
      "maxTokens": 1
    }
  },
  "llmTestConfigurationId": "string",
  "llmTestConfigurationName": "string",
  "llmTestGradingCriteria": {
    "passThreshold": 100
  },
  "llmTestSuiteId": "string",
  "passPercentage": 0,
  "useCaseId": "string"
}

Responses¶

Status	Meaning	Description	Schema
202	Accepted	Successful Response	LLMTestResultResponse
422	Unprocessable Entity	Validation Error	HTTPValidationErrorResponse

To perform this operation, you must be authenticated by means of one of the following methods:

BearerAuth

DELETE /api/v2/genai/llmTestResults/{llmTestResultId}/¶

Delete an existing LLM test result.

Code samples¶

# You can also use wget
curl -X DELETE https://app.datarobot.com/api/v2/genai/llmTestResults/{llmTestResultId}/ \
  -H "Accept: application/json" \
  -H "Authorization: Bearer {access-token}"

Parameters

Name	In	Type	Required	Description
llmTestResultId	path	string	true	The ID of the LLM Test Result to delete.

Example responses¶

422 Response

{
  "detail": [
    {
      "loc": [
        "string"
      ],
      "msg": "string",
      "type": "string"
    }
  ]
}

Responses¶

Status	Meaning	Description	Schema
204	No Content	Successful Response	None
422	Unprocessable Entity	Validation Error	HTTPValidationErrorResponse

To perform this operation, you must be authenticated by means of one of the following methods:

BearerAuth

GET /api/v2/genai/llmTestResults/{llmTestResultId}/¶

Retrieve an existing LLM test result.

Code samples¶

# You can also use wget
curl -X GET https://app.datarobot.com/api/v2/genai/llmTestResults/{llmTestResultId}/ \
  -H "Accept: application/json" \
  -H "Authorization: Bearer {access-token}"

Parameters

Name	In	Type	Required	Description
llmTestResultId	path	string	true	The ID of the LLM Test Result to retrieve.

Example responses¶

200 Response

{
  "creationDate": "2019-08-24T14:15:22Z",
  "creationUserId": "string",
  "creationUserName": "string",
  "errorMessage": "An unknown error occurred while generating a LLM test result.",
  "errorResolution": "Please validate that the submitted LLM Test Configuration is valid. If so, try submitting a new LLM Test Result. Otherwise contact the DataRobot team for assistance.",
  "executionStatus": "NEW",
  "gradingResult": "PASS",
  "id": "string",
  "insightEvaluationResults": [
    {
      "aggregationType": "average",
      "aggregationValue": 0,
      "chatId": "string",
      "chatName": "string",
      "evaluationDatasetConfigurationId": "string",
      "evaluationDatasetName": "string",
      "evaluationName": "string",
      "executionStatus": "NEW",
      "gradingResult": "PASS",
      "id": "string",
      "insightGradingCriteria": {
        "passThreshold": 100
      },
      "lastUpdateDate": "2019-08-24T14:15:22Z",
      "llmTestResultId": "string",
      "maxNumPrompts": 0,
      "metricName": "string",
      "promptSamplingStrategy": "random_without_replacement"
    }
  ],
  "llmBlueprintId": "string",
  "llmBlueprintSnapshot": {
    "description": "string",
    "id": "string",
    "llmId": "azure-openai-gpt-3.5-turbo",
    "llmSettings": {
      "maxCompletionLength": 0,
      "systemPrompt": "string",
      "temperature": 0,
      "topP": 0
    },
    "name": "string",
    "playgroundId": "string",
    "promptType": "CHAT_HISTORY_AWARE",
    "snapshotDate": "2019-08-24T14:15:22Z",
    "vectorDatabaseId": "string",
    "vectorDatabaseSettings": {
      "addNeighborChunks": false,
      "maxDocumentsRetrievedPerPrompt": 1,
      "maxTokens": 1
    }
  },
  "llmTestConfigurationId": "string",
  "llmTestConfigurationName": "string",
  "llmTestGradingCriteria": {
    "passThreshold": 100
  },
  "llmTestSuiteId": "string",
  "passPercentage": 0,
  "useCaseId": "string"
}

Responses¶

Status	Meaning	Description	Schema
200	OK	Successful Response	LLMTestResultResponse
422	Unprocessable Entity	Validation Error	HTTPValidationErrorResponse

To perform this operation, you must be authenticated by means of one of the following methods:

BearerAuth

Schemas¶

AggregationType

"average"

AggregationType

Properties¶

Name	Type	Required	Restrictions	Description
AggregationType	string	false		The type of the metric aggregation.

Enumerated Values¶

Property	Value
AggregationType	[`average`, `percentYes`, `classPercentCoverage`, `ngramImportance`, `guardConditionPercentYes`]

AggregationValue

{
  "item": "string",
  "value": 0
}

AggregationValue

Properties¶

Name	Type	Required	Restrictions	Description
item	string	true		The name of the item.
value	number	true		The value associated with the item.

CommonLLMSettings

{
  "maxCompletionLength": 0,
  "systemPrompt": "string",
  "temperature": 0,
  "topP": 0
}

CommonLLMSettings

Properties¶

Name	Type	Required	Restrictions	Description
maxCompletionLength	integer¦null	false		Maximum number of tokens allowed in the completion. The combined count of this value and prompt tokens must be below the model's maximum context size, where prompt token count is comprised of system prompt, user prompt, recent chat history, and vector database citations.
systemPrompt	string¦null	false	maxLength: 500000	System prompt guides the style of the LLM response. It is a "universal" prompt, prepended to all individual prompts.
temperature	number¦null	false		Temperature controls the randomness of model output, where higher values return more diverse output and lower values return more deterministic results.
topP	number¦null	false		Top P sets a threshold that controls the selection of words included in the response, based on a cumulative probability cutoff for token selection. For example, 0.2 considers only the top 20% probability mass. Higher numbers return more diverse options for outputs.

CreateLLMTestResultRequest

{
  "llmBlueprintId": "string",
  "llmTestConfigurationId": "string"
}

CreateLLMTestResultRequest

Properties¶

Name	Type	Required	Restrictions	Description
llmBlueprintId	string	true		The LLM Blueprint ID associated with the LLM Test result.
llmTestConfigurationId	string	true		The use case ID associated with the LLM Test result.

CustomModelLLMSettings

{
  "externalLlmContextSize": 128,
  "systemPrompt": "string",
  "validationId": "string"
}

CustomModelLLMSettings

Properties¶

Name	Type	Required	Restrictions	Description
externalLlmContextSize	integer¦null	false	maximum: 128000 minimum: 128	The external LLM's context size, in tokens. This value is only used for pruning documents supplied to the LLM when a vector database is associated with the LLM blueprint. It does not affect the external LLM's actual context size in any way and is not supplied to the LLM.
systemPrompt	string¦null	false	maxLength: 500000	System prompt guides the style of the LLM response. It is a "universal" prompt, prepended to all individual prompts.
validationId	string¦null	false		The validation ID of the custom model LLM.

ExecutionStatus

"NEW"

ExecutionStatus

Properties¶

Name	Type	Required	Restrictions	Description
ExecutionStatus	string	false		Job and entity execution status.

Enumerated Values¶

Property	Value
ExecutionStatus	[`NEW`, `RUNNING`, `COMPLETED`, `REQUIRES_USER_INPUT`, `SKIPPED`, `ERROR`]

GradingResult

"PASS"

GradingResult

Properties¶

Name	Type	Required	Restrictions	Description
GradingResult	string	false		Grading result.

Enumerated Values¶

Property	Value
GradingResult	[`PASS`, `FAIL`]

HTTPValidationErrorResponse

{
  "detail": [
    {
      "loc": [
        "string"
      ],
      "msg": "string",
      "type": "string"
    }
  ]
}

HTTPValidationErrorResponse

Properties¶

Name	Type	Required	Restrictions	Description
detail	[ValidationError]	false		none

InsightEvaluationResultResponse

{
  "aggregationType": "average",
  "aggregationValue": 0,
  "chatId": "string",
  "chatName": "string",
  "evaluationDatasetConfigurationId": "string",
  "evaluationDatasetName": "string",
  "evaluationName": "string",
  "executionStatus": "NEW",
  "gradingResult": "PASS",
  "id": "string",
  "insightGradingCriteria": {
    "passThreshold": 100
  },
  "lastUpdateDate": "2019-08-24T14:15:22Z",
  "llmTestResultId": "string",
  "maxNumPrompts": 0,
  "metricName": "string",
  "promptSamplingStrategy": "random_without_replacement"
}

InsightEvaluationResultResponse

Properties¶

Name	Type	Required	Restrictions	Description
aggregationType	AggregationType¦null	false		Aggregation type.
aggregationValue	any	false		Aggregation value. None indicates that the aggregation failed.

anyOf

Name	Type	Required	Restrictions	Description
» anonymous	number	false		none

or

Name	Type	Required	Restrictions	Description
» anonymous	[AggregationValue]	false		[An individual record in an itemized metric aggregation.]

continued

Name	Type	Required	Restrictions	Description
chatId	string	true		Chat ID.
chatName	string¦null	true	maxLength: 5000	Chat name.
evaluationDatasetConfigurationId	string¦null	false		Evaluation dataset configuration ID.
evaluationDatasetName	string¦null	false	maxLength: 5000	Evaluation dataset name.
evaluationName	string	true	maxLength: 5000	Evaluation name.
executionStatus	ExecutionStatus	false		The execution status of the insight evaluation result.
gradingResult	GradingResult¦null	false		The grading result for this insight evaluation result. If not specified, execution status is not COMPLETED.
id	string	true		Insight evaluation result ID.
insightGradingCriteria	InsightGradingCriteria	true		Insight grading criteria.
lastUpdateDate	string(date-time)	true		Last update date of the insight evaluation result (ISO 8601 formatted).
llmTestResultId	string	true		LLM test result ID this insight evaluation result is associated to.
maxNumPrompts	integer	true		Number of prompts used in evaluation.
metricName	string¦null	false		Name of the metric.
promptSamplingStrategy	PromptSamplingStrategy	true		Prompt sampling strategy for maxNumPrompts.

InsightGradingCriteria

{
  "passThreshold": 100
}

InsightGradingCriteria

Properties¶

Name	Type	Required	Restrictions	Description
passThreshold	integer	true	maximum: 100 minimum: 0	The percentage threshold for Pass result. Greater than or equal to this threshold indicates a Pass.

LLMBlueprintSnapshot

{
  "description": "string",
  "id": "string",
  "llmId": "azure-openai-gpt-3.5-turbo",
  "llmSettings": {
    "maxCompletionLength": 0,
    "systemPrompt": "string",
    "temperature": 0,
    "topP": 0
  },
  "name": "string",
  "playgroundId": "string",
  "promptType": "CHAT_HISTORY_AWARE",
  "snapshotDate": "2019-08-24T14:15:22Z",
  "vectorDatabaseId": "string",
  "vectorDatabaseSettings": {
    "addNeighborChunks": false,
    "maxDocumentsRetrievedPerPrompt": 1,
    "maxTokens": 1
  }
}

LLMBlueprintSnapshot

Properties¶

Name	Type	Required	Description
description	string	true	The description of the LLMBlueprint at the time of snapshotting.
id	string	true	The ID of the LLMBlueprint for which the snapshot was produced.
llmId	LanguageModelTypeId¦null	false	The ID of the LLM selected for this LLM blueprint.
llmSettings	any	false	A key/value dictionary of LLM settings.

anyOf

Name	Type	Required	Restrictions	Description
» anonymous	CommonLLMSettings	false		The settings that are available for all non-custom LLMs.

or

Name	Type	Required	Restrictions	Description
» anonymous	CustomModelLLMSettings	false		The settings that are available for custom model LLMs.

continued

Name	Type	Required	Description
name	string	true	The name of the LLMBlueprint at the time of snapshotting.
playgroundId	string	true	The playground id of the LLMBlueprint.
promptType	PromptType	true	The prompting type of the LLMBlueprint at the time of snapshotting.
snapshotDate	string(date-time)	false	The date when the snapshot was produced.
vectorDatabaseId	string¦null	false	The ID of the vector database linked to this LLM blueprint.
vectorDatabaseSettings	VectorDatabaseSettings¦null	false	A key/value dictionary of vector database settings.

LLMTestGradingCriteria

{
  "passThreshold": 100
}

LLMTestGradingCriteria

Properties¶

Name	Type	Required	Restrictions	Description
passThreshold	integer	true	maximum: 100 minimum: 0	The percentage threshold for Pass results across dataset-insight pairs.

LLMTestResultErrorMessages

"An unknown error occurred while generating a LLM test result."

LLMTestResultErrorMessages

Properties¶

Name	Type	Required	Restrictions	Description
LLMTestResultErrorMessages	string	false		Error messages for LLM Test Result.

Enumerated Values¶

Property Value

LLMTestResultErrorMessages [An unknown error occurred while generating a LLM test result., One or more types for the aggregations produced are invalid for the LLM test result., An error occurred while aggregating the LLM test results., One or more LLM blueprints specified in the LLM Test configuration are invalid and/or not found., One or more evaluation datasets specified in the LLM Test configuration are invalid and/or not found., The LLM Test configuration is invalid and/or not found., An error occurred while uploading the out-of-the-box dataset specified in the LLM Test Result.]

LLMTestResultErrorResolutions

"Please validate that the submitted LLM Test Configuration is valid. If so, try submitting a new LLM Test Result. Otherwise contact the DataRobot team for assistance."

LLMTestResultErrorResolutions

Properties¶

Name	Type	Required	Restrictions	Description
LLMTestResultErrorResolutions	string	false		Error resolutions for LLM Test Result.

Enumerated Values¶

Property Value

LLMTestResultErrorResolutions [Please validate that the submitted LLM Test Configuration is valid. If so, try submitting a new LLM Test Result. Otherwise contact the DataRobot team for assistance., Please ensure that the metrics selected in the LLM Test Configuration utilized for the LLM Test Result can produce a supported aggregation type. After correcting the LLM Test Configuration, try submitting a new LLM Test Result. If the issue persists, contact the DataRobot team for assistance., Please validate that the submitted LLM Test Configuration is valid. Please verify that the dataset evaluations in the LLM Test Configuration are valid for compliance testing. After correcting the LLM Test Configuration, if needed, try submitting a new LLM Test Result. Otherwise contact the DataRobot team for assistance., Please ensure that the LLM blueprints specified in the LLM Test Configuration still exist and are valid. If the issue persists, contact the DataRobot team for assistance., Please ensure that the evaluation datasets specified in the LLM Test Configuration still exist and are valid. If the issue persists, contact the DataRobot team for assistance., Please ensure that the LLM Test Configuration still exists and is valid. If the issue persists, contact the DataRobot team for assistance., Please ensure that the out-of-the-box dataset specified in the LLM Test Result is valid. Please try to re-submit the LLM Test Result as this may have been a temporary issue. If the issue persists, contact the DataRobot team for assistance.]

LLMTestResultResponse

{
  "creationDate": "2019-08-24T14:15:22Z",
  "creationUserId": "string",
  "creationUserName": "string",
  "errorMessage": "An unknown error occurred while generating a LLM test result.",
  "errorResolution": "Please validate that the submitted LLM Test Configuration is valid. If so, try submitting a new LLM Test Result. Otherwise contact the DataRobot team for assistance.",
  "executionStatus": "NEW",
  "gradingResult": "PASS",
  "id": "string",
  "insightEvaluationResults": [
    {
      "aggregationType": "average",
      "aggregationValue": 0,
      "chatId": "string",
      "chatName": "string",
      "evaluationDatasetConfigurationId": "string",
      "evaluationDatasetName": "string",
      "evaluationName": "string",
      "executionStatus": "NEW",
      "gradingResult": "PASS",
      "id": "string",
      "insightGradingCriteria": {
        "passThreshold": 100
      },
      "lastUpdateDate": "2019-08-24T14:15:22Z",
      "llmTestResultId": "string",
      "maxNumPrompts": 0,
      "metricName": "string",
      "promptSamplingStrategy": "random_without_replacement"
    }
  ],
  "llmBlueprintId": "string",
  "llmBlueprintSnapshot": {
    "description": "string",
    "id": "string",
    "llmId": "azure-openai-gpt-3.5-turbo",
    "llmSettings": {
      "maxCompletionLength": 0,
      "systemPrompt": "string",
      "temperature": 0,
      "topP": 0
    },
    "name": "string",
    "playgroundId": "string",
    "promptType": "CHAT_HISTORY_AWARE",
    "snapshotDate": "2019-08-24T14:15:22Z",
    "vectorDatabaseId": "string",
    "vectorDatabaseSettings": {
      "addNeighborChunks": false,
      "maxDocumentsRetrievedPerPrompt": 1,
      "maxTokens": 1
    }
  },
  "llmTestConfigurationId": "string",
  "llmTestConfigurationName": "string",
  "llmTestGradingCriteria": {
    "passThreshold": 100
  },
  "llmTestSuiteId": "string",
  "passPercentage": 0,
  "useCaseId": "string"
}

LLMTestResultResponse

Properties¶

Name	Type	Required	Restrictions	Description
creationDate	string(date-time)	true		LLM test result creation date (ISO 8601 formatted).
creationUserId	string	true		ID of the user that created this LLM test result.
creationUserName	string	true		The name of the user who created this LLM result.
errorMessage	LLMTestResultErrorMessages¦null	false		The error message if the LLM Test Result failed.
errorResolution	LLMTestResultErrorResolutions¦null	false		The error resolution message if the LLM Test Result failed.
executionStatus	ExecutionStatus	true		The LLM Test execution status.
gradingResult	GradingResult¦null	false		The grading result based on the llm test grading criteria. If not specified, execution status is not COMPLETED.
id	string	true		LLM test result ID.
insightEvaluationResults	[InsightEvaluationResultResponse]	true		The Insight evaluation results.
llmBlueprintId	string	true		LLM Blueprint ID.
llmBlueprintSnapshot	LLMBlueprintSnapshot	true		A snapshot of the llm blueprint entity at the time of LLM Test execution.
llmTestConfigurationId	string	true		LLM test configuration ID this LLM result is associated to.
llmTestConfigurationName	string¦null	true	maxLength: 5000 minLength: 1 minLength: 1	Name of the LLM test configuration this LLM result is associated to.
llmTestGradingCriteria	LLMTestGradingCriteria	true		LLM test grading criteria.
llmTestSuiteId	string¦null	false		LLM test suite ID to which the LLM test configuration is associated to.
passPercentage	number¦null	false		The percentage of underlying insight evaluation results that have a PASS grading result. If not specified, execution status is not COMPLETED.
useCaseId	string	true		Use case ID this LLM test result belongs to.

LanguageModelTypeId

"azure-openai-gpt-3.5-turbo"

LanguageModelTypeId

Properties¶

Name	Type	Required	Restrictions	Description
LanguageModelTypeId	string	false		The ID that defines the type of the LLM.

Enumerated Values¶

Property	Value
LanguageModelTypeId	[`azure-openai-gpt-3.5-turbo`, `azure-openai-gpt-3.5-turbo-16k`, `azure-openai-gpt-4`, `azure-openai-gpt-4-32k`, `azure-openai-gpt-4-turbo`, `azure-openai-gpt-4-o`, `amazon-titan`, `anthropic-claude-2`, `anthropic-claude-3-haiku`, `anthropic-claude-3-sonnet`, `anthropic-claude-3-opus`, `google-bison`, `google-gemini-1.5-flash`, `google-gemini-1.5-pro`, `custom-model`]

ListLLMTestResultResponse

{
  "count": 0,
  "data": [
    {
      "creationDate": "2019-08-24T14:15:22Z",
      "creationUserId": "string",
      "creationUserName": "string",
      "errorMessage": "An unknown error occurred while generating a LLM test result.",
      "errorResolution": "Please validate that the submitted LLM Test Configuration is valid. If so, try submitting a new LLM Test Result. Otherwise contact the DataRobot team for assistance.",
      "executionStatus": "NEW",
      "gradingResult": "PASS",
      "id": "string",
      "insightEvaluationResults": [
        {
          "aggregationType": "average",
          "aggregationValue": 0,
          "chatId": "string",
          "chatName": "string",
          "evaluationDatasetConfigurationId": "string",
          "evaluationDatasetName": "string",
          "evaluationName": "string",
          "executionStatus": "NEW",
          "gradingResult": "PASS",
          "id": "string",
          "insightGradingCriteria": {
            "passThreshold": 100
          },
          "lastUpdateDate": "2019-08-24T14:15:22Z",
          "llmTestResultId": "string",
          "maxNumPrompts": 0,
          "metricName": "string",
          "promptSamplingStrategy": "random_without_replacement"
        }
      ],
      "llmBlueprintId": "string",
      "llmBlueprintSnapshot": {
        "description": "string",
        "id": "string",
        "llmId": "azure-openai-gpt-3.5-turbo",
        "llmSettings": {
          "maxCompletionLength": 0,
          "systemPrompt": "string",
          "temperature": 0,
          "topP": 0
        },
        "name": "string",
        "playgroundId": "string",
        "promptType": "CHAT_HISTORY_AWARE",
        "snapshotDate": "2019-08-24T14:15:22Z",
        "vectorDatabaseId": "string",
        "vectorDatabaseSettings": {
          "addNeighborChunks": false,
          "maxDocumentsRetrievedPerPrompt": 1,
          "maxTokens": 1
        }
      },
      "llmTestConfigurationId": "string",
      "llmTestConfigurationName": "string",
      "llmTestGradingCriteria": {
        "passThreshold": 100
      },
      "llmTestSuiteId": "string",
      "passPercentage": 0,
      "useCaseId": "string"
    }
  ],
  "next": "string",
  "previous": "string",
  "totalCount": 0
}

ListLLMTestResultResponse

Properties¶

Name	Type	Required	Description
count	integer	true	The number of records on this page.
data	[LLMTestResultResponse]	true	The list of records.
next	string¦null	true	The URL to the next page, or `null` if there is no such page.
previous	string¦null	true	The URL to the previous page, or `null` if there is no such page.
totalCount	integer	true	The total number of records.

PromptSamplingStrategy

"random_without_replacement"

PromptSamplingStrategy

Properties¶

Name	Type	Required	Restrictions	Description
PromptSamplingStrategy	string	false		The prompt sampling strategy for the evaluation dataset configuration.

Enumerated Values¶

Property	Value
PromptSamplingStrategy	[`random_without_replacement`, `first_n_rows`]

PromptType

"CHAT_HISTORY_AWARE"

PromptType

Properties¶

Name	Type	Required	Restrictions	Description
PromptType	string	false		Determines whether chat history is submitted as context to the user prompt.

Enumerated Values¶

Property	Value
PromptType	[`CHAT_HISTORY_AWARE`, `ONE_TIME_PROMPT`]

ValidationError

{
  "loc": [
    "string"
  ],
  "msg": "string",
  "type": "string"
}

ValidationError

Properties¶

Name	Type	Required	Restrictions	Description
loc	[anyOf]	true		none

anyOf

Name	Type	Required	Restrictions	Description
» anonymous	string	false		none

or

Name	Type	Required	Restrictions	Description
» anonymous	integer	false		none

continued

Name	Type	Required	Restrictions	Description
msg	string	true		none
type	string	true		none

VectorDatabaseSettings

{
  "addNeighborChunks": false,
  "maxDocumentsRetrievedPerPrompt": 1,
  "maxTokens": 1
}

VectorDatabaseSettings

Properties¶

Name	Type	Required	Restrictions	Description
addNeighborChunks	boolean	false		Add neighboring chunks to those that the similarity search retrieves, such that when selected, search returns i, i-1, and i+1.
maxDocumentsRetrievedPerPrompt	integer¦null	false	maximum: 10 minimum: 1	The maximum number of chunks to retrieve from the vector database.
maxTokens	integer¦null	false	maximum: 51200 minimum: 1	The maximum number of tokens to retrieve from the vector database.

更新しました January 24, 2025

このページは役に立ちましたか？

ありがとうございます。どのような点が役に立ちましたか？

より良いコンテンツを提供するには、どうすればよいでしょうか？

アンケートにご協力いただき、ありがとうございました。