Validate custom model LLMs¶
If you have a custom model deployment that serves as an LLM, you need to validate it before using it in LLM blueprints. Validation ensures the deployment can properly handle LLM requests. Use datarobot.CustomModelLLMValidation to validate your deployments.
Start LLM validation¶
Validate a deployment as a custom model LLM. Specify the following:
deployment_id: The ID of the deployment to validate.name: An optional name for the validation.prompt_column_name: The name of the column the deployed model uses for prompt text input (required for non-chat API deployments).target_column_name: The name of the column the deployed model uses for prediction output (required for non-chat API deployments).chat_model_id: The model ID to specify when calling the chat completion API (for deployments that support the chat completion API).wait_for_completion: If set to True, the code will wait for the validation job to complete before returning results.
import datarobot as dr
deployment = dr.Deployment.get(deployment_id)
use_case = dr.UseCase.get(use_case_id)
validation = dr.genai.CustomModelLLMValidation.create(
deployment_id=deployment.id,
use_case=use_case.id,
name="My Custom LLM",
prompt_column_name="prompt",
target_column_name="response",
wait_for_completion=True
)
validation
For deployments that support the chat completion API, use chat_model_id:
validation = dr.genai.CustomModelLLMValidation.create(
deployment_id=deployment.id,
name="Chat API LLM",
chat_model_id="model-id-from-deployment",
wait_for_completion=True
)
Check validation status¶
If you don't wait for completion, poll for status using datarobot.CustomModelLLMValidation.get():
validation = dr.genai.CustomModelLLMValidation.create(
deployment_id=deployment.id,
name="My Custom LLM",
prompt_column_name="prompt",
target_column_name="response",
wait_for_completion=False
)
while validation.validation_status == "TESTING":
import time
time.sleep(5)
validation = dr.genai.CustomModelLLMValidation.get(validation.id)
if validation.validation_status == "PASSED":
print("Validation passed!")
print(f"Access data: {validation.deployment_access_data}")
else:
print(f"Validation failed: {validation.error_message}")
Use a validated LLM in a blueprint¶
Once validation passes, use the validation ID in an LLM blueprint:
validation = dr.genai.CustomModelLLMValidation.get(validation_id)
use_case = dr.UseCase.get(use_case_id)
playground = dr.genai.Playground.create(
name="Custom LLM Playground",
use_case=use_case.id
)
blueprint = dr.genai.LLMBlueprint.create(
playground=playground.id,
name="Custom Model Blueprint",
llm="custom-model",
llm_settings={
"system_prompt": "You are a helpful assistant.",
"validation_id": validation.id
}
)
blueprint
Update validation settings¶
Modify validation parameters using datarobot.CustomModelLLMValidation.update():
validation = dr.genai.CustomModelLLMValidation.get(validation_id)
validation.update(
name="Updated Custom LLM Name",
prediction_timeout=60
)