# Custom model validation
URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/genai/custom-model-validation.html

# Validate custom model LLMs

If you have a custom model deployment that serves as an LLM, you need to validate it before using it in LLM blueprints. Validation ensures the deployment can properly handle LLM requests. Use [datarobot.CustomModelLLMValidation](https://docs.datarobot.com/en/docs/api/reference/sdk/gen-llm-generation.html#datarobot.models.genai.custom_model_llm_validation.CustomModelLLMValidation) to validate your deployments.

## Start LLM validation

Validate a deployment as a custom model LLM. Specify the following:

- deployment_id : The ID of the deployment to validate.
- name : An optional name for the validation.
- prompt_column_name : The name of the column the deployed model uses for prompt text input (required for non-chat API deployments).
- target_column_name : The name of the column the deployed model uses for prediction output (required for non-chat API deployments).
- chat_model_id : The model ID to specify when calling the chat completion API (for deployments that support the chat completion API).
- wait_for_completion : If set to True, the code will wait for the validation job to complete before returning results.

```
import datarobot as dr
deployment = dr.Deployment.get(deployment_id)
use_case = dr.UseCase.get(use_case_id)
validation = dr.genai.CustomModelLLMValidation.create(
    deployment_id=deployment.id,
    use_case=use_case.id,
    name="My Custom LLM",
    prompt_column_name="prompt",
    target_column_name="response",
    wait_for_completion=True
)
validation
```

For deployments that support the chat completion API, use `chat_model_id`:

```
validation = dr.genai.CustomModelLLMValidation.create(
    deployment_id=deployment.id,
    name="Chat API LLM",
    chat_model_id="model-id-from-deployment",
    wait_for_completion=True
)
```

## Check validation status

If you don't wait for completion, poll for status using [datarobot.CustomModelLLMValidation.get()](https://docs.datarobot.com/en/docs/api/reference/sdk/gen-llm-generation.html#datarobot.models.genai.custom_model_llm_validation.CustomModelLLMValidation.get):

```
validation = dr.genai.CustomModelLLMValidation.create(
    deployment_id=deployment.id,
    name="My Custom LLM",
    prompt_column_name="prompt",
    target_column_name="response",
    wait_for_completion=False
)
while validation.validation_status == "TESTING":
    import time
    time.sleep(5)
    validation = dr.genai.CustomModelLLMValidation.get(validation.id)

if validation.validation_status == "PASSED":
    print("Validation passed!")
    print(f"Access data: {validation.deployment_access_data}")
else:
    print(f"Validation failed: {validation.error_message}")
```

## Use a validated LLM in a blueprint

Once validation passes, use the validation ID in an LLM blueprint:

```
validation = dr.genai.CustomModelLLMValidation.get(validation_id)
use_case = dr.UseCase.get(use_case_id)
playground = dr.genai.Playground.create(
    name="Custom LLM Playground",
    use_case=use_case.id
)
blueprint = dr.genai.LLMBlueprint.create(
    playground=playground.id,
    name="Custom Model Blueprint",
    llm="custom-model",
    llm_settings={
        "system_prompt": "You are a helpful assistant.",
        "validation_id": validation.id
    }
)
blueprint
```

## Update validation settings

Modify validation parameters using [datarobot.CustomModelLLMValidation.update()](https://docs.datarobot.com/en/docs/api/reference/sdk/gen-llm-generation.html#datarobot.models.genai.custom_model_llm_validation.CustomModelLLMValidation.update):

```
validation = dr.genai.CustomModelLLMValidation.get(validation_id)
validation.update(
    name="Updated Custom LLM Name",
    prediction_timeout=60
)
```

---
