Skip to content

On-premise users: click in-app to access the full platform documentation for your version of DataRobot.

Deploy an LLM from the playground

Use an LLM playground in a Use Case to create an LLM blueprint. Set the blueprint configuration, specifying the base LLM and, optionally, a system prompt and vector database. After testing and tuning the responses, the blueprint is ready for registration and deployment.

You can create a text generation custom model by sending the LLM blueprint to the Registry's model workshop. The generated custom model automatically implements the OpenAI chat completion API, which is particularly useful for building conversational applications.

Follow the steps below to add the LLM blueprint to the model workshop:

  1. In a Use Case, from the Playgrounds tab, click the playground containing the LLM you want to register as a blueprint.

  2. In the playground, compare LLMs to determine which LLM blueprint to send to the model workshop, then, do either of the following:

    • In the Comparison panel, on the LLM blueprints tab, click the Actions menu , and then click Send to model workshop.

    • In the chat comparison window, on the blueprint's header, click LLM blueprint actions , and then click Send to model workshop.

  3. In the Send to model workshop modal, select up to twelve evaluation metrics (and any configured moderations).

  4. Next, select any Compliance tests to send. Then, click Send to model workshop:

    When you select compliance tests, they are sent to the model workshop and included as part of the custom model registration. They are also included in any generated compliance documentation.

    Compliance tests in the model workshop

    The selected compliance tests are linked to the custom model in the model workshop by the LLM_TEST_SUITE_ID runtime parameter. If you modify the custom model code significantly in the model workshop, set the LLM_TEST_SUITE_ID runtime parameter to None to avoid running compliance documentation intended for the original model on the modified model.

  5. In the lower-right corner of the LLM playground, notifications appear as the LLM is queued and registered. When notification of the registration's completion appears, click Go to the model workshop:

    The LLM blueprint opens in the Registry's model workshop as a custom model with the Text Generation target type:

  6. On the Assemble tab, in the Runtime Parameters section, configure the key-value pairs required by the LLM, including the LLM service's credentials and other details. To add these values, click the edit icon next to the available runtime parameters.

    To configure Credential type Runtime Parameters, first, add the credentials required for the LLM you're deploying to the Credentials Management page of the DataRobot platform:

    Microsoft-hosted LLMs: Azure OpenAI GPT-3.5 Turbo, Azure OpenAI GPT-3.5 Turbo 16k, Azure OpenAI GPT-4, Azure OpenAI GPT-4 32k, Azure OpenAI GPT-4 Turbo, and Azure OpenAI GPT-4o

    Credential type: API Token (not Azure)

    The required Runtime Parameters are:

    Key Description
    OPENAI_API_KEY Select the API Token credential, created on the Credentials Management page, for the Azure OpenAI LLM API endpoint.
    OPENAI_API_BASE Enter the URL for the Azure OpenAI LLM API endpoint.
    OPENAI_API_DEPLOYMENT_ID Enter the name of the Azure OpenAI deployment of the LLM, chosen when deploying the LLM to your Azure environment. For more information, see the Azure OpenAI documentation on how to Deploy a model. The default deployment name suggested by DataRobot matches the ID of the LLM in Azure OpenAI (for example, gpt-35-turbo). Modify this parameter if your Azure OpenAI deployment is named differently.
    OPENAI_API_VERSION Enter the Azure OpenAI API version to use for this operation, following the YYYY-MM-DD or YYYY-MM-DD-preview format (for example, 2023-05-15). For more information on the supported versions, see the Azure OpenAI API reference documentation.
    PROMPT_COLUMN_NAME Enter the prompt column name from the input .csv file. The default column name is promptText.

    Amazon-hosted LLM: Amazon Titan, Anthropic Claude 2.1, Anthropic Claude 3 Haiku, Anthropic Claude 3 Opus, and Anthropic Claude 3 Sonnet

    Credential type: AWS

    The required Runtime Parameters are:

    Key Description
    AWS_ACCOUNT Select an AWS credential, created on the Credentials Management page, for the AWS account.
    AWS_REGION Enter the AWS region of the AWS account. The default is us-west-1.
    PROMPT_COLUMN_NAME Enter the prompt column name from the input .csv file. The default column name is promptText.

    Google-hosted LLM: Google Bison, Google Gemini 1.5 Flash, and Google Gemini 1.5 Pro

    Credential type: Google Cloud Service Account

    The required Runtime Parameters are:

    Key Description
    GOOGLE_SERVICE_ACCOUNT Select a Google Cloud Service Account credential created on the Credentials Management page.
    GOOGLE_REGION Enter the GCP region of the Google service account. The default is us-west-1.
    PROMPT_COLUMN_NAME Enter the prompt column name from the input .csv file. The default column name is promptText.
  7. In the Settings section, ensure Network access is set to Public.

  8. After you complete the custom model assembly configuration, you can test the model or create new versions. DataRobot recommends testing custom LLMs before deployment.

  9. Next, click Register model, provide the registered model or version details, and then click Register model again to add the custom LLM to the Registry.

    The registered model version opens in the Model directory.

  10. From the Model directory, in the upper-right corner of the registered model version panel, click Deploy and configure the deployment settings.

    For more information on the deployment functionality available for generative models, see Monitoring support for generative models.

For more information on this process, see the Playground deployment considerations.


Updated November 12, 2024