# LLM gateway model configuration > LLM gateway model configuration - Add approved LLMs to the LLM gateway to leverage agentic > functionality in situations where company policy restricts the LLMs available to employees. This Markdown file sits beside the HTML page at the same path (with a `.md` suffix). It summarizes the topic and lists links for tools and LLM context. Companion generated at `2026-04-24T16:03:56.656959+00:00` (UTC). ## Primary page - [LLM gateway model configuration](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/llm-gateway-config.html): Full documentation for this topic (HTML). ## Sections on this page - [Set up LLM credentials](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/llm-gateway-config.html#set-up-llm-credentials): In-page section heading. - [Via secure configuration](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/llm-gateway-config.html#via-secure-configuration): In-page section heading. - [Directly during installation](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/llm-gateway-config.html#directly-during-installation): In-page section heading. - [Provider-specific configuration](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/llm-gateway-config.html#provider-specific-configuration): In-page section heading. - [Azure OpenAI](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/llm-gateway-config.html#azure-openai): In-page section heading. - [Amazon Bedrock](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/llm-gateway-config.html#amazon-bedrock): In-page section heading. - [Single-tenant SaaS (STS) configuration](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/llm-gateway-config.html#sts-bedrock): In-page section heading. - [Google Vertex AI](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/llm-gateway-config.html#google-vertex-ai): In-page section heading. - [Anthropic](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/llm-gateway-config.html#anthropic): In-page section heading. - [Cerebras](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/llm-gateway-config.html#cerebras): In-page section heading. - [TogetherAI](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/llm-gateway-config.html#togetherai): In-page section heading. ## Related documentation - [Reference documentation](https://docs.datarobot.com/en/docs/reference/index.html): Linked from this page. - [GenAI reference](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/index.html): Linked from this page. - [Azure OpenAI LLM availability](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/llm-availability.html#azure-openai): Linked from this page. ## Documentation content # LLM gateway model configuration > [!NOTE] Note > The ability to add LLMs to the LLM gateway is a function controlled by the install administrator. For multi-tenant SaaS users, contact [DataRobot Support](mailto:support@datarobot.com) or visit the [Support site](https://support.datarobot.com/) to request assistance. A self-managed or single-tenant SaaS org admin can add LLMs to the LLM gateway of their installation, making them available for users. In this way, you can leverage the agentic functionality in situations where, for example, company policy restricts the LLMs available to its employees. DataRobot's GenAI service supports managed LLMs from a variety of providers. To enable these models, refer to the LLM provider documentation for provisioning LLM resources: - Azure OpenAI - Amazon Bedrock - Google Vertex AI - Anthropic - Cerebras - TogetherAI ## Set up LLM credentials Once LLMs are configured on the provider side, set up provider credentials in the DataRobot GenAI service to access the various LLMs. For user-managed credentials (always used for self-managed, and often, single-tenant SaaS installations), best practice suggests using [secure configurations](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/llm-gateway-config.html#via-secure-configuration) in DataRobot for maximum flexibility and security. In certain cases, for instance a single-tenant SaaS environment with DataRobot-managed credentials, an admin can [set up credentials directly](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/llm-gateway-config.html#directly-during-installation) in `values.yaml`. ### Via secure configuration Prior to DataRobot installation, ensure the respective provider is enabled under the `llmGateway` configuration section within the `buzok-llm-gateway` sub-chart, for example: ``` buzok-llm-gateway: enabled: true llmGateway: providers: : enabled: true credentialsSecureConfigName: genai--llm-credentials ``` When configuring, use the following list of supported providers and corresponding LLM credentials "`SecureConfig`" display names: ``` anthropic: credentialsSecureConfigName: genai-anthropic-llm-credentials aws: credentialsSecureConfigName: genai-aws-llm-credentials azure: credentialsSecureConfigName: genai-azure-llm-credentials cerebras: credentialsSecureConfigName: genai-cerebras-llm-credentials google: credentialsSecureConfigName: genai-gcp-llm-credentials togetherai: credentialsSecureConfigName: genai-togetherai-llm-credentials ``` > [!NOTE] Note > You can include the following providers' secure configurations, but they are not yet supported in DataRobot: > > cohere ( > genai-cohere-llm-credentials > ) > openai ( > genai-openai-llm-credentials > ) > groq ( > genai-groq-llm-credentials > ) After installation is complete, provide model credentials to DataRobot from the Secure Configuration page: **Access from NextGen:** From the system admin icon, to the left of the profile icon: [https://docs.datarobot.com/en/docs/images/secure-config-ng.png](https://docs.datarobot.com/en/docs/images/secure-config-ng.png) **Access from Classic:** From the user profile icon, in the APP ADMIN section: [https://docs.datarobot.com/en/docs/images/secure-config-classic.png](https://docs.datarobot.com/en/docs/images/secure-config-classic.png) 1. Click+ Add a secure configuration. 2. Complete the fields to create the configuration. FieldDescriptionSecure configuration typeFrom the dropdown, select the credential provider. For ease in locating provider options, enterGenAIin the search field. Example:[GenAI] Anthropic LLM CredentialsSecure configuration display nameEnter the display name, listed in eachprovider sectionbelow. Example:genai-anthropic-llm-credentialsLLM CredentialsAdd the credentials in JSON format by copying and pasting the JSON—from{to}—from theprovider-specific configuration sectionsbelow. Be sure to update with your actual credentials. 3. ClickSave. The new secure configuration, with credentials, is added to the list of secure configurations. 4. UnderActionsin the configuration table, click the share icon to share the newly created config with the GenAI Admin system user (genai-admin@datarobot.com). This user is automatically created during installation. You must assign either theOwnerorEditorrole to the admin. > [!NOTE] Note > The Generative AI Service caches credentials retrieved from secure configuration for a short period of time. When rotating credentials, it can take a few minutes for the new credentials to be applied. ### Directly during installation > [!NOTE] Note > Using this method, updating credentials requires reinstalling or updating DataRobot. Because the credentials are visible for the person performing the installation, this is considered a less secure and flexible way of setting up credentials for the service. Credentials can be specified directly in the `buzok-llm-gateway` configuration section. Secure configuration won't be used in this case. The example below is how a configuration for AWS credentials; exact keys differ for various providers, see the [provider-specific configuration](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/llm-gateway-config.html#provider-specific-configuration) sections for credentials structure details: ``` buzok-llm-gateway: enabled: true llmGateway: providers: aws: enabled: true credentialsSecureConfigName: "" credentials: endpoints: - region: "" access_key_id: "" secret_access_key: "" ``` ## Provider-specific configuration The following sections describe configuration for supported providers: - Azure OpenAI - Amazon Bedrock - Google Vertex AI - Anthropic - Cerebras - TogetherAI ### Azure OpenAI Azure OpenAI models must be configured with specific deployment names, where deployment name can be: - Equal to the model name, for example, a deployment for the gpt-35-turbo-16k model can be named gpt-35-turbo-16k . - Different from the model name, such as creating a deployment named gpt-4o-2024-11-20 for the model gpt-4o . For a list of deployment names, see the chat model ID on the [Azure OpenAI LLM availability](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/llm-availability.html#azure-openai) page. After configuring an Azure OpenAI LLM deployment, format the credentials using the following JSON structure: ``` { "endpoints": [ { "region": "", "api_type": "azure", "api_base": "https://.openai.azure.com/", "api_version": "2024-10-21", "api_key": "" } ] } ``` When creating a secure configuration with these credentials, use: | Field | Value | | --- | --- | | Secure configuration type | [GenAI] Azure OpenAI LLM Credentials | | Secure configuration display name | genai-azure-llm-credentials | > [!NOTE] Note > Currently, "2024-10-21" is the latest stable version of the Azure API. To use the latest models, you may be required to use a preview version of the API. Refer to the official [Azure API documentation](https://learn.microsoft.com/en-us/azure/ai-services/openai/api-version-lifecycle) for details on Azure Inference API versions. The secure configuration section for Azure should follow the YAML structure outlined below: ``` buzok-llm-gateway: enabled: true llmGateway: providers: azure: enabled: true credentialsSecureConfigName: genai-azure-llm-credentials ``` ### Amazon Bedrock The Generative AI Service supports the various first-party (e.g., `Amazon Nova`) and third-party (e.g., `Anthropic Claude`, `Meta Llama`, and `Mistral`) models from Amazon Bedrock. There are two options of setting up Amazon Bedrock models: - IAM roles - Static AWS credentials Using IAM roles ( [IRSA](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html)) is recommended as it is a more secure option. This is because it uses dynamic short-lived AWS credentials. However, if your Kubernetes cluster is not in the AWS cloud, setting up this authorization mechanism requires setting up Workload Identity Federation. For this reason, the Generative AI Service also supports static AWS credentials. For security reasons, when using static credentials, it is recommended to create a separate AWS user that only has permissions to access AWS Bedrock. After enabling model access in AWS Bedrock, format the credentials using the following JSON structure for using static credentials: ``` { "endpoints": [ { "region": "", "access_key_id": "", "secret_access_key": "", "session_token": null, } ] } ``` Use the following JSON structure when using a role different from IRSA (ensure there is a policy that allows IRSA to assume this role): ``` { "endpoints": [ { "region": "", "role_arn": "" } ] } ``` Using IRSA to access Bedrock is available in 11.1.2 and later. Set `"use_web_identity": true` to use `AssumeRoleWithWebIdentity` from the pod's service account token. ``` { "endpoints": [ { "region": "", "role_arn": "", "use_web_identity": true } ] } ``` Using FIPS-enabled endpoints is available in 11.1.2 and later. Set `"use_fips_endpoint": true` to do that. Note, that model availability may vary between FIPS-enabled and regular endpoints. ``` { "endpoints": [ { "region": "", "role_arn": "", "use_fips_endpoint": true } ] } ``` #### Single-tenant SaaS (STS) configuration For single-tenant SaaS (STS) installations, the Bedrock credential configuration differs from the standard IRSA setup. Use the following JSON structure: ``` { "endpoints": [ { "region": "", "role_arn": "" } ] } ``` Note the following differences from the standard IRSA configuration: - The role_arn value must be the IAM role on the customer's AWS account—not the IAM role currently used by the STS cluster. - Do not include use_web_identity in the configuration. The STS cluster handles authentication through its own role assumption mechanism rather than web identity federation. When creating a secure configuration with these credentials, use: | Field | Value | | --- | --- | | Secure configuration type | [GenAI] AWS Bedrock LLM Credentials | | Secure configuration display name | genai-aws-llm-credentials | The secure configuration section for Amazon Bedrock should follow the YAML structure outlined below: ``` buzok-llm-gateway: enabled: true llmGateway: providers: aws: enabled: true credentialsSecureConfigName: genai-aws-llm-credentials ``` ### Google Vertex AI The Generative AI Service supports multiple models from Google Vertex AI, including first-party Gemini models and third-party Claude/Llama/Mistral models. After provisioning the model, you should receive a JSON file with access credentials. To provide credentials to DataRobot Generative AI, format the contents using the following JSON structure: ``` { "endpoints": [ { "region": "us-central1", "service_account_info": { "type": "service_account", "project_id": "", "private_key_id": "", "private_key": "----- -----\n", "client_email": ".iam.gserviceaccount.com", "client_id": "", "auth_uri": "https://accounts.google.com/o/oauth2/auth", "token_uri": "https://oauth2.googleapis.com/token", "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", "client_x509_cert_url": "https://.iam.gserviceaccount.com", "universe_domain": "googleapis.com" } } ] } ``` When creating a secure configuration with these credentials, use: | Field | Value | | --- | --- | | Secure configuration type | [GenAI] Google VertexAI LLM Credentials | | Secure configuration display name | genai-gcp-llm-credentials | The secure configuration section for Google Vertex AI should follow the YAML structure outlined below: ``` buzok-llm-gateway: enabled: true llmGateway: providers: google: enabled: true credentialsSecureConfigName: genai-gcp-llm-credentials ``` ### Anthropic The Generative AI Service provides first-party integration with Anthropic, giving access to various `Claude` models. To configure access to Anthropic, provide the API key using the following JSON structure: ``` { "endpoints": [ { "region": "us", "api_key": "" } ] } ``` When creating a secure configuration with these credentials, use: | Field | Value | | --- | --- | | Secure configuration type | [GenAI] Anthropic LLM Credentials | | Secure configuration display name | genai-anthropic-llm-credentials | The secure configuration section for Anthropic should follow the YAML structure outlined below: ``` buzok-llm-gateway: enabled: true llmGateway: providers: anthropic: enabled: true credentialsSecureConfigName: genai-anthropic-llm-credentials ``` ### Cerebras The Generative AI Service provides integration with Cerebras, giving access to high-performance inference models. To configure access to Cerebras, sign in to the [Cerebras platform](https://cloud.cerebras.ai/platform/) and create a new API key. Format the credentials using the following JSON structure: ``` { "endpoints": [ { "region": "global", "api_key": "" } ] } ``` When creating a Secure Configuration with these credentials, use: | Field | Value | | --- | --- | | Secure configuration type | [GenAI] Cerebras LLM Credentials | | Secure configuration display name | genai-cerebras-llm-credentials | The secure configuration section for Cerebras should follow the YAML structure outlined below: ``` buzok-llm-gateway: enabled: true llmGateway: providers: cerebras: enabled: true credentialsSecureConfigName: genai-cerebras-llm-credentials ``` ### TogetherAI The Generative AI Service provides integration with TogetherAI, offering access to a wide range of open-source language models. To configure access to TogetherAI, sign in to the [TogetherAI console](https://api.together.ai/) and create a new API key. Format the credentials using the following JSON structure: ``` { "endpoints": [ { "region": "global", "api_key": "" } ] } ``` When creating a Secure Configuration with these credentials, use: | Field | Value | | --- | --- | | Secure configuration type | [GenAI] TogetherAI LLM Credentials | | Secure configuration display name | genai-togetherai-llm-credentials | The secure configuration section for TogetherAI should follow the YAML structure outlined below: ``` buzok-llm-gateway: enabled: true llmGateway: providers: togetherai: enabled: true credentialsSecureConfigName: genai-togetherai-llm-credentials ```