Configure LLM providers with metadata¶

One of the key components of an LLM agent is the underlying LLM provider. DataRobot allows users to connect to virtually any LLM backend for their agentic workflows. LLM connections can be simplified by using the DataRobot LLM gateway or a DataRobot deployment (including NIM deployments). Alternatively, you can connect to any external LLM provider that supports the OpenAI API standard.

DataRobot agent templates provide multiple methods for defining an agent LLM:

• Use the DataRobot LLM gateway as the agent LLM, allowing you to use any model available in the gateway.
• Connect to use a previously-deployed custom model or NIM using the DataRobot API by providing the deployment ID.
• Connect directly to an LLM provider API (such as OpenAI, Anthropic, or Gemini) by providing the necessary API credentials, enabling access to providers supporting a compatible API.

This document focuses on configuring LLM providers using environment variables and Pulumi (infrastructure-level configuration). This approach allows you to switch between different LLM provider configurations without modifying your agent code. The infrastructure configuration files use the build_llm() helper function from datarobot_genai package, which automatically handles deployment detection, gateway configuration, and credential management. This simplifies deployment and credential management by leveraging DataRobot's secure credential system.

Infrastructure-level LLM configuration¶

You can configure LLM providers at the infrastructure level using environment variables and symlinks. This allows you to switch between different LLM provider configurations without modifying your agent code. Both methods create a symbolic link from infra/infra/llm.py to your chosen configuration file in infra/configurations/llm/. The available configuration options include:

Manual symlink¶

Create the symlink manually for explicit control and immediate visibility of your active configuration. This method is recommended for development.

To use this method, navigate to the infra/infra folder and create a symbolic link to your chosen configuration:

cd infra/infra
ln -sf ../configurations/llm/<chosen_configuration> llm.py

Replace <chosen_configuration> with one of the available configuration files (e.g., gateway_direct.py, blueprint_with_llm_gateway.py, etc.).

After creating the symlink, you can optionally edit infra/infra/llm.py to adjust model parameters (temperature, top_p, etc.) or select a specific model.

Environment variable¶

Set the configuration dynamically using an environment variable. The symlink is automatically created when you run Pulumi commands. This method is recommended for deployment and different environments.

To use this method, uncomment the relevant section in your .env file. The following examples from .env.template show how to configure each LLM provider type. Each configuration may require additional environment variables:

# Your DataRobot API token.
# Refer to https://docs.datarobot.com/en/docs/api/api-quickstart/index.html#configure-your-environment for help.
DATAROBOT_API_TOKEN=

# The URL of your DataRobot instance API.
DATAROBOT_ENDPOINT=https://app.datarobot.com/api/v2

# Local endpoint for writer agent component
WRITER_AGENT_ENDPOINT=http://localhost:8842

# The Pulumi stack name to use for this project.
PULUMI_STACK_NAME=agent-application-dev

# If empty, a blank passphrase will be used
PULUMI_CONFIG_PASSPHRASE=123

# If empty, a new use case will be created
DATAROBOT_DEFAULT_USE_CASE=

# If empty, a new execution environment will be created for each agent using the docker_context folder
DATAROBOT_DEFAULT_EXECUTION_ENVIRONMENT="[DataRobot] Python 3.11 GenAI Agents"

# Required: Random string for Web application security. We recommend a long password generated securely such as:
# `python -c "import os, binascii; print(binascii.hexlify(os.urandom(64)).decode('utf-8'))"`
SESSION_SECRET_KEY=

# OAuth configurtion: See README instructions for getting Google and Box OAuth Apps
# GOOGLE_CLIENT_ID=
# GOOGLE_CLIENT_SECRET=

# BOX_CLIENT_ID=
# BOX_CLIENT_SECRET=

# LLM Configuration:
# Agent templates support multiple flexible LLM options including:
# - LLM Gateway Direct (default)
# - External LLM
# - Already Deployed Text Generation model in DataRobot
#
# You can edit the LLM configuration by manually changing which configuration is
# active (recommended option).
# Simply run `ln -sf ../configurations/<chosen_configuration> llm.py`
# from the `infra/infra` folder
#
# If you want to do it dynamically however, you can also set it as a configuration value with:
# INFRA_ENABLE_LLM=<chosen_configuration>
# from the list of options in the infra/configurations/llm folder
# Here are some examples of each of those configuration using the dynamic option described above:

# If you want to use the LLM Gateway direct (default)
# INFRA_ENABLE_LLM=gateway_direct.py

# If you want to choose an existing LLM Deployment in DataRobot
# uncomment and configure these:
# TEXTGEN_DEPLOYMENT_ID=<your_deployment_id>
# INFRA_ENABLE_LLM=deployed_llm.py

# If you want to configure an LLM with an external LLM provider
# like Azure, Bedrock, Anthropic, or VertexAI (or all 4). Here we provide
# an Azure AI example, see:
# https://docs.datarobot.com/en/docs/gen-ai/playground-tools/deploy-llm.html
# for details on other providers and details:
# INFRA_ENABLE_LLM=blueprint_with_external_llm.py
# LLM_DEFAULT_MODEL="azure/gpt-4o"
# OPENAI_API_VERSION='2024-08-01-preview'
# OPENAI_API_BASE='https://<your_custom_endpoint>.openai.azure.com'
# OPENAI_API_DEPLOYMENT_ID='<your deployment_id>'
# OPENAI_API_KEY='<your_api_key>'

# MCP server configurations
MCP_SERVER_PORT=9000
APP_LOG_LEVEL=CRITICAL

When you run task infra:build or task infra:deploy, the system reads INFRA_ENABLE_LLM, automatically creates or updates the symlink to the specified configuration file, and manages credentials through DataRobot's secure credential system.

LLM configuration options¶

Configure the DataRobot LLM gateway¶

The LLM gateway provides a streamlined way to access LLMs proxied via DataRobot. The gateway is available for both cloud and on-premise users.

You can retrieve a list of available models for your account using the following methods:

curl -X GET -H "Authorization: Bearer $DATAROBOT_API_TOKEN" "$DATAROBOT_ENDPOINT/genai/llmgw/catalog/" | jq | grep 'model":'

import datarobot
print("\n".join(datarobot.genai.LLMGatewayCatalog.get_available_models()))

Use multiple models¶

In your agent.py file, the MyAgent class provides an llm() method that accepts a preferred_model parameter. Each agent in your workflow can specify a different model by calling self.llm(preferred_model="model_name"). If preferred_model is not provided, the method falls back to the default model from your configuration. This allows you to optimize for different capabilities—for example, using faster, cost-effective models for planning tasks and more capable models for content generation. The example below uses the CrewAI framework.

In your MyAgent class, you can define agents with different models:

@property
def agent_planner(self) -> Agent:
    """Content Planner agent."""
    return Agent(
        role="Content Planner",
        goal="Plan engaging and factually accurate content on {topic}",
        llm=self.llm(preferred_model="datarobot/azure/gpt-4o-mini"),  # Fast, cost-effective model
        tools=self._mcp_tools,
    )

@property
def agent_writer(self) -> Agent:
    """Content Writer agent."""
    return Agent(
        role="Content Writer",
        goal="Write insightful and factually accurate opinion piece about the topic: {topic}",
        llm=self.llm(preferred_model="datarobot/bedrock/anthropic.claude-sonnet-4-20250514-v1:0"),  # More capable model
        tools=self._mcp_tools,
    )

@property
def agent_editor(self) -> Agent:
    """Editor agent."""
    return Agent(
        role="Editor",
        goal="Edit a given blog post to align with the writing style of the organization.",
        llm=self.llm(),  # Uses default_model from configuration when preferred_model is not specified
        tools=self._mcp_tools,
    )

The default model is defined in your LLM configuration file (e.g., infra/configurations/llm/gateway_direct.py) and is used as a fallback:

default_model: str = "datarobot/azure/gpt-4o-mini"  # Fallback for agents that don't specify a model

With LLM gateway, the default model is automatically configured and rarely needs to be changed since agents can request any available model dynamically.

DataRobot hosted LLM deployments¶

You can easily connect to DataRobot-hosted LLM deployments as an LLM provider for your agents. DataRobot hosted LLMs provide access to moderations, guardrails, and advanced monitoring to help you manage and govern your models. When using a deployed LLM, all agents use the same deployment. The configuration automatically sets the correct model identifier—no manual model configuration needed. You can create LLM deployments in several ways:

• From the DataRobot Playground: Deploy an LLM from the DataRobot Playground
• Hugging Face Models: Host Hugging Face models as LLM deployments

After deployment, copy the Deployment ID and add it to the .env file:

TEXTGEN_DEPLOYMENT_ID=<your_deployment_id>
INFRA_ENABLE_LLM=deployed_llm.py

The default model is automatically configured to match the deployment. All agents will use this deployment.

Configure external LLMs¶

When configured with an external LLM like Azure OpenAI, Amazon Bedrock, etc. all agents in the workflow use the model specified in your configuration. Unlike the LLM gateway, this connection is to a single specific model deployment.

# If you want to configure an LLM with an external LLM provider
# like Azure, Bedrock, Anthropic, or VertexAI (or all 4). Here we provide
# an Azure AI example, see:
# https://docs.datarobot.com/en/docs/gen-ai/playground-tools/deploy-llm.html
# for details on other providers and details:
INFRA_ENABLE_LLM=blueprint_with_external_llm.py
LLM_DEFAULT_MODEL="azure/gpt-4o"
OPENAI_API_VERSION='2024-08-01-preview'
OPENAI_API_BASE='https://<your_custom_endpoint>.openai.azure.com'
OPENAI_API_DEPLOYMENT_ID='<your deployment_id>'
OPENAI_API_KEY='<your_api_key>'

INFRA_ENABLE_LLM=blueprint_with_external_llm.py
AWS_ACCESS_KEY_ID='<your_access_key>'
AWS_SECRET_ACCESS_KEY='<your_secret_key>'
AWS_REGION_NAME='us-east-1'

external_model_id: str = "bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0"

INFRA_ENABLE_LLM=blueprint_with_external_llm.py
GOOGLE_SERVICE_ACCOUNT='<your_service_account_json>'
# or
GOOGLE_APPLICATION_CREDENTIALS='/path/to/service-account.json'
GOOGLE_REGION='us-west1'

external_model_id: str = "vertex_ai/gemini-2.5-pro"

INFRA_ENABLE_LLM=blueprint_with_external_llm.py
ANTHROPIC_API_KEY='<your_api_key>'

external_model_id: str = "anthropic/claude-3-5-sonnet-20241022"

Other LLM providers¶

The template supports 100+ LLM providers through LiteLLM, including:

• Cohere: Configure with COHERE_API_KEY
• Together AI: Configure with TOGETHERAI_API_KEY
• Hugging Face: Configure with appropriate provider credentials
• Ollama: For locally hosted models
• OpenAI Direct: For direct OpenAI API access

For providers not natively supported by the template's configuration files, you can implement direct LiteLLM integration by modifying the agent code and using LLM gateway Direct configuration with custom def llm() implementation.

Direct LiteLLM integration with DataRobot credential management¶

This approach gives you full control over LLM initialization while securely storing credentials in DataRobot's credential management system. The example below illustrates using LangGraph with direct Cohere integration.

Select the LLM gateway direct configuration in the .env file:

INFRA_ENABLE_LLM=gateway_direct.py

Add a credential to model-metadata.yaml (e.g., agent_langgraph/custom_model/model-metadata.yaml):

runtimeParameterDefinitions:
  - fieldName: COHERE_API_KEY
    type: credential
  - fieldName: COHERE_MODEL
    type: string
    defaultValue: "command-r-plus"

Update config.py to load the credential (e.g., agent_langgraph/custom_model/config.py):

from pydantic import Field
class Config(BaseConfig):
    # ... existing fields ...
    cohere_api_key: str
    cohere_model: str = "command-r-plus"

Replace the llm() method in agent_langgraph/custom_model/agent.py:

from langchain_openai import ChatOpenAI
def llm(
    self,
    preferred_model: str | None = None,
    auto_model_override: bool = True,
) -> ChatOpenAI:
    """Returns a ChatOpenAI instance configured to use Cohere via LiteLLM.
    Args:
        preferred_model: The model to use. If None, uses COHERE_MODEL from config.
        auto_model_override: Ignored for direct LiteLLM integration.
    """
    return ChatOpenAI(
        model=self.config.cohere_model,  # Override the model with the configured one
        api_key=self.config.cohere_api_key,
        base_url="https://api.cohere.ai/v1",  # Cohere's OpenAI-compatible endpoint
        timeout=self.timeout,
    )

Next, add Pulumi credential management in infra/configurations/llm/gateway_direct.py:

...existing code
# Create the Cohere credential
cohere_credential = datarobot.ApiTokenCredential(
    resource_name=f"{pulumi.get_project()} Cohere API Key Credential",
    api_token=os.environ.get("COHERE_API_KEY"),
)
# Update the runtime parameters arrays
app_runtime_parameters = [
    datarobot.ApplicationSourceRuntimeParameterValueArgs(
        key="COHERE_API_KEY",
        type="credential",
        value=cohere_credential.id,
    ),
    datarobot.ApplicationSourceRuntimeParameterValueArgs(
        key="COHERE_MODEL",
        type="string",
        value=os.environ.get("COHERE_MODEL", "command-r-plus"),
    ),
]
custom_model_runtime_parameters = [
    datarobot.CustomModelRuntimeParameterValueArgs(
        key="COHERE_API_KEY",
        type="credential",
        value=cohere_credential.id,
    ),
    datarobot.CustomModelRuntimeParameterValueArgs(
        key="COHERE_MODEL",
        type="string",
        value=os.environ.get("COHERE_MODEL", "command-r-plus"),
    ),
]

Finally, set environment variables in the .env file:

INFRA_ENABLE_LLM=gateway_direct.py
COHERE_API_KEY='<your_api_key>'
COHERE_MODEL='command-r-plus'

This approach gives you full control over LLM initialization and credential management while still using DataRobot's secure credential storage. The credentials are managed as runtime parameters and never hard coded in your application code.

Deploy configuration changes¶

After changing any LLM configuration (updating .env variables, switching configuration files, or modifying configuration parameters), you must deploy the updated configuration before running your agent:

task infra:deploy

This command updates the deployment with the latest configuration and ensures your agent connects to the proper LLM with the correct credentials. Run this command:

• After switching between LLM configurations
• After updating credentials or API keys
• After modifying model parameters (temperature, top_p, etc.)
• Before running your agent locally with the new configuration

Advanced LLM configuration¶

In addition to the .env file settings, you can directly edit the respective llm.py configuration file to make additional changes such as:

• Temperature settings (applied to the deployed blueprint)
• top_p values (applied to the deployed blueprint)
• Timeout configurations (useful for GPU-based models)
• Other model-specific parameters