# GenAI feature considerations

> GenAI feature considerations - Things to consider when working with DataRobot GenAI capabilities.

This Markdown file sits beside the HTML page at the same path (with a `.md` suffix). It summarizes the topic and lists links for tools and LLM context.

Companion generated at `2026-05-01T23:10:48.101787+00:00` (UTC).

## Primary page

- [GenAI feature considerations](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/genai-consider.html): Full documentation for this topic (HTML).

## Sections on this page

- [General considerations](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/genai-consider.html#general-considerations): In-page section heading.
- [Vector database considerations](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/genai-consider.html#vector-database-considerations): In-page section heading.
- [Playground considerations](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/genai-consider.html#playground-considerations): In-page section heading.
- [Playground deployment considerations](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/genai-consider.html#playground-deployment-considerations): In-page section heading.
- [Bolt-on Governance API](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/genai-consider.html#bolt-on-governance-api): In-page section heading.
- [LLM evaluation and moderation](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/genai-consider.html#llm-evaluation-and-moderation): In-page section heading.
- [Trial user considerations](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/genai-consider.html#trial-user-considerations): In-page section heading.

## Related documentation

- [Reference documentation](https://docs.datarobot.com/en/docs/reference/index.html): Linked from this page.
- [GenAI reference](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/index.html): Linked from this page.
- [Troubleshooting](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/genai-troubleshooting.html): Linked from this page.
- [supported LLM base models](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/llm-availability.html): Linked from this page.
- [vector databases](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/index.html): Linked from this page.
- [vector database custom model](https://docs.datarobot.com/en/docs/workbench/nxt-registry/nxt-model-workshop/nxt-create-custom-model.html#vector-databases): Linked from this page.
- [custom model resource allocation settings](https://docs.datarobot.com/en/docs/platform/admin/manage-entities/manage-orgs.html#set-custom-model-resource-allocation): Linked from this page.
- [Metadata filtering](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/rag-chatting.html#metadata-filtering): Linked from this page.
- [Playgrounds](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/index.html): Linked from this page.
- [adding a deployed LLM](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/build-llm-blueprints.html#add-a-deployed-llm): Linked from this page.
- [evaluation and moderation](https://docs.datarobot.com/en/docs/workbench/nxt-registry/nxt-model-workshop/nxt-configure-evaluation-moderation.html): Linked from this page.
- [LLM evaluation and moderation](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/playground-eval-metrics.html): Linked from this page.
- [custom metric](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-custom-metrics.html): Linked from this page.
- [NVIDIA NIM](https://docs.datarobot.com/en/docs/agentic-ai/genai-integrations/genai-nvidia-integration.html): Linked from this page.

## Documentation content

# GenAI feature considerations

The following sections describe things to consider when working with generative AI capabilities in DataRobot. Note that as the product continues to develop, some considerations may change. See [Troubleshooting](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/genai-troubleshooting.html) for an overview of common errors and their handling.

> [!NOTE] Considerations for trial users
> See the considerations specific to the [DataRobot free trial](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/genai-consider.html#trial-user-considerations), including [supported LLM base models](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/llm-availability.html).

> [!NOTE] LLM gateway rate limits
> Depending on the configuration, your organization may be subject to rate limits on total number of chat completion calls. If the application returns a message that your maximum has been reached, it will reset in 24 hours. The time to reset is indicated in the error message. To remove the limit, contact your administrator or DataRobot representative to manage your organization's pricing plan.

## General considerations

- If a multilingual dataset exceeds the limit associated with the multilingual model, DataRobot defaults to using thejinaai/jina-embedding-t-en-v1embedding model.
- Deployments created from custom models with training data attached that have extra columns cannot be used unless column filtering is disabled on the custom model.
- When using LLMs that are either BYO or deployed from the playground and require a runtime parameter to point to the endpoint associated with their credentials, be aware of the vendor's model versioning and end-of-life schedules. As a best practice, use only endpoints that are generally available when deploying to production. (Models provided in the playground manage this for you.)
- Note that an API key named[Internal] DR API Access for GenAI Experimentationis created for you when you access the playground or vector database in the UI.
- When using GPUs, BYO embeddings functionality is available for self-managed users only. Note that when many users run vector database creation jobs in parallel, if using BYO embeddings, LLM playground functionality may be degraded until vector database creation jobs complete. Using CPUs with a custom model that contains the embeddings model is supported in all environments.
- Only one aggregated metric job can run at a time. If an aggregation job is currently running, theConfigure aggregationbutton is disabled and the "Aggregation job in progress; try again when it completes" tooltip appears.

## Vector database considerations

The following describes considerations related to [vector databases](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/index.html). See also the [supported dataset types](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/llm-availability.html#supported-dataset-types), below.

> [!NOTE] GPU usage for Self-Managed users
> When working with datasets over 1GB, Self-Managed users who do not have GPU usage configured on their cluster may experience serious delays. Email [DataRobot Support](mailto:support@datarobot.com), or visit the [Support site](https://support.datarobot.com/), for installation guidance.

- Creation:
- Deployment:
- Token budget:
- Chunking:
- Metadata filtering:

## Playground considerations

- Playgroundscan be shared for viewing, and users with editor or owner access can perform additional actions within the shared playground, such as creating blueprints. While non-creators cannot prompt an LLM blueprint in the playground, they can make a copy and submit prompts to that copy.
- You can only prompt LLM blueprints that you created (i.e., in both configuration and comparison view). To see the results of prompting another user’s LLM blueprint in a shared Use Case, copy the blueprint, and then you can chat with the same settings applied.
- Deleted prompts and responses are counted toward daily limits, although only successful prompt/response pairs are counted. Bring-your-own (BYO) LLM calls are not part of the count.
- For self-managed users (STS, VPC, on-premise): The number of prompts per day, across all LLMs, is dependent on your payment plan. For users on the consumption-based pricing plan, the limit is set by your organization's requirements. All others can submit 5,000 LLM prompts per day. Limits for trial users are different, as describedhere.

## Playground deployment considerations

Consider the following when registering and deploying LLMs from the playground:

- Setting API keys through the DataRobot credential management system is supported. Those credentials are accessed as environment variables in a deployment.
- Registration and deployment is supported for:
- The creation of a custom model version from an LLM blueprint associated with a large vector database (500MB+) can be time-consuming. You can leave the workshop while the model is being created and will not lose your progress.

### Bolt-on Governance API

- When using the Bolt-on Governance API with a deployed LLM blueprint, seeLLM availabilityfor the recommended values of themodelparameter. Alternatively, specify a reserved value,model="datarobot-deployed-llm", to let the LLM blueprint select the relevant model ID automatically when calling the LLM provider's services. In Workbench, whenadding a deployed LLMthat implements thechatfunction, the playground uses the Bolt-on Governance API as the preferred communication method. Enter theChat model IDassociated with the LLM blueprint to set themodelparameter for requests from the playground to the deployed LLM. Alternatively, enterdatarobot-deployed-llmto let the LLM blueprint select the relevant model ID automatically when calling the LLM provider's services.
- Configuringevaluation and moderationfor the custom model negates the effect of streaming responses in the chat completion API, since guardrails evaluate the complete response of the LLM and return the response text in one chunk.
- The followingOpenAI parametersare not supported in the Bolt-on Governance API:functions,tool,tool_choice,logprobs,top_logprobs.

## LLM evaluation and moderation

The following describes considerations related to [LLM evaluation and moderation](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/playground-eval-metrics.html):

- You cangenerate synthetic datasetsin both theUIand API. Use GPT-4, if possible, as it best follows the format DataRobot expects for the output format. Otherwise, the LLM might not generate question-answer pairs.
- Metrics:
- Moderations:
- Aggregation:

## Trial user considerations

The following considerations apply only to DataRobot free trial users:

- You can create up to 15 DataRobot-hosted and third-party connected vector databases, computed across multiple Use Cases. Deleted vector databases are included in this count. BYO vector databases are not included in the count.
- You can make 1,000 LLM API calls, where deleted prompts and responses are also counted. However, only successful prompt response pairs are counted.
- Trial users do not have access toNVIDIA NIMor GPUs.