GenAI feature considerations¶

The following sections describe things to consider when working with generative AI capabilities in DataRobot. Note that as the product continues to develop, some considerations may change. See Troubleshooting for an overview of common errors and their handling.

Trial users: See the considerations specific to the DataRobot free trial, including supported LLM base models.

Availability¶

The sections below describe support for the various elements (LLMs, embeddings, data types, sharing) that are part of GenAI model creation. See also:

• Specific LLM deprecation and retirement dates.
• brief descriptions of each available LLM.

LLM availability¶

Note the following when working with LLMs and the LLM gateway:

The following tables list LLMs by provider.

• Amazon Bedrock
• Azure OpenAI
• Google VertexAI

• Anthropic

In the tables below, which lists LLM availability by provider, note the following:

Amazon Bedrock¶

Anthropic¶

Azure OpenAI¶

Google VertexAI¶

Deprecated and retired LLMs¶

In the quickly advancing agentic AI landscape, LLMs are constantly improving, with new versions replacing older models. To address this, DataRobot's LLM deprecation process marks LLMs and LLM blueprints with a badge to indicate upcoming changes. The goal is to help protect experiments and deployments from unexpected removal of provider support. Note that retirement dates are set by the provider and are subject to change.

Badges for deprecated LLMs are shown in the LLM blueprint creation panel:

Or if built, affected LLM blueprints are marked with a warning or notice, with dates provided on hover:

• When an LLM is in the deprecation process, support for the LLM will be removed in 90 days. Badges and warnings are present, but functionality is not restricted.

• When retired, assets created from the retired model are still viewable, but the creation of new assets is prevented. Retired LLMs cannot be used in single or comparison prompts.

Some evaluation metrics, for example faithfulness and correctness, use an LLM in their configuration. For those, messages are displayed when viewing or configuring the metrics, as well as in the prompt response.

If an LLM has been deployed, because DataRobot does not have control over the credentials used for the underlying LLM, the deployment will fail to return predictions. If this happens, replace the deployed LLM with a new model.

The following LLMs are currently, or will soon be, deprecated and removed:

If an LLM has been deployed, because DataRobot does not have control over the credentials used for the underlying LLM, the deployment will fail to return predictions. If this happens, replace the deployed LLM with a new model.

Embeddings availability¶

DataRobot supports the following types of embeddings for encoding data; all are transformer models trained on a mixture of supervised and unsupervised data.

* Input Dimension = max_sequence_length

 Supported languages:

      "Afrikaans",
        "Amharic",
        "Arabic",
        "Assamese",
        "Azerbaijani",
        "Belarusian",
        "Bulgarian",
        "Bengali",
        "Breton",
        "Bosnian",
        "Catalan",
        "Czech",
        "Welsh",
        "Danish",
        "German",
        "Greek",
        "English",
        "Esperanto",
        "Spanish",
        "Estonian",
        "Basque",
        "Persian",
        "Finnish",
        "French",
        "Western Frisian",
        "Irish",
        "Scottish Gaelic",
        "Galician",
        "Gujarati",
        "Hausa",
        "Hebrew",
        "Hindi",
        "Croatian",
        "Hungarian",
        "Armenian",
        "Indonesian",
        "Icelandic",
        "Italian",
        "Japanese",
        "Javanese",
        "Georgian",
        "Kazakh",
        "Khmer",
        "Kannada",
        "Korean",
        "Kurdish",
        "Kyrgyz",
        "Latin",
        "Lao",
        "Lithuanian",
        "Latvian",
        "Malagasy",
        "Macedonian",
        "Malayalam",
        "Mongolian",
        "Marathi",
        "Malay",
        "Burmese",
        "Nepali",
        "Dutch",
        "Norwegian",
        "Oromo",
        "Oriya",
        "Panjabi",
        "Polish",
        "Pashto",
        "Portuguese",
        "Romanian",
        "Russian",
        "Sanskrit",
        "Sindhi",
        "Sinhala",
        "Slovak",
        "Slovenian",
        "Somali",
        "Albanian",
        "Serbian",
        "Sundanese",
        "Swedish",
        "Swahili",
        "Tamil",
        "Telugu",
        "Thai",
        "Tagalog",
        "Turkish",
        "Uyghur",
        "Ukrainian",
        "Urdu",
        "Uzbek",
        "Vietnamese",
        "Xhosa",
        "Yiddish",
        "Chinese"

Sharing and permissions¶

The following table describes GenAI component-related user permissions. All roles (Consumer, Editor, Owner) refer to the user's role in the Use Case; access to various function are based on the Use Case roles. For example, because sharing is handled on the Use Case level, you cannot share only a vector database (vector databases do not define any sharing rules).

Supported dataset types¶

When uploading datasets for use in creating a vector database, the supported formats are either .zip or .csv. Two columns are mandatory for the files—document and document_file_path. Additional metadata columns, up to 50, can be added for use in filtering during prompt queries. Note that for purposes of metadata filtering, document_file_path is displayed as source.

For .zip files, DataRobot processes the file to create a .csv version that contains text columns (document) with an associated reference ID (document_file_path) column. All content in the text column is treated as strings. The reference ID column is created automatically when the .zip is uploaded. All files should be either in the root of the archive or in a single folder inside an archive. Using a folder tree hierarchy is not supported.

Regarding file types, DataRobot provides the following support:

• .txt documents

• PDF documents

  • Text-based PDFs are supported.
  • To extract text from image-based PDFs, you must use the Python API client. Extracting text from image-based PDFs via the GUI is not fully supported.
  • Documents with mixed image and text content are supported; only the text is parsed.
  • Single documents consisting only of images result in empty documents and are ignored.
  • Datasets consisting of image-only documents (no text) are not processable.

• .docx documents are supported but older .doc format is not supported.

• .md documents, and the .markdown variant, are supported.

• A mix of all supported document types in a single dataset is allowed.

General considerations¶

• If a multilingual dataset exceeds the limit associated with the multilingual model, DataRobot defaults to using the jinaai/jina-embedding-t-en-v1 embedding model.

• Deployments created from custom models with training data attached that have extra columns cannot be used unless column filtering is disabled on the custom model.

• When using LLMs that are either BYO or deployed from the playground and require a runtime parameter to point to the endpoint associated with their credentials: Be aware of the vendor's model versioning and end-of-life schedules. As a best practice, use only endpoints that are generally available when deploying to production. (Models provided in the playground manage this for you.)

• Note that an API key named [Internal] DR API Access for GenAI Experimentation is created for you when you access the playground or vector database in the UI.

• When using GPUs, BYO embeddings functionality is available for self-managed users only. Note that when many users run vector database creation jobs in parallel, if using BYO embeddings, LLM playground functionality may be degraded until vector database creation jobs complete. Using CPUs with a custom model that contains the embeddings model is supported in all environments.

• Only one aggregated metric job can run at a time. If an aggregation job is currently running, the Configure aggregation button is disabled and the "Aggregation job in progress; try again when it completes" tooltip appears.

Playground considerations¶

• Playgrounds can be shared for viewing, and users with editor or owner access can perform additional actions within the shared playground, such as creating blueprints. While non-creators cannot prompt an LLM blueprint in the playground, they can make a copy and submit prompts to that copy.

• You can only prompt LLM blueprints that you created (i.e., in both configuration and comparison view). To see the results of prompting another user’s LLM blueprint in a shared Use Case, copy the blueprint, and then you can chat with the same settings applied.

• Each user can submit 5000 LLM prompts per day across all LLMs, where deleted prompts and responses are also counted. However, only successful prompt response pairs are counted and bring-your-own (BYO) LLM calls are not part of the count. Limits for trial users are different, as described here.

Vector database considerations¶

The following describes considerations related to vector databases. See also the supported dataset types, below.

• Creation:

  • By default, DataRobot uses the Facebook AI Similarity Search (FAISS) vector database.

  • For internal (FAISS) vector databases, a 10GB dataset limit is applied during vector database creation and resulting vector database asset size (text after extraction).

  • The following apply to Pinecone or Elasticsearch connected (external) vector databases:

    • They support up to 100GB.
    • You cannot create a version (child) from a connected vector database.
    • You cannot create a connected vector database from a parent vector database.
    • You can add data to a connected vector database “in place” without creating a new version.

• Deployment:

  • When deploying vector databases for 10GB datasets, the vector database custom model may require more RAM than the default of 4GB. The custom model resource allocation settings are configurable by the organization administrator.

• Token budget:

  • When determining the number of contexts to retrieve from the vector database, DataRobot allocates 3/4 of the excess token budget (the context size for the LLM) to retrieved documents and the rest to chat history (if applicable).

  • The token budget is comprised of system prompt, user prompt, and max completion length. The excess token budget is context size - (max completion length + system prompt + user prompt).

  • If there is no chat history, the whole excess budget is used for document retrieval. Similarly, if there is no vector database, excess budget is used for history.

• Chunking:

  • Vector database creation with semantic chunking can fail when individual documents in the dataset contain very large texts. The exact limits are not known, but if you experience the error, use recursive chunking instead.

• Metadata filtering:

  • Metadata filtering is only supported in RAG playgrounds.

  • Metadata filtering only supports exact pattern matching (no partial strings or relative expressions).

  • When multiple strings are entered, DataRobot applies an implicit AND. No other operators are supported.

  • Vector databases created before the introduction of metadata filtering do not support this feature. To use filtering with them, create a version from the original and configure the LLM blueprint to use the new vector database instead.

  • The following are internal column names and should not be used to define metadata column: chunk_id, start_index, page, similarity_score, pagebreak_indices, content, _doc_vector, and chunk_size.

  • For purposes of metadata filtering, the document_file_path column name is displayed as source.

  • Metadata filtering for BYO vector databases, like all BYO functionality, requires additional configuration. Because the BYO component must be a standalone drop-in replacements for DataRobot internal vector databases, it must implement the complete vector database functionality: handle an input dataframe containing columns for the query and search parameters k, filter and add_neighbor_chunks, and return the matching most similar documents, including potential metadata in an unstructured format.

See also supported dataset types.

Playground deployment considerations¶

Consider the following when registering and deploying LLMs from the playground:

• Setting API keys through the DataRobot credential management system is supported. Those credentials are accessed as environment variables in a deployment.

• Registration and deployment is supported for:

  • All base LLMs in the playground.

  • LLMs with vector databases.

• The creation of a custom model version from an LLM blueprint associated with a large vector database (500MB+) can be time-consuming. You can leave the workshop while the model is created and will not lose your progress.

Bolt-on Governance API¶

• When using the Bolt-on Governance API with a deployed LLM blueprint, see LLM availability for the recommended values of the model parameter. Alternatively, specify a reserved value, model="datarobot-deployed-llm", to let the LLM blueprint select the relevant model ID automatically when calling the LLM provider's services. In Workbench, when adding a deployed LLM that implements the chat function, the playground uses the Bolt-on Governance API as the preferred communication method. Enter the Chat model ID associated with the LLM blueprint to set the model parameter for requests from the playground to the deployed LLM. Alternatively, enter datarobot-deployed-llm to let the LLM blueprint select the relevant model ID automatically when calling the LLM provider's services.

• Configuring evaluation and moderation for the custom model negates the effect of streaming responses in the chat completion API, since guardrails evaluate the complete response of the LLM and return the response text in one chunk.

• The following OpenAI parameters are not supported in the Bolt-on Governance API: functions, tool, tool_choice, logprobs, top_logprobs.

LLM evaluation and moderation¶

The following describes considerations related to LLM evaluation and moderation:

• You can generate synthetic datasets in both the UI and API. Use GPT-4, if possible, as it best follows the format DataRobot expects for output format. Otherwise, the LLM might not generate question-answer pairs.

• Metrics:

  • For NeMo metrics, the blocked_terms.txt file is shared between the prompt and response metrics. As a result, modifying blocked_terms.txt in the prompt metric will modify it for the response metric and vice versa.

  • All metrics can be copied and duplicates can exist, with the following exception: Only two NeMo stay on topic metrics can exist in a custom model, one for input and one for output (NeMo metric prompt and one NeMo response metric).

  • The Faithfulness and Correctness metrics will return 0 if the LLM you chose does not produce the correct output format.

  • When transferring metrics to a production environment, if the guard for a metric is not enabled in the playground it is transferred as a report guard to production.

• Moderations:

  • The Report moderation method triggers a warning for an evaluation metric when the guard condition is met. The Report and block moderation method triggers a warning and displays a moderation message, defined for each metric. The Replace moderation method is not available in the playground.

  • When a playground evaluation metric and moderation configuration is sent to the workshop, the evaluation metric is created as a custom metric, including the guard condition (if enabled). Moderation settings do not need to be configured for a playground evaluation metric to create a custom metric and log the base metric scores during the export to the workshop.

  • When a playground evaluation metric and moderation configuration is sent to the workshop, the moderation configuration is applied after the first custom model version is created. As a result, any evaluation metric exported from the playground includes a second custom model version containing the moderation configuration. This additional step must be complete, and the second version of the custom model must be available, before the custom model is ready to be used with moderations.

• Aggregation:

  • In the evaluation dataset aggregation table, the Current configuration only toggle compares only those metrics sharing the configuration currently displayed in LLM tab of the Configuration sidebar. Old aggregation records may not contain the LLM blueprint configurations used and will default to the LLM blueprint configuration migration that occurred in September 2024. All new aggregation records moving forward track the LLM blueprint configuration used for computation.

  • If multiple LLM blueprints are part of a request, DataRobot computes aggregation blueprint-by-blueprint, sequentially, to avoid LLM limit issues.

Trial user considerations¶

The following considerations apply only to DataRobot free trial users:

• You can create up to 15 vector databases, computed across multiple Use Cases. Deleted vector databases are included in this count.

• You can make 1000 LLM API calls, where deleted prompts and responses are also counted. However, only successful prompt response pairs are counted.