GenAI feature considerations¶

When working with generative AI capabilities in DataRobot, consider the following. Note that as the product continues to develop, some considerations may change.

Trial users: See the considerations specific to the DataRobot free trial, including supported LLM base models.

Availability¶

The following sections describe support for the various elements (LLMs, embeddings, data types, sharing) that are part of GenAI model creation.

LLM availability¶

The following table describes the availability of LLMs:

* Available for trial users.
† Due to EU regulations, Claude 3 Opus model access is disabled for Cloud users on the EU platform.

Embeddings availability¶

DataRobot supports the following types of embeddings for encoding data; all are transformer models trained on a mixture of supervised and unsupervised data.

* Input Dimension = max_sequence_length

 Supported languages:

      "Afrikaans",
        "Amharic",
        "Arabic",
        "Assamese",
        "Azerbaijani",
        "Belarusian",
        "Bulgarian",
        "Bengali",
        "Breton",
        "Bosnian",
        "Catalan",
        "Czech",
        "Welsh",
        "Danish",
        "German",
        "Greek",
        "English",
        "Esperanto",
        "Spanish",
        "Estonian",
        "Basque",
        "Persian",
        "Finnish",
        "French",
        "Western Frisian",
        "Irish",
        "Scottish Gaelic",
        "Galician",
        "Gujarati",
        "Hausa",
        "Hebrew",
        "Hindi",
        "Croatian",
        "Hungarian",
        "Armenian",
        "Indonesian",
        "Icelandic",
        "Italian",
        "Japanese",
        "Javanese",
        "Georgian",
        "Kazakh",
        "Khmer",
        "Kannada",
        "Korean",
        "Kurdish",
        "Kyrgyz",
        "Latin",
        "Lao",
        "Lithuanian",
        "Latvian",
        "Malagasy",
        "Macedonian",
        "Malayalam",
        "Mongolian",
        "Marathi",
        "Malay",
        "Burmese",
        "Nepali",
        "Dutch",
        "Norwegian",
        "Oromo",
        "Oriya",
        "Panjabi",
        "Polish",
        "Pashto",
        "Portuguese",
        "Romanian",
        "Russian",
        "Sanskrit",
        "Sindhi",
        "Sinhala",
        "Slovak",
        "Slovenian",
        "Somali",
        "Albanian",
        "Serbian",
        "Sundanese",
        "Swedish",
        "Swahili",
        "Tamil",
        "Telugu",
        "Thai",
        "Tagalog",
        "Turkish",
        "Uyghur",
        "Ukrainian",
        "Urdu",
        "Uzbek",
        "Vietnamese",
        "Xhosa",
        "Yiddish",
        "Chinese"

Sharing and permissions¶

The following table describes GenAI component-related user permissions. All roles (Consumer, Editor, Owner) refer to the user's role in the Use Case; access to various function are based on the Use Case roles. For example, because sharing is handled on the Use Case level, you cannot share only a vector database (vector databases do not define any sharing rules).

Supported dataset types¶

When uploading datasets for use in creating a vector database, the supported formats are either .zip or .csv. Two columns are mandatory for the files—document and document_file_path. Additional metadata columns, up to 50, can be added for use in filtering during prompt queries. Note that for purposes of metadata filtering, document_file_path is displayed as source.

For .zip files, DataRobot processes the file to create a .csv version that contains text columns (document) with an associated reference ID (document_file_path) column. All content in the text column is treated as strings. The reference ID column is created automatically when the .zip is uploaded. All files should be either in the root of the archive or in a single folder inside an archive. Using a folder tree hierarchy is not supported.

Regarding file types, DataRobot provides the following support:

• .txt documents

• PDF documents

  • Text-based PDFs are supported.
  • Image-based PDFs are not fully supported. That is, images are generally ignored but do not lead to errors.
  • Documents with mixed image and text content are supported; only the text is parsed.
  • Single documents consisting only of images result in empty documents and are ignored.
  • Datasets consisting of image-only documents (no text) are not processable.

• .docx documents are supported but older .doc format is not supported.

• .md documents, and the .markdown variant, are supported.

• A mix of all supported document types in a single dataset is allowed.

General considerations¶

• If a multilingual dataset exceeds the limit associated with the multilingual model, DataRobot defaults to using the jinaai/jina-embedding-t-en-v1 embedding model.

• Deployments created from custom models with training data attached that have extra columns cannot be used unless column filtering is disabled on the custom model.

• When using LLMs that are either BYO or deployed from the playground and require a runtime parameter to point to the endpoint associated with their credentials: Be aware of the vendor's model versioning and end-of-life schedules. As a best practice, use only endpoints that are generally available when deploying to production. (Models provided in the playground manage this for you.)

• Note that an API key named [Internal] DR API Access for GenAI Experimentation is created for you when you access the playground or vector database in the UI.

• When using GPUs, BYO embeddings functionality is available for self-managed users only. Note that when many users run vector database creation jobs in parallel, if using BYO embeddings, LLM playground functionality may be degraded until vector database creation jobs complete. Using CPUs with a custom model that contains the embeddings model is supported in all environments.

• Only one aggregated metric job can run at a time. If an aggregation job is currently running, the Configure aggregation button is disabled and the "Aggregation job in progress; try again when it completes" tooltip appears.

Playground considerations¶

• Playgrounds can be shared for viewing, and users with editor or owner access can perform additional actions within the shared playground, such as creating blueprints. While non-creators cannot prompt an LLM blueprint in the playground, they can make a copy and submit prompts to that copy.

• You can only prompt LLM blueprints that you created (i.e., in both configuration and comparison view). To see the results of prompting another user’s LLM blueprint in a shared Use Case, copy the blueprint, and then you can chat with the same settings applied.

• Each user can submit 5000 LLM prompts per day across all LLMs, where deleted prompts and responses are also counted. However, only successful prompt response pairs are counted and bring-your-own (BYO) LLM calls are not part of the count. Limits for trial users are different, as described here.

Vector database considerations¶

The following describes considerations related to vector databases. See also the supported dataset types, below.

• Creation:

  • By default, DataRobot uses the Facebook AI Similarity Search (FAISS) vector database.

  • Globally, a 10GB dataset limit is applied during vector database creation and resulting vector database asset size (text after extraction).

• Deployment:

  • When deploying vector databases for 10GB datasets, the vector database custom model may require more RAM than the default of 4GB. The custom model resource allocation settings are configurable by the organization administrator.

• Token budget:

  • When determining the number of contexts to retrieve from the vector database, DataRobot allocates 3/4 of the excess token budget (the context size for the LLM) to retrieved documents and the rest to chat history (if applicable).

  • The token budget is comprised of system prompt, user prompt, and max completion length. The excess token budget is context size - (max completion length + system prompt + user prompt).

  • If there is no chat history, the whole excess budget is used for document retrieval. Similarly, if there is no vector database, excess budget is used for history.

• Chunking:

  • Vector database creation with semantic chunking can fail when individual documents in the dataset contain very large texts. The exact limits are not known, but if you experience the error, use recursive chunking instead.

• Metadata filtering:

  • Metadata filtering only supports exact pattern matching (no partial strings or relative expressions).

  • When multiple strings are entered, DataRobot applies an implicit AND. No other operators are supported.

  • Vector databases created before the introduction of metadata filtering do not support this feature. To use filtering with them, create a version from the original and configure the LLM blueprint to use the new vector database instead.

  • The following are internal column names and should not be used to define metadata column: chunk_id, start_index, page, similarity_score, pagebreak_indices, content, _doc_vector, and chunk_size.

  • For purposes of metadata filtering, the document_file_path column name is displayed as source.

  • Metadata filtering for BYO vector databases, like all BYO functionality, requires additional configuration. Because the BYO component must be a standalone drop-in replacements for DataRobot internal vector databases, it must implement the complete vector database functionality: handle an input dataframe containing columns for the query and search parameters k, filter and add_neighbor_chunks, and return the matching most similar documents, including potential metadata in an unstructured format.

See also supported dataset types.

Playground deployment considerations¶

Consider the following when registering and deploying LLMs from the playground:

• Setting API keys through the DataRobot credential management system is supported. Those credentials are accessed as environment variables in a deployment.

• Registration and deployment is supported for:

  • All base LLMs in the playground.

  • LLMs with vector databases.

• The creation of a custom model version from an LLM blueprint associated with a large vector database (500+ MB) can be time-consuming. You can leave the model workshop while the model is created and will not lose your progress.

• Streaming responses in the chat completion API are only supported by LLM blueprints that use Azure OpenAI LLMs. Configuring evaluation and moderation for the custom model negates the effect of streaming, since guardrails evaluate the complete response of the LLM and return the response text in one chunk.

• The following OpenAI parameters are not supported in the chat completion API: functions, tool, tool_choice, logprobs, top_logprobs.

LLM evaluation and moderation¶

The following describes considerations related to LLM evaluation and moderation:

• You can generate synthetic datasets in both the UI and API. Use GPT-4, if possible, as it best follows the format DataRobot expects for output format. Otherwise, the LLM might not generate question-answer pairs.

• Metrics:

  • For NeMo metrics, the blocked_terms.txt file is shared between the prompt and response metrics. As a result, modifying blocked_terms.txt in the prompt metric will modify it for the response metric and vice versa.

  • All metrics can be copied and duplicates can exist, with the following exception: Only two NeMo stay on topic metrics can exist in a custom model, one for input and one for output (NeMo metric prompt and one NeMo response metric).

  • The Faithfulness and Correctness metrics will return 0 if the LLM you chose does not produce the correct output format.

  • When transferring metrics to a production environment, if the guard for a metric is not enabled in the playground it is transferred as a report guard to production.

• Moderations:

  • The Report moderation method triggers a warning for an evaluation metric when the guard condition is met. The Report and block moderation method triggers a warning and displays a moderation message, defined for each metric. The Replace moderation method is not available in the playground.

  • When a playground evaluation metric and moderation configuration is sent to the model workshop, the evaluation metric is created as a custom metric, including the guard condition (if enabled). Moderation settings do not need to be configured for a playground evaluation metric to create a custom metric and log the base metric scores during the export to the model workshop.

  • When a playground evaluation metric and moderation configuration is sent to the model workshop, the moderation configuration is applied after the first custom model version is created. As a result, any evaluation metric exported from the playground includes a second custom model version containing the moderation configuration. This additional step must be complete, and the second version of the custom model must be available, before the custom model is ready to be used with moderations.

• Aggregation:

  • In the evaluation dataset aggregation table, the Current configuration only toggle compares only those metrics sharing the configuration currently displayed in LLM tab of the Configuration sidebar. Old aggregation records may not contain the LLM blueprint configurations used and will default to the LLM blueprint configuration migration that occurred in September 2024. All new aggregation records moving forward track the LLM blueprint configuration used for computation.

  • If multiple LLM blueprints are part of a request, DataRobot computes aggregation blueprint-by-blueprint, sequentially, to avoid LLM limit issues.

Trial user considerations¶

The following considerations apply only to DataRobot free trial users:

• You can create up to 15 vector databases, computed across multiple Use Cases. Deleted vector databases are included in this count.

• You can make 1000 LLM API calls, where deleted prompts and responses are also counted. However, only successful prompt response pairs are counted.

See also the section on LLM availability.