Build LLM blueprints¶

LLM blueprints represent the full context for what is needed to generate a response from an LLM, the resulting output is what can then be compared within the playground.

To create an LLM blueprint, select that action from either the LLM blueprints tab or, if it is your first blueprint, the playground welcome screen.

Clicking the create button brings you to the configuration and chatting tools:

You can also create an LLM blueprint by copying an existing blueprint.

Set the configuration¶

The configuration panel is where you define the LLM blueprint. From here:

• Select an LLM and choose settings.
• Select or add a vector database.
• Set the prompting strategy.

LLM selection and settings¶

DataRobot offers a variety of preloaded LLMs, with availability dependent on your cluster and account type.

Alternatively, you can add a deployed LLM to the playground, which, when validated, is added to the Use Case and available to all associated playgrounds. In either case, selecting a base LLM exposes some or all of the following configuration options, dependent on the selection:

Each base LLM has default configuration settings. As a result, the only required selection before starting to chat is to choose the LLM.

Add a deployed LLM¶

To add a custom LLM deployed in DataRobot, click Create LLM blueprint to add a new blueprint to the playground. Then, from the playground's blueprint Configuration panel, in the LLM drop-down, click Add deployed LLM:

In the Add deployed LLM dialog box, enter a deployed LLM Name, then select a DataRobot deployment in the Deployment name drop-down. Depending on if the selected deployment supports the Bolt-on Governance API, configure the following:

After you add a custom LLM and validation is successful, back in the blueprint's Configuration panel, in the LLM drop-down, click Deployed LLM, and then select the Validation ID of the custom model you added:

Finally, you can configure the Vector database and Prompting settings, and click Save configuration to add the blueprint to the playground.

Deprecated and retired LLMs¶

In the quickly advancing GenAI landscape, LLMs are constantly improving, with new versions replacing and disabling older models. To address this, DataRobot's LLM deprecation process marks LLMs with a badge to indicate upcoming changes. The goal is to help protect experiments and deployments from unexpected removal of vendor support.

Badges for deprecated LLMs are shown in the LLM blueprint creation panel:

Or if built, affected LLM blueprints are marked with a warning or notice, with dates provided on hover:

• When an LLM is in the deprecation process, support for the LLM will be removed in two months. Badges and warnings are present, but functionality is not curtailed.

• When retired, assets created from the retired model are still viewable, but the creation of new assets is prevented. Retired LLMs cannot be used in single or comparison prompts.

Some evaluation metrics, for example faithfulness and correctness, use an LLM in their configuration. For those, messages are displayed when viewing or configuring the metrics, as well as in the prompt response.

If an LLM has been deployed, because DataRobot does not have control over the credentials used for the underlying LLM, the deployment will fail to return predictions. If this happens, replace the deployed LLM with a new model.

See the full list of deprecated and retired LLMs on the LLM availability page.

Add a vector database¶

From the Vector database tab, you can optionally select a vector database. The selection identifies a database comprised of a collection of chunks of unstructured text and corresponding text embeddings for each chunk, indexed for easy retrieval. Vector databases are not required for prompting but are used for providing relevant data to the LLM to generate the response. Add a vector database to a playground to experiment with metrics and test responses.

The following table describes the fields of the Vector database tab:

The dropdown

Retriever methods¶

The retriever you select defines how the LLM blueprint searches through, and retrieves, the most relevant chunks from the vector database. They inform which information is provided to the language model. Select one of the following methods:

Use Add Neighbor Chunks to control whether to add neighboring chunks within the vector database to the chunks that the similarity search retrieves. When enabled, the retriever returns i, i-1, and i+1 (for example, if the query retrieves chunk number 42, chunks 41 and 43 are also retrieved).

Notice also that only the primary chunk has a similarity score. This is because the neighbor chunks are added, not calculated, as part of the response.

Also known as context window expansion or context enrichment, this technique includes surrounding chunks adjacent to the retrieved chunk to provide more complete context. Some reasons to enable this include:

• A single chunk may be cut off mid-sentence or may miss important context.
• Related information might span multiple chunks.
• The response might require context from surrounding or chunks.

Enter a value to set the Retrieval limits, which control the number of returned documents.

The value you set for Top K (nearest neighbors) instructs the LLM on how many relevant chunks to retrieve from the vector database. Chunk selection is based on similarity scores. Consider:

• Larger values provide more comprehensive coverage but also require more processing overhead and may include less relevant results.
• Smaller values provide more focused results and faster processing, but may miss relevant information.

Max tokens specifies:

• The maximum size (in tokens) of each text chunk extracted from the dataset when building the vector database.
• The length of the text that is used to create embeddings.
• The size of the citations used in RAG operations.

Set prompting strategy¶

The prompting strategy is where you configure context (chat history) settings and optionally add a system prompt.

Set context state¶

There are two states of context. They control whether chat history is sent with the prompt to include relevant context for responses.

See context-aware chatting for more information.

Set system prompt¶

The system prompt, an optional field, is a "universal" prompt prepended to all individual prompts for this LLM blueprint. It instructs and formats the LLM response. The system prompt can impact the structure, tone, format, and content that is created during the generation of the response.

See an example of system prompt application in the LLM blueprint comparison documentation.

Actions for LLM blueprints¶

The actions available for an LLM blueprint can be accessed from the actions menu next to the name in the left-hand LLM blueprints tab or from LLM blueprint actions in a selected LLM blueprint.

Copy LLM blueprint¶

You can make a copy of an existing LLM blueprint to inherit the settings. Using this approach makes sense when you want to compare slightly different blueprints, duplicate a blueprint for which you are not the owner in a shared playground, or replace a soon-to-expire LLM.

Make a copy in one of two ways:

Change LLM blueprint configuration¶

To change the configuration of an LLM blueprint, choose Configure LLM blueprint from the actions menu in the LLM blueprints tab. The LLM blueprint configuration and chat history display. Change any of the configuration settings and Save configuration

When you make changes to an LLM blueprint, the chat history associated with it, if the configuration is context-aware, is also saved. All the prompts within a chat persist through LLM blueprint changes:

• When you submit a prompt, the history included is everything within the most recent chat context.
• If you switch the LLM blueprint to No context, each prompt is its own chat context.
• If you switch back to Context-aware, that starts a new chat context within the chat.

Note that chats in the configuration view are separate from chats in the Comparison view—the histories don't mingle.