Skip to content

On-premise users: click in-app to access the full platform documentation for your version of DataRobot.

LLM blueprint comparison

Generative AI > LLM blueprint comparison

LLM blueprint comparison

The playground's Comparison tab allows you to compare LLM blueprints side-by-side. Note that you cannot compare drafts—you must first save the draft as an LLM blueprint to compare.

To use the comparison:

  1. Create two or more LLM blueprints in the playground.
  2. Select the Comparison tab and select up to three LLM blueprints for comparison.
  3. Send a prompt. Each of the blueprints receives the prompt and responds, allowing you to compare responses.

The following provides a simple example of comparing multiple LLM blueprints.

Build blueprints for comparison

This section describes building blueprints specifically for comparison purposes. See the fundamentals of building a blueprint if necessary. See also the note on system prompts.

The following example builds three LLM blueprints, each using a different style of response. Each uses this prompt: Describe the TV show Game of Thrones.

  1. Enter the system prompt Answer using emojis.

  2. Enter the system prompt Answer in the style of a news headline.

  3. Enter the system prompt Answer as a haiku.

Compare LLM blueprints

Note

In contrast to chats with an LLM blueprint, comparison prompts are not history-aware.

When LLM blueprints are built:

  1. Select Comparison and check the box next to the blueprints you want to compare.

  2. Send a prompt (Describe DataRobot). Each LLM blueprint responds in their configured style:

  3. Try different prompts (Describe a fish taco, for example) to identify the LLM that best suits the business use case.

  4. Choose the Info tab to view configuration information for each blueprint:

Changing comparison blueprints

You can add blueprints to the comparison at any time, although the maximum allowed for comparison at one time is three. To add an LLM blueprint, select the checkbox to the left of its name. If three are already selected, remove a current selection first.

The Comparison tab retrieves the comparison history. Because responses have not been returned for the new LLM blueprint, DataRobot provides a button to initiate that action. Click Generate to include the new results.

Consider system prompts

Note that system prompts are not guaranteed to be followed completely, and that wording is very important. For example, consider the comparison using the prompt Answer using emojis (EmojiGPT) and Answer using only emojis (OnlyEmojiGPT):

Updated February 5, 2024