DataRobot experimentation plugin¶

The DataRobot experimentation plugin (dr xp) exposes a local dashboard during agentic application development. The dashboard runs on port 8090 by default and organizes observability into tabs. The Assist, Entities, Tracing, and Logs tabs in the left sidebar are on by default. Additional tabs, such as Evaluation, are enabled with dr xp command-line flags, and some also require an optional component in the project; Choose your path lists them.

The landing page depends on the entity configuration. With no entity configured, the dashboard opens on the Assist tab; with an entity configured, it opens on the Tracing tab. When the Assist tab is disabled (see Assist tab), a start without an entity opens on the Entities tab instead. See Start without an entity.

The plugin uses the same OpenTelemetry standard that carries telemetry across the rest of the stack. Local traces follow the same tracing model as deployed agents on the DataRobot platform, so local inspection reflects what the agent actually did during development without deploying to the platform or switching to a separate UI tab.

Access the dashboard¶

Complete these steps to open the local experimentation dashboard:

1. Install the CLI—curl https://cli.datarobot.com/install | sh; verify with dr self version. See Install the CLI and authenticate.
2. Authenticate—dr auth login.
3. Install the plugin—dr plugin install xp; verify with dr plugin list. See Install the experimentation plugin.
4. Create or open a project—dr start from an Agentic Starter template; confirm DATAROBOT_USE_CASE_ID in pulumi_config.json. See Set up the App Framework project.
5. Start the agent and dashboard—Run dr run dev from the project root. To launch the dashboard separately, start the agent with task agent:dev in one terminal, then run dr xp in another; the dashboard reads the use case ID from pulumi_config.json when present. See Start the experimentation dashboard.
6. Open the GUI—browse to http://localhost:8090. See Open the GUI in a browser.
7. Confirm traces—send a test message to the agent, refresh the Tracing tab. See Confirm end-to-end behavior.

For batch evaluation workflows, also install the evaluation component and set DR_EXPERIMENT_ENABLE_EVALUATION=true in .env. See Batch agent evaluation.

Choose your path¶

Preview capabilities are functional during development but still change between releases; their pages list the current limitations.

All paths share the same setup: install the CLI, install the xp plugin, set up an App Framework project, and open the GUI.

Assist tab¶

The Assist tab embeds an OpenCode coding assistant wired to the DataRobot LLM gateway. The assistant is scoped to the directory dr xp was launched from, so it answers questions about the agent project before an entity is selected.

The tab is on by default and requires the dr opencode plugin. When the plugin is missing, dr xp installs it at startup; set DR_EXPERIMENT_ASSIST_NO_INSTALL=1 to skip the automatic install. The assistant requires DATAROBOT_API_TOKEN and DATAROBOT_ENDPOINT to reach the LLM gateway; dr auth login sets both.

To disable the tab, pass --enable-assist=false or set DR_EXPERIMENT_ENABLE_ASSIST=0.

Start without an entity¶

--entity-id is optional. When no entity is configured, dr xp opens on the Assist tab, and the Entities tab lists the Use Cases, deployments, custom applications, and workloads you can select. Entity-scoped tabs, such as Tracing, stay disabled until you select an entity.

The entity ID resolves in the following order: the --entity-id flag, the DR_EXPERIMENT_ENTITY_ID environment variable, the .dr-xp.yaml config file, DATAROBOT_USE_CASE_ID in pulumi_config.json, then DATAROBOT_DEFAULT_USE_CASE. When an entity is configured through any of these sources, the dashboard opens on the Tracing tab instead.

The Entities tab is on by default. To disable it, pass --enable-entities=false or set DR_EXPERIMENT_ENABLE_ENTITIES=0; --entity-id is then required.

Local development stack¶

Agentic Starter templates run several services during local development. The commands and ports referenced throughout this page assume this stack:

Ports 8090, 8080, and 5173 are fixed in Agentic Starter templates. The agent port (8842) is set during the dr start wizard.

Quick start¶

<USE_CASE_ID> below is your App Framework project's use case ID—see Prerequisites. It is optional; a bare dr xp opens on the Assist tab, and the Entities tab lists the entities to choose from.

# Install the plugin
dr plugin install xp

# Show available options
dr xp --help

# Start without an entity; pick one later from the Entities tab
dr xp

# Start with a use case (opens on the Tracing tab)
dr xp --entity-id <USE_CASE_ID>

# Enable batch evaluation workflows in the GUI
dr xp --enable-evaluation --entity-id <USE_CASE_ID>

# Enable trace analysis (trace analysis controls on the **Tracing** page)
dr xp --enable-trace-analysis --entity-id <USE_CASE_ID>

# Start with a deployment on a custom port
dr xp --entity-type deployment --entity-id <DEPLOYMENT_ID> --port 8091

When dr run dev runs from an Agentic Starter project, the local tracing dashboard starts automatically on port 8090. The bundled dashboard starts with traces only unless DR_EXPERIMENT_ENABLE_EVALUATION=true is set in .env.

Access the GUI¶

The experimentation plugin serves a web UI from a local HTTP server. The CLI installs and launches the plugin; the browser connects to the server URL after startup completes. With no entity configured, the GUI opens on the Assist tab; with an entity configured, it opens on the Tracing tab.

Prerequisites¶

Before starting the dashboard, confirm the following:

After confirming the prerequisites, complete the following sections in order (or jump straight to a step using the links in Access the dashboard above).

Install the CLI and authenticate¶

1. Install the DataRobot CLI:

curl https://cli.datarobot.com/install | sh

1. Verify the installation:

dr self version

1. Authenticate with DataRobot:

dr auth login

The plugin requires authentication. The CLI passes DATAROBOT_ENDPOINT and DATAROBOT_API_TOKEN to the plugin process at launch.

Install the experimentation plugin¶

Install the xp plugin once per machine:

dr plugin install xp

Verify discovery:

dr plugin list

Confirm the list includes xp with a path under ~/.config/datarobot/plugins/xp/ (Linux and macOS) or the equivalent Windows config directory.

Set up the App Framework project¶

1. Create or open an Agentic Starter project:

   dr start
   

2. Confirm pulumi_config.json at the project root contains a use case ID:

   "DATAROBOT_USE_CASE_ID": "<USE_CASE_ID>"
   

3. (Optional) To run batch evaluation workflows in the GUI, add the evaluation component:

   dr component add https://github.com/datarobot-community/af-component-evaluation .
   

4. From the evaluation component directory (for example evaluations/), install dependencies:

   task install
   

See Batch agent evaluation for prerequisites and setup details.

Configure the dashboard¶

Set options with flags, environment variables, or a project-level .dr-xp.yaml file. Flag values take precedence over environment variables, then config files, then pulumi_config.json.

The following table summarizes the most common settings:

Example project-level config file:

# .dr-xp.yaml (project root)
entity_type: experiment_container
port: 8090
# enable_evaluation: true   # or set DR_EXPERIMENT_ENABLE_EVALUATION=true in .env

For Agentic Starter projects, add evaluation support to .env:

# .env (project root) — enables Evaluation tab with dr run dev / task infra:dev
DR_EXPERIMENT_ENABLE_EVALUATION=true
# Optional: Trace analysis controls on the Tracing page (preview)
DR_EXPERIMENT_ENABLE_TRACE_ANALYSIS=true

For judge-based evaluations, also set credentials in .env:

DATAROBOT_API_TOKEN=<API_TOKEN>
DATAROBOT_ENDPOINT=https://app.datarobot.com/api/v2

Judge-free benchmarks run without judge credentials. See Batch agent evaluation.

Start the local agent¶

The dashboard visualizes traces from agent activity. Batch evaluations require a running agent endpoint.

Start the full development stack from the project root:

dr run dev

This typically starts the agent on http://localhost:8842/v1 and may also launch the dashboard on port 8090. See Quick start for the flag/env var that enables the Evaluation tab.

To start services individually:

task agent:dev      # Agent on port 8842.
task infra:dev      # Dashboard on port 8090.

When task dev runs, dr xp may start from both agent:dev and infra:dev. The message Already running at http://127.0.0.1:8090 is expected.

Start the experimentation dashboard¶

Choose one of the following options.

Standalone start¶

From the App Framework project root:

dr xp

The dashboard reads the Use Case ID from pulumi_config.json, when present. To point to a different entity, pass --entity-id <USE_CASE_ID>, or pick one from the Entities tab after startup. Add --enable-evaluation when the Evaluation tab is needed. See Enable batch evaluation workflows.

Infra task (Agentic Starter projects)¶

task infra:dev

This runs dr xp --plain-output without --enable-evaluation. To enable batch evaluation workflows, set DR_EXPERIMENT_ENABLE_EVALUATION=true in the project .env.

Bundled with full dev¶

dr run dev

Expected startup output:

Starting server...
Local Experimentation running on http://127.0.0.1:8090 for experiment_container:<USE_CASE_ID>

If a server is already listening on the port, the CLI prints:

Already running at http://127.0.0.1:8090 — no new server started.

The server runs in the foreground. Stop it with Ctrl+C.

Verify the server (optional)¶

Confirm the dashboard responds:

curl -s http://127.0.0.1:8090/api/info

Example response with evaluation enabled:

{"entity_type":"experiment_container","entity_id":"<USE_CASE_ID>","features":["traces","evaluation"]}

When evaluation is disabled, features contains "traces" only and evaluation API routes return 404.

Open the GUI in a browser¶

Open one of the following URLs:

The page title is Experimentation UI. Available tabs depend on enabled features:

Confirm end-to-end behavior¶

1. Send a test message to the agent (for example at http://localhost:5173).
2. Refresh the dashboard. A new trace appears in the Tracing tab. See Local tracing.
3. (Optional) When evaluation is enabled, open the Evaluation tab, select a pipeline, dataset, and agent endpoint (http://localhost:8842/v1), then start a run. See Batch agent evaluation.

GUI access troubleshooting¶

Configuration reference¶

Use these options when non-default entity types, ports, or environment-based configuration is required.

Command options¶

dr xp --entity-type [TYPE] --entity-id [ID] --port [PORT] [--enable-evaluation] [--enable-trace-analysis] [--plain-output]

The server runs in the foreground and logs to the terminal. Stop it with Ctrl+C. For a step-by-step walkthrough, see Access the GUI.

• --entity-id is optional. Without one, the value falls back to DATAROBOT_USE_CASE_ID in pulumi_config.json, then to DATAROBOT_DEFAULT_USE_CASE. When no source sets an entity, the dashboard opens on the Assist tab; pick an entity from the Entities tab. See Start without an entity.
• --entity-type defaults to experiment_container (use case). Also accepts the alias use_case.
• --enable-assist controls the Assist tab (on by default). Disable with --enable-assist=false. See Assist tab.
• --enable-entities controls the Entities tab (on by default). Disable with --enable-entities=false; --entity-id is then required.
• --enable-evaluation enables batch evaluation workflows in the GUI when the evaluation component is present in the project. Without this flag, the Evaluation tab is unavailable.
• --enable-trace-analysis adds trace analysis controls to the Tracing page for traces recorded by a Tensile-instrumented agent (preview). Run dr xp from the agent project directory. See Trace analysis.
• --plain-output streams logs directly without the persistent TUI panel (used by task infra:dev).
• Value precedence: flag > environment variable > config file (.dr-xp.yaml) > pulumi_config.json > DATAROBOT_DEFAULT_USE_CASE > default.

Run dr xp --help for the full flag list.

Environment variables¶

Add these variables to the .env file as needed.

• DR_EXPERIMENT_ENTITY_TYPE (default: experiment_container)
• DR_EXPERIMENT_ENTITY_ID
• DATAROBOT_DEFAULT_USE_CASE (used as the entity ID when no other source sets one)
• DR_EXPERIMENT_PORT (default: 8090)
• DR_EXPERIMENT_ENABLE_ASSIST (default: true; set to 0 to disable the Assist tab)
• DR_EXPERIMENT_ENABLE_ENTITIES (default: true; set to 0 to disable the Entities tab)
• DR_EXPERIMENT_ASSIST_PORT (default: 4099)
• DR_EXPERIMENT_ASSIST_NO_INSTALL (set to true to skip the automatic dr opencode plugin install)
• DR_EXPERIMENT_ENABLE_EVALUATION (set to true for evaluation workflows)
• DR_EXPERIMENT_ENABLE_TRACE_ANALYSIS (set to true for trace analysis)
• DR_EXPERIMENT_TENSILE_NO_INSTALL (set to true to keep dr xp from installing tensile the first time you analyze a trace)

Supported entity types¶

See also¶

• Local tracing—inspect OpenTelemetry spans during development.
• Batch agent evaluation—score agent responses on a fixed dataset.
• Trace analysis—analyze recorded trajectories with Tensile and OpenCode (preview).
• Implement tracing—add custom spans and attributes.
• DataRobot CLI getting started—install and configure the CLI.
• af-component-evaluation repository—evaluation component source and reference docs.