# Trace analysis

> Trace analysis - Analyze recorded agent trajectories from the local experimentation dashboard with
> Tensile and OpenCode to find and fix agent mistakes during development.

This Markdown file sits beside the HTML page at the same path (with a `.md` suffix). It summarizes the topic and lists links for tools and LLM context.

Companion generated at `2026-10-23T15:20:58.914487+00:00` (UTC).

## Primary page

- [Trace analysis](https://docs.datarobot.com/en/docs/agentic-ai/cli/local-experimentation/experimentation-trace-analysis.html.md): Full documentation for this topic (Markdown sidecar).

## Sections on this page

- [How it works](https://docs.datarobot.com/en/docs/agentic-ai/cli/local-experimentation/experimentation-trace-analysis.html.md#how-trace-analysis-works): In-page section heading.
- [Enable trace analysis](https://docs.datarobot.com/en/docs/agentic-ai/cli/local-experimentation/experimentation-trace-analysis.html.md#enable-trace-analysis): In-page section heading.
- [Prerequisites](https://docs.datarobot.com/en/docs/agentic-ai/cli/local-experimentation/experimentation-trace-analysis.html.md#trace-analysis-prerequisites): In-page section heading.
- [Instrument the agent](https://docs.datarobot.com/en/docs/agentic-ai/cli/local-experimentation/experimentation-trace-analysis.html.md#instrument-the-agent): In-page section heading.
- [What gets captured](https://docs.datarobot.com/en/docs/agentic-ai/cli/local-experimentation/experimentation-trace-analysis.html.md#what-gets-captured): In-page section heading.
- [Analyze a trace](https://docs.datarobot.com/en/docs/agentic-ai/cli/local-experimentation/experimentation-trace-analysis.html.md#analyze-a-trace): In-page section heading.
- [Analyze all traces on a page](https://docs.datarobot.com/en/docs/agentic-ai/cli/local-experimentation/experimentation-trace-analysis.html.md#analyze-all-traces-on-a-page): In-page section heading.
- [Resume or restart a session](https://docs.datarobot.com/en/docs/agentic-ai/cli/local-experimentation/experimentation-trace-analysis.html.md#resume-or-restart-a-session): In-page section heading.
- [Whichtensileruns](https://docs.datarobot.com/en/docs/agentic-ai/cli/local-experimentation/experimentation-trace-analysis.html.md#which-tensile-runs): In-page section heading.
- [Tensile skills and concepts](https://docs.datarobot.com/en/docs/agentic-ai/cli/local-experimentation/experimentation-trace-analysis.html.md#tensile-skills-and-concepts): In-page section heading.
- [Trace analysis troubleshooting](https://docs.datarobot.com/en/docs/agentic-ai/cli/local-experimentation/experimentation-trace-analysis.html.md#trace-analysis-troubleshooting): In-page section heading.
- [See also](https://docs.datarobot.com/en/docs/agentic-ai/cli/local-experimentation/experimentation-trace-analysis.html.md#see-also): In-page section heading.

## Related documentation

- [Agentic AI](https://docs.datarobot.com/en/docs/agentic-ai/index.html.md): Linked from this page.
- [DataRobot CLI](https://docs.datarobot.com/en/docs/agentic-ai/cli/index.html.md): Linked from this page.
- [DataRobot experimentation plugin](https://docs.datarobot.com/en/docs/agentic-ai/cli/local-experimentation/index.html.md): Linked from this page.
- [Local tracing](https://docs.datarobot.com/en/docs/agentic-ai/cli/local-experimentation/experimentation-tracing.html.md): Linked from this page.
- [OpenCode](https://docs.datarobot.com/en/docs/agentic-ai/cli/opencode-plugin.html.md): Linked from this page.
- [Batch agent evaluation](https://docs.datarobot.com/en/docs/agentic-ai/cli/local-experimentation/experimentation-evaluation.html.md): Linked from this page.
- [Implement tracing](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-tracing-code.html.md): Linked from this page.

## Documentation content

> [!NOTE] Preview
> Trace analysis is in preview and is off by default. Start `dr xp` with `--enable-trace-analysis` (or `DR_EXPERIMENT_ENABLE_TRACE_ANALYSIS=true`) to enable it. Without the flag, none of the controls described on this page appear, and the Tracing page behaves as described in [Local tracing](https://docs.datarobot.com/en/docs/agentic-ai/cli/local-experimentation/experimentation-tracing.html.md). Behavior, prompts, and outputs may change between releases.

On the Tracing page of the local experimentation dashboard, the trace analysis controls hand a trace to a coding agent that reports what your agent did, points out mistakes, and can propose or test fixes to the agent's prompts or code. The controls appear in three places: a ✨ icon beside the status badge of every trace that can be analyzed, an Analyze trace button at the top of an open trace, and an Analyze traces button in the toolbar that analyzes every analyzable trace on the page together.

A trace is the record of one request the agent handled, made up of spans for its LLM calls, tool calls, and other steps. The Tracing page lists one row per trace exported to the Use Case or deployment you passed to `dr xp` with `--entity-id`, from any agent or run. A trajectory is the LLM-call subset of a trace, recorded in full by Tensile so the coding agent can read every prompt and response. Traces without a ✨ were produced by a run that recorded no LLM calls—an agent without Tensile instrumentation, an unsupported LLM client, or a trace that made no LLM call at all.

Analysis works on traces recorded by an agent instrumented with [Tensile](https://pypi.org/project/tensile/): every LLM call the agent makes is stored in its DataRobot trace exactly as sent and received. Clicking a control pulls that recording down as a trajectory, starts an [OpenCode](https://docs.datarobot.com/en/docs/agentic-ai/cli/opencode-plugin.html.md) session with the Tensile agent skills installed, and pre-fills a `/tensile` prompt for you to review and send.

For installation, GUI access, and port configuration, see the [DataRobot experimentation plugin](https://docs.datarobot.com/en/docs/agentic-ai/cli/local-experimentation/index.html.md) landing page. For the Tracing page itself, see [Local tracing](https://docs.datarobot.com/en/docs/agentic-ai/cli/local-experimentation/experimentation-tracing.html.md).

## How it works

Trace analysis connects three components:

| Component | Role |
| --- | --- |
| Tensile | A DataRobot Python package that records an agent's LLM interactions as replayable trajectories and provides agent skills for analyzing them, detecting recurring mistakes, and testing remediations with statistical replays. |
| OpenCode (dr opencode plugin) | The coding agent that runs the Tensile skills. It reads the pulled trajectory, reports what the agent did, and can edit the agent's prompts or code. Models come from the DataRobot LLM gateway. |
| dr xp (Tracing page) | A tool that detects which traces carry a recorded trajectory, pulls them on demand, and embeds the OpenCode session in the dashboard. |

When you click ✨ or Analyze trace on a trace, `dr xp` does the following in the directory it was launched from:

1. Prepares a Tensile project. Runs tensile check and, when needed, tensile init --refresh --agent opencode . This writes .tensile/config.yaml , installs the Tensile agent skills under .opencode/skills/ , and adds both to .gitignore . Nothing is written until the first click.
2. Pulls the trajectory. Runs tensile trace pull for the trace, using the entity, endpoint, and token dr xp already holds. The result is .tensile/trajectories/trace-<first 12 characters of the trace ID>.jsonl.gz . If the file already exists, it is reused. It is refreshed automatically when DataRobot has more spans than the local copy.
3. Opens OpenCode. Starts the shared dr opencode serve process if it is not already running (the same one the Assist tab uses), then opens the analysis view with /tensile trace-<prefix> pre-filled in the composer. The prompt is not sent, so you can edit it or switch models first.

## Enable trace analysis

Trace analysis is controlled by the `--enable-trace-analysis` flag. Without it, none of the trace analysis controls appear on the Tracing page, `dr xp` never checks DataRobot for recorded trajectories, never runs or installs `tensile`, and never starts OpenCode on demand.

Pass the flag when starting the dashboard from the agent project directory:

```
dr xp --enable-trace-analysis --entity-id <USE_CASE_ID>
```

Alternatively, set the environment variable before launch:

```
export DR_EXPERIMENT_ENABLE_TRACE_ANALYSIS=true
dr xp --entity-id <USE_CASE_ID>
```

For Agentic Starter projects that use `dr run dev` or `task infra:dev`, add the variable to the project root `.env` file instead. Taskfiles load `.env` automatically:

```
# .env (project root)
DR_EXPERIMENT_ENABLE_TRACE_ANALYSIS=true
```

When trace analysis is enabled, `/api/info` lists `"trace_analysis"` in `features`. When disabled, the `/api/tensile/*` routes return `404`, and the analysis views redirect to the Tracing page.

> [!NOTE] Rundr xpfrom the agent project
> Trace analysis writes its files to, and reads the agent's `tensile` installation from, the directory `dr xp` was launched from. Start the dashboard from the root of the agent project you want to analyze so that OpenCode sees the agent's code and the project's own `tensile` version pulls the traces it recorded.

## Prerequisites

| Requirement | Notes |
| --- | --- |
| Experimentation plugin with trace analysis | Run dr xp --help and confirm --enable-trace-analysis is listed. Update with dr plugin update xp if it is missing. |
| OpenCode plugin | dr plugin install opencode. If it is missing, dr xp installs it the first time you analyze a trace. Set DR_EXPERIMENT_ASSIST_NO_INSTALL=true to disable the automatic install. See DataRobot OpenCode plugin. |
| LLM Gateway access | OpenCode runs the analysis with a model from the DataRobot LLM Gateway. |
| Credentials | DATAROBOT_API_TOKEN and DATAROBOT_ENDPOINT in the environment or project .env; dr auth login provides them when dr xp runs through the CLI. |
| tensile CLI | Found in the agent project's virtual environment, on PATH, or installed automatically the first time you analyze a trace (requires uv). See Which tensile runs. |
| Instrumented agent | The agent must call tensile.otel.install() and export traces to DataRobot. See Instrument the agent. |

## Instrument the agent

The trace analysis controls appear only on traces that carry a recorded trajectory—one `tensile.envelope` log record per LLM call. To record them, the agent must install Tensile's OpenTelemetry hook. These steps are also available in the dashboard at `/tensile/setup`, linked from the Analyze traces tooltip and from every analysis error state.

1. Add the packages to the agent project: uvadd"tensile>=0.12.0"opentelemetry-sdkopentelemetry-exporter-otlp-proto-http Agents built from an Agentic Starter template already trace to DataRobot; they only need thetensiledependency inagent/pyproject.toml.
2. Export traces to DataRobot. Point a tracer provider athttps://<your-datarobot-host>/otel/v1/traceswith the headersX-DataRobot-Api-Key(your API token) andX-DataRobot-Entity-Id(where traces land:experiment_container-<USE_CASE_ID>ordeployment-<DEPLOYMENT_ID>). Alternatively, setDATAROBOT_API_TOKEN,DATAROBOT_ENDPOINT, andDATAROBOT_ENTITY_IDin the environment or.env, and the hook in the next step bootstraps a provider for you.
3. Call the hook once, after the providers are configured and before the first LLM client is created: importtensile.oteltensile.otel.install()
4. Emit spans. An envelope attaches to the span that is current when the LLM call runs, so wrap each request in a root span or turn on your framework's OpenTelemetry instrumentation (for example,Agent.instrument_all()in PydanticAI). Without a span, there is nothing to attach to.

The minimal shape of an instrumented agent:

```
load_dotenv()          # DATAROBOT_* variables
configure_otel()       # tracer provider aimed at DataRobot

import tensile.otel    # after the providers, before any LLM client

tensile.otel.install()

with tracer.start_as_current_span("my-agent.request"):
    client.chat.completions.create(model="azure/gpt-5-mini", messages=[...])
```

### What gets captured

Tensile's hook records calls made through the OpenAI Python SDK (1.x to 3.x) or LiteLLM (1.70 up to 2.0), to paths ending in `/chat/completions`, `/responses`, or `/llmgw/chat`. That covers the DataRobot LLM Gateway, OpenAI, any OpenAI-compatible server, and frameworks built on those clients (PydanticAI, LangChain's OpenAI integration, `datarobot-genai`).

Not captured: the Anthropic SDK or Anthropic through LiteLLM ( `/v1/messages`), and the native Gemini, Bedrock, and Vertex SDKs. Those traces get no trace analysis controls. Envelopes over 3 MB are dropped with a warning in the agent's log.

Envelopes arrive 7 to 30 seconds after their spans, so a trace that has just finished gains its ✨ within about half a minute.

## Analyze a trace

1. From the agent project root, start the dashboard with trace analysis enabled, then open the Tracing page. See Enable trace analysis .
2. Send requests to the agent and wait for traces to appear. Traces with a recorded trajectory show a ✨ icon beside the status badge. Open a trace to see the same control as theAnalyze tracebutton at the top of the trace, next to the fullscreen and close icons.
3. Click ✨ orAnalyze trace. The dashboard reportsPreparing Tensile, pulling the trajectory from DataRobot and starting OpenCode…while it prepares the project, pulls the trajectory, and starts OpenCode. The first click afterdr xpstarts takes a few seconds; later clicks on a settled trace return in about a second.
4. Review the pre-filled prompt (/tensile trace-<prefix>) in the OpenCode composer. Edit it to add context—for example, what the agent was supposed to do—or leave it as is.
5. If the model picker shows a free-tier OpenCode model instead of a DataRobot LLM Gateway model, pick a gateway model once (a Claude Sonnet or stronger model is recommended). OpenCode remembers the choice for later sessions.
6. Send the prompt. Thetensileskill checks the installation, lists trajectories, starts an analysis session, reads the extracted trajectory, and reports a step-by-step assessment: whether the agent succeeded, which tool calls were appropriate, and any mistakes it found.
7. Continue the conversation. Ask follow-up questions about the trajectory, or ask the agent to remediate an issue. Thetensileskill escalates into thetensile-remediateworkflow, which createsmotifs(pattern detectors for the mistake), measures baseline rates with replays, and tests prompt or tool changes before applying them to the agent. SeeTensile skills and concepts.

> [!NOTE] Replays consume tokens
> Remediation workflows replay LLM requests to measure whether a fix changes the agent's behavior. Replays go to the LLM endpoint the agent used when the trajectory was recorded—the DataRobot LLM Gateway if the agent called it, or whatever other endpoint the agent was configured with—and can consume a substantial number of tokens there. The initial `/tensile` analysis does not replay anything.

### Analyze all traces on a page

Analyze traces in the Tracing page toolbar is the page-level version. It pulls every captured trace currently shown on the page (in parallel, reusing files that already exist) and opens one OpenCode session seeded with:

```
/tensile Inspect my recent trajectories. How is the agent performing?
```

The coding agent, not `dr xp`, chooses which trajectories to analyze together. Edit the prompt before sending to state a different intent (for example, "the latest 4" or "anything that called tool X"). Use the date, status, or search filters to limit which traces get pulled.

The button is disabled when no trace on the page has a recorded trajectory. Its tooltip explains why and links to the setup guide. While a trace is open, the toolbar button is hidden and use the open trace's Analyze trace button instead.

Every pulled trajectory file is backdated to its newest step's timestamp, so `tensile ls` reports when the agent ran, not when the file was pulled, and "recent" means what you expect.

### Resume or restart a session

Returning to a trace's analysis view resumes its OpenCode session rather than starting over. The mapping from trajectory to session is kept in browser local storage. Click Start fresh in the analysis header to discard the remembered session and open a new one with the prompt re-seeded.

## Which tensile runs

`dr xp` looks for the `tensile` CLI in this order:

1. An activated virtual environment ( $VIRTUAL_ENV ).
2. Any .venv or venv up to three levels below the launch directory that contains tensile (for example, agent/.venv ), shallowest first.
3. PATH , then ~/.local/bin , ~/.cargo/bin , and $UV_TOOL_BIN_DIR .
4. A copy dr xp installs for itself.

The agent project's own copy comes first so the version that pulls a trace is the one that recorded it. That directory is also put first on the OpenCode subprocess's `PATH`, so the coding agent's own `tensile` commands use the same version.

When none of those has `tensile`, the first time you analyze a trace, `dr xp` installs `tensile>=0.12.0` with `uv tool install` into a directory `dr xp` owns:

| Platform | Location |
| --- | --- |
| macOS | ~/Library/Application Support/dr-experimentation/tensile/ |
| Linux | ~/.config/dr-experimentation/tensile/ |

The install takes roughly 20 to 60 seconds once, requires [uv](https://docs.astral.sh/uv/) on the machine, and is reported in the notes above the analysis frame. The copy is never upgraded automatically; delete the directory to reinstall. A project virtual environment always takes precedence over it.

Set `DR_EXPERIMENT_TENSILE_NO_INSTALL=true` to prevent `dr xp` from installing anything. Without `uv`, install `tensile` yourself (for example, `pipx install "tensile>=0.12.0"`, or `pip install "tensile>=0.12.0"` into a virtual environment you activate before launching `dr xp`), then analyze the trace again.

## Tensile skills and concepts

`tensile init` installs the Tensile agent skills into `.opencode/skills/`. Trace analysis uses two of them:

| Skill | Use |
| --- | --- |
| /tensile | The default entry point. Inspects recorded trajectories for lightweight analysis, debugging agent behavior, identifying behavioral patterns, or latency and token analysis. This is the skill the Analyze trace prompt invokes. |
| /tensile-remediate | A rigorous three-phase workflow: mistake identification, motif creation, and statistical replay remediation. Invoked when you ask for remediation or trajectory repair, or when escalating from /tensile. |

Key concepts the analysis reports use:

- Trajectory. A recorded log of every LLM interaction during one agent run. Each step contains the messages sent to the LLM and its response (text and tool calls). Trajectories are saved as .jsonl.gz files under .tensile/trajectories/ .
- Session. Groups one or more trajectories for analysis. Starting a session extracts the system prompt, user prompt, tool definitions, and a readable summary for each trajectory into .tensile/extracted/ , and scans them against existing motifs.
- Motif. A reusable pattern detector for a specific class of agent behavior, typically a recurring mistake. Motifs accumulate annotations (trajectory and step pairs where the pattern was observed) so you can measure how often a mistake occurs and whether a fix works.
- Replay. Feeds the conversation history up to an annotated step back to the LLM and samples a fresh response. Comparing detection rates across many replays, with and without a change, makes remediation quantitative rather than anecdotal.
- Remediation. A change to the agent's system prompt, tool definitions, skills, or code, tested with replays before it is applied to the real agent.

For the full CLI and skill reference, see the [Tensile package on PyPI](https://pypi.org/project/tensile/).

## Trace analysis troubleshooting

> [!NOTE] How detection works
> `dr xp` asks DataRobot, not the local filesystem, which traces carry a trajectory: one logs API probe per trace on the page, cached in the `dr xp` process. For two minutes after a trace is first seen, a negative answer is rechecked every 10 seconds (envelopes land 7 to 30 seconds after their spans), then every 60 seconds. The controls appear even if `.tensile/trajectories/` doesn't exist in the current directory.

| Symptom | Fix |
| --- | --- |
| No trace analysis controls anywhere; /tracing/<id>/analyze redirects to the Tracing page | The dashboard was started without the flag. Restart with --enable-trace-analysis or set DR_EXPERIMENT_ENABLE_TRACE_ANALYSIS=true in .env. Confirm with curl -s http://127.0.0.1:8090/api/info; features must include "trace_analysis". |
| No ✨ on any trace | Fix tracing first if the Tracing page is empty. Then check the agent's log for tensile.otel lines: openai adapter installed means capture is active; no supported LLM SDK found or is outside the supported range means the client is not captured (see What gets captured). Confirm the agent's X-DataRobot-Entity-Id matches the entity dr xp shows in its banner; a dr run dev session and a deployment are different entities. |
| ✨ missing on a trace that just finished | Envelopes land up to 30 seconds after their spans. Wait about half a minute and refresh. |
| Analyze traces is disabled | No trace on the current page has a recorded trajectory. Hover the button for the reason and a link to the setup guide. Adjust the filters or instrument the agent. |
| Could not check this page for recorded trajectories | The probe to DataRobot failed; check the dr xp terminal log, credentials, and DATAROBOT_ENDPOINT. |
| tensile is not installed | DR_EXPERIMENT_TENSILE_NO_INSTALL is set, or uv is missing. Install tensile into the agent's virtual environment or on PATH, then analyze the trace again. See Which tensile runs. |
| none of these traces has a recorded trajectory | The traces on the page lost their envelopes or the pull returned no steps. Run tensile trace pull --entity <ENTITY_ID> --trace <TRACE_ID> --wait 60 from the agent directory to check. If a trace older than a minute still pulls no recorded LLM calls, the agent's envelopes never reached DataRobot; see the No ✨ on any trace row above. |
| /tensile is not recognized in OpenCode | OpenCode discovers skills when it starts. If the Assist tab (--enable-assist) started OpenCode before your first analysis created .opencode/skills/, restart dr xp. |
| Composer shows a free-tier model | Pick a DataRobot LLM Gateway model from the model picker once; OpenCode remembers the choice. |
| Analysis says the trajectory is empty or stale | dr xp re-pulls a trace when DataRobot reports more spans than the local file, and refreshes young pulls until their envelopes settle. Analyze the trace again, or delete the file under .tensile/trajectories/ to force a fresh pull. Locally recorded trajectories (from an agent run in the same directory with tensile init done) are never replaced. |
| Logs tab is flooded with tensile.envelope records | The Logs tab always hides tensile.envelope records, regardless of the flag, so this is a bug. Report it. Span details still show each span's envelope. |

## See also

- DataRobot experimentation plugin —install, configure, and open the GUI.
- Local tracing —inspect OpenTelemetry spans on the Tracing page.
- Batch agent evaluation —score agent responses on a fixed dataset.
- DataRobot OpenCode plugin —the coding agent that runs the Tensile skills.
- Implement tracing —add custom spans and attributes.
- Tensile on PyPI —package, CLI reference, and skill descriptions.