# Implement tracing

> Implement tracing - Learn how to add OpenTelemetry tracing and custom span instrumentation to your
> agent tools for monitoring, debugging, and observability.

This Markdown file sits beside the HTML page at the same path (with a `.md` suffix). It summarizes the topic and lists links for tools and LLM context.

Companion generated at `2026-04-24T16:03:56.226747+00:00` (UTC).

## Primary page

- [Implement tracing](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-tracing-code.html): Full documentation for this topic (HTML).

## Sections on this page

- [Add custom tracing to tools](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-tracing-code.html#add-custom-tracing-to-tools): In-page section heading.
- [Tool examples](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-tracing-code.html#tool-examples): In-page section heading.
- [Create nested spans](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-tracing-code.html#create-nested-spans): In-page section heading.
- [Add events to spans](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-tracing-code.html#add-events-to-spans): In-page section heading.
- [Add custom tracing to agent](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-tracing-code.html#trace-agent-configuration): In-page section heading.
- [Map spans and attributes to the tracing table](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-tracing-code.html#map-spans-and-attributes-to-the-tracing-table): In-page section heading.
- [Surface tool names in the tracing table](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-tracing-code.html#surface-tool-names-in-the-tracing-table): In-page section heading.
- [Best practices](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-tracing-code.html#best-practices): In-page section heading.

## Related documentation

- [Agentic AI](https://docs.datarobot.com/en/docs/agentic-ai/index.html): Linked from this page.
- [Build](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/index.html): Linked from this page.
- [View logs and traces for a deployed agent](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-development.html#view-logs-and-traces-for-a-deployed-agent): Linked from this page.

## Documentation content

# Implement tracing

OpenTelemetry (OTel) provides comprehensive observability for your agents, allowing you to monitor, trace, and debug agent execution in real-time. This guide explains how to add custom tracing to your agent tools to capture detailed execution information.

OpenTelemetry tracing helps:

- Monitor agent performance and execution flow.
- Debug issues by tracking detailed execution traces.
- Understand tool execution patterns and timing.
- View custom attributes and metadata from your tools.

The agent templates already include OpenTelemetry instrumentation for frameworks like CrewAI, LangGraph, and Llama-Index. This instrumentation automatically captures spans for:

- Agent execution
- Tool invocations
- LLM API calls
- HTTP requests

You can enhance this default tracing by adding custom spans and attributes in your tools.

## Add custom tracing to tools

Add custom OpenTelemetry tracing to your tools to capture additional information about tool execution. This allows you to track custom attributes, intermediate outputs, and execution details that are specific to your use case.

The basic pattern for adding custom tracing to a tool is:

```
from opentelemetry import trace

tracer = trace.get_tracer(__name__)

# Within your tool's execution
with tracer.start_as_current_span("my_custom_span_name"):
    current_span = trace.get_current_span()
    current_span.set_attribute("tool_name", "my_tool_name")
    current_span.set_attribute("gen_ai.prompt", "input passed to this step")
    current_span.set_attribute("datarobot.moderation.cost", 0.0)

    # Your tool logic here
    result = perform_tool_action()

    current_span.set_attribute("gen_ai.completion", str(result))
    # Optionally add more attributes about the result
    current_span.set_attribute("result.status", "success")
    current_span.set_attribute("result.size", len(result))

    return result
```

### Tool examples

See the code examples below to learn how to add custom OpenTelemetry tracing to agentic tools:

**CrewAI:**
```
import requests
from crewai.tools import BaseTool
from opentelemetry import trace

tracer = trace.get_tracer(__name__)

class WeatherTool(BaseTool):
    name: str = "weather_tool"
    description: str = (
        "Fetches the current weather for a specified city. "
        "Requires an API key from OpenWeatherMap."
    )

    def _run(self, city: str) -> str:
        with tracer.start_as_current_span("weather_tool_fetch"):
            current_span = trace.get_current_span()
            current_span.set_attribute("tool_name", "weather_tool")
            current_span.set_attribute("gen_ai.prompt", f"weather lookup for {city}")
            current_span.set_attribute("datarobot.moderation.cost", 0.0)

            # Set custom attributes
            current_span.set_attribute("weather.city", city)
            current_span.set_attribute("weather.api", "openweathermap")

            api_key = "YOUR_API_KEY"  # Replace with your API key
            base_url = "http://api.openweathermap.org/data/2.5/weather"
            params = {"q": city, "appid": api_key, "units": "metric"}

            try:
                response = requests.get(base_url, params=params, timeout=10)
                response.raise_for_status()

                data = response.json()
                weather = data['weather'][0]
                main = data['main']

                # Add result attributes
                current_span.set_attribute("weather.temperature", main['temp'])
                current_span.set_attribute("weather.condition", weather['main'])

                result = (
                    f"Current weather in {data['name']}, {data['sys']['country']}:\n"
                    f"Temperature: {main['temp']}°C (feels like {main['feels_like']}°C)\n"
                    f"Condition: {weather['main']} - {weather['description']}\n"
                    f"Humidity: {main['humidity']}%\n"
                    f"Pressure: {main['pressure']} hPa"
                )
                current_span.set_attribute("gen_ai.completion", result)

                return result

            except requests.exceptions.RequestException as e:
                current_span.set_attribute("weather.error", str(e))
                err = f"Error fetching weather data: {str(e)}"
                current_span.set_attribute("gen_ai.completion", err)
                return err
```

**LangGraph:**
```
import requests
from langchain.tools import BaseTool
from opentelemetry import trace

tracer = trace.get_tracer(__name__)

class WeatherTool(BaseTool):
    name: str = "weather_tool"
    description: str = (
        "Fetches the current weather for a specified city. "
        "Requires an API key from OpenWeatherMap."
    )

    def _run(self, city: str) -> str:
        with tracer.start_as_current_span("weather_tool_fetch"):
            current_span = trace.get_current_span()
            current_span.set_attribute("tool_name", "weather_tool")
            current_span.set_attribute("gen_ai.prompt", f"weather lookup for {city}")
            current_span.set_attribute("datarobot.moderation.cost", 0.0)

            # Set custom attributes
            current_span.set_attribute("weather.city", city)

            api_key = "YOUR_API_KEY"  # Replace with your API key
            base_url = "http://api.openweathermap.org/data/2.5/weather"
            params = {"q": city, "appid": api_key, "units": "metric"}

            try:
                response = requests.get(base_url, params=params, timeout=10)
                response.raise_for_status()

                data = response.json()

                # Add result attributes
                current_span.set_attribute("weather.temperature", data['main']['temp'])

                result = f"Temperature in {city}: {data['main']['temp']}°C"
                current_span.set_attribute("gen_ai.completion", result)
                return result

            except requests.exceptions.RequestException as e:
                current_span.set_attribute("weather.error", str(e))
                err = f"Error: {str(e)}"
                current_span.set_attribute("gen_ai.completion", err)
                return err
```

**Llama-Index:**
```
import requests
from llama_index.core.tools import FunctionTool
from opentelemetry import trace

tracer = trace.get_tracer(__name__)

def _weather_run(city: str) -> str:
    """Fetches the current weather for a specified city. Requires an API key from OpenWeatherMap."""
    with tracer.start_as_current_span("weather_tool_fetch"):
        current_span = trace.get_current_span()
        current_span.set_attribute("tool_name", "weather_tool")
        current_span.set_attribute("gen_ai.prompt", f"weather lookup for {city}")
        current_span.set_attribute("datarobot.moderation.cost", 0.0)

        # Set custom attributes
        current_span.set_attribute("weather.city", city)

        api_key = "YOUR_API_KEY"  # Replace with your API key
        base_url = "http://api.openweathermap.org/data/2.5/weather"
        params = {"q": city, "appid": api_key, "units": "metric"}

        try:
            response = requests.get(base_url, params=params, timeout=10)
            response.raise_for_status()

            data = response.json()

            # Add result attributes
            current_span.set_attribute("weather.temperature", data['main']['temp'])

            result = f"Temperature in {city}: {data['main']['temp']}°C"
            current_span.set_attribute("gen_ai.completion", result)
            return result

        except requests.exceptions.RequestException as e:
            current_span.set_attribute("weather.error", str(e))
            err = f"Error: {str(e)}"
            current_span.set_attribute("gen_ai.completion", err)
            return err

def WeatherTool() -> FunctionTool:
    return FunctionTool.from_defaults(
        fn=_weather_run,
        name="weather_tool",
        description=(
            "Fetches the current weather for a specified city. "
            "Requires an API key from OpenWeatherMap."
        ),
    )
```


## Create nested spans

Create nested spans to represent complex tool execution with multiple steps:

```
from opentelemetry import trace

tracer = trace.get_tracer(__name__)

def complex_tool_workflow(input_data):
    with tracer.start_as_current_span("complex_tool_main"):
        current_span = trace.get_current_span()
        current_span.set_attribute("input.size", len(input_data))

        # First step in the workflow
        with tracer.start_as_current_span("data_processing"):
            processed_data = process_data(input_data)
            trace.get_current_span().set_attribute("processed_items", len(processed_data))

        # Second step in the workflow
        with tracer.start_as_current_span("data_validation"):
            validated_data = validate_data(processed_data)
            trace.get_current_span().set_attribute("validated_items", len(validated_data))

        # Third step in the workflow
        with tracer.start_as_current_span("result_generation"):
            result = generate_result(validated_data)
            current_span.set_attribute("result.size", len(result))

        return result
```

### Add events to spans

Add events to your spans to mark important moments in tool execution:

```
from opentelemetry import trace
from datetime import datetime

tracer = trace.get_tracer(__name__)

def tool_with_events():
    with tracer.start_as_current_span("tool_execution"):
        current_span = trace.get_current_span()

        # Add an event for when processing starts
        current_span.add_event(
            "Processing started",
            {"timestamp": datetime.utcnow().isoformat()}
        )

        # Your tool logic
        intermediate_result = perform_action()

        # Add an event for mid-execution
        current_span.add_event(
            "Intermediate result ready",
            {"result_count": len(intermediate_result)}
        )

        # More processing
        final_result = complete_processing(intermediate_result)

        # Add final event
        current_span.add_event(
            "Processing completed",
            {"output_size": len(final_result)}
        )

        return final_result
```

## Add custom tracing to agent

You can set up a custom trace to capture how your agent starts up, including configurations and environment details. Follow the steps below to surface runtime parameters (like environment variables) on a span:

1. Update your.envfile to contain the following environment variable so it's available during local development and when you package the model: EXAMPLE_ENV_VAR=my_example_value
2. Add the parameter toagent/model-metadata.yamlso DataRobot can inject it when the agent runs: runtimeParameterDefinitions:-fieldName:EXAMPLE_ENV_VARtype:stringdefaultValue:SET_VIA_PULUMI_OR_MANUALLY
3. Add the parameter to your Config class inagent/agent/config.py(for example,example_env_var: str = "") so the value is available asconfig.example_env_varin your agent code.
4. Updateinfra/infra/llm.pyto forward the runtime parameter into the custom model environment: custom_model_runtime_parameters=[# ...existing parameters...datarobot.CustomModelRuntimeParameterValueArgs(key="EXAMPLE_ENV_VAR",type="string",value=os.environ.get("EXAMPLE_ENV_VAR"),),]
5. Wrap the configuration loading code in a span and attach the values withset_attributeandadd_event. The property name depends on your template (for example,agent_plannerin CrewAI templates): @propertydefagent_planner(self)->Any:withtracer.start_as_current_span("config_variables"):current_span=trace.get_current_span()current_span.set_attribute("config.example_env_var",config.example_env_var)current_span.add_event("config attribute set on span")# ...agent code continued...

When you deploy and run the agent, the trace visualizer shows a `config_variables` span with attributes such as `config.example_env_var=my_example_value`. This makes it easy to confirm that runtime parameters and other environment values were loaded correctly.

For information on viewing traces in the DataRobot UI after deployment, see [View logs and traces for a deployed agent](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-development.html#view-logs-and-traces-for-a-deployed-agent).

## Map spans and attributes to the tracing table

On a deployment, the tracing table displays several columns derived from OpenTelemetry span attributes. The following naming conventions apply across the trace:

| Tracing table column | Attribute | How it is derived |
| --- | --- | --- |
| Cost | datarobot.moderation.cost | Summed across all spans in the trace. |
| Prompt | gen_ai.prompt | If multiple spans set this attribute, the first value in trace order is used. |
| Completion | gen_ai.completion | If multiple spans set this attribute, the last value in trace order is used. |
| Tools | tool_name | Every distinct tool_name found on any span in the trace is listed. |

### Surface tool names in the tracing table

The Tools column is populated from the span attribute `tool_name`. Some frameworks set it on tool spans automatically; others do not. If your traces show tool execution in the span timeline, but Tools is empty, create a span around the tool body (or use the active span) and set `tool_name` explicitly.

**Inside awithspan:**
```
from opentelemetry import trace

tracer = trace.get_tracer(__name__)

def my_tool_impl(query: str) -> str:
    with tracer.start_as_current_span("my_tool"):
        span = trace.get_current_span()
        # Attributes that map to deployment Tracing table columns (see above)
        span.set_attribute("tool_name", "my_tool_name")
        span.set_attribute("gen_ai.prompt", query)
        span.set_attribute("datarobot.moderation.cost", 0.0)  # numeric; summed per trace
        # ... tool logic ...
        result = "result"
        span.set_attribute("gen_ai.completion", result)
        return result
```

**Current span only:**
If a span is already active (for example, from upstream instrumentation), you can set the attribute on that span:

```
from opentelemetry import trace

span = trace.get_current_span()
span.set_attribute("tool_name", "my_tool_name")
span.set_attribute("gen_ai.prompt", "user input or request text")
span.set_attribute("gen_ai.completion", "model or tool output text")
span.set_attribute("datarobot.moderation.cost", 0.0)
```


For LangGraph and similar frameworks, tool calls are sometimes wired through callbacks in a way that does not add `tool_name` to spans; manual instrumentation can allow the name to appear in the Tools column.

## Best practices

Use descriptive span names:

- Use clear, descriptive names for spans (e.g., "weather_fetch" rather than "span1" ).
- Include the tool name in the span name when relevant.

Set meaningful attributes:

- Add attributes that provide context about the execution.
- Use consistent attribute naming conventions (e.g., tool.input , tool.output , tool.error ).
- Include relevant metadata like sizes, counts, or statuses.
- To populate Cost , Prompt , Completion , and Tools in the deployment Tracing table, set datarobot.moderation.cost , gen_ai.prompt , gen_ai.completion , and tool_name on the relevant spans.

Keep spans focused:

- Create spans for significant operations, not every line of code.
- Each span should represent a meaningful unit of work.
- Use nested spans to represent sub-operations.