Self-managed install & maintenance > Enterprise monitoring > Observability with OpenTelemetry > Observability vendors

Observability vendors¶

This section shows how to configure the observability subchart for observability vendors.

Datadog¶

This section shows how to configure the chart to export telemetry to Datadog.

Requirements¶

A Kubernetes secret containing the Datadog API key must exist in the cluster (either by creating it directly or using an external secret provider, refer to the External Secrets for DataRobot Kubernetes Installation section in the installation guide). The secret name and key are configurable - they just need to match what is referenced in the secrets section of the chart configuration below.

To add the secret directly to the cluster, use the following command:

kubectl create secret -n <DR_CORE_NAMESPACE> generic datarobot-datadog \
  --from-literal=api-key="<DATADOG-API-KEY>"

Full chart configuration¶

The following configuration is added to the datarobot-prime chart values. Replace the placeholder values with the actual values.

For additional exporter configuration options, refer to the upstream OpenTelemetry documentation for the datadog exporter.

global:
  observability:
    secrets:
      - envVar: DD_API_KEY
        secretName: datarobot-datadog
        secretKey: api-key

    exporters:
      datadog:
        api:
          site: <DATADOG_SITE>
          key: ${env:DD_API_KEY}

    signals:
      logs:
        exporters: [datadog]
      metrics:
        exporters: [datadog]
      traces:
        exporters: [datadog]

Where:

<DATADOG_SITE>: the Datadog site (e.g. datadoghq.com, us5.datadoghq.com, datadoghq.eu)

The secrets section injects the DD_API_KEY environment variable from the datarobot-datadog Kubernetes secret into all collector pods. The exporter references this value using the ${env:DD_API_KEY} syntax.

Datadog agent¶

Additionally, you can install the Datadog agent to collect and export cluster-level metrics to Datadog's Infrastructure Monitoring, which has out-of-the-box dashboards for Kubernetes metrics. Note that this is a completely independent component from DataRobot and isn't distributed in the observability subcharts.

Refer to the Datadog docs; or perform a quick install with the following minimal values for its chart (note the reference of the datarobot-datadog secret mentioned above):

datadog:
  apiKeyExistingSecret: datarobot-datadog
  site: "<DATADOG_SITE>"

And installing the chart:

helm repo add datadog https://helm.datadoghq.com
helm repo update
helm install datadog-agent datadog/datadog -f datadog-agent.yaml

Splunk/SignalFx¶

This section shows how to configure the chart to export telemetry to Splunk (logs) and SignalFx (metrics and traces).

Requirements¶

Kubernetes secrets containing the Splunk HEC (HTTP Event Collector) token and the Splunk Observability Cloud (formerly SignalFx) token must exist in the cluster (either by creating them directly or using an external secret provider, refer to the External Secrets for DataRobot Kubernetes Installation section in the installation guide). The secret names and keys are configurable - they just need to match what is referenced in the secrets section of the chart configuration below.

To add the secrets directly to the cluster, use the following commands:

kubectl create secret -n <DR_CORE_NAMESPACE> generic datarobot-splunk \
  --from-literal=access_token="<HEC_ACCESS_TOKEN>"

kubectl create secret -n <DR_CORE_NAMESPACE> generic datarobot-signalfx \
  --from-literal=access_token="<SIGNALFX_ACCESS_TOKEN>"

Full chart configuration¶

The following configuration is added to the datarobot-prime chart values. Replace the placeholder values with the actual values.

For additional exporter configuration options, refer to the upstream OpenTelemetry documentation for the splunkhec and signalfx exporters.

global:
  observability:
    secrets:
      - envVar: SPLUNK_HEC_TOKEN
        secretName: datarobot-splunk
        secretKey: access_token
      - envVar: SIGNALFX_TOKEN
        secretName: datarobot-signalfx
        secretKey: access_token

    exporters:
      splunk_hec:
        token: ${env:SPLUNK_HEC_TOKEN}
        endpoint: <SPLUNK_HEC_ENDPOINT>
        source: datarobot
        sourcetype: _json
      signalfx:
        access_token: ${env:SIGNALFX_TOKEN}
        realm: <SIGNAL_FX_REALM>
        api_url: https://api.<SIGNAL_FX_REALM>.signalfx.com
        ingest_url: https://ingest.<SIGNAL_FX_REALM>.signalfx.com

    signals:
      logs:
        exporters: [splunk_hec]
      metrics:
        exporters: [signalfx]
      traces:
        exporters: [signalfx]

Where:

<SPLUNK_HEC_ENDPOINT>: the HEC endpoint, refer to the official Splunk cloud/ enterprise docs to retrieve this endpoint
<SIGNAL_FX_REALM>: the SignalFx realm (e.g. us1)

The secrets section injects the SPLUNK_HEC_TOKEN and SIGNALFX_TOKEN environment variables from their respective Kubernetes secrets into all collector pods. The exporters reference these values using the ${env:VAR_NAME} syntax.

Generic OTLP endpoint¶

This section shows how to configure the chart to export telemetry to a generic OTLP ingestion endpoint. This is suitable for any backend that supports the OTLP protocol.

Full chart configuration¶

The following configuration is added to the datarobot-prime chart values. Replace the placeholder values with the actual values.

For additional exporter configuration options, refer to the upstream OpenTelemetry documentation for the otlp and otlphttp exporters.

Using gRPC (OTLP)¶

global:
  observability:
    exporters:
      otlp:
        endpoint: <OTLP_GRPC_ENDPOINT>
        tls:
          insecure: false

    signals:
      logs:
        exporters: [otlp]
      metrics:
        exporters: [otlp]
      traces:
        exporters: [otlp]

Using HTTP (OTLP/HTTP)¶

global:
  observability:
    exporters:
      otlphttp:
        endpoint: <OTLP_HTTP_ENDPOINT>

    signals:
      logs:
        exporters: [otlphttp]
      metrics:
        exporters: [otlphttp]
      traces:
        exporters: [otlphttp]

Where:

<OTLP_GRPC_ENDPOINT>: the gRPC endpoint (e.g. otel-collector.example.com:4317)
<OTLP_HTTP_ENDPOINT>: the HTTP endpoint (e.g. https://otel-collector.example.com:4318)

With authentication headers¶

If the endpoint requires an API key or bearer token, store it in a Kubernetes secret and reference it via the secrets section:

kubectl create secret -n <DR_CORE_NAMESPACE> generic datarobot-otlp \
  --from-literal=api-key="<API_KEY>"

global:
  observability:
    secrets:
      - envVar: OTLP_API_KEY
        secretName: datarobot-otlp
        secretKey: api-key

    exporters:
      otlphttp:
        endpoint: <OTLP_HTTP_ENDPOINT>
        headers:
          Authorization: Bearer ${env:OTLP_API_KEY}

    signals:
      logs:
        exporters: [otlphttp]
      metrics:
        exporters: [otlphttp]
      traces:
        exporters: [otlphttp]