Skip to content
For Self-Managed AI Platform users running v11.1, see the on-premise platform documentation
Self-managed installation and maintenance > Installation and configuration guide > Advanced configuration > Security Best Practices

Security best practices

Introduction

This guide is designed to help configure DataRobot following security best practices. DataRobot is a complex product with a wealth of configuration options -- at the time of writing (Jan 2023), this guide covers a small number of them and may be extended over time.

To configure these options, refer to the Tuning DataRobot Environment Variables section of this guide.

Session expiration

Leaving web UI sessions active forever is considered to be a security threat. DataRobot offers flexible control over the session expiration mechanism that allows you to configure session lifetime to be relative to last user visit or to expire after a certain amount of seconds after user login. Once a webui session has expired, a user must re-login and re-authenticate before they can get access to the application again.

Control over the session expiration logic is done by the boolean configuration option WEB_UI_SESSION_ABSOLUTE_EXPIRATION_ENABLED, which determines if the session lifetime should be treated as relative to last user action (request) or absolute to user login. The lifetime (in seconds) of the session is configured by the WEB_UI_SESSION_LIFE_TIME configuration option. The default value is 0 which means that session lasts forever.

Configuration examples

The following configuration options in core.config_env_vars set session lifetime to 10 minutes relative to last user action:

# helm chart values snippet
core:
  config_env_vars:
    WEB_UI_SESSION_LIFE_TIME: 600

The following configuration options in core.config_env_vars set session lifetime to 10 minutes from the moment of user login:

# helm chart values snippet
core:
  config_env_vars:
    WEB_UI_SESSION_LIFE_TIME: 600
    WEB_UI_SESSION_ABSOLUTE_EXPIRATION_ENABLED: true

Host header protection

To prevent host header poisoning, API servers validate the host header against an allowed list.

By default, the ALLOWED_HOSTS includes:

  • EXTERNAL_WEB_SERVER_URL for the external load balancer (and for situations there is a proxy in front of the application)
  • DATAROBOT_PUBLIC_LB for the global.domain chart values setting for the application hostname when accessed via Ingress.
  • DATAROBOT_INTERNAL_LB for the service hostname within the Kubernetes cluster (e.g. datarobot-nginx) used by internal services.
  • CUSTOM_MODEL_WEB_SERVER_URL for a configurable hostname used by Custom Models for connecting to the API server

If necessary, the ALLOWED_HOSTS setting can be configured in chart values, using a comma-delimited string of hostnames:

core:
  config_env_vars:
    ALLOWED_HOSTS: "datarobot-nginx,datarobot.example.com"

Although not recommended, to disable host header protection:

core:
  config_env_vars:
    ALLOWED_HOSTS: "*"

Integrate with customer HTTP proxy

It may be required to use a HTTP proxy for providing cluster access to specific resources over the internet or within the corporate network. To accomplish this, users may have a HTTP proxy such as Squid proxy configured to route cluster traffic.

Note

This configuration requires a lot of customization and maintenance to keep up-to-date with upgrades of DataRobot or to reflect changes to your internal infrastructure.

To achieve this, network traffic from application services needs to be routed to the proxy, but ensuring that internal traffic to services within the cluster remains internal and skips the proxy. The following environment variables can be configured globally for all containers:

  • HTTP_PROXY for proxying HTTP traffic within the cluster through an external proxy
  • HTTPS_PROXY for proxying HTTPS traffic within the cluster through an external proxy
  • NO_PROXY for excluding certain hostnames from routing through the proxy. This should be a comma-separated list of hostnames.

These can be configured in the Helm chart values:

global:
  extraEnvVars:
  - name: HTTP_PROXY
    value: http://proxy.example.com:3128
  - name: HTTPS_PROXY
    value: http://proxy.example.com:3128
  - name: NO_PROXY
    value: .dr-app,.svc,.local,.internal,localhost,127.0.0.1,kubernetes,auth-server-hydra-admin,auth-server-hydra-public,blob-view-service,browser-adls,browser-bigquery,browser-controller,browser-databricks,browser-datasphere,browser-s3,browser-snowflake,build-service,buzok-llm-gateway,buzok-web,cfx-session-port-proxy,compute-jobs-service,compute-spark,custom-apps-websocket-proxy,datarobot-analyticsbroker-api,datarobot-analyticsbroker-ingest,datarobot-apigateway-apigateway,datarobot-apps-builder-api,datarobot-apps-builder-internal-api,datarobot-auth-server,datarobot-datasets-service-api,datarobot-internal-api,datarobot-mmapp,datarobot-mmqueue,datarobot-nginx,datarobot-otel-collector,datarobot-pngexport,datarobot-prediction-server,datarobot-prediction-spooler,datarobot-predictions-gateway,datarobot-public-api,datarobot-rsyslog-master,datarobot-tileservergl,datarobot-upload,datavolt-service,identity-resource-service,nbx-audit-logs,nbx-code-assistance,nbx-code-nuggets,nbx-data-retention,nbx-env-vars,nbx-exec-envs,nbx-filesystems,nbx-ingress,nbx-notebook-import,nbx-notebook-revisions,nbx-notebooks,nbx-orchestrator,nbx-permissions,nbx-scheduling,nbx-session-port-proxy,nbx-terminals,nbx-usage-tracking,nbx-users,nbx-vcs,nbx-websockets,network-policy-consumer-service,notification-service,oauth-providers-service,ocr-service,pcs-elasticsearch,pcs-elasticsearch-master-hl,pcs-mongo-0,pcs-mongo-1,pcs-mongo-2,pcs-mongo-arbiter-headless,pcs-mongo-headless,pcs-pgpool,pcs-postgresql,pcs-postgresql-headless,pcs-rabbitmq,pcs-rabbitmq-headless,pcs-redis,pcs-redis-headless,pred-environments-api,service-registration-controller

Make sure to replace proxy.example.com:3128 with your desired proxy address.

For internal service communications in Kubernetes, this covers all <service>.<namespace>.svc.cluster.local fully-qualified names for DataRobot services.

You MUST also add the following to NO_PROXY based on your configuration:

  • The namespace where DataRobot is installed (e.g. .dr-app by default; replace it if you use another namespace)
  • The ClusterIP associated with the Kubernetes API accessed by pods, as returned by kubectl get svc kubernetes (e.g. 10.100.0.1 for EKS by default)
  • The global.domain host where users access your DataRobot application (e.g. datarobot.example.com)
  • Hostnames for cloud services endpoints when deployed into a VPC.

For cloud providers' service endpoints:

  • AWS: you can specify .amazonaws.com for all regions or .<region>.amazon.aws.com to limit to a specific region such as us-east-1. See AWS Service Endpoints for the supported list of hostnames.
  • GCP: you can specify .googleapis.com to allow list all services, otherwise refer to Access Google APIs through endpoints for supported list of hostnames.
  • Azure: see Azure Private Endpoints for a list of supported hostnames.

Additionally, you SHOULD add the following if they apply to your situation:

  • Internal IP addresses for corporate networks that shouldn't be accessed through the proxy (e.g. 1.2.3.4). NOTE - CIDR ranges aren't supported due to limitations in python.
  • Internal hostnames for resources that are accessible without the tunnel proxy (e.g. .example.com)

When upgrading to 10.2 or later, make sure to remove HTTP_PROXY, HTTPS_PROXY and NO_PROXY settings from other sections of the values YAML (e.g. core.common_env_vars). When set in global.extraEnvVars the settings should correctly be applied to all pods.

Network policy for custom workloads on Long running services (LRS)

Custom workloads (custom models, custom jobs, custom apps) running on LRS have a deny-by-default NetworkPolicy setup, accompanied by Network Policies to allow specific ingress or egress with external services as needed.

These pages explain the specifics to each workload:

LRS's egress for custom workloads is governed by CUSTOM_WORKLOADS_PUBLIC_ACCESS_RULES, a JSON list of {cidr, except} rules. CUSTOM_WORKLOADS_PUBLIC_ACCESS_IGNORE_CIDRS is shorthand for the except list of a single default rule.

Default value of CUSTOM_WORKLOADS_PUBLIC_ACCESS_RULES

{"cidr": "0.0.0.0/0", "except": ["10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"]}

CUSTOM_WORKLOADS_PUBLIC_ACCESS_RULES overrides CUSTOM_WORKLOADS_PUBLIC_ACCESS_IGNORE_CIDRS entirely

Setting CUSTOM_WORKLOADS_PUBLIC_ACCESS_RULES directly replaces the default above, so CUSTOM_WORKLOADS_PUBLIC_ACCESS_IGNORE_CIDRS no longer has any effect once you do—the two settings are not merged.

Allow egress to internal destinations

Some workloads need to reach an internal destination even though the default rule above denies egress to internal networks.

Because Kubernetes network policy rules are additive, the supported way to allow egress to a specific internal destination is to add a second rule to CUSTOM_WORKLOADS_PUBLIC_ACCESS_RULES for that destination, alongside the existing rule. If you've already customized CUSTOM_WORKLOADS_PUBLIC_ACCESS_IGNORE_CIDRS (see Restrict egress to specific CIDR ranges), carry those same CIDR ranges into the first rule's except list instead of the defaults shown below—setting CUSTOM_WORKLOADS_PUBLIC_ACCESS_RULES replaces that customization entirely rather than merging with it. For example, to allow egress to a target internal service at 172.16.5.10 while keeping the default except CIDR ranges for everything else:

core:
  config_env_vars:
    CUSTOM_WORKLOADS_PUBLIC_ACCESS_RULES: '[{"cidr":"0.0.0.0/0","except":["10.0.0.0/8","172.16.0.0/12","192.168.0.0/16"]},{"cidr":"172.16.5.10/32","except":[]}]'

The first rule keeps denying egress to the standard private ranges; the second rule adds an allow for 172.16.5.10/32 specifically, regardless of the first rule's except list.

If the additional rule has no effect

Although NetworkPolicy rules are meant to be additive, some clusters don't apply a second rule this way in practice—the effective behavior can depend on the CNI plugin and how it compiles multiple rules. If egress to the destination is still denied after adding the rule above, exclude the destination from the except CIDR ranges directly instead, as described in Restrict egress to specific CIDR ranges—narrowing the except list doesn't depend on multi-rule behavior, since it changes what a single rule excludes rather than relying on a second rule to add exceptions.

Restrict egress to specific CIDR ranges

To restrict egress for LRS to specific CIDR ranges, the CUSTOM_WORKLOADS_PUBLIC_ACCESS_IGNORE_CIDRS setting can be configured in chart values:

core:
  config_env_vars:
    CUSTOM_WORKLOADS_PUBLIC_ACCESS_IGNORE_CIDRS: '10.0.0.0/8,172.16.0.0/12,192.168.0.0/16'

This should be a comma-delimited list of CIDR ranges. This setting has no effect if CUSTOM_WORKLOADS_PUBLIC_ACCESS_RULES is already set directly (see the warning above)—edit that setting's first rule's except list instead.

it's important that LRS can egress to the datarobot-nginx pod for workloads to use the Public API. Workloads should be using the internal server name datarobot-nginx instead of the external hostname (e.g. datarobot.example.com). In order for LRS to communicate with the Public API using Ingress, the Network Policy would need to be modified to allow egress to the CIDR range covering the ingress controller.

To stop restricting an entire range when nothing else in it needs to stay restricted, remove it from the except list entirely (e.g. 172.16.0.0/12), for example:

core:
  config_env_vars:
    CUSTOM_WORKLOADS_PUBLIC_ACCESS_IGNORE_CIDRS: '10.0.0.0/8,192.168.0.0/16'

To exclude specific IP addresses from an except CIDR while keeping the rest of it restricted, split the range into smaller CIDR ranges that leave those addresses out (e.g. 172.16.5.10/32). Compute the complement of the destination address against the excepted range with Python's ipaddress.address_exclude to avoid manual CIDR-math errors, for example:

import ipaddress

excepted_range = ipaddress.ip_network("172.16.0.0/12")
destination_host = ipaddress.ip_network("172.16.5.10/32")

remaining_cidrs = sorted(excepted_range.address_exclude(destination_host))
print(",".join(str(cidr) for cidr in remaining_cidrs))

This prints the comma-delimited list of CIDR ranges to use in CUSTOM_WORKLOADS_PUBLIC_ACCESS_IGNORE_CIDRS in place of 172.16.0.0/12:

172.16.0.0/22,172.16.4.0/24,172.16.5.0/29,172.16.5.8/31,172.16.5.11/32,172.16.5.12/30,172.16.5.16/28,172.16.5.32/27,172.16.5.64/26,172.16.5.128/25,172.16.6.0/23,172.16.8.0/21,172.16.16.0/20,172.16.32.0/19,172.16.64.0/18,172.16.128.0/17,172.17.0.0/16,172.18.0.0/15,172.20.0.0/14,172.24.0.0/13

core:
  config_env_vars:
    CUSTOM_WORKLOADS_PUBLIC_ACCESS_IGNORE_CIDRS: '10.0.0.0/8,172.16.0.0/22,172.16.4.0/24,172.16.5.0/29,172.16.5.8/31,172.16.5.11/32,172.16.5.12/30,172.16.5.16/28,172.16.5.32/27,172.16.5.64/26,172.16.5.128/25,172.16.6.0/23,172.16.8.0/21,172.16.16.0/20,172.16.32.0/19,172.16.64.0/18,172.16.128.0/17,172.17.0.0/16,172.18.0.0/15,172.20.0.0/14,172.24.0.0/13,192.168.0.0/16'

This narrows the except set so only 172.16.5.10 (target internal service) falls outside the excepted range, while every other address in 172.16.0.0/12—including the rest of 172.16.5.0/24—remains restricted.

Wrangler on Spark with external object storage

If your cluster is configured with an external s3-compatible object store (e.g. Minio), you must ensure that LRS pods hosting the interactive spark session for Data Wrangling have egress to the object store.

If your cluster uses Cilium CNI, create the following CiliumNetworkPolicy. Make sure to replace DATAROBOT-NAMESPACE and MINIO-API-HOSTNAME-GOES-HERE accordingly.

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: allow-lrs-spark-app-access-to-external-object-store
  namespace: DATAROBOT-NAMESPACE
spec:
  endpointSelector:
    matchLabels:
      datarobot-type: lrs
  egress:
  - toFQDNs:
    - matchName: "MINIO-API-HOSTNAME-GOES-HERE"
    toPorts:
    - ports:
      - port: "443"
        protocol: TCP

Without Cilium, configure a NetworkPolicy to allow egress to the IP addresses that the DNS hostname resolves to. Make sure to replace DATAROBOT-NAMESPACE and X.X.X.X accordingly.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-lrs-spark-app-access-to-external-object-store
  namespace: DATAROBOT_NAMESPACE
spec:
  podSelector:
    matchLabels:
      datarobot-instance-name: datarobot-lrs
      datarobot-lrs-type: spark_app
  egress:
    - ports:
        - protocol: TCP
          port: 443
      to:
        - ipBlock:
            cidr: X.X.X.X/32
  policyTypes:
    - Egress

Set up custom DNS policies

A default configuration would create two policies for DNS access from LRS pods. The first policy is for standard DNS access in k8s cluster which covers all egress from LRSes to kube-dns pods on 53/5353 TCP/UDP ports. And the second policy covers the similar traffic but for OpenShift installations. Here are both policies:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-kube-dns-egress
  namespace: <installation_namespace>
spec:
  egress:
    - ports:
        - port: 53
          protocol: UDP
        - port: 53
          protocol: TCP
        - port: 5353
          protocol: UDP
        - port: 5353
          protocol: TCP
      to:
        - namespaceSelector: {}
          podSelector:
            matchLabels:
              k8s-app: kube-dns
  podSelector:
    matchLabels:
      datarobot-type: lrs
  policyTypes:
    - Egress

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-openshift-dns-egress
  namespace: <installation_namespace>
spec:
  egress:
    - ports:
        - port: 53
          protocol: UDP
        - port: 53
          protocol: TCP
        - port: 5353
          protocol: UDP
        - port: 5353
          protocol: TCP
      to:
        - namespaceSelector: {}
          podSelector:
            matchLabels:
              dns.operator.openshift.io/daemonset-dns: default
  podSelector:
    matchLabels:
      datarobot-type: lrs
  policyTypes:
    - Egress

These policies should be enough to cover most cases. If any other custom policy is required then lrsEgressNetworkPolicies could be overwritten. For example, to define an egress DNS policy to pods with custom-label-name=custom-label-value selector on 53 UDP port in kube-system namespace. The overwritten piece should look like:

operator:
  lrsEgressNetworkPolicies:
    - name: "allow-custom-label-name-egress"
      podSelectorLabel: "custom-label-name"
      podSelectorValue: "custom-label-value"
      namespaceSelector: "kube-system"
      ports:
      - protocol: "UDP"
        port: 53

Note

The lrsEgressNetworkPolicies field can define any custom egress policy for LRSes, not only DNS-specific.