DataRobot ingress reference settings¶

Ingress in Kubernetes is an API object that manages external access to the services in a cluster, typically HTTP. For the Ingress resource to work, the cluster must have an ingress controller running. Cluster administrators are expected to select ingress controllers according to their requirements. There are dozens of ingress controllers available. Some controllers are maintained by the Kubernetes team (e.g., AWS, GCE, and Nginx controllers), while others are provided by third-party vendors.

DataRobot is compatible with a wide range of ingress controllers. Nginx Ingress, HAProxy Ingress, and OpenShift Ingress have been fully validated by the DataRobot team, but other controllers also works as long as some basic requirements are met. These requirements are documented below.

Nginx Ingress deprecation¶

Kubernetes is retiring the community Ingress NGINX project, so we recommend looking at alternatives. If you want to keep using Ingress NGINX, Chainguard maintains a CVE-free fork that works as a drop-in replacement for the deprecated version. See the announcement.

DataRobot includes selected Chainguard Ingress NGINX images with the product, and we will keep shipping up-to-date, maintained versions in future releases. This means you can stay on a maintained Ingress NGINX by making a simple configuration change. Starting with DataRobot 11.6.0, you can use: datarobot/mirror_chainguard_datarobot.com_ingress-nginx-controller-fips:v1.14.3.

You can retrieve the tag of the mirror_chainguard_datarobot.com_ingress-nginx-controller-fips image that was bundled with a given DataRobot version using the following command:

INGRESS_NGINX_TAG=$(helm show chart oci://registry-1.docker.io/datarobot/datarobot-prime --version <DataRobot-version> | grep mirror_chainguard_datarobot.com_ingress-nginx-controller-fips | cut -d: -f3)

To patch an existing ingress-nginx-controller deployment with Chainguard's image:

kubectl set image deployment/ingress-nginx-controller -n <namespace> controller=docker.io/datarobot/mirror_chainguard_datarobot.com_ingress-nginx-controller-fips:$INGRESS_NGINX_TAG

If Ingress NGINX was installed via Helm chart, the chart can be updated as follows:

helm upgrade <release-name> -n <release-namespace> ingress-nginx/ingress-nginx \
  --reuse-values \
  --set 'controller.image.registry=docker.io' \
  --set 'controller.image.image=datarobot/mirror_chainguard_datarobot.com_ingress-nginx-controller-fips' \
  --set 'controller.image.tag=$INGRESS_NGINX_TAG' \
  --set 'controller.image.digest=null'

DataRobot also provides the Ingress NGINX helm chart (maintained by Chainguard), which can be pulled from oci://registry-1.docker.io/datarobot/ingress-nginx.

Nginx Ingress reference settings¶

This section details the way DataRobot sets up the ingress-nginx controller using a helm chart. Client clusters may not use these exact settings, as each client has its own administrator. This is provided as a reference and starting point for cluster admins who want to see how DataRobot does it.

Kubernetes provides extensive documentation about the controller here: https://github.com/kubernetes/ingress-nginx/tree/main/docs. This location includes user guides, examples, hardening recommendations, and many other details.

AWS¶

Nginx Ingress Controller release can be configured with the below values.yaml.

controller:
  image:
    registry: docker.io
    image: datarobot/mirror_chainguard_datarobot.com_ingress-nginx-controller-fips
    tag: $INGRESS_NGINX_TAG
    digest: null
  admissionWebhooks:
    enabled: false
  config:
    use-proxy-protocol: "true"
    worker-processes: "2"
  extraArgs:
    annotations-prefix: nginx.ingress.kubernetes.io
  ingressClassResource:
    default: true
  service:
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: <a list of DR internal tags>
      service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
      service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "3600"
      service.beta.kubernetes.io/aws-load-balancer-internal: "true"
      service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*'
      service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:us-east-1:DR_AWS_ACCOUNT_NUMBER:certificate/DR_ACM_CERTIFICATE_ID
      service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https
      service.beta.kubernetes.io/aws-load-balancer-subnets: <comma-separated list of AWS subnets IDs>
    ports:
      http: 80
    targetPorts:
      https: http
    type: LoadBalancer
defaultBackend:
  digest: null
  enabled: true
imagePullSecrets: []

The helm command to install it's something like this, assuming the above values are inside a file named ingress-nginx_values.yaml:

helm upgrade --install ingress-nginx oci://registry-1.docker.io/datarobot/ingress-nginx --values ingress-nginx_values.yaml --namespace ingress-nginx --debug

AWS Network load balancer¶

To use AWS Network Load Balancer as the load balancer, AWS Load Balancer Controller must be installed. Nginx Ingress Controller release can be then configured with the below values.yaml.

controller:
  image:
    registry: docker.io
    image: datarobot/mirror_chainguard_datarobot.com_ingress-nginx-controller-fips
    tag: $INGRESS_NGINX_TAG
    digest: null
  admissionWebhooks:
    enabled: false
  config:
    use-forwarded-headers: "true"
    use-gzip: "true"
    use-proxy-protocol: "true"
  service:
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "60"
      service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
      service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
      service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*'
      service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
      service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https
      service.beta.kubernetes.io/aws-load-balancer-subnets: <comma-separated list of AWS subnets IDs>
      service.beta.kubernetes.io/aws-load-balancer-type: external
    externalTrafficPolicy: Local

Azure¶

Nginx Ingress Controller release can be configured with the below values.yaml.

controller:
  image:
    registry: docker.io
    image: datarobot/mirror_chainguard_datarobot.com_ingress-nginx-controller-fips
    tag: $INGRESS_NGINX_TAG
    digest: null
  admissionWebhooks:
    enabled: false
  service:
    annotations:
      service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: /healthz
      service.beta.kubernetes.io/azure-load-balancer-internal: "true"
      service.beta.kubernetes.io/azure-load-balancer-tcp-idle-timeout: "30"
    externalTrafficPolicy: Local
    type: LoadBalancer
  watchIngressWithoutClass: true

The helm command to install it's something like this, assuming the above values are inside a file named ingress-nginx_values.yaml:

helm upgrade --install ingress-nginx oci://registry-1.docker.io/datarobot/ingress-nginx --values ingress-nginx_values.yaml --namespace ingress-nginx --debug

Azure ingress-terminated TLS with cert-manager¶

See the cert-manager and LetsEncrypt for Ingress Terminated TLS section for details on the cert-manager and LetsEncrypt settings DataRobot uses for ingress terminated TLS.

Google cloud¶

Nginx Ingress Controller release can be configured with the below values.yaml.

controller:
  image:
    registry: docker.io
    image: datarobot/mirror_chainguard_datarobot.com_ingress-nginx-controller-fips
    tag: $INGRESS_NGINX_TAG
    digest: null
  admissionWebhooks:
    enabled: false
  service:
    externalTrafficPolicy: Local
    type: LoadBalancer
  watchIngressWithoutClass: true

The helm command to install it's something like this, assuming the above values are inside a file named ingress-nginx_values.yaml:

helm upgrade --install ingress-nginx oci://registry-1.docker.io/datarobot/ingress-nginx --values ingress-nginx_values.yaml --namespace ingress-nginx --debug

Google cloud ingress-terminated TLS with cert-manager¶

See the cert-manager and LetsEncrypt for Ingress Terminated TLS section for details on the cert-manager and LetsEncrypt settings DataRobot uses for ingress terminated TLS. Also, if Google provides a certificate service, it's not possible to integrate it with the Nginx ingress controller.

Compatibility with Ingress Controllers¶

DataRobot leverages the standard Kubernetes Ingress and doesn't impose strict dependencies on particular Ingress Controllers. Nevertheless, certain DataRobot HTTP endpoints come with specific non-functional prerequisites, such as maximum body size or request timeout. Administrators may need to adjust either DataRobot's ingress annotations or the ingress controller configuration to meet these requirements effectively.

DataRobot is shipped with corresponding annotations for the following ingress controllers out of the box:

• Nginx Ingress
• HAProxy Ingress
• OpenShift Ingress

For other Ingress Controllers, administrators must ensure the following requirements are met:

1. The Ingress Controller must support the websockets protocol.
2. The apigateway Ingress should support requests up to 1124M in size that take up to 600 seconds.
3. The core Ingress should support requests up to 20G in size that take up to 600 seconds.
4. The nbx-ingress Ingress should support requests up to 1124M in size.
5. The nbx-websockets Ingress should support requests up to 15M in size that take up to 3600 seconds.
6. If the nbx-websockets-notebooks-deployment deployment is configured with two or more replicas (default: 1), "sticky sessions" should be implemented at the ingress side.

OpenShift ingress configuration example¶

DataRobot supports OpenShift Ingress; please use the following values.yaml as a starting point:

global:
  domain: "datarobot.apps.example.com"
  ingressClassName: openshift-default
  ingress:
    annotations:
      route.openshift.io/termination: edge

DataRobot's domain should be a subdomain of the domain name configured at the ingress controller side (please see the OpenShift Ingress documentation for more details).

Terminate TLS at the ingress controller by adding the annotation route.openshift.io/termination: edge. Additional annotations supported by OpenShift Ingress are documented on the Route configuration page in OCP docs.

HAProxy Ingress configuration example¶

The following values.yaml snippet implements the requirements above for HAProxy Ingress. This configuration is already shipped with the app and should be treated as informational; it can be used as a reference for configuring other ingress controllers.

global:
  ingressClassName: haproxy
  ingress:
    annotations: {}  # additional global ingress annotations go here
apigateway:
  ingress:
    annotations:
      haproxy-ingress.github.io/proxy-body-size: "1124000000"
      haproxy-ingress.github.io/timeout-server: "600s"
      haproxy-ingress.github.io/timeout-server-fin: "600s"
      haproxy-ingress.github.io/timeout-client: "600s"
      haproxy-ingress.github.io/timeout-client-fin: "600s"
      haproxy-ingress.github.io/timeout-http-request: "600s"
core:
  ingress:
    annotations:
      haproxy-ingress.github.io/proxy-body-size: "20000000000"
      haproxy-ingress.github.io/timeout-server: "600s"
      haproxy-ingress.github.io/timeout-server-fin: "600s"
      haproxy-ingress.github.io/timeout-client: "600s"
      haproxy-ingress.github.io/timeout-client-fin: "600s"
      haproxy-ingress.github.io/timeout-http-request: "600s"
nbx-ingress:
  ingress:
    annotations:
      haproxy-ingress.github.io/proxy-body-size: "1124000000"
nbx-websockets:
  ingress:
    annotations:
      haproxy-ingress.github.io/proxy-body-size: "15000000"
      haproxy-ingress.github.io/timeout-server: "3600s"
      haproxy-ingress.github.io/timeout-server-fin: "3600s"
      haproxy-ingress.github.io/timeout-client: "3600s"
      haproxy-ingress.github.io/timeout-client-fin: "3600s"
      haproxy-ingress.github.io/timeout-http-request: "3600s"
      cookie-persistence: "nbx-websockets-session-persistence-cookie"

Istio VirtualService¶

DataRobot supports Istio as an alternative to a traditional Kubernetes Ingress controller. When enabled, DataRobot renders Istio VirtualService resources instead of Kubernetes Ingress resources, routing external traffic through the Istio ingress gateway.

Prerequisites¶

• Istio control plane (istiod) installed in the cluster.
• An Istio ingress gateway deployed and an Istio Gateway resource configured to accept HTTPS traffic.
• The DataRobot TLS certificate Secret accessible to the ingress gateway pod (see TLS certificate management).

Enabling VirtualService mode¶

Set global.ingress.type to virtualservice in your values.yaml:

global:
  ingress:
    type: virtualservice
  istio:
    gateway: datarobot-gw  # name of your Istio Gateway resource

global.istio.gateway accepts a plain name or a namespace-qualified reference such as istio-system/my-gateway. It defaults to datarobot-gw.

Gateway resource¶

Create an Istio Gateway resource that binds to the ingress gateway and terminates TLS using the DataRobot certificate. The namespace should match wherever you deploy the ingress gateway — see TLS certificate management for both deployment options.

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: datarobot-gw
  namespace: <ingress-gateway-namespace>
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 443
        name: https
        protocol: HTTPS
      tls:
        mode: SIMPLE
        credentialName: datarobot-acme-cert
      hosts:
        - "your.domain.com"

TLS certificate management¶

The Istio ingress gateway reads TLS certificates from the namespace where the gateway pod runs. The credentialName in the Gateway resource must match a Secret in that same namespace. DataRobot creates a certificate named datarobot-acme-cert in the application namespace (for example, dr-app) during installation. There are two approaches for making this Secret accessible to the gateway.

Option 1: Deploy the ingress gateway in the DataRobot namespace (recommended)¶

Install the ingress gateway directly into the DataRobot application namespace using Istio's Helm charts, which separate the control plane from the gateway. The gateway pod reads secrets from its own namespace, so datarobot-acme-cert is available without any additional steps.

# Control plane in istio-system
helm install istiod istio/istiod -n istio-system --wait

# Ingress gateway co-located with DataRobot
helm install istio-ingressgateway istio/gateway -n <datarobot-namespace>

The Gateway resource and datarobot-acme-cert Secret can both remain in the DataRobot namespace with no cross-namespace secret access required.

Option 2: Mirror the certificate to istio-system using Reflector¶

If the ingress gateway must remain in istio-system (for example, when using a shared gateway across multiple applications), use Reflector to automatically mirror datarobot-acme-cert from the DataRobot namespace into istio-system.

Install Reflector:

helm upgrade --install reflector \
  oci://ghcr.io/emberstack/helm-charts/reflector \
  --namespace kube-system

Annotate the DataRobot certificate Secret to enable mirroring:

kubectl annotate secret datarobot-acme-cert -n <datarobot-namespace> \
  "reflector.v1.k8s.emberstack.com/reflection-allowed=true" \
  "reflector.v1.k8s.emberstack.com/reflection-allowed-namespaces=istio-system" \
  "reflector.v1.k8s.emberstack.com/reflection-auto-enabled=true"

Reflector keeps a copy of datarobot-acme-cert in istio-system in sync with the source. Deploy the Gateway resource in istio-system in this case.

AWS NLB configuration¶

When running on AWS with the AWS Load Balancer Controller, annotate the istio-ingressgateway Service to provision a Network Load Balancer in TCP passthrough mode. The NLB must forward raw TCP traffic to Istio so that Istio terminates TLS. A TLS listener on the NLB terminates TLS before it reaches Istio, causing a handshake failure.

apiVersion: v1
kind: Service
metadata:
  name: istio-ingressgateway
  namespace: <ingress-gateway-namespace>
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: external
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
    service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
    # Keep connections open long enough for long-running WebSocket sessions
    service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "3600"
    # Distribute traffic evenly across nodes in all availability zones
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
    # Health check against Istio's dedicated readiness port/path instead of a data port
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: HTTP
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: "15021"
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-path: /healthz/ready

Sticky sessions¶

Istio does not provide sticky sessions via VirtualService alone. Session affinity requires a DestinationRule with a consistentHash load balancing policy. DataRobot's notebook execution service requires session affinity so that WebSocket connections are consistently routed to the same backend pod. Create a DestinationRule in the DataRobot namespace targeting the relevant service:

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: nbx-websockets
  namespace: <datarobot-namespace>
spec:
  host: nbx-websockets
  trafficPolicy:
    loadBalancer:
      consistentHash:
        httpCookie:
          name: INGRESSCOOKIE
          ttl: 0s