Self-managed installation and maintenance > Installation and configuration guide > DataRobot advanced configuration section > Advanced TLS > Custom Certificate Authority (CA) for TLS

Custom Certificate Authority (CA) for TLS¶

Starting with 10.2 release, DataRobot supports configuring Custom CA bundles for managing the root certificates across the platform. This enables customers with Public Key Infrastructure (PKI) or Private CA issuers to generate TLS certificates for use by DataRobot or to integrate with external systems using them.

How it works¶

A ConfigMap is created within the DataRobot namespace containing all root CA certificates. This ConfigMap contains PEM and JKS formatted certificate bundles which are mounted into the containers for all workloads in the platform, mounting over the system trust store locations.

The volume mounts cover a set of paths specific to Red Hat (UBI8) base images used within the platform, as well as several other popular variants (Debian-based distributions, OpenJDK Java Trust Store, etc.).

This effectively gives the administrator control over configuring which CA root certs to trust and overrides the defaults within the container images.

Pre-requisites¶

Prior to installing DataRobot, you must create a ConfigMap containing all the root CA certificates, both Public CA and Private CA, and any needed intermediate certifcates to be used within the platform. This trust bundle must contain PEM formatted and JKS formatted certificates.
If using a Private Container Registry, your Kubernetes cluster must be configured to pull images. Depending on the container runtime configured, you will need to configure the registry settings with the Private CA bundle to use with TLS. For example, containerd can configure a registry. This configuration needs to be applied to every node in the cluster.

Creating the CA bundle¶

Options:

(Recommended) - install and use Trust Manager for managing trust bundles in Kubernetes clusters.
Manually create and manage the CA bundle

Why Trust Manager?

Component of cert-manager, a CNCF member project and industry standard within the Kubernetes ecosystem
Open source
Easy to install using Helm
Integrates with cert-manager Issuer and ClusterIssuer CA certificates
Automated to keep trusted certs up-to-date and rotate to avoid expiration
Makes it easy to create CA bundles that combine Public CA and Private CA certificates
Supports PEM and JKS formatted certificates
Supports creation of ConfigMap or Secret targets with namespace scope
Supports syncing CA bundles between cluster namespaces

If you choose to not install Trust Manager, it is possible to manually create the CA bundle but it requires being responsible for ensuring certificates are formatted correctly, include all necessary certs, and rotating expired certs.

Using Trust Manager¶

Please see the Trust Manager Installation Guide for the most up-to-date install commands.

We recommend you install Trust Manager into the cert-manager namespace and set app.trust.namespace=cert-manager for the Trust Namespace where Bundles are created and managed.

This allows the cluster admin to restrict access using RBAC permissions for who can create and modify the sources for the CA certificates.

The resulting CA bundle target would be created as a ConfigMap within the DataRobot namespace so that it can be mounted into workloads.

Create a ca-cert-bundle.yaml file:

apiVersion: trust.cert-manager.io/v1alpha1
kind: Bundle
metadata:
  name: ca-cert-bundle
  annotations:
    kubernetes.io/description: Custom CA root cert bundle for DataRobot
spec:
  sources:
  # copies Public CA cert bundle
  - useDefaultCAs: true
  # this secret named "private-ca-secret" must existing within the Trust Namespace (e.g. trust-manager)
  # and contain a PEM formatted CA certificate as `ca.crt` key.
  - configMap:
      # If the k8s API uses a Private CA, this will allow workloads in the platform
      # to work using TLS.
      name: kube-root-ca.crt
      key: "ca.crt"
  - secret:
      # The Private CA certificate only
      name: "private-ca-secret"
      key: "ca.crt"
  target:
    configMap:
      # The ConfigMap will be named tls-ca-cert-bundle with this key
      # containing PEM formatted certificates (not base64 encoded)
      key: "pem_formatted"
    additionalFormats:
      jks:
        # The ConfigMap will also contain this key containing binary encoded
        # Java Key Store (JKS).
        key: "jks_formatted"
      pkcs12:
        key: "pkcs12_formatted"
        profile: "Modern2023"
    namespaceSelector:
      matchExpressions:
        - key: "kubernetes.io/metadata.name"
          operator: In
          values:
          - datarobot # TODO - update this to match the namespace in which DataRobot will be installed!

Create the resource within the cluster in the Trust Namespace:

kubectl apply --namespace cert-manager -f ca-cert-bundle.yaml

Trust Manager will process this Bundle resource request and produce a ConfigMap named ca-cert-bundle within the datarobot namespace with the pem_formatted, jks_formatted, and pkcs12_formatted keys.

See Trust Manager documentation for additional customization options.

Manually creating the CA bundle with a Private CA issuer¶

Pre-requisites:

Linux
openssl command line tool
Java (OpenJDK)
keytool command line tool
python3
pip3 command line tool
certifi python package
kubectl command line tool
Kubernetes user with RBAC access to create a ConfigMap in the DataRobot namespace

Create a custom_ca.sh script:

#!/usr/bin/env bash

####################################################################################################
# Creates TLS certificates and CA root cert bundles
#
# NOTE: certifi is used to provide the Public CA root cert bundle (PEM formatted)
# but the JKS formatted Public CA bundle instead comes from the JDK installation, so these
# may differ.
####################################################################################################

# -------------------------------------------------------------------------------------------------- {: # }
# TODO: configure settings for CA certificate
export COUNTRY=US
export STATE=Massachusetts
export LOCALE=Boston
export ORGANIZATION=DataRobot
export ORGANIZATION_UNIT=DataRobotDev
export COMMON_NAME=Custom CA
export CA_DAYS_TO_EXPIRE=3650
export CA_KEY_PASSWORD=notverysecure

export SERVER_CERT_DAYS_TO_EXPIRE=1095

# -------------------------------------------------------------------------------------------------- {: # }

# make sure JAVA_HOME is set so we can locate cacerts
if [[ -z $JAVA_HOME ]]; then
  export JAVA_HOME=$(java -XshowSettings:properties -version 2>&1 | grep "java.home" | sed 's/.*java.home = \(.*\)/\1/')
fi
export JAVA_CACERTS_PATH=$JAVA_HOME/lib/security/cacerts

# Create a Custom CA for issuing TLS certs
mkdir -p ./certs/ca

if [[ (! -r "./certs/ca") || (! -w "./certs/ca") ]]; then
  echo "Make sure ./certs/ folder is writeable"
fi

rm ./certs/ca/*.jks || true

# Generate a private key for the CA
openssl genpkey \
  -algorithm RSA \
  -aes256 \
  -out ./certs/ca/ca.key \
  -pass env:CA_KEY_PASSWORD

# Generate the CA certificate
openssl req -new -x509 \
  -key ./certs/ca/ca.key \
  -passin env:CA_KEY_PASSWORD \
  -out ./certs/ca/ca.crt \
  -days $CA_DAYS_TO_EXPIRE \
  -subj "/C=${COUNTRY}/ST=${STATE}/L=${LOCALE}/O=${ORGANIZATION}/OU=${ORGANIZATION_UNIT}/CN=${COMMON_NAME}"

# Create combined cert bundle with Public and Private CA
pip3 install certifi
CERTIFI_CA_BUNDLE=$(python3 -c "import certifi; print(certifi.where());")
cat $CERTIFI_CA_BUNDLE ./certs/ca/ca.crt > ./certs/ca/ca-cert-bundle.crt
# remove blank lines and commented lines since its not technically valid PEM format
sed -i.bak '/^\s*$/d; /^\s*#/d' ./certs/ca/ca-cert-bundle.crt && rm ./certs/ca/ca-cert-bundle.crt.bak

# Create JKS formatted CA bundle for java trust store
keytool -import \
  -alias custom-ca \
  -file ./certs/ca/ca.crt \
  -keystore ./certs/ca/ca.jks \
  -storetype JKS \
  -storepass changeit \
  -trustcacerts \
  -noprompt

# copy the JDK's cacerts bundle
cp $JAVA_CACERTS_PATH ./certs/ca/ca-cert-bundle.jks

# make sure the copy is writeable
chmod 666 ./certs/ca/ca-cert-bundle.jks

# Import the Private CA cert to the Java Trust Store (which already contains all Public CA certs)
# NOTE: doesn't work with importing chains, only the first cert is exported to the key store
keytool -import \
  -alias custom-ca \
  -file ./certs/ca/ca.crt \
  -keystore ./certs/ca/ca-cert-bundle.jks \
  -storetype JKS \
  -storepass changeit \
  -trustcacerts \
  -noprompt

# Create PKCS12 formatted CA bundle which provides more future-proof FIPS compliant cryptographic algorithms.
keytool -importkeystore \
    -srckeystore ./certs/ca/ca-cert-bundle.jks \
    -destkeystore ./certs/ca/ca-cert-bundle.p12 \
    -srcstoretype JKS \
    -deststoretype PKCS12 \
    -srcstorepass changeit \
    -deststorepass changeit \
    -noprompt

Run this script to create the Private CA issuer:

chmod +x custom_ca.sh
./custom_ca.sh

This produces the following output:

certs/
  ca/
    ca.key                   - CA private key
    ca.crt                   - CA certificate (PEM format)
    ca.jks                   - CA certificate (JKS format)
    ca-cert-bundle.crt       - CA cert bundle (PEM format)
    ca-cert-bundle.jks       - CA cert bundle (JKS format)
    ca-cert-bundle.p12       - CA cert bundle (PKCS12 format)

Create the ConfigMap for the ca-cert-bundle within the datarobot namespace:

kubectl create configmap ca-cert-bundle \
  --namespace=datarobot \
  --from-file=pem_formatted=./certs/ca/ca-cert-bundle.crt \
  --from-file=jks_formatted=./certs/ca/ca-cert-bundle.jks \
  --from-file=pkcs12_formatted=./certs/ca/ca-cert-bundle.p12

Manually creating TLS certificates using a Private CA issuer¶

Following the steps above to create the Custom CA, you can generate any additional TLS server certificates needed for services within the cluster.

If you already have a Private CA issuer that generates TLS certificates, skip this section.

The following is an illustrative example for Minio.

Make sure the current directory is writable so the ./certs subfolder can be created.

mkdir -p ./certs/minio

Create a server private key server.key:

openssl genrsa -out ./certs/minio/server.key 2048

Create the Certificate Signing Request (CSR) as server.csr:

openssl req -new \
  -key ./certs/minio/server.key \
  -out ./certs/minio/server.csr \
  -nodes \
  -subj "/C=${COUNTRY}/ST=${STATE}/L=${LOCALE}/O=${ORGANIZATION}/OU=${ORGANIZATION_UNIT}/CN=minio"

Verify the CSR:

openssl req -in certs/minio/server.csr -text -noout

Create an extensions file certs/minio/subjects.ext to specify the Subject Alternate Names.

This TLS server certificate would cover hostnames minio and minio.MINIO-NAMESPACE.svc.cluster.local. Make sure to replace MINIO-NAMESPACE with the actual namespace where Minio is installed.

subjectAltName=DNS:minio,DNS:minio.MINIO-NAMESPACE.svc.cluster.local
````

Sign the CSR using the CA and output the TLS certificate `server.crt`:

```shell
openssl x509 -req -in ./certs/minio/server.csr \
  -CA ./certs/ca/ca.crt \
  -CAkey ./certs/ca/ca.key \
  -passin env:CA_KEY_PASSWORD \
  -CAcreateserial \
  -out ./certs/minio/server.crt \
  -days $SERVER_CERT_DAYS_TO_EXPIRE \
  -sha256 \
  -extfile ./certs/minio/subjects.ext

Verify the server cert:

openssl x509 -in certs/minio/server.crt -text -noout

This produces the following output:

certs/
  minio/
    server.key               - Private key for Minio server
    server.crt               - TLS certificate for Minio server
    server.csr               - TLS certificate signing request

You can create a k8s secret for the Minio TLS certificate:

kubectl create secret generic tls-minio-secret \
  --namespace=minio-namespace \
  --from-file=ca.crt=./certs/ca/ca.crt \
  --from-file=tls.crt=./certs/minio/server.crt \
  --from-file=tls.key=./certs/minio/server.key

and then follow the Minio documentation for configuring TLS on Ingress.

Configuring the Custom CA bundle¶

Within the chart values used for helm install:

global:
  customCARootCertBundle:
    configMap: ca-cert-bundle
    pem: pem_formatted
    jks: jks_formatted
    pkcs12: pkcs12_formatted

where ca-cert-bundle is a ConfigMap within the same namespace as the DataRobot installation, containing a key named pem_formatted containing plaintext PEM-formatted root CA certificates and a key named jks_formatted containing a JKS-formatted Java Key Store with all root CA certificates.

Using a ConfigMap is recommended since the CA bundle contains public certs and not any sensitive data.

However, it is also possible to use a Secret for storing the CA bundle:

global:
  customCARootCertBundle:
    secret: ca-cert-bundle
    pem: pem_formatted
    jks: jks_formatted
    pkcs12: pkcs12_formatted

Note the use of secret instead of configMap. This assumes a secret with the specified name exists within the namespace.

Dockerfile modifications for image builds¶

When building container images that perform network operations requiring TLS (e.g., pip install, npm install from Artifactory), Dockerfiles must be modified to use the custom CA certificates mounted by BuildKit.

Add ARG and ENV declarations at the top of your Dockerfile for CA certificate environment variables:

ARG SSL_CERT_FILE
ARG REQUESTS_CA_BUNDLE
ARG CURL_CA_BUNDLE
ARG AWS_CA_BUNDLE
ARG NODE_EXTRA_CA_CERTS

ENV SSL_CERT_FILE=${SSL_CERT_FILE}
ENV REQUESTS_CA_BUNDLE=${REQUESTS_CA_BUNDLE}
ENV CURL_CA_BUNDLE=${CURL_CA_BUNDLE}
ENV AWS_CA_BUNDLE=${AWS_CA_BUNDLE}
ENV NODE_EXTRA_CA_CERTS=${NODE_EXTRA_CA_CERTS}

Add --mount=type=secret,id=ca_cert to any RUN commands that perform network operations:

FROM python:3.11-slim

ARG SSL_CERT_FILE
ARG REQUESTS_CA_BUNDLE
ARG CURL_CA_BUNDLE
ARG AWS_CA_BUNDLE
ARG NODE_EXTRA_CA_CERTS

ENV SSL_CERT_FILE=${SSL_CERT_FILE}
ENV REQUESTS_CA_BUNDLE=${REQUESTS_CA_BUNDLE}
ENV CURL_CA_BUNDLE=${CURL_CA_BUNDLE}
ENV AWS_CA_BUNDLE=${AWS_CA_BUNDLE}
ENV NODE_EXTRA_CA_CERTS=${NODE_EXTRA_CA_CERTS}

RUN --mount=type=secret,id=ca_cert \
    pip install --index-url https://artifactory.example.com/artifactory/api/pypi/python-all/simple requests==2.31.0

CMD ["python", "--version"]

This ensures the CA certificate is available during the build process without being baked into the image layers.

Considerations¶

It is recommended that the CA cert bundle contain ONLY root certs to be trusted and does NOT contain intermediate certs because that complicates rotation of these intermediates.
Application-level TLS is managed separately and does not need to include the sdtk-ca issuer in the Custom CA bundle. Application-level TLS is configured explicitly to mount and use the sdtk-ca CA certificate for constructing TLS requests.
Configuring CA root certificates at the infrastructure level (cluster nodes) is not supported. This would require mounting a host path into containers which is prohibited by Container Security policies.
Configuring additional CA certificates to extend the existing trust store within containers is not supported because that is prohibited by the Container Security policy for read-only file systems and running containers as non-root users. That is what motivated the solution to volume mount over the system trust store instead.
When deploying Long Running Services (LRS) for Custom Models or Custom Apps to namespaces outside the main DataRobot namespace, the CA bundle ConfigMap will be copied into the target namespace if it doesn't already exist. If the CA bundle sources are updated, these copied ConfigMaps will not receive updates automatically and will require manually copying them into all the target namespaces.

Migrating from self-signed certs¶

For customers upgrading from 10.1 or prior releases who were using self-signed certs and disabling TLS verification with ALLOW_SELF_SIGNED_CERTS, we strongly recommend migrating away from that insecure configuration.

Instead, follow the steps above to create a Private CA issuer (either using cert-manager or manually) and regenerate each of the TLS certificates that were previously self-signed.

Since the application was configured to disable TLS verification, rotating the TLS server certs should not disrupt any workloads.

However, to take advantage of the security provided by verifying TLS server certificates, follow the steps above to create the Custom CA bundle and configure the application to utilize this ConfigMap.

Upgrade notes¶

Upgrading to 11.2+ Release¶

For improved FIPS compatibility, PKCS12 formatted trust bundle support was added with "Modern2023" profile containing FIPS 140-2 and FIPS 140-3 compatible algorithms.

Note that the JKS format has been deprecated in favor of PKCS12—JKS uses non FIPS-approved algorithms (PBEwithMD5AndDES). As an alternative, Bouncy Castle FIPS keystore (BCFKS) can be used and is based on JKS-formatted inputs but with PKCS12 stronger encryption (PBES2) algorithms and a broader support than Java.

Ensure the Bundle and its corresponding, generated ConfigMap contain the pkcs12_formatted key.

If using Trust Manager, the Bundle definition must be updated, as indicated in the section above ,to automatically produce the necessary formatted output.
If manually setting up Trust Bundle, follow the steps indicated in the script to create it.

Set the global.customCARootCertBundle.pkcs12 Helm chart value to pkcs12_formatted so that it will use the corresponding ConfigMap key that contains the PKCS12 bundle.

Troubleshooting¶

SSL_ERROR_SSL:tls_process_server_certificate:certificate verify failed¶

This indicates that the TLS server certificate could not be verified.

That means the CA bundle does not contain the CA certificate chain to verify the issuer of the server's certificate.

Things to check:

The server certificate is not expired
The server certificate covers the hostname used in the connection (within Subject Alternate Names section of certificate)
The CA bundle when used directly with openssl verify can verify the server certificate.
If successful, that means the service is misconfigured and not using the correct CA bundle.
If not successful, that means the CA bundle does not have the certificate chain for the issuer. See the Intermediate certs, AIA chasing section below.

SSL_ERROR_SSL: OPENSSL_internal:WRONG_VERSION_NUMBER¶

This likely indicates that the client attempted to connect to the server using an insecure channel (e.g. HTTP instead of HTTPS). Check the URL and port used for the connection.

SSL certificate problem: unable to get local issuer certificate¶

This likely indicates a problem with the CA bundle. See the intermediate certs and AIA chasing sections below.

Intermediate certs¶

The CA bundle should not contain intermediate certs! This makes it difficult to later rotate these CA authorities without downtime. Instead, the CA bundle should contain only root certs to be trusted, and the servers (e.g., Ingress controller with nginx) should use TLS server certs that contain the certificate chain with the leaf node (corresponding to the service) followed in sequential order by the intermediate cert that issued the preceding cert in the chain.

Example:

-----BEGIN CERTIFICATE-----
...leaf node for the server
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
...intermediate 2 that issued lead node cert above
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
...intermediate 1 that issued the intermediate cert 2 above
-----END CERTIFICATE-----

See Validating certificate chain for how to make sure the certificate chain is formatted correctly between subjects and issuers.

Inspecting TLS certificate¶

Using openssl x509 command:

openssl x509 -text -in <cert.pem>

where <cert.pem> corresponds to the TLS certificate in PEM format, which may contain one or more certificates (as a bundle). This can be used for inspecting root, intermediate and leaf certs.

Inspect TLS certificate chain with issuers and subjects¶

openssl x509 -text -in <cert.pem> | grep -E '(Subject|Issuer):'

Verify a certificate file with a CA cert bundle¶

Using openssl verify command:

openssl verify -CAfile <ca_bundle.pem> -show_chain <cert.pem>

where <ca_bundle.pem> contains a PEM formatted file containing one or more certificates for the trust chain, containing the intermediate certs and possibly root certs, and <cert.pem> corresponds to the cert (lead or intermediate) to validate against the chain.

Optionally, you may also specify the -verify_hostname <hostname> argument for the DNS name to check. This would correspond to the server’s hostname that would be serving this TLS leaf certificate, and should specify a Common Name corresponding to the hostname, use a wildcard hostname like *.example.com to cover all subdomains, or specify a list of “Subject Alternative Name” for hostnames covered by the cert. You can inspect the cert using openssl x509 -text to see the Common Name and Subject Alternative Name fields.

Verify a server certificate with a client request¶

Using openssl s_client command:

openssl s_client -connect <host>:<port> -CAfile <ca_bundle.pem> -showcerts

will connect to host on port and verify the server’s TLS certificate using <ca_bundle.pem> containing root certs (and possibly intermediate certs).

The -showcerts will output the TLS certificate provided by the server.

If connecting through port forwarding or a proxy, you may need to set -servername <hostname> to use Server Name Indication (SNI).

Check whether AIA chasing is used¶

Some customers with more sophisticated PKI infrastructure may utilize multiple intermediate certs, and be using AIA TLS extensions to dynamically resolve the intermediate certs. This is problematic because not all clients fully support it, and expect to have the full chain of intermediates and roots available without external requests.

You can use openssl x509 to inspect the cert and look for a section like:

            Authority Information Access:
                CA Issuers - URI:http://example.org/some_intermediate_issuer.crt
                CA Issuers - URI:ldap:///CN=Example Org Issuing Cert in LDAP,CN=AIA,CN=...,DC=...
                OCSP - URI:http://example.org/ocsp

In this case, you will need to download the intermediate certs and either:

Add these intermediate certs to the Custom CA root cert bundle
Add these intermediate certs to the TLS server cert's chain so they are returned by the server explicitly

Where should CA root certs be located in containers?¶

Red Hat distros: /etc/pki/tls/certs/ca-bundle.crt Debian distros: /etc/ssl/certs/ca-certificates.crt DataRobot specific: /datarobot/tls/ca-cert-bundle.crt

Java Key Store (JKS)¶

Java has its own trust store and doesn’t use the system trust store.

The keytool command is provided for this purpose and can be used to update the JKS trust store with additional certificates. See the section above for creating the Custom CA bundle.