Upgrade from DataRobot v11.x to the latest release¶

This section provides a checklist of tasks that are required when upgrading from any DataRobot 11.x version to the latest release version.

Upgrade to latest release¶

In the latest release, the PCS and DataRobot charts have been consolidated into a single chart. This change eliminates code duplication and simplifies the installation process.

Update values¶

The latest DataRobot release introduces minor changes to the YAML configuration schema. Therefore, adjust the existing configuration to match the new schema.

Minimal values files¶

For platform-specific configuration examples, refer to the minimal values files located within the datarobot-prime/override Helm chart artifact directory.

tar xzf datarobot-prime-X.X.X.tgz
cd datarobot-prime/override 

Use the current values_dr.yaml file as a reference when populating the minimal values files.

Upgrade GenAI¶

Enable the LLM gateway service in values_dr.yaml. For details, see the Generative AI documentation.

Move forward-proxy settings to global.proxy¶

If your cluster reaches the internet only through a forward proxy, replace the hand-written proxy environment variables in global.extraEnvVars with the global.proxy block. DataRobot then injects HTTP_PROXY, HTTPS_PROXY, and NO_PROXY (in both upper and lower case) into every platform service pod, and supplies you with the internal-service NO_PROXY defaults, so you no longer need to maintain the long exclusion list of DataRobot service names:

global:
  proxy:
    httpProxy: http://proxy.example.com:3128
    httpsProxy: http://proxy.example.com:3128
    noProxy:
      - .dr-app                # the namespace where DataRobot is installed
      - datarobot.example.com  # the global.domain host
      - 10.100.0.1             # ClusterIP of the Kubernetes API (kubectl get svc kubernetes) 

noProxy lists destinations that pods must reach directly, because the forward proxy cannot route to them or should not carry that traffic. The built-in defaults already exclude the cluster-internal suffixes and the DataRobot service names, for example:

# built-in NO_PROXY defaults (excerpt)
- .svc
- .cluster.local
- datarobot-public-api
- pcs-rabbitmq
- ... 

However, the defaults cannot predict values that are specific to your environment—that is why the three entries above are needed: services address each other as <service>.<namespace>, so the namespace entry keeps that traffic inside the cluster; services also call back into DataRobot through its global.domain URL, which the proxy typically cannot route back inside; and pods reach the Kubernetes API server by its ClusterIP, which is unreachable from the proxy. A missing entry sends that traffic to the proxy, where it is denied or dropped—this usually surfaces as connection timeouts or readiness-probe failures rather than an explicit proxy error.

Entries in noProxy are appended to the built-in defaults—they are not a replacement for them. Only list what is specific to your environment. Remove the old HTTP_PROXY, HTTPS_PROXY, and NO_PROXY entries from global.extraEnvVars (and from any per-service section such as core.common_env_vars) so that the two mechanisms cannot disagree.

For a full description of proxy behavior, including which destinations must bypass the proxy, see Integrate with customer HTTP proxy.

Set the air-gap flag for offline installations¶

If your cluster cannot download DataRobot's internet-hosted assets during installation, set the platform-wide air-gap flag:

global:
  airgap:
    enabled: true 

This flag and global.proxy are independent, and a restricted network commonly needs both. The flag states that internet-hosted assets are unreachable, so services must use their offline paths; global.proxy states how egress happens when it is permitted at all. Set only the flag for a fully offline cluster, set only the proxy when everything remains reachable through it, and set both when a forward proxy allows a curated set of destinations, such as package indexes, but not the DataRobot CDN.

When this flag is set, services that would otherwise download assets from the internet during installation switch to their offline behavior. In particular, the Generative AI static-file and execution-environment migration jobs, which fetch model weights from the DataRobot CDN, are skipped rather than failing the release, and the Memory Service reads embedding models from blob storage instead of the CDN. You no longer need to disable those jobs individually, although an explicit setting still takes precedence in either direction if your environment permits egress to a specific host. See Static files migration and Memory Service.

Review external PCS PostgreSQL extensions¶

If you use external PCS and plan to enable the Memory Service (available in 11.9 and later), your external PostgreSQL instance must provide the pgvector and pg_trgm extensions. On AWS RDS and Google Cloud SQL these are available by default and are created automatically when DataRobot connects as a superuser. Two cases require manual action before you enable the service:

• Azure Database for PostgreSQL blocks CREATE EXTENSION until VECTOR and PG_TRGM are allowlisted in the server parameters.
• Limited-permission database users that cannot run CREATE EXTENSION require the extensions to be created manually.

For the exact steps, see the cloud-specific guide: AWS, Azure, or Google Cloud.

Review external Elasticsearch configuration¶

If Datavolt (LLM monitoring) or the datarobot-otel-collector chart is configured to use an external Elasticsearch cluster instead of the one provided by PCS, review those values before upgrading. Datavolt version 11.9.58 and later reads the connection URL from component.app.elasticsearch and the credentials from the envs map; the top-level DATAVOLT_ELASTIC_SEARCH_URL and DATAVOLT_ELASTIC_SEARCH_USERNAME values used by earlier 11.x releases are deprecated. The OTel collector follows the same pattern through its top-level elasticsearch key and envs map. For the exact values and secret configuration, see:

• Configure Datavolt for external Elasticsearch
• Configure the DataRobot OTel Collector for external Elasticsearch

Migrate Workload API data from the admin database¶

This applies only if you used the Workload API during its Private Preview (in 11.9 or 11.10), run external PCS with a full MONGODB_URI override, and are upgrading from those releases. If that override's connection string pointed at the admin database, Workload API data was written to admin; run the one-off migrateFromAdmin job during the upgrade to relocate it. See Migrate Workload API data from the admin database. Most installations are unaffected and can skip this.

Review auth server system secrets¶

Earlier releases could create the auth-server-system-secrets Secret with a shared default password, such as insecure-default-must-be-changed, in both of its fields, secretsSystem and secretsCookie. A default password is not safe and must not be used to encrypt. Because the chart reuses the Secret already in the cluster, upgrading does not replace it.

After the upgrade, follow the steps provided in Rotate auth server system secrets. It starts with a check that shows whether either field still uses the default.

New installations are unaffected. On a standard helm install the chart now generates these keys and reuses them on later upgrades. ArgoCD and other helm template-based workflows must supply them explicitly—see Auth server system secrets.

Review file storage prefix configuration¶

In earlier releases, the datarobot-modeling-envvars ConfigMap ignored global.filestore.environment.FILE_STORAGE_PREFIX and fell back to the release namespace name. The ConfigMap now resolves FILE_STORAGE_PREFIX in the following order:

1. core.config_env_vars.FILE_STORAGE_PREFIX, if set.
2. global.filestore.environment.FILE_STORAGE_PREFIX, if set.
3. The release namespace name.

As a result, the monolith and services that read the ConfigMap (such as Buzok) now use the same prefix as the datarobot-filestore Secret.

Action is required only if both of the following apply to your installation:

• global.filestore.environment.FILE_STORAGE_PREFIX is set to a value other than your namespace name.
• core.config_env_vars.FILE_STORAGE_PREFIX is not set.

In that case, the effective prefix changes on upgrade. Objects stored under the old path are not migrated and will not be found after the upgrade. No data is deleted.

To check whether you are affected, compare the prefix in the ConfigMap with the prefix in the Secret:

kubectl -n ${NAMESPACE} get cm datarobot-modeling-envvars -o jsonpath='{.data.FILE_STORAGE_PREFIX}'
kubectl -n ${NAMESPACE} get secret datarobot-filestore -o jsonpath='{.data.FILE_STORAGE_PREFIX}' | base64 -d 

If the values differ, you are affected. To keep your current storage layout, set core.config_env_vars.FILE_STORAGE_PREFIX to the value currently in the ConfigMap before upgrading. This value takes precedence over the global value:

core:
  config_env_vars:
    FILE_STORAGE_PREFIX: <existing-prefix> 

Review custom CA bundle configuration¶

If you are using Trust Manager and deploying custom CA bundles to manage root certificates across the DataRobot platform, verify the structure of the ca-cert-bundle.yaml file. For details, see configuring Custom CA upgrade notes.

Review notebook configuration¶

The naming for notebook chart values was updated. This change may impact your process. For detailed instructions, see the Notebooks Upgrade Guide.

Grant Artifact Registry delete to the image build service (GKE)¶

This applies to installations on Google Cloud whose image build service pushes to Google Artifact Registry. Earlier releases asked for roles/artifactregistry.writer on the service account the image build service authenticates with (DOCKERHUB_PASSWORD in the build-service values). Writer allows push and pull but not delete, so the image cleanup that removes images of deleted custom models, applications, jobs, and environments fails against Artifact Registry with 403 FORBIDDEN and never frees registry storage.

Before upgrading, grant that service account roles/artifactregistry.repoAdmin on the repository (writer plus delete; it does not allow creating or deleting repositories or changing their IAM policy):

gcloud artifacts repositories add-iam-policy-binding ${REPOSITORY} \
  --project=${PROJECT_ID} --location=${REGION} \
  --member="serviceAccount:${SERVICE_ACCOUNT_EMAIL}" \
  --role="roles/artifactregistry.repoAdmin" 

If the repository binding is managed by Terraform with the DataRobot module, update to a module release that grants repoAdmin instead of writer and apply. No build-service values change is required. See GKE values.

Application upgrade steps¶

Set DataRobot namespace¶

export NAMESPACE="DATAROBOT_NAMESPACE" 

Delete Elasticsearch certificate¶

Skip this step when upgrading from 11.0. On 11.0, nothing recreates this secret until after the Elasticsearch upgrade, so restarted Elasticsearch pods cannot start. The upgrade replaces the 11.0 certificate automatically and, after the upgrade, restarts all Elasticsearch pods together when the certificate authority changed. When upgrading from 11.1 or later, delete the certificate:

kubectl -n ${NAMESPACE} delete secret pcs-elasticsearch-master-crt 

Temporarily disable PostgreSQL HA¶

If High Availability (HA) is configured for PostgreSQL, reduce the PostgreSQL replicaCount to prevent potential conflicts during the upgrade:

postgresql-ha:
    postgresql:
        replicaCount: 1 

Verify that the current PostgreSQL primary node is pcs-postgresql-0:

kubectl exec -i -t -n ${NAMESPACE}  pcs-postgresql-0 -c postgresql -- bash -c "/opt/bitnami/scripts/postgresql-repmgr/entrypoint.sh repmgr cluster show -f /opt/bitnami/repmgr/conf/repmgr.conf --compact" 

Scale down the PostgreSQL StatefulSet to 1 replica:

kubectl scale sts pcs-postgresql --replicas=1 -n ${NAMESPACE} 

Apply API gateway CRDs¶

The latest release introduces the API gateway as a subchart of the datarobot-prime chart. Helm only installs the custom resource definitions (CRDs) located in a chart's crds/ directory during the initial installation, not during a helm upgrade. Therefore, apply the API gateway CRDs manually before running the Helm upgrade to ensure they are present in the cluster.

Extract the datarobot-prime chart artifact and apply the API gateway CRDs from the bundled subchart:

tar xzf datarobot-prime-X.X.X.tgz
kubectl apply --server-side --force-conflicts -f datarobot-prime/charts/api-gateway/crds/ 

Upgrade Helm to latest version¶

helm upgrade --install dr datarobot-prime-X.X.X.tgz \
--namespace ${NAMESPACE} \
--values values_dr.yaml \
--debug \
--timeout 2h 

Post-upgrade tasks¶

After your helm upgrade command completes successfully and the application pods are running, perform any applicable final steps from the sections below to complete the upgrade process.

Scale PostgreSQL replica¶

If High Availability (HA) for PostgreSQL is required, scale the PostgreSQL replica count back up to the desired value to restore HA functionality.

kubectl scale statefulset pcs-postgresql --replicas=3 -n ${NAMESPACE} 

postgresql-ha:
    postgresql:
        replicaCount: 3 

Upgrade GenAI¶

Every installation that includes a GenAI subsystem requires a static files migration. This is done automatically for installations with internet access; however, offline clusters must do this manually:

• Migrate static files.

Execution environments are installed by a Helm hook job pulling a pre-built image and require no manual steps, even offline (see Execution environments).

Delete orphaned OTEL collector volumes¶

When datarobot-otel-collector upgrades, it creates new PersistentVolumeClaims (PVCs) with a -v2 suffix. The original PVCs (without the -v2 suffix) are no longer used and should be deleted. These volumes only hold a cache for otel collector metrics/traces/logs and are safe to delete after the upgrade.

List the existing PVCs to identify the ones without the v2 suffix:

kubectl get pvc -n ${NAMESPACE} | grep -i otel
exporter-persistent-storage-datarobot-otel-collector-exporter-0      Bound     pvc-dd4544f4-ff3e-4776-934b-26d7e3a5e9f0   1Gi        RWO            gp3-csi        <unset>                 179d
exporter-persistent-storage-datarobot-otel-collector-exporter-1      Bound     pvc-41755fd6-5b06-482e-9d9f-3bf9926de8be   1Gi        RWO            gp3-csi        <unset>                 48d
exporter-persistent-storage-datarobot-otel-collector-exporter-v2-0   Bound     pvc-9d11d327-2436-4c63-9189-2a240d4db68e   5Gi        RWO            gp3-csi        <unset>                 43h
exporter-persistent-storage-datarobot-otel-collector-exporter-v2-1   Bound     pvc-c78c7e1b-9a37-4104-b67c-ec1c9dbbaf8e   5Gi        RWO            gp3-csi        <unset>                 43h 

Delete each orphaned PVC, the ones without v2, the originals that should be 1Gi in size:

kubectl delete pvc -n ${NAMESPACE} exporter-persistent-storage-datarobot-otel-collector-exporter-0
kubectl delete pvc -n ${NAMESPACE} exporter-persistent-storage-datarobot-otel-collector-exporter-1 

The corresponding PersistentVolumes (PVs) are automatically reclaimed once the PVCs are deleted. Verify that no orphaned PVs remain:

kubectl get pv | grep -i otel 

Delete orphaned notebook network policies¶

This step applies if both of the following are true:

• You upgraded from a release earlier than v11.13 that had network policies enabled (global.networkPolicy.enabled: true).
• Network policies are disabled after the upgrade..

Earlier releases annotated notebook session network policies with helm.sh/resource-policy: keep, so Helm doesn’t remove them when it stops rendering them. Those releases also included a service-wide notebooks policy, nbx-internal, that allowed traffic between notebook services. In v11.13, that policy has a new name, so the upgrade removes nbx-internal. Without it, the leftover session policies block traffic from nbx-ingress to notebook and codespace sessions.

While these policies remain, every codespace session request hangs for roughly 60 seconds and then fails with 412 "Unable to access notebook's environment". Codespaces can’t be opened, and the codespace file upload step of the post-install checks fails.

To list the leftover notebook network policies, run:

kubectl get networkpolicy -n ${NAMESPACE} | grep -E 'notebook-(resource|operator)' 

Delete the orphaned policies:

kubectl delete networkpolicy -n ${NAMESPACE} --ignore-not-found \
  notebook-resource-runner-network-policy \
  notebook-resource-runner-internet-network-policy \
  notebook-resource-kernel-network-policy \
  notebook-resource-kernel-internet-network-policy \
  notebook-operator-network-policy 

If your cluster runs Cilium, also delete the orphaned kernel deny policy:

kubectl delete ciliumnetworkpolicy -n ${NAMESPACE} --ignore-not-found notebook-resource-kernel-deny-network-policy 

Verify that no notebook network policies remain:

kubectl get networkpolicy -n ${NAMESPACE} | grep -E 'notebook-(resource|operator)'