Rotate auth-server system secrets¶

The auth-server-system-secrets Secret has two fields—secretsSystem and secretsCookie. Each field holds one or more passwords, separated by commas. The first password in a field is the main one and is used to encrypt. All passwords in a field are used to decrypt.

If the auth server runs in FIPS 140-3 mode, the Secret has a third field, secretsPagination. It only protects the tokens the auth server uses to page through long lists, so rotate it the same way as secretsCookie. Replace its contents with the new password. The only cost is that list requests that are already in progress have to start again from the first page.

Rotate these passwords if a main password is insecure (for example, the shared default insecure-default-must-be-changed that earlier releases could set), if you suspect a password was exposed, or if your security policy requires it. A secure main password is a random 44-character value.

The two fields of a Secret are rotated differently:

• secretsCookie: Replace the field’s contents with the new password only.
• secretsSystem: Add the new password at the front of the field and keep the existing passwords after it.

Back up the Secret before you change it:

kubectl -n ${NAMESPACE} get secret auth-server-system-secrets -o yaml > auth-server-system-secrets-backup.yaml

Where Secret is managed¶

The steps below patch the Secret directly in the cluster. If your values file sets auth-token-customization.hydra.secret.secretsSystem and secretsCookie, or an external secret store supplies the Secret, make the change there instead. Otherwise, the next upgrade or sync reverts it.

Before you rotate, check for a Secret named auth-server-hydra-secrets, which releases prior to DataRobot 10.2 created. On every upgrade, the chart overwrites auth-server-system-secrets with the values from auth-server-hydra-secrets, which undoes your rotation. If auth-server-hydra-secrets still exists, back it up and then delete it.

1. Check the content of both fields:

   SECRET=$(kubectl -n ${NAMESPACE} get secret auth-server-system-secrets -o yaml)
   for i in secretsCookie secretsSystem; do
     DATA=$(echo "${SECRET}" | grep "$i:" | awk '{print $2}' | base64 -d)
     echo "$i: ${DATA}"
   done
   

   Warning

   This command prints the secrets in plain text. Make sure you are not capturing the output anywhere that persists.

   Example output:

   secretsCookie: insecure-default-must-be-changed
   secretsSystem: "insecure-default-must-be-changed","old-insecure-default-must-be-changed"
   

   In this example, the main password in both fields is not secure, so reset secretsCookie and rotate secretsSystem. If a main password is already a long random value, that field needs no change unless you are rotating it for another reason, such as suspected exposure.

2. Reset secretsCookie. The command generates a random password:

   kubectl -n ${NAMESPACE} patch secret auth-server-system-secrets \
     --type=merge \
     -p "{\"data\":{\"secretsCookie\":\"$(openssl rand -hex 22 | tr -d '\n' | base64)\"}}"
   

3. Rotate secretsSystem. The commands generate a new password and add it in front of the existing passwords:

   OLD_VALUE=$(kubectl -n ${NAMESPACE} get secret auth-server-system-secrets \
     -o jsonpath='{.data.secretsSystem}' | base64 -d)
   if [ -n "$OLD_VALUE" ]; then
     NEW_SECRET=$(openssl rand -hex 22)
     NEW_VALUE="${NEW_SECRET},${OLD_VALUE}"
     kubectl -n ${NAMESPACE} patch secret auth-server-system-secrets --type=merge \
       -p "{\"data\":{\"secretsSystem\":\"$(printf '%s' "$NEW_VALUE" | base64 | tr -d '\n')\"}}"
   else
     echo "secretsSystem is empty or unreadable; nothing changed"
   fi
   

   Do not remove the old passwords

   Keep the existing passwords in secretsSystem. The auth server still needs them to read data encrypted before the rotation; removing them leaves it unable to issue or validate tokens. Removing insecure passwords is not recommended.

4. Restart the auth server. It reads these values only on startup. The restart command finds the auth server Deployment with jq:

   HYDRA=$(kubectl -n ${NAMESPACE} get deployment -o json \
     | jq -r '.items[] | select(any(.spec.template.spec.containers[]; .name == "hydra")) | .metadata.name')
   kubectl -n ${NAMESPACE} rollout restart deployment "$HYDRA"
   kubectl -n ${NAMESPACE} rollout status deployment "$HYDRA"
   

5. Run the check from Step 1 again. secretsCookie should show a single 44-character password; secretsSystem should show a new 44-character password first, followed by the past passwords.