Skip to content

Backup inference servers

DataRobot uses inference servers as part of its serverless predictions offering. If the associated serverless resources are not backed up, customers will be unable to make predictions using model deployments after the restore.

To ensure inference servers are properly backed up, you must include the following Kubernetes Custom Resource Definitions (CRDs) in your backup procedure:

  • executionenvironments.predictions.datarobot.com
  • inferenceservers.predictions.datarobot.com

Prerequisites

  • Utility kubectl of version 1.23 is installed on the host where backup will be created

  • Utility kubectl is configured to access the Kubernetes cluster where the DataRobot application is running, verify this with kubectl cluster-info command.

Backup Kubernetes CRDs

Assuming the desired Kubernetes namespace and the backup location environment variables are set:

export DR_CORE_NAMESPACE=dr11p1
export BACKUP_LOCATION=~/datarobot-backups/

The Kubernetes CRDs can be backed using the following script:

echo "executionenvironments.predictions.datarobot.com backup started"
mkdir -p $BACKUP_LOCATION/predictions/executionenvironments
for ee in $(kubectl get executionenvironments.predictions.datarobot.com -n $DR_CORE_NAMESPACE -o name); do
    echo $ee;
    kubectl get ${ee} -n $DR_CORE_NAMESPACE -o yaml > $BACKUP_LOCATION/predictions/executionenvironments/${ee#*/}.yaml;
done
echo "executionenvironments.predictions.datarobot.com backup finished"


echo "inferenceservers.predictions.datarobot.com backup started"
mkdir -p $BACKUP_LOCATION/predictions/inferenceservers
for is in $(kubectl get inferenceservers.predictions.datarobot.com -n $DR_CORE_NAMESPACE -o name); do
    echo $is;
    kubectl get ${is} -n $DR_CORE_NAMESPACE -o yaml > $BACKUP_LOCATION/predictions/inferenceservers/${is#*/}.yaml;
done
echo "inferenceservers.predictions.datarobot.com backup finished"