Skip to content
For Self-Managed AI Platform users running v11.1, see the on-premise platform documentation

title: Restoring Custom Models description: Prerequisites:

Restoring Custom Models

Prerequisites:

  • Utility kubectl of version 1.23 is installed on the host where backup is located
  • Utility kubectl is configured to access the Kubernetes cluster where DataRobot application is running, verify this with kubectl cluster-info command.

DataRobot Custom Models are executed on a Kubernetes cluster. Every custom model is backed by DataRobot S3-compatible Object Storage and MongoDB data. This means thats: * A backup of all custom models is represented by a backed up state of DataRobot S3-compatible Object Storage and MongoDB. * Restoring Custom Models to a running state is fully possible using this data.

In order to recover all custom models data into a running state on a target installation where the restore is happening, DataRobot provides an additional Python script that should be called after: * Filestore and Mongo have been restored * Application is in a running state. * Installation Admin API token is available, required for the script execution.

The images will be rebuilt as needed and pushed to the new repository as part of the restore process.

Running The Restore Script

The restore scipt must be started within the DataRobot application container.

Export name of DataRobot application Kubernetes namespace in DR_CORE_NAMESPACE variable: 

export DR_CORE_NAMESPACE=<namespace>

Run the restore script 

kubectl -n $DR_CORE_NAMESPACE exec deploy/mmapp-app -it -- \
  /entrypoint tools/custom_model/restore_custom_models.py \
  --webserver-address "$EXTERNAL_WEB_SERVER_URL" \
  --admin-api-token <ADMIN_API_TOKEN>

A similar output will be presened where each eligible custom model will be restored and reenabled one by one.

2021-11-16 00:40:24 INFO     Custom Model LRS list to start (1): ['6192d2abdb1a2688cb94b968']
2021-11-16 00:40:24 INFO     Starting Custom Model LRS 6192d2abdb1a2688cb94b968
2021-11-16 00:40:35 INFO     Started Custom Model LRS 6192d2abdb1a2688cb94b968
2021-11-16 00:40:35 INFO     Started: 1 models. Failed: 0 models

The script is idempotent and will not alter custom models that were already running at the time of execution.