Skip to content
For Managed AI Platform users, see the cloud platform documentation
Self-managed installation and maintenance > Backup and restore > Database restore procedure > Restoring Custom Models

Restoring Custom Models

Prerequisites

  • Utility kubectl version 1.23 is installed on the host where the backup is located
  • Utility kubectl is configured to access the Kubernetes cluster where the DataRobot application is running. Verify this with the kubectl cluster-info command.

DataRobot Custom Models are executed on a Kubernetes cluster. Every custom model is backed by DataRobot S3-compatible Object Storage and MongoDB data. This means that:

  • A backup of all custom models is represented by a backed up state of DataRobot S3-compatible Object Storage and MongoDB.
  • Restoring Custom Models to a running state is fully possible using this data.

In order to recover all custom models data into a running state on a target installation where the restore is happening, DataRobot provides an additional Python script that should be called after: * Filestore and Mongo have been restored * Application is in a running state. * Installation Admin API token is available, required for the script execution.

The images will be rebuilt as needed and pushed to the new repository as part of the restore process.

Running The Restore Script

The restore script must be started within the DataRobot application container.

Export name of DataRobot application Kubernetes namespace in DR_CORE_NAMESPACE variable: 

export DR_CORE_NAMESPACE=<namespace>

Run the restore script 

kubectl -n $DR_CORE_NAMESPACE exec deploy/mmapp-app -it -- \
  /entrypoint tools/custom_model/restore_custom_models.py \
  --webserver-address "$EXTERNAL_WEB_SERVER_URL" \
  --admin-api-token <ADMIN_API_TOKEN>

Output similar to the below will be returned, where each eligible custom model is restored and then re-enabled.

2021-11-16 00:40:24 INFO     Custom Model LRS list to start (1): ['6192d2abdb1a2688cb94b968']
2021-11-16 00:40:24 INFO     Starting Custom Model LRS 6192d2abdb1a2688cb94b968
2021-11-16 00:40:35 INFO     Started Custom Model LRS 6192d2abdb1a2688cb94b968
2021-11-16 00:40:35 INFO     Started: 1 models. Failed: 0 models

The script is idempotent and will not alter custom models that were already running at the time of execution.