Restore custom models¶

DataRobot Custom Models are executed on a Kubernetes cluster. Every custom model is backed by DataRobot's S3-compatible Object Storage and MongoDB data. This means that:

• A backup of all custom models is represented by the backed-up state of the S3-compatible Object Storage and MongoDB.
• You can fully restore Custom Models to a running state using this data.

To recover all custom models data to a running state on a target installation, DataRobot provides an additional Python script. You should run this script after the following conditions are met:

• Filestore (S3-compatible Object Storage) and MongoDB have been successfully restored.
• The DataRobot application is in a running state.
• An Installation Admin API token is available (this is required for script execution).

Custom model images are rebuilt as needed and pushed to the new repository as part of the restore process.

Run the restore script¶

You must start the restore script from within a DataRobot application container (specifically, a pod running the mmapp-app deployment).

Run the restore script:

kubectl -n $NAMESPACE exec deploy/mmapp-app -it -- \
    /entrypoint tools/custom_model/restore_custom_models.py \
    --webserver-address "$EXTERNAL_WEB_SERVER_URL" \
    --admin-api-token "ADMIN_API_TOKEN"

The script produces output similar to the following; each eligible custom model is restored and re-enabled one by one:

2021-11-16 00:40:24 INFO     Custom Model LRS list to start (1): ['6192d2abdb1a2688cb94b968']
2021-11-16 00:40:24 INFO     Starting Custom Model LRS 6192d2abdb1a2688cb94b968
2021-11-16 00:40:35 INFO     Started Custom Model LRS 6192d2abdb1a2688cb94b968
2021-11-16 00:40:35 INFO     Started: 1 models. Failed: 0 models

The script is idempotent and doesn't alter custom models that were already running at the time of execution.