Restore Elasticsearch¶

Due to specific security settings, use the curl utility within the Elasticsearch containers for all following operations. For example, attach to the container in the pcs-elasticsearch-master stateful set:

kubectl -n $NAMESPACE exec sts/pcs-elasticsearch-master -- /bin/bash

Then retrieve the cluster information:

curl -k -u elastic:$ELASTICSEARCH_PASSWORD "https://localhost:9200/"

Restore from filesystem snapshot¶

Starting with 10.2, configure an additional mount point to store backups locally. If the backup is stored under the local filesystem in Elasticsearch, follow the steps below:

1. Remove any old snapshot files. Once completed, copy the latest local backup (for example, es_snapshot_merged_final.tar.gz), onto all three pods. (See the backup instructions for more information.)

   for pod in pcs-elasticsearch-master-0 pcs-elasticsearch-master-1 pcs-elasticsearch-master-2; do
     echo "Copying to target $pod..."
     kubectl cp ./es_snapshot_merged_final.tar.gz $NS/$pod:/tmp/es_snapshot_merged_final.tar.gz
     echo "Extracting on target $pod..."
     kubectl exec -it $pod -n $NS -- bash -c "rm -rf /snapshots/* && tar -xzvf /tmp/es_snapshot_merged_final.tar.gz -C /"
   done
   

2. Export the master pod to the target. When complete, open an interactive shell session inside the master pod.

   kubectl exec -it pcs-elasticsearch-master-0 -n $NS -- bash -c 'export ELASTICSEARCH_PASSWORD=$(cat /opt/iamguarded/elasticsearch/secrets/elasticsearch-password); curl -X GET -k -u elastic:$ELASTICSEARCH_PASSWORD "https://localhost:9200/_cat/master?v"'
   

3. Delete the old snapshot registry and register a new snapshot directory.

   curl -k -X DELETE -u elastic:$ELASTICSEARCH_PASSWORD \
   "https://localhost:9200/_snapshot/dr_repository?pretty"
   curl -k -X PUT -u elastic:$ELASTICSEARCH_PASSWORD \
   "https://localhost:9200/_snapshot/dr_repository?verify=false&pretty" \
   -H "Content-Type: application/json" \
   -d '{"type": "fs", "settings": {"location": "/snapshots"}}'
   

4. For a clean restore, disable existing protective rules and delete any existing indices on the target.

   curl -k -X PUT -u elastic:$ELASTICSEARCH_PASSWORD \
   "https://localhost:9200/_cluster/settings?pretty" \
   -H "Content-Type: application/json" \
   -d '{
     "persistent": {
       "action.destructive_requires_name": false
     }
   }'
   
   curl -k -X DELETE -u elastic:$ELASTICSEARCH_PASSWORD \
   "https://localhost:9200/*?pretty"
   

5. Run the restore.

   curl -k -X POST -u elastic:$ELASTICSEARCH_PASSWORD \
   "https://localhost:9200/_snapshot/dr_repository/dr_snapshot/_restore?pretty" \
   -H "Content-Type: application/json" \
   -d '{
     "indices": ["*", "-.ds-*", "-.*"],
     "ignore_unavailable": true,
     "include_global_state": false
   }'
   

6. Verify that the restore was successful.

   curl -k -u elastic:$ELASTICSEARCH_PASSWORD \
   "https://localhost:9200/_cat/indices?v&h=index,status,health,docs.count,store.size"
   

   The output must show, for all indices, a health state of green.

   bash-5.3$ curl -k -u elastic:$ELASTICSEARCH_PASSWORD \
   "https://localhost:9200/_cat/indices?v&h=index,status,health,docs.count,store.size"
   index                                                status health docs.count store.size
   storage-org-69a195679c20cec5b786c048-geopoint-000001 open   green         868    421.5kb
   datavolt_esmigrate_version                           open   green           1      8.9kb
   storage-org-69a195679c20cec5b786c048-text-000001     open   green        1342      1.6mb
   

Register a snapshot repository¶

You can configure Elasticsearch to read snapshots from different external locations: AWS S3 bucket, Azure Blob Storage, shared NFS volume, etc.

Shared filesystem (NFS) repository¶

Elasticsearch distribution delivered with the DataRobot application allows you to configure Elasticsearch to keep snapshots on an NFS volume. Check Snapshot and restore operations for more information on using this method. Note that this method requires an NFS server continuously available in your network.

Other repository types¶

Elasticsearch can also store snapshots on S3, Google Cloud, or Azure Blob Storage. If your snapshots are stored using one of these methods, refer to the appropriate section of the Register a snapshot repository guide.

To configure a snapshot registry, refer to the backup guide. This restore guide assumes that the backup snapshot is accessible from the same registry to perform restore.

Restore data from snapshot¶

Snapshots can be manually restored according to the Restore a snapshot guide. Follow the Restore Entire Cluster section.

If the snapshot is available in the S3 registry, follow the steps below:

1. Get a list of all indices.

   kubectl -n $NAMESPACE exec -it pcs-elasticsearch-master-0 -- /bin/bash
   curl -k -X GET -u elastic:$ELASTICSEARCH_PASSWORD "https://localhost:9200/_cat/indices"
   

2. Remove all existing indices if restoring onto an existing cluster with data. A single command doesn't work and throws an error:

   curl -k -XDELETE -u elastic:$ELASTICSEARCH_PASSWORD "https://localhost:9200/<index-name-from-step-1>"
   

3. Once deleted, run restore using the backup snapshot in the S3 repository:

   curl -k -X POST -u elastic:$ELASTICSEARCH_PASSWORD "https://localhost:9200/_snapshot/repository_name/<snapshot-name>/_restore" -H "Content-Type: application/json" -d '{
     "indices": "*",
     "ignore_unavailable": true,
     "include_global_state": true
   }'