Step by Step guide to increase partition size of MongoDB, PostgreSQL and Redis¶

This document provides a guide to increase the size of storage partitions/volumes of a running cluster in On-Prem DataRobot9 and above

Make sure storage class has volume expansion to true¶

$ kubectl get sc -n NAMESPACE

NAME                     PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
ebs-standard (default)   ebs.csi.aws.com         Delete          WaitForFirstConsumer   true                   189d
ebs-standard-retain      ebs.csi.aws.com         Retain          WaitForFirstConsumer   true                   189d
gp2                      kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer   false                  189d

Get the pvc volumes and ensure the storage class is the one that has disk expansion to true¶

$ kubectl get pvc -n NAMESPACE

NAME                                             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
app-persistent-storage-prediction-server-app-0   Bound    pvc-669377db-2aa9-4e27-be5e-b2ad4873c9a1   5Gi        RWO            ebs-standard   50m
app-persistent-storage-tileservergl-app-0        Bound    pvc-80f18162-c663-4ffb-9d5d-d59202a4c26b   15Gi       RWO            ebs-standard   50m
app-persistent-storage-tileservergl-app-1        Bound    pvc-c3f074d8-2c07-4511-9e58-3790c0e09c33   15Gi       RWO            ebs-standard   44m
data-pcs-elasticsearch-master-0                  Bound    pvc-3a4c3f89-abdf-4a56-97f8-0041d60ba8aa   20Gi       RWO            ebs-standard   56m
data-pcs-elasticsearch-master-1                  Bound    pvc-c6e78c6f-7903-4a73-9ad6-5e4662e201ba   20Gi       RWO            ebs-standard   56m
data-pcs-elasticsearch-master-2                  Bound    pvc-b3307cfa-672f-4312-9a19-0dee9d41e434   20Gi       RWO            ebs-standard   56m
data-pcs-postgresql-0                            Bound    pvc-1b00fb84-224d-4fb4-b097-5c643fed3939   250Gi      RWO            ebs-standard   56m
data-pcs-postgresql-1                            Bound    pvc-dd874463-9b5a-4947-8e59-2e4582e34f5d   250Gi      RWO            ebs-standard   56m
data-pcs-postgresql-2                            Bound    pvc-0ecb19fc-581c-4d8a-9d7b-e99ac61f07a4   250Gi      RWO            ebs-standard   56m
data-pcs-rabbitmq-0                              Bound    pvc-3a979f05-c1cc-444a-b122-fa495791b610   8Gi        RWO            ebs-standard   56m
datadir-pcs-mongo-0                              Bound    pvc-e3ccc6af-3cba-4147-bc55-785caab2a990   250Gi      RWO            ebs-standard   56m
datadir-pcs-mongo-1                              Bound    pvc-57d1a5cf-9a08-4f66-9e86-df3beff45e3b   250Gi      RWO            ebs-standard   56m
datadir-pcs-mongo-2                              Bound    pvc-2f748e6e-b8d6-402e-857e-473035a96250   250Gi      RWO            ebs-standard   55m
redis-data-pcs-redis-node-0                      Bound    pvc-dfac02a6-35bb-4874-988c-1d70e7b161d6   8Gi        RWO            ebs-standard   56m
redis-data-pcs-redis-node-1                      Bound    pvc-2106221a-4bcf-4425-857e-b12173c9676b   8Gi        RWO            ebs-standard   56m
redis-data-pcs-redis-node-2                      Bound    pvc-7d5d171a-e4f1-4ec8-891c-2db97dd9814d   8Gi        RWO            ebs-standard   55m

Modify the pvc to expand the disk on each node. Modify pvc for each node.¶

Note: Below example explained for redis with 3 nodes. This will be applicable to other databases as well.

Set spec:resources:requests:storage and status:capacity:storage to desired size

$ kubectl edit pvc redis-data-pcs-redis-node-0 -n NAMESPACE

spec:
 accessModes:
 - ReadWriteOnce
 resources:
   requests:
     storage: 10Gi
 storageClassName: ebs-standard
 volumeMode: Filesystem
 volumeName: pvc-dfac02a6-35bb-4874-988c-1d70e7b161d6
status:
 accessModes:
 - ReadWriteOnce
 capacity:
   storage: 10Gi
 phase: Bound

Follow the same for all other nodes

$ kubectl edit pvc redis-data-pcs-redis-node-0 -n NAMESPACE

persistentvolumeclaim/redis-data-pcs-redis-node-0 edited

$ kubectl edit pvc redis-data-pcs-redis-node-1 -n NAMESPACE

persistentvolumeclaim/redis-data-pcs-redis-node-1 edited

$ kubectl edit pvc redis-data-pcs-redis-node-2 -n NAMESPACE

persistentvolumeclaim/redis-data-pcs-redis-node-2 edited

Verify the update¶

$ kubectl get pvc -n NAMESPACE | grep redis

redis-data-pcs-redis-node-0                      Bound    pvc-dfac02a6-35bb-4874-988c-1d70e7b161d6   10Gi       RWO            ebs-standard   77m
redis-data-pcs-redis-node-1                      Bound    pvc-2106221a-4bcf-4425-857e-b12173c9676b   10Gi       RWO            ebs-standard   76m
redis-data-pcs-redis-node-2                      Bound    pvc-7d5d171a-e4f1-4ec8-891c-2db97dd9814d   10Gi       RWO            ebs-standard   75m