PVC increase | Database mirroring

Question

I am increasing the persistent volume of our INT database from 30Gi to 40Gi upon releasing helm changes I am getting below error:

Error: UPGRADE FAILED: cannot patch "dbname-dbname-db" with kind StatefulSet: StatefulSet.apps "dbname-dbname-db" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden

I did a bit of research and I am figuring out that I need to stop the pod connecting to the database and/or delete the current service name dbname-dbname-db to apply the changes.

This seems to be pausing risks of downtime in our environment.

Can you please suggest possible ways of updating PVC capacity without downtime in the environment?

I read about database mirroring but it's not clear to me how to implement this. we run our application in a Kubernetes cluster I would appreciate it if one could show me the kubectl commands to create database mirror

below is part of the values.YAML.gotmpl file for the helm.

volumeMounts: {{- toYaml .volumeMounts | nindent 4 }}
  volumes: {{- toYaml .volumes | nindent 4 }}

  postgresql:
    postgresqlUsername: "user1"
    postgresqlDatabase: {{ .database.name | quote }}
    existingSecret: {{ .database.existingSecret | quote }}
    persistence:
      enabled: true
      size: 30Gi  ----> value being changed to **40Gi**
    nameOverride: "dbname-db"
    metrics:
      enabled: true
    networkPolicy:
      enabled: false
      allowExternal: true

  image:
    replicaCount: {{ .image.replicaCount }}
    repository: itops.company.io
    imageName: company/app-backend
    tag: {{ .image.tag | quote }}

Answer 1

Let's divide your issue into two different issues:

1. Increase your volume size
2. Update the value in your values.YAML.gotmpl file to reflect the actual size.

According to the error you get, it is apparently that the size in the values file is templated to the statefulset configuration under the field volumeClaimTemplates. However, currently this field cannot be patched (currently (April 2022) there is an open issue in kubernetes about it).

The way to resize the PVC is by patching the PVC itself.

Kubernetes can let you increase the PVC size without the need to restart the pod that uses it. This depends on what version of kubernetes you have, what type of storage and what flags are set by the cluster administrator.

You can try the following, supposing you have a kubectl access to the namespace of your database:

1. Figure out the number of pods in your statefulset: kubectl get sts dbname-dbname-db
2. Figure out the PVC names by runing kubectl describe statefulset dbname-dbname-db. You should see a section Volume Claims. the name field in this section is the prefix for the PVC name. the actual PVC name should be <prefix>-dbname-dbname-db-<i> when i is the serial number of the pod (out of the number of the pods for this statefulset), starting with 0.
3. For each PVC, run kubectl patch pvc <pvc-name> -p '{"spec":{"resources":{"requests":{"storage":"40Gi"}}}}}'

If the previous commands are successful, you can track the status of the resizing process by running kubectl describe pvc <pvc-name> and looking in the conditions field. It can take several minutes for the process to complete, make sure it is completed before proceeding with the next steps.

The above process leaves you with the second problem - the code in your values file is not aligned with the actual volume size.

The solution for this problem is as follows, and must be done after the previous process is completed :

1. Make sure you are ready to deploy your changes to the values file (30Gi -> 40Gi).
2. Delete the statefulset object, without deleting any running pod, by running kubectl delete statefulset dbname-dbname-db --cascade=orphan. this way, the working pods continue to work without interruption.
3. Deploy your changes. The statefulset will be created again and should be automatically associated with the running pod(s).

Reference: Expanding persistent volumes claims

Answer 2

Thanks, @Meir I followed your steps but encounter an error could not upgrade PVC it's attached to the node, you have to detach/stop the node.

After some research, we did the below steps and the PVC increase was successful, we had to delete the pod associated with the PVC thus in our case we destroyed the entire release via Helm.

• Destroy release
• apply this command:: kubectl patch pvc <pvc-name> -p '{\"spec\":{\"resources\":{\"requests\":{\"storage\":\"100Gi\"}}}}'
• After applying the patch the value may not reflect as an update, you will see a message waiting for the pod to restart in this case, we deployed the release again as this will create the pods and run kubectl get pvc -n namespace the change should reflect.

May not be the best solution as there is a downtime for at least 10 minutes but worked for us.