How to increase storage in YugabyteDB cluster deployed in k8s?

Question

[Question posted by a user on YugabyteDB Community Slack]

Can anyone suggest how to increase storage on my kube cluster, I have deployed with the official helm chart. And when I use the set command it says forbidden:

helm upgrade  --set storage.master.size=100Gi,storage.tserver.size=100Gi yb1 yugabytedb/yugabyte --namespace yb
Error: UPGRADE FAILED: cannot patch "yb-master" with kind StatefulSet: StatefulSet.apps "yb-master" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy' and 'minReadySeconds' are forbidden && cannot patch "yb-tserver" with kind StatefulSet: StatefulSet.apps "yb-tserver" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy' and 'minReadySeconds' are forbidden

Answer 1

As the helm error message says, we cannot update volume claim templates once created, this is how a k8s statefulset works. If the StorageClass allows volume expansion, then you can directly update the relevant pvcs’.

It's a manual process today as Statefulset doesn’t yet allow the volume expansion through its own manifest.

1. Get the content of the sts - kubectl get sts/<sts_name> -o yaml > out.yaml
2. Delete it without deleting the pods - kubectl delete sts/<sts_name> --cascade=orphan
3. Update the PVC with the required volume size
4. Recreate the statefulset with the updated volume size from the output of step-1
5. The sts manifest will take over the pods (through selectors) and update the storage spec without recreating.

You can track the status of the feature request here in kubernetes repo.