Kubernetes Multi-Attach error for volume "pvc "Volume is already exclusively attached to one node and can't be attached to another"

Question

Kubernetes version: V1.22.2

Cloud Provider Vsphere version 6.7

Architecture:

• 3 Masters
• 15 Workers

What happened: One of the pods for some "unknown" reason went down, and when we try to lift him up, it couldn't attach the existing PVC. This only happened to a specific pod, all the others didn't have any kind of problem.

What did you expect to happen: Pods should dynamically assume PVCs

Validation: First step: The connection to Vsphere has been validated, and we have confirmed that the PVC exists. Second step: The Pod was restarted (Statefulset 1/1 replicas) to see if the pod would rise again and assume the pvc, but without success. Third step: Made a restart to the services (kube-controller, kube-apiserve, etc) Last step: All workers and masters were rebooted but without success, each time the pod was launched it had the same error ""Multi-Attach error for volume "pvc......" Volume is already exclusively attached to one node and can't be attached to another""

When I delete a pod and try to recreate it, I get this warning: Multi-Attach error for volume "pvc-xxxxx" The volume is already exclusively attached to a node and cannot be attached to another

Anything else we need to know: I have a cluster (3 master and 15 nodes)

Temporary resolution: Erase the existing PVC and launch the pod again to recreate the PVC. Since this is data, it is not the best solution to delete the existing PVC.

Multi-Attach error for volume "pvc-xxx" Volume is already exclusively attached to one node and can't be attached to another