failed to get sandbox ip: check network namespace closed

Question

Today I found the kubernetes (version 1.21) pods stay terminating forever. The events output look like this:

error determining status: rpc error: code = Unknown desc = failed to get sandbox ip: check network namespace closed: remove netns: unlinkat /var/run/netns/cni-b12fd5b4-6d07-691e-103f-fe7b95531e78: device or resource busy

why did this happen? what should I do to fixed this problem? Now the kubernetes cluster only have 1 node. the kubelet log look like this:

May 18 21:45:08 k8smasterone kubelet[18945]: E0518 21:45:08.811693   18945 kuberuntime_manager.go:1003] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = failed to get sandbox ip:
May 18 21:45:08 k8smasterone kubelet[18945]: E0518 21:45:08.812351   18945 kuberuntime_manager.go:1003] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = failed to get sandbox ip:
May 18 21:45:08 k8smasterone kubelet[18945]: E0518 21:45:08.813075   18945 kuberuntime_manager.go:1003] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = failed to get sandbox ip:
May 18 21:45:08 k8smasterone kubelet[18945]: E0518 21:45:08.813673   18945 kuberuntime_manager.go:1003] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = failed to get sandbox ip:
May 18 21:45:08 k8smasterone kubelet[18945]: E0518 21:45:08.814290   18945 kuberuntime_manager.go:1003] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = failed to get sandbox ip:
May 18 21:45:08 k8smasterone kubelet[18945]: E0518 21:45:08.814914   18945 kuberuntime_manager.go:1003] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = failed to get sandbox ip:
May 18 21:45:08 k8smasterone kubelet[18945]: E0518 21:45:08.815539   18945 kuberuntime_manager.go:1003] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = failed to get sandbox ip:
May 18 21:45:08 k8smasterone kubelet[18945]: E0518 21:45:08.816164   18945 kuberuntime_manager.go:1003] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = failed to get sandbox ip:
May 18 21:45:08 k8smasterone kubelet[18945]: E0518 21:45:08.816786   18945 kuberuntime_manager.go:1003] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = failed to get sandbox ip:
May 18 21:45:08 k8smasterone kubelet[18945]: E0518 21:45:08.817397   18945 kuberuntime_manager.go:1003] "PodSandboxStatus of sandbox for pod" err="rpc error: code = Unknown desc = failed to get sandbox ip:
May 18 21:45:09 k8smasterone kubelet[18945]: E0518 21:45:09.522075   18945 pod_workers.go:190] "Error syncing pod, skipping" err="rpc error: code = Unknown desc = failed to get sandbox ip: check network nam

the calico log look like this:

2022-05-18 10:46:25.634 [INFO][1] watchercache.go 96: Watch channel closed by remote - recreate watcher ListRoot="/calico/ipam/v2/assignment/"
2022-05-18 11:17:36.476 [INFO][1] resources.go 377: Terminating main client watcher loop
2022-05-18 11:17:36.489 [INFO][1] resources.go 349: Main client watcher loop
2022-05-18 11:39:19.412 [INFO][1] watchercache.go 96: Watch channel closed by remote - recreate watcher ListRoot="/calico/ipam/v2/assignment/"
2022-05-18 11:43:54.495 [INFO][1] watchercache.go 96: Watch channel closed by remote - recreate watcher ListRoot="/calico/resources/v3/projectcalico.org/nodes"
2022-05-18 11:53:08.825 [INFO][1] resources.go 377: Terminating main client watcher loop
2022-05-18 11:53:08.837 [INFO][1] resources.go 349: Main client watcher loop
2022-05-18 12:23:21.779 [INFO][1] resources.go 377: Terminating main client watcher loop
2022-05-18 12:23:21.789 [INFO][1] resources.go 349: Main client watcher loop
2022-05-18 12:34:28.086 [INFO][1] watchercache.go 96: Watch channel closed by remote - recreate watcher ListRoot="/calico/ipam/v2/assignment/"
2022-05-18 12:37:57.917 [INFO][1] watchercache.go 96: Watch channel closed by remote - recreate watcher ListRoot="/calico/resources/v3/projectcalico.org/nodes"
2022-05-18 12:53:43.531 [INFO][1] resources.go 377: Terminating main client watcher loop
2022-05-18 12:53:43.671 [INFO][1] resources.go 349: Main client watcher loop
2022-05-18 13:05:42.290 [INFO][1] watchercache.go 96: Watch channel closed by remote - recreate watcher ListRoot="/calico/ipam/v2/assignment/"
2022-05-18 13:11:55.529 [INFO][1] watchercache.go 96: Watch channel closed by remote - recreate watcher ListRoot="/calico/resources/v3/projectcalico.org/nodes"
2022-05-18 13:39:11.175 [INFO][1] resources.go 377: Terminating main client watcher loop
2022-05-18 13:39:11.196 [INFO][1] resources.go 349: Main client watcher loop
2022-05-18 13:41:50.476 [INFO][1] watchercache.go 96: Watch channel closed by remote - recreate watcher ListRoot="/calico/ipam/v2/assignment/"
2022-05-18 14:00:02.143 [INFO][1] watchercache.go 96: Watch channel closed by remote - recreate watcher ListRoot="/calico/resources/v3/projectcalico.org/nodes"