'Kubernetes: Unable to mount a projected volume, no space on device when clearly there is

I have a local k3d cluster in stalled on Mac (Big Sur 12) that I am attempting to install calico (their default manifest https://k3d.io/v5.3.0/usage/advanced/calico/). In the logs for the calico-kube-controller pod, I get this:

Warning FailedMount 44m kubelet MountVolume. SetUp failed for volume "kube-api-access-vfdd9" : write /var/lib/kubelet/pods/faa7d654-6424-4774-bc40-71de88c1d337/volumes/kubernetes.io~projected/kube-api-access-vfdd9/..2022_03_09_20_46_13.100604692/token: no space left on device

There is clearly plenty of space:

/var/lib/kubelet/pods/faa7d654-6424-4774-bc40-71de88c1d337/volumes/kubernetes.io~projected # df -h .

Filesystem Size Used Avail Use% Mounted on

/dev/vda1 79G 7.7G 67G 11% /var/lib/kubelet

Google searching has yielded nothing effective.

Kubernetes version: K8s Rev: v1.22.4+k3s1

Docker version:

Server: Docker Desktop 4.5.0 (74594)

Engine: Version: 20.10.12

API version: 1.41 (minimum version 1.12)

Go version: go1.16.12

Git commit: 459d0df

Built: Mon Dec 13 11:43:56 2021

OS/Arch: linux/amd64

Experimental: false

containerd: Version: 1.4.12

GitCommit: 7b11cfaabd73bb80907dd23182b9347b4245eb5d

runc: Version: 1.0.2

GitCommit: v1.0.2-0-g52b36a2

docker-init: Version: 0.19.0

GitCommit: de40ad0

Any tips/docs/analysis would be much appreciated!

kubernetesvolumescalico


Solution 1:[1]

double check your container resources configuration for errors. Try to increase the values.

      resources:
        limits:
          cpu: '1'
          memory: 500m
        requests:
          cpu: 50m
          memory: 100m

Mine started working when removed resources section completely. (Docker Desktop 4.3.2)

Solution 2:[2]

You can also check if there is a resourcequota configured.

kubectl get resourcequota

somequota 14d pods: 0/2, requests.cpu: 0/1, requests.memory: 0/1024m
limits.cpu: 0/4, limits.memory: 0/4096m

If you can are allowed to delete it then do and it will probably work

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 Ognjen Miletic
Solution 2 Serve Laurijssen

Related Questions