Docker driver "overlay2" failed to remove root filesystem: unlinkat - device or resource busy

Question

When trying to remove a docker container (for example when trying to docker-compose down) I always get these errors:

ERROR: for <my_container>  container d8424f80ef124c2f3dd8f22a8fe8273f294e8e63954e7f318db93993458bac27: driver "overlay2" failed to remove root filesystem: unlinkat /var/lib/docker/overlay2/64311f2ee553a5d42291afa316db7aa392a29687ffa61971b4454a9be026b3c4/merged: device or resource busy

Common advice like restarting the docker service, pruning or force removing the containers doesn't work. The only thing that I found that works is manually unmounting with sudo umount /home/virtfs/xxxxx/var/lib/docker/overlay2/<container_id>/merged, then I'm able to remove the container.

My OS is CentOS Linux release 7.9.2009 (Core) with kernel version 3.10.0-1127.19.1.el7.x86_64. I thought this was maybe due to overlay2 clashing with CentOS, but according to this page my CentOS/kernel version should work. It would be great to find a solution to this, because I ideally want to docker-compose down without having to use elevated privileges to umount beforehand.

Answer 1

From the error log, it is observed that the file mounting is involved Execute the following command to view the related processes

grep 64311f2ee553a5d42291afa316db7aa392a29687ffa61971b4454a9be026b3c4 /proc/*/mountinfo

ps -ef | grep "The process ID obtained by the grep command above" 

Stop the occupied process

Then delete the container

Answer 2

Often this happens when there are no processes listed as blocking, then you know it's a kernel module blocking it.

The most likely culprits are nfs (not sure why you'd run that in docker), or files inside docker that are bind-mounted, sometimes the automatic ones, such as perhaps ones created by systemd-networkd.

Overlay2 was phased out by Ubuntu for a reason. CentOS is at its end of life, so this problem might be resolved already in your most likely upgrade path, to Rocky Linux. Alternately you can enter the jungle of migrating your docker storage engine.

Or you can just get rid of the package or software hogging it in the first place, if you can. But you'll have to share more info on what it is for help on that.