Could not pull image - caused by: failed to register layer: error creating overlay mount to var/lib/docker/overlay2 :too many levels of symbolic links

Question

We are facing one issue related to the IoT Edge Module pull in one of the hardware. The Gateway is not able to pull all the edge modules as per the deployment manifest file. While downloading some modules, the below error is observed in the IoT Edge runtime logs.

Could not pull image XXXXXX/YYYY:6.0.0-amd64

\<4\>2022-03-28T08:56:44Z \[WARN\] - 
caused by failed to register layer: 
error creating overlay mount to var/lib/docker/overlay2/24593016a6b6bf0eaf6543d5ec82d94244d5fcb6d25e3be62ed0da70761daacd/merged: 
too many levels of symbolic links

The IoT Edge runtime version is 1.1.6

Could you please let us know the cause for this error and how to recover from this error when the devices are installed at sites during production?

Thank You. Best Regards, Supreet

We are not expecting any errors while IoT Edge modules are being pulled as per the deployment manifest file. All the modules should be pulled and running.

Answer 1

See below for my resolution; not sure if this will work for you.

Specifications

• Hardware: Jetson Nano
• O/S: Ubuntu 18.04
• Package: nvidia-jetpack
• Version: 4.6-b199
• Architecture: arm64
• Iotedge version: 1.2
• Iotedge Agent: 1.1.12.53348281

Background

I am experiencing the same issue on one (1) of our IoTEdge modules that is running a custom base image. I have checked the base image and it is valid and has not been updated. Also, this same Image/Module is running without issue on 13 other almost identical devices. Overall running nine modules on these machines (includes edgeAgent & edgeHub).

When I inspect the edgeAgent Logs I see the following errors:

Could not create module EnergyCalc
Apr 11 10:31:37 agl-a-f2-h2-u001 aziot-edged[32020]: 2022-04-11T14:31:37Z [WARN] -         caused by: error creating overlay mount to /var/lib/docker/overlay2/8c5855e126f9b652d63bf02f293b2382767077eacb00490bd754db0d0c4085c7-init/merged: no such file or directory

This device did experience a "disk" full scenario. This issue was resolved and then I have uninstalled/reinstalled IoTEdge several times without resolution. Created/Deployed a new image (EnergyCalc2) of the same module on a test machine-works perfectly—however when I deploy to this machine, same problem exists.

Brute force method

• Stopped/Removed IoTEdge, Moby-cli, Moby-Engine
• Deleted /var/lib/docker directory:

  rm -r /var/lib/docker
  

• Re-installed Moby-Engine, IoTEdge

I would venture there is something in the docker configuration that is causing the problem; I did not had a chance to investigate fully.