'How to solve 'no space left on device' error when running airflow on docker
I'm running Airflow on docker, and the scheduler container stops running after some time with the following error
Error response from daemon: Cannot restart container airflow-scheduler_1: mkdir /var/snap/docker/common/var-lib-docker/overlay2/3355a6abc0c5f0bde0eba9026fd04a9d627562d6c6f069c4a564c5fe7738721f/merged: no space left on device
I have used
docker system prune --volumes
but this is effective for a short period of time before the container stops running again and displaying the same error.
What I'm I missing?
Solution 1:[1]
You've got a few options:
- Increase the space on the docker instance itself
- Have a repeating script or job that prunes the volumes
- Map your DAGs or logs folder in the docker instance to a non-docker mount point
The third option is what we do, all our logs and code files are held outside the docker instances, so they don't get filled up unnecessarily.
We've found that the logs can sometimes be very verbose, so you might also want to look at filtering your logs or adding a log lifecycle.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 |