Unable to execute a command in Ubuntu container, from Airflow container using DockerOperator

Question

I am very new to Docker + Airflow. Below is what I am trying to accomplish.

I have 4 services as shown in the below compose file. 3 are relevant to Airflow and one as a test Ubuntu instance. The Airflow related containers: airflow-database, airflow-webserver, airflow-scheduler are able to communicate with each other and I am able to run the example DAGs. Now I added a 4th service (ubuntu) to which I am trying to send a simple command "/bin/sleep 10" from a DAG using DockerOperator (Below is the DAG file). But for some reason I am getting the Permission Denied message (Attached the DAG error file as well).

It works if I run Airflow from localhost instead of from inside a docker container Unable to figure out what I am missing. Below are some of the ways I tried:

• In docker_url:

1. replaced unix://var/run/docker.sock with tcp://172.20.0.1 thinking it would be able to resolve via the docker host ip

2. used gateway.host.internal

3. Even removed docker_url option from the operator, but realized it anyways gets defaulted to unix://var/run/docker.sock

4. Tried bunch of combinations, tcp://172.20.0.1:2376, tcp://172.20.0.1:2375

5. Mapped port of host to Ubuntu, like 8085:8085, etc.

• May be the airflow user of Airflow web server is getting kicked out by Ubuntu
• So created a group in the Ubuntu container and added airflow user to it -- Nope did not work as well
• api_version: The option 'auto' as well does not work and keeps giving version not found error. So had to hard-code with 1.41 as I found this in the docker version command. Not sure if that is what it supposed to be.

Thanks in advance for any help in what else I can try to make this work :)

docker-compose.yml

version: '3.2'

services:
# Ubuntu Container  
  ubuntu:
    image: ubuntu
    networks:
      - mynetwork

# Airflow Database
  airflow-database:
    image: postgres:12
    env_file:
      - .env
    ports:
      - 5432:5432
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./airflow/database/data:/var/lib/postgresql/data/pgdata
      - ./airflow/database/logs:/var/lib/postgresql/data/log
    command: >
     postgres
       -c listen_addresses=*
       -c logging_collector=on
       -c log_destination=stderr
       -c max_connections=200
    networks:
      - mynetwork

# Airflow DB Init
  initdb:
      image: apache/airflow:2.0.0-python3.8
      env_file:
        - .env
      depends_on:
        - airflow-database
      volumes:
        - /var/run/docker.sock:/var/run/docker.sock
        - ./airflow/metadata-airflow/dags:/opt/airflow/dags
        - ./airflow/logs:/opt/airflow/logs
      entrypoint: /bin/bash
      command: -c "airflow db init && airflow users create --firstname admin --lastname admin --email [email protected] --password admin --username admin --role Admin"
      networks:
        - mynetwork

# Airflow Webserver
  airflow-webserver:
    image: apache/airflow:2.0.0-python3.8
    env_file:
      - .env
    depends_on:
      - airflow-database
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./airflow/metadata-airflow/dags:/opt/airflow/dags
      - ./airflow/logs:/opt/airflow/logs
    ports:
      - 8080:8080
    deploy:
      restart_policy:
        condition: on-failure
        delay: 8s
        max_attempts: 3
    command: webserver
    healthcheck:
      test: ["CMD-SHELL", "[ -f /opt/airflow/airflow-webserver.pid ]"]
      interval: 30s
      timeout: 30s
      retries: 3
    networks:
      - mynetwork

# Airflow Scheduler
  airflow-scheduler:
    image: apache/airflow:2.0.0-python3.8
    env_file:
      - .env
    depends_on:
      - airflow-database
      - airflow-webserver
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./airflow/metadata-airflow/dags:/opt/airflow/dags
      - ./airflow/logs:/opt/airflow/logs
    deploy:
      restart_policy:
        condition: on-failure
        delay: 8s
        max_attempts: 3
    command: scheduler
    networks:
      - mynetwork

networks:
  mynetwork:

DAG File

from datetime import timedelta
from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from airflow.providers.docker.operators.docker import DockerOperator
from airflow.utils.dates import days_ago


default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'email': ['[email protected]'],
    'email_on_failure': False,
    'email_on_retry': False,
}

dag = DAG(
    'docker_sample',
    default_args=default_args,
    schedule_interval=None,
    start_date=days_ago(2),
)

t1 = DockerOperator(
    task_id='docker_op_tester',
    api_version='auto', 
    image='ubuntu',
    docker_url='unix://var/run/docker.sock',
    auto_remove=True,
    command=[
        "/bin/bash",
        "-c",
        "/bin/sleep 30; "],
    network_mode='bridge',
    dag=dag,
)


t1

DAG Error Log

*** Reading local file: /opt/airflow/logs/docker_sample/docker_op_tester/2021-01-09T05:16:17.174981+00:00/1.log
[2021-01-09 05:16:26,726] {taskinstance.py:826} INFO - Dependencies all met for <TaskInstance: docker_sample.docker_op_tester 2021-01-09T05:16:17.174981+00:00 [queued]>
[2021-01-09 05:16:26,774] {taskinstance.py:826} INFO - Dependencies all met for <TaskInstance: docker_sample.docker_op_tester 2021-01-09T05:16:17.174981+00:00 [queued]>
[2021-01-09 05:16:26,775] {taskinstance.py:1017} INFO - 
--------------------------------------------------------------------------------
[2021-01-09 05:16:26,776] {taskinstance.py:1018} INFO - Starting attempt 1 of 1
[2021-01-09 05:16:26,776] {taskinstance.py:1019} INFO - 
--------------------------------------------------------------------------------
[2021-01-09 05:16:26,790] {taskinstance.py:1038} INFO - Executing <Task(DockerOperator): docker_op_tester> on 2021-01-09T05:16:17.174981+00:00
[2021-01-09 05:16:26,794] {standard_task_runner.py:51} INFO - Started process 1057 to run task
[2021-01-09 05:16:26,817] {standard_task_runner.py:75} INFO - Running: ['airflow', 'tasks', 'run', 'docker_sample', 'docker_op_tester', '2021-01-09T05:16:17.174981+00:00', '--job-id', '360', '--pool', 'default_pool', '--raw', '--subdir', 'DAGS_FOLDER/example_docker.py', '--cfg-path', '/tmp/tmp4phq52dv']
[2021-01-09 05:16:26,821] {standard_task_runner.py:76} INFO - Job 360: Subtask docker_op_tester
[2021-01-09 05:16:26,932] {logging_mixin.py:103} INFO - Running <TaskInstance: docker_sample.docker_op_tester 2021-01-09T05:16:17.174981+00:00 [running]> on host 367f0fc7d092
[2021-01-09 05:16:27,036] {taskinstance.py:1230} INFO - Exporting the following env vars:
[email protected]
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=docker_sample
AIRFLOW_CTX_TASK_ID=docker_op_tester
AIRFLOW_CTX_EXECUTION_DATE=2021-01-09T05:16:17.174981+00:00
AIRFLOW_CTX_DAG_RUN_ID=manual__2021-01-09T05:16:17.174981+00:00
[2021-01-09 05:16:27,054] {taskinstance.py:1396} ERROR - ('Connection aborted.', PermissionError(13, 'Permission denied'))
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/home/airflow/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 392, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.8/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.8/http/client.py", line 950, in send
    self.connect()
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/transport/unixconn.py", line 43, in connect
    sock.connect(self.unix_socket)
PermissionError: [Errno 13] Permission denied

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/home/airflow/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 726, in urlopen
    retries = retries.increment(
  File "/home/airflow/.local/lib/python3.8/site-packages/urllib3/util/retry.py", line 410, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/home/airflow/.local/lib/python3.8/site-packages/urllib3/packages/six.py", line 734, in reraise
    raise value.with_traceback(tb)
  File "/home/airflow/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/home/airflow/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 392, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.8/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.8/http/client.py", line 950, in send
    self.connect()
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/transport/unixconn.py", line 43, in connect
    sock.connect(self.unix_socket)
urllib3.exceptions.ProtocolError: ('Connection aborted.', PermissionError(13, 'Permission denied'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1086, in _run_raw_task
    self._prepare_and_execute_task_with_callbacks(context, task)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1260, in _prepare_and_execute_task_with_callbacks
    result = self._execute_task(context, task_copy)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1300, in _execute_task
    result = task_copy.execute(context=context)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/docker/operators/docker.py", line 286, in execute
    if self.force_pull or not self.cli.images(name=self.image):
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/api/image.py", line 89, in images
    res = self._result(self._get(self._url("/images/json"), params=params),
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/api/client.py", line 230, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/home/airflow/.local/lib/python3.8/site-packages/requests/sessions.py", line 543, in get
    return self.request('GET', url, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', PermissionError(13, 'Permission denied'))
[2021-01-09 05:16:27,073] {taskinstance.py:1433} INFO - Marking task as FAILED. dag_id=docker_sample, task_id=docker_op_tester, execution_date=20210109T051617, start_date=20210109T051626, end_date=20210109T051627
[2021-01-09 05:16:27,136] {local_task_job.py:118} INFO - Task exited with return code 1

Specs: Docker: Version: 20.10.2 API version: 1.41

Airflow image: apache/airflow:2.0.0-python3.8

Host system: MacOS BigSur

Answer 1

I think I got it - source: https://tomgregory.com/running-docker-in-docker-on-windows

1. Check the Group ID of root:

   docker run --rm -v /var/run/docker.sock:/var/run/docker.sock debian:buster-slim stat -c %g /var/run/docker.sock

returns "1001" for me.

2. Add the group_add statements referring to this group id into your docker-compose.yml:

    image: apache/airflow:2.0.0-python3.8
    group_add:
      - 1001
   

I added it to webserver and scheduler (not sure if both need it) and it seems to work for me now (at least it crashes at a later point ;-)

Edit:

you also need to add

AIRFLOW__CORE__ENABLE_XCOM_PICKLING=True

as environment variable in Airflow, otherwise your container crashes when exiting (https://github.com/apache/airflow/issues/13487).

Answer 2

DISCLAIMER: Not a production level solution.

Its kind of late but here is what I did to get it resolved.

In same lines as outlined by @lungben, what I have learned is its related to file access conflict between the user group of the docker daemon running in the host and the one running inside the containers.

In my Mac (local): I started the containers by giving the airflow user (inside the containers) root privlieges (AIRFLOW_UID=0 and AIRFLOW_GID=0), basically same as the root of the local machine. (NO NO for Production). Tried adding my Mac username to docker group, but for some reason it didn't work.

In the Ubuntu server:

• Added the ubuntu user to docker group sudo usermod -aG docker $USER(create if docker group doesn't exist)
• If this does not resolve, chmod 777 the /var/run/docker.sock & start docker compose as sudo (Again NONO in Production)

Since, I no more use DockerOperator, rolled back all the above changes.