'How to setup Kubernetes Executor on Airflow 2.x running Locally on Ubuntu-- NO HELM CHART

can anyone tell me how to config kuberenetes executor in local airflow deployment. I created a kind cluster named airflow-cluster and created the pod_template.yaml and made the following changes in the airflow.cfg.

[kubernetes]
# Path to the YAML pod file that forms the basis for KubernetesExecutor workers.
pod_template_file = /home/caxe/airflow/logs/yamls/pod_template.yaml
worker_container_repository = apache/airflow
worker_container_tag = 2.2.3
namespace = airflow
in_cluster = False
cluster_context = kind-airflow-cluster
config_file = /home/caxe/.kube/config

pod_template.yaml

---
apiVersion: v1
kind: Pod
metadata:
  name: dummy-name
spec:
  containers:
    - env:
        - name: AIRFLOW__CORE__EXECUTOR
          value: LocalExecutor
        # Hard Coded Airflow Envs
        - name: AIRFLOW__CORE__SQL_ALCHEMY_CONN
          value: postgresql+psycopg2://airflow:airflow@localhost:5432/airflow
      image: apache/airflow:2.2.3
      imagePullPolicy: IfNotPresent
      name: base
      volumeMounts:
        - mountPath: "/opt/airflow/logs"
          name: airflow-logs
        - mountPath: "/opt/airflow/dags"
          name: airflow-dags
          readOnly: true
        - mountPath: "/opt/airflow/airflow.cfg"
          name: airflow-config
          readOnly: true
          subPath: airflow.cfg
  restartPolicy: Never
  securityContext:
    runAsUser: 50000
    fsGroup: 50000
  serviceAccountName: airflow
  volumes:
    - name: airflow-logs
      persistentVolumeClaim:
        claimName: logs-pv-claim
    - name: airflow-dags
      persistentVolumeClaim:
        claimName: dag-pv-claim
    - configMap:
        name: k8s-config 
      name: airflow-config

Doesnt execute. On running kubectl get pods -n airflow

NAME                                                               READY   STATUS    RESTARTS   AGE
examplebashoperatoralsorunthis.dd577351d4554c87923bc1eabe5e617e    0/1     Pending   0          114s
examplebashoperatorrunme0.afd364b8033643549a29ab536e9fc83f         0/1     Pending   0          116s
examplebashoperatorrunme1.47c97859639543bcab04a2ef0001ee9a         0/1     Pending   0          116s
examplebashoperatorrunme2.7296c3f011624f5ab62c1777187a006f         0/1     Pending   0          115s
examplebashoperatorthiswillskip.b9474f2673524a538ed2fddb6af00dd0   0/1     Pending   0          113s

I'm not a kubernetes guy, I have created the persistent volume and claim for logs and dags, but I think there could be a problem with the non cluster postgres connection. Because i have not configured postgres in cluster other than providing values in config and yaml file. Moreover the psycopg2 (apache-airflow[postgres]) is installed in local airflow, but since i havent modified the base image apace/airflow:2.2.3 could it be missing?

linuxairflowubuntu-20.04airflow-2.x


Solution 1:[1]

Setting up Kubernetes Executor needed Postgres to be running as a service in the cluster with the port forwarded, so that the Executor pod has access to it. It also required to install the additional package apache-airflow['postgres'] by creating a Dockerfile and using the base image of airflow. We also have to create persistent volumes and persistent volume claims where our dags and logs can be stored. Make sure the image we are using has the same version of python as in our local sytstem.

requirements.txt

apache-airflow-providers-cncf-kubernetes==3.0.1
apache-airflow-providers-postgres==2.4.0

Dockerfile

FROM apache/airflow:2.2.3-python3.8 
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY test.py /opt/airflow/dags/

create the custom image with the tag eg. airflow-custom:1.0.0 using docker build -t airflow-custom:1.0.0 . and specify it in the pod template file as well.

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1

Related Questions