위 버전과 저장소는 오래된 것이라 저장소가 존재하지 않을 수 있다.
Airflow 2.1.4 이미지와 당시 Helm 차트로 올린 기록이다. 현행은 Airflow Helm 차트 설치 가이드 를 볼 것. Airflow 3.x 에서는 차트의webserver.*값이apiServer.*로 바뀌었고AIRFLOW__KUBERNETES__*환경 변수는AIRFLOW__KUBERNETES_EXECUTOR__*다.
DAG 저장소를 읽을 SSH 키를 secret 으로 만든다.
kubectl create secret generic airflow-ssh-git-secret --from-file=gitSshKey=/home/encore/gitlab_key/id_rsa
helm repo add apache-airflow https://airflow.apache.org
helm fetch --untar apache-airflow/airflow
cd airflow
vi values.yaml
바꾼 항목만 적는다.
# dags.gitSync
sshKeySecret: ${REDACTED} #encore
wait: 10 #60 encore
#private repo for kpo
defaultAirflowRepository: encore.sec/library/apache/airflow #encore
#executor change
executor: "KubernetesExecutor" #encore
#env for airflow.cfg
env:
- name: "AIRFLOW__KUBERNETES__WORKER_CONTAINER_REPOSITORY"
value: "encore.sec/library/apache/airflow"
- name: "AIRFLOW__KUBERNETES__WORKER_CONTAINER_TAG"
value: "2.1.4"
- name: "AIRFLOW__KUBERNETES__RUN_AS_USER"
value: "50000"
- name: "AIRFLOW__KUBERNETES__DAGS_IN_IMAGE"
value: "True"
gitSync:
enabled: true #encore
repo: ssh://git@172.17.172.2/hakgeon.kim/airflow-dags.git #encore
branch: main #encore
subPath: "" #"tests/dags"
helm upgrade --install airflow apache-airflow/airflow -n airflow-webinar -f ./values.yaml --debug
NAME READY STATUS RESTARTS AGE
pod/airflow-postgresql-0 1/1 Running 0 116m
pod/airflow-scheduler-fd899887f-mvcxh 3/3 Running 0 103m
pod/airflow-statsd-84f4f9898-hrfr5 1/1 Running 0 117m
pod/airflow-webserver-868d75f456-7ppmq 1/1 Running 0 103m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
service/airflow-postgresql ClusterIP 10.233.29.55 <none> 5432/TCP
service/airflow-postgresql-headless ClusterIP None <none> 5432/TCP
service/airflow-statsd ClusterIP 10.233.44.221 <none> 9125/UDP,9102/TCP
service/airflow-webserver ClusterIP 10.233.40.66 <none> 8080/TCP
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/airflow-scheduler 1/1 1 1 117m
deployment.apps/airflow-statsd 1/1 1 1 117m
deployment.apps/airflow-webserver 1/1 1 1 117m
NAME READY AGE
statefulset.apps/airflow-postgresql 1/1 117m
웹서버를 NodePort 로 연 기록.
cd ./script/
kubectl apply -f airflow-webserver-svc-np.yaml
# service/airflow-webserver-svc-np created
kubectl get svc -n airflow-webinar
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
airflow-postgresql ClusterIP 10.233.29.55 <none> 5432/TCP 120m
airflow-postgresql-headless ClusterIP None <none> 5432/TCP 120m
airflow-statsd ClusterIP 10.233.44.221 <none> 9125/UDP,9102/TCP 120m
airflow-webserver ClusterIP 10.233.40.66 <none> 8080/TCP 120m
airflow-webserver-svc-np NodePort 10.233.16.30 <none> 8080:31394/TCP 11s
kubectl get event -n airflow-webinar -w
https://marclamberti.com/blog/airflow-on-kubernetes-get-started-in-10-mins/