본문 바로가기
Dev/Airflow

[airflow, docker] log 가 안나오는 이슈

by 석맨.class 2024. 3. 7.
반응형

Airflow 로그가 출력되지 않는 이슈

docker-compose 로 airflow 를 실행하는데 로그가 안나오고 다음과 같은 에러만 발생

 

 

!!!! Please make sure that all your Airflow components (e.g. schedulers, webservers, workers and triggerer) have the same 'secret_key' configured in 'webserver' section and time is synchronized on all your machines (for example with ntpd) See more at https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#secret-key *** Could not read served logs: Client error '403 FORBIDDEN' for url 'http://6a8606fb0e23:8793/log/dag\_id=dag\_mysql\_operator\_sample/run\_id=manual\_\_2024-03-07T00:15:48.001488+00:00/task\_id=test\_logger/attempt=1.log' For more information check: https://httpstatuses.com/403

 

원인

airflow 디렉토리를 dags 하위 common 폴더에 몰아넣기 위해 compose 파일의 volume 을 다음과 같이 설정해서 생긴 문제였다.
docker-compose.yaml volumes의 경로를 common 하나만 남겨두어서 logs volume 을 인식하지 못함.
x-airflow-common: &airflow-common # In order to add custom dependencies or upgrade provider packages you can use your extended image. # Comment the image line, place your Dockerfile in the directory where you placed the docker-compose.yaml # and uncomment the "build" line below, Then run `docker-compose build` to build the images. image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.8.2} # build: . environment: &airflow-common-env AIRFLOW__CORE__EXECUTOR: CeleryExecutor AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://airflow:airflow@postgres/airflow AIRFLOW__CELERY__BROKER_URL: redis://:@redis:6379/0 AIRFLOW__CORE__FERNET_KEY: '' AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true' AIRFLOW__CORE__LOAD_EXAMPLES: 'false' AIRFLOW__API__AUTH_BACKENDS: 'airflow.api.auth.backend.basic_auth,airflow.api.auth.backend.session' # yamllint disable rule:line-length # Use simple http server on scheduler for health checks # See https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/check-health.html#scheduler-health-check-server # yamllint enable rule:line-length AIRFLOW__SCHEDULER__ENABLE_HEALTH_CHECK: 'true' # WARNING: Use _PIP_ADDITIONAL_REQUIREMENTS option ONLY for a quick checks # for other purpose (development, test and especially production usage) build/extend Airflow image. _PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-} volumes: - ${AIRFLOW_PROJ_DIR:-.}/common:/opt/airflow/dags/common

 

해결 방법

volumes 설정에 logs 위치를 추가하여 해결
volumes: - ${AIRFLOW_PROJ_DIR:-.}/common:/opt/airflow/dags/common - log-volume:/opt/airflow/logs

 

 

'Dev > Airflow' 카테고리의 다른 글

[Airflow/python] airflow Variable  (0) 2024.03.05