반응형
Airflow 로그가 출력되지 않는 이슈
docker-compose 로 airflow 를 실행하는데 로그가 안나오고 다음과 같은 에러만 발생
!!!! Please make sure that all your Airflow components (e.g. schedulers, webservers, workers and triggerer) have the same 'secret_key' configured in 'webserver' section and time is synchronized on all your machines (for example with ntpd) See more at https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#secret-key *** Could not read served logs: Client error '403 FORBIDDEN' for url 'http://6a8606fb0e23:8793/log/dag\_id=dag\_mysql\_operator\_sample/run\_id=manual\_\_2024-03-07T00:15:48.001488+00:00/task\_id=test\_logger/attempt=1.log' For more information check: https://httpstatuses.com/403
원인

airflow 디렉토리를 dags 하위 common 폴더에 몰아넣기 위해 compose 파일의 volume 을 다음과 같이 설정해서 생긴 문제였다.
docker-compose.yaml volumes의 경로를 common 하나만 남겨두어서 logs volume 을 인식하지 못함.
x-airflow-common:
&airflow-common
# In order to add custom dependencies or upgrade provider packages you can use your extended image.
# Comment the image line, place your Dockerfile in the directory where you placed the docker-compose.yaml
# and uncomment the "build" line below, Then run `docker-compose build` to build the images.
image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.8.2}
# build: .
environment:
&airflow-common-env
AIRFLOW__CORE__EXECUTOR: CeleryExecutor
AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://airflow:airflow@postgres/airflow
AIRFLOW__CELERY__BROKER_URL: redis://:@redis:6379/0
AIRFLOW__CORE__FERNET_KEY: ''
AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
AIRFLOW__API__AUTH_BACKENDS: 'airflow.api.auth.backend.basic_auth,airflow.api.auth.backend.session'
# yamllint disable rule:line-length
# Use simple http server on scheduler for health checks
# See https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/check-health.html#scheduler-health-check-server
# yamllint enable rule:line-length
AIRFLOW__SCHEDULER__ENABLE_HEALTH_CHECK: 'true'
# WARNING: Use _PIP_ADDITIONAL_REQUIREMENTS option ONLY for a quick checks
# for other purpose (development, test and especially production usage) build/extend Airflow image.
_PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-}
volumes:
- ${AIRFLOW_PROJ_DIR:-.}/common:/opt/airflow/dags/common
해결 방법
volumes 설정에 logs 위치를 추가하여 해결volumes:
- ${AIRFLOW_PROJ_DIR:-.}/common:/opt/airflow/dags/common
- log-volume:/opt/airflow/logs
'Dev > Airflow' 카테고리의 다른 글
[Airflow/python] airflow Variable (0) | 2024.03.05 |
---|