To resolve issues related to increased memory usage and worker pod evictions in your Cloud Composer 2 environment, the following steps are recommended:
* Increase Memory Available to Airflow Workers:
* By increasing the memory allocated to Airflow workers, you can handle more memory-intensive tasks, reducing the likelihood of pod evictions due to memory limits.
* Increase Maximum Number of Workers and Reduce Worker Concurrency:
* Increasing the number of workers allows the workload to be distributed across more pods, preventing any single pod from becoming overwhelmed.
* Reducing worker concurrency limits the number of tasks that each worker can handle simultaneously, thereby lowering the memory consumption per worker.
Steps to Implement:
* Increase Worker Memory:
* Modify the configuration settings in Cloud Composer to allocate more memory to Airflow workers. This can be done through the environment configuration settings.
* Adjust Worker and Concurrency Settings:
* Increase the maximum number of workers in the Cloud Composer environment settings.
* Reduce the concurrency setting for Airflow workers to ensure that each worker handles fewer tasks at a time, thus consuming less memory per worker.
Reference Links:
* Cloud Composer Worker Configuration
* Scaling Airflow Workers