How do I install custom packages in my Amazon MWAA environment?

3 minute read
0

I want to install custom packages with plugins.zip in my Amazon Managed Workflows for Apache Airflow (Amazon MWAA) environment.

Short description

Use the requirements.txt and plugins.zip files to install Python libraries in Amazon MWAA. If you use the requirements.txt file to install packages, then the packages are installed from the Python Package Index (from the PyPI website) by default. If you ship libraries (.whl files) with compiled artifacts, then use the plugins.zip file to install these Python wheels.

Install custom Apache Airflow operators, hooks, sensors, or interfaces with the plugins.zip file. This file is written on the backend Amazon ECS Fargate containers at the location /usr/local/airflow/plugins/. Plugins can also be used to export environment variables and authentication and config files, like .crt and .yaml.

Resolution

Install libraries with Python Wheels

A Python wheel is a package file with compiled artifacts. To install this package, place the (.whl) file in a plugins.zip and then refer to this file in requirements.txt. If you update the environment after you add the .whl file to the plugins.zip, then the .whl is shipped to location /usr/local/airflow/plugins/. This location is in the underlying Amazon Elastic Container Service (Amazon ECS) Fargate containers.

To install libraries with Python Wheels, see How do I install libraries in my Amazon MWAA environment?

Install custom operators, hooks, sensors, or interfaces

Amazon MWAA supports Apache Airflow's built-in plugin manager that allows you to use custom Apache Airflow operators, hooks sensors, or interfaces. These custom plugins can be placed in the plugins.zip file with both a flat and nested directory structure. For examples of custom plugins, see Examples of custom plugins.

Create a custom plugin to generate runtime environment variables

You can also create a custom plugin that generates environment variables at runtime on your Amazon MWAA environment. Then, you can use these environment variables in your Directed Acyclic Graph (DAG) code. For more information, see Creating a custom plugin that generates runtime environment variables.

Export PEM, .crt, and configuration files

If you don't need specific files to be continuously updated when your environment runs, then use plugins.zip to ship these files. Also, you can place files that you don't need to grant access to users that write DAGs. These files include certificate (.crt) files, PEM files, and configuration YAML files. After you zip the files into plugins.zip, upload plugins.zip to S3. Then, update the environment. The files are replicated with required permissions to access /usr/local/airflow/plugins.

To zip custom CA certificates into the plugins.zip file, run the following command:

$ zip plugins.zip ca-certificates.crt

The file is now located at /usr/local/airflow/plugins/ca-certificates.crt. To zip the kube_config.yaml into the plugins.zip file, run the following command:

$ zip plugins.zip kube_config.yaml

The file is now located at /usr/local/airflow/plugins/kube_config.yaml.

Troubleshoot the installation process

If you have issues with the installation of these packages, you can test your DAGs, custom plugins, or Python dependencies locally with aws-mwaa-local-runner.

To troubleshoot issues with installation of Python packages with the plugins.zip file, you can view the log file (requirements_install_ip) from either the Apache Airflow Worker or Scheduler log groups.

Important: It's a best practice to test the Python dependencies and plugins.zip file with the Amazon MWAA CLI utility (aws-mwaa-local-runner) before you install the packages or plugins.zip file on your Amazon MWAA environment.

Related information

Plugins

AWS OFFICIAL
AWS OFFICIALUpdated 7 months ago