apache/airflow

Getting started

Visit the official Airflow website documentation (latest stable release) for help with installing Airflow, getting started, or walking through a more complete tutorial.

For more information on Airflow Improvement Proposals (AIPs), visit the Airflow Wiki.

Documentation for dependent projects like provider distributions, Docker image, Helm Chart, you'll find it in the documentation index.

Installing from PyPI

We publish Apache Airflow as apache-airflow package in PyPI. Installing it however might be sometimes tricky because Airflow is a bit of both a library and application. Libraries usually keep their dependencies open, and applications usually pin them, but we should do neither and both simultaneously. We decided to keep our dependencies as open as possible (in pyproject.toml) so users can install different versions of libraries if needed. This means that pip install apache-airflow will not work from time to time or will produce unusable Airflow installation.

To have repeatable installation, however, we keep a set of "known-to-be-working" constraint files in the orphan constraints-main and constraints-2-0 branches. We keep those "known-to-be-working" constraints files separately per major/minor Python version. You can use them as constraint files when installing Airflow from PyPI. Note that you have to specify correct Airflow tag/version/branch and Python versions in the URL.

While it is possible to install Airflow with tools like Poetry or pip-tools, they do not share the same workflow as pip - especially when it comes to constraint vs. requirements management. Installing via Poetry or pip-tools is not currently supported.

If you wish to install Airflow using those tools, you should use the constraint files and convert them to the appropriate format and workflow that your tool requires.

• Installing just Airflow:
• Installing with extras (i.e., postgres, google)

pip install 'apache-airflow==3.3.2' \
 --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-3.3.2/constraints-3.10.txt"

Installation

For comprehensive instructions on setting up your local development environment and installing Apache Airflow, please refer to the INSTALLING.md file.

Requirements

Note: SQLite is used in Airflow tests. Do not use it in production. We recommend using the latest stable version of SQLite for local development.

Note: Airflow currently can be run on POSIX-compliant Operating Systems. For development, it is regularly tested on fairly modern Linux Distros and recent versions of macOS. On Windows you can run it via WSL2 (Windows Subsystem for Linux 2) or via Linux Containers. The work to add Windows support is tracked via #10388, but it is not a high priority. You should only use Linux-based distros as "Production" execution environment as this is the only environment that is supported. The only distro that is used in our CI tests and that is used in the Community managed DockerHub image is Debian Bookwo

• Python — 3.11, 3.12, 3.13, 3.14 — 3.10, 3.11, 3.12, 3.13, 3.14 — 3.10, 3.11, 3.12
• Platform — AMD64/ARM64 — AMD64/ARM64 — AMD64/ARM64(\)
• Kubernetes — 1.31, 1.32, 1.33, 1.34, 1.35, 1.36, 1.37 — 1.30, 1.31, 1.32, 1.33, 1.34, 1.35 — 1.26, 1.27, 1.28, 1.29, 1.30
• PostgreSQL — 14, 15, 16, 17, 18 — 14, 15, 16, 17, 18 — 12, 13, 14, 15, 16
• MySQL — 8.0, 8.4, Innovation — 8.0, 8.4, Innovation — 8.0, Innovation
• SQLite — 3.15.0+ — 3.15.0+ — 3.15.0+

Approach to dependencies of Airflow

Airflow has a lot of dependencies - direct and transitive, also Airflow is both - library and application, therefore our policies to dependencies has to include both - stability of installation of application, but also ability to install newer version of dependencies for those users who develop Dags. We developed the approach where constraints are used to make sure airflow can be installed in a repeatable way, while we do not limit our users to upgrade most of the dependencies. As a result we decided not to upper-bound version of Airflow dependencies by default, unless we have good reasons to

The constraint mechanism of ours takes care about finding and upgrading all the non-upper bound dependencies automatically (providing that all the tests pass). Our main build failures will indicate in case there are versions of dependencies that break our tests - indicating that we should either upper-bind them or that we should fix our code/tests to account for the upstream changes from those dependencies.

Whenever we upper-bound such a dependency, we should always comment why we are doing it - i.e. we should have a good reason why dependency is upper-bound. And we should also mention what is the condition to remove the binding.

Approach for dependencies for Airflow Core

There are few dependencies that we decided are important enough to upper-bound them by default, as they are known to follow predictable versioning scheme, and we know that new versions of those are very likely to bring breaking changes. We commit to regularly review and attempt to upgrade to the newer versions of the dependencies as they are released, but this is manual process.

• SQLAlchemy: upper-bound to specific MINOR version (SQLAlchemy is known to remove deprecations and
• introduce breaking changes especially that support for different Databases varies and changes at
• various speed)
• Alembic: it is important to handle our migrations in predictable and performant way. It is developed
• together with SQLAlchemy. Our experience with Alembic is that it very stable in MINOR version
• Flask: We are using Flask as the back-bone of our web UI and API. We know major version of Flask
• are very likely to introduce breaking changes across those so limiting it to MAJOR version makes sense
• werkzeug: the library is known to cause problems in new versions. It is tightly coupled with Flask
• libraries, and we should update them together
• celery: Celery is a crucial component of Airflow as it used for CeleryExecutor (and similar). Celery
• follows SemVer, so
• we should upper-bound it to the next MAJOR version. Also, when we bump the upper version of the library,
• we should make sure Celery Provider minimum Airflow version is updated.
• kubernetes: Kubernetes is a crucial component of Airflow as it is used for the KubernetesExecutor

Approach for dependencies in Airflow Providers and extras

The main part of the Airflow is the Airflow Core, but the power of Airflow also comes from a number of providers that extend the core functionality and are released separately, even if we keep them (for now) in the same monorepo for convenience. You can read more about the providers in the Providers documentation. We also have set of policies implemented for maintaining and releasing community-managed providers as well as the approach for community vs. 3rd party providers in the providers document.

Those extras and providers dependencies are maintained in provider.yaml of each provider.

By default, we should not upper-bound dependencies for providers, however each provider's maintainer might decide to add additional limits (and justify them with comment).

Principles

• Dynamic: Pipelines are defined in code, enabling dynamic dag generation and parameterization.
• Extensible: The Airflow framework includes a wide range of built-in operators and can be extended to fit your needs.
• Flexible: Airflow leverages the Jinja templating engine, allowing rich customizations.