Back to index
OtherSelf-paced, around 20 to 40 hours to become competent·Completely free and open source

Apache Airflow Official Documentation and Tutorials

3.8

Comprehensive documentation for a tool that everyone in data engineering has to learn, written in a way that explains every feature and never tells you how to design a pipeline.

What We Liked

  • Core concepts section explains the execution model properly, which is essential
  • Fundamental tutorial gets a working pipeline running quickly
  • Provider and operator reference is exhaustive and current
  • Best practices page contains genuinely hard won advice
  • Free, and the skill is directly requested in a large number of job listings

What Could Be Better

  • The scheduling and data interval model confuses nearly everyone at first
  • Local setup is heavier than it should be for learning purposes
  • Version differences across major releases make external guidance unreliable
  • Documentation covers features, not pipeline design, which is the actual skill
  • Testing and debugging guidance is thin relative to how painful both are

Detailed review

Airflow is one of those tools that nobody chooses because they love it and everybody ends up using because it is what the industry settled on. It orchestrates workflows, which in practice means it runs your data pipelines on a schedule, handles dependencies between steps, retries what fails, and gives you a view of what ran and what did not. Every machine learning system that needs data to arrive reliably has something doing this job, and more often than not it is Airflow. The documentation is comprehensive and it is the right place to learn from, because the alternative sources are worse.

The core concepts section is where to start and where to spend real time. It explains what a directed acyclic graph is in this context, what a task is, what an operator is, how the scheduler decides what to run, and how execution dates and data intervals work. That last one is the concept that determines whether you will fight this tool for a year or work with it, and it is where almost everyone goes wrong initially. A pipeline scheduled daily does not run for today's date, it runs for the interval that just closed, and the run happens at the end of the period it covers.

This is correct behaviour for data processing, it is not what most people expect from a scheduler, and misunderstanding it produces pipelines that quietly process the wrong day's data. The documentation explains it. Read that section twice, and if you think you have understood it after one pass, read it again. The fundamental tutorial is well judged.

It builds a simple pipeline with real tasks and real dependencies and gets you to a running result quickly, which is the right shape for a first encounter. From there the taskflow API material shows the more modern way of writing pipelines with decorated Python functions, which is considerably more pleasant than the older operator instantiation style and is what new work should use. The provider reference is exhaustive. Airflow's usefulness comes largely from the enormous library of integrations for cloud services, databases, transfer tools and everything else, and the reference documents them all with parameters and examples.

When you need to know exactly what a particular operator does, the answer is there. The best practices page is better than most projects manage and worth reading before you write anything real. Keep tasks idempotent so a rerun is safe. Do not put heavy computation in the file that defines your pipeline, because the scheduler parses it constantly.

Design for backfilling from the start. These are lessons normally learned by breaking production and it is good that someone wrote them down. Now the problems. Local setup for learning is heavier than it needs to be.

Between the scheduler, the web server, the metadata database and the executor, getting a development environment running takes longer than a person evaluating the tool wants to spend. There are container based options and a standalone mode and they are less prominent in the documentation than they should be. Version fragmentation is a practical nuisance. Airflow has been through major releases with real breaking changes, and the internet is full of tutorials, forum answers and blog posts written for versions that behaved differently.

Import paths moved, defaults changed, and features were added and deprecated. The official docs are versioned and correct, and anything you find elsewhere may not be, so check what version it targets before trusting it. The deepest limitation is one the documentation cannot really fix. It teaches you Airflow.

It does not teach you pipeline design, and pipeline design is the actual skill. How to decompose a workflow into tasks of the right size. When to split a pipeline and when to keep it together. How to handle late arriving data, partial failures and dependencies on systems you do not control.

What to do when yesterday's run needs rerunning and downstream consumers have already read the bad output. None of that is in the documentation and all of it determines whether your pipelines are maintainable. That knowledge is currently transmitted by working alongside someone who has been burned, which is a poor distribution mechanism. Testing and debugging deserve more than they get.

Testing pipelines is genuinely awkward, the guidance is brief, and debugging a task that failed in a scheduled run three hours ago is a specific skill involving logs, the interface and a certain amount of intuition. Given how much time practitioners spend on exactly this, the coverage is thin. My three point eight is for thorough, current, well organised documentation for a tool that is unavoidable in this line of work, marked down for a scheduling model that traps everyone despite being documented, for a setup process that discourages casual learning, and for teaching the software rather than the craft. Learn it because you will need it, and expect the useful understanding to arrive the first time something breaks at three in the morning.

[ final ]

The verdict.

Learn it because the jobs ask for it, read the core concepts twice, and expect the real understanding to come from operating something in production rather than from any documentation.