Three to four engineers. More than 1,000 active Dags. On the Data Flowcast, Matt Stavinga of Idelic (now part of Descartes) shares how a small team runs Airflow on Astro at that scale: → A shared framework that generates task groups for each ETL phase → Consistent logging across every integration → A separate Airflow environment for each pull request 🎧 Listen: astronomer.io/podcast/scalin… #ApacheAirflow #DataEngineering

Calling an LLM takes one line of code. Running that call every day in production takes retries, guardrails, structured outputs, and a way for a human to review the result. That second part is orchestration. Our new 100+ page guide covers: → Orchestrating any model with Airflow, with any harness → Patterns for LLM calls, agents, and multi agent workflows → Guardrails, structured outputs, and human review for AI tasks Code snippets, Dag patterns, and a companion GitHub repo included. 📘 Get the guide: astronomer.io/ebooks/ai-orch… #ApacheAirflow #DataEngineering

A pipeline that never fails never alerts. Cargill's William Orgertrice III has seen one run 10 hours instead of one. On the Data Flowcast, William shares how Cargill runs Airflow on Astro for 60+ teams, and how to stop silent failures: → One purpose per task → Notifications on success and failure → Monitor run duration, not only status Read the recap: astronomer.io/airflow/airflo… #ApacheAirflow #DataEngineering

One data engineer. A fintech's entire Airflow platform. A cloud bill that stays small. On the latest Data Flowcast, Aniruddha Sengupta of Insignis joins Kenten Danas to break down how a data team of one runs a low cost, scalable Airflow platform. What they cover: 🔍 Observability that works when there is no one else on call 🧩 Dag level practices that keep the platform maintainable 💰 Tariff and Maestro, two internal tools Insignis built to keep costs in check 🤖 Where AI fits on the roadmap 🎧 Watch or listen: youtube.com/watch?v=BuPgvpEC… #ApacheAirflow #DataEngineering

📝 Reminder that the annual Airflow Survey is open! We're reaching out to gather your valuable feedback, and we'd love to hear from you. 🚀 As key members of the Airflow community, your insights are crucial in helping us understand how Airflow is being used in your day-to-day operations and identifying areas for improvement. This is everyone's opportunity to directly influence the future of the project! 🎟️ The survey takes about 7 minutes, and as a thank-you, we're offering participants a free AI Orchestration Fundamentals Certification Exam or Dag Authoring Certification (valued at $150 each). 👉 You can take the survey here: astronomer.typeform.com/airf…

When a Dag fails, the answer is spread across task logs, lineage, code changes, and upstream data. Airflow is where all of that context lives, which makes it the right foundation for an AI agent that diagnoses pipeline issues. We're excited to sponsor Airflow Summit Online Reconnect (Nov 4-5), where Stephanie Niu and Christine S. of Astronomer will present "How We Taught an Agent Airflow," a standout from #AirflowSummit Austin. Catch the session to see: ✅ How live Airflow context makes AI troubleshooting more accurate ✅ How Otto, our data engineering agent, draws on years of real-world Airflow operations ✅ How one team cut resolution time by 95% and shifted from firefighting to reviewing fixes ✅ A framework for deciding which parts of self-healing pipelines to build yourself Register now: lnkd.in/eQ7cjmKm

When a Dag fails, the answer is spread across task logs, lineage, code changes, and upstream data. Airflow is where all of that context lives. That makes it the right foundation for an AI agent that diagnoses pipeline issues. We're sponsoring Airflow Summit Online Reconnect on November 4 and 5, where Stephanie Niu and Christine S. of Astronomer will present "How We Taught an Agent Airflow", a standout session from #AirflowSummit Austin. Catch the session to see: ✅ How live Airflow context makes AI troubleshooting more accurate ✅ How Otto, our data engineering agent, draws on years of real world Airflow operations ✅ How one team cut resolution time by 95% and shifted from firefighting to reviewing fixes ✅ A framework for deciding which parts of self-healing pipelines to build yourself Register now: airflowsummit.org/onlinereco…

What you should know about ETL/ELT pipelines in 2026. Join Airflow expert Marc Lamberti on October 15 to learn the latest Airflow features that make ETL/ELT orchestration easier today. We'll cover core best practices, plus how to handle new challenges like prepping data for both humans and agents, so you're fully up to speed on modern data engineering. Register here: bit.ly/3W9RS5E

Last chance to register for our webinar tomorrow! You’ve probably already heard from your data engineering team that most agents aren’t very good at Airflow work. We’re putting a data engineering agent (built on context from 9 years of managing Airflow) to the test in a live session with Airflow experts on September 29. See for yourself how the agent performs against serious Airflow challenges including upgrading to Airflow 3, reviewing pipeline code, and the in’s and out’s of daily maintenance. Register now: bit.ly/4e37ttU

The definitive guide for data engineers who are serious about orchestrating AI in production. A Data Engineer’s Guide to AI Orchestration with Apache Airflow®, a 100+ page technical guide that teaches: 🔧 How to orchestrate any AI model with Airflow (any model, any harness) ⚡ Patterns for LLM, agent orchestration, and multi-agent workflows 🧠 How to add guardrails, structured outputs, and human review to AI tasks Written by Airflow experts, based on real world examples, and complete with code snippets, Dag patterns, example architectures, and a companion GitHub repo. Download now: bit.ly/4AveQUF

Can an agent automatically debug your gnarliest pipeline failures? What about rewrite your Dag code for Airflow 3? We’re putting a data engineering agent to the test in a live session with Airflow experts on September 29. See for yourself how the agent holds up against serious Airflow challenges, including: - Upgrading from Airflow 2.x to 3 - Reviewing pipeline code with hidden issues like provider conflicts - Day to day Airflow maintenance Register now: bit.ly/4iGvf26

Today's the day! Orchestrate Everything kicks off shortly at 11a ET with @marclambertiml's AI Orchestration Fundamentals crash course (with $150 value exam code for free!), followed by real production stories from data leaders at Ramp, Wix, Mercor, Lyft, Aampe, and Astronomer! We'll close the day with practical orchestration training from Airflow expert Tamara Fingerlin, complete with a public GitHub repo you can use. Free to register, virtual to watch, video-on-demand coming soon if you can't join us live. Tap in: astronomer.io/events/orchest…

Most AI coding assistants know how to write code, but they don't know how *your* team writes code. Wix, Yarden Wolf's team closed this gap by embedding organizational knowledge directly into an agentic coding platform, built for their 100+ data engineers running 8,000+ Airflow Dags. The estimated return on that investment has been 850 engineering hours back every month. At Orchestrate Everything on September 16, Yarden will walk through the architecture journey alongside data leaders from Ramp, Lyft, Aampe, Mercor, and more, all sharing real production stories. Free, virtual, and it comes with Airflow Expert Marc Lamberti's Airflow AI crash course plus a $150 exam code. Register now: bit.ly/4gkvWMX

Somewhere at Ramp right now, a model is deciding a business's credit limit. It's one of 100+ ML models running in production there, covering everything from credit risk assessment to sales lead valuation. Ryan Stevens, Director of Applied Sciences at Ramp, will be at Orchestrate Everything on Sept 16 to talk through how Airflow orchestrates the full lifecycle behind those decisions, from feature creation to data quality checks to large-batch inference. Register now with the link in the comments to hear from Ryan, as well as data leaders from Lyft, Mercor, Wix, Aampe, and more. It's free, virtual, and registration comes with Marc Lamberti's Airflow AI crash course plus a $150 exam code. Register now: astronomer.io/events/orchest…

🤯 Just 3% of your Dags cause two-thirds of your failures. And almost none of them trace back to a deploy. Airflow expert Constance Martineau analyzed 400 million production pipeline runs to find out where the real risk lives. Join us October 1 at 11am ET to see what she found. You'll learn: 👀 Where most pipeline failures actually start 🔧 The 4 moves to stop chasing the wrong DAG ⚡ What to check instead of the last deploy when something breaks Register now: bit.ly/4gV93yD #airflow #webinar #dataengineering

Every AI model gets graded by a human against a rubric. At Mercor, that grade becomes a line item. Multiply across thousands of experts and hundreds of domains and what you have is a flood of inconsistent task data that still has to become a correct invoice, every time, for $4M/day in payouts. Patrick Old will present at Orchestrate Everything (September 16, free to register) on how Airflow + dbt Cosmos on Astro turn that into one canonical source of truth. Register now using for more real production war stories from data leaders at companies like Wix, Lyft, Aampe, Ramp, and more, as well as practical Airflow for AI training from experts at Astronomer: bit.ly/4gkvWMX

Orchestrate Everything is all about production use cases: Ramp orchestrating 100+ ML models in production, Aampe orchestrating millions of agents, 200 billion decisions a week. Every session comes with proof. Airflow expert Tamara Fingerlin's session comes with a public repo. On September 16 she will walk through the Dag pattern behind every stage of a pipeline, from basic ETL to putting AI into your Dags with @ task.llm and @ task.agent. Even better, she'll share code for every example that's yours to run. Sept 16, free and virtual + Marc Lamberti's Airflow AI crash course and a $150 exam code to kick things off. Register today: bit.ly/4gkvWMX

Most agent demos show you one agent doing one task; Aampe runs millions of them in production. One per user, each deciding when and what to send, continuously. 200 billion decisions a week behind apps like Swiggy and Grab. Hundreds of Airflow Dags underneath it, in production, right now. CTO and cofounder Sami Abboud will walk through the architecture that make it all possible at Orchestrate Everything, Sept 16. Register now: astronomer.io/events/orchest… #dataengineering #AI #virtualconference