Here’s a structured, useful blog post about — written for data engineers who want to move beyond basic tasks and build real DAGs. Mastering XComs in Apache Airflow: Cross‑Task Communication Without the Pain One of the first surprises when learning Airflow is that tasks run isolated from each other. You can’t just set task_2.data = task_1.data . So how do you pass a value from one task to another? XComs .
XComs are not designed for large data. Default size limit is 1 MB (configurable, but don’t). Use them for IDs, file paths, dates, small JSON – not DataFrames or images. The Two Ways to Use XComs 1. Implicit XComs via return Any Python function decorated with @task (TaskFlow API) automatically pushes its return value as an XCom. xcom in airflow
@task def process(user_data: dict) -> str: return f"Processed user user_data['name']" Here’s a structured, useful blog post about —
from airflow.operators.python import PythonOperator def push_function(**context): context['ti'].xcom_push(key='user_id', value=123) So how do you pass a value from one task to another
XCom (short for cross‑communication ) is Airflow’s built‑in mechanism for exchanging small pieces of data between tasks. When used wisely, they unlock powerful patterns. When abused, they break your DAGs. Let’s see how to use them correctly. XComs are key‑value pairs stored in Airflow’s metadata database. A task can push an XCom (write a value under a key), and another task can pull that value (read it).
@task def extract() -> dict: return "user_id": 123, "name": "Alice" # pushed automatically