Definition
A data pipeline is an automated series of steps that collects data from one or more sources, processes it, and delivers it to a destination such as a data warehouse, a dashboard, a machine learning model or another application. Each step may filter, clean, combine, enrich or reshape the data. Pipelines run on a schedule, every hour or every night, or continuously as events stream in. ETL and ELT are specific kinds of data pipeline.
In a company, data pipelines are the plumbing behind every automated report and every AI feature. One pipeline lands billing data in the warehouse. Another streams product events so dashboards update in near real time. Another prepares documents so an AI assistant can search them. Orchestration tools such as Airflow, Dagster and Prefect schedule the steps, retry failures and alert when something breaks. In mature data teams, pipelines are treated like software: versioned, tested and monitored.
The misconception is that a pipeline, once built, stays built. Source systems change a field name, an API adds a limit, a file arrives late, and the pipeline silently produces wrong numbers or none at all. Monitoring data quality, not just whether the job ran, is what separates reliable pipelines from fragile ones. In 2026 the fastest-growing pipelines are those feeding AI: keeping the documents and records an agent relies on fresh and correct is now a core operational task.
In practice
A company's churn dashboard showed zero cancellations for three weeks and the team celebrated. The pipeline had broken when the billing provider renamed a field. The real churn, when discovered, was higher than usual. Data quality alerts now flag any metric that falls outside its normal range.
Why it matters
Data pipelines are invisible when they work and expensive when they fail quietly. Ask your team how they would know if a pipeline started producing wrong numbers, not just whether it ran. The answer tells you how much to trust your dashboards.
Frequently asked questions
- What is the difference between a data pipeline and ETL?
- ETL, extract, transform, load, is one specific type of data pipeline aimed at feeding a data warehouse. Data pipeline is the broader term for any automated flow of data between systems, including real-time streaming, machine learning feature preparation, and syncing data between applications. All ETL is a pipeline; not every pipeline is ETL.
- How do you monitor a data pipeline?
- At two levels. Operational monitoring checks that each job ran, how long it took and whether it failed. Data quality monitoring checks the output: row counts, missing values, unexpected duplicates and metrics that fall outside normal ranges. The second level catches the silent failures that cause wrong business decisions.