Data
Data pipeline architecture
Data pipeline architecture is the design of how data moves from source systems to its users through ingestion, storage, transformation and serving.
Teams design a data pipeline architecture when the number of pipelines outgrows the people who remember how each one works. Agreeing once on the stages, the storage and the standards lets the fiftieth pipeline look like the first.
What are the stages of a data pipeline?
The diagram at the top of this page shows the five stages every pipeline passes through, whatever tools run them.
- Sources. Application databases, SaaS products and event streams. The pipeline does not control them, so their schemas change without warning.
- Ingestion. Getting data out of the sources: scheduled batch loads, change data capture from database logs, or events published to a stream.
- Storage. Where data rests. A data lake, data warehouse or lakehouse holds data for analysis, and an event log holds data in motion. Storage is usually layered, most often as a medallion architecture.
- Transformation. Cleaning, joining and modeling. In batch this is scheduled SQL; in streaming it is a processor that handles one event at a time.
- Serving. Delivering data to its users: dashboards, model training, operational applications and reverse ETL back into business tools.
The band at the bottom of the diagram runs across all five.
- Orchestration. A scheduler that runs jobs in dependency order and retries failures.
- Data quality checks. Tests on volume, nulls, duplicates and freshness, run before bad data moves on.
- Lineage and catalog. What feeds what, which is data lineage, and a searchable list of tables.
- Monitoring and alerts. The observability of the pipeline itself.
How do batch and streaming pipelines differ?
| Batch | Streaming | |
|---|---|---|
| Runs | On a schedule | Continuously |
| Data is | Minutes to hours old | Seconds old |
| Built from | An orchestrator and SQL in a warehouse | An event log and a stream processor |
| Good for | Reporting, finance, model training | Fraud checks, monitoring, live personalization |
| Hard part | Late and partial loads | Ordering, duplicates and state |
Most enterprises run both, which is why the hero diagram has two rows. Batch is simpler and cheaper, so it should be the default. Streaming earns its complexity only where a decision depends on data that is seconds old. The event log on the streaming path is usually Kafka; see Kafka architecture for how it stores and replays events.
What are the common data pipeline architecture patterns?
| Pattern | How it works | Use it when |
|---|---|---|
| ETL | Transform data before loading it | Data must be masked or reshaped before it is stored |
| ELT | Load raw data, then transform it in the warehouse | You run a cloud warehouse or lakehouse, which is most teams |
| Lambda | A batch layer and a speed layer, merged when queried | You need complete history and live results, and can afford two code paths |
| Kappa | One streaming path that replays the log to reprocess | Events are the source of truth and the log keeps enough history |
| Medallion | Bronze, silver and gold layers in storage | Many teams share one lake and need to know which tables to trust |
These are not five alternatives. ETL and ELT decide where transformation happens, and the ETL pipeline page covers that choice. Lambda and Kappa decide how batch and streaming are combined. Medallion decides how storage is layered. A typical architecture picks one of each.
How do you read a data pipeline architecture diagram?
Read the hero diagram a row at a time. On the top row, SaaS apps and application databases are pulled by batch loads every hour or night, land in the lake or warehouse as raw copies, are modeled by scheduled SQL and reach dashboards and model training.
On the bottom row, event streams and database changes arrive within seconds. The dashed arrow is change data capture, which streams the application database as well. Events land in a replayable log, a stream processor handles them one by one and real-time applications act on the results. The arrow from the event log up to the lake archives every event, so history can also be queried in batch.
Then ask what the diagram does not say. Where is the raw copy kept, and for how long? Which stage stops a bad load? What happens when a source adds a column?
How do you design a data pipeline architecture?
- Start from the consumers. List the dashboards, models and applications, and how fresh each one needs its data.
- Classify the sources. Note the volume, the rate of change and how each can be read: API, database log or files.
- Choose batch or streaming per flow. Decide by the freshness the consumer needs.
- Pick the storage and its layers. One place for raw data, one for modeled data and a rule for which is which.
- Standardize transformation. One framework, in version control, with tests.
- Make every job idempotent and observable. Any run can be repeated, and every run reports volume and freshness.
- Draw it. Keep a data flow diagram of the whole architecture beside the code.
What are common data pipeline architecture mistakes?
- Streaming everything. A nightly finance report does not need a stream processor.
- No raw copy. Without one, a transformation bug cannot be fixed by rerunning.
- One-off pipelines. When every team ingests its own way, nobody can cover for anyone else.
- Ignoring schema change. Sources will add, rename and drop fields. Detect it at ingestion, not on a dashboard.
What tools help with data pipeline architecture?
- Ingestion and streaming platforms. Managed connectors for SaaS and database sources, change data capture tools and event logs.
- Storage, transformation and orchestration tools. Cloud warehouses and lakehouses, SQL transformation frameworks with tests, and schedulers that run the jobs in order.
- AI diagramming tools. Tools like Eraser turn a description of your sources, stages and consumers into a data flow diagram or an architecture diagram you can keep editing as text, so the design can be reviewed before the first job is written.
For more on this kind of diagram, see data flow diagrams.
Frequently asked questions
-
What is the difference between a data pipeline and an ETL pipeline?
A data pipeline is any automated flow of data from one system to another. An ETL pipeline is one kind: it extracts from sources, transforms and loads an analytical store, usually in batches. Streaming flows, database replication and reverse ETL back into applications are data pipelines that are not ETL.
-
What is the difference between batch and streaming pipelines?
A batch pipeline processes the data that has accumulated since its last run, on a schedule, so results are minutes to hours old. A streaming pipeline processes each event as it arrives, so results are seconds old, at the cost of more moving parts. Choose by how fresh the data must be for the decision it feeds.
-
What is the difference between Lambda and Kappa architecture?
Lambda architecture runs a batch layer and a speed layer side by side and merges their results when a query arrives. Kappa architecture uses a single streaming path and reprocesses history by replaying the event log. Lambda means maintaining the same logic twice; Kappa needs a log that keeps enough history to replay.
-
What is change data capture?
Change data capture, or CDC, reads a database's transaction log and emits every insert, update and delete as an event. It gives a pipeline each change within seconds without running queries against the tables, which is why it is the usual way to stream an application database into a warehouse or an event log.
