Data lake architecture

Teams build a data lake when the data they want to keep no longer fits a warehouse: clickstreams, logs, documents and sensor readings that arrive faster and messier than anyone can model in advance. The lake stores all of it cheaply now and lets the structure be decided later.

Why does data lake architecture matter?

• Cost. Object storage costs a fraction of warehouse storage, so keeping every record is affordable.
• Any format. Tables, JSON, logs, images and audio sit side by side.
• One copy, many engines. SQL queries, batch jobs and model training all read the same files.
• Structure decides the outcome. The storage is the easy part. Without zones, a catalog and owners, a lake turns into a swamp.

What are the components of a data lake?

The diagram at the top of this page shows the parts and how they connect.

• Ingestion. Batch loads and streams that land data as files without reshaping it. The wider data pipeline architecture covers how they are built.
• Object storage. The lake itself: a cloud object store holding files in folders organized by source, dataset and date.
• Zones. Raw, cleaned and curated areas, each holding a more refined version of the data. Named bronze, silver and gold, they are the medallion architecture.
• File formats. Columnar files such as Parquet for tabular data past the raw zone, and a table format where transactions are needed.
• Catalog. The index that maps files to tables: schemas, file locations, partitions and owners. It is what lets an engine run SQL over a folder.
• Compute engines. SQL query engines, batch and stream processors and machine learning frameworks. They run outside the lake, scale separately from it and cost money only while they run.
• Governance. Access control down to the table and column, encryption, audit logs and data lineage.
• Consumers. Analysts, dashboards and data scientists, who reach the data through an engine.

How does a data lake compare with a data warehouse and a lakehouse?

	Data warehouse	Data lake	Lakehouse
Stores	Structured tables	Files of any format	Files under a table format
Schema applied	When data is loaded	When data is read	When data is loaded, on files
Best at	Fast, governed SQL	Cheap storage, machine learning	Both, from one copy
Weak at	Unstructured data, cost at scale	Consistency and governance	Maturity and tuning effort

A lake and a data warehouse often run together: the lake keeps everything, and the warehouse holds the curated subset analysts query. A lakehouse removes the second copy. A table format over the lake’s files adds transactions and schema enforcement, so the curated zone can serve dashboards directly.

How do you read a data lake architecture diagram?

Start on the left of the hero diagram. Databases, SaaS apps and event streams land in the raw zone as files, exactly as they arrived. Jobs then promote the data downward: the cleaned zone holds validated, columnar files, and the curated zone holds tables modeled for analysis.

The dashed arrows show each zone registering its tables in the catalog. That step matters more than it looks. A file that is not in the catalog is invisible to everyone but the person who wrote it.

On the right, the compute engines hold no data of their own. An engine asks the catalog where a table’s files are, reads them from object storage and writes its results back, which is the two-way arrow. Consumers work through an engine, never against the files.

Reading it this way raises the questions a review should ask. Who can read the raw zone? What happens to a file nobody cataloged? How long is raw data kept?

How do you design a data lake?

1. Decide what it is for. List the workloads: reporting, data science, archiving. They determine the zones and engines you need.
2. Define the zones. Write down what each one promises and who may read and write it.
3. Lay out the storage. One folder per source and dataset, partitioned by what queries filter on, usually date. Use columnar formats past the raw zone.
4. Stand up the catalog first. Nothing lands without a catalog entry and an owner.
5. Keep compute separate. Pick engines per workload and let each scale and shut down on its own.
6. Govern by zone. Restrict the raw zone, which often holds personal data, and open the curated zone widely.
7. Draw it. An architecture diagram of zones, catalog, engines and consumers shows a new team where their data belongs.

What are common data lake mistakes?

• No catalog. Files without schemas and owners are storage, not data.
• Loading without owners. Every dataset needs a team that answers for it.
• Millions of small files. Streaming writes create tiny files that slow every query. Compact them on a schedule.
• BI on the raw zone. Dashboards built on raw JSON break whenever a source changes a field.
• One zone. When raw and curated data share a folder, nobody knows which tables to trust.
• Keeping everything forever. Storage is cheap, but unowned data still costs money and carries compliance risk. Set retention by zone.

What tools help with a data lake?

• Object storage and table formats. Every major cloud offers an object store, and open table formats add transactions and schema enforcement on top of its files.
• Catalogs and query engines. A metadata catalog plus one or more engines that read from it: SQL for analysts, batch and stream processors for pipelines.
• AI diagramming tools. Tools like Eraser turn a description of your sources, zones and engines into an architecture diagram you can keep editing as text, which keeps the picture current as datasets are added.

For more on this kind of diagram, see architecture diagrams.

Frequently asked questions