Data fabric

Teams look at a data fabric when their data is spread across on-premises databases, several clouds and dozens of SaaS tools, and moving all of it into one place is no longer realistic. A fabric leaves the data where it is and connects it through a layer of metadata.

Why does a data fabric matter?

• Data stays put. Regulated, very large or rarely used data does not have to be copied to be usable.
• One place to look. People search one catalog, not five systems.
• Less integration built by hand. The fabric uses what it knows about the data to suggest and generate pipelines.
• One set of rules. A masking policy is defined once and applies whichever system holds the column.

What are the components of a data fabric?

The diagram at the top of this page shows the layers between the sources and the people who use them.

• Data catalog. An inventory of every dataset in every connected system, with its schema, owner and description.
• Active metadata. Usage, query logs, lineage and schema changes, collected continuously and analyzed. The FAQ below covers how it differs from documentation.
• Knowledge graph. The metadata stored as a graph of datasets, columns, people and business terms, so the fabric can tell that cust_id in one system is customer_key in another.
• Integration. Three ways to deliver data: query it in place, replicate it on a schedule or stream it as it changes.
• Governance. Access, masking and retention policies, defined once and enforced at every source.
• Automation. Recommendations and generated pipelines that come from analyzing the metadata, often with machine learning.
• Access layer. One interface, usually SQL or an API, through which consumers reach any connected dataset.

How does a data fabric work?

Follow a request across the hero diagram. An analyst searches the catalog for claims data and finds it in two places, an on-premises database and the cloud warehouse. The active metadata shows which copy is fresher and which one other analysts use, so the choice is informed.

The analyst queries the dataset through the fabric. For an occasional question, the integration layer reads the source in place. If the same query starts running every hour, the fabric recommends replicating the table, and can build that pipeline itself. Either way, the governance layer masks the personal columns before the result comes back.

Every step adds metadata. The query, the join and the user are recorded, which is what makes the next recommendation better. Data lineage is one output of the same record.

How does a data fabric compare with a data mesh?

	Data fabric	Data mesh
Starts from	Technology	Organization
Who integrates the data	An automated metadata layer	Domain teams, as data products
Central team	Runs the fabric	Runs the platform, not the data
Main risk	Metadata too thin to automate anything	Domains without the skills to own data

The two are often presented as rivals, but they answer different questions. A fabric is about how data is connected. A data mesh is about who is responsible for it. A mesh still needs a catalog, integration and policy enforcement, and a fabric can provide them.

How does a data fabric compare with a data lake or warehouse?

	Data lake or warehouse	Data fabric
What is centralized	The data	The metadata
Adding a source	Build a pipeline to copy it	Connect it and catalog it
Suits	Analytics on data you can move	Data spread across systems that cannot all move

A data lake and a data warehouse centralize the data. A fabric centralizes the knowledge about the data and leaves the storage distributed. Most fabrics include a lake or a warehouse as one of their sources, and the data pipeline architecture that feeds it becomes one of the integration styles the fabric manages.

How do you build a data fabric?

1. Inventory the sources. List the systems that hold data people ask for, and who owns each.
2. Stand up the catalog. Connect it to every source and harvest schemas automatically.
3. Collect active metadata. Add query logs, lineage and usage, so the catalog reflects what is used and not only what exists.
4. Add integration by need. Query in place first, and replicate or stream only where performance requires it.
5. Define policies once. Classify sensitive data and attach access and masking rules to the classification, not to each table.
6. Automate gradually. Start with recommendations a person approves, then let the proven ones run unattended.
7. Draw it. An architecture diagram of sources, fabric layers and consumers shows what is connected and what is not.

What are common data fabric mistakes?

• Buying a tool and calling it a fabric. Without connected, current metadata, a platform is only another silo.
• A catalog filled in by hand. Passive documentation goes stale within months and automates nothing.
• Virtualizing everything. Large joins across live sources are slow and load the systems underneath.
• Ignoring ownership. A fabric can show that data is wrong, but not who fixes it.
• Connecting everything at once. Start with the few systems people ask about most, prove the catalog is used, then widen.

What tools help with a data fabric?

• Catalogs and metadata platforms. Tools that harvest schemas, lineage and usage from every source into one searchable graph.
• Integration and virtualization tools. Query federation engines, replication services and streaming platforms, driven from the same metadata.
• AI diagramming tools. Tools like Eraser turn a description of your sources, fabric layers and consumers into an architecture diagram you can keep editing as text, which helps explain a design that spans many systems.

For more on this kind of diagram, see architecture diagrams.

Frequently asked questions