Decentralized Architecture
Introduction
Data mesh is a sociotechnical method for building a decentralized data architecture by leveraging a self-service domain-oriented design (in a software development perspective), and borrows the theory of domain-driven design from Eric Evans[1] and the theory of team topologies from Manuel Pais and Matthew Skelton.[2] Data mesh is all about the data itself, taking the data lake and pipelines as a concern. secondary.[3] The primary proposition is to scale analytics data through domain-oriented decentralization.[4] With data mesh, responsibility for analytics data is transferred from the central data team to domain teams, supported by a data platform team that provides a domain-independent data platform.[5] This helps better organize data and avoids having separate isolated areas of data. It is due to the presence of a central system that makes sure that everyone follows the same basic rules within the data network, allowing data to be shared across different locations.
History
The term data mesh was first defined by Zhamak Dehghani in 2019[6] while working as the principal consultant at technology company Thoughtworks.[7][8] Dehghani introduced the term in 2019 and then provided more details on the principles and logical architecture throughout 2020. The process was predicted to be a "big contender" for companies in 2022.[9][10] Some of the companies that have implemented data meshes are Zalando,[11] Netflix,[12] Intuit,[13] VistaPrint, JPMorgan Chase,[14] PayPal[15] and others.
In 2022, Dehghani left Thoughtworks to found Nextdata Technologies and focus on decentralized data.[16].
Beginning
The data mesh is based on four fundamental principles:[5].
In addition to these principles, Dehghani writes that data products created by each domain team must be discoverable, addressable, trustworthy, possess self-describing semantics and syntax, be interoperable, secure, and governed by global standards and access controls.[18] In other words, data must be treated as a trusted, ready-to-use product.[9].