Decentralized architecture
Introduction
Data mesh is a sociotechnical method for building a decentralized data architecture by leveraging a self-service domain-oriented design (in a software development perspective), and borrows the theory of domain-driven design from Eric Evans[1] and the theory of team topologies from Manuel Pais and Matthew Skelton.[2] Data mesh is all about the data itself, taking the data lake and pipelines as a concern. secondary.[3] The primary proposition is to scale analytics data through domain-oriented decentralization.[4] With data mesh, responsibility for analytics data is transferred from the central data team to domain teams, supported by a data platform team that provides a domain-independent data platform.[5] This helps better organize data and avoids having separate isolated areas of data. It is due to the presence of a central system that makes sure that everyone follows the same basic rules within the data network, allowing data to be shared across different locations.
History
The term data mesh was first defined by Zhamak Dehghani in 2019[6] while working as the principal consultant at technology company Thoughtworks.[7][8] Dehghani introduced the term in 2019 and then provided more details on the principles and logical architecture throughout 2020. The process was predicted to be a "big contender" for companies in 2022.[9][10] Some of the companies that have implemented data meshes are Zalando,[11] Netflix,[12] Intuit,[13] VistaPrint, JPMorgan Chase,[14] PayPal[15] and others.
In 2022, Dehghani left Thoughtworks to found Nextdata Technologies and focus on decentralized data.[16].
Beginning
The data mesh is based on four fundamental principles:[5].
In addition to these principles, Dehghani writes that data products created by each domain team must be discoverable, addressable, trustworthy, possess self-describing semantics and syntax, be interoperable, secure, and governed by global standards and access controls.[18] In other words, data must be treated as a trusted, ready-to-use product.[9].
Community
Scott Hirleman has started a data networking community in his Slack channel "Slack (software)") that contains more than 7,500 people.[19].
References
- [1] ↑ Evans, Eric (2004). Domain-driven design : tackling complexity in the heart of software. Boston: Addison-Wesley. ISBN 0-321-12521-5. OCLC 52134890.: https://www.worldcat.org/oclc/52134890
- [2] ↑ Skelton, Matthew (2019). Team topologies : organizing business and technology teams for fast flow. Manuel Pais. Portland, OR. ISBN 978-1-942788-84-3. OCLC 1108538721.: https://www.worldcat.org/oclc/1108538721
- [3] ↑ Machado, Inês Araújo; Costa, Carlos; Santos, Maribel Yasmina (1 de enero de 2022). «Data Mesh: Concepts and Principles of a Paradigm Shift in Data Architectures». Procedia Computer Science. International Conference on ENTERprise Information Systems / ProjMAN - International Conference on Project MANagement / HCist - International Conference on Health and Social Care Information Systems and Technologies 2021 (en inglés) 196: 263-271. ISSN 1877-0509. doi:10.1016/j.procs.2021.12.013.: https://es.wikipedia.org//portal.issn.org/resource/issn/1877-0509
- [4] ↑ «Data Mesh Architecture». datamesh-architecture.com (en inglés). Consultado el 13 de junio de 2022.: https://datamesh-architecture.com/
- [5] ↑ a b Dehghani, Zhamak (2022). Data Mesh. Sebastopol, CA. ISBN 978-1-4920-9236-0. OCLC 1260236796.: https://www.worldcat.org/oclc/1260236796
- [6] ↑ «How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh». martinfowler.com. Consultado el 28 de enero de 2022.: https://martinfowler.com/articles/data-monolith-to-mesh.html