Data Mesh: concepts and principles of a paradigm shift in data architectures

Abstract

Inherent to the growing use of the most varied forms of software (e.g., social applications), there is the creation and storage of data that, due to its characteristics (volume, variety, and velocity), make the concept of Big Data emerge. Big Data Warehouses and Data Lakes are concepts already well established and implemented by several organizations, to serve their decision-making needs. After analyzing the various problems demonstrated by those monolithic architectures, it is possible to conclude about the need for a paradigm shift that will make organizations truly data-oriented. In this new paradigm, data is seen as the main concern of the organization, and the pipelining tools and the Data Lake itself are seen as a secondary concern. Thus, the Data Mesh consists in the implementation of an architecture where data is intentionally distributed among several Mesh nodes, in such a way that there is no chaos or data silos, since there are centralized governance strategies and the guarantee that the core principles are shared throughout the Mesh nodes. This paper presents the motivation for the appearance of the Data Mesh paradigm, its features, and approaches for its implementation.