Just over a month on from the crowds and excitement at Big Data London (BDL). One of the key themes, which even had its own theatre was Data Mesh, and most of the two days were spent debating Data Mesh’s merits. While the first and second Data Mesh principles of “Data as a Product” and “Decentralised Data Domains” stole the limelight in London, there was little debate about the fourth, and arguably the most important, principle, that of Federated Data Governance.
It is arguably the most important and least understood Data Mesh principle. This is odd, given that it will be critical to Data Mesh success in particular, but will fundamentally change how Data Governance is delivered going forward. Why do I believe this to be the case?
Let’s consider the theory and how data governance is delivered in most organisations through the usual lens of People, Process, and Technology.
The Theory
Data Mesh assumes that the overarching Data Governance policies, procedures, and standards will be agreed upon and written at the Federated level. This is the level above the Data Domains themselves so effectively by a central function or team. These Data Governance artifacts will then be deployed into a Data Mesh solution through largely automated mechanisms to ensure that the data being used by data domain users is accessible, available, complete, accurate, and fit for purpose.
The Reality
This sounds great at a conceptual level, but when you sit that alongside what is being done in many organisations to implement Data Governance at the moment, it is a million miles away from how businesses are typically trying to deploy their data governance framework.
If we believe that Data Mesh will become the new data paradigm, then those doing Data Governance will need to change their approach and behaviours not just slightly but materially. Let’s consider this in terms of People, Process and Technology.
People
In the event of Data Governance being delivered through more automated mechanisms, it is evident that the skills needed will move into a more technically led space. Currently, many Data Governance practitioners have fallen into the discipline because they are good with numbers, have loads of common sense, and can talk to stakeholders. Successful Data Governance professionals in the future will need not just these skills but equivalent technical skills – well beyond the doing a bit of SQL that most of us have seen in the past.
We are already starting to see this shift in recruiting Data Governance professionals who need strong technical and non-technical skills.
Process
A Data Governance framework is currently designed and deployed through a blend of Website, Intranet, Training & Education type mechanisms. In Data Mesh, policies, standards, and procedures will be far more automated, so the processes underpinning good Data Governance will have to change. In the same way, those doing Data Governance will need to have a good grounding in designing and building these automated processes.
Technology
As the Technology underpinning Data Mesh can be expected to use a heavy recycling or realignment of existing technologies – with the odd addition of faster query engines such as Starburst – this is likely to have the least impact on current Data Governance best practice. So long as Data Governance practitioners understand the value and implications of their respective Data Governance technical solutions, this area is least likely to evolve.
So is Data Governance at an evolutionary tipping point? We are now starting to see changes in how data governance is delivered to organisations. Adoption of Data Mesh can and will escalate that evolution. It remains to be seen who will be the winners and who will be the losers.
Andrew Sharp is a Senior Data Governance Consultant at The Oakland Group