Data Engineering and storage costs are spiraling (increasing by 30% in the past year!) (US and European cloud prices set to soar in 2023 – Techzine Europe), so data leaders are naturally looking at ways to maintain the velocity of delivery, whilst also bucking up their data governance efforts and keeping an eye on ever-increasing budgets. Enter Data Products! The concept of data products is spreading through the world of data like wildfire..
By definition, “a data product is a reusable data asset, built to deliver a trusted dataset, for a specific purpose”. Or in the simplest terms, data products help get the right data in the right package to the right people. That doesn’t sound too revolutionary, though, after all, we have had “self-serve” datasets for many years now. The uniqueness of data products come when you adopt a federated ownership model – no longer is there a centralised data team that deals with all requests from all corners of the business. Now, there is a product team per “domain” – a logical grouping of data that pertains to a single use case, similar to a subject area. These teams become responsible for the whole value chain of data within their domains, from production all the way through to consumption and sharing.
Domain driven ownership
Figure 1 A Domain-Oriented Ownership model
Source: https://www.starburst.io/blog/data-mesh-and-starburst-domain-oriented-ownership-architecture/
Why would I move to Data Products?
A shift to a data product approach to data delivery can help with a range of challenges, including:
1. Untrustworthy data
Because of the ownership model, there is now accountability for maintaining the quality of the data presented.
2. Siloed/Undiscoverable data
Data Products should contain usable managed data, enabling people to self-serve autonomously — getting the data they need when needed rather than waiting for a central team of engineers working through an already unmanageable backlog.
3. Security Risks
Optimising for risk and compliance ensure you are applying the appropriate control levels for the situation. Your domain teams are responsible for managing and the right levels of access and sharing policies.
Cards on the table, the data product approach isn’t necessarily appropriate for less mature data estates, as it relies on highly mature processes, skills and infrastructure. Despite this, by introducing this approach as a long-term strategic goal, you can start to augment your existing data practices whilst laying positive foundations for data product delivery (e.g. documenting data assets consistently).
Incorporating a data product approach into your data strategy has people, process and technology implications – so the following tips are designed to provide you with some ideas of steps you might want to take if you are considering taking the plunge:
“Sounds good, where do I start?”
We mentioned earlier the concept of domain driven ownership. Given the nature of data products being focused on how each domain can solve issues for the business, your first port of call is deciding how to divide the organisation up into domains.
A good starting point for this is to investigate the existing subject areas that sit within your data estate – most organisations will have these. Once you have a picture of what your domains are, you can start to look at the process of building a data product.
If your data strategy is led from IT or Data, this is where you start to find friends in the domain areas. Once you have identified a domain that is open to the data product approach, you can work with them to define the data products that would support the business needs from that domain.
Defining a Data Product
You don’t see people on Dragon’s Den pitching a product that doesn’t solve a specific problem. In fact, the most successful companies will think about the problem first, and work to build a solution to that problem. Data Products are no different, and building a strategy for these data products should align with that view.
You will want to build products that solve specific problems and challenges within the business, which can be marketed and maintained. Get to know your consumers, do your research, understand the pains they feel when engaging with your data. Establish processes that enable consumers to contact and feedback to data product teams so that every data product is built towards either toward solving a specific problem or responding to market demand quickly by creating new business opportunities.
You can then set out a prioritisation exercise to understand where to focus efforts to deliver your first data products!
Deliver Value Rapidly and Iterate
As with any new process within an organisation, at Oakland – we find that delivering value rapidly is critical. We can do this by taking a lighthouse approach, where you take a narrow but deep slice of your data estate and design a small, but valuable data product.
With this lighthouse project, you set up the rituals and roles of product management to ensure that they fit within your existing operating model. Once the lighthouse project has been delivered, the processes and roles that supported its delivery should be assessed – as well as the value gained from the product. You can then embed the learnings into your Target Operating Model, ensuring the product management approach is embedded into the organisation’s ways of working.
The key here is that the approach is tested and it drives value for your organisation with more moderate investment. This lighthouse data product and way of working can then be demonstrated to other domains, increasing appetite and excitement!
Breaking the Chains of a Project Mindset
This is where the strategy element of “data product strategy” becomes vital because it involves cultural and process change. Once you have delivered a data product, you can rapidly see it fall back into the traditional cycle of standard data development because the organisation’s mentality hasn’t moved forward with the technology.
Data is unfortunately often seen as an output of a project, be it a report, dashboard, or ML algorithm. The project is delivered, signed off by stakeholders and then handed over to the business unit. Inevitably the deliverables from these projects are outdated pretty quickly – and the response? Set up another project, build another report – rinse and repeat.
You need to stop thinking of data as an output and start to think about data as a product.
Products need to be maintained, updated, and associated with SLAs/data contracts that guarantee a certain level of uptime and data quality as part of a standard product lifecycle, by the product management team. Taking the approach that you learn from in the lighthouse project and embedding it into a TOM helps to ensure that the accountability for data product maintenance is prioritised.
The whole aim of data products is they are built and maintained in order to be reusable, shareable assets!
Value, Value, Value!
It’s been mentioned a few times, but data products need to be focused on delivering specific outcomes for a specific use-case. To build out the roadmap of your data strategy, you need to think about which use-cases will be tackled and in which order.
How do we know which products will provide the most value? By engaging with consumers of course! The engagement with business users is absolutely critical to the success of a data product approach.
Through engaging with consumers, you should get a view of the shape of data products that can drive value to the organisation, aligned with business goals. This forms the backbone of your data strategy – regular workshops, clinics and meetings between product management teams and consumers. They will guide the development requirements of data products and encourage wider adoption.
In summary, using data products to drive your strategy will enable you to drive widespread adoption and accountability for data within your organisation. There are some layers of complexity from a technological standpoint, but they can be overcome with data product enabling tools such as Starburst. The complexity of people and processes can be mitigated by having regular touchpoints with consumers who can feed back into the development and prioritisation of data products.
If you’d like to learn more about a data product approach then drop us a line at hello@theoaklandgroup.co.uk
Luke Sharma is a consultant at Oakland