A UK Public Service Broadcaster

Siloed Data

Typically in these situations, an organisation might build a centralised data store in the form of a data warehouse, data lake, or core data analytics platform. But these solutions can soon morph into silos – largely driven by the data capabilities being built and having lost sight of the true challenge of what it means for the organisation to be data-centric or insight-driven.

Every organisation desires easy access to its most vital information, but data is commonly spread across disparate silos according to business function.

Technology grasped this challenge to the point of cleaned and structured data models either in a central data warehouse or in enriched data marts. Whilst these technology solutions and practices can reduce complexity and simplify ways of working at a departmental or functional level, they become a headache when generating insights and solutions that demand a unified view of data across the wider business, and data consumers are still facing challenges such as:

1. A common business definition of performance indicators and data to evaluate them consistently.
2. An easy way to share between data consumers in a controlled and secure way.
3. Time to have data ready to be consumed: With new data sources constantly integrated through partnerships, capacity is reaching its limit to ensure all relevant data are ready to be consumed through widely available solutions such as PowerBI.

From Conceptual Model to Curated Data Products Layer (and Beyond)

Oakland started with an enterprise-wide conceptual model to help our client develop its goal of a unified data strategy aligned with the business strategy. Oakland adopted a top-down approach with business interviews to capture the needs for metrics, analysis, and how the KPIs are connected.

The model became a vehicle for understanding the essential data domains and entities required to answer the most critical questions posed by the business, along with a foundation for data ownership and governance.

But most importantly, the conceptual model provided the foundation for a curated Data Products layer for designing and sharing analytics and reporting.

With a base curated data products layer, our client could start thinking about cross-enterprise ‘data products’ and other analytical solutions that would tackle pressing strategic and operational challenges.

By collapsing data silos into a unified data architecture, our client could also find ways for disparate teams of business and data specialists to collaborate and establish a true data culture and community.

Proving the Concept

Once our client recognised the strategic value of the conceptual data model and unified data architecture, an Oakland Proof of Concept (PoC) was approved to build the business-curated data products and all relevant technologies.

There were several goals for the initial PoC:

1. From needs to access: Create a use case demonstrating a proven reduction in time and effort for analysts looking to produce insights and answers.
2. Data Product aligned with business needs: Demonstrate how the conceptual model would lead to a single, unifying query methodology across the business.
3. Quality and Lineage: Improve data quality and provide an accurate lineage of insights and KPIs.
4. Data exploration: Unlock new opportunities for exciting new business capabilities, e.g. adopting and extending data products that truly manage data as an asset.
5. Data sharing: Ensure secure and appropriate access to data in a controlled environment.
6. Reduction of shadow data: Enable data products to include any data within the organisation. With connectors and federated queries, the data products can include highly curated data from the data warehouse or any other data sources, including APIs, removing barriers to physical access.

The final goal was to validate the suitability of Starburst and its operation with other technical components.

Introducing Starburst

The Starburst product was the main ‘workhorse’ of the PoC based on its suitability for data product design and data analytics provision.

Starburst delivered a single data access point across the enterprise, giving our client the perfect platform to test several analytical and operational use cases with a unified query method and language (SQL).

Where traditional SQL engines can struggle when querying scattered data sources or complex data lakes, Starburst employs a massively parallel analytical engine to break down complicated queries and return rapid results.

Unlike more traditional data architectures, which rely on costly and complex data pipelines that feed into central storage areas (e.g. data warehouses), Starburst instead adopts a ‘data mesh’ style of distributed data architecture. This allowed our client to experiment with an entirely new data analytics capability, moving away from a pipeline and centralised approach to a truly federated model.

Traditionally, the business would need to wait for engineering to link up disparate data via pipelines and central storage before they could ask questions about the data. With the curated data layer approach, our client is developing a more agile and creative discovery method.

Outcome of the Data Platform Accelerator PoC

Data Products are foundational to changing how data producers, experts, and consumers collaborate.

We found Starburst to be highly effective at delivering a unified query environment that supports our client’s vision of a modern, data product-focused data architecture. Leveraging existing data assets based on in-memory query enables quick implementation of data products to drive how business consumers use data for a purpose.

Starburst adapted well to the needs of multiple user groups (e.g. admin, business users, SQL data analysts, etc.), with plenty of connectors ‘under the hood’ for seamless access to data stores required for this PoC and future operational scenarios.

We discovered that by incorporating our approach of a conceptual model and curated data products layer, enabled through a modern data mesh solution such as Starburst, our client could accelerate its data strategy and solve real business challenges around data access, exploration, iteration, re-use and management of data as an asset through Data Products, without the trouble of complex migration or organisational changes.