Driving growth for Elvie with a data platform

“We just thought that Oakland’s presentation was one of the most organised. We liked the comprehensive plans that we saw.”

The Goal

Elvie asked for Oakland’s help to build a data analytics platform that would extract, store, and prepare data for reporting and insights.

Historically, Elvie had struggled to report on various KPIs across the business easily. Existing reports were often time-consuming to prepare, with data manually collated from siloed sources.

The platform would therefore need to extract data from multiple sources on a scheduled (likely daily) basis. Elvie needed to store the data in a structured database designed to be enterprise-ready as a template for any future business logic and data modelling that Elvie would take forward.

Initial transformations in the platform would need to cleanse and check the quality of the data, producing trusted insights via the custom Tableau reporting interface.

To drive better management decisions, access to organised data would provide richer insights into hard facts on profit and loss, stock, sales, and other operational priorities.

“Previously, people were downloading and transforming data in different ways across the business. And because of that, people on different teams were ending up with different numbers.”

Tailoring the solution

At Oakland, we design every data platform around the organisation’s unique business challenges and data landscape.

With so many options available, Elvie sought guidance on which components and approaches were right for them. The key considerations to balance were cost, complexity, scalability and extensibility. The Elvie team were looking for a solution built to last with foundations set ready to expand in the future.

An example architecture was proposed as part of the initial discussions, outlining a Google Cloud Platform (GCP) solution with various orchestration and integration options for the data sources.

From a pricing and complexity perspective, some options were more favourable than others, so more consideration was required before a final approach could be agreed upon.

Oakland and Elvie decided to launch a rapid proof-of-concept (PoC) platform that would demonstrate and assure the functionality of the options available.

Certain parts of the solution, such as the cloud platform, could be clarified as part of conversations, but how data would be extracted and transformed was an essential consideration that the PoC platform could address.

The final components selected can be seen in the diagrams below:

High-Level Components

“Oakland was confident enough to advise us in the right direction, while still being flexible to what we as a business actually needed.”

Delivering a ‘ready-to-operate’ platform

One of our key goals was creating a solution that Elvie could readily adopt and maintain. There would be no benefit to the organisation if the data platform remained unused or developed issues the Elvie team could not address.

As with most of Oakland’s projects, we aimed to develop the solution in technical collaboration with Elvie, bringing the Elvie Business Intelligence (BI) team up to speed as it progressed and involving those on the ground in each design decision.

This collaborative approach all the way through to handover ensured Elvie was equipped with Oakland’s experience and best practices in configuring, maintaining and utilising their new platform.

Ultimately, Oakland crafted a platform suited to a smaller team new to maintaining data platforms. Some of the more complex initial concepts may have provided additional benefits, but this becomes a challenge as they become increasingly difficult to support.

The result was a data platform supported by detailed documentation handed over to an Elvie BI team that was skilled and confident in developing and maintaining the data platform themselves.

“Oakland gave us a total package solution, which we needed because of our very small BI team.”

The Data Platform Solution
_{(Technical Components)}

Elvie’s data platform was developed using a Tech Stack with the following components:

Data Extraction – FiveTran

Oakland tested several extraction solutions, but FiveTran’s simplicity and ease at configuring new data sources made it an excellent solution for Elvie’s small, ‘low-code’ BI team.

The tool came pre-packaged with connectors for almost every data source in Elvie’s landscape. Its MAR pricing model also enabled Elvie to keep costs to a minimum.

Cloud Platform – GCP

Elvie had already adopted GCP. With no need for features specific to other cloud platforms, it made sense to build the platform using GCP as a foundation.

Data Storage – BigQuery

Although Snowflake was considered a viable solution, we selected BigQuery as the Data Warehouse of choice. Elvie was already rooted in GCP, and selecting BigQuery provided easier integration with other native GCP components. Again, the serverless and cost-effective nature of BigQuery made it ideal for Elvie’s BI team.

Data Transformation – DBT

DBT was chosen as the transformation tool for several reasons.

Firstly, the dbt models could be written in SQL, making it ideal for Elvie’s BI team, whose main area of technical expertise was SQL.

Secondly, Elvie’s relatively low data volumes and complexity meant heavy-duty tools like Spark would have led to an over-engineered solution.

Finally, adopting dbt meant Elvie could easily write SQL-based transformation scripts and leverage the power of BigQuery for maximum performance.

Data Visualisation – Tableau

Tableau was already in use as the visualisation tool of choice at Elvie.

To get the maximum benefit from the new data platform, the Data Warehouse was designed around three layers – raw, staging and reporting.

The reporting layer contained heavily curated pre-processed datasets resulting from several transformations as the data moved through the raw and staging layers. This design meant that any heavy processing was executed in BigQuery, leaving Tableau to focus on what it does best – visualisation.

Data Orchestration – Apache Airflow

There were a wide variety of data sources at Elvie, each updated periodically at different intervals.

To effectively manage and monitor the tasks to extract and transform Elvie’s data, we selected Apache Airflow.

Airflow was the perfect tool of choice as its industry ubiquity meant all use cases were supported, with the advantage of an extensive online knowledge base to help the small BI team.

It was also easy to get Airflow up and running through GCP’s managed service, Cloud Composer – another big win for Elvie.

Infrastructure as Code – Terraform

Finally, we used Terraform to provision the infrastructure required for Elvie’s data platform.

Because we had to tear down and re-create several environments throughout the project, Terraform became a vital tool. Having all infrastructure configured in a Terraform repo made environment changes easy to repeat.

When Oakland delivered the final platform to Elvie, the entire configuration history was available in a GitHub repo. Without this, Elvie may have wondered how Oakland had built the data platform in the first place.

“The fact that Oakland set up our foundations for us is huge because it just meant that we can now actually move forward. I don’t think we could have really done everything without Oakland.”