When building a data platform for your business, you need to anticipate and plan for any potential challenges you may encounter. That’s where a data engineering specialist can help.
With decades of data experience behind us, we’ve seen and dealt with practically every problem you might encounter during data platform development. In this guide, we’ll cover common difficulties you might face when developing your data platform, with advice on how to handle each to make your data platform journey as smooth as possible.
To understand more about the complexities of a data platform and the process of building one, check out our data platform guide.
What is a Data Platform?
Put simply, a data platform is a central hub that collects, organises, transforms and applies your data. You can learn more about data platforms through our expert data platform guide.
What Challenges Can Impact Building a Data Platform?
It would be lovely if everything could be easy, wouldn’t it? But unfortunately, many of the best things in life come with a few struggles along the way to attain them. Luckily, our data engineering experts can help you navigate the following common challenges.
Challenge 1: Excessively Tech-Centric Focus
It’s easy to think of your data platform initiative as a technical project; after all, you’ll soon be designing, launching and modifying an expensive chunk of technology real estate.
But think back to the failed data management initiatives you’ve observed in past organisations – what did they have in common? Chances are, they got bogged down in the tech aspect of building a data platform at the expense of the business strategy.
Technical teams often prefer to solve technical problems rather than get involved in the messy business of persuading people with different objectives to collaborate.
The big risk in being overly tech-focused is that if your data platform does not meet user needs (such as data availability and usability), users won’t use it. You will achieve minimal adoption, and the platform will fail to deliver tangible business outcomes.
Therefore, your data platform aspirations should form the ‘pointy end’ of a data strategy – it’s where the rubber hits your digital transformation roadmap.
Whenever you feel the narrative swinging too far over to the tech, bring it back with questions such as:
- What does this tech mean to our business model and value proposition?
- How will the technical direction impact our strategic goals?
- How will the tech impact the customer experience?
Without this, you run the risk of low adoption, low involvement from business users, and ultimately, low value delivered, if any.
Challenge 2: Departmental Data Silos
In an attempt to solve a tactical or near-term challenge, departments or cross-business functions can often be swayed by a solution vendor’s shiny offering. The department then commissions a localised solution that seemingly fits their needs but doesn’t take stock of the wider data strategy or business needs.
Other teams then struggle to extract this new data, particularly if the department has used off-the-shelf solutions.
The result is an ever-increasing technical burden that becomes difficult to unravel and migrate in the future.
To prevent this from happening, it’s crucial to implement solutions in a consultative and collaborative manner. What do you need from your solution? Is there one out there that can benefit a greater range of stakeholders?
Challenge 3: Poor-Quality Data
One of the most common issues we have to deal with is poor-quality data. This is important since your data platform will only deliver your business goals if the data ingested is high enough in quality otherwise your platform could sink without a trace.
But what is data quality? Data quality can be measured by the following six principles, which you should always keep in the back of your mind when building a data platform:
- Accuracy: Does your data correctly represent the intended entities and events? Are your data sources reliable?
- Consistency: Is your data uniform across systems and data sets?
- Validity: Does your data conform to predefined rules regarding data structure and values?
- Completeness: Have you included all the expected data values and types?
- Timeliness: Is your data current and available to use when needed?
- Uniqueness: Have you ensured no duplicate records within a single data set? Can your data record be uniquely identified?
To aid you in understanding how your data measures up to these principles, Oakland offers data quality assessments with advice on improvement as part of our data governance service.
Why is data quality important? Poor quality data leads to inaccurate analytics, which, in turn, leads to bad business strategies. This can result in missed sales opportunities, decreased customer satisfaction, significant monetary fines over incorrect reporting and a lack of trust from your corporate executives and business managers. See our blog on why you should invest in data quality to see the benefits it can provide for your business.
Challenge 4: Lack of Data Governance
According to Gartner, “Data governance is the specification of decision rights and an accountability framework to ensure the appropriate behavior in the valuation, creation, consumption and control of data and analytics.”
The demand for data governance originally emerged from the shift toward more robust regulatory controls in the banking and insurance sectors.
Today, data governance is pervasive across all industries, and now you can even buy data governance platforms off the shelf! Yet, many data platform initiatives stutter or fail when Data Governance is immature or lacks key components suited to Data Platform strategy and management.
The impact of poor data governance can include:
- Business and technical users lose trust in the data
- Designing and maintaining the platform takes far longer
- Frustrating ‘Turf wars’ over data ownership erupt or remain unresolved
To find out more about data governance and how you can enable it properly when designing a data platform, check out our guide: How to Launch a Data Governance Initiative by Stealth.
Challenge 5: Failing to consider the complexity and cost implications of the legacy data landscape
Your data platform is not an island; it needs careful integration with existing systems and processes.
At Oakland, we’ve been around a long time and one of the recurring trends we’ve seen is the case of the ‘over-optimistic’ target vendor.
Despite the glossy marketing blurb, no data platform is a true plug-and-play solution, so avoid anything that sounds too simple to be true. The last thing you want is to get locked into a vendor, meaning that you are forced to keep using a low-quality platform because switching would be impractical.
Many aspects of vendor lock-in are inevitable, and using a vendor’s portfolio of cloud-native services increases lock-in. However, access to integrated services and increased discounts can be a plus. To prevent lock-in (e.g. open source software solutions), make decisions on a case-by-case basis and vet the total cost of ownership (TCO) of solutions intended.
Creating a new data platform in any enterprise requires a careful analysis of what approaches have gone before and now require direct integration with your new data architecture.
We’ve parachuted into several data platform recoveries in which the complexity of integration was overlooked and soon became the mother of all obstacles to going live.
In short, don’t overlook the essential brownfield discovery tasks that some vendors like to gloss over to get you over the finishing line.
Challenge 6: Not prioritising requirements
As you get deeper into data platform delivery, stakeholders asking, “So, what are we building again?” can become a common challenge.
The problem is the modern data platform strategy can support a range of use cases, including:
- BI (Business Intelligence) /MI (Management Information) capabilities
- Self-service reporting
- Integration
- Data catalogue
- Customer Data Platform, e.g. single voice of the customer, employee, product, service or asset
- Executive or regulatory reporting
- Advanced analytics and AI
- Real-time analytics
- Assessment management
- Adjunct to a core system.
It’s easy for your data platform to lack clarity and prioritisation around its core function, especially as different groups begin to see it as a data ‘dumping ground’. The practice of “let’s keep it in case we need it” can lead to a bloated data platform, further complicating the task of extracting insights from your data.
To prevent a toxic swamp of data, we prefer to phase the delivery of a data platform with a regular cycle of ‘Lighthouse Projects’ that solve burning issues within the business but still align to an overarching data strategy and architecture with a clear transition to an enterprise solution.
Start small, think big, and act fast.
You get to demonstrate the benefits of each release, garnering support as you deliver each successful project, and helping justify further initiatives’ investment.
Challenge 7: Creating the case for change
Creating a compelling business case for a modern data platform can be challenging for many organisations, particularly when faced with a legacy of delivery struggles.
Deploying the next generation of data platforms has multiple benefits and use cases that will appeal to various leadership sponsors.
We’ve found that the key is smaller, faster pilot projects that deliver rapid and sustained gains without over-investment and risk while building data capabilities at the same time. You can find the right use cases for your pilot project with Oakland’s complementary use case workshop, where we’ll explore what the business wants to know and where insights are needed.
In short, focusing on delivering the right data at the right time to support specific business outcomes will quickly gain support for the future of your data platform.
Build your perfect data platform with Oakland and explore our other services to take your data even further. You can also contact us with all your burning data platform questions. Want to learn more about what we do? Take a peek at our blog or give our podcast a listen.
