Oakland Group

Can you really build a cloud data platform in six weeks?

In short: yes, but no, but yes…sort of…it depends.

For a better answer, let’s start by describing what we mean by a cloud data platform, as it can mean a few different things. Generally, it refers to a series of different components and resources deployed in the cloud to ingest, store, process, and report upon data. The main aim could be for analysis or storage purposes, say to bring data from many different sources to one location. There may be other elements or angles, but this is a usual setup for many.

Back to that six-week deadline.

This can be threatened by a number of different factors, which may demand a longer turnaround time, and have a wider range than just “it takes a while to configure.”

The main reason we see is down to how much of a data platform you are trying to “build” in a six-week period. It is unrealistic to think designing, building, deploying, and embedding a full end-to-end cloud platform in this timeframe is possible. Could you potentially have rotating teams to allow for a 24/7 turnaround, including your stakeholders and a development team? This isn’t for the fainthearted there is a lot of work to complete and factors to consider which are wrapped into achieving your deadline.

Diagram representing three options for building a data platform in six weeks. All options go from Data Source to Ingesting Data, Storing Data, Processing Data, Serving Data and then Reporting. Option one spreads the time evenly across activities. Option two focuses on ingesting multiple sources with minimal processing. Option three focuses on processing and serving data from on source.

Establish what is it you’re trying to build.

Gathering enough information and finalising requirements alone (should!) take weeks or even months to truly ensure the designed solution is suitable. To hit a 6-week deadline, it should be clearly defined what the overall aim is and how you’re going to get there. An end-to-end data platform also regularly has multiple data sources, transformation steps and aspects to complete before handover or go-live is on the cards.

But, if we change the view to focus just on a specific use case, and a narrow element of the data platform development process – then we start to see results in that six-week timeframe. It suddenly becomes a lot more realistic to build elements or the foundations of a cloud data platform under these conditions. When you acknowledge that the solution will not be a complete one but rather a start or focused build then there are definitely big achievements that can be made in shorter time periods.

Keep the scope TIGHT.

The main hurdle is to ensure that both the development team and the stakeholders agree on the defined, tight scope. If requirements have been gathered, it is possible to develop a small proof of concept (PoC) or a minimal viable product (MVP) which would get a solution off the ground and help realise value quickly. This could be one or maybe two of the following examples:

Ideally there needs to be a balance between speed and delivery to meet the deadline. A consideration needs to be made on the longevity of the solution as you should be expecting to deliver work that is long lasting – there is rarely a point in investing in building something that falls apart tomorrow.

These smaller projects should hold their own against the longer, more transformative programmes which help develop capability over a longer period of time. Doing a “drains up” review, redesign, and redevelopment is often necessary but getting buy in, proving value and ensuring the business doesn’t grind to a halt in the meantime means there is a place for quicker “lighthouse projects”.

Diagram describing the differences between Rapid Deployment and Strategic. Rapid Deployment, represented by a lighthouse icon, is used for Lighthouse Projects. You identify "narrow slice" use cases that will engage and excite the business, add value quickly, build momentum and de-risk the change. Strategic, represented by a brick wall icon, is used for Foundation Development. You plan, build, and manage the changes required across people, process, and technology to deliver on the target capability.

Tight deadlines require an experienced, disciplined, but flexible delivery team.

Building a modern data platform brings its own swathe of challenges, especially when up against the clock. The delivery challenges around building cloud data platforms from a project or programme perspective are broad and warrant their own blog post, but it is worth noting some key factors at play here.

The team developing the platform need to be equipped, informed, capable and flexible.

In summary…

Building a cloud data platform in six weeks is a challenging task, and it largely depends on the scope and complexity of the ‘platform’. However, it may not be enough time to build a fully-featured and scalable cloud data platform that meets all of the desired requirements. It requires careful planning, a skilled team, and a well-defined scope. To increase the chances of success, it’s important to have a well-defined project plan, target architecture and a skilled team with relevant expertise in cloud infrastructure, data engineering, and software development.

Assuming that you have these alongside a clear understanding of the requirements and specifications of the cloud data platform, six weeks may actually be enough time to develop a basic version of a platform for continued future development later.

If you’d like to know more about cloud data platforms? Get in touch by emailing hello@theoaklandgroup.co.uk

Jack Evans is a Principal Consultant at Oakland.