In short: yes, but no, but yes…sort of…it depends.
For a better answer, let’s start by describing what we mean by a cloud data platform, as it can mean a few different things. Generally, it refers to a series of different components and resources deployed in the cloud to ingest, store, process, and report upon data. The main aim could be for analysis or storage purposes, say to bring data from many different sources to one location. There may be other elements or angles, but this is a usual setup for many.
Back to that six-week deadline.
This can be threatened by a number of different factors, which may demand a longer turnaround time, and have a wider range than just “it takes a while to configure.”
The main reason we see is down to how much of a data platform you are trying to “build” in a six-week period. It is unrealistic to think designing, building, deploying, and embedding a full end-to-end cloud platform in this timeframe is possible. Could you potentially have rotating teams to allow for a 24/7 turnaround, including your stakeholders and a development team? This isn’t for the fainthearted there is a lot of work to complete and factors to consider which are wrapped into achieving your deadline.
Establish what is it you’re trying to build.
Gathering enough information and finalising requirements alone (should!) take weeks or even months to truly ensure the designed solution is suitable. To hit a 6-week deadline, it should be clearly defined what the overall aim is and how you’re going to get there. An end-to-end data platform also regularly has multiple data sources, transformation steps and aspects to complete before handover or go-live is on the cards.
But, if we change the view to focus just on a specific use case, and a narrow element of the data platform development process – then we start to see results in that six-week timeframe. It suddenly becomes a lot more realistic to build elements or the foundations of a cloud data platform under these conditions. When you acknowledge that the solution will not be a complete one but rather a start or focused build then there are definitely big achievements that can be made in shorter time periods.
Keep the scope TIGHT.
The main hurdle is to ensure that both the development team and the stakeholders agree on the defined, tight scope. If requirements have been gathered, it is possible to develop a small proof of concept (PoC) or a minimal viable product (MVP) which would get a solution off the ground and help realise value quickly. This could be one or maybe two of the following examples:
- Deploying a series of cloud services which are part of a wider architecture
- Setting up an initial data source connection and outputting data in a cloud database
- Developing transformation steps for pre-existing data sets
- Creating a simple data product
Ideally there needs to be a balance between speed and delivery to meet the deadline. A consideration needs to be made on the longevity of the solution as you should be expecting to deliver work that is long lasting – there is rarely a point in investing in building something that falls apart tomorrow.
These smaller projects should hold their own against the longer, more transformative programmes which help develop capability over a longer period of time. Doing a “drains up” review, redesign, and redevelopment is often necessary but getting buy in, proving value and ensuring the business doesn’t grind to a halt in the meantime means there is a place for quicker “lighthouse projects”.
Tight deadlines require an experienced, disciplined, but flexible delivery team.
Building a modern data platform brings its own swathe of challenges, especially when up against the clock. The delivery challenges around building cloud data platforms from a project or programme perspective are broad and warrant their own blog post, but it is worth noting some key factors at play here.
The team developing the platform need to be equipped, informed, capable and flexible.
- Equipped: Developers need have the required access to any necessary source systems, cloud platforms or infrastructure. Fast builds require rapid access and little work often gets done when waiting for weeks for support tickets to be fulfilled. Ensure the developers have what they need to do their job.
- Informed: This can take many forms, but often this is both being aware and comfortable with the delivery plan/target architecture but also having access to stakeholders to ask important questions which arise. Governance and decision making are important factors to stop “cowboy development” and deliver the right solution but waiting for emails for weeks throttles delivery and that 6-week deadline. It’s also rare that “too much documentation” is ever provided ahead of delivery – so this alongside efficient communication is often key.
- Capable: The delivery team need to be technically competent when it comes to building the platform but also when it comes to managing it. Data platform implementations draw on a range of talents from developers, architects and cloud engineers to ensure the solution is built efficiently to specification but – let’s not forget the need for experienced project/programme managers and business analysts. These are vital to clarifying the specification and handling a plethora of unforeseen changes and risks which could threaten that 6-week deadline.
- Flexible: Plans change and every project has surprises – it’s the way they’re handled which decides the success of the project. The delivery team need to work in a tightly managed agile approach, adapting and overcoming whatever changes or unforseen factors jump in the way. Responding to stakeholder feedback and managing expectations fall into this category to deal with what can happen with the solution as it develops. Working with the cloud helps a lot given the very adaptive nature of building with technology – but this is always within reason as any change takes time.
In summary…
Building a cloud data platform in six weeks is a challenging task, and it largely depends on the scope and complexity of the ‘platform’. However, it may not be enough time to build a fully-featured and scalable cloud data platform that meets all of the desired requirements. It requires careful planning, a skilled team, and a well-defined scope. To increase the chances of success, it’s important to have a well-defined project plan, target architecture and a skilled team with relevant expertise in cloud infrastructure, data engineering, and software development.
Assuming that you have these alongside a clear understanding of the requirements and specifications of the cloud data platform, six weeks may actually be enough time to develop a basic version of a platform for continued future development later.
If you’d like to know more about cloud data platforms? Get in touch by emailing hello@theoaklandgroup.co.uk
Jack Evans is a Principal Consultant at Oakland.