There is Something Human about Waste
Let’s start by setting the scene – an all too familiar one. Humans love waste. We throw away around 100 billion pieces of plastic every year in the UK, and around 1.9 billion tonnes of food that can be eaten is discarded on an annual basis, and even on a household level – between 9-16% of energy is wasted on standby.
This fascinating graphic from No Planet B by Mike Berners-Lee shows the world supply chain for food based on energy loss at each stage. As you can see, at every level of the picture, our food waste. The depressing part can be seen in the bottom three rungs; even if we’re lucky enough to have food we still waste around 50% of it.
The impact of this level of overconsumption is evidenced in the world around us: fires in Maui, droughts in Uruguay, and extreme flooding in Bangladesh – all directly attributed to climate change driven largely by Greenhouse gas emissions.
But what does this have to do with data?
It doesn’t come as much of a surprise that physical human behaviour also maps onto digital human behaviour. The attitude towards waste and overconsumption has transcended from the physical to the digital in the case of data – and that has led to some huge inefficiencies in the way that we currently manage data – directly resulting in some pretty stark impacts on the environment. We talk about inefficiency through a few lenses – the first being the underutilisation of resources which we can see represented by the fact that the utilisation rate of on-premises servers is only around 18%, and it doesn’t get much better when we talk about cloud storage and usage, where around 33% of paid-for services are considered waste.
Combine this with the fact that 55% of stored data is considered dark (currently unused, with no future use-case) – a frightening statistic! That means stored and processed data that will never realistically be used for operational or analytical purposes. It’s one of the darker legacies of the big data wave of 2017!
Finally let’s not forget the onslaught of Artificial Intelligence. The amount of data we are creating is out of control.
Data Centres – The Lords of Darkness
When you consider the artificially inflated needs of server racks, the ramifications of this wastage are very real – data centres are being constructed at a rate of knots and are having pretty dire consequences on the environment.
Think about all the utilities that you need to maintain a data centre, primarily you might just think about the energy costs of powering your servers. Ignoring the energy requirements for cooling and lighting, server energy usage on its own has the potential for improvement. Server configuration and distribution of workloads can often drive higher than necessary energy usage.
Due to the sheer amount of energy digital technologies power through, data centres have become the world’s second largest source of greenhouse gases (2.7%) – behind petrochemicals but ahead of the poster boy of climate change, the airline industry (2%).
With no signs of slowing down, the environmental costs of increasing energy consumption will be around 14% of the world’s share by 2030.
All of this has led to data and digital becoming a primary concern for those interested in sustainability initiatives. Most large businesses require decarbonisation initiatives to meet regulatory requirements (e.g. SECR or CSRD). As the issue becomes more and more severe, these regulations are expanding from not only being focused on direct emissions (e.g. owned vehicles, building emissions) but also including scope 3 or supply chain emissions. In the context of data, on-premises data centres can utilise as much as 25% of an organisation’s total energy expenditure, so when you add your cloud usage into the picture – you can see the scope for improvement.
So what?
So we’ve set the scene – but why should businesses care? The way Oakland views this, there are four main drivers of why you should care as a business – boiled down into direct and indirect impacts. If you are considering or are currently involved in any kind of digital transformation, these are the metrics you should consider.
Cost
Direct impacts are associated with cost and regulation. With wastage comes excess fees – you can see enormous waste in resource use across cloud estates, and there is an associated cost with this resource wastage. Waste is the key word here – it’s cost that isn’t being spent on business outcomes.
The main concern with many sustainability initiatives is that they cost a lot of money and negatively impact business processes. However, with data, many of the issues are down to businesses not operating as efficiently as possible, meaning both environmental and financial costs are spiraling for everyone! By tackling these problems, you could potentially have an environmental sustainability initiative that maintains business processes and operations and positively impacts your bottom line.
Regulatory
Regulations are coming in that expect businesses (corporate sustainability reporting directive) to report not only their emissions for scope 2 and 3 (direct emissions and supply chain emissions) but also carbon reduction initiatives. You become more compliant if you can show how you plan to reduce carbon emissions across scopes.
Reputation
Reputation is critical – businesses are now becoming more attractive to prospective employees and customers because of their sustainability credentials. Imagine a data engineer who wants to go and work for two competing organisations. Both roles are bog-standard data engineering, but one of the organisations is committed to its environmental footprint by making its data infrastructure as carbon neutral as possible. The same goes for customers; if you can offer a sustainable option against one that hasn’t even thought about it, there’s a clear winner.
Doing what is right for the planet
Arguably THE most important driver – the long-term ecological disasters that are likely to increase due to the ever-increasing number of data centres and uncontrollable data estates. We mentioned them earlier, but on a more local level, we are seeing huge impacts of data centre development in places like Ireland (18% of energy is used by data centres); and in the US, where new data centres are being constructed in sunny, arid areas to take advantage of PV, we’re seeing reliant water consumption leading to severe droughts.
So really we need to tackle this at the level that we do business and change how we do things.
How to Approach Change
We need to talk about a new way to look at data operations. Can you support the delivery of both cloud and on-premises data estates in the most efficient possible way and have a more muted impact on the environment whilst having a more positive impact on your finances?
Let’s talk about FinOps
There’s been an obsession with FinOps in the past five years – and what that tends to focus on is cloud efficiency, but we saw from the stats before that it’s not necessarily working.
Rates of adoption are pretty low, action is limited to just IT or data teams, and cost becomes once again “a data problem.” When things are just “a data problem,” we don’t see the action which is needed.
So, let’s widen the conversation – How is sustainability baked into your data strategy? – that’s one of the key drivers to create buy-in to data across the business. As we’ve seen the next levels of both regulation and CSR being rolled into key strategic goals for organisations, we can leverage our approach to data in a way that appeals to these initiatives. Cut the waste and your cost can drop whilst meeting your strategic sustainability goals.
The answer? Include Sustainability
So, we approach data with a fourth lens – a lens of sustainability.
This acknowledges there is an issue with how we do business: by handling processes more efficiently we can save energy and carbon emissions, and by driving down energy spend and resource usage, cost savings can be achieved!
By revolutionising how we look at and assess the efficiency of our data estate, as a vehicle to support organisational sustainability initiatives we can drive the value from our data and drive the business outcomes needed from the data itself! This revolution moves us from looking at FinOps, to GreenOps.
Alas this isn’t easy – although we can quite easily say “how much” we spend financially on data, it is difficult to get a full picture of what our energy actually costs.
Oakland and our partners at Interact have devised a four-step process that businesses and data teams can use to start to adopt a GreenOps approach.
There are four main stages: Display, Diagnose, Decide, Deliver.
Display
You can’t take action without first understanding where you currently are – think of this as a baselining stage, but there are some major challenges in this phase – starting with understanding the impact your cloud is having. You need to consider many things, such as replication factors and networking – things that aren’t necessarily covered by existing cloud reporting.
Diagnose
A much more straightforward process – once you have completed your assessment, the next stage is to identify the key areas that are having the biggest impact without reaping benefits. One of the critical things from a cloud perspective could be where you do the majority of your computing, on-premises you want to look at utilisation rates and potential for consolidation to provide short term wins.
An example of this would be when we talk about the statistics of average utilisation of servers; combined with the linear nature of power usage vs. utilisation, you can triple the existing “utilisation” of that server (making it 54% utilised), whilst only using around 50% more energy.
That could result in a 300% increase in productivity for just a 50% increase in energy usage. Think about the carbon impact of this change and the cost savings that can be attached to your energy bills! With the high cost of energy, this is well worth an investigation.
Decide
Take the insights and diagnosis which enable you to start making decisions, reviewing what processes are necessary and if it is possible to change the location of your servers and the times of your heaviest usage.
Consider the wider impact that this has on your business – are there applications or data that you store on-premises that would be better served by migrating to the cloud? Or alternatively, you may even find potential for saving energy and money by moving your “archival” data back to on-premises servers.
The decisions you make here can be broken down into two lenses – short-term and long-term. Short-term decisions are those largely around consolidation and immediate fixes, such as changing the time of an ETL process to when a grid is being run on more renewable energy. To make these kinds of impacts, you don’t always have to fundamentally change what you’re doing in the short term – in fact, with Oakland’s carbon efficiency tool, we have seen that by shifting your heavy compute to one day rather than another has the potential to drop associated carbon emissions by around 10% – which costs nothing!
Your longer-term view of things might be migrating on-premises data to the cloud, but to do this most efficiently you need to understand the potential environmental impact – but the most important thing is to create a plan.
You want to focus on the areas you see the most potential to support your business and data strategies. If you’re going for a big push on governance, perhaps you need to initially consider where all that pesky dark data is sat and create a plan to decommission it, if you’ve got some horribly inefficient applications that aren’t optimised for the cloud – then create a plan to refactor them.
Deliver
Migrate, consolidate, and even delete! You need to deliver to the plan. When it comes to the delivery of these larger scale migration projects, you need to consider business continuity – how are you going to maintain services whilst doing this migration? One way to approach this is through introducing new tooling – for example, our partners at Starburst provide a distributed analytics engine that enables you to continually query from both on-premises and cloud instances to maintain services during a migration process.
What challenges might you face?
A lack of transparency from cloud providers on the carbon impact their services are making (for example, when we talk about replication factors, you need to consider that AWS lambda instances need to be replicated six times across different servers – meaning your carbon impact is six-fold).
A lack of publicly available data to support on-premises data assessments. Without support and data, it will take a long time to understand the existing impact that your large on-prem data centres are having.
We face a number of challenges as an industry, and to take on a GreenOps approach we need to be equipped with the tools and capability to assess and benchmark where we are, from an on-premises perspective and a consolidated view across both cloud and on-premises.
What are the next steps?
To support businesses in taking their next steps towards GreenOps, Oakland and Interact have partnered to create a revolutionary service based on the four-step method we’ve talked about here. Interact are experts in on-premises data centre efficacy assessments and how you can save money and energy by consolidating and reconfiguring servers.
Oakland can support organisations in reporting on, understanding, and investigating their cloud estate using our carbon efficiency tool – an enhancement from the well-regarded Thoughtworks cloud carbon footprint tool to create an estimate of the carbon footprint of your data estate.
We work together to identify how you can streamline your data estate and optimise for both carbon and cost perspectives. It’s important to note that the platform itself is only a part of the battle; you need governance to ensure that retention policies are being fulfilled, which can have a huge impact on your dark data, which needs to be baked into your data strategy as a means of alignment, and from an analytics perspective, you have to start to report on data sustainability as a KPI to support carbon reduction initiatives.
As an industry, we need to do better for ourselves and the planet by working together to minimise the environmental impact of our data in a cost-efficient way.
If you’d like more information about Oakland’s revolutionary GreenOps service please get in touch by emailing Luke.sharma@theoaklandgroup.co.uk