Technology is not an entity that stands still, this is evident from the myriad of new releases, both small and large, that can be seen from companies like AWS and Azure, who continuously push the capabilities of what they can offer. However, this constant barrage of incremental gains can seem overwhelming and often will not provide huge improvements to organisations looking to change not what they do but how they do it. As such, for this scale of impact, companies look to broader, more platform orientated releases that can encompass whole elements of their strategies or operations, such as improving data quality or improved use of companywide analytics. This wider scale transformation will likely become even more evident as we enter the new year, with companies either trying to hold onto productivity or recoup losses as the pandemic lessens its grip.
With this in mind, Oakland has looked at some of the key players in the technology landscape. Assessing some of the new and interesting releases that will likely find their way into general use in 2021.
Microsoft Azure – Purview
Almost every organisation has a goal relating to data in some form, moving databases to the cloud, improving reporting, or developing advanced analytics capabilities. However, to do these things properly, you need a baseline to start from, understanding what data you have, where it exists, what it is used for, and its lineage. Without this, you run a considerable risk of disrupting what currently works and introducing new complications. This is where Purview, a new Azure service specifically designed for data cataloguing and governance, can offer some support. Purview has been designed to improve Azures data cataloguing capabilities by introducing both standard and more bespoke methods for data cataloguing. In terms of standard features, Purview can scan both cloud and on-premise systems to map what data is available, how it moves from source system to use, and what transformations occur. These standard scans can then be supplemented with more bespoke data cataloguing through additional interfaces, giving a fuller view. Once complete, users can utilise a web interface to see where data originates, what actions are performed, and how it is used. With this new data baseline, companies are much better placed to transform elements of their data usage and the associated governance, knowing that they can always view where the data came from. https://azure.microsoft.com/en-us/services/purview/
Amazon Web Services (AWS)– Amazon Monitron
Principles such as Total Productive Maintenance have been utilised by manufacturing companies since the last century. However, these more established processes often can be difficult to integrate into the more digital, real-time age properly, with machine readings commonly stored in databases and analysed as and when. There are clear areas for improvement, especially when considering the increase in manufacturing processes’ complexity relating to everyday technology advancements. The reasoning behind the lack of adoption is understandable, as it requires teams of highly specialised people to move and analyse the data. As such, AWS has looked to radically simplify this process by developing its Monitron service, which does exactly what you’d expect it to, it monitors. Using a series of IoT sensors attached physically to machinery and wirelessly to a routing hub, data is collected on machinery in real-time and fed into AWS for analysis. This analysis uses customised algorithms to detect potential faults in various machine types, which are then fed back to users through a web platform or mobile app. This product’s key feature is the simplicity it brings, allowing a company with an AWS account, an internet service, and a mobile phone to understand their machine efficiency in real-time, without the outlay of multiple technical staff. Because of this, it’s likely that the service fully rolled out in December 2020 will be further adopted by companies, especially after positive customer feedback from organisations such as General Electric and RS Components. https://aws.amazon.com/monitron/
Databricks – SQL Analytics
Databrick’s have most recently been trying to solve a common problem associated with numerous organisations. How do we reduce the complexity associated with data lakes and their multiple ETL jobs and data warehouses? Their solultion is what they term the Data Lakehouse architecture, a core component of which being the Delta Lake system they have engineered. By utilising Delta Lake, an open-source, standardised technology, organisations can house and transform data in the same place whilst still adhering to required standards on governance and stability. However, this architecture’s new key element is blending fast, reporting-based analytics from this layer, which is where SQL analytics comes in. This new set of features provides a dedicated SQL workspace for data analysts, a range of connectors for hooking data up to reporting tools and visualisation software, and optimised query performance through a new Spark engine built purely for rapid SQL queries. Through the combination of Delta Lake and SQL analytics, Databricks users will be able to rapidly access stable data via standard queries, delivering a single version of the truth regarding data reporting. Due to the increase in companies utilising the Databricks platform and the ubiquitous need for reporting, we see this as a technology that could well be on the rise through 2021.
In summary, there are some interesting new technologies available for 2021, with major players pushing at least one large release. However, while the solutions are available, there are still the ever-present hard yards to really understand the source of the issues and how best to integrate these innovations to offer the best outcome. The skillsets to implement, maintain, and use this advancing technology will continue to be in demand as organisations realise that the technology alone can’t deliver the results they require. https://databricks.com/product/sql-analytics
Richard Louden is a Senior Engineer at The Oakland Group
