Last week I attended the Spark + AI Summit Europe 2019 in Amsterdam, whilst a few days in Amsterdam raised mentions of ‘jolly’ and ‘mini-breaks’ with the team this couldn’t have been further from the truth. With over 2,000 avid Spark users from all over Europe gathered together to get the latest updates and detailed information about Artificial Intelligence, Data Science and Data Engineering.
And WOW what an event, filled with exceptional people at the bleeding edge of tech. This leads me to think I am caught between 2 worlds. Those who understand the tech and are madly enthusiastic, utilising data at the cutting edge, and attending this event looking for the next big thing. And everyone else! I am somewhere in the middle. An engineer by trade but very business focused in my view of the world and building a fast growing Leeds based analytics business. I am now 3 years into a campaign to help businesses understand how to get value from this technology (and 3 years spent realising how much there is to learn).
Returning from the summit I am reinvigorated to build bridges or if you are to build a bridge you need to understand the start and endpoint.
So what were the key points I took away?
It’s no surprise how fast things are moving, what is a pleasant surprise is that instead of the usual hoo-ha around shiny product launches there was much more focus around adding business value. Databricks is making data science even more accessible by unifying data science, engineering and business. There were exciting announcements about open-sourcing, Delta and the launch of the latest version of MLflow which will help to build the maturity of the tech and add the value too often missing to business. It was notable the announcements were related to both infrastructure and organising the work from a business point of view.
There seem to be fewer people questioning the move to the cloud, and who would believe that Hadoop is already dead. And it is interesting the Delta Lake is about organising the data and combining the best features of data warehouses and data lakes and MLflow helps orchestrate moving from model developments and POC ‘s into production. These tools might not get the average man in the street excited but they are important advancements in both tech and business.
For the masses, the excitement of tech conferences such as these is often difficult to translate to the day to day with a new silver bullet being fired at each one. Even if you take a passing interest in the tech, keeping up is bewildering. When we hear large corporations talking about their latest advances in AI these are still isolated deployments and still yet to have an impact at the core of most businesses. There is a profound need for the data science team to work with and influence management at every level, as when the two work together is when huge strides forward can be made.
What does good look like?
From the event, the best use cases were where these two worlds have come together often doing basic stuff in a combined way.
For example US pet retailer PetSmart and their clever tech automatically classifying customer complaints to help make the customer experience better or the sales forecast modelling from Electrolux combining historic data with opinion in a structured way to benefit the customer.
These use cases studies highlight good science is much more than good data scientists. It’s how you structure your team to do great stuff, build better relationships and keep delivering for your customers in a world where there is no such thing as loyalty.
The success of the profession will be driven by how well real-world business cases can be made, where we can build repeatable solutions (approaches for how to tackle common problems so not reinventing the wheel and where we can combine these approaches with genuine insight into the business operation.