MIT’s recently published report concludes by stating that in 2025, ‘95% of generative AI pilots at companies are failing’ – and it doesn’t surprise me one bit. Thousands of AI proof of concepts are being made under the guise of massive easy cost savings and robot magic simply finding all the answers – but many are doomed to fail. We’ve seen the amount of ‘gotchas!’ and obstacles first-hand and it’s all too easy to fall into traps and end up with something which is just quite ‘meh’.
So, how do you make sure your generative AI project actually brings value and still exists 12 months later? I’ve written out some learnings from our experience scoping, building and deploying AI solutions at Oakland Everything Data. (I then ran this blog through CoPilot to make sure it read okay, but I decided to ignore a lot of the suggestions. I secretly worry all content on the internet will sound the same in a few years, but Nicola, our Marketing Manager, assures me it won’t…)
What is Generative AI?
Generative AI is an artificial intelligence that creates original, new content. Think text, images, videos, music. It learns from huge datasets of information that already exist, spotting patterns and structures in the data to then produce novel outputs. When using a GenAI model, users give it a prompt. It then creates the original content off the back of that – ChatGPT is the most well-known example and one you’ve probably already used.
Solve the Right Problem with the Right Tool
So, how should businesses choose the right use-case to solve with AI?
Firstly, think about what you’re trying to do and consider if you even need AI. Hype is powerful and budgets often get approved on buzzwords, like GenAI. But, spending money on a tool which will make a ‘pretty interesting’ case study and nothing else won’t lead to a successful, money saving solution.
At Oakland, we often run workshops with clients to discuss where and how they could use AI in their business. The shortlist usually ends up at around 10 or 20 project candidates. More often than not, 80% of these problems are just general or more advanced analytics problems, with some golden AI problems buried amongst them. This isn’t an issue as they could be brilliant candidates in their own right – maybe even solvable using some element of AI to enhance the output. But getting the right problem to solve is important.
“Choose a challenge which will have a high impact on something lower risk. Avoid regulated areas, critical workflows, or things which require more exact output. Look for problems which are time consuming, automatable, and areas where AI can add to existing human processes, rather than fully replacing them.”
Jack Evans, Principal Consultant
Making work easier so individuals can achieve more or be more impactive is the key, as is choosing an existing process to improve rather than creating something completely new.
AI Solutions can be More than Chatbots
There’s nothing wrong with a good chatbot. In fact, they’re often an ideal way for a user to interact with an AI model. The problem is that people often think this is the best and only way to use AI within an organisation, because it’s the most visible format for LLMs. But they’re not the only way!
AI can come in many different forms. The best solutions use the best parts of different tech to solve a problem and have a bigger impact. Examples include:
- Using AI alongside machine learning, intelligent forecasting, or even Power BI reports to summarise findings
- AI automatically generating or updating documents
- AI acting as the ‘glue’ between processes, triggering automation workflows between systems
(Agentic systems are the dream to bring real value and change through multi-step autonomous actions rather than just Q&A, but these can get complicated quickly. So let’s take things slow!)
Just thinking that the only way to surface AI is through a chatbot is limiting. Working through a problem and exploring the different outputs and user interaction are important to building something brilliant.
Appropriately Scale the Generative AI Solution
So, you have a decent user case in mind and you want to start developing an AI solution. I feel like this wouldn’t be a good blog if I only called out the need to “plan a lot” (I feel like that’s obvious for any IT project) – but the main thing is to plan with some specific considerations in the AI space.
Plan for future phases, with a high-level roadmap of where to begin (smaller aspirations) to what comes next (bigger aspirations). This will help from a cost management and output standpoint. It’s also a great way to highlight the need for iterative development and the need for potential commitment as you do with any proof of concept IT project.
A PoC will not (and should not) solve all problems or immediately become a production solution without extra steps. Start with small, accessible, non-sensitive, and good quality data sources to minimise potential obstacles – otherwise, you could trip at many hurdles along the way.
“Remember that Generative AI works best on firm data foundations. It’s vital to make sure that you’re confident in both the quality of the data it’s based on, as well as the types of insights you want.”
Jack Evans, Principal Consultant
Technical Architecture Considerations
Tooling in this space is evolving fast. Very fast. You don’t want to find out the method you’ve chosen is out of date or not suitable before you’ve even finished the development. Seek advice from vendors on the right approach, keep an eye on roadmaps and research as best as you can to make things scalable and extensible.
Many pilots do prove the concept, but fail to take the project through to the production stage. This is important, because getting a solution through to production means it has more structure and controls, whilst also having more chance of being adopted.
Governance, technical design authorities, and processes are not always ready to handle AI solutions. Plan for this potential lag and guide people through the ins and outs of the project – whether this be explaining potential concerns around security, data privacy, or Skynet scenarios, or just walking through the selected tech architecture.
Measure GenAI Project Success Accordingly (and be Realistic)
With the pressure to justify budget requests, aspirations can often overreach – especially when discussing new technology and the promises that follow. AI has a reputation for being a way to find the answer to many corporate problems, but this is not always the case! Asking a chatbot to “find and fix all the problems” is unlikely to yield the best results out of the box.
Success measures are, however, important to track progress. Without these guides (as with any iterative process), it’s hard to know when the aims have even been met and when to stop.
Success is often defined as producing relevant benefits. Yet these can be difficult to quantify with AI and regularly aren’t directly financial. For instance:
- A greater understanding of information
- New, smoother, more efficient processes
- Improved transparency to data
If you choose particular use cases, it can be easier to align to financial measures – but that requires a specific type of problem you’re solving. On top of this, it’s a good idea not to promise the world. Keep any targets within the realms of possibility (e.g. 10% increased throughput of tickets).
Tailor, Build, and Test GenAI – A Lot
Building an AI solution requires more than just adding data to an LLM and asking it a question. Don’t get me wrong, you would likely get an answer, but it might not be the level of detail you’d like. The “out of the box” experience for a lot of LLMs takes time and configuration to improve and develop. Prompt engineering is very much a necessary commitment to make sure the AI model has everything it needs to answer the best it can. Allocate time to this activity and test, iterate and then test some more.
As an example, setting how much speculation an AI model should be able to do and clarifying terminology will help tailor a response.
- Generating legal documents? Dial down the free will and force the model to reference real facts.
- Summarising historical spend? A bit more flexibility could be given to provide more options on how the user can query the AI model.
This work helps the user feel like the answer was written for them with the context of how they wanted the response. If they wanted a general answer, they would’ve used ChatGPT or CoPilot.
GenAI Integration and Business Change
As with any IT project, developing the solution is only half the problem – getting people to use it in the way it was intended is a whole other challenge. Proof of concepts regularly have less time allocated to business change, but this makes sense given the smaller scope. The issue is often around the user case being selected, because if this is right, then it really helps with adoption later on.
“If you’re building a generative AI solution, you should aim to look at existing processes to augment and improve with AI. Developing a whole new process can often add complications, as it’s easier to jump onto the existing structure and setup that users are familiar with.”
Jack Evans, Principal Consultant
The other consideration is getting users to adopt the fancy new AI solution which has been developed. Chatbots, as an example, often rely on user input to thrive and if no suggestions are given, then a user can sometimes have no idea what to ask. This results in the blinking cursor of doom facing a user, with a powerhouse of wonder and capability sitting within a model that someone doesn’t know how to interact with.
Bake suggestions and prompts into the tool, plan a process to push insights to users wherever possible, and help guide users to find what they want easier. People will always need upskilling, but the more heavy lifting the solution can do in this space, the better.
Impactful Generative AI Projects
With any luck, thinking about some of the above will increase the chance that your AI solution will exist six months after development starts. Patience is already wearing thin and we’re seeing an increased push of ROI reflections creeping into the thoughts of budget holders.
“Just build some AI” needs more consideration and, unfortunately, doesn’t hold the same weight for immediate £££ signoff and guaranteed efficiencies once built. If you’re not careful, it will simply be added to the pile of novelty solutions that didn’t actually save any money or improve any processes. Or worse, it was everything everyone asked for but no one knows it exists or how to use it!For support with scoping out a meaningful generative AI user-case for your organisation, please speak to us.
