‘Big Data,’ Analytics and Statistical Process Control (SPC)
“Poor little old SPC – with ‘big data’ and advanced analytics, we don’t need it anymore.” WRONG, VERY WRONG!
Improving product quality by reducing process variation has always relied on data. However, SPC whilst being a long-established and powerful tool was developed over 70 years ago in a very different environment. Since then business has exploded with specialised technology and analytics which support data analysis, however, integrating these tools is much more difficult than everyone was led to believe. How can we unlock the next wave of improvements promised by SPC, making the most of the new whilst remembering the learnings of the past?
We’ve gone from problems in the 20th century of infrequent data to lots of very frequent data, in fact, we’re generating terabytes of the stuff. How should we effectively manage real-time applications to collect, clean and provide useable data, and then analyse it correctly to make better decisions to improve our product performance?
We answer these questions and provide guidance on the application of SPC and using tools for analysing big data in this article recently published in Quasar https://www.therqa.com/resources/quasar/ by the Research Quality Association https://www.therqa.com and available to you know as a download.
Whether we are digitalised or not, we are harnessed to the fact that we must understand the processes we operate and the nature of their variation, in terms of:
- accuracy versus precision;
- special and random causes of variation;
- capability and control; and
- performance versus improvement.
the questions we need answers to are:
- how does big data relate to SPC?
- can there arise a conflict between big data and SPC?
- how can SPC be modified (modernised?) and used with big data to strengthen the analysis?
In the full article, you will be able to see how, when designing and using any large complex system involving data streams, machine learning, robotics, AI, data collection/aggregation, data analytics and algorithms, it is important to measure its performance, in terms of the system’s ‘correctness’ – how often it gives the right answer, the right decision/instruction. This dimension of system performance may be usefully addressed using the ‘confusion matrix’ and its related measures.
The adoption of big data and machine learning, robotics, AI and the internet of things (IoT) is greatly impacting industry and business models. The amount of data is constantly growing but learning from more and better data depends massively on the data being collected and used correctly if we are not going to see increases in variation in processes through lack of understanding of aggregated or filtered data with the right algorithms. That is why the application of modified SPC approaches, methods and tools is still an essential part of making this transformation in the world of Big Data.
Read the full article here
