Toward Fast and Exact Machine Learning Platform for Big Data

摘要

Data has become the foundation of knowledge, and many companies are growing interested in harnessing AI-based data analysis to unlock its value. The volume of digital data is increasing at an unprecedented pace: market research reports estimate that global data volume, approximately 12.5 zettabytes in 2014, will reach around 180 zettabytes by 2025. Extracting patterns and trends from such big data is crucial for enabling data-driven decision-making. However, a key challenge lies in the enormous computational costs required for large-scale analysis, due to the inherent complexities of the task. Approximate methods are often employed to reduce these costs, but they inevitably trade exactness for efficiency. To overcome this limitation, our research aims to develop a machine learning platform that delivers both speed and accuracy. The core of our platform is computational pruning. This talk will introduce three representative pruning strategies. Specifically, it first introduces a pruning method that uses upper and lower bounds to omit computations. This method efficiently identifies unnecessary processes by using upper and lower bounds of scores to skip unnecessary computations. Next, this talk introduces a method to terminate computations that cannot yield solutions. This method maintains patterns that failed during the search process to avoid repeated futile searches. This talk finally introduces a method that prunes computations through optimistic processing. This method temporarily removes a constraint to find a solution quickly and then verifies if the obtained solution meets the constraint. These strategies can open the path to data analysis techniques that are both efficient and exact, ultimately empowering companies to make more reliable and timely decisions in today's increasingly data-driven world.