Augmented Intelligence
Apr 18, 2016

Analytic rigor is the key to operationalizing machine learning

It is an exciting time for the field of machine learning, an area of research that involves finding latent patterns in data. It is making striking advances in an array of fields, including speech translation, web search, social media, clinical diagnosis, manufacturing, driverless cars, customer marketing, fraud detection, and, more broadly, all business operations.

Machine learning can help transform industries by enabling the reimagination of processes end to end, and generating impact from these new processes in ways not previously possible. A foundational step of pattern search in machine learning can be done using data as diverse as text, images, audio recordings, smartphone sensor readings, industrial sensor output, and social graphs. Moreover, it can combine this data and new forms that emerge.

Pattern detection itself, as enabled by the algorithms that power machine learning, operate in stark contrast to traditional computer programming and by extension robotic automation, where patterns had to first be described for a system to be useful. A loan application is a good example as it was traditionally accepted on a set of predefined generic rules and steps versus advanced data-driven decisions made by weighting data specific to an individual at the time of application.

To build increasingly advanced systems, we are presented with opportunities and challenges: how do we conduct the research design with the requisite analytic rigor that allows us to capitalize from machine learning's advanced pattern-detection capabilities to solve business challenges?

Tools are not enough
In light of these new capabilities, the key to unlocking value is the ability to think of solutions to problems that were previously not solvable. The essential bridge to leveraging machine learning becomes the coupling of advanced pattern capability with research design.

Bringing a machine learning system online requires different experiments for each new application. Conducting such experiments requires analytic rigor, which is the lynchpin to success, particularly as machine learning systems must be trained with data and in context. In addition, analytic rigor fosters stability of solutions and is therefore highly advantageous for both the deployment and continued operation of high-impact machine learning applications in the enterprise.

It's indeed very exciting that application programming interfaces (APIs) have greatly facilitated access to raw pattern detection capabilities. For example, how many people can tell the difference between a Pembroke Welsh corgi and the Cardigan Welsh corgi? [1] The analytic rigor required to solve business challenges, however, cannot be so cleanly encapsulated.

Let me provide an example, one that is almost a legend in the machine learning community. It presents the multiple types of challenges that appear when attempting to operationalize a fundamentally new form of pattern detection.

  • In a military application, researchers leveraged an advanced machine learning approach, called a neural network, to detect whether images did or did not have tanks in them. This dichotomy is common in classification challenges, where a machine learning system is trained by being presented with examples of distinct classes of labeled data. The system is then free during training to extract and learn the most useful patterns to make future classifications. For this challenge, the system worked superbly in testing environments, but failed when the military attempted to operationalize it. It turned out that the photos of the tanks had been taken on cloudy days while the photos without tanks had been taken on sunny days. As a result, the neural network had learned to distinguish the two sets of images essentially by the color of the sky. [2]

Here, the eventual failure was non-obvious as even experts can make errors in analytic rigor and research design, such as not carefully selecting training data to enable problem solving in a general way, for example, accounting for the primary yet spurious signal in the data or using data that has been properly sampled or gathered with sufficient diversity. This underscores the need for a range of analytics professionals to be involved in the operationalization of advanced capabilities with machine learning, particularly given its dependence on data and the need for science to anchor discovery during solution development.

Analysts, data engineers, data scientists, developers, and domain specialists must work together to ensure that machine learning systems come online to help solve problems we couldn't before.

Access is only part of the puzzle
To make real impact across industries, machine learning must be leveraged for specific domains. Resolving domain-specific nuances in operational environments can be a challenge, as with the most advanced methods you don't automatically get insight as to why a classification is made a certain way, although improving this is an active area of research.[3]  In addition to the different levels of opaqueness from machine learning methods and the critical step of data selection, other key challenges remain that are also not directly related to domain of algorithms: data quality, noise, bias, model development, and human intuition failing in high dimensions.[4]

Therefore, although advances have been made to make machine learning algorithms more accessible, this is not to be confused with making them easier to use, as research design with analytic rigor is still required to set up problems to be solved, and ensure the results are interpretable to drive meaningful action. New impact in every industry depends on this discipline to capture opportunities leveraging previously unknown patterns in data, whether attempting to understand customer preferences in a consumer goods setting, accelerating drug discovery, boosting efficiency in a supply chain, managing risk in a portfolio, or detecting system failure in a manufacturing plant.

Beyond the hype
The exuberance surrounding machine learning is similar to that experienced by big data several years ago. [5] Big data has been hyped so much that it has been seen as a cure for everything, but it has been poorly operationalized and few organizations today have truly advanced big data analytics capabilities.

Machine learning presents a bigger challenge, as leveraging advanced methods at scale (e.g. deep learning [6]) is also a big data challenge, which presents the obstacles of data management and processing in addition to algorithmic understanding and testing.

It's still early days for machine learning to deliver impact, yet I remain optimistic that the democratization of education about methods, greater data access, continued algorithmic advances, and an increased availability of computing power will accelerate companies' success with disciplined solutions and application development.

In closing, I offer the following statement as a beacon to practitioners, researchers, domain specialists, leaders, and those engaged in this global transformation for productivity and discovery:

  • We have reached a time in our civilization where we have begun to teach machines to learn. The implications are profound, as machines can learn things previously unknown to us. We must ensure that this learning happens responsibly, as meaningful impact for industry and society depends upon it.

  1. Project Adam: Building an Efficient and Scalable Deep Learning Training System
  2. What Artificial Experts Can and Cannot do
  3. Opening the black box: low-dimensional dynamics in high-dimensional recurrent neural networks
  4. A Few Useful Things to Know about Machine Learning
  5. Gartner 2015 Hype Cycle: Big Data is Out, Machine Learning is in
  6. Computer science: The learning machines. Using massive amounts of data to recognize photos and speech, deep-learning computers are taking a big step towards true artificial intelligence

About the author

Armen Kherlopian

Armen Kherlopian

Ph.D., Chief Science Officer & VP, Analytics & Research

Follow Dr. Armen Kherlopian on LinkedIn