Analytics & Big Data
Oct 26, 2016

No, let's not blame the algorithm!

In the first blog, we discussed the growth and usefulness of algorithms. This time we'll ask: While there's great debate about humans vs. algorithms, are they really pitted against each other?

There are two different schools of thought here: One school believes that algorithms will take over, the other that humans and machines need to co-exist in order to improve lives.

But in my opinion machines or algorithms are ultimately for the betterment of our lives—they are created by us for our own benefit. Humans will make the machines and algorithms smarter day by day, in order to drive productivity and make intelligent decisions—just please do bear in mind that algorithms need data. Wherever data is scarce or absent, human judgment comes into play. And there is always going to be an insufficiency of data in certain scenarios, which lends itself to human decision making. That imputation logic then becomes learning for the machines. And therefore there is a case to be made that while algorithms are powerful, they cannot completely take over.

There is also an argument that algorithms cannot do a great job of personalization, especially for industries that thrive on customization and consumer targeting. The debate here is how much of a human element is required in such cases, that is, whether it can be completely left to algorithms.

For example, let's compare the approaches of Apple Music1 and Spotify2. On the one hand, Spotify's "Discover Weekly" uses algorithms to understand the favorite artists of users through analysis of profiles and song selection. From there, by comparing users who have ostensibly similar tastes, Spotify is able to recommend new songs to many users. Some industry experts observe that the experience has been good so far.

Apple, on the other hand, insists that machines cannot understand human emotions, nor can they overcome this inability through algorithms. That's why Apple engages popular DJs and artists to curate specifically themed music in a way that algorithms never could. While this emphasizes the need for human curation of certain jobs, does it also mean that we can safely say human intervention is better than using algorithms alone? Certainly, we will all agree, to find solutions for particular business situations using algorithms, there would always be multiple methods. An algorithm created by a person or team will be influenced by the choice of the methodology that is preferred by that individual or team. So can algorithms escape human biases? And is there a way to create a non-influenced algorithm?

Let's consider a case from predictive analytics, which is as much an art as it is a science. Netflix uses an algorithm for recommending movies to its users. It threw open a challenge – Netflix prize3 — to develop a better algorithms than the one it currently uses. Netflix shared a data set showing how nearly half a million people had rated approximately 18,000 movies. The goal was to use these ratings as the basis for predictive analytics capable of rating other, as-yet-unrated, movies with a higher degree of accuracy than Netflix current algorithm.

Top prize? $1 million.

Two teams worked separately on this. The first team created a complex algorithm that worked using the given data set. The second team created a simple algorithm, but added more data points from the Internet Movie Database (IMDB)—and wound up walking away with the grand prize. The point here is that adding more, independent data can even beat out better-designed algorithms that are working off of an existing data set.

To sum up, while algorithms obviously can be intimidating for humans, it's also important to note that their effectiveness is dubious when there is a lack of adequate, relevant data and human intervention. And while the input humans bring sometimes might be marginal–that input might make a huge difference towards overall effectiveness.

There is also the general sentiment that increasing our reliance on machines and algorithms is going to put the universe under tremendous security threat. Some talk about cyber security—the story about the Google Car being hacked—and others think that it can create a moral dilemma. In my considered view, analyzing big data sets to forecast trends is a joint effort of humans and algorithms. Algorithms can rapidly establish patterns, whereas humans can set parameters and analyze the results. Both have their strengths.

As technology advances, there will be mistakes and failures, but let's not forget that tools are just created for our comfort and convenience. We shouldn't transfer blame or compete with our tools—or our algorithms.

Let's be pragmatic and not fear the algorithm!


About the author

Sudhanshu Singh

Sudhanshu Singh

Senior Vice President & Global Practice Leader, Analytics & Research

Follow Sudhanshu Singh on LinkedIn