Big Data Comes of Age
Written by J. Walker Smith, Executive Chairman, The Futures Company
The biggest winner in the US Presidential election was not the incumbent, Barack Obama. It was New York Times’ political blogger, Nate Silver. His success tells us something important about the way the world is heading in marketing as well as politics.
Silver established his quantitative bona fides with a Moneyball-style model for predicting the performance (thus, the value) of Major League Baseball players before turning to politics. Silver first made his mark there when his blog, FiveThirtyEight (now at The New York Times) accurately called 49 of 50 states in the 2008 Presidential election, along with all 35 Senate races. In this year’s fiercely contested Presidential contest, Silver had a clean sweep – 50 of 50 states plus the District of Columbia. As a result, his star is rising.
So how does he do this? He is not a pundit or pollster, nor former politician or campaign operative. He is a statistician – a quant.
Silver is a poll aggregator who uses the polls of other firms as a data stream that he analyzes and models. His methodology is highly developed, but it boils down to a handful of things. First, he looks at all available polls, except those with patently flawed methodologies. Second, he weights these polls on factors affecting their accuracy. Third, he fits regression curves and trend lines that bring the various polls together. Finally, he runs simulations on key parameters in his models. From all of this, he calculates several results, including two crucial ones – a predicted outcome and the odds of that prediction coming true.
Pundits versus quants
Throughout the 2012 political season, Silver posted daily updates, along with detailed blog posts explaining and comparing his results. By Silver’s reckoning, President Obama never had less than a 59% chance of winning reelection, peaking just above 90% the day before the vote. There were shifts along the way, but Silver always had the President ahead, which made him a target for a legion of patronizing pundits.
Politico’s Dylan Byers said Silver could be “a one-term celebrity.” MSNBC’s “Morning Joe” Scarborough called Silver an “ideologue” and a “joke,” prompting Silver to bet him $2,000. New York Times columnist David Brooks took Silver to task for presuming that polls could “understand the future.”
But pundit after pundit called the election wrong. Why? Because of that gut-feel that pundits get paid for. In her Monday pre-election column, Peggy Noonan made light of “the weighting of the polls,” asserting instead that “the American people were quietly cooking something up,” namely a Romney victory. With tea leaves like these, it’s no wonder that Philip Tetlock’s much-referenced 20-year study of expert predictions found them to be less accurate than just flipping a coin.
Of course, it’s not as if every quant got it right. In a post-election Slate scorecard, the third worst prediction came from a poll aggregator. But only one other forecast matched Silver’s and that was another poll aggregator. Like pundits, quants can get it wrong. But when done properly, quants beat pundits every time.
There is a long history to real-world, real-time modeling of elections; the use of social science methods in reporting, known as precision journalism, goes back to the 1960s. But with Nate Silver, quants have suddenly come of age, and not just because of his Twitter-era notoriety in a closely scrutinized election but because the cultural trajectory of quants parallels that of Big Data. All of a sudden, the world itself is one made to order for quants, a world that, indeed, only they can parse.
The coming data deluge
Like Noah and The Flood, magazine covers have been foretelling the coming data deluge. “Getting Control of Big Data” resounded from the cover of the October 2012 Harvard Business Review above a drawing of a lion tamer reeling backwards, hat flying. In 2010, The Economist published a special section on “The Data Deluge and How to Handle It” with cover art of a businessman funneling the downpour of data to water a blooming flower.
This mind-boggling reality of exabytes, zettabytes and, soon to come, yottabytes, is well beyond the grasp of our intuition. An Economist video reports that the quantity of global data is forecast to be an staggering 7,910 exabytes by 2015, over 60 times greater than 2005. Twitter alone generates over 230 million tweets each day, equivalent to 46 megabits of data per second. In this future, says The Economist, people will live in a world of sensors and software in which their “every move is instantly digitized and added to the flood of public data.” This is where quants come to the rescue. In a world flooded with data, good, solid quantitative analytics will be table stakes for success. This is obvious already in marketing.
For example, one of the big new developments in marketing is dynamic pricing, or pricing that varies in real-time to reflect on-going shifts in buying patterns, competitive pricing, and contextual factors. The stream of data now available makes it possible for marketers to identify previously unrecognized pricing inefficiencies, or situations in which consumers are willing to pay more and others in which consumers would buy if the price were lower.
As a Wall Street Journal story reported, online prices for everything from white goods to children’s clothing to consumer electronics to shoes to jewelry to detergents to razor blades are in constant flux. One online pricing software company, Mercent Corp., says it changes the prices of two million items every hour. The era of ‘the programmable store’ comes ever closer.
The redefinition of expertise
Nate Silver’s success has put quants on everyone’s radar. A Forbes news analysis the day after the election noted three ways Silver has made a lasting impact in politics, ways that are also true more broadly.
First, quants are now a permanent part of the scene, and they focus on numbers not on presumptions and rules-of-thumb. Their job is to get the prediction right, and that means pet theories take a back seat to hypothesis-testing and empirical results.
Second, quantitative analysis removes much of the mystery that has shrouded elections, marketing and other futures. What once plagued decision-makers as uncertainty and volatility can now be understood as ignorance of underlying patterns and regularities, not some intractable characteristic of the marketplace or the world at large.
Finally, expertise is being redefined. Processing, analyzing and modeling Big Data demands a much higher level of quantitative analytics than ever before. Knowing how to handle Big Data will be essential to proficiency and credibility. Pundits will no longer be excused for innumeracy. Better knowledge of statistical concepts will be mandatory.
Gurus and gut-feel won’t cut it in a world of Big Data, not in politics and especially not in marketing. Marketers are managing brands in a world of complexity well beyond the imagination of the 1960s advertising doyens depicted in Mad Men. Big Data requires quants who can wrestle it to the ground. Nate Silver showed this to the world on one of its biggest stages, and now the world is demanding more of the show.