In a break from reviewing ‘self-help’ books on investing and market histories, this month we look at a New York Times bestseller which investigates the use of big data, regressions and randomisations to solve puzzles.

If you’ve ever watched Who wants to be a millionaire? you may have noticed that ‘Ask the Audience’ option has a better record of getting the right answer than ‘Phone a Friend’. In fact, asking the audience produces the right answer more than 90% of the time while phoning a friend produces the right answer less than two thirds of the time.

The reason for this is that the average guess or estimate of a group of people is more likely to be closer to the right answer than any individual estimate, as the author James Surowiecki showed in his 2004 book The Wisdom of Crowds.

In Super Crunchers, author and Yale law professor Ian Ayers takes this idea of collective wisdom further and dives into the world of data crunching to show how statistical analysis is displacing traditional experts and how this is affecting our everyday lives.

Super crunching isn’t just changing the way that decisions are made, it’s changing the decisions themselves.

DINNER IS ON THE HOUSE AND LOVE IS IN THE AIR

For every customer who goes to a Harrah’s casino and picks up a ‘Total Rewards’ swipeable card, Harrah’s knows on a hand-by-hand or slot-by-slot basis how much each customer is winning or losing in real real-time.

It combines this data with other information like the customer’s age and the average income in the area where they live to predict how much that customer can lose and still enjoy the experience enough to come back.

It calls this amount the ‘pain point’. If a customer gets close to that point, a manager from Harrah’s approaches with commiserations and the offer of a free steak dinner for them and their partner in the company’s steak-house. The pain of losing gives way to a pleasurable experience, and Harrah’s data shows that this is often enough to encourage the customer to return.

One of the most active super crunchers is eHarmony which uses statistics rather than peoples’ conscious and articulated preferences to find a compatible match.

At the heart of its match-making process is a regression model which takes raw historical data and estimates how various factors influence one single question or variable. In eHarmony’s case the single variable is how compatible a couple is likely to be.

The raw historical data is comprised of decades of information on married people to see what types of personalities are actually happy together rather than how long people who just think they are compatible stay together.

The firm keeps its model and its data to itself but its ability to predict a match is remarkable. So much so that it claims that 70% of men and women will meet their spouse on eHarmony within a year.

WHAT’S IN YOUR WALLET?

US credit card provider Capital One, which used the phrase above in its marketing several years ago, combines regression analysis with randomisation to drive its business.

For more than a decade, Capital One has been running thousands of experiments every year for new products, new advertising approaches and new contract terms.

It regularly sends different groups of prospective customers different offers – for example a low introductory rate or a limited-time offer – to see which has the highest success rate.

Because each group is over 50,000 people, at which point the distribution of each group is almost identical, the difference in their responses can only be down to the offer they receive.

Sometimes price isn’t everything. In one experiment, adding a photo of a smiling woman to the corner of an offer letter raised the response rate of male customers by the same amount as dropping the interest rate by nearly 5%.

GETTING IN ON THE GAME

Unsurprisingly, US companies are well ahead of their UK and European counterparts when it comes to super crunching.

Two of the few UK companies to really mine their customer data are Tesco (TSCO) and ASOS (ASC:AIM) .

Thanks to its Clubcard scheme, which was set up over 20 years ago, Tesco knows the shopping habits of 19m households or 60% of the UK population.

Together with the data from its mobile telephone and banking services, Tesco knows who are its most valuable customers and how to keep them spending.

ASOS is successfully using ‘machine learning’ – essentially computers crunching masses of data, running regressions and randomisations – to predict which products and styles its customers will like.

Companies which rely on consumers for their business need to get ahead and start crunching data now or risk losing out to rivals who better understand the value of data and are better at forecasting what customers are likely to want in the future.

‹ Previous2019-07-18Next ›