Survivorship Bias

There’s a probably apocryphal story of a man who wrote to a number of wealthy people telling him that if they paid him a dollar he would accurately predict the winner of an upcoming NFL game. If he was wrong, he would refund their money.

He got over a hundred takers and he sent half of them the prediction of the home team winning, the other half the guest team. He refunded the half that he got wrong and sent a follow-up letter to the half he got correct offering them a prediction of the next game for 5 dollars.

Again half got one prediction, the other half the other. Another set of refunds for the one, and another set of letters continuing to increase the price to the other half. After a few more rounds of this, he had two people left, both of whom had just had this guy predict correctly 6 games in a row. This time he offered to sell them the correct prediction for $100,000 with the same caveat that he would refund the money if he was wrong. Both took him up on it, but he got greedy and just ran away with the $200,000 instead of refunding half of it and so he was arrested on fraud.

Consider this from the perspective of those remaining two people, who felt they had found a modern-day fortune-teller, because they didn’t see all the wrong guesses only that he was always right. This is often called the survivorship bias – we only look at what remains standing to assess what is true and not recognizing all of those who died along the way.

This bias crops up everywhere. The investment company that for the last 10 years had better returns than the S&P 500, so they must be awesome neglecting the fact that there were thousands of other companies and so statistically one of them should have this record.  The business book that looks at the most successful companies with 100,000 surveys neglecting all those who tried their sage wisdom and did not succeed. We commit this fallacy when we decide to start an enterprise pointing at the Mark Zuckerbergs or Larry Page and Sergey Brin’s as proof that billion dollar dreams can come true (ignoring the fact that hundreds of thousands who all had good ideas failed to deliver).

This is part of the WYSIATI (What You See Is All There Is) bias which does not allow us to consider all the events that we don’t know about or who are no longer around. When we find ourselves pointing at good examples, it’s important to try to consider whether there are any who might be dead who tried those same things. Not to depress ourselves, but simply as a reality check to the conclusions we are likely to make when all we see is the survivors.

We may think our conclusions are can’t miss.  After all, up to the final prediction, our con-artist has gotten every single prediction correct. Presumably the people taking our fortune teller’s offer were using that information to make even larger bets on those games — trying to make some quick money. Yet even after so many successful predictions in a row, one of the parties still got the wrong prediction at the end. When we find ourselves in “can’t lose” situations because statistically we’ve never seen anything that would indicate otherwise, we should naturally wonder what we are missing, and seek out the data that has died.

Photo Credit:  Tristan Martin