The problem of the missing cases
Survivorship bias is the error of drawing conclusions from the cases that made it through a selection process while ignoring the ones that did not. The survivors are visible, and the others are not, so the visible group looks like the whole picture.
In sport, esports and gaming, success stories are everywhere: the player who trained for ten hours a day, the streamer who quit a job to go full time, the team that took a risky strategy and won. They make good stories because they ended well. The trouble is that we rarely hear from the equally determined people who did the same things and did not succeed.
A classic example
A frequently told example comes from the Second World War. Analysts examined returning aircraft and mapped where they had been hit by enemy fire, with the idea of adding armour to the most-hit areas. The statistician Abraham Wald is usually credited with pointing out a flaw: the planes being studied were the ones that returned. Hits in other areas probably belonged to planes that did not make it back. The areas with few holes on returning planes may be exactly the ones where damage was fatal. Details of the story vary between tellings, but the lesson is the useful part: the sample was filtered by survival.
An illustrative sports example
Suppose 1,000 young players each practise a particular intense routine for years. Imagine, purely as an illustration, that 10 of them reach a professional level. When those ten are interviewed, many of them credit the routine, and the routine seems to be the secret.
But those figures say nothing about the other 990, who did the same routine and did not reach that level. If, in a different group, players who did not follow the routine reached the professional level at a similar rate, the routine would not explain much at all. The interviews cannot tell us, because they only include the survivors.
Real careers are shaped by talent, health, timing, coaching, opportunity, money and luck, as well as effort. A success story can honestly mention effort and still leave out the contribution of the others.
Where it appears
- Career advice. “Quit and go all in” sounds like a winning approach if you only hear from those it worked for.
- Strategies in games. If a risky opening is celebrated because it won a famous tournament, it is worth asking how often it lost in earlier rounds.
- Records and lists. “Best players ever” lists include only those who reached the top, which can make certain traits seem more common than they are among all who tried.
- Old equipment and tactics. Things that appear to have lasted well are often the ones that did not fail, with the failures long discarded.
- Leaderboards. Players who quit after a bad run drop out of the data, so those who remain show better average results than the starting group would have.
Link to other statistical pitfalls
Survivorship bias is a form of selection, and it often travels with other errors. A claim that successful players share a trait is a correlation, and the question of correlation versus causation applies: the trait may not cause success. If only the best examples are quoted, it overlaps with cherry-picked statistics. And the thinking is related to base rate neglect, since the relevant base rate is how many people tried the approach in the first place, not just how many succeeded.
A related point concerns luck. When a large number of people try something with uncertain results, some will do well by chance. A few exceptional performers among thousands is what we would expect from randomness alone, even if there were no skill differences. The very fact that someone is visible does not show their method was better, and an outstanding run is also subject to regression to the mean.
How to correct for it
- Ask for the denominator. How many people tried this, and how many succeeded?
- Look for the failures. What happened to those who did the same and fell away? Are they discussed anywhere?
- Ask what the successful group has in common with the unsuccessful one. A trait shared by both does not explain the difference.
- Consider the filter. What had to happen for someone to appear in this story? Who was never going to be interviewed?
- Separate effort from outcome. A good process can have a bad result, and a poor process can have a good one.
- Treat advice as a hypothesis. Even honest advice from a successful person describes what they believe worked for them.
A balanced takeaway
None of this means success stories are useless or that effort does not matter. Often, hard work and good habits do improve the odds. The point is narrower: a success story alone is weak evidence about what caused the success, because it comes from a filtered sample. Learning from winners is fine, as long as you also ask about those who are not in the room.
The same caution applies to any activity where results are uncertain. Outcomes are never guaranteed by following someone else’s route, and it is wise to weigh the cost of trying, in time and money, against what you can afford to lose.