I have just finished Philip Tetlock’s fine book about forecasting the future in an intelligent way “Superforecasting: the Art and Science of Prediction”. His keen insights on this subject in his previous book “Expert Political Judgment” suggested that most political pundits we see in the media are no better at making precise, time bounded predictions about the future than a pack of chimps throwing darts at a dart board to generate choices.
But he has also found there is a group of people who are very good at forecasting, and his latest project realated in his new book has been to harness their skills in a forecasting tournament for the US intelligence community to make literally thousands of precise, time bounded predictions with precise estimates of likelihood to the nearest percentage point and then figure out who is great at this over endless iterations and how they do it.
Key to the super-forecasters is that they are Foxes not Hedgehogs as per Isaiah Berlin’s saying from the Greeks: “The fox knows many things, the hedgehog just one.” His foxes do not have one over-arching view of how the world is, but many and they use multiple perspectives and are evidence based, and avoid thinking anything is inevitable. They also learn from their mistakes because they are precise not vague in their forecasts. And they revise their forecasts appropriately in the light of new data as it emerges. They are highly curious, eager to learn and willing to get into the detail and understand causal forces. And they avoid what the media audiences love: confidence and certainty in commentaries unsupported by evidence or real thought and completely without accountability when their forecasts fail aka punditry.
At the end of this book he adds some rules his superforecasters used:
- Focus on prediction questions where the hard work is likely to pay off aka triage
- Break down seemingly intractable problems into tractable sub-problems
- Strike the right balance between inside (the precise situation involved) and outside views (similar situations in the past).
- Strike a balance between under and over reacting to new data
- Look for the clashing causal forces in each problem
- Strive to distinguish as many degrees of doubt as the problem permits, but no more
- Strike the right balance between under and over confidence, between prudence and decisiveness
- Look for the erros behind your mistakes but beware of rearview mirror hindsight biases
- Bring out the best in others and let others bring out the best in you
- Master the error-balancing bicycle
- Don’t treat these commandments as commandments
Philip’s sites: http://goodjudgment.com/gjp/