Outfoxing the Pundits

A groundbreaking study sheds light on how experts make predictions—and why they often are so calamitously wrong. 

In his groundbreaking study Expert Political Judgment: How Good Is It? How Can We Know?, Philip Tetlock employs the research done by psychologists, neuroscientists and game theorists to show how we make political decisions. The fundamental properties of the human mind—preference for simplicity, aversion to ambiguity and dissonance, belief in a controllable world, misunderstanding of probability theory—lead us to rely on narratives, the simple story lines that integrate and interpret information, both making sense of complex realities and trying to predict the future. Understanding how such narratives are used in politics sheds light on the biggest foreign-policy calamities of our times.

Not Outfoxed

There is no direct correlation between the intelligence and knowledge of the political expert and the quality of his analyses and forecasts, argues Tetlock, a political psychologist at UC Berkeley. What the pundit was thinking mattered less than how he was thinking.

To prove his point, Tetlock gathered and analyzed more than eighty thousand forecasts by academics and journalists about a variety of global issues, including the collapse of the Soviet Union and the end of the apartheid system in South Africa, discovering—surprise, surprise—that the experts were wrong more often than blind chance.

But occasionally Tetlock encountered an analyst who got it right more times than he got it wrong. And contrary to what the conventional wisdom would suggest, getting it right on, say, whether the Soviet Union would collapse or not, had very little to do with whether one had received a PhD with distinction in Russian Studies from Harvard. In fact, knowing too much about a subject and having strong personal commitment to the issue could be a major obstacle to getting it right. Or to put it differently, great minds don't necessarily make for great pundits.

Utilizing a typology of classifying political thinking and behavior introduced by the late British political philosopher Isaiah Berlin, Tetlock demonstrated the usefulness of sorting political experts along a rough cognitive-style continuum anchored at one hand by his prototypical “hedgehog” and at the other by his prototypical “fox.”

Hedgehogs are intellectuals who “know one big thing,” one Big Idea that they tend to apply in dealing with policy issues. They integrate the many pieces of information into a mega narrative that reflects a long-held theory and ideology, dismissing facts that don’t fit into their dogma. Foxes, on the other hand, know many things. They tend to examine the many facts before them, and try to draw the outline of their policy analyses based on the reality on the ground and not on a grand theory that may sound very profound but doesn’t provide an explanation that could produce workable solutions to concrete challenges.

The intellectually aggressive hedgehog is an ideologue who seeks, under the banner of parsimony, to expand the explanatory power of that Big Idea to “cover” new cases. He toils devotedly within one tradition and imposes formulaic solutions on ill-defined problems.

The more eclectic foxes are empiricists who are content to improvise with ad hoc solutions to keep pace with a rapidly changing world. Drawing from various and sometimes contradictory ideas and traditions, she is better able to improvise in response to changing events and more successful in predicting the future than the hedgehog.

Paradoxically, Tetlock noted, there was a perversely inverse relationship between the scientific indicators of good judgment and the qualities that the media most prized in pundits—an “expertise” in a specific field and the single-minded determination required to prevail in ideological combat. Hence, when it comes to the op-ed pages and the television news shows, it's the hedgehog that rises to the top.

The public and the elites, including the media, tend to be attracted to the assertive hedgehogs and their Big Ideas, which make for good copy and sound bites that make it easier to employ a battle cry to mobilize people to “war” on this or that problem. The foxes sound a bit hesitant and irresolute and they do a lot of zigzags, use a lot of “on one hand” and “on the other hand,” which doesn’t sound so great on television.

Rise of the Hedgehogs?

Life has been good for the hedgehogs since Tetlock completed his study. Consider our contemporary media environment, in which thousands of Internet sites and blogs, Facebook posts and tweets, and even the surviving representatives of the mainstream media are competing for our attention—and providing us 24/7 with instant narratives.

This media environment is more conducive to the kind of simplistic narratives that reflect the one-dimensional thinking of the hedgehogs. These narratives can accommodate very complicated foreign-policy issues—often involving many players, interests, geographic locales and cultures—into a brief blog post or a catchy tweet. These hedgehog narratives thus transform an intricate global crisis into a morality play, a suspenseful plot dominated by the a clash between “good guys” and “bad guys,” a cosmic struggle between grandiose ideas and great civilizations.