LET’S TEST our metaexpertise by focusing on the last debate—a slightly lower-profile argument than the others but one in which the political-philosophical stakes could hardly be higher. The winners will shape the policies that employers must adopt to guarantee equality of opportunity in their workplaces. If unconscious biases are as potent and pervasive as some experts claim in law reviews, journal articles and court testimony, society may need to resort to more draconian measures to achieve equal-employment opportunity—in particular, numerical goals and quotas. In short, this debate pivots on whether the classic distinction between equality of opportunity and equality of results is sustainable.

In one corner of this debate, we find the race pessimists who include an array of prominent psychologists, social scientists and law professors (too numerous to enumerate) who make three sets of bold claims:

1. They have invented a device—they call it the Implicit Association Test (IAT)—that allows them to measure not only prejudices that people balk at acknowledging but also prejudices that they are flat-out unaware of having. The experts compare their new technique for unconscious mind reading to such revolutionary scientific breakthroughs as the development of the telescope. They declare the IAT to be a 100 percent–pure measure of prejudice. And they imply that the scientific debate is essentially over.

2. They have discovered that, although most people in early-twenty-first-century America claim to be unprejudiced on a conscious level, the IAT reveals that on an unconscious level this is very much not the case. Whereas only 10 or 15 percent of Americans endorse explicitly anti–African American sentiments, 70 or 80 percent register as biased against African Americans on an unconscious level.

3. And some even testify under oath as expert witnesses that unconscious biases will insinuate themselves into employment decisions whenever managers have “excessive discretion” in deciding whom to hire or promote—and that the only way to check such distortions is to hold managers accountable for achieving numerical goals (quotas in all but name) for the advancement of African Americans.

In the other corner, we find the relative optimists who warn of a precipitous rush to judgment about unconscious bias and of the dangers of discounting the extraordinary progress the country has made in the almost half century since the Civil Rights Act of 1964. They suspect that it does not help race relations to label the vast majority of the white population as antiblack on the basis of flimsy evidence. And they see themselves as defending scientific rigor and integrity against an onslaught of hyped-up claims (of course their detractors see them as reactionary apologists for residual racism).

The stage has now been set for the acid test: What help can our authors give us—and indeed federal judges and regulators—in becoming better appraisers of expertise? How do we figure out whether the unconscious-bias proponents have indeed made a revolutionary discovery that is now beyond reasonable scientific doubt, or whether we are looking at yet another faddish claim to fame, or whether the truth lurks somewhere in between?

OUR STATISTICALLY wary author of Proofiness Charles Seife is the perfect starting-off point, for he is ever concerned that experts distort their findings to mislead Joe Q. Public. He tells us to follow the logical trail from the measurement process that generates the numbers to the policy’s conclusion. We need to look out especially for data cherry-picking that disguises weaknesses to make them look like strengths.

So, on to the IAT. Everyone should now be curious, ideally equally curious, about where the numerical estimates of unconscious bias come from. And luckily the answer can be found at the IAT website which has been visited by millions of test takers—and which has over recent years issued varying but consistently high estimates of how much unconscious antiblack prejudice lurks in the American population.

Zero bias in IAT research has a precise meaning: it means recognizing flattering words linked to black faces just as fast as to white faces—and unflattering words linked to black faces just as fast as to white faces (if it helps to be evenhanded, think of the test as aimed at unconscious anti-Americanism—and measuring how fast you recognize flattering words linked to the face of Osama bin Laden versus George W. Bush). One can qualify as biased if one responds as little as one-tenth of a second faster to combinations of white-good and black-bad than to combinations of black-good and white-bad.

But a good Seife-ian worries about long, ultraspecific numbers for fuzzy concepts—and wants more than the expert’s word that a 166 millisecond response differential on a stimulus-recognition task (i.e., the IAT) qualifies one as highly racially biased. A good Seife-ian recognizes that every real-world number comes with a unit attached to it. And, in a thus-far-mysterious fashion, the experts have transformed a unit of time into a unit of prejudice. The smart question is: How do you know that zero bias on the test means zero bias in behavior?

The tacit assumption underlying the test is that people scoring zero are most likely to judge others on their merits—not on prejudice. But is that assumption justified? Can we safely glide from a 70 percent “fail” rate on the unconscious-prejudice test to the conclusion that 70 percent of the population is predisposed to discriminate against African Americans whenever they think they can get away with it?

In fact, when we crank up the microscope to the next level and explore how the actual propensity to discriminate against a group correlates with where one is located along the IAT scoring continuum, we find a messier picture than the test’s proponents imply. In some studies, the correlations run in the predicted direction (although they are not large). But in others, although the vast majority of subjects score as antiblack on the IAT, there is still, on average, a pro-black behavioral bias across a wide range of IAT scores, including at the zero point. In yet other studies, the more antiblack one’s IAT score, the more pro-black one’s behavior.

To clean up messes of this sort in complex research literatures, experts often rely on a technique known as meta-analysis which statistically summarizes and integrates the conflicting results. Suffice it to say, the meta-analyses have sparked only more controversy—but that has not stopped many experts from announcing to the world, and testifying under oath, that the debate about the IAT’s efficacy is effectively over.

AND YET even with all this data sifting—all this statistical skepticism—a further danger exists still: that ideological-bias problem. Common sense tells us (and if that is not enough, scientific studies reinforce the point) that liberals are likelier to resonate to pro-IAT arguments that emphasize the pervasiveness and potency of unconscious biases—and to see in these arguments compelling reasons for ramping up regulatory pressure on companies to promote African Americans. By contrast, conservatives are likelier to roll their eyes at liberal academics up to their usual mischief—and deplore pseudoscience that falsely labels as racist many fair-minded Americans and sets the policy stage for quotas.

If we are serious about de-biasing ourselves, we need to master the art of “turnabout thought experiments.” Confronted by controversial evidence, liberals and conservatives alike should learn, as a matter of course, to imagine reversing roles: Suppose that the same researchers used exactly the same standards of evidence not for identifying covert antiblack prejudice but instead for identifying covert anti-Americanism among American Muslims—the sort of national-security application likely to appeal to conservatives but make liberals nervous. And the unconscious-anti-Americanism researchers claim to find plenty of it among American Muslims. Again, common sense and science dictate that it is now the turn of the ACLU liberals to roll their eyes—and deplore reactionary pseudoscience that falsely impugns the patriotism of American Muslims.

The point of this thought experiment is to ensure that liberals as well as conservatives will be motivated skeptics, thereby checking the temptation to give a free pass to slippery science as long as only the other side’s ox is being gored.

In short, we have now gone as deep as even a devout Seife-ian would be willing to go (without being paid handsomely to continue). We have tried hard to rein in our own capacity for self-deception, the omnipresent temptation to believe what we want to be true. We have probed the meaning of zero bias in psychological measurement. And the fact is, we remain confronted with a big psychological barrier. Most citizens will economize on mental effort and accept numbers on faith if they resonate as true and come from a trusted expert.

DAVID FREEDMAN, however, encourages a more ruthless brand of skepticism—and some tempting shortcuts for people who don’t have the time for a PhD-level tutorial in psychometrics. It is not just that data can be manipulated. We must worry about the very incentives “experts” have for fudging their results. He builds on the rather sound premise that a disturbingly large percentage of this purportedly professional advice is flawed—and there are systematic reasons why many expert communities go offtrack. All too often, scientific journals, grant agencies and tenure committees put a premium on surprising (“counterintuitive”) findings that we discover on sober reflection are difficult to replicate. Which result is more likely to excite the pulse of liberal-leaning academic reviewers: to learn about previously hidden unconscious causes of our behavior that threaten to derail progress on civil rights; or to learn that representative-sample surveys over the last several decades have been getting it roughly right and that overt forms of prejudice that people are willing to express have been in steep decline—although significant pockets of prejudice remain?