Nearly a decade and a half ago, as our nation’s leaders pondered the possibility of war with Iraq, the US Intelligence Community published a set of judgments on whether Iraq was hiding WMD programs despite international prohibitions. The individual elements of the analytic case were each perfectly reasonable: that Iraq had produced and used chemical weapons in the past, that it had not been forthcoming with inspectors seeking to verify compliance with UN Resolutions, that President Saddam Hussein was a brutal and deceptive dictator with a history of hiding illicit weapons programs, and that several intelligence sources indicated that Iraq had ongoing programs. The conclusion that Iraq was “continuing and in some areas expanding its chemical, biological, nuclear and missile programs contrary to UN resolutions” was widely shared in both the US and Europe.[1] But it proved incorrect. In retrospect, analysts should have been more circumspect about their judgments and more open to alternative explanations of the evidence.

Today we wrestle with another vexing and politically charged analytic problem: Did Russia interfere in the US presidential election to aid the candidacy of Donald Trump? On the surface, the case against Moscow is intuitively obvious. Information detrimental to Hillary Clinton was clearly stolen from Democratic National Committee and other sensitive computer servers and then leaked to the media. Forensic data traceable to Russia were found in the intrusions. The operations were consistent with cyber techniques that Russia has used repeatedly in the past against both the US and other countries, and Moscow had an undeniable preference for one candidate over the other in the election.

The conclusion that Russia hacked its way toward a Trump victory is no slam dunk, however, despite its plausibility. Although the Intelligence Community has not published its evidence or analysis regarding this case, the analytic lessons learned from post-mortem reviews of the Iraq WMD failure argue for approaching the matter with a great deal of caution.  Applying these lessons to the case of the election intrusions – an analytic “pre-mortem,” so to speak – is one of the best means of ensuring that we do not fall into the same cognitive traps.

Lesson One: Explore Alternative Explanations. One of the most significant problems facing intelligence analysts is that nearly always, the information available to them is consistent with multiple explanations. In Iraq, the most famous example was a communications intercept cited by Secretary of State Colin Powell, which quoted Baghdad as telling officials at an Iraqi military base that was about to be visited by UN inspectors to “clean out all the areas, the scrap areas, the abandoned areas. Make sure there is nothing there.”[2] The meaning seemed clear: remove WMD before inspectors arrive. But in fact, Baghdad merely wanted base officials to remove traces of old, destroyed material that might have been misleading to inspectors. The intercept was not as conclusive as Powell or others suggested. Although it was used to support the judgment that Iraq was hiding illicit WMD stockpiles, the intercept was equally consistent with the hypothesis that Iraq had destroyed the stockpiles but was ambivalent about revealing this fact to the world.

In the case of Russia today, it is possible that the Intelligence Community has classified information that shows directly and conclusively that the Russian government ordered the intrusions and deployed the stolen data with the specific intent of aiding Trump’s candidacy.  Illustrative examples of such conclusive evidence might include an intercepted communication in which a Russian government official directed or approved the operations, or a pilfered Russian government policy paper of good provenance outlining their approach to influencing the US elections. But public comments from individuals briefed on the matter suggest that the available evidence is circumstantial rather than diagnostic.[3] Such a situation demands examination of alternative explanations of the evidence surrounding alleged Russian election hacking.

Take, for example, the forensic data on the DNC intrusion. In the world of cyber operations, attribution – determining who is responsible for penetration of a computer network – is a particularly difficult problem, because hackers can easily mask their locations and identities through the use of proxy systems and “botnets,” computers belonging to others that the hackers have electronically hijacked for the purpose of using them in an intrusion. Cyber operations rarely feature the equivalent of fingerprints or DNA evidence. Given the technologies that are available to hackers, “false flag” operations – which make it appear that an intrusion has originated in one country when in fact another is responsible – are fairly easy to pull off.[4][5]

This argues for caution in assessing the evidence surrounding the DNC intrusions. According to analysis published by the cyber security firm CrowdStrike, hired by the DNC to investigate the breach of their servers, several clues point toward Russia’s responsibility: the tactics of the intruders closely resembled those typically used by two hacking groups thought to be Russian by numerous cyber experts; the activity by the intruders on the DNC network tended to take place during Moscow working hours; and some of the stolen documents released to the media contained signs that Russian speakers were involved.[6][7]

While each of these facts indeed supports the judgment that the Russian government was behind the operations, each is also consistent with alternative explanations, including that it was a false flag effort or conducted by a private hacking group with the aim of selling the stolen information to the Russian government or others.

Lesson Two:  Look for Disconfirming Information.  The temptation to regard a piece of evidence as diagnostic when in fact it is consistent with multiple explanations is a type of “confirmation bias” – the tendency to see what we expect to see – to which all humans are prone. In the case of Iraq, this bias was evident in analysts’ gravitational attraction to reporting that aligned with their well-founded suspicions that Baghdad was hiding WMD stockpiles, and in their reluctance to give weight to reports that Iraq had destroyed them. This tendency was so strong that the WMD Commission report said analysts simply “disregarded evidence that did not support their hypotheses.”[8]

One of the best ways that analysts can mitigate their susceptibility to confirmation bias is actively to seek information that is inconsistent with their leading hypotheses. In the case of the DNC intrusions, press reporting suggests that cyber investigators have two interrelated “what” and “why” hypotheses: that the Russian government directed or approved the hacks, and that it purposively used the stolen data to bolster the candidacy of Trump.[9] Is there information available that is inconsistent with these hypotheses?

The public record indicates that there is. According to CrowdStrike’s report, the two hacking groups that penetrated the DNC (which it dubbed “Fancy Bear” and “Cozy Bear”) have engaged in “extensive targeting of defense ministries and other military victims … that closely mirrors the strategic interests of the Russian government.”[10] In other words, the DNC hackers probably worked for the Russian government because they have a track record of technically sophisticated operations against targets relevant to the Russian state.

But have they also targeted organizations that would seemingly be irrelevant to – or even inconsistent with – Russian national interests?  Yes. The CrowdStrike report is mum on this matter, but other cyber investigators point out that both Fancy Bear and Cozy Bear have engaged in a wide variety of targeting that includes web service providers and finance companies.[11] Such operations are explainable – they could for example reflect efforts to gather information that could be useful in separate attempts to penetrate national security targets – but they could also be a sign that the DNC hackers are a diverse group of cyber entrepreneurs who may or may not have Russian government connections and who generate their own target lists independent of outside direction. The targeting history of the purported DNC hackers does not by itself disprove Russian government involvement, but it raises questions about how confident we can be of that involvement.

The CrowdStrike report includes a second red flag: the DNC was breached at least twice, first in the summer of 2015, and then again in March 2016.  Each intrusion was conducted by a separate hacking group, and each stole many of the same documents. CrowdStrike acknowledges that this is unusual; the more often an organization is targeted, the more likely the intrusion will be detected and blocked. Failure to coordinate what information was taken suggests a lack of central direction. But CrowdStrike explains this anomaly as the product of inter-service rivalry between Russia’s military intelligence directorate (the GRU) and its civilian intelligence agency (the FSB), each of which presumably wanted in on the DNC action. This explanation is not implausible – Russia is at least as prone to bureaucratic squabbles as any other government – but the implication that the Russian leadership would compromise operational security in the interest of managing lower-level infighting begs for alternative explanations. Might the impetus for the hacks have come independently from the hackers themselves rather than the Kremlin? Or might Moscow have wanted the intrusions detected, perhaps to send a signal to Washington that it was retaliating for perceived US interference in Russian elections?