The Buzz

The Data-Driven Transformation of Intelligence

When “little green men” invaded Crimea in early 2014, they left a data trail that went largely unnoticed by the U.S. Intelligence Community (IC).  Distracted by a large Russian exercise to the west, the IC did not connect the digital dots that indicated the impending invasion.  In the Information Age, the “dots” are more plentiful and glaring as everyone now leaves a data trail.  Given that, how can intelligence analysts better gather, share, organize, and view data to reveal intent, more accurately predict behavior, and make better decisions with limited resources?

Intelligence is, for the most part, production focused, meaning many analysts are often forced to make quick assessments from limited text-based sources to meet deadlines.  As such, they can be trapped into making predictions that put too much weight on personal experience and cognitive biases.  This is a big problem considering the difficulty of forecasting the behavior of every rival the U.S. faces in the global arena or on the battlefield.  The best analysis derives from the right combination of art and science.  It is dangerous to favor the former at the expense of the latter, even for the most experienced and skilled analysts. 

If the U.S. government is to deal with deceptive and elusive adversaries, the IC must make data, structured in a way to tell a story, the foundation for its intelligence assessments.  Data analytics are changing how we approach economics, elections, health, security, business, and our daily routines.  It is time for intelligence to catch up.  Analysts must pair their intuition and expertise with data science to give decision-makers the best possible intelligence.  

In turn, IC and defense leaders must accelerate this data-driven transformation of intelligence tradecraft, and demand a more structured, scientific approach to the intelligence problems we face.

Data correlation has always been part of intelligence tradecraft, however, it is largely still a product of analysts laboriously reading through of mountains of intelligence reports.  There is a limit to the type of reports available to analysts, who have limited time to make assessments.  There is a better way for analysts to tackle intelligence problems given the amount of data that exists—or could exist with the right focus—on just about anything or anyone. 

Imagine if in the post-mortem of the Crimea invasion analysts, working with data scientists, could take the datasets tied to the missed indicators and fold them into algorithms that drive artificial intelligence (AI).  This AI could then help analysts predict future outcomes and focus resources.  As this human-machine ”red” teaming matures, AI could learn to think like an adversary by looking for data, in specific combinations and quantities, to anticipate when little green men will be on the move, for example. 

Many in the IC are already moving down this path.  Analysts are using applications similar to Waze, a service where users input real-time road activity, to record and geo-tag an adversary’s movements and actions.  Focusing on activity patterns versus individual pieces of intelligence is helping analysts see the connections between the dots rather than just the dots themselves.  

Analysts are also beginning to use apps to identify and catalog intelligence objects (people, places, and things of interest) into databases similar to the real estate platform Zillow, which allows users to assign data attributes to a property (numbers of bedrooms for example), or link the property to data feeds like  Similarly, there are intelligence apps that allow analysts to tag an object based on a new discovery, or add intelligence feeds such as imagery streams or social media, to the object’s database entry.  This method essentially creates a digital bucket to store relevant intelligence data about an object.  

By focusing on data performance vis-à-vis activity and objects, analysts can better understand and anticipate the value of different types and combinations of data to inject precision in intelligence tradecraft.  In other words, understanding data performance allows analysts to play Moneyball with limited resources to generate the optimal type, quantity, and combination of data needed to answer specific intelligence questions.  Sports teams are using data science to focus as much on themselves as they do the competition.  How can the IC do the same?

Statistically speaking, the star of a team may not be the best lead-off hitter or even the best player in certain circumstances.  When it comes to intelligence, however, analysts and decision-makers tend to bias toward a single source more often than not.  The preferred intelligence source today is a full-motion video (FMV) feed from a remotely-piloted aircraft (RPA).  In other words, when leaders and analysts are faced with a new crisis or intelligence problem, they reflexively ask for an eye-in-the-sky.