Background: 

I met with Professor Paul Pillar (PP) of Georgetown University on February 1, 2012 to discuss intelligence analysis, analytic methodology, and producer/consumer relations as part of my ongoing dissertation research.  The conversation was illuminating in several ways, particularly with respect to relations between analysts and policymakers.  PP joined Georgetown University after a 28-year career in the US intelligence community, and also maintains an excellent blog on current foreign policy and national security issues.

Conversation Summary and Thoughts: 

I started out our conversation with several general questions regarding the relationship between analysts and policymakers, also referred to as producer/consumer relations in the intelligence literature.  I was particularly interested in the ways relations between the two affect how analysts go about their work, and select methodologies in their effort to address the needs and interests of policymakers.

PP noted much of the challenge for intelligence analysts is a result of the busy and dynamic nature of policymakers’ responsibilities.  While many of the idealizations of intelligence work assume that policymakers can articulate their intelligence needs to analysts and collectors, he noted that reality is far more complex.  Policymakers’ ideas, interests, and thoughts about issues can evolve quickly and unexpectedly.  This makes it difficult for them to anticipate precisely what their intelligence needs are or will be in the future.  Moreover, because policymakers have little time, they cannot always educate themselves on the issues that intelligence analysts have expertise on.  Alternatively, intelligence analysts cannot replicate the dynamic demands on policymakers, which are critical to understanding how consumers’ ideas, needs, and perspectives will evolve.

A follow-on question was on the differences between what qualifies a high-quality finished intelligence product within the intelligence community compared amongst their policymaking consumers.  PP noted that internally, features such as the intellectual cogency and elegance of an assessment are deemed important.  Alternatively, consumers may care more about the usefulness of the assessment, and wonder why it could not have been produced sooner, or if anything could have been done to resolve outstanding uncertainties and gaps in information.

One of PP’s most insightful comments came as he emphasized the primacy of consumer interests.  Importantly, he noted that tradecraft and methodology increased in importance whenever analysts received less guidance from policymakers about their interests and requirements.  Thus, analysis is often a straightforward act when sufficient guidance regarding priorities and interests are provided.  However, in the absence of these things, analysis becomes increasingly subject to methodological considerations in order to expose the assumptions and logic behind inferences given their uncertain relevance to consumer’s needs.  PP’s note on the importance of methodology as a substitute for external guidance also led to a related but inverted observation—that increased customer interest and guidance in a subject often reduced the extent to which analysts could explore alternatives.

PP’s observations suggest an interesting tension that is not unfamiliar to evolutionary and complex systems that must balance the use of resources between exploration and exploitation.  In this case, too much interest and guidance from consumers focuses analysts on exploiting a limited set of information and perspectives that will satisfy the interests of policymakers at the expense of searching for alternative points of views and additional collection.  As a result, analysts may lose the ability to explore alternatives (spending resources on potentially uninteresting or unproductive lines of reasoning), missing out on potentially important discoveries or opportunities to reframe questions.  By comparison, if policymakers take no interest in an intelligence problem, then analysts may constantly explore alternative frameworks, gather data, and engage in an endless search process that may produce conflicting assessments that do not build towards a shared understanding of the issues or character of a target or opportunity.  Thus, too little exploration leaves analysts and policymakers vulnerable to surprise, while too much exploration does not provide any consistent theme or approach around which an analytic foundation can be developed that will warrant the sustained interest of consumers.

A related question that I asked PP had to do with the contributions of Structured Analytic Techniques (SATs) (formerly known as Alternative Analysis).  SATs have been introduced into tradecraft in an effort to make analytic assessments more transparent with respect to the assumptions of the analysts, the data they used, and the logic of their arguments—making the relationships between evidence and inference explicit.  Additionally, they seek to extend the number of perspectives under consideration, ensuring that analysts reduce the psychological tendency to satisfice and accept the first theory that explains a particular situation or answers a question satisfactorily.  On the whole, PP believed that such techniques were generally helpful and should be encouraged, but also observed that their implementation was not simple or straightforward.  Specifically, it was noted that particular pieces intended to stimulate thinking or advance a case for a given hypothesis, regardless of its merits, could invite cherry-picking.  PP noted examples of CIA analysis during Vietnam, where a particular assessment was provided to the White House to answer a question regarding whether a particular case for success could be made, and was then treated as a considered analytic judgment once stripped of the caveats that characterized it as an intellectual exercise.  Likewise, he referred to a similar piece was written regarding an effort to assassinate Pope John Paul, laying out how to interpret the available intelligence to make a case for Soviet covert action—yet, PP noted that no consumer had asked for an analysis setting out the case against Soviet sponsorship of the assassination attempt.  Thus, it was suggested that in some cases, SATs and extensive explorations of alternatives could exacerbate certain producer/consumer pathologies, and that when such explorations were beneficial, packing them in a larger, comprehensive product could hedge against cherry picking single perspectives or scenarios if presented as individual products.

PP argued that the successful employment of SATs required managerial engagement and support.  Because of the agency’s effort to advance a corporate analytic line, managers needed to be disciplined regarding how and when alternatives were presented to consumers and ensure that Devil’s Advocacy, Red Cell, Scenarios, and other products were developed and advanced in the context of corporate perspectives and needs.  Absent managerial support, analysts could not advance products without risking isolation or employing resources in unproductive ways.  This suggested that a bottom-up approach to analytic production was unlikely to improve the overall performance of the community.  Finally, PP noted that there remain limits to what tradecraft and methodology can offer, and that achieving the levels of predictive success that many outsiders desire may not be possible.

On the issue of Analytic Integrity Standards and PDB guidance, PP did not feel that internal editorial practices significantly affected analytic work and were generally positive.  Specifically, they helped to ensure that the policy preferences of the analysts themselves did not shade their judgments, and made sure that the use of language employed by analysts was consistent across products—guaranteeing that everyone knew the “strike zone.”

In regards to the relationships between collectors and analysts, PP noted that this was an area where the community has significantly improved over the last couple of decades and that communications and increased flows of information across boundaries had helped a lot.  He noted that analysts have increasingly supported collection through the use of targeting analysis, and that collectors have played a stronger role in analysis by validating the use of data by analysts, ensuring that it use is appropriate based on the context of its collection (PP specifically noted the case of Curveball as a case where analysts would have benefitted from the guidance of collectors in order to understand how to use/not use data).  On this topic, he noted that the relationship is generally good, but unstructured and informal and that the use of computational simulations might add structure and sophistication to this relationship and reduce the use of scarce resources by limiting the amount of unnecessary information pulled into the system.

The final point about unnecessary information served as a prelude to questions about technology and its role in analysis.  PP argued that the most important challenge facing analysts was the massive volume of information that must be processed and examined, much of which coming from open sources.  He argued that this problem would become increasingly acute as human resources become scarcer and that simply providing analysts with the tools they want would not necessarily solve the institutional problems created by the massive quantities of data that required evaluation.

Another question that I asked dealt with the question how intelligence analysts’ work distinguishes them from scholars who study similar topics.  PP noted that analysts’ closeness to policymakers had a significant impact on their work processes, and resulted in elevated expectations, the need to address policy relevant concerns in real-time, and professional and personal pressure and responsibility for failures.  He noted that no scholar risks the headline “scholarly failure” if their predictions fail to materialize.

Interestingly, PP’s emphasis on constant contact and need to support consumers evoked another evolutionary model advanced by Herbert Simon and his work on complex systems.  Simon used the analogy of two watchmakers, each constructing watches with the same number of parts.  The difference in their work, however, was that one assembled the watches as a single unit, and, therefore, if disturbed, would lose his work and have to start from the beginning.  By comparison, the second watchmaker developed watches in modules, such at any disruption would only affect the module under development, leaving those that were already constructed unaffected.  The result was that the second, modular watchmaker was more productive.

This metaphor may capture some of the salient differences between intelligence analysts and scholars.  Because analysts are constantly in contact with policymakers, they are subjected to many “disruptions” and must therefore do their work in smaller pieces and cannot wait until the “end” in order to reach a conclusion or advance a judgment.  This may also mean that analysts’ views or beliefs may appear to be inconsistent over time, simply because the evolution of their thinking will be transparent and displayed because of the need to constantly publish their best estimate at that moment in order to support consumers.  By contrast, scholars may hide their evolutionary path from others, meaning that what appears as a single, integrated coherent and consistent final result may be the product of their internal learning and deliberations that remain private or shared only among peers, but never subjected to demands or disruptions of outsiders during the research process.

A final question concerned the prospects for new models of producer/consumer relations.  Specifically, new models have been advanced such as consulting, net assessment, or even joint production and assessment.  PP felt that the limitations of new models were already known and that preserving the distance between producers and consumers remained an important and necessary part of intelligence in the future.  He pointed to the conclusions of the Butler Report that suggested the two were too integrated in the UK in the buildup to the Iraq War.  There was an acknowledgement that wargames and simulation might serve as a bridge between the two and allow each to work together without going as far as “joint publication” which would necessarily politicize intelligence.  PP also noted that policymakers who had backgrounds in intelligence were different consumers, more informed about intelligence capabilities and processes, but were very few in number.