At the heart of effective statecraft lies the burden of ascertaining the best available truth about the capabilities and intentions of a state’s allies and adversaries. Equal to the high stakes of intelligence performance is the difficulty of the tasks involved. The importance of knowing one’s enemies confronts the enemies of intelligence in a contest that often enough favors the latter. Because intelligence failure is inevitable over time, intelligence agencies are subject to intense external and internal scrutiny.[1] Feeling burned by previous intelligence failures, political leaders may question, sometimes repeatedly, the veracity of the conclusions their intelligence analysts provide; in the extreme, whole-scale organizational and procedural reforms can be imposed so to avoid the commission of similar mistakes in the future. One under-studied form of internally-derived correction that comes in the wake of major intelligence failures is the degree of urgency with which intelligence judgments are rendered.
H-Diplo | ISSF Article Review 119
Editors: Diane Labrosse and Seth Offenbach
Web and Production Editor: George Fujii
Cullen G. Nutt. “Proof of the Bomb: The Influence of Previous Failure on Intelligence Judgments of Nuclear Programs.” Security Studies (2019): 1-39. DOI: https://doi.org/10.1080/09636412.2019.1551569.
Published by ISSF on 14 June 2019
https://issforum.org/articlereviews/119-proof
PDF: https://issforum.org/ISSF/PDF/ISSF-AR119.pdf
Review by Spencer D. Bakich, Virginia Military Institute
At the heart of effective statecraft lies the burden of ascertaining the best available truth about the capabilities and intentions of a state’s allies and adversaries. Equal to the high stakes of intelligence performance is the difficulty of the tasks involved. The importance of knowing one’s enemies confronts the enemies of intelligence in a contest that often enough favors the latter. Because intelligence failure is inevitable over time, intelligence agencies are subject to intense external and internal scrutiny.[1] Feeling burned by previous intelligence failures, political leaders may question, sometimes repeatedly, the veracity of the conclusions their intelligence analysts provide; in the extreme, whole-scale organizational and procedural reforms can be imposed so to avoid the commission of similar mistakes in the future. One under-studied form of internally-derived correction that comes in the wake of major intelligence failures is the degree of urgency with which intelligence judgments are rendered.
In his important recent article, “Proof of the Bomb,” Cullen Nutt maintains that variation in the urgency of definitive judgment should complement intelligence accuracy in broader performance assessments of intelligence agencies. Nutt bounds his domain of inquiry tightly, focusing on variations in judgment definitiveness on nuclear weapons proliferation. How an intelligence agency judges the status of an opponent’s covert nuclear weapons program—whether it judges ’urgently’ or ’skeptically’—is a function of the type of major failure that the agency experienced in the recent past. If it experienced a recent “false positive” failure (an incorrect conclusion that an opponent had a viable nuclear weapons program), then the agency will judge skeptically in the present. The high costs of repeating a false positive error will incline the agency to construct an airtight case, a substantial raising of the evidentiary bar for a definitive conclusion. If the agency experienced a recent “false negative” failure (an incorrect conclusion that an opponent did not have a viable nuclear weapons program), then it will judge urgently in the present. The agency will be incentivized to boost its collection and analytical efforts, entertain far-fetched possibilities, and lower its standard of proof for a definitive conclusion.
The degree of failure the agency experienced in the past matters significantly for Nutt, who argues that only major failures induce changes to the urgency with which judgments are reached. For a case to be considered a major failure “both policymakers and intelligence agencies must agree that a lapse occurred. They must also agree that the state either suffered or was in danger of suffering high costs as a result of this error” (9). Major failures have a significant effect on judgment because they trigger organizational concerns about the possible loss of influence with policymakers, and individual psychological fears about being terribly wrong again. Additionally, Nutt casts variation in definitiveness in judgment (urgency versus skepticism) comparatively. That is, if an intelligence agency judges urgently, it renders definitive judgments earlier than other similarly situated intelligence agencies. Judgment definitiveness is thus neither an absolute construct, nor compared against the agency’s own past performance.
Nutt tests his ‘past failure’ theory by examining how American and Israeli intelligence agencies evaluated the development of the Libyan and Syrian nuclear weapons programs in the 2000s. After lengthy negotiations with America’s CIA and Britain’s MI6, in December 2003 Colonel Muammar Gaddafi agreed to dismantle his nuclear weapons program in return for the U.S. agreeing to remove Libya from its list of state sponsors of terrorism and to commit to economic investment in that country. When this arrangement was announced, Israeli intelligence had no idea that Libya had an active program, completely missing the years-long collaboration between Pakistani nuclear scientist A. Q. Khan and Gaddafi’s regime. This false negative failure was jarring to the Israelis and would factor significantly in Israeli intelligence judgments regarding Syria’s program. U.S. and UK intelligence, however, had been focused on the Khan network since at least 1996. Over the course of the next seven years, the CIA and MI6 cultivated sources with knowledge both about that proliferation vector and about Libya’s program. These two intelligence targets merged in October 2003 resulting in the seizure of a ship carrying centrifuges to Tripoli.
According to Nutt, Israeli intelligence judged the prospects of a Libyan nuclear program skeptically, while American intelligence was far more urgent in its assessments. Israeli skepticism discounted the seemingly outrageous notion that a single foreign scientist could be a one-man proliferator of the world’s most dangerous weapons technology. American urgency, on the other hand, had its roots in the failure to predict the al Qaeda attack on September 11, 2001. In the wake of the 9/11 false negative failure, the CIA devoted substantially more resources to counterproliferation and was willing to run greater operational risks to penetrate the Khan network. Standards of proof for further investigation were lowered and the CIA’s Counterproliferation Division, flush with new organizational influence, was given the freedom to pursue even farfetched leads. Finally, the CIA lowered American standards of proof for definitive judgment. For example, the 2001 National Intelligence Estimate declared that Khan’s activities were directly linked to Libya’s program, despite the fact that CIA had substantially less insight into activities occurring internally to Libya than they did externally. The analytical inference made by U.S. intelligence was based on worst-case analysis; clear evidence, Nutt contends, of urgent judgment.
In the late 1990s, the CIA concluded that despite Syria’s aggressive development of chemical weapons it was not pursuing nuclear weapons capabilities. Around the time of President Hafez al-Assad’s death in 2000, Syria and North Korea arranged for the construction of nuclear reactor in eastern Syria at a site called Al Kibar. Despite the construction of the site, which American intelligence monitored closely, the CIA held firm in its conclusion that nothing of significance had changed with respect to Syria’s nuclear ambitions or activities. This judgment was upended in 2007 when Israel’s intelligence agency Mossad provided the CIA with photos from inside a mysterious building near the Euphrates River showing clear evidence of Syrian and North Korean scientists working together at a nuclear reactor.
The CIA’s skepticism did not result from a lack of information (indeed, American and Israeli intelligence was very similar), but from an unwillingness take large inferential steps prior to spring 2007. For its part, Israeli intelligence monitored Syrian activities assiduously and rendered urgent judgments about its neighbor’s aspirations and activities. The differences between U.S. and Israeli intelligence in the Syrian case, Nutt contends, are explained by ‘past failure’ theory. Coming on the heels of the false negative failure in the Libya episode, Syria’s activities drew close scrutiny by Israeli intelligence, attention that was not necessarily warranted prior to the acquisition of clear photographic evidence. American skepticism, on the other hand, was born from the false positive failure in the Iraq Weapons of Mass Destruction (WMD) episode of 2003. Interestingly, even when evidence was presented that showed Syria did have a nuclear weapons program, U.S. intelligence officials refused to conclude that it pointed clearly to such an effort. Refusing to attach “high confidence” to any analytic judgment in the absence of clearly confirming evidence, U.S. officials stated that they believed with “low confidence” that Syria had a nuclear weapons program (31). The insistence on downgrading confidence in assessments in the absence of supporting evidence is, for Nutt, the hallmark of skeptical judgment.
“Proof of the Bomb” is an important contribution to the literature on intelligence performance for two reasons. First, Nutt makes a compelling case that the differences between U.S. and Israeli intelligence judgments resulted neither from the interests and/or threat perceptions of civilian leaders, nor the underlying intelligence that each side had at the outset. With respect to civilian influence, for example, Nutt shows that despite persistent queries and insinuations, Vice President Dick Cheney was incapable of moving the U.S. intelligence community from its skeptical position regarding Syria’s covert nuclear weapons program. This finding is important, specifically in this case, because it demonstrates the limits that powerful leaders confront when challenging the judgments of their Intelligence Communities (ICs). In so doing, Nutt adds another lens to view “intelligence-policy relations.”[2] Second, Nutt carefully distinguishes between judgment definitiveness and intelligence accuracy, though his concept construction is shrewd enough to open space for causal connections between the two. Judgment and accuracy are two sides of the same coin, but skepticism can bias agencies toward underestimation, while urgency tends to favor overestimation. Even so, Nutt strays into murky waters by suggesting that post-Iraq War intelligence skepticism “served the cause of accuracy” in the 2007 case of Iran’s purported nuclear weapons program (36). The problem is that such a telling blurs the causal story presented here, that failure in the past, and not a paucity of evidence in the present, induces skepticism.
Nevertheless, a measure of skepticism about Nutt’s argument is warranted. There is reason to question the stipulated effects of major intelligence failures on judgment definitiveness, a problem resulting from Nutt’s truncating his universe of cases to include only possible nuclear proliferation. Even though he maintains that major intelligence failures are significant events, they are even more consequential—and may have greater and more complex results—than Nutt allows. The reforms to the U.S. intelligence community after 9/11, for example, were comprehensive, resulting in significant bureaucratic and procedural changes that affected most of U.S. intelligence activity. Given the far-reaching effects of major failure, it is reasonable to widen the aperture to draw comparisons among intelligence activities in multiple domains (Nutt does not explain why his domain is limited to possible proliferation, nor why the reactions to failure by counterproliferation intelligence officials would be different that those in other domains.)[3] In addition to weapons of mass destruction, Iraqi President Saddam Hussein’s purported links to terrorist organizations constituted a critical component of the Bush administration’s casus belli.[4] And yet, despite the false negative failure that resulted in the 9/11 terrorist attacks—a false negative that had far more to do with terrorism than with nuclear weapons—U.S. intelligence judged skeptically on the question of Iraq-al Qaeda ties. As Robert Jervis notes,
While information pointing to a connection [between Iraq and al Qaeda] was unreliable, there were enough scattered reports that someone who had a different reading of the regime could have placed more faith in them, as the vice president and many civilians in the Defense Department did. In fact, because of differences in background beliefs, terrorism analysts in the IC were more prone to see links than were those with regional expertise.[5]
Urgent judgments were not passed by U.S. intelligence, but by political and defense officials. According to ‘past failure’ theory, U.S. intelligence should have entertained far-fetched scenarios and lowered the standard of proof for definitive judgment. It did neither.
The difference between U.S. counterterrorism judgments (about Iraq) and counterproliferation judgments (about Libya, and possibly Iraq) in response to the same major intelligence failure requires an explanation. For Nutt, while individual and organizational learning are different, organizational learning (which is manifest in variations in judgment) occurs through individual-level processes. Quoting Jack Levy, Nutt writes that organizations “learn only through individuals who serve in those organizations, by encoding individually learned inferences from experience into organizational routines” (11-12).[6] The encoding process, however, needs greater specification. Did the individuals in the counterterrorism and counterproliferation divisions learn different lessons in response to the false negative failure of 9/11? If so, then the content of each division’s set of pre-existing beliefs would play a mediating between the shock of failure and subsequent behavior.[7] Or was there wide variation in the lessons learned by individuals within both divisions, but process of aggregating those lessons differed for some yet unidentified reason? The upshot is that Nutt presents a plausible argument, but one that applies only under certain organizationally-determined conditions.
Overall, “Proof of the Bomb” is an important contribution to the literature on intelligence performance. Cullen Nutt’s ‘past failure’ theory is a productive first step to explaining cross-national variation in intelligence judgment.
Spencer D. Bakich is an associate professor of political science and the Director of the National Security Program at the Virginia Military Institute. He is the author of Success and Failure in Limited War: Information and Strategy in the Korean, Vietnam, Persian Gulf and Iraq Wars (University of Chicago Press, 2014).
©2019 The Authors | Creative Commons Attribution-NonCommercial-NoDerivs 3.0 United States License
[1] Richard K. Betts, Enemies of Intelligence: Knowledge and Power in American National Security (New York: Columbia University Press, 2007), 28-32.
[2] Joshua Rovner, Fixing the Facts: National Security and the Politics of Intelligence (Ithaca: Cornell University Press, 2011), 6-8.
[3] In a footnote, Nutt stipulates that “Judgment by intelligence agencies in domains outside of possible proliferation may also vary” (7).
[4] Spencer D. Bakich, Success and Failure in Limited War: Information and Strategy in the Korean, Vietnam, Persian Gulf and Iraq Wars (Chicago: University of Chicago Press, 2014), 188-189.
[5] Robert Jervis, Why Intelligence Fails: Lessons from the Iranian Revolution and the Iraq War (Ithaca: Cornell University Press, 2010), 147, 161.
[6] Jack S. Levy, “Learning and Foreign Policy: Sweeping a Conceptual Minefield,” International Organization 48:2 (Spring 1994): 287. DOI: https://doi.org/10.1017/S0020818300028198.
[7] Jeffrey W. Legro, Rethinking the World: Great Power Strategies and International Order (Ithaca: Cornell University Press, 2007).