Matthew Connelly’s The Declassification Engine: What History Reveals about America’s Top Secrets explores what it means for Americans to live in a society whose governing institutions are incentivized to keep secrets. The book stems from Connelly’s tenure as lead investigator at Columbia University’s History Lab, in which a team of historians and data scientists spent several years using machine-learning tools to mine an extensive repository of declassified government records. Drawing on History Lab’s findings, The Declassification Engine traces the origins of what Connelly calls the American “dark state” (19), which classifies far more information than it makes public. In the book’s opening chapters, Connelly argues that the US government established a self-perpetuating culture of secrecy in the wake of World War II, when the nation assumed a more expansive role in world affairs. Subsequent chapters follow a case-study model, tracking the influence of classification imperatives on core government functions such as surveillance, diplomacy, and scientific research.
H-Diplo | Robert Jervis International Security Studies Forum
Roundtable Review 15-39
Matthew Connelly. The Declassification Engine: What History Reveals about America’s Top Secrets. New York: Pantheon Books, 2023. ISBN: 9781101871577 (hardcover, $32.50).
22 April 2024 |PDF: https://issforum.org/to/jrt15-39 | Website: rjissf.org | Twitter: @HDiplo
Editor: Diane Labrosse
Commissioning Editor: Daniel R. Hart
Production Editor: Christopher Ball
Pre-Production Copy Editor: Bethany Keenan
Contents
Introduction by Brian Hochman, Georgetown University. 2
Review by Krista Ferrante, MITRE Corporation. 5
Review by Heidi Kitrosser, Northwestern University, Pritzker School of Law.. 10
Review by John Krige, Georgia Institute of Technology, emeritus. 16
Review by Suzanne J. Piotrowski, Rutgers University. 22
Review by Aviel Roshwald, Georgetown University. 26
Response by Matthew Connelly, Columbia University. 31
Introduction by Brian Hochman, Georgetown University
Matthew Connelly’s The Declassification Engine: What History Reveals about America’s Top Secrets explores what it means for Americans to live in a society whose governing institutions are incentivized to keep secrets. The book stems from Connelly’s tenure as lead investigator at Columbia University’s History Lab, in which a team of historians and data scientists spent several years using machine-learning tools to mine an extensive repository of declassified government records. Drawing on History Lab’s findings, The Declassification Engine traces the origins of what Connelly calls the American “dark state” (19), which classifies far more information than it makes public. In the book’s opening chapters, Connelly argues that the US government established a self-perpetuating culture of secrecy in the wake of World War II, when the nation assumed a more expansive role in world affairs. Subsequent chapters follow a case-study model, tracking the influence of classification imperatives on core government functions such as surveillance, diplomacy, and scientific research.
Connelly’s book appeared just as classified documents began turning up in unexpected places. First there was the collection of top-secret files stashed in former President Donald Trump’s Mar-a-Lago club, which are now the basis for a pending criminal trial. Then, a private attorney discovered classified materials in a corporate office once occupied by President Joe Biden, and authorities soon located a cache of records in former Vice President Mike Pence’s home garage. Classified documents were everywhere—or, at least, they were everywhere they were not supposed to be. The revelations garnered national attention.
Has a work of academic scholarship enjoyed a more unexpected bump of publicity? My initial encounter with The Declassification Engine suggests not. Shortly after news of the Pence documents broke, an editor at the Washington Post sent me a harried request to review the book. Connelly’s publisher had already pitched the newspaper for advance coverage. But the accumulation of headlines now meant that The Declassification Engine was necessary reading. The situation was urgent enough for the editor to offer to drive a promotional copy to my house, with a deadline for a 1200-word review set for the following week. I opted for an electronic version and got to work.[1]
Despite the swirl of related news, what struck me upon reading The Declassification Engine was not the book’s timeliness. To be sure, Connelly offers a necessary backstory to present-day controversies. His research shows how the US government’s impulse to keep secrets has turned self-defeating, creating a universe of classified information so vast that the task of managing proprietary records has become impossible. But there is also something more subtle in the book’s pages: a slower, sadder story about the future of history. When so much of the state’s business takes place behind closed doors, all of us—citizens and scholars alike—lose the basic ability to understand the past. As classified records accumulate at exponential rates, and as government bureaucracies allow their declassification mechanisms to languish, historians face an existential challenge to their work. Computer-assisted methods offer a potential solution to the twin problems of access and scale. Yet the prospects for historical inquiry still seem dismal, as Connelly’s melancholy concluding chapter on the wholesale deletion of classified files suggests.
This roundtable looks past the headlines to highlight the broader implications of The Declassification Engine for the study of secrecy and the practice of history. Reflecting the reach of Connelly’s argument, the contributing reviewers—Krista Ferrante, Heidi Kitrosser, John Krige, Suzanne Piotrowski, and Aviel Roshwald—approach the book from a wide range of disciplinary orientations: from diplomatic history and security studies to archival science and public records administration. Taken together, they engage with two main features of Connelly’s project.
First, the reviewers evaluate the conclusions that Connelly draws from History Lab’s experiments with its aptly named Freedom of Information Archive. For Krige, The Declassification Engine offers a new look at how national security protocols have, since the 1940s, come to colonize all manner of quotidian state functions, even those that seem tangential to the project of defending the homeland from external threats. For Roshwald, Connelly’s account of pervasive bureaucratic secrecy sheds fresh light on the “dilemmas surrounding foreign-policy and military decision-making in a democracy,” although he takes issue with the book’s claims about the “original secret” (36) of America’s entry into World War II.
Second, the reviews step back to assess how The Declassification Engine might influence the study of history in the twenty-first century. Ferrante, Kitrosser, and Piotrowski all distinguish Connelly’s data-driven approach as both a political provocation and an intellectual model. In Piotriowski’s estimation, the book may well emerge as a “standard in the field,” particularly as developments in artificial intelligence offer new avenues for processing vast amounts of primary source material. Since the 1980s, archivists and historians have witnessed the US government’s gradual abandonment of the infrastructures that support the regular maintenance and release of classified information. New technologies provide renewed hope for the cause of transparency—which is to say, the cause of history. But all the reviewers remain as cautious as Connelly about the future to come. Technology is never a panacea, and there is much more work to be done.
In an extended response, Connelly’s engages with the two main threads of this dossier of reviews, answering queries about his conclusions and reflecting on the problem of history in the age of overclassification. He concludes with a call to action. It is worth reproducing his words here to establish the stakes of the conversation that follows:
While we may continue to disagree about how, specifically, to meet the immense challenge posed by the exponential growth in secret records, all of us agree that, at the very least, public officials should comply with the law. And archivists and historians are uniquely positioned to insist that, even if our leaders escape every other kind of accountability, they should at least be judged in the court of history. If that is ever to occur, we need to speak with one voice, starting now, in insisting they stop destroying the evidence.
Contributors:
Matthew Connelly is a Professor of International and Global History at Columbia University, and Director of the Centre for the Study of Existential Risk at the University of Cambridge. Connelly is also the principal investigator of History Lab, a project that uses data science to analyze state secrecy. His publications include A Diplomatic Revolution: Algeria’s Fight for Independence and the Origins of the Post-Cold War Era, Fatal Misconception: The Struggle to Control World Population, and The Declassification Engine: What History Reveals about America’s Top Secrets. Connelly received his BA from Columbia in 1990 and earned his PhD from Yale in 1997.
Brian Hochman is Hubert J. Cloke Director of American Studies and Professor of American Studies and English at Georgetown University. He is the author of two books. The most recent, The Listeners: A History of Wiretapping in the United States (Harvard University Press, 2022), traces the history of wiretapping and electronic eavesdropping from the mid-nineteenth century to the present. The research for this project received support from the National Endowment for the Humanities Public Scholars Program and the Library of Congress John W. Kluge Center, and it was named one of Publisher’s Weekly’s 20 Best Nonfiction Books of 2022. His first book, Savage Preservation: The Ethnographic Origins of Modern Media Technology (University of Minnesota Press, 2014), was a finalist for the American Studies Association’s Lora Romero Prize in 2015. His research on electronic surveillance has been featured in Humanities Magazine, IEEE Spectrum Magazine, Smithsonian Magazine, the Wall Street Journal, the Washington Post, and Wired.
Krista Ferrante is the Corporate Archivist at the MITRE Corporation, a not-for-profit dedicated to solving problems for a safer world. Through private-public partnerships and federally funded R&D centers, MITRE works across the government in partnership with industry to tackle challenges to the safety, stability, and well-being of the nation. In 2022, she co-published a white paper through the MITRE Center for Data-Driven Policy called Considerations for NARA on The Future of Recordkeeping. She is the immediate past chair of the Committee on Public Policy for the Society of American Archivists.
Heidi Kitrosser is the William W. Gurley Professor of Law at the Northwestern University Pritzker School of Law. An expert on the constitutional law of federal government secrecy and on separation of powers and free speech law more broadly, she has written, spoken, and consulted widely on these topics. Her book, Reclaiming Accountability: Transparency, Executive Power, and the US Constitution (University of Chicago Press, 2015) was awarded the 2014 IIT Chicago-Kent College of Law/Roy C. Palmer Civil Liberties Prize. She is a 2017 recipient of a Guggenheim Fellowship.
John Krige is the Kranzberg Professor Emeritus at the Georgia Institute of Technology. His research lies at the intersection of the history of science, technology and US foreign relations. His most recent publications include Knowledge Regulation and National Security in Postwar America (University of Chicago Press, 2022), co-authored with Mario Daniels, and an edited collection entitled Knowledge Flows in a Global Age. A Transnational Approach (University of Chicago Press, 2022).
Suzanne J. Piotrowski is a Professor of Public Affairs and Administration at Rutgers University-Newark and Director of the Transparency and Governance Center. Dr. Piotrowski researches freedom of information, transparency and open government issues with a strong focus on connecting with communities of practice. In 2018, Professor Piotrowski was appointed by the Archivist of the United States to serve on the Freedom of Information Act Advisory Committee for a two-year term. She served as the independent assessor of the United States national action plan for the Open Government Partnership. She founded the Global Conference on Transparency Research (Rutgers University-Newark, May 2011). In 2022, MIT Press’ Information Policy Series published her coauthored project The Power of Partnership in Open Government? Multistakeholder Governance Reform and the Open Government Partnership.
Aviel Roshwald is Professor of History at Georgetown University, Washington, DC, where he has taught since 1991. He obtained his PhD at Harvard University in 1987. He is the author of the following books: Occupied: European and Asian Responses to Axis Conquest, 1937–1945 (Cambridge University Press, 2023); The Endurance of Nationalism: Ancient Roots and Modern Dilemmas (Cambridge University Press, 2006); Ethnic Nationalism and the Fall of Empires: Central Europe, Russia and the Middle East, 1914–1923 (Routledge, 2001); Estranged Bedfellows: Britain and France in the Middle East during the Second World War (Oxford University Press, 1990). He is the co-editor, with Richard Stites, of European Culture in the Great War: The Arts, Entertainment, and Propaganda, 1914–1918 (Cambridge University Press, 1999). With Matthew D’Auria and Cathie Carmichael, he is a co-editor of the two-volume Cambridge History of Nationhood and Nationalism (to be published by Cambridge University Press by the end of 2023).
Review by Krista Ferrante, MITRE Corporation
The dynamics of power and knowledge are central to the conception of political thought. Knowledge and intentional ignorance are powerful mechanisms. In the years since the invention of the nuclear bomb, the stakes of political power have never been higher. The systems and bureaucracy around protecting the national secrets of the United States have grown increasingly complex as regulations and laws are layered on top of each other. As sociologist Lukas Griessl posits, “Ignorance can become a powerful and productive force that brings about new forms of ignorance and secrecy.”[2]
In his new book, The Declassification Engine: What History Reveals About America’s Top Secrets, Matthew Connelly writes about his adventures in building an artificial intelligence (AI) tool to try to crack the secrets of the United States government. The tool known as “The Declassification Engine” uses machine-learning techniques and a band of teaching assistants at Columbia University’s History Lab to analyze leaked and declassified documents. Statistics and probability offer one way to understand the unknown by using equations to predict the likelihood of something occurring. Using natural-language processing and machine-learning algorithms, Connelly has pieced together unknown stories of government programs. The computers analyzed hundreds of unredacted and redacted documents and ranked them based on the use of certain words that were classified in unredacted texts in order to locate the most sensitive documents. Connelly did this with as much government approval as he could gain. His team needed to sign non-disclosure agreements similar to the lifelong agreements that cleared individuals must make with the United States of America. Several agencies have shown interest in the tool, and in this book, Connelly shares the reaction of government officials to his results.
The calls for techniques and methods to classify and declassify information like the Declassification Engine have become louder in recent years. In a May 2021 letter to President Joseph Biden, the Public Interest Declassification Board asserted that the “classification and declassification system is in crisis and near failure… [and] at a breaking point.”[3] Director of National Intelligence Avril Haines recently noted that “deficiencies in the current classification system undermine our national security, as well as critical democratic objectives, by impeding our ability to share information in a timely manner.”[4] Connelly is correct in his assessment that “the government itself desperately needs to automate and accelerate archiving and declassification” (397).
The challenges of classification are myriad, and the current system is not scalable. In fact, there are so many people with a clearance that Congress does not even know how to count them.[5] Cleared individuals with eligibility who do not even work with classified information are cleared just in case they gain access in order to protect against future liability, a phenomenon that sociologist Linsey McGoey refers to as a type of “strategic ignorance… [to] deny liability in the aftermath of disaster.”[6] Even less is known about the declassification officers who are tasked to take on the Sisyphean task of reviewing the mountains of classified information page by page. The primary recommendation in a 2022 MITRE paper on how to better understand the system was that the relevant agencies report on the number, grade, experience, and cost of declassification officers.[7]
Declassification is the exclusive purview of the creating agencies. Though Connelly is correct that the Department of Energy (DOE) requires its own reviews (92), it is more complicated than that. Every classifying agency, not just the DOE, requires its own declassification reviews and has its own declassification office. Furthermore, the rules for what is deemed classified and what is deemed unclassified are spelled out in Security Classification Guides (SCG), which are themselves classified. The SCGs are jealously guarded even between agency declassification offices, which claim that the other agency does not have a need to know what is or is not classified by another. In 2022, through the Information Security Oversight Office (ISOO) at the National Archives and Records Administration (NARA), the Public Interest Declassification Board (PIDB)[8] prepared a feasibility study to examine what it would take to review and declassify information related to the testing of nuclear weapons on the Marshall Islands.[9] They found that it would be feasible to conduct a “topical-based declassification project” at a cost of $100-200 million. Importantly, a declassification project by topic and not by agency would require multiple interagency agreements to make it possible. The agencies would need to agree to share the relevant SCGs. They would also need to agree to what is restricted but unclassified across agencies, and they would need to agree on the logistics on where to do the reviews and how. The study shows that it is feasible, but has many hurdles to overcome.
Connelly’s exploration of why records are still protected by classification also looks at the roles played by scholars, archivists, and other agency officials. He asserts that scholars have “never stopped pushing for timely release of state secrets” (394). Archivists are a natural ally in the pursuit of transparency. Yet he derides the role of government archivists by stating that “the archivists’ creed can therefore conceal darker arts. When all is said and done, government archivists are functionaries of the state” (351). The code of ethics of the Society of American Archivists states that “[a]rchivists endeavor to ensure that materials entrusted to their care will be accessible over time.”[10] Archivists periodically are in a situation where they cannot provide information to researchers. The archival content might be protected by laws such as the Family Educational Rights and Privacy Act (FERPA),[11] which protects private student information, or the content might be subject to the legal terms of a collecting agreement, where a donor restricts access for a predetermined number of years. Ultimately, the goal is always to protect and hold high-value information sources so they can eventually be made available in time.
Though archivists are functionaries of the institutions that employ them, they are also in a unique position to promote transparency. Recent efforts to recover and better describe the archival collection materials of underrepresented communities show a commitment to push the envelope on what information can and should be revealed and how to do so.[12] Archivists are champions for openness, so long as it falls within the limits of the governance boundaries of the information they release. It can take a very long time before information can be released, sometimes as long as 80 to 100 years. But it is the archive that will remain committed to protecting the materials during that time, document the releasability details through generations of archivists, and then follow through on the day that it can be made available for researchers.
Archives also have an equal obligation to reject materials. Archivists appraise materials for future research value, and if something can never be released for research, there is no point in spending the resources to retain it indefinitely. NARA will no longer take classified permanent records from agencies, and agencies are required to declassify documents before they are transferred to the National Archives.[13] The National Archives do not possess the resources to take on the responsibility of caring for a backlog of records to be declassified that may never see the light of day. Archivists may also reject materials because they hold no future research value. In the federal government these are known as temporary records. NARA sets the schedules or rules for what agencies should designate as permanent records to send to NARA, and which records are temporary. These rules are outlined in the general records schedules (GRS) and each time a new GRS is generated, it is open for public comment through the Federal Register.[14] In other words, the rules around agency records are not a state secret. Rather, there is a great deal of transparency on how record policy is set and there are opportunities for the American public to comment.
Historians who use primary sources as data points might argue that every piece of documentation about government business should be kept. But keeping everything produced comes at a tremendous cost. The United States federal government is the largest employer in the country, with around 2.2 million civilian employees and another 2.2 million employees in the military.[15] This does not include the millions more who work as contractors to the federal government. Connelly refers repeatedly to “The Government” as if it were a faceless monolith, but there are millions of individual Americans who work as federal employees, individuals who we can assume are doing their best. The government as a monolith does not exist and it is not a decisionmaking body on to itself. Rather it is a vastly complex organization with over 400 agencies and sub-agencies.[16] What would it entail to retain the emails and draft working documents of the entire federal workforce? How much storage would be needed? Who would that benefit? It neither benefits the agencies nor the American public to keep petabytes of data forever. Keeping everything and destroying nothing would only benefit data historians and would be an example of socialized cost for privatized gain. Luckily, there are clear rules for federal employees about what they can destroy in the form of temporary records.[17]
In chapter 9, “There is No There There,” Connelly discusses notorious leaks of documents like WikiLeaks and the Pentagon Papers (316-346). He concludes that since much of the information exposed was already available in the news media, it is possible that it should not be considered secret, but this is not how the system works. The availability of information elsewhere does not automatically make it public, nor does it make it declassified. In fact, cleared individuals are barred from even reading leaked documents because they do not have a need to know that leaked information. The context of the information is also important to consider. Often it is the analysis of multiple public sources that makes it relevant for how it is classified. For example, an open library collection is open to everyone, but it is the contextual analysis and gathering by a scholar that makes it something relevant and marketable. The same holds for intelligence analysis. It is secret because of how it was compiled. The fact that pieces and parts of the information can be available publicly elsewhere is expected and irrelevant.
Second, the inconsistencies in the application of classified restrictions have come to be expected because of the siloed SCGs across agencies. While Connelly finds the inconsistencies to be evidence that secrecy can be used to cover up misdeeds or obscure an embarrassing moment, the inconsistencies actually reveal the human flaws in a complicated system that spans a very large organization. When government is solely viewed through its most notorious leaks and failures, then a logical conclusion is that the point of classification is to conceal incompetence. However, it is unlikely that a secret government program that worked reliably, efficiently, and protected millions of lives without incident would garner headlines because it is not newsworthy. It is clear that that government secrecy has been misused, but there are many more untold stories about the public good done and average daily work well executed under the so-called “veil of secrecy.”
Furthermore, because a document was classified at a certain time does not mean that it has enduring importance in the future. In other words, there can be temporary classified records. It is useful to consider private information like bank passwords. Certainly, the integrity of personal assets relies on the ability to protect passwords. But what about bank passwords from accounts closed five years ago? This obsolete information does not have enduring value. Following this logic, it is evident that not everything that has been classified needs to be declassified. I think for declassification to be scalable and most relevant to the American public, topical-based declassification projects should become the norm, not the exception. Separate grants or defense appropriations could be allocated for specific projects like the Marshall Islands nuclear testing case. In my opinion, topical-based declassification could broadly appeal to the American public. Impacted communities could learn the full breadth of information in a timelier manner. It would give the American public better insight and congress more input into the priorities of the agency declassification offices. In addition, topical-based declassification could enable the public to learn as much as possible about a subject from across the government, instead of only getting one agency’s perspective and would give a fuller picture of the context surrounding individual documents or data points. On the other side of the time scale, it could lead to improved appraisal of records at the time of creation. For example, one could deduce now that records created around the US involvement in Ukraine will be important in the future and steps could be taken to preserve and protect those now for future public consumption. It would be prudent for the government to strategically consider information that is created during important campaigns as permanent records at the time of creation and to find ways to designate those documents for eventual declassification and transfer to NARA.
Archivists have a professional calling to promote the transparency and openness of the information in their care.[18] Government agency archivists, records managers, historical research offices, and NARA staff, all of whom are members of what Connelly calls the “dark state” (374), are the ones who are pushing against the tide of secrecy. But as Connelly correctly points out, “the Archives [NARA] have been so neglected and underfunded that staff no longer have the technical capacity to recover email of our forty-first president. So we can’t handle the truth, quite literally, because the government is losing its capacity to hand over historical records” (59). Archivists and historical scholars need to continue to work together in order to build awareness and promote government transparency.
Review by Heidi Kitrosser, Northwestern University, Pritzker School of Law
At the heart of Matthew Connelly’s excellent and important new book, The Declassification Engine: What History Reveals about America’s Top Secrets, are the deep ties between US history and government secrecy. These ties fall into two categories. First, national security secrecy, particularly the growth of a behemothic classification system over the twentieth and twenty-first centuries, is an integral part of American history. Second, the ability to discover, write, and teach about history depends, in a substantial part, on the creation, preservation, and availability of government records. The existence of such records, let alone historians’ ability to access them, faces growing threats from an increasingly ungovernable secrecy system.
Connelly’s passion for these issues leaps off the book’s pages, and the sentiment is well and hard earned. Connelly is a historian, and he also founded and runs the History Lab at Columbia University (xv-xvi).[19] The Lab created the book’s namesake Declassification Engine—a platform comprised of high-powered computers and sophisticated algorithms—to data-mine millions of declassified government documents. (To avoid confusion, hereinafter the Lab’s engine will be referred to as “the Engine” and its namesake book as The Declassification Engine.) The generated computational outputs are analyzed by historians and national security experts to detect and evaluate patterns in classification and declassification decisions. The ultimate goal, according to Connelly, is to “train algorithms to look for sensitive records requiring the closest scrutiny and accelerate the release of everything else” (x).
This essay is divided into three topics: Secrecy’s History, The End of History, and The Future of History. Secrecy’s History refers to the book’s uncovering of key historical episodes and insights involving national security secrecy. The End of History entails Connelly’s discussion of the ways in which the secrecy system undermines the ability to preserve and to study history. The Future of History focuses on the book’s conveyance of hope that there may be a bright future for the enterprise of history, and what must be done to realize that promise.
In the book, Connelly uses Engine-driven discoveries, combined with “old-school archival spadework” and a “close reading of key documents,” (33) to fill the historical lacunae. Much of the uncovered information involves national security secrecy, which makes for some fascinating, layered stories and insights. In a chapter on Pearl Harbor, for example, Connelly asks how “can we explain why American forces in Pearl Harbor were so completely surprised” by the attack? (53). A major factor, he concludes, was the failure of information sharing. In a misplaced effort to protect national security, the military kept a number of intercepted Japanese communications from President Franklin D. Roosevelt, and also failed to share them with “the commander of the US Pacific Fleet…or the general in charge of defending Pearl Harbor” (46, 53-54). Following the attack, records that might have revealed this failure were covered up for decades (55). Connelly posits that “[w]hen government officials realize they have been incompetent, they treat the potential revelation of that fact as a grave danger to national security” (55). Officials have an additional incentive to pretend that an attack could not have been anticipated: a surprise attack can justify more and bigger secret programs. After World War II, “the military insisted on the need to spend heavily on complex, redundant, and highly classified systems that would supposedly make any new surprise attack impossible” (55).
Among the themes that recur is the self-defeating nature of excessive secrecy and compartmentalization. This is central, of course, to the story of Pearl Harbor. The same point is evident in Connelly’s intriguing account of how accidental transparency might have kept the US safe from attack by the Soviet Union during the height of the Cold War. The unintentional openness stemmed from massive security breaches that resulted, among other things, in Soviet officials’ ability to read “most if not all encrypted communications in and out of the US Embassy in Moscow from 1953 to 1964” (127). This security failure might have prevented a Soviet nuclear strike. As Connelly explains, Soviet leaders “knew that they would have timely warning of any American attack, and no need to launch a pre-emptive attack of their own” (128).
Excessive secrecy also can be self-defeating in the sense famously observed by Supreme Court Justice Potter Stewart in The New York Times Company v. United States (i.e., the “Pentagon Papers case”): “[W]hen everything is classified, nothing is classified.”[20] Truly sensitive secrets are simply harder to protect when the secrecy system is stretched thin, and elaborate secret-keeping measures can themselves generate breaches. Among the more striking examples of these phenomena in The Declassification Engine are the multiple points of porousness created when the National Security Agency (NSA) consolidated its headquarters in Fort Meade, Maryland in the late 1950s (106-108). Pressure to fill positions at the fast-growing agency was such that the NSA often did not investigate the information that it gathered on prospective employees (115-116). Connelly notes that “[i]n fact, this background screening itself created another vulnerability: The NSA Security Section was systematically gathering incriminating information about every NSA employee in one place—a place with the lowest level of security in the building” (116).
Elaborate systems of secrecy also are far from immune from racial and other biases, and this too can undermine national security. Connelly writes of acts of sabotage within the NSA that went undetected for years, despite the red flags in the saboteurs’ lives, including drug and alcohol abuse and profligate spending well beyond their means. He posits that “men who conformed to a certain version of normality appeared dependable… On the other hand, when the NSA finally tried to diversify its ranks—it was still 89 percent white in 1993—longtime employees claimed that the new Black, Hispanic, and Asian hires posed a security risk” (125).
Connelly reserves his starkest warnings for the classification system’s impact on our ability to preserve history. In the post-World War II rise of what he calls “the dark state” (6) and defines as a permanent, leviathan-like national security state steeped in secrecy, including a wildly bloated classification system, Connelly sees the potential “end of history” (377). By history’s end, Connelly means the decimation of meaningful record-keeping or access to records on all matters remotely related to US national security.
Connelly looks at multiple aspects of history’s end. Statistics give some sense of the classification system’s breadth and of how well overmatched current declassification tools are to their task. Connelly reveals that today, “twenty-seven million cubic feet of federal records await review and processing” for declassification—“almost twice as many as in 1985”—while the budget of the National Archives, which is tasked with managing declassification, has been steadily shrinking (361). Electronic recordkeeping, which was once thought to be the magic bullet to improve declassification and archival preservation more broadly,[21] has turned out to be a substantial part of the problem as electronic records have become unfathomably voluminous. “[W]hereas there were thirteen million electronic records in 1991, now there are 21.5 billion, with two billion more every year” (361). Indeed, the Information Security Oversight Office (ISOO), an office within the National Archives that is responsible for yearly reporting on the state of the classification system, has capitulated on compiling recent classification and declassification statistics. In gathering statistics from agencies to create the 2018 report, ISOO concluded that “the figures failed to ‘reflect how agencies are actually operating in the digital environment.’ They were not even worth compiling” (387). The public has no current information “about the number of times officials classified information, no data on the number of pages declassified, and no clue how much the government has been spending on secrecy” (387).
More alarming is the fact that many records have simply disappeared. The culprit is often the same neglect and lack of resources that plague declassification efforts. In 2007, for example, the National Archives concluded that it possessed too many “Central Foreign Policy Files” to review for page-by-page processing. They chose instead to delete entire groups of documents based on the reviews of unrepresentative samples. In some cases, they destroyed entire subject categories based on presumed historical insignificance (361-362).
The decision to erase whole categories of information due to their relative historical insignificance—a decision borne out of desperation given understaffing and an impossible workload—bespeaks a myopia that disserves the enterprise of history. Connelly writes that “with millions of cables on passports and visas”—all of which the archives deleted—“researchers might have been able to develop a vastly more sophisticated understanding of global migration. This opportunity is now lost forever” (362). Connelly also notes that “[d]eleting immigration records…means deleting the stories just one such record might tell about a whole family’s history” (362). He illustrates the latter point with a chilling example that is worth quoting at length, as it demonstrates his skill in melding individual historical accounts with broader, even meta, observations:
[W]e still have some of the private letters of one of the greatest historians of the twentieth century, Marc Bloch. That is how we know that, in December 1940, he was offered an American visa that might have allowed him to escape occupied France, and the Holocaust. But he turned it down because he could not get visas for his mother and two grown children. He therefore decided to stay, and joined the Resistance. He was finally captured and executed in 1944, just before he could complete a remarkable masterpiece, The Historian’s Craft, which was to include a chapter on the history of the future. How many stories, how much future history, might be lost and impossible to recover because the National Archives are not preserving the records? (362-363)
In other cases, records are destroyed or hidden to hide mistakes or abuses. For example, Lt. Col. Oliver North of the National Security Council and National Security Advisor John Poindexter infamously deleted and shredded thousands of documents to cover their tracks in the Iran-Contra affair (357-358).[22] Connelly also cites suspicious gaps among the diplomatic cables archived by the “government’s first ‘Automated Data System’ to store textual records” (353). “Most of the missing cables,” whose absence “the State Department has never been able to explain,” do not date to the system’s very beginning, when one might expect technical problems to distort the record (354). Rather, “they date to 1975-76, when then Secretary of State Henry Kissinger was coming under increasing criticism for amoral conduct. Intriguingly, the cables missing from the database tend to be more highly classified, and often involve Kissinger and his more senior staff” (354-355).
Though the cover-ups involving high-level officials garner more headlines, it is the institutional factors that support secrecy, including neglect, mismanagement, and under-resourcing, that are more determinative. But there are other features that tend to make a large and elaborate secrecy system attractive to those within the national security establishment. This includes the psychological appeal of exclusivity. A quotation from long-time Central Intelligence Agency (CIA) officer Gail Donnalley is especially evocative of this factor. Donnalley reported that a 1974 executive order requiring the CIA to review records for declassification had a “‘traumatic effect’ on rank-and-file intelligence officers” (334). On practical level, a bloated secrecy system empowers those who are “in the club” to shape what the public knows, both by hiding information and by releasing information selectively and strategically. Connelly devotes an entire chapter to the phenomenon of “spin” enabled by leaking (284-315), but perhaps the most striking example of this phenomenon is detailed earlier in the book, in a chapter on the Manhattan Project.[23] US Army Lt. General Leslie Groves directed the project and its massive production of secret information, but gave little thought to how that information would be managed over time—how and when declassification decisions might be made, for instance, and how records would eventually be archived (72). “Rather than setting in motion an actual system for declassifying all information that was no longer sensitive,” Connelly writes, “Groves and his team prepared a package of press releases, authorized stories, and official reports. It would drop right after the atomic bomb dropped on Hiroshima” (72). “’Publicity Day’” turned out to be a “great success” (72). After all, “[w]ith no other information available—and official notification that nothing more could be revealed about the bomb—the media and the public eagerly consumed the stories crafted by the Manhattan Project PR team” (72-73).
Racism and sexism also impact who is afforded access to official records, and thus with the ability to impact what, and whose, histories will be written. Of the 89 professional staff members hired in 1937 by the National Archives’ first director, “when the National Archives was still exempt from having to fill positions through competitive civil service exams[,] there was only a single woman,” and no Black employees (24). The second archivist insisted that the Archives “had to destroy historically insignificant materials to make room for what really mattered” (24). As a result, Connelly writes, “those who went to archives decades later in search of records that would recapture the experience of less powerful people and marginalized communities would find only ‘silences, erasures, and distortions’” (25).
For all of its well-supported warnings about history’s end, The Declassification Engine does not run solely on despair. Connelly tells us that history still has a shot at a future. His optimism stems partly from a look back at America’s distant past. The Declassification Engine’s very first chapter, “The Radical Transparency of the American Republic” (3-35), serves as a useful reminder that much of the secrecy infrastructure that seems inevitable today is relatively new. Even in the midst of the Civil War, “Congress directed the government to begin compiling a complete historical record of wartime decision making…It was published starting fifteen years later, and eventually totaled hundreds of thousands of pages” (17). This contrasts sharply with the present, in which the declassification process is so broken that, if current trends continue, historians in 2060 “will have just begun to have access to the full historical record of the immediate post-Cold War era” (396). The existence of a permanent classification system marks another sharp divide between past and present. The first government-wide classification system was not created until 1940, on the eve of America’s entry into World War II (25). When President Harry S Truman re-established the classification system with a new executive order in 1951, the move was highly controversial, with “[e]ditors’ associations call[ing] it a ‘dangerous instrument of news suppression’” (290).
If there is any chance to recapture the “radical transparency” of America’s past, Connelly believes that it lies partly in the technology of the present and future. Throughout The Declassification Engine, he reveals what technology, coupled with close readings by experts, exposes about the history of secrecy. In his concluding chapter, Connelly reflects more broadly on what the History Lab and other researchers can accomplish by harnessing technology. He asks rhetorically: “What future is there for official secrets when we can create a virtual archive of millions of formerly secret documents and start mining it for information?” (399).
But Connelly is no Pollyana, and he is mindful that whatever technology can accomplish, there remain serious political and legal challenges (399). Indeed, he has seen both types of challenges up close. With respect to political hurdles, he recounts the mixed results that he and his colleagues experienced when they traveled to Washington, D.C. to share declassification technology with several agencies including the State Department, the National Declassification Center, the CIA, the Public Interest Declassification Board, and the Office of the Director of National Intelligence (ODNI) (378). Although many of the officials were enthusiastic—a State Department official, for example, told Connelly’s group that “the need for this kind of technology was ‘frighteningly clear’” (379)—lack of funding and other logistical hurdles precluded them from creating partnerships”(378-380). The most disappointing, and most telling, meeting took place at “an agency known as the Intelligence Advanced Research Projects Activity (IARPA), which had been tasked to work with the National Archives to develop new technology to deal with the declassification problem” (380). There, Connelly’s group was informed that IARPA simply was not interested in declassification technology, as “it would bring an ‘insufficient return on investment’” (385). In other words, Connelly realized, “we cannot assign a dollar value to democratic accountability” (385).
For History Lab members and others who aspire to work on technology like the Engine, the looming hurdles are not only political, but legal as well. Connelly begins the book’s preface by sharing his own stark encounter with this reality. A foundation to which he initially applied for a grant to pursue the Engine would fund the project only under stultifying conditions—including life-long confidentiality agreements—because it feared that the work might run afoul of the 1917 Espionage Act, presumably by uncovering still-classified information (ix, xiv-xv).[24] Although the project commenced due to the support of the MacArthur Foundation and Columbia University, Connelly viewed the incident as demonstrative of a recalcitrant system of official secrecy: “Even in the halls of academia, and inside elite foundations, people have come to fear prosecution just for doing research on state secrecy….[E]ven the perception of official displeasure can be enough to drive them away from this kind of work” (xv).
The 1917 Espionage Act is the tip of the enormous iceberg that is national security secrecy, but that tip is dangerously sharp. Over the past two decades, the act has increasingly been used to target media sources. Indeed, the record of President Donald J. Trump’s administration of prosecuting sources under the act discredits Trump’s complaint that his own recent indictment under the act constitutes political persecution. More important, “the gulf between Trump’s alleged behavior and that of the media sources prosecuted by his administration illustrates the extraordinary breadth of the act.”[25] The relevant provisions of the act “effectively draw…no distinction between a spy, a whistleblower, and a former president who hoards national security information for his own amusement or private gain.”[26] And now, as Connelly’s experience shows, historians may be deterred from doing work like the History Lab’s for fear that they too could be targeted.
History’s future thus depends in part on persuading today’s leaders that radical changes—at minimum, to the classification system and to laws that make it easy to prosecute the publication of closely held information, even when publication serves the public interest—are necessary and desirable. Connelly shows that, while “[s]ome officials want to destroy history; others just don’t seem to care” (371). My own sense is that, while there will always be those who wish to destroy history, the most formidable problem is apathy; the sense, as Connelly was told at IARPA, that transparency brings “an ‘insufficient return on investment’” (384). Such sentiments can only be surmounted with an alternative vision, one that understands that the costs to excessive secrecy can be every bit as great, if not greater, than those of leaked secrets. The Declassification Engine goes a long way toward advancing this vision. It does its part, in short, to nudge us away from the darkness, toward a future filled with rich and illuminating histories.
Review by John Krige, Georgia Institute of Technology, emeritus
In April 2023, the New York Times reported that over the course of several months, hundreds of pages of operational briefs compiled by the Pentagon’s Joint Staff that used reports from the government’s intelligence community had been posted on a social media platform called Discord. The reports were later posted on Twitter and Telegram.[27] The task of finding the source of this major intelligence breach promised to be difficult: one senior US official claimed that hundreds, if not thousands, of military and other American government officials had the security clearances that were needed to gain access to the documents. The leaked material revealed US intelligence on Ukrainian military preparedness, as well battlefield commands from the Kremlin. It also divulged sensitive briefing material on several US allies, including Canada, Israel and South Korea, notably a domestic debate on whether Seoul should provide ammunition shells for the United States’ ongoing support for Kiev.[28]
It did not take long, however, for the Federal Bureau of Investigation (FBI) to arrest a prime suspect. Discord was popular with video gamers, and was extensively used by a 21-year old Massachusetts Air National Guardsman, Jack Teixeira.[29] Teixeira was indicted by a federal grand jury on June 21 on six counts of retaining and transmitting classified national defense information, and accused of retrieving top secret information from computers at an intelligence unit at the Cape Cod Air Base. He had escaped detection for months, even while taking notes and conducting deep-dive searches for classified material and posting information online under the noses of his superiors.[30]
The Teixeira case is a timely entrée to some of the key themes developed in Matthew Connelly’s new and important book. It is a peephole through the curtain that shrouds what he calls the “dark state” (6) of secrecy, a regime produced by government officials who classify documents and other official material, often independently of each other and using different and sometimes contradictory criteria, and of whom some 1.3 million have top secret security clearance. For Teixeira to have gained access to the category of very “sensitive compartmentalized information,” he probably had first to prove “unquestionable loyalty to the United States … strength of character, reliability, judgment, and trustworthiness,” before being “indoctrinated” into how to handle different kinds of information (317). Once having crossed that line, recruits “typically” find that “there is no there there,” that “‘secret intelligence’ is not very secret, and much of what remains secret is not very intelligent” (319).
Indeed, the classified numbers Teixeira posted for Russian and Ukrainian casualties were within the margin of error from numbers available in the public domain. His prime motive for leaking sensitive information seems to have been to enhance his prestige among his fellow chatroom members, the prestige that came from having access to the “vast vacuous expanse” (319) of classified material, relishing in “the power of secrecy and the secrecy of power” (319). It is not clear why his superiors turned a blind eye to what he was posting. Some of the content was, after all, highly sensitive, not because of the raw data that it revealed but because it showed that America was both spying on its allies and had penetrated the Russian intelligence system (173). Were some of these officials using the “weaponization of secrecy and leaks to win political battles,” (150) by undermining President Joseph Biden’s ongoing support for Ukrainian President Volodymyr Zelensky with a warning to the Russians that their intelligence agencies were compromised, and that they should no longer post sensitive information on the war? We may never know, though, as Connelly warns us, this genre of conspiracy theory readily flourishes in the absence of government transparency.
My reading of Teixeira’s behavior is a takeaway from the extensive research by Connelly and his History Lab at Columbia Library—“a team of data scientists and social scientists dedicated to exploring the past in order to discover lessons for the present and the future” (xvi). Analyzing “the world’s largest data base of declassified documents,” using high speed computers coupled with artificial intelligence (AI) and sophisticated algorithms (a “declassification engine”) they sought to establish “what the government did not want us to know, and why they did not want us to know it” (xvi). The team’s aim was not simply to hold the government accountable for not respecting traditional values of American freedom and openness by shielding an ever-increasing and uncontrollable volume of putatively “sensitive knowledge” behind high walls, if not simply destroying it altogether (between 1947 and 1957 “the government created some seven hundred million pages of classified material” [294]). Rather, the History Lab wanted to see if there were patterns in classification choices in the past that could be used to shape future decisions about what information and knowledge needed to be kept secret in the name of national security, and what could be safely released or erased.
This is both an intellectual and a political project. Current indiscriminate and opaque policies for defining secrets “pose a manifold threat to both democracy and national security, which must be the first and main focus of this book. But over the longer term,” Connelly insists, “the stakes include the future of history itself” (7). Billions of government paper and electronic records remain unprocessed. In 2012, for example, one intelligence agency was producing a petabyte of classified data every eighteen months (30). If no systematic way is found to store and catalog this information, it will become impossible for future historians to produce reliable, empirically grounded accounts of the past. The National Archives will no longer even accept paper records. The vast trove of secret documents still held by the dark state may even be destroyed unless digitized intelligently (370). Connelly’s hope is that his declassification engine will harness modern technology and AI with data science to select a reliable and representative set of archival records, a memory of the past for future historians.
The task is formidable, not only because of the sheer volume of secrets that are produced but also because of the circumstances leading to their withdrawal from the public domain in the first place. The analyses described in his book reveal that “secrecy was never just about national security. It served the interests of people who wanted to avoid democratic accountability” (7) and it “goes all the way to the top” (6). Beginning with President Harry Truman, and persisting with all his successors without exception (though with some nuances), US presidents have successfully used executive orders to resist “congressional oversight or judicial review in deciding what is secret and what can be revealed” (6). Many presidents have entered office determined to oppose undue classification. President Jimmy Carter instructed officials that information was not to be classified unless its “unauthorized disclosure reasonably could be expected to cause at least identifiable damage to the national security.”[31] Presidents Bill Clinton and Barack Obama expressed much the same view, using much of the same language. Notwithstanding this instruction, the amount of classified material increased steadily year after year during their respective times in office, with Obama “really out[doing] his predecessors” (308). Although his executive order on national security information in 2009 was a “greatest-hits list from previous orders,” new government secrets reached more than 95 million in 2012, his Justice Department “prosecuted more people for unauthorized leaks than all his predecessors put together,” and journalists reported that their “sources were terrified to talk about anything related to national security for fear of prosecution” (307-308). Even though every president realizes that the dark state will pay little attention to demands to relax classification, “upon gaining executive power, [they] recognize the political capital that they can gain from defining what is secret while maintaining the appearance of transparency” (296). Every president is ultimately obliged to divulge secrets of their choosing to advance their agendas, while deflecting attention away from what is classified. As Connelly puts it, “leaked information is the lubricant that allows [the White House] to spin” (311).
Historically this has not always been the case. The turning point came in the 1940s. President Franklin Roosevelt’s narrative that the assault on Pearl Harbor was unexpected and unprovoked produced a morbid fear of surprise attack among the American people. The development of lethal nuclear weapons by Soviet Communism heightened their fears of vulnerability. Their anxiety could have been handled differently. Roosevelt chose not to reveal deliberate attempts to provoke the Japanese so as to justify the US entering World War Two. At the last minute, the Atomic Energy Act of 1946 shrouded the nuclear program in secrecy—against the wish of many scientists who pleaded for openness and international cooperation—so as to secure the United States’ monopoly on nuclear weapons for as long as possible. A dark state operating behind a veil of secrecy and with ever more expansive powers was put in place “not only to protect ‘national security’, but also, and especially, the security of particular people who were jealous of their privileges” (30).
Connelly’s strategy in the book is to use his declassification engine to throw new light on the operation of the dark state behind the public face of presidential declarations, government agencies, and private economic actors. The first and possibly the most dramatic concerns the “surprise” bombing of Pearl Harbor. During the period leading up to the attack, “the American public was totally unaware of crucial facts about US-Japan diplomacy,” Connelly avers. Contrary to officially sanctioned accounts published after the war had ended,[32] Roosevelt made efforts to provoke Japan into giving him a reason that would overwhelm the “isolationist” objections to the nation’s going to war (39). Obviously, the president never imagined anything like what happened. However, after analyzing about 765,000 pages of declassified documents at all the presidential libraries since the 1970s, Connelly’s team came across a stunning memo, dated 26 June 1954. It was buried deep in the Dwight D. Eisenhower Library in Abilene, Kansas, whose full contents were only accessible in 2002. In the memo, Henry Cabot Lodge, Jr. (then the American Ambassador to the United Nations) recounted what British Prime Minister Winston Churchill told him about the reactions of his American guests to the news of the Japanese attacks at a dinner party in his country house on 7 December 1941. In Lodge’s telling, Churchill said that when the news broke, the American Ambassador to the United Kingdom, John Winant, and President Roosevelt’s personal envoy to Europe, W. Averell Harriman, “got up and embraced each other and danced around the room in delight,” even while the base at Pearl Harbor was a smoldering ruin and the sea was awash with dead (40). The intensity of the emotions displayed by the US officials highlights just how misleading Roosevelt’s claim was to Congress and the American people that a “treacherous” enemy had made an “unprovoked and dastardly attack” on the United States (57)—and heightens one’s indignation at the subsequent surge in racist hostility throughout the country, underscored by Roosevelt’s Executive Order 9066 of February 1942, (i.e., a couple of months later) that resulted in the incarceration of thousands of Japanese Americans during World War II.[33] If he had been more frank with the American people, they might not have tolerated this policy.
The next chapter on the atomic bomb describes how the declassification engine is being used to help the government declassify (and sometimes reclassify) millions of nuclear secrets rather than simply abandoning the project altogether for lack of funds and personnel. The task facing the Department of Energy (DoE) is colossal, and the risk that sensitive nuclear information may be published online is of great concern. Using a variety of approaches, and working with the DoE, Connelly’s History Lab has made considerable progress in helping to draw the line between the public’s need to know and the government’s need to protect particular nuclear secrets—and in showing how that line has moved over time, and whose interests it has sometimes served.
Connelly and his colleagues also developed smart techniques to gain insight into the extent of government surveillance of private citizens in the United States. The History Lab approached this issue indirectly by exploiting the metadata extracted from classified cables that monitored political dissent in foreign countries. Starting with the Iranian revolution of 1978–1979, the study was extended to an analysis of 1.7 million cables from 1973 to 1977. This gave an indication of just what information the government deemed particularly sensitive and that remained classified for extended periods of time. The top hit emerged to be a classified surveillance program put in place after affiliates of the Palestinian militant group Black September murdered Israeli athletes at the Munich Olympics in 1972. Further study led Connelly’s team to conclude that secret US surveillance programs are alive and well and tend to focus on marginalized communities, a finding that he correlates with the lack of diversity in the FBI and the National Security Agency (NSA). Institutional homogeneity (87 percent of NSA employees in 1996 were white [206]), fosters distrust and suspicion of the unknown “other” and shapes surveillance priorities. When one realizes how little is known about surveillance that is “authorized in secret courts that issue secret rulings, or by White House lawyers with no judicial or Congressional oversight,” one cannot help but wonder about the breadth and depth of the warrantless surveillance of private citizens (209).
In chapter Seven, Connelly and the History Lab examine the role of business in defining the priorities of American foreign policy. They take their cue from the observation by an ex-CIA agent, Philip Agee, “that US national security, as preached by US leaders, is the security of the capitalist class in the US, not the security of the rest of the people—certainly not the security of the poor, except by way of reinforcing poverty” (247). So, is the business of America business? Are intelligence agencies involved in “dirty” secret missions abroad predominantly to secure the fortunes of the “capitalist class”? To help answer this question, the team developed techniques to analyze the online archive of the State Department’s Foreign Relations of the United States (FRUS) between 1932 and 1980.[34] Connelly’s major conclusion is that private interests were deeply involved in illegal operations abroad, and that there were systematic efforts to cover them up for decades afterwards (272). However, there is little evidence to suggest that either Democratic or Republican administrations actively colluded in these clandestine activities: “business promotion and economic diplomacy” have been prioritized by both, but “for the most part that agenda is not hidden” (279).
About twenty years ago historian of science Peter Galison wrote that “[t]he closed world is not a small strongbox in the corner of our collective house of codified and stored knowledge. It is we in the open world—we who study the world lodged in our libraries, from aardvarks to zymurgy, we who are living in a modest information booth facing outwards, our unseeing backs to a vast and classified empire we barely know.”[35] To limit this review to a reasonable length I have specifically chosen to give a few examples of the technological, data science, and social scientific methods used by the History Lab to lift the veil on this “vast and classified empire.” Connelly attributes the resilience of this system, its ability to routinely violate the law with impunity, and its resistance to reform to its “culture of secrecy—its rituals, its ethnography, its belief systems” (207). This culture was produced, at least since World War II, by a dark state not only to protect “national security… but also and especially, the security of particular people who were jealous of their privileges” (30). That said, his book focuses far more on the latter, personal motivations, than on their appeal to “national security” to justify their actions. I would have liked to have seen more attention to this justification, and so to the broader, changing ideological and geo-political framework in which the United States president and the government acted to protect its institutions at home and to project its power abroad, and to the changing practices of the dark state as it molded the concept of national security to justify its secret behavior.
By exploring personal motivations for classifying information in the name of national security, Connelly has added a quite different, and decidedly subversive, dimension to Melvyn Leffler’s definition of national security. To wit: “National security policy encompasses the decisions and actions deemed to protect domestic core values from external threats.”[36] Andrew Preston has already historicized Leffler’s general concept, giving it a specific meaning in the Cold War. Preston writes, “Americans pursued the Cold War under the aegis of ‘national security’ as the pursuit of total security.” With this totalizing extension, the ideology of national security “lost all sense of proportion.” Americans seemed to be facing “limitless threats;” they were “finding monsters everywhere.”[37] As the dark state extended its reach during the Reagan administration, in particular, those monsters were increasingly used to justify new laws of secrecy and of surveillance that led to the steady erosion of the First and Fourth Amendments. Testifying to a Subcommittee of the Congressional Committee on the Judiciary in hearings in September 1984—symbolically entitled 1984: Civil Liberties and the National Security State—John Shattuck (the Vice President for Government, Community and Public Affairs for Harvard University) emphasized the changing and ever more expansive meaning of the policy. “National security,” Shattuck suggested, “had become a very broad concept with very little definition.”[38] A “dangerously expanding concept of national security,” he remarked in his written submission, now posed a major threat to civil liberties, allowing for “greater secrecy, more censorship, a CIA with more domestic authority, an FBI with fewer restraints, and a National Security Agency with broader powers than we have ever had in our history.”[39]
Within a few years the US government extended the scope of the concept again, now to embrace the concept of “economic security” in response to the competitive threat from Japanese semiconductor industries.[40] As Bobby R. Inman and Daniel F. Burton wrote in the April 1990 issue of Foreign Affairs, “national security can no longer be viewed exclusively in military terms; economic security and industrial competitiveness are also vital considerations.”[41] In President Donald Trump’s White House, these were the dominant consideration: Trump is cited as saying that “Economic security is national security” (my emphasis).[42] With each expansion of the concept of national security, the dark state creates new monsters that have to be slayed, new threats that have to be hidden behind the veil of secrecy, in the ceaseless and futile attempt to gain “total knowledge about all possible enemies, foreign and domestic” that has defined its mission ever since the 1940s (37). Hopefully Connelly and his History Lab will eventually be able to correlate changes in the meaning of national security with changes in the classification practices that it engenders, situating them in the broader historical sweep of US domestic and foreign policy. In the interim, this important book has already provided us with new tools to critically analyze the performance of US domestic and global power, rendering democracy—and history—an important service.
Review by Suzanne J. Piotrowski, Rutgers University
Understanding the actions governments undertake is critically important to democratic accountability. Access to information regarding government operations and decisionmaking is necessary for historical analysis and to better understand how policies are implemented. Although legislative bodies mandate the disclosure of some data, governments often fail to release these data either due to disinterest, lack of funding, or both.[43] Since access to government information is essential at all levels of government in the US and around the world, there is an international open government movement to encourage governments to be more transparent, accountable, and participatory.[44]
It is within this context that Matthew Connelly’s The Declassification Engine: What History Reveals about American’s Top Secrets is particularly relevant. Within The Declassification Engine, Connelly and his team at the History Lab, a multi-disciplinary research lab, work to bridge two fields: history and computer science. Connelly is motivated by democratic accountability and is interested in understanding what can be learned from applying machine learning tools to declassified documents from the US federal government.
In the book’s preface, Connelly establishes his subject and gives the origin story of the book. In this engaging introduction, he explains the initial hesitancy he encountered in trying to raise funding for this project. In the initial book chapter, “The Radical Transparency of the American Republic: A Reintroduction,” Connelly gives an engaging tour of the history of American secrets and surveillance. He argues that the United States in the early 1800s worked to create an “ever-more-radical experiment in open government” (10). He shows the ups and downs of US government secrecy and surveillance, explaining that secrecy, surveillance, and limitations on access to information are not equally distributed and that certain groups, including Black Americans, Native Americans, and women, are affected differently. While this understanding is vitally important, it does present a tension with his thesis of radical transparency in the US.
Connelly utilizes a case-study approach throughout the book. Each chapter weaves a story of chosen events or topics, revealing how they relate to secrecy, classified documents, and the evolution of declassification in the US federal government. In chapter 1, “Pearl Harbor: The Original Secret,” Connelly presents a compelling story about secrecy in the lead-up to the attack on Pearl Harbor (36-59). He explains how the dark state was conceived out of this attack and how those patterns continue to this day. (Throughout the book Connelly juxtaposes the terms “dark state” and “deep state.”) Connelly expertly uses these stories to explain the development of the classification system.
In chapter 2, “The Bomb: Born Secret,” Connelly presents the history of how the Atomic Energy Act of 1946 created the concept of “restricted data” (60). These data are restricted no matter their respective origin. He gives examples of students developing draft plans for atomic bombs and how those documents were automatically restricted. He also explains the category of “formerly restricted data” and notes that both are categories in which the information is “born secret” (61). At this point, Connelly reveals both the methodology employed by the Declassification Engine and the efforts undertaken to obtain the documentation needed for analysis. In one analysis, his team worked with 117,000 documents that had been previously declassified and were available from presidential libraries. They ran a series of analyses to better understand what they could learn about which documents get classified.
With a subset of the documents, the team ran a subsequent analysis to understand how well they could predict which documents were likely to have restricted data and formerly restricted data classifications. Connelly gives a full account of the strengths and weaknesses of the tools he uses in his analysis. He concludes that if the US government used similar tools to predict which documents could be declassified, it would be more successful than his team because its agencies would have a much large set of documents to analyze, including those documents that are still restricted.
The final eight chapters of the book each tackle a different topic and use different methods. In chapter 3, Connelly and his team use a massive trove of documents available to them, using algorithms to analyze the words which are immediately before or after redacted text in the Foreign Relations of the United States series. They find that the redacted text in these documents is most likely to be associated with codes, code making, code breaking, ciphers, and cryptanalysis.
Chapter 4 explains the systems surrounding the codes to set off nuclear bombs. Connelly notes that the ultimate authority to start a nuclear war is frighteningly unclear. Surveillance and questionable legal authority are the subjects of chapter 5. Connelly’s team again uses data-mining techniques to look at the US federal government’s internal communications around surveillance. One particularly poignant story is how the Federal Bureau of Investigation (FBI) spied on the National Security Archive, a nonprofit organization based out of George Washington University, because of its proficient use of the Freedom of Information Act (FOIA).[45]
Chapter 6, “Weird Science: Secrets that are Stranger than Fiction,” provides examples of secret scientific information that was subsequently released, which includes examples as varied as invisible ink to supersonic bombers. In “Following the More: Trade Secrets” (chapter 7) Connelly looks at the intersection of national security and business interests by turning algorithms on the History Lab’s data show how business interests intersect with government interests. With each chapter, Connelly expertly weaves anecdotal evidence with the analysis of documents using machine learning.
In “Spin: The Flipside of Secrecy” (chapter 8), Connelly describes how presidents have used executive orders to control public access to information concerning national security. FOIA plays a large part of the story here. Connelly offers a stinging critique of the costs of maintaining these secrets and how they are withheld by presidents. Connelly also reveals that classified information is already in the public sphere and has been printed in newspapers or industry magazines (chapter 9, “There is No There There”). In the penultimate chapter of the book (chapter 10, “Deleting the Archives”), he addresses the role of government archives in keeping records, underscoring the tension between the need to preserve records for history and the management of the sheer amount of material produced every year. This point comes full circle to the beginning of the book where he makes the case for more funding for the National Archives, explaining how the organization is overwhelmed with the document retrieval of electronic records (59).
The entire book is a testament to the potential of governmental transparency and access to information as the documents which the History Lab team analyzed were all former government documents subsequently declassified and released. With the publication of this book, Connelly and The History Lab act as an “infomediary,”[46] or a person or organization who takes a large amount of information or data and translates it in a way that a larger group of people can access and understand it. By gathering government data and packaging it in a more usable format, Connelly acts as an infomediary to the general public. To the author’s great credit, he realizes the forward-looking nature of this exercise and the potential value for the public and governments in this type of study. While in the book’s introduction he acknowledges the critics of his work, the reality is that the aggregation of released public documents and machine learning are here to stay.
Even in a 500-page book, there are limits to what can be covered. For readers who are not as familiar with government documents and how to access them, more background on presidential libraries and what documents are held in them would have been helpful. Similarly, a more fundamental description of what the US FOIA is, and the documents to which it does and does not apply, would be useful for a layperson. Connelly, who is clearly an expert in these tools, has named the searchable depository at the History Lab the Freedom of Information Archive (FOIArchives). Connelly and his team may have already laid this groundwork in other projects.
My approach and methods of studying open government and transparency differ from that of Connelly, and I enjoyed learning about his perspective. In the preface, he frames the work he is doing as one of democratic accountability[47] and links his work to open government. “To save America’s old and honorable tradition of open government—and to save history itself,” Connelly declares, “we must arm ourselves with the power that knowledge gives, including the power of artificial intelligence” (xvii). He is correct in his emphasis on the role that government documents play in holding government officials accountable for their actions. As a scholar of public administration, I was left thinking about how this work fits within the broader open-government framework. Open government is a broad concept which not only includes transparency but also other values, such as accountability and participation.[48] Open government has been described as giving citizens both “vision and voice.”[49] Connelly and his team have laid the groundwork for future scholars to explicitly link their analysis these literatures. Other scholars may also replicate his methods and apply them in different contexts and to other documents. I would be interested in knowing what we could learn from released government contracts using these tools. Another application would be to apply these tools and methods on minutes from local open-public meetings. Local governments go into closed sessions for a host of reasons, and the notes from those meetings are only occasionally released to the public.
The Declassification Engine is an essential contribution to the work of scholars who study government transparency.[50] The interdisciplinary approach to understanding government declassification will become a standard in the field. Ultimately though, Connelly’s most important audience is not historians or scholars of government transparency. His real contribution is exposing the history of government secrets to the general public, most of whom have likely never thought seriously about government records and classification. Connelly’s use of stories from the US history places his sophisticated analysis in context and takes all readers on a tour de force of the history of government classification and secrets.
Review by Aviel Roshwald, Georgetown University
This book’s catchy title actually undersells the importance of its author’s findings. Although The Declassification Engine does include a number of hitherto unknown or underreported nuggets of once-secret information, its central preoccupation is with how much historical data has gone temporarily or permanently missing from the American public record. More broadly, it wrestles with fundamental questions about the vexed relationships among information access/control, political accountability, national security, and historical understanding in a democracy.
Matthew Connelly argues that prior to the Second World War, peacetime operations of the US government were characterized by an unusually high degree of transparency, even when it came to matters of foreign and military affairs. All this changed in the aftermath of the Second World War, when wartime preoccupations with secrecy went on to become a seemingly permanent and ever more prevalent norm. This, of course, had much to do with the development of nuclear weapons, the onset of the Cold War, and the country’s emergence as a globally engaged superpower. But Connelly argues convincingly that the obsession with classifying information rapidly took on a manic life of its own, to the point of becoming self-defeating from a national-security standpoint, let alone by the standard of democratic values. The endemic problem of American governance, he argues, lies not with that bogeyman of the radical right—the so-called “deep state.” Instead, he points his finger at what he dubs the “dark state” (19) —the ever-metastasizing institutional sector of the government whose operations are cloaked by a veil of secrecy in real time and much of whose historical record is being permanently expunged, be it deliberately or through willful negligence.
In tracing the history of the “dark state” from 1941 to the present, Connelly opens with a case study that is likely to prove one of the more controversial in this volume: the lead-up to Pearl Harbor, which he labels as “the original secret” in chapter 1, and whose role in the narrative approximates that of an original sin. To be clear, the sin in question is not that of Japan’s aggression but of the President Franklin Roosevelt administration’s alleged incitement of it.[51] An outsized role in this interpretation is played by Connelly’s discovery (via a team effort involving the use of a computer algorithm to compare redacted and unredacted versions of declassified documents across multiple databases) of a fully unredacted version of a formerly classified document relating an anecdote recounted by Prime Minister Winston Churchill in 1954. The setting was a closed-door, post-dinner conversation at the White House, where Churchill was President Dwight D. Eisenhower’s guest. The anecdote concerned how, in 1941, US Ambassador to Britain John Winant and President Roosevelt’s personal envoy Averell Harriman had responded to news of the attack on Pearl Harbor. The fact that these two figures were socializing with Churchill at the prime-ministerial country residence Chequers that evening is well known. What Churchill’s 1954 version adds to the story is the claim that upon learning of the attack, Winant and Harriman jumped to their feet in excitement and danced a little jig of joy. The record of the 1954 White House conversation also includes Churchill’s teasing Walter Bedell Smith (US Army colonel and secretary of the General Staff at the time of the Pearl Harbor attack) over the delay between American military intelligence’s discovery that Japan had decided to break off negotiations, and the conveying of that information to the Army chief of staff (General George Marshall) and on to American bases in the Pacific. The latter was obviously not something to which Churchill had borne witness; he had learned about it subsequently (40-42).
Connelly uses this story as a lead-in to his reassertion of the “back-door theory” about Pearl Harbor and American entry into the Second World War—the argument that Roosevelt deliberately avoided a possible diplomatic compromise with an aggressively militarist Japan in order to maneuver Tokyo into an initiation of hostilities with the United States that would, in turn, facilitate the United States’ joining the war against Hitler’s Germany. Yet it is not clear how much credence we should lend Churchill’s version of events, given that he related the story while likely inebriated after dinner and perhaps eager to goad his American hosts by means of some narrative embellishment. That said, few historians would question the notion that President Roosevelt was indeed looking for a way to enter the conflict against Germany in the face of resistance from Congressional and public opinion, but before a Nazi victory in Europe could create a geostrategic nightmare for the United States. Whether or not American negotiations with Japan came to be shaped by anticipation that a Japanese attack in the Pacific could jump-start US entry into the war’s Atlantic theater remains a matter of how one interprets a mass of circumstantial evidence.[52] Connelly’s account ultimately depends on that same body of evidence, to which Churchill’s hitherto redacted account adds little. While Connelly ends up rejecting the extreme version of the back-door theory whereby Roosevelt is baselessly accused of having had prior knowledge of the specific targeting of Pearl Harbor, he remains disturbed by the notion that, one way or another, the White House was secretly advancing the United States’ entry into a war that much of the American public wished to steer clear of. He goes so far as to write in defense of Charles Beard, the influential American historian who was one of the originators of the back-door theory and who gained notoriety for his vehement attacks on Roosevelt’s foreign policy both during and after the war (57).[53]
How to understand the relationship between Roosevelt’s approaches to the Pacific and Atlantic theaters over the course of 1941 is an area of legitimate debate—a debate that is all the more interesting because it can likely never be decisively resolved. But before we rehabilitate Beard, it is worth recalling that he was not merely opposed to the manner in which Roosevelt led the country into the conflict. Throughout the remainder of his career, Beard remained adamant that American involvement in the war had been a mistake because there was little to distinguish the evil of, and threat posed by, the Nazi regime from that of the Stalinist one.[54] It was not just the means of Roosevelt’s policies, but more fundamentally its ends, that Beard condemned. Yet surely a world in which a genocidal Hitler regime and its allies emerged triumphant would have been a far worse one than that of the Cold War, the latter’s many evils notwithstanding.
Indeed, Connelly himself takes the opposite stance from Beard’s, conceding that American victory in World War II “…was a triumph for all Americans and indeed [for] the cause of democracy itself” (58). From Connelly’s perspective, it is the lack of transparency surrounding Roosevelt’s policy that remains fundamentally troubling because it reflects a still-prevalent situation in which “our leaders think we can’t handle the truth and wouldn’t support their plans if we knew what they were” (57). But surely what is most troubling about the story of American entry into the Second World War is the fact that it seems to illustrate the wisdom of these leaders’ assumption. In the face of a powerful America First movement and deeply divided public, would an openly pro-war stance have facilitated Roosevelt’s task or set it back by years? If the latter, might that not have left the United States confronting an aggressively triumphant alignment of Axis powers under circumstances even more daunting than those of December 1941? What the history of these events reflects, surely, is the complexity of the dilemmas surrounding foreign-policy and military decision-making in a democracy, rather than illustrating a clear-cut case in favor of transparency no matter what.
In any case, the Pearl Harbor case takes up just one of ten chapters in what is overall a richly informative, engagingly written, and intellectually provocative book. The unearthing of Churchill’s 1954 after-dinner conversation is just one of many finds and insights gained via the application of an innovative digital-history technology that was developed by a team formed under Connelly’s leadership at Columbia University. Originally known as History Lab and more recently achieving more permanent institutional status as the Freedom of Information Archive, this joint effort by historians, social scientists, data analysts, and computer scientists has been employing machine-learning and algorithmic techniques to analyze massive amounts of declassified government documents (including their metadata where available) in an effort to discern patterns that would otherwise remain hidden to the human eye.
In some cases, this method can facilitate intelligent guessing about the likely subject matter of material that remains classified. More consistently, by comparing textual content with the original classification status of declassified documents, History Lab has revealed what kinds of subject matter are the likeliest to be stamped with classification labels. Many of the insights defy certain conventional expectations. Perhaps not so astonishing is the revelation that anything related to cryptography as well as decrypting methods is the most likely to be stamped with the highest classification markings. Potentially more surprising for advocates of reductionist theories about US foreign policy as little more than a front for the pursuit of corporate profits, government communications about global business interests and international commerce are the least likely to be labeled secret.
Connelly’s powerfully made, central argument is that the government’s ever-growing, reflexive tendency to classify all manner of information is fundamentally self-defeating in a variety of ways, chief among them the following two: it undercuts the basic principles of democratic accountability, and it actually makes it harder to protect that small sub-set of classified information that really does need to be kept under wraps for legitimate national-security reasons. Information is power, and the more tightly controlled information is, the more powerful those privy to it feel. Connelly describes the self-perpetuating culture of secrecy that quickly took on a life of its own in American political life from the 1940s on, starting at the very top with the actions of presidents—among whom those most initially vocal about the need to make government more transparent have sometimes proven the most compulsively committed to classifying information and trying to suppress leaks.
The abuses of power on the part of US government agencies and individual political leaders in the name of national security and under the cover of secrecy have been legion. Connelly takes his readers through a rich selection of some of the most notoriously egregious examples, ranging from Central Intelligence Agency (CIA) drug experiments on unwitting subjects to efforts by the American military to circumvent technical curbs put in place to prevent the unauthorized initiation of nuclear warfare. The History Lab algorithms have been able to establish that a disproportionate subset of foreign policy and national security documents from the 1970s that have “gone missing” pertain to communications involving National Security Advisor Henry Kissinger on the subject of US policy in Southeast Asia. The history of civil and human rights violations and their cover-up, all in the name of national security, has been periodically marked by the apparent triumph of transparency efforts, as in the Senate Intelligence Committee’s hearings of half a century ago. But Connelly points out that committee chair Frank Church was himself the target of surveillance by a National Security Agency (NSA) that apparently lacked all sense of irony (188-189). Over the long haul of the last eight decades, this book argues, the arc of American national-security policy has bent toward such opacity that scholars’ ability to write well-documented histories of it may be in peril.
Not only is the government’s pervasive culture of secrecy at odds with democratic principles, Connelly argues, but it can paradoxically serve to undercut the very ability to protect truly crucial national security information. America is now in a situation, he explains, in which the subset of government information and communication subject to classification is so absurdly vast that some 1.3 million people need to have top-secret clearances in order to do their jobs. That makes 1.3 million potential holes in a vessel of confidentiality that is theoretically supposed to be hermetically sealed. To be sure, compartmentalization, which is itself a hindrance to effective cooperation across agencies and personnel, is supposed to help contain the damage caused by any given spill. But the series of major data breaches over the past few years, of which the Wikileaks episode is the most notorious, has illustrated how vulnerable digital-information-storage systems are to penetration on a massive scale.[55]
Having made that point, Connelly goes on to suggest that the revelations associated with some of those breaches have highlighted the egregious overkill of classification practices. In the case of Wikileaks, for instance, while a small subset of the information released to the public compromised the safety of human sources, the overwhelming bulk of it was so mundane that its major revelation was what Connelly dubs “the banality of secrecy” (330). Conversely, he speculates that the ability of Soviet intelligence to penetrate American communications during the Cold War may have had the effect of reassuring the Kremlin about Washington’s intentions during crises when greater uncertainty might have led to a descent into nuclear war by miscalculation (155). One could conclude that in this respect the compulsion to classify information had a net benefit insofar as its secret status would have enhanced its credibility in Soviet eyes, to a degree that no intentional form of communication could have achieved.
An additional liability of the over-classification syndrome, Connelly points out, is its tendency to hinder high-quality scientific research under government auspices. Quality research, after all, is normally the product of a broad and open give-and-take among scholars, is subject to peer review, and becomes publicly available for replication. Research that is effectively non-falsifiable cannot by definition meet the bar for credible science. The best scientists are likely to stay away from government employment or contracts by virtue of this fact, making it likelier that these positions will be filled by mediocrities. The ethical violations and human rights abuses associated with past government and military research efforts (such as the deliberate exposure of US troops to high levels of radiation at nuclear test sites) have also contributed, Connelly conjectures, to the widespread skepticism about science and governmental public health initiatives that has plagued American society in recent years (243).
On the latter point, I am not convinced. The impact of outrages such as the infamous Tuskegee experiment on trust in government-sponsored public health research among African American communities is well documented and readily understandable.[56] But phenomena such as anti-vaxxing, anti-masking, and climate change denial on the right wing of the American political spectrum seem like part of a broader revolt against the fundamental tenets of reason and principles of social responsibility. Careful research would be needed to establish whether and how these tendencies are associated with awareness of past secret programs undertaken by government-affiliated institutions. That said, the era of former president Donald Trump may indeed both reflect and catalyze a fundamental realignment of political cultures in this country, such that it is the center and center-left that now harbors the greatest respect for the proper handling of secret information, whereas the far right cultivates scorn for intelligence agencies and their norms—as symbolized by the classified documents strewn about the Mar-a-Lago estate.
Among Connelly’s most disheartening findings is his discovery of how extensive a practice the permanent destruction of historical records has become. Most of the book’s revelations are the product of past practices of declassification, including the regular release of material subject to time limits on its secret status as well as information pried loose from the government via application of the 1967 Freedom of Information Act. But the exponential increase in the sheer volume of communication (most of it now electronic) accompanied by the ever more reflexive nature of classification practices has not been accompanied by commensurate investment in staff and resources devoted to the process of declassification. Government agencies have therefore relied on haphazard methods not only to determine what and when to declassify but what material to bother retaining at all.
Vast troves of messages and memoranda are being designated as unimportant and therefore subject to permanent deletion from the historical record on the basis of random sampling of their contents and the application of inconsistent standards of what constitutes importance. This is where the methods developed by Connelly and his team have the potential to bear their richest fruit. He calls for the development of algorithmic systems such as those employed by History Lab to be used as tools in the declassification of documents whose quantity far exceeds the ability of humans to go through and assess them in anything approaching a timely and systematic fashion. Not taking such action, he warns, risks the further destruction of a historical record that may melt away as rapidly as polar ice caps subjected to global warming (374, 387, 396).
One does not need to be a historian of American foreign policy to grasp the broader importance of these findings, not just from the perspective of professional scholars but for the survival of a functioning democracy. At the same time, Connelly has succeeded in shedding a remarkably bright light onto the often perverse dynamics of classification and declassification, even as some of those processes themselves are supposed to remain hidden from the public eye. It is to be hoped that future researchers will apply the methods he and his team have developed to the analysis of other countries’ archival collections. Triangulating his findings with the assessment of data and metadata from the documentary records of America’s Five Eyes partners,[57] for instance, could conceivably take scholars even closer to grasping that portion of the classified-information iceberg that remains submerged beneath the surface of history. The Declassification Engine certainly constitutes an inspiring model of how to approach this titanic undertaking.
Response by Matthew Connelly, Columbia University
As Brian Hochman points out in his excellent introduction to this forum, I was lucky enough to have a book about secrecy and history come out in the middle of a historic controversy about how the United States keeps its secrets. I would joke at the time that no other author had both a president and a former president—even former vice president Mike Pence!—drumming up publicity for their book launch. But as my generous H-Diplo reviewers have noted, The Declassification Engine is about more than just “America’s Top Secrets,” even if it is the secrets that seem interesting to most people. My book is about the future of history, and how historians have to rethink the way we explore the past when so much of the historical record is being deleted.
So after the hubbub died down—there were so many special counsels and criminal indictments even I struggled to keep count—I waited with anxiety to see how scholars less caught up in the moment might analyze my arguments. In The Declassification Engine, I describe the many scandals involving the theft, destruction, or leaking of ostensibly dangerous information as surface indicators of a much bigger and deeper problem, one that cannot be fully comprehended without “computer vision” and artificial intelligence. Back in February, in an otherwise favorable newspaper review of my book, one journalist—who emailed me his review from a Hotmail.com account—felt provoked to defend old-fashioned investigative work, and set that in opposition to what I was trying to uncover using algorithms.[58] I was anxious because academics tend to be even less receptive to strong claims about the need to adopt new methods, especially when those methods might require retraining. In the 1970s, the first examples of computational scholarship elicited some excitement in the popular press, until the most influential scholars of the day set about strangling these babies in the crib. I still assign their reviews in the digital history classes I teach, just so my students know what they might be in for.
So it was with real relief that I read the H-Diplo reviews, all of which acknowledge that, in fact, the challenge we face is “titanic,” as Aviel Roshwald puts it. Scholars, archivists, journalists, and data scientists all have roles to play. If we do not use all our intelligence, including artificial intelligence, how will we avoid the disaster represented by the sinking mass of secret historical records? And, as Suzanne Piotrowski points out, there is so much more we can learn by mining the textual data produced by contemporary governance, and not just at the level of “top secret.” My great hope is that more scholars will join with a new generation of data journalists, and combine these tools with time-tested investigative techniques. As Heidi Kitrosser notes, this is a “bright future,” and, considering the catastrophic job market in our two professions, surely both journalists and historians need a positive vision and aspirational goals.
But the reviewers also pay me respect by pushing back against other arguments and reading my work closely enough to see the inner tensions and inescapable complexity. As Piotrowski notes, I fully acknowledge that the radical transparency of the American republic during its first 150 years did not apply to most Americans at the time. It was only for those deemed worthy of exercising the full rights of citizenship—for example, to education, to information, and to participation in their own governance. I would just note that both things can be true: The US was indeed an outlier, in the full breadth of these rights and the limited capacity of the state to surveil and coerce its citizens compared to other countries. But especially when it came to Black Americans, and Native Americans, the US had a whole-of-society approach, in which large numbers of private citizens participated in surveilling and repressing those they deemed inferior. For the most part, they did not even make it a secret.
John Krige rightly points out that, when narrating the rise of what I call the Dark State since World War II, I focus more on the culture of secrecy—including its particular ethnography—and devote less attention to its ideological justification. I agree that the intellectual history of “national security” is fascinating, and important. But there are already talented historians like Andrew Preston who are hard at work unpacking the concept.[59] Moreover, time and again I found the invocation of national security as a justification for specific secrecy practices to be unconvincing, when not demonstrably false. In myriad ways, security clearance investigations, “special access programs,” and the arcana of codewords and color-coded badges served the interests of those who decided that only they had a “need to know,” even when that made the American people less safe.
So I found myself asking whether all the “exceptions”—all the people who were denied or stripped of their security clearances, and all the withheld or redacted documents—actually prove a different rule, one that is nowhere spelled out in the otherwise voluminous clearance forms and classification manuals. If this secrecy did not actually work to make Americans more secure, what work was it doing?
It took me some time to understand, and still more time to rewrite chapters I had drafted, which may be why Piotrowski perceives this theme as a “tension” rather than an integral part of a streamlined narrative. In the end, I realized that both secrecy and a particular definition of national security served to exclude people who were treated with suspicion only because they did not conform to a stereotypical ideal of a loyal American—an ideal type that happened to resemble the kind of American who had long monopolized political and economic power. And yes, powerful people did everything possible to keep that particular secret.
Yet at no point do I argue for “transparency no matter what,” contrary to Roshwald’s characterization. As Kitrosser points out, I instead show that secrets that are illegitimate or self-serving make it harder to protect information that really could kill people. I offer many examples, like sniper manuals, and the names of confidential informants. And as a diplomatic historian, I am extremely familiar with Krista Ferrante’s observation that information that is public might still be officially classified. Contrary to the transparency advocates who falsely recite these facts as examples of official stupidity, we need to understand that in order to avoid diplomatic incidents, the US government must sometimes refrain from acknowledging things that are already well known, even notorious.
The example Roshwald focuses on is a hard case: President Franklin Roosevelt’s secret strategy to bring the US into World War II by provoking a Japanese attack. We agree that Roosevelt had to do everything possible to ensure the defeat of Nazi Germany. I do not know if there was any alternative to deceiving the American people, who largely opposed what they perceived to be a war of choice. What I do know is that hard cases make bad law, worse still when one example—German Führer Adolf Hitler—is taken to prove some law of nature requiring a massive military industrial complex and eighteen different intelligence agencies. I also think that, after all these years, Americans need to stop deceiving themselves: 7 December 1941 was no surprise attack.[60] What is therefore required is not “transparency no matter what.” Transparency is not a good in and of itself. Rather we need to realize that no one—not even a beloved president—should forever escape accountability.
Roshwald is also skeptical about my argument that secret and unethical government research helps explain why many Americans are suspicious of the guidance they get from government researchers, even when it could save their lives. I acknowledge that this is a big claim. But the examples that Roshwald cites actually buttress my case. For instance, while some people have opposed vaccinations almost from the beginning of vaccination programs, there was nothing like the contemporary anti-vax movement before the 1970s. Even after some 40,000 American children were accidentally infected with polio in 1955, the Cutter Incident, there was no concerted opposition to vaccination. This did not start until 1976, when a few hundred people developed Guillain-Barré syndrome during the Swine Flu affair. Could it be because the Cutter Incident occurred when the public had tremendous faith in government scientists, whereas the Swine Flu Affair came amidst a whole series of revelations of official malfeasance, such as the Church Committee’s discovery that the Central Intelligence Agency (CIA) had destroyed the records of a research program to control the minds of unwitting human subjects? But again, this is a complex history, and I agree it requires follow-up research.[61]
I have a harder time understanding Ferrante’s critique, starting with her claim that my book “derides the role of government archivists.” I am sometimes critical of specific decisions archivists make, but only because I deeply respect their role. Rather than treating them as if they were part of a “faceless monolith,” I highlight the work of archivists as foundational to any functioning democracy. In my acknowledgements as well, I thank these “noble men and women,” especially the “true masters” who taught me about the challenges they face in operating the time machines that allow the rest of us to explore history and preserve it for the future. I have repeatedly advocated for increasing the resources for archivists to do their essential work relative to the billions spent every year on secrecy and called on other government officials to respect archivists’ legal authorities.
Ironically, Ferrante’s review describes archivists as part of a faceless monolith, in the form of “the millions of individual Americans who work as federal employees, individuals who we can assume are doing their best.” Why should we assume these millions of people are all doing their best? That is not how accountable government works. And as Ferrante usefully points out, as a profession, archivists have their own ethical obligations. It would therefore be unprofessional, even irresponsible, not to acknowledge those instances in which archivists have not met those obligations and try to understand why.
Take, for instance, the decision by the National Archives and Records Administration (NARA) to grant Immigration and Customs Enforcement (ICE) authorization to destroy five-year old records on the treatment of asylum seekers, despite tens of thousands of critical public comments, including from senators and members of Congress. These records documented civil rights violations and shoddy medical care. A federal judge ordered that the records be saved after finding that NARA had violated its own rules.[62] Then, after President Donald Trump was declared the loser in the 2020 election, ICE officials decided to destroy these records anyway, apparently with impunity.[63] That, too, is not how accountable government is supposed to work. When historians point to these problems, and call for the National Archives to follow its own rules and meet its legal obligations, it seems unfair to dismiss this as a pursuit of “privatized gain.”
Should we be concerned that historians like me will lead the public to view their government only “through its most notorious leaks and failures”? In fact, I use one of the most notorious security failures, Wikileaks’ publication of a quarter of a million diplomatic cables, as a case study to demonstrate how overclassification has led us to wrongly expect the worst of anonymous officials who operate in secret. I also use it to show that “transparency no matter what” can actually be destructive to the privacy rights of individuals and human rights more generally (333-335).
But there is a much bigger reason to doubt Ferrante’s concern that works like mine will make the public miss “many more untold stories about the public good done and average daily work well executed” by the Pentagon, intelligence agencies, and the State Department. The reason is as big as the government’s own public relations budget. We do not even know how big it is. But in 2016, the Pentagon alone spent some $626 million on public relations, six times more than what the entire government spends on its systematic declassification program (317). This is one more reason why I think there is a need to confront the enormous challenge of ensuring democratic accountability in an age when so much of the historical record is endangered. If we are successful, then all the stories—good and bad—can be told. If we are not, no one will even know what they are missing.
In one way I completely agree with Ferrante: I too think historians and archivists are natural allies in the struggle to preserve the historical record and keep our government accountable. We should start acting that way, considering how we are badly outnumbered, and pitifully under-resourced compared to the forces arrayed against us. To take just one more example: after largely succeeding in destroying all the records of all their meetings in 1974, in blatant defiance of the Federal Records Act, the Joint Chiefs of Staff have refused to keep any account of their deliberations, another clear violation of federal regulations (133). Historians and archivists should join in demanding that our military leaders stop running the Pentagon like a numbers racket.
While we may continue to disagree about how, specifically, to meet the immense challenge posed by the exponential growth in secret records, all of us agree that, at the very least, public officials should comply with the law. And archivists and historians are uniquely positioned to insist that, even if our leaders escape every other kind of accountability, they should at least be judged in the court of history. If that is ever to occur, we need to speak with one voice, starting now, in insisting they stop destroying the evidence.
[1] Brian Hochman, “The Real Classified Documents Scandal is Larger than Any Ex-President,” Washington Post 16 February 2023, https://www.washingtonpost.com/books/2023/02/16/declassifcation-enginge-connelly-review/.
[2] Lukas Griessl, “The Power of Secret Knowledge: The RAND Corporation, Ignorance Studies and Sociology,” The American Sociologist (March 2023), DOI: https://doi.org/10.1007/s12108-023-09567-2.
[3] Ezra Cohen, Chair, Public Interest Declassification Board, to Joseph R. Biden, President of the United States, 3 May 2021.
3 Dustin Volz, “Vast Troves of Classified Info Undermine National Security, Spy Chief Says,” Wall Street Journal, 27 January 2022.
[5] “Congress Doesn’t Know How to Count the Number of National Security Professionals and That’s a Problem,” Government Executive, 25 May 2023, https://www.govexec.com/workforce/2023/05/congress-doesnt-know-how-count-number-national-security-professionals-thats-problem/386734/.
[6] Linsey McGoey, “The Logic of Strategic Ignorance,” The British Journal of Sociology 63:3 (2012): 533–576, DOI: https://onlinelibrary.wiley.com/doi/10.1111/j.1468-4446.2012.01424.x..
[7] Eliot Wilczek, Krista Ferrante, and Mike Fleckenstein, “Considerations for NARA on the Future of Recordkeeping,” MITRE, 8 July 2022, https://www.mitre.org/sites/default/files/2022-07/pr-22-2103-considerations-for-nara-on-the-future-of-recordkeeping.pdf.
[8] The Public Interest Declassification Board (PIDB) is a bipartisan board appointment by the President and Congress. They are administered by the National Archives and Records Administration (NARA) through the ISOO.
[9] “Declassification of Records Relating to Nuclear Weapons Testing and Cleanup Activities in the Marshall Islands Feasibility Study,” 2022, https://www.archives.gov/files/pidb/recommendations/marshall-islands-feasibility-study-2022-.pdf.
[10] Society of American Archivists, “SAA Core Values Statement and Code of Ethics,” August 2020, https://www2.archivists.org/statements/saa-core-values-statement-and-code-of-ethics.
[11] US Department of Education, “Family Educational Rights and Privacy Act (FERPA),” US Department of Education 1 (August 2021), http://www.ed.gov/policy/gen/guid/fpco/ferpa/index.html.
[12] Joyce Gabiola; Gracen Brilmyer, Michelle Caswell, and Jimmy Zavala, “‘It’s a Trap’: Complicated Representation in Community-Based Archives,” The American Archivist 85:1 (Spring/Summer 2022), DOI: https://doi.org/10.17723/2327-9702-85.1.60.
[13] Executive Order 13526, “Classified National Security Information,” Section 3.2(d), 29 December 2009.
[14] Arian D. Ravanbakhsh, “Getting Notifications When Schedules Are Open for Comment,” Records Express, 11 January 11 2021, https://records-express.blogs.archives.gov/2021/01/11/getting-notifications-when-schedules-are-open-for-comment/.
[15] “Federal Employer,” US Department of Labor, https://www.dol.gov/agencies/odep/program-areas/employers/federal-employment#:~:text=As%20the%20nation%27s%20largest%20employer.
[16] “A-Z Index of U.S. Government Departments and Agencies,” USA.Gov, https://www.usa.gov/agency-index.
[17] “Documenting Your Public Service,” National Archives, 15 August 2016, https://www.archives.gov/records-mgmt/publications/documenting-your-public-service.html.
[18] Society of American Archivists, “SAA Core Values Statement and Code of Ethics,” August 2020, https://www2.archivists.org/statements/saa-core-values-statement-and-code-of-ethics.
[19] See also History Lab, http://history-lab.org/about.
[20] New York Times Co. v. United States, 403 US 713 (1971), https://supreme.justia.com/cases/federal/us/403/713/.
[21] Fred Schulte, “Why the Promise of Electronic Health Records Has Gone Unfulfilled,” National Public Radio, 18 March 2019, https://www.npr.org/sections/health-shots/2019/03/18/704475396/why-the-promise-of-electronic-health-records-has-gone-unfulfilled.
[22] On Iran-Contra see Malcolm Byrne, Iran-Contra: Reagan’s Scandal and the Unchecked Abuse of Presidential Power (Lawrence: University Press of Kansas, 2014).
[23] On the Manhattan Project see Cynthia Kelly, The Manhattan Project: The Birth of the Atomic Bomb in the Words of Its Creators, Eyewitnesses, and Historians (New York: Leventhal, 2007).
[24] The Espionage Act of 1917 was originally found in Title 50 of the US Code, but is now found under Title 18, Ch. 37. See Cornell Law School, Legal Information Institute, “US Code Chapter 37: Espionage and Censorship,” https://www.law.cornell.edu/uscode/text/18/part-I/chapter-37.
[25] Heidi Kitrosser, “The Espionage Act After the Mar-a-Lago Indictment,” Lawfare, 13 June 2023, https://www.lawfaremedia.org/article/the-espionage-act-after-the-mar-a-lago-indictment.
[26] Kitrosser, “Donald Trump and the Espionage Act,” Lawfare, 18 August 2022, https://www.lawfaremedia.org/article/donald-trump-and-espionage-act.
[27] The material regarding the Teixeira case is from a series of reports in the New York Times: Julian E. Barnes, Helene Cooper, Thomas Gibbons-Neff, Michael Schwirtz and Eric Schmitt, “Leaked Documents Reveal Depth of US Spy Efforts and Russia’s Military Struggles,” New York Times, 8 April 2023, https://www.nytimes.com/2023/04/08/us/politics/leaked-documents-russia-ukraine-war.html; Aric Toler, Christiaan Triebert, Haley Willis, Malachy Browne, Schwirtz and Riley Mellen, “The Airman Who Gamers a Real Taste of War,” New York Times, 13 April 2023, https://www.nytimes.com/2023/04/13/world/europe/jack-teixeira-pentagon-leak.html; Eric Nagourney, “A Quick Guide to What the Leaked U.S. Intelligence Documents Say,” New York Times, 27 April 2023; https://www.nytimes.com/article/leaked-documents-ukraine-russia-war.html; Toler, Robin Stein, Glenn Thrush, Mellen and Ishaan Jhaveri “War, Weapons and Conspiracy Theories: Inside Airman Teixeira’s Online World,” New York Times, 6 May 2023, https://www.nytimes.com/2023/05/06/us/politics/jack-teixeira-leaks-discord-messages.html; Glenn Thrush, “Airman Who Leaked Files is Indicted on Charges of Mishandling Secrets,” New York Times, 15 June 2023, https://www.nytimes.com/2023/06/15/us/politics/jack-teixeira-indicted-document-leaks.html; Beth Treffeisen and Thrush, “Airman Pleads Not Guilty to Federal Charges in Leaks Case,” New York Times, 21 June 2023, https://www.nytimes.com/2023/06/21/us/politics/airman-jack-teixeira-leaks-case.html.
[28] Nagourney, “A Quick Guide to What the Leaked U.S. Intelligence Documents Say.”
[29] Toler, et al., “The Airman Who Gamers a Real Taste of War.”
[30] Thrush, “Airman Who Leaked Files is Indicted on Charges of Mishandling Secrets,” New York Times, 15 June 2023.
[31] Jimmy Carter, Executive Order 12065, National Security Information, 28 June 1978, Section 1-302, accessed 19 July 2023, https://www.presidency.ucsb.edu/documents/executive-order-12065-national-security-information.
[32] As cited by Connelly, Herbert Feis, The Road to Pearl Harbor: The Coming of War Between the United States and Japan (Princeton: Princeton University Press, 1950): William Langer and S. Everett Gleason, The Undeclared War, 1940-1941 (New York: Council on Foreign Relations, 1953).
[33] “Executive Order 9066: Resulting in Japanese-American Incarceration (1942),” National Archives, accessed 23 July 2023, https://www.archives.gov/milestone-documents/executive-order-9066. See also John Howard, Concentration Camps on the Home Front. Japanese Americans in the House of Jim Crow (Chicago: University of Chicago Press, 2008).
[34] Foreign Relations of the United States, eds. Daniel J. Lawler and Erin R. Mahan (Washington: Government Printing Office, 2010), https://history.state.gov/historicaldocuments.
[35] Peter Galison, “Removing Knowledge,” Critical Inquiry 31:1 (Autumn 2004): 229-243, at 231.
[36] Melvyn P. Leffler, “National Security,” in Michael J. Hogan and Thomas C. Paterson, eds., Explaining the History of American Foreign Relations, (Cambridge: Cambridge University Press, 2004), 123.
[37] Andrew Preston, “Monsters Everywhere: A Genealogy of National Security,” Diplomatic History 38:3 (June 2014): 472-500, at 499.
[38] John Shattuck, Hearings Before the Subcommittee on Courts, Civil Liberties and the Administration of Justice of the Committee on the Judiciary, 1984: Civil Liberties and the National Security State, 98th Congress 1st and 2nd Sessions, November 2, 3, 1983 and January 24, April 5 and September 26, 1984 (Washington D.C.: US Government Printing Office, 1984), September 1984, 333-338, at 336. After working for the American Civil Liberties Union from 1971 to 1984, Shattuck moved to Harvard University.
[39] Shattuck, written statement, Hearings, 339–347, at 345, 346.
[40] For details see Mario Daniels and John Krige, Knowledge Regulation and National Security in Postwar America (Chicago: University of Chicago Press, 2022), chapter 7.
[41] B. R. Inman and Daniel F. Burton, “Technology and Competitiveness. The New Policy Frontier,” Foreign Affairs 69:2 (1990): 116-134, at 133. For a history of the notion of economic security see Mario Daniels and John Krige, Knowledge Regulation and National Security in Postwar America (Chicago: University of Chicago Press, 2022), chapter 7.
[42] The White House, National Security Strategy of the United States of America, 17 December 2017, https://trumpwhitehouse.archives.gov/wp-content/uploads/2017/12/NSS-Final-12-18-2017-0905.pdf
[43] Dana DiFilippo, “A Surprising Lawbreaker in Mandated Public Reporting? New Jersey’s Government,” New Jersey Monitor, 28 July 2023, accessed 8 August 2023. https://newjerseymonitor.com/2023/07/28/a-surprising-lawbreaker-in-mandated-public-reporting-the-government/.
[44] Suzanne Piotrowski, Daniel Berliner, and Alex Ingrams, The Power of Partnership in Open Government: Multistakeholder Governance Reform and the Open Government Partnership (Cambridge: MIT Press Information Policy Series, 2022).
[45] National Security Archive, George Washington University, Washington, D.C., accessed 7 August 2023, https://nsarchive.gwu.edu/about.
[46] Becky Carter, GSDRC Helpdesk Research Report 1347: Infomediaries and Accountability (Birmingham, UK: GSDRC, University of Birmingham, 2016).
[47] See Robert D Behn, Rethinking Democratic Accountability (Washington, D.C.: Brookings Institution Press, 2001).
[48] See J. Ramon Gil-Garcia et al, “Making Sense of Open Government: A Conceptual Framework and Ideas for Future Research,” Perspectives on Public Management and Governance 6:2-3 (2023): 80-93, https://doi.org/10.1093/ppmgov/gvad005; Alex Ingrams, “Administrative Reform and the Quest for Openness: A Popperian Review of Open Government,” Administration & Society 52:2 (2020): 319-340. https://doi.org/10.1177/0095399719875460; and Suzanne Piotrowski, “The ‘Open Government Reform’ Movement: The Case of the Open Government Partnership and US Transparency Policies,” The American Review of Public Administration 47:2 (2017): 155-171.
[49] Albert Meijer, Deirdre Curtin, and Maarten Hillebrandt, “Open Government: Connecting Vision and Voice,” International Review of Administrative Sciences 78:1 (2012): 10-29.
[50] See Suzanne Piotrowski, Governmental Transparency in the Path of Administrative Reform (Albany: SUNY Press, 2007), and Gregory Porumbescu, G., Albert Meijer, and Stephan Grimmelikhuijsen, Government Transparency: State of the Art and New Perspectives (Cambridge: Cambridge University Press, 2022).
[51] On the Japanese government’s record of expansionism and militarism, as well as its muddled decision-making process in the lead-up to Pearl Harbor, see Eri Hotta, Japan 1941: Countdown to Infamy (New York: Knopf, 2013).
[52] The case in favor of the “back-door” theory is succinctly articulated in Marc Trachtenberg, The Craft of International History (Princeton: Princeton University Press, 2006), chapter 4.
[53] Charles A. Beard, “We’re Blundering into War,” The American Mercury, April 1939, 398; Beard, A Foreign Policy for America (New York: Knopf, 1940); Beard, President Roosevelt and the Coming of the War, 1941: Appearances and Realities (Livingston, NJ: Transaction Publishers, 2003 [1948]).
[54] Beard, Roosevelt and the Coming of the War, 576-577.
[55] David Leigh, Wikileaks: Inside Julian Assange’s War on Secrecy (New York: Public Affairs, 2011).
[56] Susan Reverby, Examining Tuskegee: The Infamous Syphilis Study and its Legacy (Chapel Hill: University of North Carolina Press, 2009).
[57] The Five Eyes is an intelligence alliance comprising Australia, Canada, New Zealand, the United Kingdom, and the United States.
[58] Unnamed, email message to author, 8 February 2023.
[59] See, for example, Andrew Preston, “Monsters Everywhere: A Genealogy of National Security,” Diplomatic History 38:3 (2014): 477-500.
[60] See chapter one in The Declassification Engine. As Roshwald notes, my analysis is indebted to Marc Trachtenberg, The Craft of International History (Princeton: Princeton University Press, 2006), chapter 4.
[61] I thank Leah Benque for sharing the research she has already begun on the Cutter Incident and the Swine Flu Affair. For background, see Paul A. Offit, The Cutter Incident: How America’s First Polio Vaccine Led to the Growing Vaccine Crisis (Yale University Press, 2007), and Richard E. Neustadt and Harvey V. Fineberg, The Swine Flu Affair: Decision-Making on a Slippery Disease (U.S. Department of Health, Education, and Welfare, 1978).
[62] Justine Coleman, “Judge Rules ICE Must Keep Records About Sexual Assault and Detainee Deaths,” 14 March 2021, The Hill, https://thehill.com/regulation/court-battles/543160-judge-rules-ice-must-keep-records-about-sexual-assault-and-detainee/.
[63] Office of the Inspector General, DHS, “ICE Needs to Improve Its Oversight of Segregation Use in Detention Facilities,” 13 October 2021, http://www.oig.dhs.gov/sites/default/files/assets/2021-10/OIG-22-01-Oct21.pdf.