Analytic Culture in the U.S. Intelligence Community (PDF) - CIA
Analytic Culture in the U.S. Intelligence Community (PDF) - CIA
Analytic Culture in the U.S. Intelligence Community (PDF) - CIA
You also want an ePaper? Increase the reach of your titles
YUMPU automatically turns print PDFs into web optimized ePapers that Google loves.
All statements of fact, op<strong>in</strong>ion, or analysis expressed <strong>in</strong> this book<br />
are those of <strong>the</strong> authors. They do not necessarily reflect official<br />
positions of <strong>the</strong> Central <strong>Intelligence</strong> Agency or any o<strong>the</strong>r US government<br />
entity, past or present. Noth<strong>in</strong>g <strong>in</strong> <strong>the</strong> contents should be<br />
construed as assert<strong>in</strong>g or imply<strong>in</strong>g US government endorsement<br />
of <strong>the</strong> authors’ factual statements and <strong>in</strong>terpretations.<br />
The Center for <strong>the</strong> Study of <strong>Intelligence</strong><br />
The Center for <strong>the</strong> Study of <strong>Intelligence</strong> (CSI) was founded <strong>in</strong> 1974 <strong>in</strong><br />
response to Director of Central <strong>Intelligence</strong> James Schles<strong>in</strong>ger’s desire to create<br />
with<strong>in</strong> <strong>CIA</strong> an organization that could “th<strong>in</strong>k through <strong>the</strong> functions of<br />
<strong>in</strong>telligence and br<strong>in</strong>g <strong>the</strong> best <strong>in</strong>tellects available to bear on <strong>in</strong>telligence problems.”<br />
The Center, compris<strong>in</strong>g professional historians and experienced practitioners,<br />
attempts to document lessons learned from past operations, explore<br />
<strong>the</strong> needs and expectations of <strong>in</strong>telligence consumers, and stimulate serious<br />
debate on current and future <strong>in</strong>telligence challenges.<br />
To support <strong>the</strong>se activities, CSI publishes Studies <strong>in</strong> <strong>Intelligence</strong> and books<br />
and monographs address<strong>in</strong>g historical, operational, doctr<strong>in</strong>al, and <strong>the</strong>oretical<br />
aspects of <strong>the</strong> <strong>in</strong>telligence profession. It also adm<strong>in</strong>isters <strong>the</strong> <strong>CIA</strong> Museum<br />
and ma<strong>in</strong>ta<strong>in</strong>s <strong>the</strong> Agency’s Historical <strong>Intelligence</strong> Collection.<br />
Comments and questions may be addressed to:<br />
Center for <strong>the</strong> Study of <strong>Intelligence</strong><br />
Central <strong>Intelligence</strong> Agency<br />
Wash<strong>in</strong>gton, DC 20505<br />
Pr<strong>in</strong>ted copies of this book are available to requesters outside <strong>the</strong><br />
US government from:<br />
Government Pr<strong>in</strong>t<strong>in</strong>g Office (GPO)<br />
Super<strong>in</strong>tendent of Documents<br />
P.O. Box 391954<br />
Pittsburgh, PA 15250-7954<br />
Phone: (202) 512-1800<br />
E-mail: orders@gpo.gov<br />
ISBN: 1-929667-13-2
ANALYTIC CULTURE IN THE US<br />
INTELLIGENCE COMMUNITY
The Center for <strong>the</strong> Study of <strong>Intelligence</strong><br />
Central <strong>Intelligence</strong> Agency<br />
Wash<strong>in</strong>gton, DC 20505<br />
Johnston, Rob<br />
Library of Congress Catalogu<strong>in</strong>g-<strong>in</strong>-Publication data<br />
<strong>Analytic</strong> <strong>Culture</strong> <strong>in</strong> <strong>the</strong> US <strong>Intelligence</strong> <strong>Community</strong>: An Ethnographic Study/<br />
Dr. Rob Johnston<br />
Includes bibliographic references.<br />
ISBN 1-929667-13-2 (pbk.:alk paper)<br />
1. <strong>Intelligence</strong>—United States. 2. <strong>Intelligence</strong> analysis.<br />
3. <strong>Intelligence</strong> policy. 4. <strong>Intelligence</strong> tra<strong>in</strong><strong>in</strong>g.<br />
Typeset <strong>in</strong> Times and Ariel.<br />
Pr<strong>in</strong>ted by Imag<strong>in</strong>g and Publication Support, <strong>CIA</strong>.<br />
Cover design: Imag<strong>in</strong>g and Publication Support, <strong>CIA</strong>.<br />
The pensive subject of <strong>the</strong> statue is Karl Ernst von Baer (1792–1876), <strong>the</strong><br />
Prussian-Estonian pioneer of embryology, geography, ethnology, and physical<br />
anthropology (Jane M. Oppenheimer, Encyclopedia Brittanica).
ANALYTIC CULTURE IN THE<br />
US INTELLIGENCE COMMUNITY<br />
AN ETHNOGRAPHIC STUDY<br />
DR. ROB JOHNSTON<br />
Center for <strong>the</strong> Study of <strong>Intelligence</strong><br />
Central <strong>Intelligence</strong> Agency<br />
Wash<strong>in</strong>gton, DC<br />
2005
ACKNOWLEDGEMENTS<br />
There are literally slightly more than 1,000 people to thank for <strong>the</strong>ir help <strong>in</strong><br />
develop<strong>in</strong>g this work. Most of <strong>the</strong>m I cannot name, for one reason or ano<strong>the</strong>r,<br />
but my thanks go out to all of those who took <strong>the</strong> time to participate <strong>in</strong> this<br />
research project. Thank you for your trust. Particular thanks are due <strong>the</strong> o<strong>the</strong>r<br />
researchers who coauthored chapters: Judith Meister Johnston, J. Dexter<br />
Fletcher, and Stephen Konya.<br />
There is a long list of <strong>in</strong>dividuals and <strong>in</strong>stitutions deserv<strong>in</strong>g of my gratitude,<br />
but, at <strong>the</strong> outset, for mak<strong>in</strong>g this study possible, I would like to express my<br />
appreciation to Paul Johnson and Woody Kuhns, <strong>the</strong> chief and deputy chief of<br />
<strong>the</strong> Central <strong>Intelligence</strong> Agency’s Center for <strong>the</strong> Study of <strong>Intelligence</strong>, and<br />
<strong>the</strong>ir staff for <strong>the</strong>ir support and to John Phillips and Tom Kennedy of <strong>the</strong> <strong>Intelligence</strong><br />
Technology Innovation Center who, along with <strong>the</strong>ir staff, adm<strong>in</strong>ister<br />
<strong>the</strong> Director of Central <strong>Intelligence</strong> Postdoctoral Fellowship Program.<br />
I would like to thank Greg Treverton and Joe Hayes for <strong>the</strong>ir help throughout<br />
this project and for <strong>the</strong>ir will<strong>in</strong>gness to give of <strong>the</strong>ir time. Dr. Forrest<br />
Frank, Charles Perrow, and Mat<strong>the</strong>w Johnson deserve recognition for <strong>the</strong><br />
material <strong>the</strong>y contributed. Although it was not possible to cite <strong>the</strong>m as references<br />
for those contributions, <strong>the</strong>se are <strong>in</strong>dicated <strong>in</strong> <strong>the</strong> footnotes, and I give<br />
<strong>the</strong>m full credit for <strong>the</strong>ir work and efforts.<br />
I would also like to thank Bruce Berkowitz, Mike Warner, Fritz Ermarth,<br />
Gordon Oehler, Jeffrey Cooper, Dave Kaplan, John Morrison, James Wirtz,<br />
Robyn Dawes, Chris Johnson, Marilyn Peterson, Drew Cukor, Dennis<br />
McBride, Paul Chatelier, Stephen Marr<strong>in</strong>, Randy Good, Brian Hear<strong>in</strong>g, Phil<br />
Williams, Jonathan Clemente, Jim Wilson, Dennis, Kowal, Randy Murch,<br />
Gordie Boezer, Steve Holder, Joel Resnick, Mark Stout, Mike Vlahos, Mike<br />
Rigdon, Jim Silk, Karl Lowe, Kev<strong>in</strong> O’Connell, Dennis Gormley, Randy<br />
Pherson, Chris Andrew, Daniel Serfaty, Tom Armour, Gary Kle<strong>in</strong>, Brian<br />
Moon, Richard Hackman, Charlie Kisner, Matt McKnight, Joe Rosen, Mike<br />
Yared, Jen Lucas, Dick Heuer, Robert Jervis, Pam Harbourne, Katie Dorr,<br />
v
Dori Akerman, and, most particularly, my editors: Mike Schneider, Andy<br />
Vaart, and Barbara Pace. Special thanks are also due Adm. Dennis Blair, USN<br />
(Ret.), and Gen. Larry Welch, USAF (Ret.), and <strong>the</strong>ir staff.<br />
There were 50 peer reviewers who made sure I did not go too far afield <strong>in</strong><br />
my research and analysis. Aga<strong>in</strong>, <strong>the</strong>re are mitigat<strong>in</strong>g reasons why I cannot<br />
thank <strong>the</strong>m by name. Suffice it to say, <strong>the</strong>ir work and <strong>the</strong>ir time were <strong>in</strong>valuable,<br />
and I appreciate <strong>the</strong>ir efforts.<br />
Because I cannot name specific <strong>in</strong>dividuals, I would like to thank <strong>the</strong> organizations<br />
of <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong> that gave me access to perform this research<br />
and made available research participants: Air Force <strong>Intelligence</strong>; Army <strong>Intelligence</strong>;<br />
Central <strong>Intelligence</strong> Agency; Defense <strong>Intelligence</strong> Agency; Department of<br />
Energy; Department of Homeland Security; Bureau of <strong>Intelligence</strong> and Research,<br />
Department of State; Department of <strong>the</strong> Treasury; Federal Bureau of Investigation;<br />
Mar<strong>in</strong>e Corps <strong>Intelligence</strong>; National Geospatial <strong>Intelligence</strong> Agency;<br />
National Reconnaissance Office; National Security Agency; Navy <strong>Intelligence</strong>.<br />
Thanks are also due <strong>the</strong> follow<strong>in</strong>g: Institute for Defense Analyses; <strong>the</strong> Sherman<br />
Kent Center and <strong>the</strong> Global Futures Partnership at <strong>the</strong> <strong>CIA</strong> University; <strong>the</strong><br />
<strong>CIA</strong>’s Publications Review Board; Office of Public Affairs, National Archives;<br />
Jo<strong>in</strong>t Military <strong>Intelligence</strong> College; Advanced Research and Development Activity,<br />
Defense Advanced Research Projects Agency (DARPA); International Association<br />
of Law Enforcement <strong>Intelligence</strong> Analysts; Drug Enforcement<br />
Adm<strong>in</strong>istration and <strong>the</strong> DEA Academy; FBI Academy; National Military <strong>Intelligence</strong><br />
Association; Association of Former <strong>Intelligence</strong> Officers; MITRE; RAND;<br />
<strong>Analytic</strong> Services, Inc. (ANSER); Potomac Institute; Center for Strategic and<br />
International Studies; Woodrow Wilson International Center; Booz Allen Hamilton;<br />
Naval Postgraduate School; Columbia University; Dartmouth College; University<br />
of Pittsburgh; Georgetown University; Carnegie Mellon University;<br />
Cambridge University; Johns Hopk<strong>in</strong>s University and <strong>the</strong> Advanced Physics<br />
Laboratory; George Mason University; Harvard University; Yale University;<br />
American Anthropological Association; Society for <strong>the</strong> Anthropology of Work;<br />
Society for Applied Anthropology; National Association for <strong>the</strong> Practice of<br />
Anthropology; Inter-University Sem<strong>in</strong>ar on Armed Forces and Society; Royal<br />
Anthropological Institute, and <strong>the</strong> national laboratories.<br />
I express my s<strong>in</strong>cere apologies if I have failed to <strong>in</strong>clude any <strong>in</strong>dividuals or<br />
organizations to which thanks are due. Moreover, any errors of commission or<br />
omission are my own. God knows, with this much help, <strong>the</strong>re is no one to<br />
blame but myself. Mostly, though, I would like to thank my long-suffer<strong>in</strong>g<br />
wife, to whom this book is dedicated. Thanks, Jude.<br />
vi
CONTENTS<br />
Foreword by Gregory F. Treverton ........................................................... xi<br />
Introduction ............................................................................................. xiii<br />
Background ...................................................................................... xiv<br />
Scope ............................................................................................... xvii<br />
A Work <strong>in</strong> Progress .......................................................................... xix<br />
Part I: Research F<strong>in</strong>d<strong>in</strong>gs<br />
Chapter One: Def<strong>in</strong>itions ........................................................................... 3<br />
Chapter Two: F<strong>in</strong>d<strong>in</strong>gs................................................................................ 9<br />
The Problem of Bias ........................................................................ 10<br />
F<strong>in</strong>d<strong>in</strong>g: Secrecy Versus Efficacy ..................................................... 11<br />
F<strong>in</strong>d<strong>in</strong>g: Time Constra<strong>in</strong>ts ................................................................ 13<br />
F<strong>in</strong>d<strong>in</strong>g: Focus on Current Production .............................................. 15<br />
F<strong>in</strong>d<strong>in</strong>g: Rewards and Incentives ...................................................... 16<br />
F<strong>in</strong>d<strong>in</strong>g: “Tradecraft” Versus Scientific Methodology ..................... 17<br />
F<strong>in</strong>d<strong>in</strong>g: Confirmation Bias, Norms, and Taboos ............................ 21<br />
F<strong>in</strong>d<strong>in</strong>g: <strong>Analytic</strong> Identity ................................................................ 25<br />
F<strong>in</strong>d<strong>in</strong>g: <strong>Analytic</strong> Tra<strong>in</strong><strong>in</strong>g ............................................................... 28<br />
Part II: Ethnography of Analysis<br />
Chapter Three: A Taxonomy of <strong>Intelligence</strong> Variables ........................... 33<br />
<strong>Intelligence</strong> Analysis ......................................................................... 34<br />
Develop<strong>in</strong>g <strong>the</strong> Taxonomy ................................................................ 37<br />
Systemic Variables ........................................................................... 39<br />
Systematic Variables ......................................................................... 40<br />
Idiosyncratic Variables ..................................................................... 41<br />
Communicative Variables ................................................................. 42<br />
Conclusion ........................................................................................ 43<br />
Chapter Four: Test<strong>in</strong>g <strong>the</strong> <strong>Intelligence</strong> Cycle<br />
Through Systems Model<strong>in</strong>g and Simulation ......................................... 45<br />
The Traditional <strong>Intelligence</strong> Cycle ................................................... 45<br />
Systematic Analysis .......................................................................... 47<br />
F<strong>in</strong>d<strong>in</strong>gs Based on Systematic Analysis ........................................... 47<br />
Systemic Analysis ............................................................................. 50<br />
vii
F<strong>in</strong>d<strong>in</strong>gs Based on Systems Analysis ................................................54<br />
Recommendations ............................................................................55<br />
Part III: Potential Areas for Improvement<br />
Chapter Five: Integrat<strong>in</strong>g Methodologists <strong>in</strong>to Teams of Experts ............61<br />
Becom<strong>in</strong>g an Expert .........................................................................61<br />
The Power of Expertise .....................................................................63<br />
The Paradox of Expertise ..................................................................64<br />
The Burden on <strong>Intelligence</strong> Analysts ................................................66<br />
The Pros and Cons of Teams .............................................................68<br />
Can Technology Help? ......................................................................71<br />
<strong>Analytic</strong> Methodologists ...................................................................72<br />
Conclusion ........................................................................................72<br />
Chapter Six: The Question of Foreign <strong>Culture</strong>s:<br />
Combat<strong>in</strong>g Ethnocentrism <strong>in</strong> <strong>Intelligence</strong> Analysis ...............................75<br />
Case Study One: Tiananmen Square .................................................76<br />
Case Study Two: The Red Team ......................................................81<br />
Conclusion and Recommendations ...................................................84<br />
Chapter Seven: Instructional Technology:<br />
Effectiveness and Implications for <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong> ...........87<br />
Background ........................................................................................88<br />
Meta-analysis Demonstrates <strong>the</strong> Effectiveness of<br />
Instructional Technology ............................................................90<br />
Current Research on Higher Cognitive Abilities ...............................92<br />
Discussion ..........................................................................................94<br />
Conclusion .........................................................................................96<br />
Chapter Eight: Organizational <strong>Culture</strong>:<br />
Anticipatory Socialization and <strong>Intelligence</strong> Analysts ............................97<br />
Organizational Socialization .............................................................98<br />
Anticipatory Socialization ................................................................99<br />
Consequences of <strong>Culture</strong> Mismatch ................................................101<br />
Anticipatory Socialization <strong>in</strong> <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong> .............102<br />
Conclusion and Recommendations .................................................104<br />
Chapter N<strong>in</strong>e: Recommendations ...........................................................107<br />
The First Step: Recogniz<strong>in</strong>g A Fundamental Problem ....................107<br />
Performance Improvement Infrastructure .......................................108<br />
Infrastructure Requirements ............................................................108<br />
Research Programs ..........................................................................111<br />
The Importance of Access ...............................................................115<br />
viii
Part IV: Notes on Methodology<br />
Chapter Ten: Survey Methodology ........................................................ 119<br />
Methodology ................................................................................... 120<br />
Demographics ................................................................................. 124<br />
Chapter Eleven: Q-Sort Methodology ................................................... 127<br />
Chapter Twelve: The “File-Drawer” Problem and<br />
Calculation of Effect Size .................................................................... 129<br />
Appendix: Selected Literature ............................................................... 133<br />
<strong>Intelligence</strong> Tools and Techniques ................................................. 133<br />
Cognitive Processes and <strong>Intelligence</strong> ............................................. 134<br />
Tools and Techniques as Cognitive Processes ............................... 134<br />
<strong>Intelligence</strong> Analysis as Individual Cognitive Process ................... 135<br />
Error ................................................................................................ 135<br />
Language and Cognition ................................................................. 136<br />
Bibliography............................................................................................ 139<br />
Published Sources ........................................................................... 139<br />
Web resources ................................................................................. 157<br />
Afterword by Joseph Hayes ................................................................... 159<br />
The Author .............................................................................................. 161<br />
ix
FOREWORD<br />
Gregory F. Treverton<br />
It is a rare season when <strong>the</strong> <strong>in</strong>telligence story <strong>in</strong> <strong>the</strong> news concerns <strong>in</strong>telligence<br />
analysis, not secret operations abroad. The United States is hav<strong>in</strong>g such<br />
a season as it debates whe<strong>the</strong>r <strong>in</strong>telligence failed <strong>in</strong> <strong>the</strong> run-up to both September<br />
11 and <strong>the</strong> second Iraq war, and so Rob Johnston’s wonderful book is perfectly<br />
timed to provide <strong>the</strong> back-story to those headl<strong>in</strong>es. The <strong>CIA</strong>’s Center<br />
for <strong>the</strong> Study of <strong>Intelligence</strong> is to be commended for hav<strong>in</strong>g <strong>the</strong> good sense to<br />
f<strong>in</strong>d Johnston and <strong>the</strong> courage to support his work, even though his conclusions<br />
are not what many <strong>in</strong> <strong>the</strong> world of <strong>in</strong>telligence analysis would like to<br />
hear.<br />
He reaches those conclusions through <strong>the</strong> careful procedures of an anthropologist—conduct<strong>in</strong>g<br />
literally hundreds of <strong>in</strong>terviews and observ<strong>in</strong>g and participat<strong>in</strong>g<br />
<strong>in</strong> dozens of work groups <strong>in</strong> <strong>in</strong>telligence analysis—and so <strong>the</strong>y<br />
cannot easily be dismissed as mere op<strong>in</strong>ion, still less as <strong>the</strong> bitter mutter<strong>in</strong>gs<br />
of those who have lost out <strong>in</strong> <strong>the</strong> bureaucratic wars. His f<strong>in</strong>d<strong>in</strong>gs constitute not<br />
just a strong <strong>in</strong>dictment of <strong>the</strong> way American <strong>in</strong>telligence performs analysis,<br />
but also, and happily, a guide for how to do better.<br />
Johnston f<strong>in</strong>ds no basel<strong>in</strong>e standard analytic method. Instead, <strong>the</strong> most common<br />
practice is to conduct limited bra<strong>in</strong>storm<strong>in</strong>g on <strong>the</strong> basis of previous analysis,<br />
thus produc<strong>in</strong>g a bias toward confirm<strong>in</strong>g earlier views. The validat<strong>in</strong>g of<br />
data is questionable—for <strong>in</strong>stance, <strong>the</strong> Directorate of Operation’s (DO) “clean<strong>in</strong>g”<br />
of spy reports doesn’t permit test<strong>in</strong>g of <strong>the</strong>ir validity—re<strong>in</strong>forc<strong>in</strong>g <strong>the</strong> tendency<br />
to look for data that confirms, not refutes, prevail<strong>in</strong>g hypo<strong>the</strong>ses. The<br />
process is risk averse, with considerable managerial conservatism. There is<br />
much more emphasis on avoid<strong>in</strong>g error than on imag<strong>in</strong><strong>in</strong>g surprises. The analytic<br />
process is driven by current <strong>in</strong>telligence, especially <strong>the</strong> <strong>CIA</strong>’s crown jewel<br />
analytic product, <strong>the</strong> President’s Daily Brief (PDB), which might be caricatured<br />
xi
as “CNN plus secrets.” Johnston doesn’t put it quite that way, but <strong>the</strong> <strong>Intelligence</strong><br />
<strong>Community</strong> does more report<strong>in</strong>g than <strong>in</strong>-depth analysis.<br />
None of <strong>the</strong> analytic agencies knows much about <strong>the</strong> analytic techniques of<br />
<strong>the</strong> o<strong>the</strong>rs. In all, <strong>the</strong>re tends to be much more emphasis on writ<strong>in</strong>g and communication<br />
skills than on analytic methods. Tra<strong>in</strong><strong>in</strong>g is driven more by <strong>the</strong><br />
dru<strong>the</strong>rs of <strong>in</strong>dividual analysts than by any strategic view of <strong>the</strong> agencies and<br />
what <strong>the</strong>y need. Most tra<strong>in</strong><strong>in</strong>g is on-<strong>the</strong>-job.<br />
Johnston identifies <strong>the</strong> needs for analysis of at least three different types of<br />
consumers—cops, spies, and soldiers. The needs of those consumers produce<br />
at least three dist<strong>in</strong>ct types of <strong>in</strong>telligence—<strong>in</strong>vestigative or operational, strategic,<br />
and tactical.<br />
The research suggests <strong>the</strong> need for serious study of analytic methods across<br />
all three, guided by professional methodologists. Analysts should have many<br />
more opportunities to do fieldwork abroad. They should also move much<br />
more often across <strong>the</strong> agency “stovepipes” <strong>the</strong>y now <strong>in</strong>habit. These movements<br />
would give <strong>the</strong>m a richer sense for how o<strong>the</strong>r agencies do analysis.<br />
Toge<strong>the</strong>r, <strong>the</strong> analytic agencies should aim to create “communities of practice,”<br />
with mentor<strong>in</strong>g, analytic practice groups, and various k<strong>in</strong>ds of on-l<strong>in</strong>e<br />
resources, <strong>in</strong>clud<strong>in</strong>g forums on methods and problem solv<strong>in</strong>g. These communities<br />
would be l<strong>in</strong>ked to a central repository of lessons learned, based on<br />
after-action post-mortems and more formal reviews of strategic <strong>in</strong>telligence<br />
products. These reviews should derive lessons for <strong>in</strong>dividuals and for teams<br />
and should look at roots of errors and failures. Oral and written histories<br />
would serve as o<strong>the</strong>r sources of wherewithal for lessons. These communities<br />
could also beg<strong>in</strong> to reshape organizations, by reth<strong>in</strong>k<strong>in</strong>g organizational<br />
designs, develop<strong>in</strong>g more formal socialization programs, test<strong>in</strong>g group configurations<br />
for effectiveness, and do<strong>in</strong>g <strong>the</strong> same for management and leadership<br />
practices.<br />
The agenda Johnston suggests is a daunt<strong>in</strong>g one, but it f<strong>in</strong>ds echoes <strong>in</strong> <strong>the</strong><br />
work of small, <strong>in</strong>novative groups across <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong>—groups<br />
more tolerated than sponsored by agency leaders. With <strong>the</strong> challenge workforce<br />
demographics poses for <strong>the</strong> <strong>Community</strong>—<strong>the</strong> “gray-green” age distribution,<br />
which means that large numbers of new analysts will lack mentors as old<br />
hands retire—also comes <strong>the</strong> opportunity to refashion methods and organizations<br />
for do<strong>in</strong>g <strong>in</strong>telligence analysis. When <strong>the</strong> f<strong>in</strong>ger-po<strong>in</strong>t<strong>in</strong>g <strong>in</strong> Wash<strong>in</strong>gton<br />
subsides, and <strong>the</strong> time for serious change arrives, <strong>the</strong>re will be no better place<br />
to start than with Rob Johnston’s f<strong>in</strong>e book.<br />
xii
INTRODUCTION<br />
In August 2001, I accepted a Director of Central <strong>Intelligence</strong> postdoctoral<br />
research fellowship with <strong>the</strong> Center for <strong>the</strong> Study of <strong>Intelligence</strong> (CSI) at <strong>the</strong><br />
Central <strong>Intelligence</strong> Agency. The purpose of <strong>the</strong> fellowship, which was to<br />
beg<strong>in</strong> <strong>in</strong> September and last for two years, was to identify and describe conditions<br />
and variables that negatively affect <strong>in</strong>telligence analysis. Dur<strong>in</strong>g that<br />
time, I was to <strong>in</strong>vestigate analytic culture, methodology, error, and failure<br />
with<strong>in</strong> <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong> us<strong>in</strong>g an applied anthropological methodology<br />
that would <strong>in</strong>clude <strong>in</strong>terviews (thus far, <strong>the</strong>re have been 489), direct and<br />
participant observation, and focus groups.<br />
I began work on this project four days after <strong>the</strong> attack of 11 September, and<br />
its profound effect on <strong>the</strong> professionals <strong>in</strong> <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong> was<br />
clearly apparent. As a whole, <strong>the</strong> people I <strong>in</strong>terviewed and observed were<br />
patriotic without pageantry or fanfare, <strong>in</strong>telligent, hard work<strong>in</strong>g, proud of <strong>the</strong>ir<br />
profession, and angry. They were angry about <strong>the</strong> attack and that <strong>the</strong> militant<br />
Islamic <strong>in</strong>surgency about which <strong>the</strong>y had been warn<strong>in</strong>g policymakers for<br />
years had murdered close to 3,000 people <strong>in</strong> <strong>the</strong> United States itself. There<br />
was also a sense of guilt that <strong>the</strong> attack had happened on <strong>the</strong>ir watch and that<br />
<strong>the</strong>y had not been able to stop it.<br />
Hav<strong>in</strong>g occurred under <strong>the</strong> dark shadow of that attack, this study has no<br />
comparable basel<strong>in</strong>e aga<strong>in</strong>st which its results could be tested, and it is difficult<br />
to identify biases that might exist <strong>in</strong> <strong>the</strong>se data as a result of 11 September. In<br />
some ways, post-9/11 data may be questionable. For example, angry people<br />
may have an ax to gr<strong>in</strong>d or an agenda to push and may not give <strong>the</strong> most reliable<br />
<strong>in</strong>terviews. Yet, <strong>in</strong> o<strong>the</strong>r ways, post-9/11 data may be more accurate.<br />
When people become angry enough, <strong>the</strong>y tend to blurt out <strong>the</strong> truth—or, at<br />
least, <strong>the</strong>ir perception of <strong>the</strong> truth. The people I encountered were, <strong>in</strong> my judgxiii
INTRODUCTION<br />
ment, very open and honest; and this, too, may be attributable to 9/11. In any<br />
case, that event is now part of <strong>the</strong> culture of <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong>, and<br />
that <strong>in</strong>cludes whatever consequences or biases resulted from it.<br />
Background<br />
The opportunity to do this research presented itself, at least <strong>in</strong> part, as a<br />
result of my participation <strong>in</strong> a multiyear research program on medical error<br />
and failure for <strong>the</strong> Defense Advanced Research Projects Agency (DARPA). 1<br />
The DARPA research focused on team and <strong>in</strong>dividual error <strong>in</strong> m<strong>in</strong>imally <strong>in</strong>vasive<br />
or laparoscopic surgical procedures. This research revealed that <strong>in</strong>dividual<br />
errors were cognitive ra<strong>the</strong>r than purely psychomotor or skill-based. For<br />
example, some surgeons had trouble navigat<strong>in</strong>g three-dimensional anatomical<br />
space us<strong>in</strong>g <strong>the</strong> exist<strong>in</strong>g laparoscopic technology, with <strong>the</strong> result that <strong>the</strong>se<br />
surgeons would identify anatomical structures <strong>in</strong>correctly and perform a surgical<br />
procedure on <strong>the</strong> wrong body part.<br />
O<strong>the</strong>r <strong>in</strong>dividual errors were discovered dur<strong>in</strong>g <strong>the</strong> DARPA studies, but, for<br />
<strong>the</strong> most part, <strong>the</strong>se were spatial navigation and recognition problems for<br />
which <strong>the</strong>re were technological solutions. Team errors, unlike <strong>in</strong>dividual<br />
errors, proved to be more challeng<strong>in</strong>g. The formal and <strong>in</strong>formal hierarchical<br />
structures of operat<strong>in</strong>g rooms did not lend <strong>the</strong>mselves to certa<strong>in</strong> performance<br />
<strong>in</strong>terventions. Generally, junior surgical staff and support personnel were not<br />
will<strong>in</strong>g to confront a senior staff member who was committ<strong>in</strong>g, or was about<br />
to commit, an error.<br />
The culture of <strong>the</strong> operat<strong>in</strong>g room, coupled with <strong>the</strong> social and career structure<br />
of <strong>the</strong> surgical profession, created barriers to certa<strong>in</strong> k<strong>in</strong>ds of communication.<br />
For a surgical resident to <strong>in</strong>form a senior surgeon <strong>in</strong> front of <strong>the</strong> entire<br />
operat<strong>in</strong>g room staff that he was about to cut <strong>the</strong> wrong organ could result <strong>in</strong><br />
career “suicide.” Such a confrontation could have been perceived by <strong>the</strong><br />
senior surgeon as a form of mut<strong>in</strong>y aga<strong>in</strong>st his authority and expertise and a<br />
challenge to <strong>the</strong> social order of <strong>the</strong> operat<strong>in</strong>g room. Although not universal,<br />
this taboo is much more common than surgeons would care to admit. Unlike<br />
<strong>in</strong>dividual errors, purely technological solutions were of little value <strong>in</strong> try<strong>in</strong>g<br />
to solve team errors <strong>in</strong> a surgical environment.<br />
The DARPA surgical research was followed up by a multiyear study of<br />
<strong>in</strong>dividual and team performance of astronauts at <strong>the</strong> National Aeronautics<br />
and Space Adm<strong>in</strong>istration’s (NASA) Johnson Space Center. Results of <strong>the</strong><br />
NASA study, also sponsored by DARPA, were similar to <strong>the</strong> surgical study<br />
1<br />
Rob Johnston, J. Dexter Fletcher and Sunil Bhoyrul, The Use of Virtual Reality to Measure Surgical<br />
Skill Levels.<br />
xiv
INTRODUCTION<br />
with regard to team <strong>in</strong>teractions. Although, on <strong>the</strong> face of it, teams of astronauts<br />
were composed of peers, a social dist<strong>in</strong>ction never<strong>the</strong>less existed<br />
between commander, pilots, and mission specialists.<br />
As with surgery, <strong>the</strong>re was a dis<strong>in</strong>centive for one team member to confront<br />
or criticize ano<strong>the</strong>r, even <strong>in</strong> <strong>the</strong> face of an impend<strong>in</strong>g error. Eighty percent of<br />
<strong>the</strong> current astronauts come from <strong>the</strong> military, which has very specific rules<br />
regard<strong>in</strong>g confrontations, dissent, and criticism. 2 In addition to <strong>the</strong> similarities<br />
<strong>in</strong> behavior aris<strong>in</strong>g from <strong>the</strong>ir common backgrounds, <strong>the</strong> “criticism” taboo<br />
was cont<strong>in</strong>ually re<strong>in</strong>forced throughout <strong>the</strong> astronaut’s career. Virtually any<br />
negative comment on an astronaut’s record was sufficient for him or her to be<br />
assigned to ano<strong>the</strong>r crew, “washed out” of an upcom<strong>in</strong>g mission and recycled<br />
through <strong>the</strong> tra<strong>in</strong><strong>in</strong>g program, or, worse still, released from <strong>the</strong> space program<br />
altoge<strong>the</strong>r.<br />
Taboos are social markers that prohibit specific behaviors <strong>in</strong> order to ma<strong>in</strong>ta<strong>in</strong><br />
and propagate an exist<strong>in</strong>g social structure. Generally, <strong>the</strong>y are unwritten<br />
rules not available to outside observers. Insiders, however, almost always perceive<br />
<strong>the</strong>m simply as <strong>the</strong> way th<strong>in</strong>gs are done, <strong>the</strong> natural social order of <strong>the</strong><br />
organization. To confront taboos is to confront <strong>the</strong> social structure of a culture<br />
or organization.<br />
I mention <strong>the</strong> surgical and astronautical studies for a number of reasons.<br />
Each serves as background for <strong>the</strong> study of <strong>in</strong>telligence analysts. Astronauts<br />
and surgeons have very high performance standards and low error rates. 3 Both<br />
studies highlight o<strong>the</strong>r complex doma<strong>in</strong>s that are <strong>in</strong>terested <strong>in</strong> improv<strong>in</strong>g <strong>the</strong>ir<br />
own professional performance. Both studies reveal <strong>the</strong> need to employ a variety<br />
of research methods to deal with complicated issues, and <strong>the</strong>y suggest that<br />
<strong>the</strong>re are lessons to be learned from o<strong>the</strong>r doma<strong>in</strong>s. Perhaps <strong>the</strong> most tell<strong>in</strong>g<br />
connection is that, because lives are at stake, surgeons and astronauts experience<br />
tremendous <strong>in</strong>ternal and external social pressure to avoid failure. The<br />
same often holds for <strong>in</strong>telligence analysts.<br />
In addition, surgery and astronautics are highly selective and private discipl<strong>in</strong>es.<br />
Although <strong>the</strong>ir work is not secret, both groups tend to be shielded from<br />
<strong>the</strong> outside world: surgeons for reasons of professional selection, tra<strong>in</strong><strong>in</strong>g, and<br />
<strong>the</strong> fiscal realities of malpractice liability; astronauts because <strong>the</strong>ir community<br />
2<br />
National Aeronautics and Space Adm<strong>in</strong>istration, Astronaut Fact Book.<br />
3<br />
NASA has launched missions with <strong>the</strong> shuttle fleet 113 times s<strong>in</strong>ce 1981 and has experienced<br />
two catastrophic failures. It is probable that both of those were mechanical/eng<strong>in</strong>eer<strong>in</strong>g failures<br />
and not <strong>the</strong> result of astronaut error. Surgical report<strong>in</strong>g methods vary from hospital to hospital,<br />
and it is often difficult to determ<strong>in</strong>e <strong>the</strong> specific causes of morbidity and mortality. One longitud<strong>in</strong>al<br />
study of all surgical procedures <strong>in</strong> one medical center puts <strong>the</strong> surgical error rates at that center<br />
between 2.7 percent and 7.5 percent. See Hunter McGuire, Shelton Horsley, David Salter, et al.,<br />
“Measur<strong>in</strong>g and Manag<strong>in</strong>g Quality of Surgery: Statistical vs. Incidental Approaches.”<br />
xv
INTRODUCTION<br />
is so small and <strong>the</strong> selection and tra<strong>in</strong><strong>in</strong>g processes are so demand<strong>in</strong>g. 4 <strong>Intelligence</strong><br />
analysts share many of <strong>the</strong>se organizational and professional circumstances.<br />
The <strong>Intelligence</strong> <strong>Community</strong> is relatively small, highly selective, and<br />
largely shielded from public view. For its practitioners, <strong>in</strong>telligence work is a<br />
cognitively-demand<strong>in</strong>g and high-risk profession that can lead to public policy<br />
that streng<strong>the</strong>ns <strong>the</strong> nation or puts it at greater risk. Because <strong>the</strong> consequences<br />
of failure are so great, <strong>in</strong>telligence professionals cont<strong>in</strong>ually feel significant<br />
<strong>in</strong>ternal and external pressure to avoid it. One consequence of this pressure is<br />
that <strong>the</strong>re has been a long-stand<strong>in</strong>g bureaucratic resistance to putt<strong>in</strong>g <strong>in</strong> place a<br />
systematic program for improv<strong>in</strong>g analytical performance. Accord<strong>in</strong>g to<br />
71 percent of <strong>the</strong> people I <strong>in</strong>terviewed, however, that resistance has dim<strong>in</strong>ished<br />
significantly s<strong>in</strong>ce September 2001.<br />
It is not difficult to understand <strong>the</strong> historical resistance to implement<strong>in</strong>g<br />
such a performance improvement program. Simply put, a program explicitly<br />
designed to improve human performance implies that human performance<br />
needs improv<strong>in</strong>g, an allegation that risks considerable political and <strong>in</strong>stitutional<br />
resistance. Not only does performance improvement imply that <strong>the</strong> system<br />
is not optimal, <strong>the</strong> necessary scrut<strong>in</strong>y of practice and performance would<br />
require exam<strong>in</strong><strong>in</strong>g sources and methods <strong>in</strong> detail throughout <strong>the</strong> <strong>Intelligence</strong><br />
<strong>Community</strong>. Although this scrut<strong>in</strong>y would be wholly <strong>in</strong>ternal to <strong>the</strong> community,<br />
<strong>the</strong> concept runs counter to a culture of secrecy and compartmentalization.<br />
The conflict between secrecy, a necessary condition for <strong>in</strong>telligence, and<br />
openness, a necessary condition for performance improvement, was a recurr<strong>in</strong>g<br />
<strong>the</strong>me I observed dur<strong>in</strong>g this research. Any organization that requires<br />
secrecy to perform its duties will struggle with and often reject openness, even<br />
at <strong>the</strong> expense of efficacy. Despite this, and to <strong>the</strong>ir credit, a number of small<br />
groups with<strong>in</strong> <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong> have tasked <strong>the</strong>mselves with creat<strong>in</strong>g<br />
formal and <strong>in</strong>formal ties with <strong>the</strong> nation’s academic, non-profit, and <strong>in</strong>dustrial<br />
communities. In addition, <strong>the</strong>re has been an appreciable <strong>in</strong>crease <strong>in</strong> <strong>the</strong><br />
use of alternative analyses and open-source materials.<br />
These efforts alone may not be sufficient to alter <strong>the</strong> historical culture of<br />
secrecy, but <strong>the</strong>y do re<strong>in</strong>force <strong>the</strong> idea that <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong> itself<br />
has a responsibility to reconsider <strong>the</strong> relationship between secrecy, openness,<br />
and efficacy. This is especially true as it relates to <strong>the</strong> community’s performance<br />
and <strong>the</strong> occurrence of errors and failure. External oversight and public<br />
debate will not solve <strong>the</strong>se issues; <strong>the</strong> desire to improve <strong>the</strong> <strong>Intelligence</strong> Com-<br />
4<br />
There are currently 109 active US astronauts and 36 management astronauts. See National Aeronautics<br />
and Space Adm<strong>in</strong>istration-Johnson Space Center career astronaut biographies.<br />
xvi
INTRODUCTION<br />
munity’s performance needs to come from with<strong>in</strong>. Once <strong>the</strong> determ<strong>in</strong>ation has<br />
been found and <strong>the</strong> necessary policy guidel<strong>in</strong>es put <strong>in</strong> place, it is <strong>in</strong>cumbent<br />
upon <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong> to f<strong>in</strong>d and utilize <strong>the</strong> <strong>in</strong>ternal and external<br />
resources necessary to create a performance improvement <strong>in</strong>frastructure.<br />
Scope<br />
This project was designed explicitly as an applied research program. In<br />
many respects, it resembles an assessment of organizational needs and a gap<br />
analysis, <strong>in</strong> that it was <strong>in</strong>tended to identify and describe conditions and variables<br />
that affect <strong>in</strong>telligence analysis and <strong>the</strong>n to identify needs, specifications,<br />
and requirements for <strong>the</strong> development of tools, techniques, and<br />
procedures to reduce analytic error. Based on <strong>the</strong>se f<strong>in</strong>d<strong>in</strong>gs, I was to make<br />
recommendations to improve analytic performance.<br />
In previous human performance-related research conducted <strong>in</strong> <strong>the</strong> military,<br />
medical, and astronautic fields, I have found <strong>in</strong> place—especially <strong>in</strong> <strong>the</strong> military—a<br />
large social science literature, an elaborate tra<strong>in</strong><strong>in</strong>g doctr<strong>in</strong>e, and welldeveloped<br />
quantitative and qualitative research programs. In addition to<br />
research literature and programs, <strong>the</strong>se three discipl<strong>in</strong>es have substantial performance<br />
improvement programs. This was not <strong>the</strong> case with <strong>the</strong> <strong>Intelligence</strong><br />
<strong>Community</strong>.<br />
This is not to say that an <strong>in</strong>telligence literature does not exist but ra<strong>the</strong>r that<br />
<strong>the</strong> literature that does exist has been focused to a greater extent on case studies<br />
than on <strong>the</strong> actual process of <strong>in</strong>telligence analysis. 5 The vast majority of<br />
<strong>the</strong> available literature is about history, <strong>in</strong>ternational relations, and political<br />
science. Texts that address analytic methodology do exist, and it is worth not<strong>in</strong>g<br />
that <strong>the</strong>re are quantitative studies, such as that by Robert Folker, that compare<br />
<strong>the</strong> effectiveness of different analytic methods for solv<strong>in</strong>g a given<br />
analytic problem. Folker’s study demonstrates that objective, quantitative, and<br />
controlled research to determ<strong>in</strong>e <strong>the</strong> effectiveness of analytic methods is possible.<br />
6<br />
The literature that deals with <strong>the</strong> process of <strong>in</strong>telligence analysis tends to be<br />
personal and idiosyncratic, reflect<strong>in</strong>g an <strong>in</strong>dividualistic approach to problem<br />
solv<strong>in</strong>g. This is not surpris<strong>in</strong>g. The <strong>Intelligence</strong> <strong>Community</strong> is made up of a<br />
variety of discipl<strong>in</strong>es, each with its own analytic methodology. The organizational<br />
assumption has been that, <strong>in</strong> a multidiscipl<strong>in</strong>ary environment, <strong>in</strong>telli-<br />
5<br />
There are exceptions. See <strong>the</strong> appendix.<br />
6<br />
MSgt. Robert D. Folker, <strong>Intelligence</strong> Analysis <strong>in</strong> Theater Jo<strong>in</strong>t <strong>Intelligence</strong> Centers. Folker’s<br />
study conta<strong>in</strong>s a methodological flaw <strong>in</strong> that it does not describe one of <strong>the</strong> <strong>in</strong>dependent variables<br />
(<strong>in</strong>tuitive method), leav<strong>in</strong>g <strong>the</strong> dependent variable (test scores) <strong>in</strong> doubt.<br />
xvii
INTRODUCTION<br />
gence analysts would use analytic methods and tools from <strong>the</strong>ir own doma<strong>in</strong><br />
<strong>in</strong> order to analyze and solve <strong>in</strong>telligence problems. When <strong>in</strong>terdiscipl<strong>in</strong>ary<br />
problems have arisen, <strong>the</strong> organizational assumption has been that a variety of<br />
analytic methods would be employed, result<strong>in</strong>g <strong>in</strong> a “best fit” syn<strong>the</strong>sis.<br />
This <strong>in</strong>dividualistic approach to analysis has resulted <strong>in</strong> a great variety of<br />
analytic methods—I identified at least 160 <strong>in</strong> my research for this paper—but<br />
it has not led to <strong>the</strong> development of a standardized analytic doctr<strong>in</strong>e. That is,<br />
<strong>the</strong>re is no body of research across <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong> assert<strong>in</strong>g that<br />
method X is <strong>the</strong> most effective method for solv<strong>in</strong>g case one and that method Y<br />
is <strong>the</strong> most effective method for solv<strong>in</strong>g case two. 7<br />
The utility of a standardized analytic doctr<strong>in</strong>e is that it enables an organization<br />
to determ<strong>in</strong>e performance requirements, a standard level of <strong>in</strong>stitutional<br />
expertise, and <strong>in</strong>dividual performance metrics for <strong>the</strong> evaluation and development<br />
of new analytic methodologies. 8 Ultimately, without such an analytic<br />
basel<strong>in</strong>e, one cannot assess <strong>the</strong> effectiveness of any new or proposed analytic<br />
method, tool, technology, reorganization, or <strong>in</strong>tervention. Without standardized<br />
analytic doctr<strong>in</strong>e, analysts are left to <strong>the</strong> ra<strong>the</strong>r slow and tedious process<br />
of trial and error throughout <strong>the</strong>ir careers.<br />
Generally, <strong>in</strong> research literature, one f<strong>in</strong>ds a taxonomy, or matrix, of <strong>the</strong><br />
variables that affect <strong>the</strong> object under study. Taxonomies help to standardize<br />
def<strong>in</strong>itions and <strong>in</strong>form future research by establish<strong>in</strong>g a research “road map.”<br />
They po<strong>in</strong>t out areas of <strong>in</strong>terest and research priorities and help researchers<br />
place <strong>the</strong>ir own research programs <strong>in</strong> context. In my search of <strong>the</strong> <strong>in</strong>telligence<br />
literature, I found no taxonomy of <strong>the</strong> variables that affect <strong>in</strong>telligence analysis.<br />
Follow<strong>in</strong>g <strong>the</strong> literature review, I undertook to develop work<strong>in</strong>g def<strong>in</strong>itions<br />
and a taxonomy <strong>in</strong> order to systematize <strong>the</strong> research process. Readers will f<strong>in</strong>d<br />
<strong>the</strong> work<strong>in</strong>g def<strong>in</strong>itions <strong>in</strong> <strong>the</strong> first chapter. The second chapter highlights <strong>the</strong><br />
<strong>the</strong> broader f<strong>in</strong>d<strong>in</strong>gs and implications of this ethnographic study. Because <strong>the</strong><br />
first two chapters conta<strong>in</strong> many quotes from my <strong>in</strong>terviews and workshops,<br />
<strong>the</strong>y illustrate <strong>the</strong> tone and nature of <strong>the</strong> post-9/11 environment <strong>in</strong> which I<br />
worked.<br />
The taxonomy that grew out of this work was first described <strong>in</strong> an article for<br />
<strong>the</strong> CSI journal, Studies <strong>in</strong> <strong>Intelligence</strong>, and is presented here as Chapter<br />
7<br />
There is no s<strong>in</strong>gle <strong>Intelligence</strong> <strong>Community</strong> basic analytic tra<strong>in</strong><strong>in</strong>g program. There is, however,<br />
community use of advanced analytic courses at both <strong>the</strong> <strong>CIA</strong> University and <strong>the</strong> Jo<strong>in</strong>t Military<br />
<strong>Intelligence</strong> College. The Generic <strong>Intelligence</strong> Tra<strong>in</strong><strong>in</strong>g Initiative is a recent attempt to standardize<br />
certa<strong>in</strong> law enforcement <strong>in</strong>telligence analysis tra<strong>in</strong><strong>in</strong>g programs through a basic law enforcement<br />
analyst tra<strong>in</strong><strong>in</strong>g curriculum. The program has been developed by <strong>the</strong> Tra<strong>in</strong><strong>in</strong>g Advisory<br />
Council, under <strong>the</strong> Counterdrug <strong>Intelligence</strong> Coord<strong>in</strong>at<strong>in</strong>g Group and <strong>the</strong> Justice Tra<strong>in</strong><strong>in</strong>g Center.<br />
8<br />
See <strong>the</strong> appendix.<br />
xviii
INTRODUCTION<br />
Three. In addition to <strong>the</strong> normal journal review process, I circulated a draft of<br />
<strong>the</strong> taxonomy among 55 academics and <strong>in</strong>telligence professionals and <strong>in</strong>corporated<br />
<strong>the</strong>ir suggestions <strong>in</strong> a revised version that went to press. This is not to<br />
assert that <strong>the</strong> taxonomy is f<strong>in</strong>al; <strong>the</strong> utility of any taxonomy is that it can be<br />
revised and expanded as new research f<strong>in</strong>d<strong>in</strong>gs become available. The chapter<br />
by Dr. Judith Meister Johnston that follows offers an alternative model—more<br />
complex and possibly more accurate than <strong>the</strong> traditional <strong>in</strong>telligence cycle—<br />
for look<strong>in</strong>g at <strong>the</strong> dynamics of <strong>the</strong> <strong>in</strong>telligence process, <strong>in</strong> effect <strong>the</strong> <strong>in</strong>terrelationships<br />
of many elements of <strong>the</strong> taxonomy<br />
The follow<strong>in</strong>g chapters, prepared by me and o<strong>the</strong>r able colleagues, were<br />
developed around o<strong>the</strong>r variables <strong>in</strong> <strong>the</strong> taxonomy and offer suggestions for<br />
improvement <strong>in</strong> those specific areas. One of <strong>the</strong>m—Chapter Five, on <strong>in</strong>tegrat<strong>in</strong>g<br />
methodologists and substantive experts <strong>in</strong> research teams—also appeared<br />
<strong>in</strong> Studies <strong>in</strong> <strong>Intelligence</strong>. Chapter N<strong>in</strong>e conta<strong>in</strong>s several broad recommendations,<br />
<strong>in</strong>clud<strong>in</strong>g suggestions for fur<strong>the</strong>r research.<br />
To <strong>the</strong> extent possible, I tried to avoid us<strong>in</strong>g professional jargon. Even so,<br />
<strong>the</strong> reader will still f<strong>in</strong>d a number of specific technical terms, and, <strong>in</strong> those<br />
cases, I have <strong>in</strong>cluded <strong>the</strong>ir discipl<strong>in</strong>ary def<strong>in</strong>itions as footnotes.<br />
A Work <strong>in</strong> Progress<br />
In some respects, it may seem strange or unusual to have an anthropologist<br />
perform this type of work ra<strong>the</strong>r than an <strong>in</strong>dustrial/organizational psychologist<br />
or some o<strong>the</strong>r specialist <strong>in</strong> professional performance improvement or bus<strong>in</strong>ess<br />
processes. The common perception of cultural anthropology is one of fieldwork<br />
among <strong>in</strong>digenous peoples. Much has changed dur<strong>in</strong>g <strong>the</strong> past 40 years,<br />
however. Today, <strong>the</strong>re are many practitioners and professional associations<br />
devoted to <strong>the</strong> application of anthropology and its field methods to practical<br />
problem-solv<strong>in</strong>g <strong>in</strong> modern or post<strong>in</strong>dustrial society. 9<br />
It is difficult for any modern anthropological study to escape <strong>the</strong> legacy of<br />
Margaret Mead. She looms as large over 20th century anthropology as does<br />
Sherman Kent over <strong>the</strong> <strong>in</strong>telligence profession. Although Franz Boas is arguably<br />
<strong>the</strong> fa<strong>the</strong>r of American anthropology and was Margaret Mead’s mentor,<br />
hers is <strong>the</strong> name everyone recognizes and connects to ethnography. 10 Chances<br />
are, if one has read anthropological texts, one has read Mead.<br />
9<br />
The Society for Applied Anthropology and <strong>the</strong> National Association for <strong>the</strong> Practice of Anthropology<br />
section of <strong>the</strong> American Anthropological Association are <strong>the</strong> two pr<strong>in</strong>cipal anthropological<br />
groups. Ano<strong>the</strong>r group is <strong>the</strong> Inter-University Sem<strong>in</strong>ar on Armed Forces and Society, a<br />
professional organization represent<strong>in</strong>g 700 social science fellows, <strong>in</strong>clud<strong>in</strong>g practic<strong>in</strong>g anthropologists,<br />
apply<strong>in</strong>g <strong>the</strong>ir research methods to issues <strong>in</strong> <strong>the</strong> military.<br />
xix
INTRODUCTION<br />
I mention Mead not only because my work draws heavily on hers, but also<br />
because of her impact on <strong>the</strong> discipl<strong>in</strong>e and its direction. She moved from traditional<br />
cultural anthropological fieldwork <strong>in</strong> <strong>the</strong> South Pacific to problemoriented<br />
applied anthropology dur<strong>in</strong>g World War II. She was <strong>the</strong> founder of<br />
<strong>the</strong> Institute for Intercultural Studies and a major contributor to <strong>the</strong> Cold War<br />
RAND series that attempted to describe <strong>the</strong> Soviet character. She also pioneered<br />
many of <strong>the</strong> research methods that are used <strong>in</strong> applied anthropology<br />
today. I mention her work also as an illustrative po<strong>in</strong>t. After two years of field<br />
research <strong>in</strong> <strong>the</strong> South Pacific, she wrote at least five books and could possibly<br />
have written more.<br />
As I look over <strong>the</strong> stacks of documentation for this study, it occurs to me<br />
that, given <strong>the</strong> various constra<strong>in</strong>ts of <strong>the</strong> fellowship, <strong>the</strong>re is more material<br />
here than I will be able to address <strong>in</strong> any one text. There are <strong>the</strong> notes from<br />
489 <strong>in</strong>terviews, direct observations, participant observations, and focus<br />
groups; <strong>the</strong>re are personal letters, e-mail exchanges, and archival material; and<br />
<strong>the</strong>re are my own notes track<strong>in</strong>g <strong>the</strong> progress of <strong>the</strong> work. Moreover, <strong>the</strong> fieldwork<br />
cont<strong>in</strong>ues. As I write this, I am schedul<strong>in</strong>g more <strong>in</strong>terviews, more observations,<br />
and yet more fieldwork.<br />
This text, <strong>the</strong>n, is more a progress report than a f<strong>in</strong>al report <strong>in</strong> any traditional<br />
sense. It reflects f<strong>in</strong>d<strong>in</strong>gs and recommendations to date and is <strong>in</strong> no way<br />
comprehensive. F<strong>in</strong>ally, based as it is on my own research <strong>in</strong>terests and<br />
research opportunities, it is but one piece of a much larger puzzle.<br />
10<br />
Boas (1858–1942) developed <strong>the</strong> l<strong>in</strong>guistic and cultural components of ethnology. His most<br />
notable work was Race, Language, and <strong>Culture</strong> (1940).<br />
xx
PART I<br />
Research F<strong>in</strong>d<strong>in</strong>gs<br />
1
eth•nog•ra•phy\n [F ethnographie, fr. ethno- + -graphie -graphy] (1834) :<br />
<strong>the</strong> study and systematic record<strong>in</strong>g of human cultures: also: a descriptive work<br />
produced from such research. (Merriam Webster’s Collegiate Dictionary,<br />
Eleventh Edition)<br />
2
CHAPTER ONE<br />
Def<strong>in</strong>itions<br />
Because I conducted human performance–related fieldwork before I came<br />
to this project, I carried <strong>in</strong>to it a certa<strong>in</strong> amount of experiential bias, or “cognitive<br />
baggage.” The research f<strong>in</strong>d<strong>in</strong>gs from those o<strong>the</strong>r studies could bias my<br />
perspective and research approach with<strong>in</strong> <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong>. For<br />
example, surgeons and astronauts do not need to deal with <strong>in</strong>tentionally<br />
deceptive data. Patients are not try<strong>in</strong>g to “hide” <strong>the</strong>ir illnesses from surgeons,<br />
and spacecraft are not th<strong>in</strong>k<strong>in</strong>g adversaries <strong>in</strong>tent on deny<strong>in</strong>g astronauts critical<br />
pieces of <strong>in</strong>formation. This one difference may mean that <strong>in</strong>telligence<br />
analysis is much more cognitively challeng<strong>in</strong>g than <strong>the</strong> o<strong>the</strong>r two cases and<br />
that <strong>the</strong> requisite psychomotor skills are significantly less important. In an<br />
effort to counteract <strong>the</strong> biases of experience, I will attempt to be explicit about<br />
my own def<strong>in</strong>itions <strong>in</strong> this work.<br />
Work<strong>in</strong>g Def<strong>in</strong>itions<br />
The three ma<strong>in</strong> def<strong>in</strong>itions used <strong>in</strong> this work do not necessarily represent<br />
def<strong>in</strong>itions derived from <strong>the</strong> whole of <strong>the</strong> <strong>in</strong>telligence literature. Although<br />
some of <strong>the</strong> def<strong>in</strong>itions used <strong>in</strong> this work are based on <strong>the</strong> Q-sort survey of <strong>the</strong><br />
<strong>in</strong>telligence literature described later, some are based on <strong>the</strong> 489 <strong>in</strong>terviews,<br />
focus groups, and two years of direct and participant observations collected<br />
dur<strong>in</strong>g this project.<br />
3
CHAPTER ONE<br />
Def<strong>in</strong>ition 1: <strong>Intelligence</strong> is secret state or group activity to understand<br />
or <strong>in</strong>fluence foreign or domestic entities.<br />
The above def<strong>in</strong>ition of <strong>in</strong>telligence, as used <strong>in</strong> this text, is a slightly modified<br />
version of <strong>the</strong> one that appeared <strong>in</strong> Michael Warner’s work <strong>in</strong> a recent<br />
article <strong>in</strong> Studies In <strong>Intelligence</strong>. 1 Warner reviews and syn<strong>the</strong>sizes a number<br />
of previous attempts to def<strong>in</strong>e <strong>the</strong> discipl<strong>in</strong>e of <strong>in</strong>telligence and comes to <strong>the</strong><br />
conclusion that “<strong>Intelligence</strong> is secret state activity to understand or <strong>in</strong>fluence<br />
foreign entities.”<br />
Warner’s syn<strong>the</strong>sis seems to focus on strategic <strong>in</strong>telligence, but it is also<br />
logically similar to actionable <strong>in</strong>telligence (both tactical and operational)<br />
designed to <strong>in</strong>fluence <strong>the</strong> cognition or behavior of an adversary. 2 This syn<strong>the</strong>sis<br />
captures most of <strong>the</strong> elements of actionable <strong>in</strong>telligence without be<strong>in</strong>g too<br />
restrictive or too open-ended, and those I asked to def<strong>in</strong>e <strong>the</strong> word found its<br />
elements, <strong>in</strong> one form or ano<strong>the</strong>r, to be generally acceptable. The modified<br />
version proposed here is based on Warner’s def<strong>in</strong>ition and <strong>the</strong> <strong>in</strong>terview and<br />
observation data collected among <strong>the</strong> law enforcement elements of <strong>the</strong> <strong>in</strong>telligence<br />
agencies. These elements confront adversaries who are not nation states<br />
or who may not be foreign entities. With this <strong>in</strong> m<strong>in</strong>d, I chose to def<strong>in</strong>e <strong>in</strong>telligence<br />
somewhat more broadly, to <strong>in</strong>clude nonstate actors and domestic <strong>in</strong>telligence<br />
activities performed with<strong>in</strong> <strong>the</strong> United States.<br />
Def<strong>in</strong>ition 2: <strong>Intelligence</strong> analysis is <strong>the</strong> application of <strong>in</strong>dividual<br />
and collective cognitive methods to weigh data and test hypo<strong>the</strong>ses<br />
with<strong>in</strong> a secret socio-cultural context.<br />
This mean<strong>in</strong>g of <strong>in</strong>telligence analysis was harder to establish, and readers<br />
will f<strong>in</strong>d a more comprehensive review <strong>in</strong> <strong>the</strong> follow<strong>in</strong>g chapter on develop<strong>in</strong>g<br />
an <strong>in</strong>telligence taxonomy. In short, <strong>the</strong> literature tends to divide <strong>in</strong>telligence<br />
analysis <strong>in</strong>to “how-to” tools and techniques or cognitive processes. This is not<br />
to say that <strong>the</strong>se items are mutually exclusive; many authors see <strong>the</strong> tools and<br />
techniques of analysis as cognitive processes <strong>in</strong> <strong>the</strong>mselves and are reluctant<br />
to place <strong>the</strong>m <strong>in</strong> different categories. Some authors tend to perceive <strong>in</strong>telligence<br />
analysis as essentially an <strong>in</strong>dividual cognitive process or processes. 3<br />
My work dur<strong>in</strong>g this study conv<strong>in</strong>ced me of <strong>the</strong> importance of mak<strong>in</strong>g<br />
explicit someth<strong>in</strong>g that is not well described <strong>in</strong> <strong>the</strong> literature, namely, <strong>the</strong> very<br />
1<br />
Michael Warner, “Wanted: A Def<strong>in</strong>ition of ‘<strong>Intelligence</strong>’,” Studies <strong>in</strong> <strong>Intelligence</strong> 46, no. 3<br />
(2002): 15–22.<br />
2<br />
US Jo<strong>in</strong>t Forces Command, Department of Defense Dictionary of Military and Associated<br />
Terms.<br />
3<br />
The appendix lists literature devoted to each of <strong>the</strong>se areas.<br />
4
DEFINITIONS<br />
<strong>in</strong>teractive, dynamic, and social nature of <strong>in</strong>telligence analysis. The <strong>in</strong>terview<br />
participants were not asked to def<strong>in</strong>e <strong>in</strong>telligence analysis as such; ra<strong>the</strong>r, <strong>the</strong>y<br />
were asked to describe and expla<strong>in</strong> <strong>the</strong> process <strong>the</strong>y used to perform analysis.<br />
The <strong>in</strong>terview data were <strong>the</strong>n triangulated with <strong>the</strong> direct and participant<br />
observation data collected dur<strong>in</strong>g this study. 4<br />
Despite <strong>the</strong> seem<strong>in</strong>gly private and psychological nature of analysis as<br />
def<strong>in</strong>ed <strong>in</strong> <strong>the</strong> literature, what I found was a great deal of <strong>in</strong>formal, yet purposeful<br />
collaboration dur<strong>in</strong>g which <strong>in</strong>dividuals began to make sense of raw<br />
data by negotiat<strong>in</strong>g mean<strong>in</strong>g among <strong>the</strong> historical record, <strong>the</strong>ir peers, and <strong>the</strong>ir<br />
supervisors. Here, from <strong>the</strong> <strong>in</strong>terviews, is a typical description of <strong>the</strong> analytic<br />
process:<br />
When a request comes <strong>in</strong> from a consumer to answer some question,<br />
<strong>the</strong> first th<strong>in</strong>g I do is to read up on <strong>the</strong> analytic l<strong>in</strong>e. [I] check <strong>the</strong><br />
previous publications and <strong>the</strong> data. Then, I read through <strong>the</strong> question<br />
aga<strong>in</strong> and f<strong>in</strong>d where <strong>the</strong>re are l<strong>in</strong>ks to previous products.<br />
When I th<strong>in</strong>k I have an answer, I get toge<strong>the</strong>r with my group and ask<br />
<strong>the</strong>m what <strong>the</strong>y th<strong>in</strong>k. We talk about it for a while and come to some<br />
consensus on its mean<strong>in</strong>g and <strong>the</strong> best way to answer <strong>the</strong> consumer’s<br />
question. I write it up, pass it around here, and send it out<br />
for review. 5<br />
The cognitive element of this basic description, “when I th<strong>in</strong>k I have an<br />
answer,” is a vague impression of <strong>the</strong> psychological processes that occur dur<strong>in</strong>g<br />
analysis. The elements that are not vague are <strong>the</strong> historical, organizational,<br />
and social elements of analysis. The analyst checks <strong>the</strong> previous written products<br />
that have been given to consumers <strong>in</strong> <strong>the</strong> past. That is, <strong>the</strong> analyst looks<br />
for <strong>the</strong> accepted organizational response before generat<strong>in</strong>g analytic hypo<strong>the</strong>ses.<br />
The organizational-historical context is critical to understand<strong>in</strong>g <strong>the</strong> mean<strong>in</strong>g,<br />
context, and process of <strong>in</strong>telligence analysis. There are real organizational<br />
and political consequences associated with chang<strong>in</strong>g official analytic f<strong>in</strong>d<strong>in</strong>gs<br />
and releas<strong>in</strong>g <strong>the</strong>m to consumers. The organizational consequences are associated<br />
with challeng<strong>in</strong>g o<strong>the</strong>r doma<strong>in</strong> experts (<strong>in</strong>clud<strong>in</strong>g peers and supervisors).<br />
The potential political consequences arise when consumers beg<strong>in</strong> to question<br />
<strong>the</strong> veracity and consistency of current or previous <strong>in</strong>telligence report<strong>in</strong>g.<br />
Accurate or not, <strong>the</strong>re is a general impression with<strong>in</strong> <strong>the</strong> analytic community<br />
4<br />
In research, triangulation refers to <strong>the</strong> application of a comb<strong>in</strong>ation of two or more <strong>the</strong>ories, data<br />
sources, methods, or <strong>in</strong>vestigators to develop a s<strong>in</strong>gle construct <strong>in</strong> a study of a s<strong>in</strong>gle phenomenon.<br />
5<br />
<strong>Intelligence</strong> analyst’s comment dur<strong>in</strong>g an ethnographic <strong>in</strong>terview. Such quotes are <strong>in</strong>dented and<br />
italicized <strong>in</strong> this way throughout <strong>the</strong> text and will not be fur<strong>the</strong>r identified; quotes attributable to<br />
o<strong>the</strong>rs will be identified as such.<br />
5
CHAPTER ONE<br />
that consumers of <strong>in</strong>telligence products require a static “f<strong>in</strong>al say” on a given<br />
topic <strong>in</strong> order to generate policy. This sort of organizational-historical context,<br />
coupled with <strong>the</strong> impression that consumers must have a f<strong>in</strong>al verdict, tends to<br />
create and re<strong>in</strong>force a risk-averse culture.<br />
Once <strong>the</strong> organizational context for answer<strong>in</strong>g any given question is understood,<br />
<strong>the</strong> analyst beg<strong>in</strong>s to consider raw data specific to answer<strong>in</strong>g <strong>the</strong> new<br />
question. In so do<strong>in</strong>g, <strong>the</strong> analyst runs <strong>the</strong> risk of confirmation biases. That is,<br />
<strong>in</strong>stead of generat<strong>in</strong>g new hypo<strong>the</strong>ses based solely on raw data and <strong>the</strong>n<br />
weigh<strong>in</strong>g <strong>the</strong> evidence to confirm or refute those hypo<strong>the</strong>ses, <strong>the</strong> analyst<br />
beg<strong>in</strong>s look<strong>in</strong>g for evidence to confirm <strong>the</strong> exist<strong>in</strong>g hypo<strong>the</strong>sis, which came<br />
from previous <strong>in</strong>telligence products or was <strong>in</strong>ferred dur<strong>in</strong>g <strong>in</strong>teractions with<br />
colleagues. The process is re<strong>in</strong>forced socially as <strong>the</strong> analyst discusses a new<br />
f<strong>in</strong>d<strong>in</strong>g with group members and superiors, often <strong>the</strong> very people who collaborated<br />
<strong>in</strong> produc<strong>in</strong>g <strong>the</strong> previous <strong>in</strong>telligence products. Similarly, those who<br />
review <strong>the</strong> product may have been <strong>the</strong> reviewers who passed on <strong>the</strong> analyst’s<br />
previous efforts.<br />
This is not to say that <strong>the</strong> exist<strong>in</strong>g <strong>in</strong>telligence products are necessarily <strong>in</strong>accurate.<br />
In fact, <strong>the</strong>y are very often accurate. This is merely meant to po<strong>in</strong>t out<br />
that risk aversion, organizational-historical context, and socialization are all<br />
part of <strong>the</strong> analytic process. One cannot separate <strong>the</strong> cognitive aspects of <strong>in</strong>telligence<br />
analysis from its cultural context.<br />
Def<strong>in</strong>ition 3: <strong>Intelligence</strong> errors are factual <strong>in</strong>accuracies <strong>in</strong> analysis<br />
result<strong>in</strong>g from poor or miss<strong>in</strong>g data; <strong>in</strong>telligence failure is systemic<br />
organizational surprise result<strong>in</strong>g from <strong>in</strong>correct, miss<strong>in</strong>g,<br />
discarded, or <strong>in</strong>adequate hypo<strong>the</strong>ses.<br />
Dur<strong>in</strong>g <strong>in</strong>terviews, participants were asked to expla<strong>in</strong> <strong>the</strong>ir understand<strong>in</strong>g of<br />
<strong>the</strong> terms <strong>in</strong>telligence error and <strong>in</strong>telligence failure. There was little consensus<br />
regard<strong>in</strong>g <strong>the</strong> def<strong>in</strong>itions of error and failure with<strong>in</strong> <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong><br />
or with<strong>in</strong> <strong>the</strong> larger <strong>in</strong>terview sample. Here are some sample responses:<br />
I don’t know what <strong>the</strong>y mean.<br />
There are no such th<strong>in</strong>gs. There’s only policy failure.<br />
You report what you know, and, if you don’t know someth<strong>in</strong>g, <strong>the</strong>n it<br />
isn’t error or failure. It’s just miss<strong>in</strong>g <strong>in</strong>formation.<br />
Failure is forecast<strong>in</strong>g <strong>the</strong> wrong th<strong>in</strong>g.<br />
Failure is report<strong>in</strong>g <strong>the</strong> wrong th<strong>in</strong>g.<br />
Error is forecast<strong>in</strong>g <strong>the</strong> wrong th<strong>in</strong>g.<br />
6
DEFINITIONS<br />
Error is report<strong>in</strong>g <strong>the</strong> wrong th<strong>in</strong>g.<br />
A failure is someth<strong>in</strong>g catastrophic, and an error is just a mistake.<br />
Error is about facts; failure is about surprise.<br />
Error is when nobody notices, and failure is when everybody<br />
notices.<br />
Some responses disavowed <strong>the</strong> existence of <strong>in</strong>telligence error and failure;<br />
some placed <strong>the</strong> terms <strong>in</strong> <strong>the</strong> broader context of policy and decisionmak<strong>in</strong>g;<br />
some <strong>in</strong>terchanged <strong>the</strong> two terms at random; some def<strong>in</strong>ed <strong>the</strong> terms accord<strong>in</strong>g<br />
to <strong>the</strong>ir outcomes or consequences. Despite <strong>the</strong> variability of <strong>the</strong><br />
responses, two trends emerged: novice analysts tended to worry about be<strong>in</strong>g<br />
factually <strong>in</strong>accurate; senior analysts, managers, and consumers, tended to<br />
worry about be<strong>in</strong>g surprised. Often, participants’ responses were not def<strong>in</strong>itions<br />
at all but statements meant to represent familiar historical examples:<br />
The attack on Pearl Harbor.<br />
The Ch<strong>in</strong>ese send<strong>in</strong>g combat troops <strong>in</strong>to Korea.<br />
The Tet Offensive.<br />
The Soviet <strong>in</strong>vasion of Afghanistan.<br />
The collapse of <strong>the</strong> Soviet Union.<br />
The Indian nuclear test.<br />
September Eleventh.<br />
The danger of def<strong>in</strong><strong>in</strong>g by example is that each case is contextually unique<br />
and can be argued ad <strong>in</strong>f<strong>in</strong>itum. What is important about <strong>the</strong>se examples as a<br />
whole is that <strong>the</strong>y all <strong>in</strong>dicate one central and recurr<strong>in</strong>g <strong>the</strong>me. Specifically, all<br />
<strong>the</strong>se examples signify surprise—<strong>in</strong> some cases, <strong>in</strong>telligence surprise; <strong>in</strong> o<strong>the</strong>r<br />
cases, military, civil, and political surprise. Even if <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong><br />
itself was not surprised by one of <strong>the</strong>se events, it was unable to conv<strong>in</strong>ce<br />
<strong>the</strong> military, civil, and political consumers of <strong>in</strong>telligence that <strong>the</strong>se events<br />
might occur; <strong>in</strong> which case, <strong>the</strong> failure was one of communication and persuasion.<br />
When I began this study, my own def<strong>in</strong>ition of error and failure derived<br />
from <strong>the</strong> psychological and cognitive discipl<strong>in</strong>es. Specifically, I took it that<br />
human error and failure are related to measures of cognitive and psychomotor<br />
accuracy, commission of error be<strong>in</strong>g at one end of <strong>the</strong> accuracy scale and<br />
omission or not perform<strong>in</strong>g <strong>the</strong> correct action be<strong>in</strong>g at <strong>the</strong> o<strong>the</strong>r. 6<br />
6<br />
See Appendix A for a list of literature on error.<br />
7
CHAPTER ONE<br />
Dur<strong>in</strong>g <strong>the</strong> <strong>in</strong>terviews for this study, I soon found that <strong>the</strong> psychological<br />
def<strong>in</strong>ition was <strong>in</strong>sufficient. The psychological def<strong>in</strong>ition took <strong>in</strong>to account <strong>the</strong><br />
cognitive and psychomotor components of task-structure, time-to-task, and<br />
accuracy-of-task as measures of errors and error rates, but it did not fully take<br />
<strong>in</strong>to account <strong>the</strong> notion of surprise. 7 Surprise is <strong>the</strong> occurrence of someth<strong>in</strong>g<br />
unexpected or unanticipated. It is not precisely commission or omission; it<br />
<strong>in</strong>dicates, ra<strong>the</strong>r, <strong>the</strong> absence of contraven<strong>in</strong>g cognitive processes. Measures<br />
of accuracy may account for factual errors <strong>in</strong> <strong>the</strong> <strong>in</strong>telligence doma<strong>in</strong>, but<br />
measures of accuracy are <strong>in</strong>sufficient to account for surprise events and <strong>in</strong>telligence<br />
failure.<br />
To put this <strong>in</strong> context, an analyst, while account<strong>in</strong>g successfully for an<br />
adversary’s capability, may misjudge that adversary’s <strong>in</strong>tention, not because<br />
of what is cognitively available, but because of what is cognitively absent.<br />
The failure to determ<strong>in</strong>e an adversary’s <strong>in</strong>tention may simply be <strong>the</strong> result of<br />
miss<strong>in</strong>g <strong>in</strong>formation or, just as likely, it may be <strong>the</strong> result of miss<strong>in</strong>g hypo<strong>the</strong>ses<br />
or mental models about an adversary’s potential behavior.<br />
7<br />
Sociological def<strong>in</strong>itions are more ak<strong>in</strong> to <strong>the</strong> def<strong>in</strong>itions proposed <strong>in</strong> this study. Failure can occur<br />
due to system complexity and miss<strong>in</strong>g data as well as through <strong>the</strong> accumulation of error. See<br />
Charles Perrow, Normal Accidents. Liv<strong>in</strong>g with High Risk Technologies. I’d like to thank Dr. Perrow<br />
for his assistance with this work.<br />
8
CHAPTER TWO<br />
F<strong>in</strong>d<strong>in</strong>gs<br />
Scientific knowledge, like language, is <strong>in</strong>tr<strong>in</strong>sically <strong>the</strong> common<br />
property of a group or else noth<strong>in</strong>g at all. To understand it we shall<br />
need to know <strong>the</strong> special characteristics of <strong>the</strong> groups that create<br />
and use it.<br />
Thomas Kuhn 1<br />
The more we learn about <strong>the</strong> world, and <strong>the</strong> deeper our learn<strong>in</strong>g, <strong>the</strong><br />
more conscious, specific, and articulate will be our knowledge of<br />
what we do not know.<br />
Karl Popper 2<br />
The purpose of this research was to identify and describe conditions and<br />
variables that negatively affect <strong>in</strong>telligence analysis, to develop relevant and<br />
testable <strong>the</strong>ory based on <strong>the</strong>se f<strong>in</strong>d<strong>in</strong>gs, and to identify areas <strong>in</strong> which strategies<br />
to improve performance may be effective. Although <strong>the</strong>re has recently<br />
been a great deal of concern that <strong>in</strong>telligence error and failure rates are <strong>in</strong>ord<strong>in</strong>ately<br />
high, <strong>in</strong> all likelihood, <strong>the</strong>se rates are similar to those of o<strong>the</strong>r complex<br />
socio-cognitive doma<strong>in</strong>s, such as analysis of f<strong>in</strong>ancial markets. The significant<br />
differences are that o<strong>the</strong>r complex doma<strong>in</strong>s employ systematic performance<br />
1<br />
Philosopher of science Thomas Kuhn described <strong>the</strong> now-common concept of paradigm shifts <strong>in</strong><br />
scientific revolutions. He posited that paradigm shifts are tied to cultural and social constructionist<br />
models, such as Vygotsky’s (See footnote 22 <strong>in</strong> Chapter Three). Thomas Kuhn, The Structure<br />
of Scientific Revolutions.<br />
2<br />
Karl Popper was one of <strong>the</strong> 20th century’s pre-em<strong>in</strong>ent philosophers of science. Karl Popper,<br />
Conjectures and Refutations: The Growth of Scientific Knowledge.<br />
9
CHAPTER TWO<br />
improvement <strong>in</strong>frastructures and that <strong>the</strong> consequences of <strong>in</strong>telligence error<br />
and failure are disproportionately high <strong>in</strong> comparison with o<strong>the</strong>r doma<strong>in</strong>s.<br />
It is evident from <strong>the</strong> literature that <strong>in</strong>telligence organizations recognize <strong>the</strong><br />
need to improve <strong>the</strong>ir performance and that it is possible to make <strong>the</strong> doma<strong>in</strong><br />
of <strong>in</strong>telligence analysis <strong>in</strong>to a coherent scientific discipl<strong>in</strong>e. The first step <strong>in</strong><br />
this transition is to identify and describe performance gaps. 3 Once gaps have<br />
been identified, it will be possible to <strong>in</strong>troduce performance improvement<br />
methods systematically and to measure <strong>the</strong> effectiveness of <strong>the</strong> results. This<br />
work is <strong>in</strong>tended to fur<strong>the</strong>r research toward creat<strong>in</strong>g <strong>in</strong>telligence organizations<br />
that are more effective.<br />
The Problem of Bias<br />
Although a researcher might pretend to be neutral and unbiased <strong>in</strong> present<strong>in</strong>g<br />
his f<strong>in</strong>d<strong>in</strong>gs and conclusions, personal biases can creep <strong>in</strong>to a f<strong>in</strong>ished<br />
product. The methods ethnographers employ to collect raw data and <strong>the</strong> use of<br />
<strong>in</strong>terpretational analysis to extract mean<strong>in</strong>g and generate <strong>the</strong>ory virtually guarantee<br />
it. In my view, one should be candid about this possibility. I noted <strong>in</strong><br />
Chapter One that ethnographers br<strong>in</strong>g a certa<strong>in</strong> amount of experiential baggage<br />
to <strong>the</strong>ir work, myself <strong>in</strong>cluded. At this po<strong>in</strong>t, before discuss<strong>in</strong>g analytical<br />
difficulties and problems I identified dur<strong>in</strong>g my research, I want to make <strong>the</strong><br />
readers aware of an additional personal bias that has developed from observ<strong>in</strong>g<br />
<strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong>.<br />
Dur<strong>in</strong>g my research, I developed a great deal of empathy for <strong>in</strong>dividual analysts<br />
and <strong>the</strong> problems <strong>the</strong>y face <strong>in</strong> try<strong>in</strong>g to perform <strong>the</strong>ir jobs. The reason for<br />
this is straightforward and someth<strong>in</strong>g every anthropologist recognizes. It is<br />
part of <strong>the</strong> process that anthropologists reach a po<strong>in</strong>t where <strong>the</strong>y can modify<br />
<strong>the</strong>ir own identity <strong>in</strong> order to ga<strong>in</strong> <strong>in</strong>sight <strong>in</strong>to a different culture. The risk is<br />
that empathy and identity modification will <strong>in</strong>duce <strong>the</strong> researcher to “go<br />
native” and produce bias <strong>in</strong> his f<strong>in</strong>d<strong>in</strong>gs.<br />
Although I may empathize with analysts personally, it is critical for <strong>the</strong>ory<br />
development to avoid parrot<strong>in</strong>g <strong>the</strong> views, kudos, or compla<strong>in</strong>ts of <strong>in</strong>dividual<br />
analysts, who may or may not be dissatisfied with <strong>the</strong>ir unique professional<br />
experience. In order to counteract <strong>the</strong> empathy bias, I employed multiple data<br />
collection techniques and <strong>the</strong>n used those data to refute or confirm each categorical<br />
f<strong>in</strong>d<strong>in</strong>g. Triangulation is not an <strong>in</strong>fallible system, however, and <strong>the</strong><br />
reader is advised to approach <strong>the</strong>se f<strong>in</strong>d<strong>in</strong>gs with both a critical eye and <strong>the</strong><br />
3<br />
Performance gaps are <strong>the</strong> difference or distance between ideal (perfect) organizational performance<br />
and actual organizational performance. In this case, ideal performance <strong>in</strong>cludes complete<br />
data sets, reportorial accuracy, and <strong>the</strong> ability to avoid strategic, operational, and tactical surprise.<br />
10
FINDINGS<br />
foreknowledge that this researcher has a number of personal and professional<br />
biases. 4<br />
F<strong>in</strong>d<strong>in</strong>g: Secrecy Versus Efficacy<br />
Secrecy and efficacy conflict. Secrecy <strong>in</strong>terferes with analytic effectiveness<br />
by limit<strong>in</strong>g access to <strong>in</strong>formation and sources that may be necessary for accurate<br />
or predictive analysis. In turn, openness <strong>in</strong>terferes with security by<br />
degrad<strong>in</strong>g <strong>the</strong> value of <strong>in</strong>formation resources and by reveal<strong>in</strong>g specific<br />
sources and methods.<br />
Perfect secrecy would ultimately be unproductive, because it would<br />
restrict <strong>in</strong>formation to one m<strong>in</strong>d or to a very small group of m<strong>in</strong>ds. Limit<strong>in</strong>g<br />
available resources <strong>in</strong> this way would produce organizational failure <strong>in</strong><br />
competition with resources available to a large and diverse group of adversaries.<br />
Perfect openness would also lead to organizational failure, because,<br />
with full access to all <strong>in</strong>formation, <strong>the</strong>re would never be an <strong>in</strong>stance of<br />
advantage for any one group over any o<strong>the</strong>r group. In addition, perfect<br />
openness would result <strong>in</strong> adversaries be<strong>in</strong>g aware <strong>the</strong>y are under observation<br />
and could lead <strong>the</strong>m to alter <strong>the</strong>ir behavior to deceive <strong>the</strong> observer if<br />
<strong>the</strong>y so desired. 5<br />
Between <strong>the</strong>se two extremes, <strong>the</strong>re is some notional po<strong>in</strong>t where secrecy<br />
and openness converge to create an optimal performance tradeoff. My perception<br />
is that, with<strong>in</strong> <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong>, more organizational emphasis<br />
is placed on secrecy than on effectiveness. It is important, <strong>in</strong> my view, that<br />
<strong>the</strong>re be a voice <strong>in</strong> favor of openness to counterbalance <strong>the</strong> many voices<br />
whose sole or primary responsibility is <strong>the</strong> advocacy and ma<strong>in</strong>tenance of<br />
4<br />
Throughout <strong>the</strong> project, my data collection method consisted of written field notes. Anthropologists<br />
traditionally <strong>in</strong>clude specific detail from participant <strong>in</strong>put or direct observation. Usually, this is<br />
<strong>in</strong> <strong>the</strong> form of precise descriptions of <strong>the</strong> actual behavior of participants and transcripts of <strong>the</strong>ir verbal<br />
<strong>in</strong>teractions. It is also standard practice <strong>in</strong> field work to capture <strong>the</strong>se data, and <strong>the</strong> data from <strong>the</strong><br />
<strong>in</strong>terviews and focus groups, on audio- or videotape. These practices were not followed <strong>in</strong> this particular<br />
case for two reasons: first, <strong>the</strong> nature of my work was not to document actual practices and<br />
procedures; ra<strong>the</strong>r, it was to derive categories of variables and <strong>in</strong>dividual variables <strong>in</strong> order to create<br />
a taxonomy, and to use <strong>the</strong> prototype taxonomy to structure <strong>the</strong> <strong>in</strong>teractions; second, <strong>the</strong> nature of<br />
<strong>in</strong>telligence work and <strong>the</strong> environment <strong>in</strong> which it occurs, as well as its professional practitioners,<br />
require that certa<strong>in</strong> data be restricted.<br />
5<br />
This has been demonstrated <strong>in</strong> <strong>the</strong> psychological literature and is referred to as <strong>the</strong> Hawthorne<br />
Effect. Derived from research that began with an experimental program at Western Electric’s<br />
Hawthorne Works conducted between 1927 and 1930, <strong>the</strong> Hawthorne Theory, broadly <strong>in</strong>terpreted,<br />
states that <strong>the</strong> behavior of subjects changes when <strong>the</strong>y are aware of be<strong>in</strong>g observed. See<br />
Fritz J. Roethlisberger and William J. Dickson, Management and <strong>the</strong> worker; Elton Mayo, The<br />
Human Problems of an Industrial Civilization.<br />
11
CHAPTER TWO<br />
Secrecy vs. Efficacy<br />
secrecy. I believe this secrecy-efficacy conflict can be stated as a <strong>the</strong>ory, along<br />
<strong>the</strong> follow<strong>in</strong>g l<strong>in</strong>es. 6<br />
The more open <strong>the</strong> system (where zero is perfect <strong>in</strong>formation access and<br />
shar<strong>in</strong>g on <strong>the</strong> X axis secrecy scale [as shown on <strong>the</strong> above graph]), <strong>the</strong> more<br />
access an analyst has to all sources of <strong>in</strong>formation with<strong>in</strong> <strong>the</strong> <strong>Intelligence</strong><br />
<strong>Community</strong> regard<strong>in</strong>g an adversary. In addition, this openness encourages<br />
<strong>in</strong>terorganizational communication, <strong>in</strong>teraction, and shar<strong>in</strong>g of <strong>in</strong>formation<br />
among analysts and <strong>in</strong>creases <strong>the</strong> likelihood that an analyst will be more efficient<br />
(<strong>in</strong> this case <strong>the</strong> Y axis efficiency scale) and <strong>the</strong>refore effective or accurate<br />
<strong>in</strong> his or her assessment of a situation.<br />
Conversely, counter-<strong>in</strong>telligence is negatively affected by zero-level<br />
secrecy and perfect openness. The less open or more compartmentalized <strong>the</strong><br />
system, <strong>the</strong> more efficient and effective are counter<strong>in</strong>telligence activities.<br />
Notionally, <strong>the</strong> two curves would meet somewhere <strong>in</strong> <strong>the</strong> tradeoff between<br />
efficiency and secrecy. Where <strong>the</strong>y meet would depend on program goals and<br />
a clear def<strong>in</strong>ition of start<strong>in</strong>g po<strong>in</strong>ts and end-states.<br />
The notional set of curves above illustrates <strong>the</strong> tradeoff between system<br />
efficiency and system secrecy and <strong>the</strong> effect that <strong>the</strong> tradeoff has on performance<br />
effectiveness, both positive and negative. In this case, <strong>the</strong> start<strong>in</strong>g and<br />
end<strong>in</strong>g po<strong>in</strong>ts of effectiveness for analysis and for counter<strong>in</strong>telligence are<br />
arbitrary and could be positioned anywhere along a cont<strong>in</strong>uum between zero<br />
6<br />
I would like to credit and thank Mat<strong>the</strong>w Johnson at <strong>the</strong> Institute for Defense Analyses for his<br />
help <strong>in</strong> formulat<strong>in</strong>g this <strong>the</strong>ory.<br />
12
FINDINGS<br />
and ten. In this <strong>the</strong>ory, analytic efficiency and effectiveness are purely functions<br />
of system openness and do not take <strong>in</strong>to account analytic methods or personnel.<br />
This <strong>the</strong>ory will require additional ref<strong>in</strong>ement, and it may or may not be<br />
represented by a tradeoff curve like <strong>the</strong> one proposed here. The <strong>the</strong>ory will<br />
also require numerous controlled quantitative experiments to test its explanatory<br />
power.<br />
F<strong>in</strong>d<strong>in</strong>g: Time Constra<strong>in</strong>ts<br />
The work itself is a 24-hour-a-day job, but it never seems like I have<br />
any time to actually analyze anyth<strong>in</strong>g when I’m at my desk. I spend<br />
most of my time read<strong>in</strong>g daily traffic, answer<strong>in</strong>g e-mail, coord<strong>in</strong>at<strong>in</strong>g<br />
papers with everybody, and writ<strong>in</strong>g. Mostly I read and write, but<br />
when <strong>the</strong> workday is over, I go home and th<strong>in</strong>k. It isn’t like I can<br />
turn off my bra<strong>in</strong>. So, I guess I do most of my real analysis on my<br />
own time.<br />
The majority of <strong>the</strong> analysts <strong>in</strong>terviewed <strong>in</strong>dicated that time was one of<br />
<strong>the</strong>ir greatest constra<strong>in</strong>ts at work. This comment triangulated with <strong>the</strong> f<strong>in</strong>d<strong>in</strong>gs<br />
from direct and participant observation. In addition, analysts <strong>in</strong>dicated that<br />
<strong>the</strong>re has been a communitywide shift toward focus<strong>in</strong>g on short-term issues or<br />
problem solv<strong>in</strong>g, <strong>the</strong>reby address<strong>in</strong>g <strong>the</strong> immediate needs of <strong>in</strong>telligence consumers.<br />
This shift <strong>in</strong> product focus, coupled with a growth <strong>in</strong> available allsource<br />
raw <strong>in</strong>telligence, has resulted <strong>in</strong> a change <strong>in</strong> <strong>the</strong> pace of analytic production.<br />
In order to generate <strong>the</strong> daily products, analysts have had to change<br />
<strong>the</strong> way <strong>the</strong>y go about do<strong>in</strong>g <strong>the</strong>ir work.<br />
I haven’t been do<strong>in</strong>g this very long, but I wish I had been a journalism<br />
major <strong>in</strong>stead of poli-sci. The pace is excruciat<strong>in</strong>g.<br />
I don’t get much sleep. It’s like cramm<strong>in</strong>g for f<strong>in</strong>als, except we do it<br />
every day.<br />
Everyth<strong>in</strong>g I do is reactive. I don’t have time to work my subject.<br />
We’re not pro-active here.<br />
I’m so busy putt<strong>in</strong>g out today’s fires, I don’t have any time to th<strong>in</strong>k<br />
about what k<strong>in</strong>d of catastrophe is <strong>in</strong> store for me a month from now.<br />
About 15 years ago, I used to have 60 percent of my time available<br />
for long-term products. Now, it’s between 20 and 25 percent.<br />
I probably have about 30 percent of my time for self-<strong>in</strong>itiated products.<br />
13
CHAPTER TWO<br />
You know, someday somebody is bound to notice that velocity isn’t<br />
a substitute for quality. We’ve gotten rid of <strong>the</strong> real analytic products<br />
that we use to make, and now we just report on current events.<br />
Not all analysts <strong>in</strong>dicated that time constra<strong>in</strong>ts and <strong>in</strong>formation load had a<br />
negative effect on <strong>the</strong>ir performance. A m<strong>in</strong>ority <strong>in</strong>dicated that <strong>the</strong>re was sufficient<br />
time to perform analytic duties and prepare analytic products.<br />
This is a tactical shop. It’s all we do. Current report<strong>in</strong>g is our job.<br />
I work a slow desk. I have plenty of time for self-<strong>in</strong>itiated products<br />
—maybe 60 percent or more.<br />
I multitask pretty well. I don’t really experience a time-crunch.<br />
Maybe I just process better than o<strong>the</strong>r people, but I don’t really feel<br />
pressed for time. Besides, I’d ra<strong>the</strong>r be at a hot desk than at a cold<br />
desk.<br />
<strong>Analytic</strong> supervisors were more evenly mixed <strong>in</strong> <strong>the</strong>ir op<strong>in</strong>ions about time<br />
constra<strong>in</strong>ts. A slight majority of <strong>the</strong> managers <strong>in</strong>terviewed said time constra<strong>in</strong>ts<br />
had negative effects on <strong>the</strong> work environment, work processes, and <strong>the</strong><br />
morale of <strong>the</strong>ir staff. A majority of <strong>the</strong>m also put analytic time constra<strong>in</strong>ts <strong>in</strong> a<br />
larger context of policy mak<strong>in</strong>g. They <strong>in</strong>dicated that <strong>the</strong> decision-cycle of policymakers<br />
was 24 hours a day and that <strong>the</strong>ir responsibility was to support that<br />
decision cycle with current <strong>in</strong>telligence.<br />
In discuss<strong>in</strong>g <strong>the</strong>ir perceptions of consumer demand, <strong>the</strong> managers’ views<br />
of <strong>the</strong> nature of those demands were mixed.<br />
I want my analysts to produce long-term products. I want <strong>the</strong>m<br />
th<strong>in</strong>k<strong>in</strong>g through <strong>the</strong>ir subjects. The decision makers want wellthought-out<br />
products, not just daily briefs.<br />
Our customers want current production. They never compla<strong>in</strong> about<br />
<strong>the</strong> daily products and, frankly, I doubt <strong>the</strong>y have time to read <strong>the</strong><br />
longer stuff.<br />
My consumers like <strong>the</strong> bigger pieces. They like hav<strong>in</strong>g <strong>the</strong> context<br />
and broader picture. They don’t want to be spoon fed.<br />
I’ve never had a customer tell me <strong>the</strong>y want more to read.<br />
Our customers want to avoid surprise. As long as we keep <strong>the</strong>m<br />
from be<strong>in</strong>g surprised, I don’t care if we do daily or long-term production.<br />
I don’t th<strong>in</strong>k <strong>the</strong>y care ei<strong>the</strong>r.<br />
14
FINDINGS<br />
F<strong>in</strong>d<strong>in</strong>g: Focus on Current Production<br />
The present daily production cycle and <strong>the</strong> focus on current <strong>in</strong>telligence<br />
also affect group <strong>in</strong>teractions and <strong>the</strong> analytic process.<br />
Group Interactions:<br />
It doesn’t matter if I’m writ<strong>in</strong>g a piece myself or if I’m coord<strong>in</strong>at<strong>in</strong>g<br />
a piece with some group. We don’t sit around and test hypo<strong>the</strong>ses,<br />
because we’re too busy writ<strong>in</strong>g. We’ve got serious deadl<strong>in</strong>es here.<br />
If, by group analysis, you mean <strong>the</strong> senior expert <strong>in</strong> <strong>the</strong> room tells<br />
everybody what he th<strong>in</strong>ks, and <strong>the</strong>n we generally agree so that we<br />
can get back to our own deadl<strong>in</strong>es, <strong>the</strong>n, sure, <strong>the</strong>re’s a group process.<br />
We used to have groups that did current report<strong>in</strong>g and different<br />
groups that did longer term products. We still have some of that, but<br />
it is very limited. I couldn’t say what happened exactly, but we’re all<br />
do<strong>in</strong>g current production now.<br />
The <strong>Analytic</strong> Process:<br />
People seem to have confused writ<strong>in</strong>g with analyz<strong>in</strong>g. They figure<br />
that if you just go through <strong>the</strong> mechanics of writ<strong>in</strong>g someth<strong>in</strong>g, <strong>the</strong>n<br />
you must have analyzed it. I don’t know about everybody else, but it<br />
doesn’t work that way for me. I need time to th<strong>in</strong>k through <strong>the</strong> problem.<br />
Our products have become so specific, so tactical even, that our<br />
th<strong>in</strong>k<strong>in</strong>g has become tactical. We’re los<strong>in</strong>g our strategic edge,<br />
because we’re so focused on today’s issues.<br />
Alternative analysis is a nice concept, but I don’t have <strong>the</strong> time to do<br />
it. I’ve got to keep up with <strong>the</strong> daily traffic.<br />
I use several analytic techniques that are relatively fast. Scenario<br />
development, red teams, compet<strong>in</strong>g hypo<strong>the</strong>ses, <strong>the</strong>y’re all too time<br />
consum<strong>in</strong>g.<br />
We’ve got Bayesian tools, simulations, all k<strong>in</strong>ds of advanced methods,<br />
but when am I supposed to do any of that? It takes all my time<br />
to keep up with <strong>the</strong> daily report<strong>in</strong>g as it is.<br />
I don’t have time to worry about formal analytic methods. I’ve got<br />
my own system. It’s more <strong>in</strong>tuitive and a lot faster.<br />
15
CHAPTER TWO<br />
F<strong>in</strong>d<strong>in</strong>g: Rewards and Incentives<br />
The shift <strong>in</strong> <strong>the</strong> analytic production cycle is not only reflected <strong>in</strong> <strong>the</strong> products<br />
and processes but also <strong>in</strong> <strong>the</strong> way analysts perceive <strong>the</strong> system by which<br />
<strong>in</strong>telligence organizations reward and promote employees. Employees see<br />
<strong>the</strong>ir opportunities for promotion as be<strong>in</strong>g tied directly to <strong>the</strong> number of daily<br />
products <strong>the</strong>y generate and <strong>the</strong> amount of social capital or direct consumer<br />
<strong>in</strong>fluence <strong>the</strong>y amass, most often when <strong>the</strong>ir work is recognized by senior policymakers.<br />
7<br />
In any given week, I could devote about 20 percent of my time to<br />
longer th<strong>in</strong>k pieces, but why should I? You can write all <strong>the</strong> th<strong>in</strong>k<br />
pieces you want, but, if you don’t write for <strong>the</strong> daily briefs, you<br />
aren’t go<strong>in</strong>g to move <strong>in</strong>to management. These days <strong>the</strong> only th<strong>in</strong>g<br />
that matters is gett<strong>in</strong>g to <strong>the</strong> customers.<br />
If I write a 12-page self-directed piece that goes out as a community<br />
product, and somebody else writes one paragraph with two bullet<br />
po<strong>in</strong>ts that goes <strong>in</strong>to a daily brief, <strong>the</strong> guy who got <strong>in</strong> <strong>the</strong> daily brief<br />
is go<strong>in</strong>g to get <strong>the</strong> recognition. Why waste my time with <strong>the</strong> big<br />
products?<br />
It isn’t really official policy, but <strong>the</strong> reality is that sheer production<br />
equals promotion. People talk about quality, but, <strong>in</strong> <strong>the</strong> end, <strong>the</strong><br />
only measurable th<strong>in</strong>g is quantity.<br />
Our group has a “team award” of 5,000 bucks. Last year, <strong>the</strong>y gave<br />
it to <strong>the</strong> one guy who published <strong>the</strong> most. I’m not sure how that one<br />
guy won a “team award,” but <strong>the</strong>re you go.<br />
Technically, I have four bosses. The only th<strong>in</strong>g that seems to keep<br />
<strong>the</strong>m all happy is volume. It’s like piece work.<br />
Quality? How do you measure quality? Quantity—now that’s someth<strong>in</strong>g<br />
you can count.<br />
Promotion is based on production—pure and simple.<br />
In sum, aside from specific tactical groups, staff positions that generate limited<br />
social capital, and <strong>in</strong>dividual cognitive differences, <strong>the</strong>re is a majority<br />
sentiment among <strong>the</strong> analysts <strong>in</strong>terviewed that <strong>the</strong> comb<strong>in</strong>ation of a shorter<br />
7<br />
Social capital refers to <strong>the</strong> set of norms, social networks, and organizations through which people<br />
ga<strong>in</strong> access to power, resources, and reciprocity and through which decisionmak<strong>in</strong>g and policy<br />
creation occur. In o<strong>the</strong>r words, whom you know is just as important as what you know. Pierre<br />
Bourdieu, “The Forms of Capital”; Robert Putnam, “The Prosperous <strong>Community</strong>” and Bowl<strong>in</strong>g<br />
Alone. See also <strong>the</strong> empirical work on social capital summarized <strong>in</strong> T<strong>in</strong>e Feldman and Susan<br />
Assaf, Social Capital: Conceptual Frameworks and Empirical Evidence.<br />
16
FINDINGS<br />
production cycle, <strong>in</strong>formation load, a shift <strong>in</strong> product focus, and organizational<br />
norms regard<strong>in</strong>g promotion have had an impact on analytic work and<br />
<strong>in</strong>telligence analysis itself.<br />
F<strong>in</strong>d<strong>in</strong>g: “Tradecraft” Versus Scientific Methodology<br />
Human be<strong>in</strong>gs do not live <strong>in</strong> <strong>the</strong> objective world alone, nor alone <strong>in</strong><br />
<strong>the</strong> world of social activity as ord<strong>in</strong>arily understood, but are very<br />
much at <strong>the</strong> mercy of <strong>the</strong> particular language which has become <strong>the</strong><br />
medium of expression for <strong>the</strong>ir society…The fact of <strong>the</strong> matter is that<br />
<strong>the</strong> “real world” is to a large extent unconsciously built upon <strong>the</strong><br />
language habits of <strong>the</strong> group…We see and hear and o<strong>the</strong>rwise experience<br />
very largely as we do because <strong>the</strong> language habits of our<br />
community predispose certa<strong>in</strong> choices of <strong>in</strong>terpretation.<br />
Edward Sapir 8<br />
The <strong>Intelligence</strong> <strong>Community</strong>, <strong>in</strong> its culture and mythos and <strong>in</strong> its literature,<br />
tends to focus on <strong>in</strong>telligence operations ra<strong>the</strong>r than on <strong>in</strong>telligence analysis.<br />
Open literature about <strong>the</strong> community certa<strong>in</strong>ly does so. Along with time constra<strong>in</strong>ts<br />
and <strong>the</strong> analytic production cycle, <strong>the</strong> private and public focus on<br />
operations has had an effect on <strong>in</strong>telligence analysts and analytic methodology.<br />
The pr<strong>in</strong>cipal effect is <strong>the</strong> spread of <strong>the</strong> concept of “tradecraft” with<strong>in</strong> <strong>the</strong><br />
analytic community.<br />
<strong>Community</strong> members quite often used <strong>the</strong> word “tradecraft” to describe<br />
<strong>in</strong>telligence analysis dur<strong>in</strong>g <strong>the</strong> <strong>in</strong>terviews, observations, tra<strong>in</strong><strong>in</strong>g programs,<br />
workshops, and actual analytic tasks that I performed for this study. Analysts,<br />
managers, <strong>in</strong>structors, and academic researchers employed <strong>the</strong> word “tradecraft”<br />
as a catchall for <strong>the</strong> often-idiosyncratic methods and techniques<br />
required to perform analysis. Although <strong>the</strong> <strong>in</strong>telligence literature often refers<br />
to tradecraft, <strong>the</strong> works tend to be a collection of suggestions and tips for writ<strong>in</strong>g<br />
and communicat<strong>in</strong>g with co-workers, supervisors, and consumers <strong>in</strong>stead<br />
of focus<strong>in</strong>g on a thorough exam<strong>in</strong>ation of <strong>the</strong> analytic process and techniques.<br />
The notion that <strong>in</strong>telligence operations <strong>in</strong>volve tradecraft, which I def<strong>in</strong>e as<br />
practiced skill <strong>in</strong> a trade or art, may be appropriate, but <strong>the</strong> analytic community’s<br />
adoption of <strong>the</strong> concept to describe analysis and analytic methods is not.<br />
The obvious logical flaw with adopt<strong>in</strong>g <strong>the</strong> idea of tradecraft as a standard of<br />
practice for analytic methodology is that, ultimately, analysis is nei<strong>the</strong>r craft<br />
nor art. Analysis, I contend, is part of a scientific process. This is an important<br />
8<br />
Edward Sapir is best known for <strong>the</strong> Sapir-Whorf hypo<strong>the</strong>sis, which asserts l<strong>in</strong>guistic/cognitive<br />
relativity (language and thought are <strong>in</strong>separable; <strong>the</strong>refore, different languages mean different<br />
ways of th<strong>in</strong>k<strong>in</strong>g). Edward Sapir, Language.<br />
17
CHAPTER TWO<br />
dist<strong>in</strong>ction, for language is a key variable <strong>in</strong> anthropology and often reveals a<br />
great deal about <strong>the</strong> cognition and culture of a community of <strong>in</strong>terest. 9<br />
The adoption by members of <strong>the</strong> analytic community of an <strong>in</strong>appropriate<br />
term for <strong>the</strong> processes and methods employed <strong>in</strong> <strong>the</strong>ir professional lives<br />
obfuscates and complicates <strong>the</strong> reality of <strong>the</strong>ir work. The adoption of <strong>the</strong><br />
word “tradecraft” demonstrates <strong>the</strong> analytic community’s need to create a professional<br />
identity separate and unique from o<strong>the</strong>r discipl<strong>in</strong>es but tied directly<br />
to <strong>the</strong> perceived prestige and cachet of <strong>in</strong>telligence operations. Adopt<strong>in</strong>g<br />
“tradecraft” as a term of reference for expla<strong>in</strong><strong>in</strong>g work practices and as a professional<br />
identity marker may seem trivial. Yet <strong>the</strong> term, and its effect on <strong>the</strong><br />
community, has unanticipated consequences.<br />
Tradecraft purposefully implies a mysterious process learned only by <strong>the</strong><br />
<strong>in</strong>itiated and acquired only through <strong>the</strong> elaborate rituals of professional <strong>in</strong>doctr<strong>in</strong>ation.<br />
It also implies that <strong>the</strong> methods and techniques of analysis are <strong>in</strong>formal,<br />
idiosyncratic, unverifiable, and perhaps even unexpla<strong>in</strong>able. “Good”<br />
methods are simply those that survive, and <strong>the</strong>n are passed on by “good” analysts<br />
to novice analysts. Unfortunately, “good” <strong>in</strong> both <strong>in</strong>stances is not an<br />
objective measure. That is, <strong>the</strong>re is no formal system for measur<strong>in</strong>g and track<strong>in</strong>g<br />
<strong>the</strong> validity or reliability of analytic methods, because <strong>the</strong>y are both perceived<br />
and employed with<strong>in</strong> <strong>the</strong> context of idiosyncratic tradecraft. When<br />
asked to describe <strong>the</strong> analytic process, analysts responded <strong>in</strong> a variety of ways.<br />
First, I figure out what I know and what I don’t know about some<br />
situation. Then, I look for <strong>in</strong>formation to fill <strong>the</strong> gap.<br />
I have a model of <strong>the</strong> situation <strong>in</strong> my head. Whenever someth<strong>in</strong>g new<br />
comes <strong>in</strong>, I see if it fits with <strong>the</strong> model. If it does, I add it to <strong>the</strong><br />
model; if it doesn’t, I try to figure out why.<br />
I’ve found a system that lets me keep up. I just look for anomalies.<br />
When I see any novel data, <strong>the</strong>n I worry.<br />
I’m always look<strong>in</strong>g for anyth<strong>in</strong>g strange or out of place. Then, I<br />
source it to see if it is mean<strong>in</strong>gful.<br />
The current data ought to fit a certa<strong>in</strong> pattern. If it doesn’t, I know<br />
someth<strong>in</strong>g is wrong.<br />
First, I pr<strong>in</strong>t <strong>the</strong> daily traffic I’m concerned with; <strong>the</strong>n I lay out all<br />
of <strong>the</strong> relevant stuff <strong>in</strong> front of me on my desk or <strong>the</strong> floor; <strong>the</strong>n I<br />
start look<strong>in</strong>g for threads.<br />
9<br />
The literature on this subject is extensive. For a representative list, see <strong>the</strong> appendix.<br />
18
FINDINGS<br />
I’m look<strong>in</strong>g for l<strong>in</strong>ks and patterns. Once I figure out <strong>the</strong> pattern, I<br />
can figure out where to look next.<br />
I use patterns. If th<strong>in</strong>gs start happen<strong>in</strong>g, out of <strong>the</strong> ord<strong>in</strong>ary th<strong>in</strong>gs, I<br />
pay attention to <strong>the</strong>m.<br />
I try to build patterns out of <strong>the</strong> data. It helps me predict what will<br />
happen next.<br />
I come up with a few scenarios and see what <strong>the</strong> evidence supports.<br />
I look for data that are diagnostic: some piece of evidence that rules<br />
out certa<strong>in</strong> possibilities.<br />
I try to weigh <strong>the</strong> evidence to see which scenario it supports.<br />
Although anomaly-detection, pattern-recognition, and weigh<strong>in</strong>g data may<br />
appear to be idiosyncratic tradecraft based on <strong>in</strong>dividual expertise and cognitive<br />
skills, <strong>the</strong>se methods can be formalized and replicated if <strong>the</strong> operat<strong>in</strong>g<br />
parameters, variables, and rules of evidence are made explicit. 10 This is to say<br />
that <strong>in</strong>telligence analysis can be reconstructed <strong>in</strong> <strong>the</strong> context of a scientific<br />
method, which is merely an articulated, formal process by which scientists,<br />
collectively and over time, endeavor to put toge<strong>the</strong>r a reliable, consistent, and<br />
nonarbitrary representation of some phenomena. Broadly, <strong>the</strong> steps <strong>in</strong>clude:<br />
• observation and description of phenomena;<br />
• formulation of hypo<strong>the</strong>ses to expla<strong>in</strong> phenomena;<br />
• test<strong>in</strong>g of hypo<strong>the</strong>ses by <strong>in</strong>dependent experts;<br />
• refutation or confirmation of hypo<strong>the</strong>ses.<br />
These steps do not suggest that any specific scientific methodology results<br />
<strong>in</strong> what is ultimately <strong>the</strong> truth, ra<strong>the</strong>r that scientific methods are merely formal<br />
processes used to describe phenomena, make predictions, and determ<strong>in</strong>e<br />
which hypo<strong>the</strong>sis best expla<strong>in</strong>s those phenomena. The pr<strong>in</strong>cipal value of any<br />
type of methodological formalism is that it allows o<strong>the</strong>r researchers to test <strong>the</strong><br />
validity and reliability of <strong>the</strong> f<strong>in</strong>d<strong>in</strong>gs of any o<strong>the</strong>r researcher by mak<strong>in</strong>g<br />
explicit, and <strong>the</strong>refore replicable, <strong>the</strong> means by which anyone reaches a specific<br />
conclusion. 11<br />
The idea that <strong>in</strong>telligence analysis is a collection of scientific methods<br />
encounters some resistance <strong>in</strong> <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong>. The <strong>in</strong>terview data<br />
analyzed <strong>in</strong> this study highlight many subtle—and not so subtle—prejudices<br />
10<br />
A corollary to <strong>the</strong>se methods can be found <strong>in</strong> <strong>the</strong> practice of radiologists. See Chapter Five for<br />
more on expertise.<br />
19
CHAPTER TWO<br />
that analysis is not a science. That is, it is an art or craft <strong>in</strong> which one can<br />
atta<strong>in</strong> skill but not a formal discipl<strong>in</strong>e with tested and validated methodology.<br />
What we do is more art and experience than anyth<strong>in</strong>g else.<br />
Science is too formal. We can’t actually run experiments here.<br />
How would you actually test a hypo<strong>the</strong>sis <strong>in</strong> <strong>in</strong>telligence?<br />
Science is what you do <strong>in</strong> a lab.<br />
We’re not scientists; we’re analysts. We don’t generate <strong>the</strong> data.<br />
We don’t worry too much about <strong>the</strong>ory; we worry about <strong>the</strong> facts.<br />
In my discipl<strong>in</strong>e, I might be a scientist, but, <strong>in</strong> <strong>in</strong>telligence, I am a<br />
practitioner.<br />
I use science for my area, but I don’t th<strong>in</strong>k <strong>in</strong>telligence analysis is<br />
science.<br />
As long as <strong>in</strong>telligence analysis cont<strong>in</strong>ues to be tradecraft, it will rema<strong>in</strong> a<br />
mystery. The quality of any tradecraft depends on <strong>the</strong> <strong>in</strong>nate cognitive capabilities<br />
of <strong>the</strong> <strong>in</strong>dividual and <strong>the</strong> good fortune one has <strong>in</strong> f<strong>in</strong>d<strong>in</strong>g a mentor who<br />
has discovered, through many years of trial and error, unique methods that<br />
seem to be effective. This process of trial and error is, <strong>in</strong> general, similar to<br />
any scientific process, except that <strong>the</strong> lessons learned <strong>in</strong> tradecraft, unlike<br />
those of o<strong>the</strong>r discipl<strong>in</strong>es, often occur without be<strong>in</strong>g captured, tested, or validated.<br />
In an oral tradition, <strong>in</strong>dividual tradecraft methods are passed on by means<br />
of apprenticeship. The consequence for any culture tied to an oral tradition is<br />
<strong>the</strong> loss of important knowledge that occurs with <strong>the</strong> loss of practitioners. In<br />
organizations, <strong>the</strong> retirement of experts and <strong>in</strong>novators leads to <strong>the</strong> loss of that<br />
expertise and <strong>in</strong>novation, unless <strong>the</strong>re is some formal written and educational<br />
system to keep that knowledge alive. 12<br />
The data collected through both <strong>in</strong>terviews and observation <strong>in</strong>dicated that<br />
<strong>the</strong>re were, <strong>in</strong> fact, general methods that could be formalized and that this process<br />
would <strong>the</strong>n lead to <strong>the</strong> development of <strong>in</strong>telligence analysis as a scientific<br />
discipl<strong>in</strong>e. The pr<strong>in</strong>cipal difficulty lies not <strong>in</strong> develop<strong>in</strong>g <strong>the</strong> methods <strong>the</strong>m-<br />
11<br />
Ra<strong>the</strong>r than engage <strong>in</strong> <strong>the</strong> longstand<strong>in</strong>g and ongo<strong>in</strong>g debate <strong>in</strong> <strong>the</strong> academic community about<br />
what is and what is not science or a scientific method, suffice it to say that any scientific method<br />
needs to be explicit, replicable, and refutable. The literature surround<strong>in</strong>g this debate is volum<strong>in</strong>ous.<br />
The philosophy of science, logic, language, and epistemology has taken this debate <strong>in</strong> a<br />
number of directions. There is, however, a general <strong>the</strong>me that replication is a key <strong>in</strong>gredient to<br />
any scientific method.<br />
12<br />
See section on Endangered Languages <strong>in</strong> Barbara Grimes, ed., Ethnologue. 14th ed.<br />
20
FINDINGS<br />
selves, but <strong>in</strong> articulat<strong>in</strong>g those methods for <strong>the</strong> purpose of test<strong>in</strong>g and validat<strong>in</strong>g<br />
<strong>the</strong>m and <strong>the</strong>n test<strong>in</strong>g <strong>the</strong>ir effectiveness throughout <strong>the</strong> community. In <strong>the</strong><br />
long view, develop<strong>in</strong>g <strong>the</strong> science of <strong>in</strong>telligence analysis is easy; what is difficult<br />
is chang<strong>in</strong>g <strong>the</strong> perception of <strong>the</strong> analytic practitioners and managers<br />
and, <strong>in</strong> turn, modify<strong>in</strong>g <strong>the</strong> culture of tradecraft.<br />
F<strong>in</strong>d<strong>in</strong>g: Confirmation Bias, Norms, and Taboos<br />
Organization is key, because it sets up relationships among people<br />
through allocation and control of resources and rewards. It draws<br />
on tactical power to monopolize or parcel out liens and claims, to<br />
channel action <strong>in</strong>to certa<strong>in</strong> pathways while <strong>in</strong>terdict<strong>in</strong>g <strong>the</strong> flow of<br />
action <strong>in</strong>to o<strong>the</strong>rs. Some th<strong>in</strong>gs become possible and likely; o<strong>the</strong>rs<br />
are rendered unlikely.<br />
Eric Wolf 13<br />
Time constra<strong>in</strong>ts affect both <strong>the</strong> general analytic production cycle and analytic<br />
methodology by contribut<strong>in</strong>g to and exacerbat<strong>in</strong>g cognitive biases.<br />
Although <strong>the</strong>re are any number of cognitive biases to which <strong>the</strong> human m<strong>in</strong>d<br />
is susceptible, one <strong>in</strong> particular became evident dur<strong>in</strong>g <strong>the</strong> triangulation phase<br />
and <strong>in</strong>terpretive analysis of <strong>the</strong> <strong>in</strong>terview and observation data of this study.<br />
The cognitive bias identified most often was confirmation bias, which is <strong>the</strong><br />
tendency of <strong>in</strong>dividuals to select evidence that supports ra<strong>the</strong>r than refutes a<br />
given hypo<strong>the</strong>sis. 14<br />
Although <strong>the</strong> psychological mechanism by which confirmation bias occurs<br />
is <strong>in</strong> debate, confirmatory behavior is a consistent f<strong>in</strong>d<strong>in</strong>g throughout <strong>the</strong><br />
experimental psychology and cognitive science literature. Ra<strong>the</strong>r than focus<br />
on <strong>the</strong> mechanism and nomenclature, <strong>the</strong> term “confirmation bias” is used <strong>in</strong><br />
this work as a description of confirmatory behavior. This behavior was<br />
13<br />
Eric Wolf was an anthropologist who focused on power, social structures, and <strong>the</strong> third world.<br />
His work on power and <strong>the</strong> lives of peasants is considered a modern anthropological classic. Eric<br />
Wolf, Pathways of Power.<br />
14<br />
There is a fair amount of disagreement <strong>in</strong> <strong>the</strong> psychological literature regard<strong>in</strong>g <strong>the</strong> mechanism<br />
by which an <strong>in</strong>dividual displays confirmatory behavior. Some researchers attribute it to motivational<br />
factors, for example, a desire to ma<strong>in</strong>ta<strong>in</strong> respect with<strong>in</strong> a group. O<strong>the</strong>r researchers attribute<br />
it to selectivity factors, an unconscious cognitive selection of data that confirms <strong>the</strong> current status<br />
quo. Some researchers attribute it to social factors, a subspecies of groupth<strong>in</strong>k (see Irv<strong>in</strong>g Janis,<br />
Groupth<strong>in</strong>k). Still o<strong>the</strong>rs ascribe it to a misapplication of heuristics, whereby an <strong>in</strong>dividual learns<br />
a set of rules that solves one problem and <strong>the</strong>n beg<strong>in</strong>s us<strong>in</strong>g that same set of rules to try to solve<br />
o<strong>the</strong>r types of problems. Although <strong>the</strong> literature is extensive, Karl Popper’s The Logic of Scientific<br />
Discovery provides a foundation for understand<strong>in</strong>g <strong>the</strong> issue. Jonathan Evans’ Bias <strong>in</strong> Human<br />
Reason<strong>in</strong>g: Causes and Consequences is still a useful and concise summary of <strong>the</strong> research<br />
related to confirmation bias.<br />
21
CHAPTER TWO<br />
described by participants dur<strong>in</strong>g <strong>the</strong> <strong>in</strong>terviews and observed dur<strong>in</strong>g direct and<br />
participant observations throughout <strong>the</strong> fieldwork.<br />
Analysts were asked to describe <strong>the</strong> work processes <strong>the</strong>y employed to<br />
answer questions, solve problems, describe and expla<strong>in</strong> phenomena, make<br />
forecasts, and develop <strong>in</strong>telligence products. The process <strong>the</strong>y described<br />
began with an exam<strong>in</strong>ation of previous analytic products developed by <strong>the</strong>ir<br />
organization <strong>in</strong> order to establish a basel<strong>in</strong>e from which <strong>the</strong>y could build <strong>the</strong>ir<br />
own analysis.<br />
When a request comes <strong>in</strong> from a consumer to answer some question,<br />
<strong>the</strong> first th<strong>in</strong>g I do is to read up on <strong>the</strong> analytic l<strong>in</strong>e.<br />
The first th<strong>in</strong>g I do is check <strong>the</strong> pervious publications, and <strong>the</strong>n I<br />
sort through <strong>the</strong> current traffic.<br />
I’ve looked at our previous products, and I’ve got a good idea of <strong>the</strong><br />
pattern; so, when I sort through <strong>the</strong> traffic, I know what I’m try<strong>in</strong>g<br />
to f<strong>in</strong>d.<br />
I try to keep up with all <strong>the</strong> products that come out of our area, so I<br />
know where to start my piece.<br />
A literature search is often <strong>the</strong> first step <strong>in</strong> any research endeavor. The utility<br />
of this practice is not merely to def<strong>in</strong>e and understand <strong>the</strong> current state of<br />
research <strong>in</strong> <strong>the</strong> field but also to determ<strong>in</strong>e major controversies and divergences<br />
of op<strong>in</strong>ion. Try<strong>in</strong>g to discern controversies and divergence <strong>in</strong> <strong>in</strong>telligence<br />
products is often difficult, because some of <strong>the</strong>m—national <strong>in</strong>telligence estimates<br />
(NIE), <strong>in</strong> particular—are specifically designed to produce a corporate<br />
consensus for an audience of high-level policymakers.<br />
These products can and do <strong>in</strong>clude divergent op<strong>in</strong>ions, <strong>in</strong> <strong>the</strong> form of footnotes,<br />
but <strong>the</strong>se tend to <strong>in</strong>dicate <strong>in</strong>ter-, ra<strong>the</strong>r than <strong>in</strong>tra-, organizational differences.<br />
Dissent<strong>in</strong>g footnotes are products of <strong>the</strong> coord<strong>in</strong>ation process, <strong>the</strong> result<br />
of an <strong>in</strong>ability on <strong>the</strong> part of one or several community organizations to conv<strong>in</strong>ce<br />
<strong>the</strong> o<strong>the</strong>rs of a particular po<strong>in</strong>t of view. Not surpris<strong>in</strong>gly, <strong>the</strong> least probable<br />
op<strong>in</strong>ion is often <strong>the</strong> hardest to defend, whereas <strong>the</strong> most probable op<strong>in</strong>ion<br />
is <strong>the</strong> easiest to support.<br />
The literature search approach may promote a logical consistency among<br />
analytic products, but it has <strong>the</strong> un<strong>in</strong>tended consequence of impos<strong>in</strong>g on <strong>the</strong><br />
analyst us<strong>in</strong>g it a preexist<strong>in</strong>g mental model of <strong>the</strong> phenomena <strong>in</strong> question. The<br />
exist<strong>in</strong>g analytic products describe, implicitly or explicitly, a set of work<strong>in</strong>g<br />
hypo<strong>the</strong>ses that an analyst may wish to reflect <strong>in</strong> his or her own work. Of<br />
course, <strong>the</strong>se exist<strong>in</strong>g hypo<strong>the</strong>ses are rarely tested each time <strong>the</strong>y are <strong>in</strong>corporated<br />
<strong>in</strong>to new products. What tends to occur is that <strong>the</strong> analyst looks for current<br />
data that confirms <strong>the</strong> exist<strong>in</strong>g organizational op<strong>in</strong>ion or <strong>the</strong> op<strong>in</strong>ion that<br />
22
FINDINGS<br />
seems most probable and, consequently, is easiest to support. As this strategy<br />
is also <strong>the</strong> most time-efficient technique, it reduces <strong>the</strong> time constra<strong>in</strong>ts associated<br />
with <strong>the</strong> daily production cycle.<br />
This tendency to search for confirmatory data is not necessarily a conscious<br />
choice; ra<strong>the</strong>r, it is <strong>the</strong> result of accept<strong>in</strong>g an exist<strong>in</strong>g set of hypo<strong>the</strong>ses, develop<strong>in</strong>g<br />
a mental model based on previous corporate products, and <strong>the</strong>n try<strong>in</strong>g to<br />
augment that model with current data <strong>in</strong> order to support <strong>the</strong> exist<strong>in</strong>g hypo<strong>the</strong>ses.<br />
Although motivational and heuristic factors and a tendency toward<br />
“groupth<strong>in</strong>k” might contribute to confirmatory behavior <strong>in</strong> <strong>in</strong>telligence analysis,<br />
my observations and <strong>in</strong>terviews dur<strong>in</strong>g this study suggest that <strong>the</strong> predom<strong>in</strong>ant<br />
<strong>in</strong>fluence is selectivity bias <strong>in</strong> order to ma<strong>in</strong>ta<strong>in</strong> a corporate judgment.<br />
The ma<strong>in</strong>tenance of a corporate judgment is a pervasive and often-unstated<br />
norm <strong>in</strong> <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong>, and <strong>the</strong> taboo aga<strong>in</strong>st chang<strong>in</strong>g <strong>the</strong> corporate<br />
product l<strong>in</strong>e contributes to confirmation biases. Once any <strong>in</strong>telligence<br />
agency has given its official op<strong>in</strong>ion to policymakers, <strong>the</strong>re exists a taboo<br />
about revers<strong>in</strong>g or significantly chang<strong>in</strong>g <strong>the</strong> official or corporate position to<br />
avoid <strong>the</strong> loss of status, trust, or respect. Often, policymakers perceive a<br />
change <strong>in</strong> judgment as though <strong>the</strong> orig<strong>in</strong>al op<strong>in</strong>ion was wrong, and, although<br />
unstated, <strong>the</strong>re are significant <strong>in</strong>ternal and external social pressures and consequences<br />
associated with be<strong>in</strong>g perceived as <strong>in</strong>correct.<br />
An analyst can change an op<strong>in</strong>ion based on new <strong>in</strong>formation or by revisit<strong>in</strong>g<br />
old <strong>in</strong>formation with a new hypo<strong>the</strong>sis; <strong>in</strong> so do<strong>in</strong>g, however, he or she<br />
perceives a loss of trust and respect among those with whom <strong>the</strong> orig<strong>in</strong>al judgment<br />
was shared. Along with this perceived loss of trust, <strong>the</strong> analyst senses a<br />
loss of social capital, or power, with<strong>in</strong> his or her group. 15<br />
It is even more difficult for an <strong>in</strong>telligence agency to change its official<br />
position once it has made its judgments known to those outside of <strong>the</strong> organization.<br />
There is a sense that chang<strong>in</strong>g <strong>the</strong> official product l<strong>in</strong>e will be seen outside<br />
of its context—<strong>the</strong> acquisition of new <strong>in</strong>formation, for <strong>in</strong>stance—and that<br />
it will be perceived by <strong>the</strong> policymakers as an example of <strong>in</strong>competence or, at<br />
least, of poor performance on <strong>the</strong> part of <strong>the</strong> <strong>in</strong>telligence agency.<br />
This perception <strong>the</strong>n carries with it <strong>the</strong> threat of a loss <strong>in</strong> status, fund<strong>in</strong>g,<br />
and access to policymakers, all of which would have a detrimental effect on<br />
<strong>the</strong> ability of <strong>the</strong> <strong>in</strong>telligence agency to perform its functions. In short, it<br />
serves <strong>the</strong> <strong>in</strong>terest of <strong>the</strong> <strong>in</strong>telligence agency to be perceived as decisive<br />
15<br />
Reciprocity <strong>in</strong> this case has to do with <strong>in</strong>formation, judgment, and trust. The classic anthropological<br />
text on social reciprocity and trust with<strong>in</strong> and between groups is Marcel Mauss’s The Gift.<br />
Orig<strong>in</strong>ally published <strong>in</strong> 1950 and based <strong>in</strong> part on <strong>the</strong> work of his uncle and mentor, Emile<br />
Durkheim, Mauss’s work (Essai sur le Don <strong>in</strong> its French version) lays <strong>the</strong> foundation for his contention<br />
that reciprocity is <strong>the</strong> key to understand<strong>in</strong>g <strong>the</strong> modern concept of social capital.<br />
23
CHAPTER TWO<br />
<strong>in</strong>stead of academic and contradictory, and that message is transmitted to <strong>the</strong><br />
analysts. In response to <strong>the</strong> organizational norm, <strong>the</strong> analyst is <strong>in</strong>cl<strong>in</strong>ed to<br />
work <strong>the</strong> product l<strong>in</strong>e ra<strong>the</strong>r than change it.<br />
Our products are company products, not <strong>in</strong>dividual products. When<br />
you publish someth<strong>in</strong>g here, it’s <strong>the</strong> official voice. It’s important for<br />
us to speak with one voice.<br />
It doesn’t do us any good if people th<strong>in</strong>k we can’t make up our m<strong>in</strong>d.<br />
Access matters; if people th<strong>in</strong>k you don’t know what you’re talk<strong>in</strong>g<br />
about, <strong>the</strong>n <strong>the</strong>y stop see<strong>in</strong>g you.<br />
We already briefed one th<strong>in</strong>g. I can’t go <strong>in</strong> <strong>the</strong>re and change it now.<br />
We’ll look like idiots.<br />
When I was new, I wrote a piece that disagreed with our l<strong>in</strong>e. Let’s<br />
just say, I’m more careful about that now.<br />
Ano<strong>the</strong>r organizational norm that contributes to confirmation bias <strong>in</strong> <strong>the</strong><br />
<strong>Intelligence</strong> <strong>Community</strong> is <strong>the</strong> selection and weigh<strong>in</strong>g of data accord<strong>in</strong>g to<br />
classification. Secrets carry <strong>the</strong> imprimatur of <strong>the</strong> organization and, <strong>in</strong> turn,<br />
have more face validity than <strong>in</strong>formation collected through open sources. 16<br />
Most analysts <strong>in</strong>dicated that <strong>the</strong>y considered “secret” data collected by<br />
covert means to be more important or mean<strong>in</strong>gful than “open” or unclassified<br />
data. Analysts said that <strong>the</strong>y rely on open sources to help fill <strong>in</strong> miss<strong>in</strong>g pieces<br />
of <strong>the</strong>ir mental models but that <strong>the</strong>y test <strong>the</strong> model’s validity with secret <strong>in</strong>formation.<br />
Choos<strong>in</strong>g to rely on classified data as more mean<strong>in</strong>gful to problem<br />
solv<strong>in</strong>g and as a tool for test<strong>in</strong>g <strong>the</strong> validity of <strong>the</strong>ir hypo<strong>the</strong>ses serves to exacerbate<br />
<strong>the</strong> confirmation bias.<br />
I’m an all-source analyst, so I use whatever I can get my hands on;<br />
but, if <strong>the</strong> traffic comes from operations, I tend to pay more attention<br />
to it than to <strong>in</strong>formation <strong>in</strong> <strong>the</strong> open literature.<br />
There is someth<strong>in</strong>g special about <strong>the</strong> word “secret” <strong>in</strong> my bus<strong>in</strong>ess.<br />
It says that it must be important because people had to go and get it<br />
ra<strong>the</strong>r than its just show<strong>in</strong>g up <strong>in</strong> <strong>the</strong> news. We tend to weigh classified<br />
material as more important than o<strong>the</strong>r sources.<br />
16<br />
In research methodology, face validity is <strong>the</strong> concept that a measurement <strong>in</strong>strument appears or<br />
seems to measure what it is actually <strong>in</strong>tended to measure and requires no <strong>the</strong>oretical support<strong>in</strong>g<br />
material. In contrast, content validity depends on <strong>the</strong> content of <strong>the</strong> doma<strong>in</strong> and established <strong>the</strong>ories<br />
to determ<strong>in</strong>e its measures of validity. See David Br<strong>in</strong>berg and Joseph McGrath, Validity and<br />
<strong>the</strong> Research Process; Edward Carm<strong>in</strong>es and Richard Zeller, Reliability and Validity Assessment;<br />
Jerome Kirk and Marc Miller, Reliability and Validity <strong>in</strong> Qualitative Research; Mark Litw<strong>in</strong>,<br />
“How to measure survey reliability and validity”; William Trochim, The Research Methods<br />
Knowledge Base.<br />
24
FINDINGS<br />
We get all k<strong>in</strong>ds of sourced material, but I th<strong>in</strong>k I trust technical collection<br />
more than <strong>the</strong> o<strong>the</strong>r INTs.<br />
I try to use everyth<strong>in</strong>g we get, but, if we are jammed, I rely on<br />
sources we collect.<br />
Our value-added is classified sourc<strong>in</strong>g. Everybody has access to <strong>the</strong><br />
Web and CNN.<br />
All our customers are analysts <strong>the</strong>se days. What we br<strong>in</strong>g to <strong>the</strong><br />
party is <strong>in</strong>formation no one else has.<br />
We’re <strong>in</strong> <strong>the</strong> bus<strong>in</strong>ess of secrets. If you see that stamped on someth<strong>in</strong>g,<br />
it must be <strong>the</strong>re for a reason.<br />
The over reliance on classified <strong>in</strong>formation for hypo<strong>the</strong>sis test<strong>in</strong>g creates a<br />
situation <strong>in</strong> which <strong>the</strong> data are screened and sorted by <strong>the</strong> organization before<br />
<strong>the</strong>y are selected and tested by <strong>the</strong> analysts. Classified <strong>in</strong>formation comes<br />
from very specific types of technical and human sources, and it is filtered<br />
through very specific report<strong>in</strong>g channels. It also has a tendency to become<br />
homogeneous because of <strong>the</strong> source types and report<strong>in</strong>g mechanisms. Because<br />
it is generated and packaged <strong>in</strong> specific formats us<strong>in</strong>g specific processes, classified<br />
<strong>in</strong>formation lacks <strong>the</strong> diversity that is <strong>in</strong>herent <strong>in</strong> open <strong>in</strong>formation, and<br />
this contributes to confirmation bias.<br />
In sum, operat<strong>in</strong>g under difficult time constra<strong>in</strong>ts, try<strong>in</strong>g to make new work<br />
accord with previous products, try<strong>in</strong>g to ma<strong>in</strong>ta<strong>in</strong> <strong>the</strong> prestige and power of<br />
<strong>the</strong> organization, and assign<strong>in</strong>g greater weight to secret <strong>in</strong>formation than to<br />
open <strong>in</strong>formation have a cumulative effect, and <strong>the</strong> analyst often f<strong>in</strong>ds himself<br />
or herself try<strong>in</strong>g to produce daily products us<strong>in</strong>g <strong>the</strong> most time-efficient strategies<br />
available <strong>in</strong>stead of generat<strong>in</strong>g or test<strong>in</strong>g hypo<strong>the</strong>ses by way of refutation.<br />
The persistence of <strong>the</strong> notion of tradecraft, coupled with organizational<br />
norms, promotes <strong>the</strong> use of disjo<strong>in</strong>ted analytic strategies by separat<strong>in</strong>g <strong>in</strong>telligence<br />
analysts from o<strong>the</strong>r scientific discipl<strong>in</strong>es. These conditions have had an<br />
effect on <strong>the</strong> self-concept of analysts and have molded <strong>the</strong> way analysts perceive<br />
<strong>the</strong>ir own identity.<br />
F<strong>in</strong>d<strong>in</strong>g: <strong>Analytic</strong> Identity<br />
The self is someth<strong>in</strong>g which has a development; it is not <strong>in</strong>itially<br />
<strong>the</strong>re, at birth, but arises <strong>in</strong> <strong>the</strong> process of social experience and<br />
activity, that is, develops <strong>in</strong> <strong>the</strong> given <strong>in</strong>dividual as a result of his<br />
relations to that process as a whole and to o<strong>the</strong>r <strong>in</strong>dividuals with<strong>in</strong><br />
that process.<br />
George Mead. 17<br />
25
CHAPTER TWO<br />
Asked to def<strong>in</strong>e <strong>the</strong>ir profession, <strong>the</strong> majority of analysts described <strong>the</strong> process<br />
of analysis ra<strong>the</strong>r than <strong>the</strong> actual profession. The question, “What is an<br />
<strong>in</strong>telligence analyst?” resulted most often <strong>in</strong> a description of <strong>the</strong> work day and<br />
of <strong>the</strong> production cycle of analytic products and very seldom <strong>in</strong> an explanation<br />
of analytic methodology or a def<strong>in</strong>ition of an analyst outside of some specific<br />
context. With very few exceptions, analysts did not describe <strong>in</strong>telligence analysis<br />
as its own discipl<strong>in</strong>e with its own identity, epistemology, and research tradition.<br />
This is not necessarily uncommon. When physicians are asked to describe<br />
<strong>the</strong>ir profession, <strong>the</strong>y tend to respond with a specific subdiscipl<strong>in</strong>e: “I’m a cardio-thoracic<br />
surgeon,” for example. When asked for a more general description,<br />
however, <strong>the</strong>y tend to respond, “I’m a doctor” or “I’m a physician.” That<br />
is, <strong>in</strong> selective, <strong>in</strong>sular professional cultures, practitioners are able to def<strong>in</strong>e<br />
<strong>the</strong>ir role <strong>in</strong> both a specific and general fashion. <strong>Intelligence</strong> analysts had difficulty<br />
def<strong>in</strong><strong>in</strong>g <strong>the</strong>ir professional identity <strong>in</strong> a general way and often relied on<br />
specific context to expla<strong>in</strong> what it is that <strong>the</strong>y do and, by extension, who <strong>the</strong>y<br />
are.<br />
The perception of <strong>in</strong>dividual analysts regard<strong>in</strong>g <strong>the</strong>ir professional identity<br />
was associated most often with <strong>the</strong>ir organization’s function or with <strong>the</strong>ir own<br />
educational background and not with <strong>in</strong>telligence analysis as its own unique<br />
discipl<strong>in</strong>e.<br />
I work counternarcotics.<br />
I work counterterrorism.<br />
I’m a military analyst.<br />
I’m a leadership analyst.<br />
I’m an economist.<br />
I’m a political scientist.<br />
In addition to <strong>the</strong>se categories, many analysts described <strong>the</strong>ir professional<br />
identity <strong>in</strong> terms of <strong>in</strong>telligence collection methods or categories.<br />
I do all-source analysis.<br />
I’m a SIGINT analyst.<br />
I’m an IMINT analyst.<br />
I’m a technical analyst.<br />
17<br />
George Mead was an American pragmatist philosopher and social psychologist, who, with John<br />
Dewey, made <strong>the</strong> University of Chicago <strong>the</strong> home of pragmatist philosophy and <strong>the</strong> “Chicago<br />
School” of sociology at <strong>the</strong> end of <strong>the</strong> 19th century. George Mead, M<strong>in</strong>d, Self, and Society.<br />
26
FINDINGS<br />
The shift <strong>in</strong> focus to daily analytic products, <strong>the</strong> changes <strong>in</strong> <strong>the</strong> production<br />
cycle, and a heterogeneously def<strong>in</strong>ed professional discipl<strong>in</strong>e have had an additional<br />
effect on <strong>the</strong> professional identity of analysts with<strong>in</strong> <strong>the</strong> <strong>Intelligence</strong><br />
<strong>Community</strong>. Analysts often commented that <strong>the</strong>y perceived <strong>the</strong>ir job and <strong>the</strong>ir<br />
daily work rout<strong>in</strong>e as more ak<strong>in</strong> to report<strong>in</strong>g than to analysis.<br />
Basically, on a day-to-day basis, it’s like work<strong>in</strong>g at CNN, only<br />
we’re CNN with secrets. Actually, it’s more like CNN’s Headl<strong>in</strong>e<br />
News.<br />
Imag<strong>in</strong>e USA Today with spies—bullet po<strong>in</strong>ts, short paragraphs,<br />
<strong>the</strong> occasional picture. You know, short and simple.<br />
I th<strong>in</strong>k of myself as a writer for <strong>the</strong> most important newspaper <strong>in</strong> <strong>the</strong><br />
world.<br />
Many analysts expressed dissatisfaction with <strong>the</strong> shift <strong>in</strong> work processes<br />
from long-term forecasts and toward current report<strong>in</strong>g and <strong>the</strong> subsequent<br />
shift <strong>in</strong> <strong>the</strong>ir own professional identity with<strong>in</strong> <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong>.<br />
The current sentiment about identity was often contrasted aga<strong>in</strong>st an idealized<br />
past that was described as be<strong>in</strong>g freer of current production practices and<br />
products.<br />
About 15 years ago, I would have described myself as a scholar.<br />
Now, I’m a reporter. I’ve got 15 people try<strong>in</strong>g to change my work<br />
<strong>in</strong>to bullet po<strong>in</strong>ts. Presumably, nobody has time to read anymore.<br />
When I jo<strong>in</strong>ed, it seemed that <strong>the</strong> word “analyst” was shorthand for<br />
“problem solver.” Now, it’s shorthand for “reporter.”<br />
I’m proud of where I work. I’m proud of <strong>the</strong> job that we do. But, it is<br />
hard to take pride <strong>in</strong> one paragraph. I have to look at <strong>the</strong> big picture,<br />
or I would get discouraged.<br />
I spend most of my wak<strong>in</strong>g hours do<strong>in</strong>g this, but I still can’t really<br />
say what an analyst is.<br />
I’m not a reporter, and I’m not an academic. I’m somewhere <strong>in</strong><br />
between.<br />
The heterogeneous descriptions and def<strong>in</strong>itions of <strong>in</strong>telligence analysis as a<br />
professional discipl<strong>in</strong>e were consistent f<strong>in</strong>d<strong>in</strong>gs dur<strong>in</strong>g this study, <strong>in</strong>dicat<strong>in</strong>g<br />
that <strong>the</strong>re needs to be a clear articulation and dissem<strong>in</strong>ation of <strong>the</strong> identity and<br />
epistemology of <strong>in</strong>telligence analysis. A clearly def<strong>in</strong>ed professional identity<br />
would help to promote group cohesion, establish <strong>in</strong>teragency ties and relationships,<br />
and reduce <strong>in</strong>tra- and <strong>in</strong>teragency communication barriers by establish<strong>in</strong>g<br />
a professional class throughout <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong>. At an<br />
<strong>in</strong>dividual level, a clearly def<strong>in</strong>ed professional identity helps to reduce job dis-<br />
27
CHAPTER TWO<br />
satisfaction and anxiety by giv<strong>in</strong>g larger mean<strong>in</strong>g to an <strong>in</strong>dividual’s daily<br />
actions. 18<br />
F<strong>in</strong>d<strong>in</strong>g: <strong>Analytic</strong> Tra<strong>in</strong><strong>in</strong>g<br />
When I started, <strong>the</strong>re wasn’t much tra<strong>in</strong><strong>in</strong>g available. There were a<br />
few advanced courses, but, for <strong>the</strong> most part, it was on <strong>the</strong> job.<br />
A professional identity is generally a discipl<strong>in</strong>ary norm, and it regularly<br />
occurs <strong>in</strong> o<strong>the</strong>r doma<strong>in</strong>s that are as cognitively demand<strong>in</strong>g as <strong>in</strong>telligence<br />
analysis, such as medic<strong>in</strong>e, aeronautics, and jurisprudence. These o<strong>the</strong>r<br />
doma<strong>in</strong>s practice a general system of professional enculturation that<br />
progresses from a basic education program to specialized tra<strong>in</strong><strong>in</strong>g. 19 These<br />
tra<strong>in</strong><strong>in</strong>g programs help to differentiate communities of practitioners from <strong>the</strong><br />
general public, create specific and unique professional identities, and develop<br />
basic communication and task-specific skills. They also help <strong>the</strong> profession to<br />
cont<strong>in</strong>ue to advance through formal research efforts.<br />
This is not <strong>the</strong> case with<strong>in</strong> <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong> as a whole. Generally,<br />
<strong>the</strong> <strong>in</strong>telligence agencies that do provide basic and advanced tra<strong>in</strong><strong>in</strong>g do<br />
so <strong>in</strong>dependently of o<strong>the</strong>r <strong>in</strong>telligence organizations. 20 A number of <strong>in</strong>telligence<br />
agencies do not provide basic analytic tra<strong>in</strong><strong>in</strong>g at all or have only<br />
recently begun to do so, rely<strong>in</strong>g <strong>in</strong>stead on on-<strong>the</strong>-job experiences and <strong>in</strong>formal<br />
mentor<strong>in</strong>g.<br />
We haven’t had a culture of tra<strong>in</strong><strong>in</strong>g analysts here <strong>in</strong> <strong>the</strong> past. It’s<br />
only <strong>in</strong> <strong>the</strong> last year or so that we’ve started to change that.<br />
When I started here, analysts were considered adm<strong>in</strong>istrative personnel.<br />
We didn’t have a tra<strong>in</strong><strong>in</strong>g program. I th<strong>in</strong>k <strong>the</strong>y just started<br />
one this year.<br />
My background was technical analysis, and we had a lot of operational<br />
tra<strong>in</strong><strong>in</strong>g where I used to work. But now that I’m do<strong>in</strong>g more<br />
strategic analysis, I’ve had to make it up as I go along.<br />
18<br />
Philip Cushman, Construct<strong>in</strong>g <strong>the</strong> Self, Construct<strong>in</strong>g America; Anthony Giddens, Modernity<br />
and Self-Identity; John P. Hewitt, Self and Society; Lewis P. H<strong>in</strong>chman and Sandra K. H<strong>in</strong>chman,<br />
Memory, Identity, <strong>Community</strong>; Carl Jung, The Undiscovered Self; George Lev<strong>in</strong>e, ed., Constructions<br />
of <strong>the</strong> Self.<br />
19<br />
Enculturation is <strong>the</strong> process or mechanism by which a culture is <strong>in</strong>stilled <strong>in</strong> a human be<strong>in</strong>g from<br />
birth until death. In this <strong>in</strong>stance, professional enculturation refers to <strong>the</strong> acquisition of a professional<br />
identity through specific cultural rituals and practices, as displayed, for example, by practitioners<br />
who have graduated from medical school, law school, and basic military tra<strong>in</strong><strong>in</strong>g.<br />
20<br />
See footnote 7 <strong>in</strong> <strong>the</strong> Introduction for several recent cross-agency tra<strong>in</strong><strong>in</strong>g <strong>in</strong>itiatives.<br />
28
FINDINGS<br />
We have a basic tra<strong>in</strong><strong>in</strong>g program, but it is different from <strong>the</strong> o<strong>the</strong>r<br />
agencies. Our mission is different. The problem is that we talk past<br />
each o<strong>the</strong>r all <strong>the</strong> time.<br />
When I got hired, I had an advanced degree. People assumed that, if<br />
I had a Masters, I could just figure out what I was supposed to do.<br />
The focus of tra<strong>in</strong><strong>in</strong>g with<strong>in</strong> <strong>the</strong> community varies widely and is shaped by<br />
<strong>the</strong> mission of <strong>the</strong> agency, such as technical, tactical, and operational. Many<br />
spend a considerable amount of time teach<strong>in</strong>g new analysts how to prepare<br />
brief<strong>in</strong>gs, write papers, and perform adm<strong>in</strong>istrative functions unique to <strong>the</strong>ir<br />
agency. This is logical from <strong>the</strong> perspective of agency managers, who naturally<br />
believe that <strong>in</strong>vestments made <strong>in</strong> personnel, tra<strong>in</strong><strong>in</strong>g, and read<strong>in</strong>ess ought<br />
to be tailored specifically for <strong>the</strong>ir own organizations.<br />
The problem with an agency-centric view is that, without a general communitywide<br />
tra<strong>in</strong><strong>in</strong>g program for <strong>in</strong>telligence analysts, agencies and <strong>the</strong>ir analysts<br />
have difficulty f<strong>in</strong>d<strong>in</strong>g, communicat<strong>in</strong>g, and <strong>in</strong>teract<strong>in</strong>g with one<br />
ano<strong>the</strong>r. 21 Analysts often said <strong>the</strong>y were dis<strong>in</strong>cl<strong>in</strong>ed to draw on resources outside<br />
of <strong>the</strong>ir own agency, <strong>in</strong>dicat<strong>in</strong>g that ei<strong>the</strong>r <strong>the</strong>y do not know whom to<br />
contact or <strong>the</strong>ir experience <strong>in</strong> <strong>the</strong> past has been <strong>in</strong>fluenced by a strict organizational<br />
focus.<br />
The media keep talk<strong>in</strong>g about <strong>in</strong>telligence failures and communication<br />
breakdowns <strong>in</strong> <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong>. What do <strong>the</strong>y<br />
expect? We don’t even speak <strong>the</strong> same language.<br />
It’s taken me 15 years to build my own network. If I didn’t have my<br />
own contacts, I wouldn’t know who to call.<br />
I don’t bo<strong>the</strong>r go<strong>in</strong>g outside. Our focus is different here.<br />
We have official channels, but it only really works if you trust <strong>the</strong><br />
person on <strong>the</strong> o<strong>the</strong>r end of <strong>the</strong> phone. That’s hard to do if you don’t<br />
know <strong>the</strong>m.<br />
Without an <strong>in</strong>clusive communitywide basic tra<strong>in</strong><strong>in</strong>g program, differentiation<br />
between <strong>the</strong> <strong>in</strong>telligence analysis discipl<strong>in</strong>e, as a whole, and o<strong>the</strong>r fields<br />
of study is unlikely. A community of practitioners will have difficulty <strong>in</strong>teract<strong>in</strong>g<br />
with one ano<strong>the</strong>r, communicat<strong>in</strong>g between and with<strong>in</strong> organizations, and<br />
establish<strong>in</strong>g a professional identity, which is a key <strong>in</strong>gredient <strong>in</strong> <strong>the</strong> development<br />
of a professional discipl<strong>in</strong>e.<br />
21<br />
Stephen Marr<strong>in</strong>, <strong>CIA</strong>’s Kent School: A Step <strong>in</strong> <strong>the</strong> Right Direction and “Improv<strong>in</strong>g <strong>CIA</strong> Analysis<br />
by Overcom<strong>in</strong>g Institutional Obstacles.”<br />
29
PART II<br />
Ethnography of Analysis<br />
31
Taxonomy of <strong>Intelligence</strong> Analysis Variables<br />
Systemic Variables Systematic Variables Idiosyncratic Variables Communicative Variables<br />
Organization User Requirements Weltanschauung Formal<br />
Internal Operations (worldview) Inter-organization<br />
Structure Information Acquisition Affiliation Hierarchical<br />
Leadership Collection Methods Familial Inter-division<br />
<strong>Culture</strong> Overt Cultural Inter-group<br />
History Covert Ethnic Intra-organization<br />
Traditions Information Reliability Religious Hierarchical<br />
Social Practice<br />
Reproducible<br />
Social<br />
Intra-division<br />
Taboo<br />
L<strong>in</strong>guistic<br />
Intra-group<br />
Consistent<br />
Group Characteristics<br />
Political<br />
Individual<br />
Information Validity<br />
Psychology<br />
Hierarchical<br />
Hierarchy<br />
Historical<br />
Bias<br />
Inter-division<br />
Resources & Incentives S<strong>in</strong>gle Source Personality Profile Intra-group<br />
Manpower Dual Source Security Trust Informal<br />
Budget Triangulation Cognitive Process<strong>in</strong>g Inter-organization<br />
Technology Information Archive Learn<strong>in</strong>g Style Hierarchical<br />
Assets<br />
Storage<br />
Information Acquisition<br />
Inter-group<br />
Inter-division<br />
R&D<br />
Access<br />
Facilities Correlation Information Process- Intra-organization<br />
Work Groups-Teams<br />
External<br />
Consumer Needs<br />
Time and Imperatives<br />
Consumer Use<br />
Consumer Structure<br />
Consumer Hierarchy<br />
Conumer Report<strong>in</strong>g<br />
Retrieval<br />
<strong>Analytic</strong>al Methodology<br />
Approach<br />
Intuitive<br />
Structured<br />
Semi-structured<br />
Information Process<strong>in</strong>g<br />
Historical Information<br />
<strong>in</strong>g<br />
Expertise<br />
Problem-solv<strong>in</strong>g<br />
Decisionmak<strong>in</strong>g<br />
Cognitive Load<br />
Speed/Accuracy<br />
Stress Effects<br />
Hierarchical<br />
Intra-division<br />
Intra-group<br />
Individual<br />
Hierarchical<br />
Inter-group<br />
Intra-group<br />
Technology<br />
Politics Current Information Education Networked Analysis<br />
Internal-Organization Decision Strategies<br />
Doma<strong>in</strong><br />
Collaboration<br />
Policy<br />
Estimative<br />
Location<br />
Tradition Predictive Mentor<br />
Taboo<br />
Report<strong>in</strong>g<br />
Tra<strong>in</strong><strong>in</strong>g<br />
Security/Access<br />
Verbal Methods<br />
Organizational<br />
External-National Written Methods Doma<strong>in</strong><br />
Law<br />
Procedural<br />
Policy<br />
Read<strong>in</strong>ess<br />
External-International<br />
Resources<br />
Security<br />
Facilities<br />
Denial<br />
Deception<br />
Policy<br />
32
CHAPTER THREE<br />
A Taxonomy of <strong>Intelligence</strong> Variables 1<br />
Science is organized knowledge.<br />
Herbert Spencer 2<br />
Aristotle may be <strong>the</strong> fa<strong>the</strong>r of scientific classification, but it was biologist<br />
Carolus L<strong>in</strong>naeus who <strong>in</strong>troduced <strong>the</strong> first formal taxonomy—k<strong>in</strong>gdom, class,<br />
order, genera, and species—<strong>in</strong> his Systema Naturae <strong>in</strong> 1735. By codify<strong>in</strong>g <strong>the</strong><br />
nam<strong>in</strong>g conventions <strong>in</strong> biology, L<strong>in</strong>naeus’s work provided a reference po<strong>in</strong>t<br />
for future discoveries. Darw<strong>in</strong>’s development of an evolutionary <strong>the</strong>ory, for<br />
example, benefited greatly from L<strong>in</strong>naeus’s creation of a hierarchical group<strong>in</strong>g<br />
of related organisms. The Systema Naturae taxonomy was not a fixed<br />
product but ra<strong>the</strong>r a liv<strong>in</strong>g document. L<strong>in</strong>naeus himself revised it through 10<br />
editions, and later biologists have cont<strong>in</strong>ued to modify it. 3<br />
In response to new discoveries and <strong>the</strong> development of new research methods<br />
<strong>in</strong> o<strong>the</strong>r doma<strong>in</strong>s, taxonomies were created to help organize those discipl<strong>in</strong>es<br />
and to assist researchers <strong>in</strong> identify<strong>in</strong>g variables that required additional<br />
study. The development of specific taxonomies—from highly structured systems,<br />
such as <strong>the</strong> periodic table of chemical elements, to less structured<br />
approaches, such as Bloom’s Taxonomy 4 —is a key step <strong>in</strong> organiz<strong>in</strong>g knowl-<br />
1<br />
A version of this chapter, “Develop<strong>in</strong>g a Taxonomy of <strong>Intelligence</strong> Analysis Variables,” orig<strong>in</strong>ally<br />
appeared <strong>in</strong> Studies <strong>in</strong> <strong>Intelligence</strong> 47, no. 3 (2003): 61–71.<br />
2<br />
Herbert Spencer’s The Study of Sociology, published <strong>in</strong> 1874, set <strong>the</strong> stage for <strong>the</strong> emergence of<br />
sociology as a discipl<strong>in</strong>e.<br />
3<br />
Ernst Haeckel <strong>in</strong>troduced phylum to <strong>in</strong>clude related classes and family to <strong>in</strong>clude related genera <strong>in</strong><br />
1866. The L<strong>in</strong>naeus taxonomy is currently be<strong>in</strong>g revised to accommodate genomic mapp<strong>in</strong>g data.<br />
33
CHAPTER THREE<br />
edge and fur<strong>the</strong>r<strong>in</strong>g <strong>the</strong> growth of <strong>in</strong>dividual discipl<strong>in</strong>es. A taxonomy differentiates<br />
doma<strong>in</strong>s by specify<strong>in</strong>g <strong>the</strong> scope of <strong>in</strong>quiry, codify<strong>in</strong>g nam<strong>in</strong>g<br />
conventions, identify<strong>in</strong>g areas of <strong>in</strong>terest, help<strong>in</strong>g to set research priorities,<br />
and often lead<strong>in</strong>g to new <strong>the</strong>ories. Taxonomies are signposts, <strong>in</strong>dicat<strong>in</strong>g what<br />
is known and what has yet to be discovered.<br />
This chapter, to which more than 100 <strong>in</strong>dividuals contributed <strong>the</strong>ir time and<br />
advice, proposes a taxonomy for <strong>the</strong> field of <strong>in</strong>telligence. It is my hope that <strong>the</strong><br />
result<strong>in</strong>g organized list<strong>in</strong>g of variables will help practitioners streng<strong>the</strong>n <strong>the</strong>ir<br />
understand<strong>in</strong>g of <strong>the</strong> analytic process and po<strong>in</strong>t <strong>the</strong>m <strong>in</strong> directions that need<br />
additional attention.<br />
<strong>Intelligence</strong> Analysis<br />
We could have talked about <strong>the</strong> science of <strong>in</strong>telligence, but . . . <strong>the</strong><br />
science of <strong>in</strong>telligence is yet to be <strong>in</strong>vented.<br />
Charles Allen 5<br />
Develop<strong>in</strong>g an <strong>in</strong>telligence taxonomy is complicated by <strong>the</strong> fact that <strong>the</strong> literature<br />
<strong>in</strong> <strong>the</strong> field is episodic and reflects specialized areas of concern. Perhaps<br />
it is best to beg<strong>in</strong> with what appears to be a key dist<strong>in</strong>ction between<br />
general analysis and <strong>in</strong>telligence analysis, that of solv<strong>in</strong>g a problem <strong>in</strong> <strong>the</strong><br />
public doma<strong>in</strong>, and solv<strong>in</strong>g a problem <strong>in</strong> a private or secret doma<strong>in</strong>.<br />
Ronald Garst articulates two arguments that are used to support this dist<strong>in</strong>ction:<br />
<strong>in</strong>telligence analysis is more time sensitive than analysis <strong>in</strong> o<strong>the</strong>r<br />
doma<strong>in</strong>s and it deals with <strong>in</strong>formation that <strong>in</strong>tentionally may be deceptive. 6<br />
The notion that <strong>in</strong>telligence is uniquely time sensitive is questionable, however.<br />
<strong>Intelligence</strong> is not <strong>the</strong> only doma<strong>in</strong> where time constra<strong>in</strong>ts can force<br />
decisions to be made before data are complete. Time is always a key variable,<br />
whe<strong>the</strong>r one is <strong>in</strong> an operat<strong>in</strong>g room or <strong>in</strong> a cockpit. To be sure, <strong>in</strong>telligence is<br />
a life and death profession, but so are medic<strong>in</strong>e and mass transportation. In<br />
each <strong>in</strong>stance, failure can mean casualties.<br />
Garst’s po<strong>in</strong>t about <strong>in</strong>tentional deception is more germane. With <strong>the</strong> possible<br />
exception of bus<strong>in</strong>ess and f<strong>in</strong>ancial markets, analysts <strong>in</strong> o<strong>the</strong>r fields seldom<br />
deal with <strong>in</strong>tentional deception. As discussed <strong>in</strong> Chapter One, Michael<br />
Warner makes a good case for secrecy be<strong>in</strong>g <strong>the</strong> primary variable dist<strong>in</strong>guish-<br />
4<br />
See Benjam<strong>in</strong> S. Bloom, Taxonomy of Educational Objectives. Bloom’s taxonomy is a classification<br />
of levels of <strong>in</strong>tellectual behavior <strong>in</strong> learn<strong>in</strong>g, <strong>in</strong>clud<strong>in</strong>g knowledge, comprehension, application,<br />
analysis, syn<strong>the</strong>sis, and evaluation.<br />
5<br />
Comment made by <strong>the</strong> Associate Director of Central <strong>Intelligence</strong> for Collection at a public sem<strong>in</strong>ar<br />
on <strong>in</strong>telligence at Harvard University, spr<strong>in</strong>g 2000.<br />
6<br />
Ronald Garst, A Handbook of <strong>Intelligence</strong> Analysis.<br />
34
A TAXONOMY OF INTELLIGENCE<br />
<strong>in</strong>g <strong>in</strong>telligence from o<strong>the</strong>r such activities. 7 He argues that <strong>the</strong> behavior of <strong>the</strong><br />
subject of <strong>in</strong>telligence changes if <strong>the</strong> subject is aware of be<strong>in</strong>g observed or<br />
analyzed. As discussed earlier, Warner’s argument is supported by a long history<br />
of psychological research, beg<strong>in</strong>n<strong>in</strong>g with an experimental program<br />
between 1927 and 1930 at Western Electric’s Hawthorne Works <strong>in</strong> Chicago. 8<br />
Intentional deception can occur outside <strong>in</strong>telligence—<strong>in</strong> connection with<br />
certa<strong>in</strong> law enforcement functions, for example—but most of <strong>the</strong> professional<br />
literature treats this as <strong>the</strong> exception ra<strong>the</strong>r than <strong>the</strong> rule. In <strong>the</strong> case of <strong>in</strong>telligence<br />
analysis, deception is <strong>the</strong> rule; <strong>the</strong> validity of <strong>the</strong> data is always <strong>in</strong><br />
doubt. Moreover, <strong>in</strong>telligence analysts are specifically tra<strong>in</strong>ed to take deception<br />
<strong>in</strong>to account as part of <strong>the</strong> analytic process—to look for anomalies and<br />
outliers <strong>in</strong>stead of focus<strong>in</strong>g on <strong>the</strong> central tendencies of distribution.<br />
The taxonomy be<strong>in</strong>g developed here requires a def<strong>in</strong>ition of <strong>in</strong>telligence<br />
analysis that is specific to <strong>the</strong> field. <strong>Intelligence</strong> pioneer Sherman Kent, who<br />
saw <strong>in</strong>telligence as a “special category of knowledge,” laid <strong>the</strong> foundation for<br />
understand<strong>in</strong>g <strong>the</strong> activities <strong>in</strong>herent <strong>in</strong> <strong>in</strong>telligence analysis by demonstrat<strong>in</strong>g<br />
that <strong>the</strong> analytic process itself was subject to be<strong>in</strong>g analyzed. 9 Kent’s approach<br />
to analysis was to reduce <strong>the</strong> process to smaller functional components for<br />
<strong>in</strong>dividual study. 10 For example, he described <strong>in</strong>telligence analysis as hav<strong>in</strong>g a<br />
basic descriptive element, a current report<strong>in</strong>g element, and an estimative element.<br />
Follow<strong>in</strong>g suit, o<strong>the</strong>r authors focused attention on <strong>the</strong> process or methodological<br />
elements of <strong>in</strong>telligence analysis. In <strong>Intelligence</strong> Research Methodology,<br />
Jerome Clauser and Sandra Weir followed Kent’s three functional areas<br />
and went on to describe basic research foundations and <strong>the</strong> <strong>in</strong>ductive and<br />
deductive models for perform<strong>in</strong>g <strong>in</strong>telligence analysis. 11 Garst’s Handbook of<br />
<strong>Intelligence</strong> Analysis conta<strong>in</strong>s less background <strong>in</strong> basic research methods than<br />
Clauser and Weir’s book, but it is more focused on <strong>the</strong> <strong>in</strong>telligence cycle. 12<br />
Bruce Berkowitz and Allan Goodman highlight <strong>the</strong> process of strategic<br />
<strong>in</strong>telligence and def<strong>in</strong>e <strong>in</strong>telligence analysis as: “[T]he process of evaluat<strong>in</strong>g<br />
and transform<strong>in</strong>g raw data <strong>in</strong>to descriptions, explanations, and conclusions for<br />
<strong>in</strong>telligence consumers.” 13 Lisa Krizan, too, focuses on process. She writes<br />
7<br />
Michael Warner.<br />
8<br />
The Hawthorne Effect. See footnote 5 <strong>in</strong> Chapter Two.<br />
9<br />
Sherman Kent, Strategic <strong>Intelligence</strong> for American World Policy.<br />
10<br />
See Chapter Seven for a fuller discussion of this approach, now usually referred to as metaanalysis.<br />
11<br />
Jerome K. Clauser and Sandra M. Weir, <strong>Intelligence</strong> Research Methodology.<br />
12<br />
See also: Morgan Jones, The Th<strong>in</strong>ker’s Toolkit. Jones’s book is a popular version of <strong>the</strong> work of<br />
Garst and Clauser and Weir <strong>in</strong> that it describes a collection of analytic methods and techniques for<br />
problem-solv<strong>in</strong>g; however, <strong>the</strong> methods are not necessarily specific to <strong>in</strong>telligence.<br />
35
CHAPTER THREE<br />
that, “At <strong>the</strong> very least, analysis should fully describe <strong>the</strong> phenomenon under<br />
study, account<strong>in</strong>g for as many relevant variables as possible. At <strong>the</strong> next<br />
higher level of analysis, a thorough explanation of <strong>the</strong> phenomenon is<br />
obta<strong>in</strong>ed, through <strong>in</strong>terpretation of <strong>the</strong> significance and effects of its elements<br />
on <strong>the</strong> whole.” 14 In addition, several authors have written about <strong>in</strong>dividual analytic<br />
approaches. 15<br />
Although <strong>the</strong> referenced works focus on methods and techniques, <strong>the</strong>y do<br />
not suggest that analysis is limited to <strong>the</strong>se devices. The view that analysis is<br />
both a process and a collection of specific techniques is explicit <strong>in</strong> <strong>the</strong> above<br />
def<strong>in</strong>itions. Analysis is seen as an action that <strong>in</strong>corporates a variety of tools to<br />
solve a problem. Different analytic methods have someth<strong>in</strong>g to offer different<br />
analytic tasks.<br />
Although largely implicit <strong>in</strong> <strong>the</strong> above def<strong>in</strong>itions, analysis is also seen as a<br />
product of cognition, and some authors directly l<strong>in</strong>k <strong>the</strong> two. Robert Mathams<br />
def<strong>in</strong>es analysis as: “[T]he break<strong>in</strong>g down of a large problem <strong>in</strong>to a number of<br />
smaller problems and perform<strong>in</strong>g mental operations on <strong>the</strong> data <strong>in</strong> order to<br />
arrive at a conclusion or generalization.” 16 Avi Shlaim writes: “S<strong>in</strong>ce <strong>the</strong> facts<br />
do not speak for <strong>the</strong>mselves but need to be <strong>in</strong>terpreted, it is <strong>in</strong>evitable that <strong>the</strong><br />
<strong>in</strong>dividual human propensities of an <strong>in</strong>telligence officer will enter <strong>in</strong>to <strong>the</strong> process<br />
of evaluation.” 17 Yet o<strong>the</strong>rs describe analysis as a process whereby:<br />
“[I]nformation is compared and collated with o<strong>the</strong>r data, and conclusions that<br />
also <strong>in</strong>corporate <strong>the</strong> memory and judgment of <strong>the</strong> <strong>in</strong>telligence analyst are<br />
derived from it.” 18<br />
Several authors make <strong>the</strong> case that analysis is not just a product of cognition<br />
but is itself a cognitive process. J. R. Thompson and colleagues write that<br />
“[I]ntelligence analysis is an <strong>in</strong>ternal, concept-driven activity ra<strong>the</strong>r than an<br />
external data-driven activity.” 19 In his Psychology of <strong>Intelligence</strong> Analysis,<br />
Heuer observes: “<strong>Intelligence</strong> analysis is fundamentally a mental process, but<br />
understand<strong>in</strong>g this process is h<strong>in</strong>dered by <strong>the</strong> lack of conscious awareness of<br />
<strong>the</strong> work<strong>in</strong>gs of our own m<strong>in</strong>ds.” 20 Ephraim Kam comments: “The process of<br />
<strong>in</strong>telligence analysis and assessment is a very personal one. There is no<br />
agreed-upon analytical schema, and <strong>the</strong> analyst must primarily use his belief<br />
13<br />
Bruce D. Berkowitz and Allan E. Goodman, Strategic <strong>Intelligence</strong> for American National Security,<br />
85. See Chapter Four for more on <strong>the</strong> <strong>in</strong>telligence cycle.<br />
14<br />
Lisa Krizan, <strong>Intelligence</strong> Essentials for Everyone.<br />
15<br />
See <strong>the</strong> apprendix for a list<strong>in</strong>g of <strong>the</strong> literature.<br />
16<br />
Robert Mathams, “The <strong>Intelligence</strong> Analyst’s Notebook.”<br />
17<br />
Avi Shlaim, “Failures <strong>in</strong> National <strong>Intelligence</strong> Estimates: The Case of <strong>the</strong> Yom Kippur War.”<br />
18<br />
John Quirk et al., The Central <strong>Intelligence</strong> Agency: A Photographic History.<br />
19<br />
J. R. Thompson, R. Hopf-Weichel, and R. Geiselman, The Cognitive Bases of <strong>Intelligence</strong><br />
Analysis.<br />
20<br />
Richards J. Heuer, Jr., Psychology of <strong>Intelligence</strong> Analysis.<br />
36
A TAXONOMY OF INTELLIGENCE<br />
system to make assumptions and <strong>in</strong>terpret <strong>in</strong>formation. His assumptions are<br />
usually implicit ra<strong>the</strong>r than explicit and may not be apparent even to him.” 21<br />
These def<strong>in</strong>itions reflect <strong>the</strong> o<strong>the</strong>r end of <strong>the</strong> spectrum from those concerned<br />
with tools and techniques. They suggest that <strong>the</strong> analytic process is a<br />
construction of <strong>the</strong> human m<strong>in</strong>d and is significantly different from <strong>in</strong>dividual<br />
to <strong>in</strong>dividual or group to group. Certa<strong>in</strong>ly, Kam goes far<strong>the</strong>st along this path,<br />
but even he does not suggest that one forgo tools; ra<strong>the</strong>r, he says that <strong>the</strong> process<br />
of choos<strong>in</strong>g <strong>the</strong> tool is governed by cognition as well.<br />
Recogniz<strong>in</strong>g that <strong>the</strong> scope of <strong>in</strong>telligence analysis is so broad that it<br />
<strong>in</strong>cludes not only methods but also <strong>the</strong> cognitive process is a significant step.<br />
View<strong>in</strong>g analysis as a cognitive process opens <strong>the</strong> door to a complex array of<br />
variables. The psychology of <strong>the</strong> <strong>in</strong>dividual analyst must be considered, along<br />
with <strong>in</strong>dividual analytic tools. In <strong>the</strong> broadest sense, this means not merely<br />
understand<strong>in</strong>g <strong>the</strong> <strong>in</strong>dividual psyche but also understand<strong>in</strong>g <strong>the</strong> variables that<br />
<strong>in</strong>teract with that psyche. In o<strong>the</strong>r words, <strong>in</strong>telligence analysis is <strong>the</strong> sociocognitive<br />
process, 22 occurr<strong>in</strong>g with<strong>in</strong> a secret doma<strong>in</strong>, by which a collection of<br />
methods is used to reduce a complex issue to a set of simpler issues.<br />
Develop<strong>in</strong>g <strong>the</strong> Taxonomy<br />
The first step of science is to know one th<strong>in</strong>g from ano<strong>the</strong>r. This<br />
knowledge consists <strong>in</strong> <strong>the</strong>ir specific dist<strong>in</strong>ctions; but <strong>in</strong> order that it<br />
may be fixed and permanent dist<strong>in</strong>ct names must be given to different<br />
th<strong>in</strong>gs, and those names must be recorded and remembered.<br />
Carolus L<strong>in</strong>naeus<br />
My research was designed to isolate variables that affect <strong>the</strong> analytic process.<br />
The result<strong>in</strong>g taxonomy is meant to establish parameters and to stimulate<br />
dialogue <strong>in</strong> order to develop ref<strong>in</strong>ements. Although a hierarchic list is artificial<br />
and rigid, it is a first step <strong>in</strong> clarify<strong>in</strong>g areas for future research. The actual<br />
variables are considerably more fluid and <strong>in</strong>terconnected than such a structure<br />
suggests. Once <strong>the</strong> <strong>in</strong>dividual elements are ref<strong>in</strong>ed through challenges <strong>in</strong> <strong>the</strong><br />
literature, <strong>the</strong>y might be better represented by a l<strong>in</strong>k or web diagram. 23<br />
To create this <strong>in</strong>telligence analysis taxonomy, I used Alexander Erv<strong>in</strong>’s<br />
applied anthropological approach, which employs multiple data collection<br />
methods to triangulate results. 24 I also drew on Robert White’s mental work-<br />
21<br />
Ephraim Kam, Surprise Attack. The Victim’s Perspective, 120<br />
22<br />
That is, analysis does not occur <strong>in</strong> a vacuum. It is socially constructed. See Lev Vygotsky, M<strong>in</strong>d<br />
and Society.<br />
23<br />
See Chapter Four for Judith Meister Johnston’s systems analysis approach to describ<strong>in</strong>g <strong>the</strong> fluidity<br />
of <strong>the</strong> <strong>in</strong>telligence process.<br />
37
CHAPTER THREE<br />
load model, David Meister’s behavioral model, and <strong>the</strong> cognitive process<br />
model of Gary Kle<strong>in</strong> and his colleagues. 25 Each model focuses on a different<br />
aspect of human performance: White’s exam<strong>in</strong>es <strong>the</strong> actual task and task<br />
requirements; Meister’s looks at <strong>the</strong> behavior of <strong>in</strong>dividuals perform<strong>in</strong>g a<br />
task; and Kle<strong>in</strong>’s uses verbal protocols to identify <strong>the</strong> cognitive processes of<br />
<strong>in</strong>dividuals perform<strong>in</strong>g a task.<br />
Survey<strong>in</strong>g <strong>the</strong> literature. My research began with a review of <strong>the</strong> literature,<br />
both for background <strong>in</strong>formation and for <strong>the</strong> identification of variables. The<br />
<strong>in</strong>telligence literature produced by academics and practitioners tends to be<br />
episodic, or case-based. This is not unique to <strong>the</strong> field of <strong>in</strong>telligence. A number<br />
of discipl<strong>in</strong>es—medic<strong>in</strong>e, bus<strong>in</strong>ess, and law, for example—are also casebased.<br />
Many of <strong>the</strong> texts were general or <strong>the</strong>oretical ra<strong>the</strong>r than episodic.<br />
Aga<strong>in</strong>, this is not an uncommon phenomenon. The review yielded 2,432 case<br />
studies, journal articles, technical reports, transcripts of public speeches, and<br />
books related to <strong>the</strong> topic. I <strong>the</strong>n narrowed <strong>the</strong> list to 374 pert<strong>in</strong>ent texts on<br />
which a taxonomy of <strong>in</strong>telligence analysis could be built, and I analyzed <strong>the</strong>m<br />
to identify <strong>in</strong>dividual variables and categories of variables that affect <strong>in</strong>telligence<br />
analysis. 26<br />
Us<strong>in</strong>g a methodology known as “Q-Sort,” by which variables are sorted and<br />
categorized accord<strong>in</strong>g to type, I read each text and recorded <strong>the</strong> variables that<br />
each author identified. 27 These variables were <strong>the</strong>n sorted by similarity <strong>in</strong>to<br />
groups. Four broad categories of analytic variables emerged from this process.<br />
28<br />
Ref<strong>in</strong><strong>in</strong>g <strong>the</strong> prototype. Next, I used <strong>the</strong> prelim<strong>in</strong>ary taxonomy derived from<br />
my read<strong>in</strong>g of <strong>the</strong> literature to structure <strong>in</strong>terviews with 51 substantive experts<br />
and 39 <strong>in</strong>telligence novices. In tandem, I conducted two focus group sessions,<br />
with five <strong>in</strong>dividuals <strong>in</strong> each group. As a result of <strong>the</strong> <strong>in</strong>terviews and focus<br />
group discussions, I added some variables to each category, moved some to<br />
different categories, and removed some that appeared redundant.<br />
Test<strong>in</strong>g <strong>in</strong> a controlled sett<strong>in</strong>g. F<strong>in</strong>ally, to compare <strong>the</strong> taxonomy with specific<br />
analytic behaviors, I watched participants <strong>in</strong> a controlled <strong>in</strong>telligence<br />
analysis–tra<strong>in</strong><strong>in</strong>g environment. Tra<strong>in</strong>ees were given <strong>in</strong>formation on specific<br />
24<br />
Alexander Erv<strong>in</strong>, Applied Anthropology. See Chapter One, note 4 for a def<strong>in</strong>ition of triangulation.<br />
25<br />
Robert White, Task Analysis Methods; David Meister, Behavioral Analysis and Measurement<br />
Methods; G. Kle<strong>in</strong>, R. Calderwood, and A. Cl<strong>in</strong>ton-Cirocco, Rapid Decision Mak<strong>in</strong>g on <strong>the</strong> Fire<br />
Ground.<br />
26<br />
A copy of <strong>the</strong> list and search criteria is available from <strong>the</strong> author.<br />
27<br />
William Stephenson, The Study of Behavior: Q-Technique and its Methodology. See Chapter<br />
Eleven for additional <strong>in</strong>formation on this methodology.<br />
28<br />
I would like to credit Dr. Forrest Frank of <strong>the</strong> Institute for Defense Analyses for his suggestions<br />
regard<strong>in</strong>g <strong>the</strong> nam<strong>in</strong>g convention for <strong>the</strong> categories of variables <strong>in</strong> <strong>the</strong> accompany<strong>in</strong>g chart.<br />
38
A TAXONOMY OF INTELLIGENCE<br />
cases and directed to use various methods to analyze <strong>the</strong> situations and to generate<br />
f<strong>in</strong>al products. Dur<strong>in</strong>g <strong>the</strong> tra<strong>in</strong><strong>in</strong>g exercises, <strong>the</strong> verbal and physical<br />
behavior of <strong>in</strong>dividuals and groups were observed and compared with <strong>the</strong> taxonomic<br />
model. I participated <strong>in</strong> a number of <strong>the</strong> exercises myself to ga<strong>in</strong> a better<br />
perspective. This process corroborated most of <strong>the</strong> recommendations that<br />
had been made by <strong>the</strong> experts and novices and also yielded additional variables<br />
for two of <strong>the</strong> categories.<br />
The result<strong>in</strong>g taxonomy is purely descriptive. It Systemic Variables<br />
is not <strong>in</strong>tended to demonstrate <strong>the</strong> weight or Organization<br />
Internal<br />
importance of each variable or category. That is,<br />
Structure<br />
<strong>the</strong> list<strong>in</strong>g is not sufficient to predict <strong>the</strong> effect of<br />
any one variable on human performance. The<br />
<strong>in</strong>tention of <strong>the</strong> enumeration is to provide a framework<br />
for aggregat<strong>in</strong>g exist<strong>in</strong>g data and to create a<br />
foundation for future experimentation. Once <strong>the</strong><br />
variables have been identified and previous f<strong>in</strong>d<strong>in</strong>gs<br />
have been aggregated, it is reasonable to consider<br />
experimental methods that would isolate and<br />
control <strong>in</strong>dividual variables and, <strong>in</strong> time, <strong>in</strong>dicate<br />
sources of error and potential remediation<br />
Systemic Variables<br />
The column of Systemic Variables <strong>in</strong>corporates<br />
items that affect both an <strong>in</strong>telligence organization<br />
and <strong>the</strong> analytic environment. Organizational<br />
variables encompass <strong>the</strong> structure of <strong>the</strong> <strong>in</strong>telligence<br />
organization; leadership, management, and<br />
management practices; history and traditions; <strong>the</strong><br />
work<strong>in</strong>g culture, social practices with<strong>in</strong> <strong>the</strong> organization,<br />
and work taboos; and organizational<br />
demographics. They also <strong>in</strong>clude <strong>in</strong>ternal politics,<br />
<strong>the</strong> hierarchical report<strong>in</strong>g structure, and material<br />
and human resources. Industrial and organizational<br />
psychology, sociology, and management<br />
studies <strong>in</strong> bus<strong>in</strong>ess have brought attention to <strong>the</strong><br />
importance of organizational behavior and its<br />
effect on <strong>in</strong>dividual work habits and practices.<br />
The works of Allison, Berkowitz and Goodman,<br />
Elk<strong>in</strong>s, Ford, Godson, and Richelson, among o<strong>the</strong>rs,<br />
exam<strong>in</strong>e <strong>in</strong> general <strong>the</strong> organizational aspects<br />
Leadership<br />
<strong>Culture</strong><br />
History<br />
Traditions<br />
Social Practice<br />
Taboo<br />
Group Characteristics<br />
Hierarchy<br />
Resources and Incentives<br />
Manpower<br />
Budget<br />
Technology<br />
Assets<br />
R&D<br />
Facilities<br />
Work Groups-Teams<br />
External<br />
Consumer Needs<br />
Time and Imperatives<br />
Consumer Use<br />
Consumer Structure<br />
Consumer Hierarchy<br />
Conumer Report<strong>in</strong>g<br />
Politics<br />
Internal-Organization<br />
Policy<br />
Tradition<br />
Taboo<br />
Security/Access<br />
External-National<br />
Law<br />
Policy<br />
External-International<br />
Security<br />
Denial<br />
Deception<br />
Policy<br />
of <strong>in</strong>telligence. 29 39
CHAPTER THREE<br />
The Systemic Variables category also focuses on<br />
environmental variables. These <strong>in</strong>clude such external<br />
<strong>in</strong>fluences on <strong>the</strong> organization as consumer<br />
needs and requirements, time limitations, and methods<br />
for us<strong>in</strong>g <strong>the</strong> <strong>in</strong>formation; and <strong>the</strong> consumer’s<br />
Overt<br />
Covert<br />
organization, political constra<strong>in</strong>ts, and security<br />
issues. The works of Betts, Hulnick, Hunt, Kam,<br />
Reproducible and Laqueur address <strong>the</strong> environmental and con-<br />
Consistent sumer issues that affect <strong>in</strong>telligence analysis. 30<br />
Systematic Variables<br />
User Requirements<br />
Operations<br />
Information Acquisition<br />
Collection Methods<br />
Information Reliability<br />
Information Validity<br />
Historical<br />
S<strong>in</strong>gle Source<br />
Dual Source<br />
Triangulation<br />
Information Archive<br />
Storage<br />
Access<br />
Correlation<br />
Retrieval<br />
<strong>Analytic</strong>al Methodology<br />
Approach<br />
Intuitive<br />
Structured<br />
Semi-structured<br />
Information Process<strong>in</strong>g<br />
Historical Information<br />
Current Information<br />
Decision Strategies<br />
Estimative<br />
Predictive<br />
Report<strong>in</strong>g<br />
Verbal Methods<br />
Written Methods<br />
Case studies that touch on various systemic variables<br />
<strong>in</strong>clude: Allison, on <strong>the</strong> Cuban missile crisis;<br />
Betts, on surprise attacks; Kirkpatrick, on World<br />
War II tactical <strong>in</strong>telligence operations; Shiels, on<br />
government failures; Wirtz, on <strong>the</strong> Tet offensive <strong>in</strong><br />
Vietnam; and Wohlstetter, on Pearl Harbor. 31<br />
Systematic Variables<br />
The Systematic Variables are those that affect<br />
<strong>the</strong> process of analysis itself. They <strong>in</strong>clude <strong>the</strong><br />
user’s specific requirements, how <strong>the</strong> <strong>in</strong>formation<br />
was acquired, <strong>the</strong> <strong>in</strong>formation’s reliability and<br />
validity, how <strong>the</strong> <strong>in</strong>formation is stored, <strong>the</strong> prescribed<br />
methods for analyz<strong>in</strong>g and process<strong>in</strong>g <strong>the</strong><br />
<strong>in</strong>formation, specific strategies for mak<strong>in</strong>g decisions<br />
about <strong>the</strong> <strong>in</strong>formation, and <strong>the</strong> methods used<br />
to report <strong>the</strong> <strong>in</strong>formation to consumers.<br />
A number of authors have written about <strong>the</strong><br />
analytic tools and techniques used <strong>in</strong> <strong>in</strong>telligence,<br />
among <strong>the</strong>m Clauser and Weir, on <strong>in</strong>telligence research methods; Jones, on<br />
analytic techniques; and Heuer, on alternative compet<strong>in</strong>g hypo<strong>the</strong>ses.<br />
29<br />
Graham T. Allison, Essence of Decision; Bruce D. Berkowitz and Allan E. Goodman, Best Truth;<br />
Dan Elk<strong>in</strong>s, An <strong>Intelligence</strong> Resource Manager’s Guide; Harold Ford, Estimative <strong>Intelligence</strong>; Roy<br />
Godson, Compar<strong>in</strong>g Foreign <strong>Intelligence</strong>; Jeffrey Richelson, The U.S. <strong>Intelligence</strong> <strong>Community</strong>.<br />
30<br />
Richard K. Betts, “Policy-makers and <strong>Intelligence</strong> Analysts: Love, Hate or Indifference”; Arthur<br />
S. Hulnick, “The <strong>Intelligence</strong> Producer-Policy Consumer L<strong>in</strong>kage: A Theoretical Approach”; David<br />
Hunt, Complexity and Plann<strong>in</strong>g <strong>in</strong> <strong>the</strong> 21st Century; Kam, Surprise Attack; Walter A. Laqueur, The<br />
Uses and Limits of <strong>Intelligence</strong>.<br />
31<br />
Allison; Richard K. Betts, Surprise Attack; Lyman B. Kirkpatrick, Jr., Capta<strong>in</strong>s Without Eyes:<br />
<strong>Intelligence</strong> Failures <strong>in</strong> World War II; Frederick L. Shiels, Preventable Disasters: Why Governments<br />
Fail; James J. Wirtz, The Tet Offensive: <strong>Intelligence</strong> Failure <strong>in</strong> War; Roberta Wohlstetter, Pearl Harbor:<br />
Warn<strong>in</strong>g and Decision.<br />
40
A TAXONOMY OF INTELLIGENCE<br />
Comparatively little work has been done<br />
compar<strong>in</strong>g structured techniques to <strong>in</strong>tuition.<br />
Robert Folker’s work is one of <strong>the</strong> exceptions; it<br />
compares <strong>the</strong> effectiveness of a modified form of<br />
alternative compet<strong>in</strong>g hypo<strong>the</strong>ses with <strong>in</strong>tuition<br />
<strong>in</strong> a controlled experimental design. His study is<br />
unique <strong>in</strong> <strong>the</strong> field and demonstrates that<br />
experimental methods are possible. Gerald<strong>in</strong>e<br />
Krotow’s research, on <strong>the</strong> o<strong>the</strong>r hand, looks at<br />
differ<strong>in</strong>g forms of cognitive feedback dur<strong>in</strong>g <strong>the</strong><br />
analytic process and makes recommendations to<br />
improve <strong>in</strong>telligence decisionmak<strong>in</strong>g. 32<br />
Idiosyncratic Variables<br />
Variables <strong>in</strong> <strong>the</strong> third column are those that<br />
<strong>in</strong>fluence <strong>in</strong>dividuals and <strong>the</strong>ir analytic performance.<br />
These <strong>in</strong>clude <strong>the</strong> sum of life experiences<br />
and enculturation—familial, cultural, ethnic, religious,<br />
l<strong>in</strong>guistic, and political affiliations—that<br />
identify an <strong>in</strong>dividual as a member of a group. I<br />
have used <strong>the</strong> German word Weltanschauung<br />
(customarily rendered <strong>in</strong> English as “world view”)<br />
to denote this concept. These idiosyncratic variables<br />
also encompass such psychological factors<br />
as biases, personality profiles, cognitive styles and<br />
process<strong>in</strong>g, cognitive loads, 33 expertise, approach<br />
to problem-solv<strong>in</strong>g, decisionmak<strong>in</strong>g style, and<br />
reaction to stress. F<strong>in</strong>ally, <strong>the</strong>re are such doma<strong>in</strong><br />
variables as education, tra<strong>in</strong><strong>in</strong>g, and <strong>the</strong> read<strong>in</strong>ess<br />
to apply knowledge, skills, and abilities to <strong>the</strong> task<br />
at hand.<br />
Idiosyncratic Variables<br />
Weltanschauung (worldview)<br />
Affiliation<br />
Familial<br />
Cultural<br />
Ethnic<br />
Religious<br />
Social<br />
L<strong>in</strong>guistic<br />
Political<br />
Psychology<br />
Bias<br />
Personality Profile<br />
Security Trust<br />
Cognitive Process<strong>in</strong>g<br />
Learn<strong>in</strong>g Style<br />
Information Acquisition<br />
Information Process<strong>in</strong>g<br />
Expertise<br />
Problem-solv<strong>in</strong>g<br />
Decisionmak<strong>in</strong>g<br />
Cognitive Load<br />
Speed/Accuracy<br />
Stress Effects<br />
Education<br />
Doma<strong>in</strong><br />
Location<br />
Mentor<br />
Tra<strong>in</strong><strong>in</strong>g<br />
Organizational<br />
Doma<strong>in</strong><br />
Procedural<br />
Read<strong>in</strong>ess<br />
Resources<br />
Facilities<br />
The relevant psychological literature is extensive. Amos Tversky and Daniel<br />
Kahneman began to exam<strong>in</strong>e psychological biases <strong>in</strong> <strong>the</strong> early 1970s. 34 Their<br />
work has found its way <strong>in</strong>to <strong>the</strong> <strong>in</strong>telligence literature through Butterfield,<br />
32<br />
Gerald<strong>in</strong>e Krotow, The Impact of Cognitive Feedback on <strong>the</strong> Performance of <strong>Intelligence</strong> Analysts,<br />
176.<br />
33<br />
“Cognitive loads” are <strong>the</strong> amount/number of cognitive tasks weighed aga<strong>in</strong>st available cognitive<br />
process<strong>in</strong>g power.<br />
34<br />
Amos Tversky and Daniel Kahneman, “The Belief <strong>in</strong> <strong>the</strong> ‘Law of Small Numbers’” and “Judgment<br />
Under Uncerta<strong>in</strong>ty: Heuristics and Biases.”<br />
41
CHAPTER THREE<br />
Davis, Goldgeier, and Heuer, among o<strong>the</strong>rs. 35 Decisionmak<strong>in</strong>g and problemsolv<strong>in</strong>g<br />
have been studied s<strong>in</strong>ce <strong>the</strong> early 1920s, and <strong>the</strong>se topics are reflected <strong>in</strong><br />
Heuer’s work as well. 36 Personality-profil<strong>in</strong>g, too,<br />
Communicative Variables is well understood and has had an impact on recent<br />
Formal <strong>in</strong>telligence practices and <strong>the</strong>ory. 37<br />
Inter-organization<br />
Hierarchical<br />
Inter-division<br />
Inter-group<br />
Intra-organization<br />
Hierarchical<br />
Intra-division<br />
Intra-group<br />
Individual<br />
Hierarchical<br />
Inter-division<br />
Intra-group<br />
Informal<br />
Inter-organization<br />
Hierarchical<br />
Inter-division<br />
Inter-group<br />
Intra-organization<br />
Hierarchical<br />
Intra-division<br />
Intra-group<br />
Individual<br />
Hierarchical<br />
Inter-group<br />
Intra-group<br />
Technology<br />
Networked Analysis<br />
Collaboration<br />
O<strong>the</strong>r well-researched areas, however, have yet<br />
to be studied <strong>in</strong> <strong>the</strong> context of <strong>in</strong>telligence. Acculturation<br />
and enculturation, educational factors,<br />
and tra<strong>in</strong><strong>in</strong>g strategies, for example, may yet yield<br />
<strong>in</strong>terest<strong>in</strong>g results and <strong>in</strong>sights <strong>in</strong>to <strong>the</strong> field of<br />
<strong>in</strong>telligence. 38<br />
Communicative Variables<br />
The fourth category conta<strong>in</strong>s variables that<br />
affect <strong>in</strong>teraction with<strong>in</strong> and among groups.<br />
Because communication is <strong>the</strong> vital l<strong>in</strong>k with<strong>in</strong><br />
<strong>the</strong> system—among processes and among <strong>in</strong>dividuals—this<br />
group of variables logically could be<br />
<strong>in</strong>cluded <strong>in</strong> each of <strong>the</strong> o<strong>the</strong>r three categories. Its<br />
broad relevance, however, makes it seem reasonable<br />
to isolate it as a dist<strong>in</strong>ct area of variability.<br />
The Communicative Variables <strong>in</strong>clude formal and<br />
<strong>in</strong>formal communications with<strong>in</strong> an organization<br />
(from products to e-mails), among organizations,<br />
and between <strong>in</strong>dividuals and <strong>the</strong> social networks<br />
<strong>the</strong>y create. In his essay on estimative probability,<br />
Kent highlights this area by describ<strong>in</strong>g <strong>the</strong> difficulty<br />
that producers of <strong>in</strong>telligence have <strong>in</strong> communicat<strong>in</strong>g<br />
<strong>the</strong> likelihood of an event to <strong>the</strong>ir consumers. 39 In addition to<br />
address<strong>in</strong>g organizational issues, case studies by Wohlstetter and o<strong>the</strong>rs touch<br />
on communication and social networks and <strong>the</strong> impact that communication<br />
35<br />
Alexander Butterfield, The Accuracy of <strong>Intelligence</strong> Assessment; Jack Davis, “Combat<strong>in</strong>g<br />
M<strong>in</strong>dset”; James M. Goldgeier, “Psychology and Security”; Heuer.<br />
36<br />
Frank H. Knight, Risk, Uncerta<strong>in</strong>ty and Profit.<br />
37<br />
Carol<strong>in</strong>e Ziemke, Philippe Loustaunau, and Amy Alrich, Strategic Personality and <strong>the</strong> Effectiveness<br />
of Nuclear Deterrence.<br />
38<br />
Acculturation is <strong>the</strong> cultural change that occurs <strong>in</strong> response to extended firsthand contact<br />
between two or more previously autonomous groups. It can result <strong>in</strong> cultural changes <strong>in</strong> groups as<br />
well as <strong>in</strong>dividuals.<br />
39<br />
Sherman Kent, “Words of Estimative Probability.”<br />
42
A TAXONOMY OF INTELLIGENCE<br />
has on <strong>the</strong> analytic process. 40 This is an area that could benefit from additional<br />
study.<br />
Conclusion<br />
There is rarely any doubt that <strong>the</strong> unconscious reasons for practic<strong>in</strong>g<br />
a custom or shar<strong>in</strong>g a belief are remote from <strong>the</strong> reasons given<br />
to justify <strong>the</strong>m.<br />
—Claude Levi-Strauss 41<br />
As it is now practiced, <strong>in</strong>telligence analysis is art, tradecraft, and science.<br />
There are specific tools and techniques to help perform <strong>the</strong> tasks, but, <strong>in</strong> <strong>the</strong><br />
end, it is left to <strong>in</strong>dividuals to use <strong>the</strong>ir best judgment <strong>in</strong> mak<strong>in</strong>g decisions.<br />
This is not to say that science is not a part of <strong>in</strong>telligence analysis. Science is<br />
born of organized knowledge, and organiz<strong>in</strong>g knowledge requires effort and<br />
time. The work on this taxonomy is <strong>in</strong>tended to help that process by spark<strong>in</strong>g<br />
discussion, identify<strong>in</strong>g areas where research exists and ought to be <strong>in</strong>corporated<br />
<strong>in</strong>to <strong>the</strong> organizational knowledge of <strong>in</strong>telligence, and identify<strong>in</strong>g areas<br />
where not enough research has been performed.<br />
There are a number of parallels <strong>in</strong> <strong>the</strong> field of medic<strong>in</strong>e, which, like <strong>in</strong>telligence,<br />
is art, tradecraft, and science. To solve problems, practitioners are<br />
trusted to use <strong>the</strong>ir best judgment by draw<strong>in</strong>g on <strong>the</strong>ir expertise. What is<br />
important to remember is that <strong>the</strong>re are numerous basic sciences driv<strong>in</strong>g medical<br />
practice. Biology, chemistry, physics, and all of <strong>the</strong> subspecialties blend<br />
toge<strong>the</strong>r to create <strong>the</strong> medical sciences, <strong>the</strong> foundation on which modern medic<strong>in</strong>e<br />
rests. The practice of medic<strong>in</strong>e has been revolutionized by <strong>the</strong> sciences<br />
that underp<strong>in</strong> its work<strong>in</strong>gs.<br />
<strong>Intelligence</strong> analysis has not experienced that revolution. Unlike medic<strong>in</strong>e,<br />
<strong>the</strong> basic sciences that underp<strong>in</strong> <strong>in</strong>telligence are <strong>the</strong> human sciences, which are<br />
considerably more multivariate and more difficult to control. Because of <strong>the</strong>se<br />
factors, it is a more complex task to measure “progress” <strong>in</strong> <strong>the</strong> human sciences.<br />
Even so, <strong>the</strong>re are numerous doma<strong>in</strong>s from which <strong>in</strong>telligence may borrow.<br />
Organizational behavior is better understood today than ever before.<br />
Problem-solv<strong>in</strong>g and decisionmak<strong>in</strong>g have been researched s<strong>in</strong>ce <strong>the</strong> 1920s.<br />
40<br />
Wohlstetter.<br />
41<br />
Claude Levi-Strauss wrote Structural Anthropology <strong>in</strong> 1958, sett<strong>in</strong>g <strong>the</strong> stage for structuralism<br />
to emerge as an analytic <strong>in</strong>terpretive method. Broadly, structuralism seeks to explore <strong>the</strong> <strong>in</strong>terrelationships<br />
(<strong>the</strong> “structures”) through which mean<strong>in</strong>g is produced with<strong>in</strong> a culture. This mean<strong>in</strong>g,<br />
accord<strong>in</strong>g to structural <strong>the</strong>ory, is produced and reproduced through various practices, phenomena,<br />
and activities that serve as systems of “signification.” A structuralist studies activities as<br />
diverse as food preparation and serv<strong>in</strong>g rituals, religious rites, games, literary and non-literary<br />
texts, and forms of enterta<strong>in</strong>ment to discover <strong>the</strong> ways <strong>in</strong> which cultural significance develops.<br />
43
CHAPTER THREE<br />
Structural anthropology addresses many of <strong>the</strong> enculturation and identity issues<br />
that affect <strong>in</strong>dividual behavior. Cognitive scientists are build<strong>in</strong>g models that can<br />
be tested <strong>in</strong> experimental conditions and used for develop<strong>in</strong>g new tools and techniques.<br />
Sociology and social <strong>the</strong>ory have much to offer <strong>in</strong> study<strong>in</strong>g social networks<br />
and communication.<br />
The organization of knowledge <strong>in</strong> <strong>in</strong>telligence is not a small task, but I<br />
believe that <strong>the</strong> effort should be undertaken for <strong>the</strong> betterment of <strong>the</strong> profession.<br />
The taxonomy proposed here could serve as a spr<strong>in</strong>gboard for a number<br />
of <strong>in</strong>novative projects, for example: development of a research matrix that<br />
identifies what is known and how that <strong>in</strong>formation may be of use <strong>in</strong> <strong>in</strong>telligence<br />
analysis, sett<strong>in</strong>g a research agenda <strong>in</strong> areas of <strong>in</strong>telligence that have<br />
been <strong>in</strong>sufficiently studied, application of research from o<strong>the</strong>r doma<strong>in</strong>s to<br />
develop additional tra<strong>in</strong><strong>in</strong>g and education programs for analysts, creation of a<br />
database of lessons learned and best practices to build a foundation for an<br />
electronic performance support system, <strong>in</strong>tegration of those f<strong>in</strong>d<strong>in</strong>gs <strong>in</strong>to new<br />
analytic tools and techniques, and development of a networked architecture<br />
for collaborative problem-solv<strong>in</strong>g and forecast<strong>in</strong>g. It is my hope that this taxonomy<br />
will help <strong>in</strong>telligence practitioners take steps <strong>in</strong> some of <strong>the</strong>se new<br />
directions.<br />
44
CHAPTER FOUR<br />
Test<strong>in</strong>g <strong>the</strong> <strong>Intelligence</strong> Cycle Through Systems<br />
Model<strong>in</strong>g and Simulation<br />
Judith Meister Johnston 1<br />
Rob Johnston<br />
Throughout <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong>, <strong>the</strong> process of analysis is represented<br />
conventionally by a model known as <strong>the</strong> <strong>Intelligence</strong> Cycle (See next page).<br />
Unfortunately, <strong>the</strong> model omits elements and fails to capture <strong>the</strong> process accurately,<br />
which makes understand<strong>in</strong>g <strong>the</strong> challenges and responsibilities of <strong>in</strong>telligence<br />
analysis much more difficult. It also complicates <strong>the</strong> tasks of recogniz<strong>in</strong>g<br />
where errors can occur and determ<strong>in</strong><strong>in</strong>g methods for change based on accurate<br />
predictions of behavior. Our analysis of <strong>the</strong> <strong>Intelligence</strong> Cycle, employ<strong>in</strong>g a systems<br />
approach and a simulation created to represent it, demonstrated <strong>the</strong>se shortcom<strong>in</strong>gs.<br />
2 Because of its wide acceptance and use <strong>in</strong> tra<strong>in</strong><strong>in</strong>g and <strong>in</strong> discussions<br />
of <strong>the</strong> analytic process, <strong>the</strong> traditional representation of <strong>the</strong> <strong>Intelligence</strong> Cycle<br />
will be closely considered <strong>in</strong> this chapter, especially with regard to its impact on<br />
analytic products, its effectiveness, and its vulnerability to error and failure.<br />
The Traditional <strong>Intelligence</strong> Cycle<br />
The <strong>Intelligence</strong> Cycle is customarily illustrated as a repeat<strong>in</strong>g process consist<strong>in</strong>g<br />
of five steps. 3 Plann<strong>in</strong>g and direction encompasses <strong>the</strong> management of<br />
1<br />
Dr. Judith Meister Johnston is an educational psychologist with expertise <strong>in</strong> human performance<br />
technology and <strong>in</strong>structional systems design. A Booz Allen Hamilton Associate, she supports<br />
human factors work for <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong>.<br />
2<br />
Simulation <strong>in</strong>volves <strong>the</strong> development of a computer-based model that represents <strong>the</strong> <strong>in</strong>ternal<br />
processes of an event or situation and estimates <strong>the</strong> results of proposed actions.<br />
45
CHAPTER FOUR<br />
<strong>the</strong> entire effort and <strong>in</strong>volves, <strong>in</strong> particular, determ<strong>in</strong><strong>in</strong>g collection requirements<br />
based on customer requests. Collection refers to <strong>the</strong> ga<strong>the</strong>r<strong>in</strong>g of raw data to<br />
meet <strong>the</strong> collection requirements. These data can be derived from any number<br />
and type of open and secret sources. Process<strong>in</strong>g refers to <strong>the</strong> conversion of raw<br />
data <strong>in</strong>to a format analysts can use. Analysis and production describes <strong>the</strong> process<br />
of evaluat<strong>in</strong>g data for reliability, validity, and relevance; <strong>in</strong>tegrat<strong>in</strong>g and<br />
analyz<strong>in</strong>g it; and convert<strong>in</strong>g <strong>the</strong> product of this effort <strong>in</strong>to a mean<strong>in</strong>gful whole,<br />
which <strong>in</strong>cludes assessments of events and implications of <strong>the</strong> <strong>in</strong>formation collected.<br />
F<strong>in</strong>ally, <strong>the</strong> product is dissem<strong>in</strong>ated to its <strong>in</strong>tended audience. 4<br />
In some ways, this<br />
The Traditional <strong>Intelligence</strong> Cycle process resembles many<br />
o<strong>the</strong>r production cycles. It<br />
is prescriptive, structured,<br />
Plann<strong>in</strong>g<br />
and direction<br />
made up of discrete steps,<br />
and expected to yield a<br />
specific product. The<br />
traditional depiction of <strong>the</strong><br />
Collection<br />
Dissem<strong>in</strong>ation<br />
process <strong>in</strong> <strong>the</strong> <strong>Intelligence</strong><br />
Cycle, however, is not an<br />
accurate representation of<br />
<strong>the</strong> way <strong>in</strong>telligence is<br />
produced. The notion of a<br />
Process<strong>in</strong>g Analysis cycle assumes that <strong>the</strong> steps<br />
and production<br />
will proceed <strong>in</strong> <strong>the</strong><br />
prescribed order and that<br />
<strong>the</strong> process will repeat itself cont<strong>in</strong>uously with reliable results. This type of<br />
representation gives <strong>the</strong> impression that all <strong>in</strong>puts are constant and flow<br />
automatically, but it does not address elements that may <strong>in</strong>fluence <strong>the</strong><br />
movement of <strong>the</strong> cycle, positively or negatively.<br />
The most significant assumption about <strong>the</strong> <strong>Intelligence</strong> Cycle model, that it<br />
provides a means for help<strong>in</strong>g managers and analysts deliver a reliable product,<br />
should be exam<strong>in</strong>ed at <strong>the</strong> outset. This can be accomplished through two types<br />
of analyses. The first is a systematic exam<strong>in</strong>ation of <strong>the</strong> elements of <strong>the</strong> process,<br />
<strong>the</strong> <strong>in</strong>puts it relies on, and <strong>the</strong> outcomes that can be expected. The second<br />
uses a systemic approach to identify<strong>in</strong>g <strong>the</strong> relationships of <strong>the</strong> elements <strong>in</strong> <strong>the</strong><br />
process and <strong>the</strong>ir <strong>in</strong>fluence on each o<strong>the</strong>r.<br />
3<br />
Central <strong>Intelligence</strong> Agency, A Consumer’s Guide to <strong>Intelligence</strong>.<br />
4<br />
Central <strong>Intelligence</strong> Agency, Factbook on <strong>Intelligence</strong>.<br />
46
TESTING THE INTELLIGENCE CYCLE<br />
Systematic Analysis<br />
Many discipl<strong>in</strong>es (for example, bus<strong>in</strong>ess process, organizational management,<br />
human performance technology, program evaluation, systems eng<strong>in</strong>eer<strong>in</strong>g,<br />
and <strong>in</strong>structional systems design) employ specific methods to analyze <strong>the</strong><br />
effectiveness of products, programs, or policy implementation. Although <strong>the</strong>y<br />
are often given different, doma<strong>in</strong>-specific names and may <strong>in</strong>volve vary<strong>in</strong>g levels<br />
of detail, <strong>the</strong>se analytic methods <strong>in</strong>volve <strong>the</strong> identification of <strong>in</strong>puts, processes,<br />
and outputs. Once <strong>the</strong>se elements are identified, <strong>the</strong> evaluation process<br />
maps <strong>the</strong> relationships of <strong>the</strong> <strong>in</strong>puts, <strong>the</strong>ir implementation <strong>in</strong> processes, and<br />
<strong>the</strong>ir impact on <strong>in</strong>tended—as opposed to actual—outputs. 5 The reason<strong>in</strong>g<br />
underly<strong>in</strong>g this approach is that an effective product, result, or action is one<br />
that matches its objectives and that <strong>the</strong>se objectives are reached by processes<br />
that logically lead from <strong>the</strong> objectives to results. Along <strong>the</strong> way, exist<strong>in</strong>g practices<br />
and barriers to reach<strong>in</strong>g goals effectively can be identified. F<strong>in</strong>ally, <strong>in</strong>terventions,<br />
which can range <strong>in</strong> complexity from simple job aids to a complete<br />
restructur<strong>in</strong>g of <strong>the</strong> process, can be proposed and implemented and <strong>the</strong>ir<br />
impacts assessed. 6<br />
This method of analysis has been employed successfully to evaluate<br />
processes that have characteristics similar to <strong>the</strong> <strong>Intelligence</strong> Cycle, and<br />
we use it here to exam<strong>in</strong>e <strong>the</strong> effectiveness of <strong>the</strong> <strong>Intelligence</strong> Cycle and<br />
its utility <strong>in</strong> represent<strong>in</strong>g <strong>the</strong> creation of sound analytic products while<br />
avoid<strong>in</strong>g failure or error.<br />
F<strong>in</strong>d<strong>in</strong>gs Based on Systematic Analysis<br />
The <strong>Intelligence</strong> Cycle is represented visually to provide an easy-to-grasp and<br />
easy-to-remember representation of a complex process. Although this type of representation<br />
may make <strong>the</strong> flow of <strong>in</strong>formation and <strong>the</strong> <strong>in</strong>terrelationships of steps<br />
easy to identify, it does not <strong>in</strong>dicate who or what may affect <strong>the</strong> completion of a<br />
step or <strong>the</strong> resources needed to beg<strong>in</strong> <strong>the</strong> next step. In its concise form, <strong>the</strong>n, <strong>the</strong><br />
visual representation of <strong>the</strong> <strong>Intelligence</strong> Cycle is reduced to a map of <strong>in</strong>formation<br />
handl<strong>in</strong>g. Without explicit descriptions of <strong>the</strong> steps <strong>in</strong> <strong>the</strong> process or <strong>the</strong> benefit of<br />
prior knowledge, it can raise questions of accuracy and completeness and can<br />
occasion misconceptions, particularly concern<strong>in</strong>g <strong>the</strong> roles and responsibilities of<br />
<strong>in</strong>telligence analysts.<br />
5<br />
Marc J. Rosenberg, “Performance technology: Work<strong>in</strong>g <strong>the</strong> system.”<br />
6<br />
Roger Kaufman, “A Holistic Plann<strong>in</strong>g Model: A Systems Approach for Improv<strong>in</strong>g Organizational<br />
Effectiveness and Impact.”<br />
47
CHAPTER FOUR<br />
Inputs, Processes, and Outputs of <strong>the</strong> <strong>Intelligence</strong> Cycle<br />
Inputs Processes Outputs<br />
Policymaker and o<strong>the</strong>r Direction Data collection<br />
stakeholder questions,<br />
requirements<br />
requirements<br />
Data collection Plann<strong>in</strong>g Task assignment,<br />
requirements, assessment<br />
potential data sources,<br />
of available resources and<br />
focus of analysis<br />
capabilities<br />
Open-source data: foreign Collection Potentially relevant data<br />
broadcasts, newspapers,<br />
periodicals, books;<br />
Classified data: case<br />
officer, diplomatic, and<br />
attaché reports,<br />
electronics, satellite<br />
photos<br />
Potentially relevant data<br />
Usable data<br />
Process<strong>in</strong>g: Reduction<br />
of data <strong>in</strong> a variety of<br />
formats to consistent<br />
pieces of usable data<br />
Analysis: Integration,<br />
evaluation, assessment<br />
of reliability, validity,<br />
and relevance of data<br />
Production: Peer<br />
review, supervisory<br />
review<br />
Usable Data<br />
F<strong>in</strong>d<strong>in</strong>gs<br />
<strong>Analytic</strong> review<br />
Written briefs, studies,<br />
long range assessments,<br />
short range assessments,<br />
oral briefs, national<br />
<strong>in</strong>telligence estimates<br />
Written briefs, studies, Dissem<strong>in</strong>ation Appropriate product to<br />
long-range assessments,<br />
address customer’s need<br />
short-range assessments,<br />
oral briefs, national<br />
<strong>in</strong>telligence estimates<br />
The table above depicts a more detailed <strong>in</strong>put, process, and output analysis<br />
and makes some relationships clearer—for example, <strong>the</strong> steps that <strong>in</strong>clude two<br />
actions (plann<strong>in</strong>g and direction, analysis and production) have been separated<br />
<strong>in</strong>to dist<strong>in</strong>ct processes—but it sill leaves a number of questions unanswered. It<br />
is difficult to see from this analysis specifically who is responsible for provid<strong>in</strong>g<br />
<strong>in</strong>puts, carry<strong>in</strong>g out <strong>the</strong> processes, and produc<strong>in</strong>g outputs; and what<br />
requirements are expected of <strong>the</strong> <strong>in</strong>puts and outputs.<br />
An important issue that this analysis only partly clarifies is <strong>the</strong> role of analysts.<br />
Nor does it demonstrate how great a burden <strong>the</strong> process places on <strong>the</strong>m,<br />
48
TESTING THE INTELLIGENCE CYCLE<br />
an especially important po<strong>in</strong>t. Assum<strong>in</strong>g that <strong>the</strong> actions identified <strong>in</strong> <strong>the</strong><br />
“Processes” column are ultimately <strong>the</strong> responsibility of <strong>the</strong> <strong>in</strong>telligence analyst,<br />
<strong>the</strong> steps of <strong>the</strong> process move from a heavy reliance on <strong>in</strong>formation com<strong>in</strong>g<br />
<strong>in</strong> from sources outside <strong>the</strong> analyst’s control to a heavy reliance on <strong>the</strong><br />
analyst to produce and manage <strong>the</strong> f<strong>in</strong>al submission of <strong>the</strong> product.<br />
Ano<strong>the</strong>r important defect <strong>in</strong> this analysis is that steps <strong>in</strong> <strong>the</strong> cycle do not<br />
accurately represent <strong>the</strong> differences <strong>in</strong> <strong>the</strong> cognitive complexity <strong>in</strong>volved <strong>in</strong><br />
prepar<strong>in</strong>g a long-range assessment or a national <strong>in</strong>telligence estimate and that<br />
required for a two-paragraph brief on a current situation. The same can be said<br />
about <strong>the</strong> process required to develop each of <strong>the</strong> products.<br />
The <strong>Intelligence</strong><br />
Treverton’s “Real” <strong>Intelligence</strong> Cycle<br />
Cycle depicts a<br />
sequential process and<br />
does not provide for<br />
iterations between<br />
steps. This is not an<br />
accurate reflection of<br />
what happens, particularly<br />
<strong>in</strong> <strong>the</strong> collection<br />
and production steps,<br />
where <strong>the</strong> challenges<br />
of def<strong>in</strong><strong>in</strong>g policymaker<br />
needs and shap<strong>in</strong>g<br />
collection<br />
necessitate repeated<br />
ref<strong>in</strong>ement of requirements<br />
by policymakers<br />
or of <strong>in</strong>ferences by <strong>the</strong><br />
<strong>Intelligence</strong> <strong>Community</strong>.<br />
A more accurate picture of <strong>the</strong> steps <strong>in</strong> <strong>the</strong> process and <strong>the</strong>ir iterative tendencies<br />
may be seen <strong>in</strong> Greg Treverton’s model, which he terms <strong>the</strong> “Real”<br />
<strong>Intelligence</strong> Cycle (above). 7<br />
Mark Lowenthal proposes ano<strong>the</strong>r model. 8 Although presented <strong>in</strong> a more<br />
l<strong>in</strong>ear fashion than Treverton’s, it focuses on <strong>the</strong> areas where revisions and<br />
reconsiderations take place, represent<strong>in</strong>g iteration <strong>in</strong> a slightly different light.<br />
Both models provide a more realistic view of <strong>the</strong> entire process. In addition,<br />
assum<strong>in</strong>g that <strong>the</strong> analyst’s role is represented by <strong>the</strong> “Process<strong>in</strong>g, Analysis”<br />
box, <strong>the</strong> Treverton model allows us to focus visually and conceptually on <strong>the</strong><br />
demands that <strong>the</strong> process can place on <strong>the</strong> analyst. However, nei<strong>the</strong>r model<br />
7<br />
Gregory F. Treverton, Reshap<strong>in</strong>g National <strong>Intelligence</strong> <strong>in</strong> an Age of Information.<br />
8<br />
Mark W. Lowenthal, <strong>Intelligence</strong>: From Secrets to Policy.<br />
49
CHAPTER FOUR<br />
provides an effective way of show<strong>in</strong>g who is responsible for what, and nei<strong>the</strong>r<br />
reflects <strong>the</strong> impact of <strong>the</strong> work on <strong>the</strong> <strong>in</strong>dividuals responsible for produc<strong>in</strong>g<br />
<strong>the</strong> reports—particularly <strong>the</strong> analyst—nor <strong>the</strong> reliance of <strong>the</strong> analyst on a variety<br />
of factors beyond his or her control.<br />
In sum, this brief evaluation of <strong>the</strong> <strong>Intelligence</strong> Cycle with respect to its<br />
<strong>in</strong>puts, processes, and outputs shows us that <strong>the</strong> traditional model:<br />
• assumes <strong>the</strong> process works <strong>the</strong> same way for all objectives, regardless of<br />
complexity and cognitive demands;<br />
• does not represent <strong>the</strong> iterative nature of <strong>the</strong> process required for meet<strong>in</strong>g<br />
objectives;<br />
• does not identify responsibilities for complet<strong>in</strong>g steps and allows for misconceptions<br />
<strong>in</strong> this regard;<br />
• does not accurately represent <strong>the</strong> impact of resource availability on analysts.<br />
To better understand <strong>the</strong>se limitations and <strong>the</strong> relationships among elements<br />
<strong>in</strong> <strong>the</strong> process, it is necessary to step back and take a longer view of <strong>the</strong> process,<br />
us<strong>in</strong>g a different method of analysis.<br />
Systemic Analysis<br />
If we th<strong>in</strong>k of <strong>the</strong> phenomenon that is be<strong>in</strong>g described by <strong>the</strong> <strong>Intelligence</strong><br />
Cycle as a system and perform a systems analysis, we may be able to derive a<br />
greater understand<strong>in</strong>g of process relationships, a better representation of <strong>the</strong><br />
variables affect<strong>in</strong>g <strong>the</strong> process, and a greater level of detail regard<strong>in</strong>g <strong>the</strong> process<br />
itself.<br />
The premise that underlies systems analysis as a basis for understand<strong>in</strong>g<br />
phenomena is that <strong>the</strong> whole is greater than <strong>the</strong> sum of its parts. A systems<br />
analysis allows for <strong>the</strong> <strong>in</strong>clusion of a variety of <strong>in</strong>fluences and for <strong>the</strong> identification<br />
of outliers that are obfuscated <strong>in</strong> o<strong>the</strong>r types of analyses but that often<br />
play major roles. A systems analysis is accomplished through <strong>the</strong> exam<strong>in</strong>ation<br />
of phenomena as cause-and-effect patterns of behavior. This approach is<br />
called a “closed feedback loop” <strong>in</strong> systems analysis. It requires a close exam<strong>in</strong>ation<br />
of relationships and <strong>the</strong>ir <strong>in</strong>fluences, provides a longer view of <strong>the</strong>se<br />
relationships, and often reveals new <strong>in</strong>sights based on trends ra<strong>the</strong>r than on<br />
discrete events. 9<br />
The systems model diagrammed below is a visual representation of <strong>the</strong> process.<br />
The elements of <strong>the</strong> <strong>Intelligence</strong> Cycle are identified <strong>in</strong> terms of <strong>the</strong>ir<br />
9<br />
Fritjof Capra, “Criteria of Systems Th<strong>in</strong>k<strong>in</strong>g”; David L. Kaufman, Jr., Introduction to Systems<br />
Th<strong>in</strong>k<strong>in</strong>g.<br />
50
TESTING THE INTELLIGENCE CYCLE<br />
relationships with each o<strong>the</strong>r, <strong>the</strong> flow of <strong>the</strong> process, and phenomena that<br />
<strong>in</strong>fluence <strong>the</strong> elements and <strong>the</strong> flow. The model uses four icons to represent<br />
actions and relationships with<strong>in</strong> <strong>the</strong> system: stocks, flows, converters, and<br />
connectors. The icons and <strong>the</strong>ir placement with<strong>in</strong> <strong>the</strong> systems model show <strong>the</strong><br />
relationships of <strong>the</strong> elements of <strong>the</strong> analyzed phenomenon.<br />
The Components of <strong>the</strong> Systems Model<br />
Icon<br />
Stock<br />
Flow<br />
Converter<br />
Flow<br />
Stock<br />
Purpose<br />
Stocks represent accumulations. These are quantities<br />
that can <strong>in</strong>crease or decrease, such as <strong>the</strong> amount of<br />
work that needs to be completed, <strong>the</strong> time available <strong>in</strong><br />
which to do it, experience one might br<strong>in</strong>g to a task.<br />
Flows represent activities. They control <strong>the</strong> fill<strong>in</strong>g or<br />
dra<strong>in</strong><strong>in</strong>g of stocks, caus<strong>in</strong>g conditions to change.<br />
Converters change <strong>in</strong>puts <strong>in</strong>to outputs. They usually<br />
represent <strong>the</strong> variables that <strong>in</strong>itiate change. In <strong>the</strong><br />
example, a converter might represent a sudden and<br />
drastic world event.<br />
Connectors l<strong>in</strong>k elements to o<strong>the</strong>r elements,<br />
represent<strong>in</strong>g assumptions about what depends on<br />
what.<br />
Converter<br />
The systems model of <strong>the</strong> <strong>Intelligence</strong> Cycle provides <strong>in</strong>sights <strong>in</strong>to <strong>the</strong> process<br />
of analysis as well as o<strong>the</strong>r factors that can <strong>in</strong>fluence <strong>the</strong> successful and timely<br />
completion of an <strong>in</strong>telligence task. It also provides a way to understand <strong>the</strong><br />
impact of change <strong>in</strong> any area of <strong>the</strong> <strong>Intelligence</strong> Cycle on o<strong>the</strong>r elements, ei<strong>the</strong>r<br />
through reflection or by apply<strong>in</strong>g ma<strong>the</strong>matical values to <strong>the</strong> <strong>in</strong>fluences and relationships<br />
and runn<strong>in</strong>g simulations of <strong>the</strong> model.<br />
Demand. As <strong>in</strong> <strong>the</strong> traditional <strong>Intelligence</strong> Cycle model, <strong>the</strong> systems model<br />
beg<strong>in</strong>s with requirements for <strong>in</strong>formation that generally come from policymakers.<br />
These requirements are represented by a stock (found <strong>in</strong> <strong>the</strong> upper<br />
left-hand quarter of <strong>the</strong> diagram) because <strong>the</strong>y can <strong>in</strong>crease or decrease based<br />
on <strong>the</strong> level of need for <strong>in</strong>formation (a flow). The change <strong>in</strong> level of need is<br />
<strong>in</strong>fluenced by national and world events, as well as by new questions or<br />
requests for clarification of items <strong>in</strong> previously delivered products. Each<br />
request does not contribute equally to <strong>the</strong> amount of work, which is <strong>in</strong>fluenced<br />
by <strong>the</strong> types of documents or products requested, <strong>the</strong> complexity of <strong>the</strong> prod-<br />
51
CHAPTER FOUR<br />
Systems Model of <strong>the</strong> <strong>Intelligence</strong> Cycle<br />
52
TESTING THE INTELLIGENCE CYCLE<br />
ucts, and <strong>the</strong> turnaround time imposed. All of <strong>the</strong>se factors determ<strong>in</strong>e <strong>the</strong> level<br />
of demand placed on <strong>the</strong> analyst.<br />
Production. This section focuses on <strong>the</strong> process of produc<strong>in</strong>g <strong>in</strong>telligence<br />
products. The elements described are tied, directly or <strong>in</strong>directly, to <strong>the</strong> flow<br />
that represents changes <strong>in</strong> <strong>the</strong> analyst’s ability to produce. In turn, <strong>the</strong>se<br />
changes cause products to be completed and requests of policymakers to be<br />
fulfilled. It is important to note that this portion of <strong>the</strong> model deals with factors<br />
that <strong>in</strong>fluence <strong>the</strong> act of analysis and does not attempt to address methods of<br />
analysis.<br />
Factors that <strong>in</strong>fluence <strong>the</strong> ability of analysts to produce are numerous and<br />
complex, as shown. First and foremost are <strong>the</strong> capabilities an analyst br<strong>in</strong>gs to<br />
<strong>the</strong> task. This is represented by a stock—usually an <strong>in</strong>creas<strong>in</strong>g one—that<br />
derives from an analyst’s education, tra<strong>in</strong><strong>in</strong>g, and experience.<br />
Ano<strong>the</strong>r <strong>in</strong>fluence is <strong>the</strong> number and frequency of evaluations and revisions<br />
imposed on a work <strong>in</strong> progress. That a draft of <strong>the</strong> product must be reviewed<br />
and edited by a number of o<strong>the</strong>rs places variable constra<strong>in</strong>ts on <strong>the</strong> time available<br />
for creat<strong>in</strong>g <strong>the</strong> orig<strong>in</strong>al draft. This factor <strong>in</strong>creases <strong>in</strong> significance when<br />
<strong>the</strong> product requested has a short deadl<strong>in</strong>e.<br />
Political and cultural values of <strong>the</strong> organization also have an <strong>in</strong>fluence, usually<br />
constra<strong>in</strong><strong>in</strong>g. Strictly follow<strong>in</strong>g traditional heuristics and methods and<br />
meet<strong>in</strong>g organizational or management expectations may <strong>in</strong>fluence both an<br />
analyst’s ability to produce and <strong>the</strong> quality of <strong>the</strong> output. The weight of <strong>the</strong>se<br />
<strong>in</strong>fluences will vary depend<strong>in</strong>g on <strong>the</strong> experience of <strong>the</strong> analyst.<br />
Ano<strong>the</strong>r factor that <strong>in</strong>fluences <strong>the</strong> analyst’s ability to produce is <strong>the</strong> amount<br />
of relevant, usable data (a stock) available. The term “relevant, usable data”<br />
describes all collected <strong>in</strong>telligence that is relevant to meet<strong>in</strong>g <strong>the</strong> request and<br />
that exists <strong>in</strong> a format that can be used to develop <strong>the</strong> product. To become<br />
usable, <strong>the</strong> data must go through steps that are <strong>in</strong>fluenced by a variety of o<strong>the</strong>r<br />
people, organizations, systems, and technologies. This process is represented<br />
by <strong>the</strong> stock and flow cha<strong>in</strong> that appears across <strong>the</strong> middle of diagram.<br />
Data are collected from a variety of sources, represented by <strong>the</strong> INTs converter.<br />
10 These data add to <strong>the</strong> stock of collected data. The ways <strong>in</strong> which<br />
accumulated collected data are converted to <strong>the</strong> stock of available data are<br />
<strong>in</strong>fluenced by <strong>in</strong>ternal research demands and specific collection requirements<br />
imposed by analysts, policymakers, and o<strong>the</strong>rs. Once <strong>the</strong> data are processed<br />
and put <strong>in</strong>to an agreed format for use by <strong>in</strong>telligence producers and consum-<br />
10<br />
INT is an abbreviation for <strong>in</strong>telligence, usually conta<strong>in</strong>ed <strong>in</strong> acronyms for <strong>the</strong> various types of<br />
<strong>in</strong>telligence collected by <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong>, for example, HUMINT (human <strong>in</strong>telligence)<br />
and SIGINT (signals <strong>in</strong>telligence).<br />
53
CHAPTER FOUR<br />
ers, <strong>the</strong>y add to <strong>the</strong> accumulation of material that affects <strong>the</strong> ability of an analysts<br />
to produce.<br />
Product Influences. The accumulation of completed <strong>in</strong>telligence products,<br />
which is represented as a stock, is not <strong>in</strong> practice an end-state for analysis. A<br />
customer may respond to a delivered product by levy<strong>in</strong>g additional or revised<br />
task<strong>in</strong>g. In all <strong>in</strong>stances, this <strong>in</strong>formation <strong>in</strong>fluences <strong>the</strong> level of need for policymaker<br />
requirements and causes <strong>the</strong> process to beg<strong>in</strong> aga<strong>in</strong>. Each iteration of<br />
<strong>the</strong> process is different, not because <strong>the</strong> steps <strong>in</strong> <strong>the</strong> process change, but<br />
because those responsible for carry<strong>in</strong>g out <strong>the</strong> steps have changed as a result<br />
of <strong>the</strong>ir participation <strong>in</strong> <strong>the</strong> previous run. These changes can <strong>in</strong>clude a greater<br />
level of experience with <strong>the</strong> process, with <strong>the</strong> customer, with <strong>the</strong> topic area, or<br />
with <strong>the</strong> quirks of <strong>the</strong> organization and its processes. The changes are a manifestation<br />
of <strong>the</strong> concept that <strong>the</strong> system is greater than <strong>the</strong> sum of its parts.<br />
F<strong>in</strong>d<strong>in</strong>gs Based on Systems Analysis<br />
Systems analysis clearly demonstrates <strong>the</strong> defects of <strong>the</strong> traditional <strong>Intelligence</strong><br />
Cycle model. To recapitulate briefly, <strong>the</strong> traditional model merely represents<br />
a simple list of steps ra<strong>the</strong>r than a dynamic closed feedback loop. In<br />
addition, although <strong>the</strong> steps are meant to be performed by several different<br />
actors, <strong>the</strong> model does not provide useful <strong>in</strong>formation about what each actually<br />
contributes to <strong>the</strong> cycle, nor does it accurately represent <strong>the</strong> path a request<br />
takes as it is addressed. Ano<strong>the</strong>r problem with <strong>the</strong> traditional model is that<br />
none of its features help identify ways of develop<strong>in</strong>g a consistent product. For<br />
example, <strong>the</strong>re is no allowance for a statement of objectives or for any formative<br />
or summative evaluations to check that objectives have been met.<br />
On <strong>the</strong> o<strong>the</strong>r hand, <strong>the</strong> model that resulted from a systems analysis provides<br />
a more complex view. That model shows cause and effect, and it shows what<br />
o<strong>the</strong>r elements have an impact on <strong>the</strong> development of <strong>in</strong>telligence products<br />
and how and why elements depend on o<strong>the</strong>r elements. These advantages of <strong>the</strong><br />
systems model are clearly apparent <strong>in</strong> consider<strong>in</strong>g <strong>the</strong> role of analysts <strong>in</strong> production,<br />
a crucial element of <strong>the</strong> cycle that <strong>the</strong> traditional model all but<br />
ignores.<br />
Impact on Production and Analyst’s Control. Study of <strong>the</strong> systems model<br />
shows that <strong>the</strong> “Analyst’s Ability to Produce” (upper right-hand quarter of <strong>the</strong><br />
diagram) is <strong>the</strong> central factor <strong>in</strong> <strong>the</strong> production cycle and <strong>the</strong> driver of <strong>the</strong> feedback<br />
loop. The systems view also makes us aware of a less obvious fact that is<br />
critically important to a discussion of analytic failure.<br />
A look at <strong>the</strong> entire system makes readily apparent <strong>the</strong> number of factors of<br />
vary<strong>in</strong>g complexity that <strong>in</strong>fluence an analyst’s ability to produce: <strong>the</strong> analyst’s<br />
capabilities; <strong>the</strong> product evaluation process; <strong>the</strong> political and cultural values of<br />
54
TESTING THE INTELLIGENCE CYCLE<br />
<strong>the</strong> organization; <strong>the</strong> amount of relevant, usable data and actions related to<br />
transform<strong>in</strong>g collected data to relevant, usable data; and <strong>the</strong> level of demand<br />
on <strong>the</strong> analyst. Of <strong>the</strong>se five factors, only one—<strong>the</strong> analyst’s capabilities—is<br />
an <strong>in</strong>ternal factor and somewhat under <strong>the</strong> analyst’s control. 11 Yet, even<br />
though <strong>the</strong> o<strong>the</strong>r factors are out of <strong>the</strong> analyst’s control, <strong>the</strong> analyst must rely<br />
on <strong>the</strong>m to accomplish <strong>the</strong> goal and to meet <strong>the</strong> expectations of customers and<br />
<strong>the</strong> organization. When <strong>the</strong> proportion of external factors to <strong>in</strong>ternal factors is<br />
as unbalanced as <strong>the</strong> systems model of <strong>the</strong> <strong>Intelligence</strong> Cycle demonstrates,<br />
<strong>the</strong> causes of stress <strong>in</strong> <strong>the</strong> analytic environment <strong>in</strong>crease, as does <strong>the</strong> possibility<br />
that stress will occur.<br />
In such a high stress environment, where <strong>the</strong> critical person is responsible<br />
for deliver<strong>in</strong>g a product whose development relies on a great number of factors<br />
beyond his or her control, <strong>the</strong>re is greater risk of error, with an <strong>in</strong>creased<br />
likelihood of <strong>in</strong>complete or <strong>in</strong>correct products. Tendencies to use shortcuts, to<br />
avoid creative th<strong>in</strong>k<strong>in</strong>g, and to m<strong>in</strong>imize <strong>the</strong> perceived impact of certa<strong>in</strong><br />
events or actions become more apparent <strong>in</strong> this situation, especially if <strong>the</strong>ir<br />
implementation means reduc<strong>in</strong>g <strong>the</strong> workload and <strong>the</strong> stressors. Results of<br />
work<strong>in</strong>g <strong>in</strong> such an environment can <strong>in</strong>clude <strong>in</strong>creased personnel turnover,<br />
missed or undervalued <strong>in</strong>formation, lack of attention to detail, decreased motivation,<br />
and a lack of creativity <strong>in</strong> approach<strong>in</strong>g analysis. Moreover, with analysts<br />
so central to <strong>the</strong> process, <strong>the</strong>ir actions may have a widespread and, thus,<br />
powerful <strong>in</strong>fluence on <strong>the</strong> entire system. This change can be positive or negative.<br />
Given <strong>the</strong> number of elements <strong>in</strong>fluenc<strong>in</strong>g <strong>the</strong> analyst that are out of his<br />
or her control, however, it is unlikely that <strong>the</strong> changes would positively affect<br />
<strong>the</strong> quality, accuracy, and number of <strong>in</strong>telligence products created.<br />
Recommendations<br />
Revisit <strong>the</strong> traditional <strong>in</strong>telligence model. The traditional <strong>Intelligence</strong> Cycle<br />
model should ei<strong>the</strong>r be redesigned to depict accurately <strong>the</strong> <strong>in</strong>tended goal, or<br />
care should be taken to discuss explicitly its limitations whenever it is used.<br />
Teach<strong>in</strong>g with an <strong>in</strong>accurate aid merely leads to misconceptions that can result<br />
<strong>in</strong> poor performance, confusion, and a need for unlearn<strong>in</strong>g and reteach<strong>in</strong>g. If<br />
<strong>the</strong> objective is to capture <strong>the</strong> entire <strong>in</strong>telligence process, from <strong>the</strong> request for<br />
a product to its delivery, <strong>in</strong>clud<strong>in</strong>g <strong>the</strong> roles and responsibilities of <strong>Intelligence</strong><br />
<strong>Community</strong> members, <strong>the</strong>n someth<strong>in</strong>g more is required. This should be a<br />
model that pays particular attention to represent<strong>in</strong>g accurately all <strong>the</strong> elements<br />
of <strong>the</strong> process and <strong>the</strong> factors that <strong>in</strong>fluence <strong>the</strong>m.<br />
11<br />
Even <strong>the</strong> factors that contribute to <strong>the</strong> analyst’s capabilities, notably experience and tra<strong>in</strong><strong>in</strong>g, may<br />
be seen to be under <strong>the</strong> control of o<strong>the</strong>rs when access to, and selection of, <strong>the</strong>m are considered.<br />
55
CHAPTER FOUR<br />
Fur<strong>the</strong>r Study. The use of simulation allows us to determ<strong>in</strong>e flaws <strong>in</strong> <strong>the</strong> system<br />
that basic <strong>in</strong>formational models cannot address. A simulation moves <strong>the</strong><br />
image of <strong>the</strong> <strong>Intelligence</strong> Cycle from a picture that selectively and <strong>in</strong>discrim<strong>in</strong>ately<br />
illustrates a series of events to a holistic and realistic representation of<br />
events, responsibilities, processes, and <strong>the</strong>ir impact on each o<strong>the</strong>r. The simulation<br />
of <strong>the</strong> <strong>Intelligence</strong> Cycle developed for this analysis is merely a first step.<br />
Fur<strong>the</strong>r work should be done with it to validate <strong>the</strong> representations, test for vulnerabilities,<br />
predict outcomes, and accurately recommend changes.<br />
Lighten<strong>in</strong>g <strong>the</strong> Analyst’s Load. The systems model reveals a serious imbalance<br />
<strong>in</strong> <strong>the</strong> work processes analysts can and cannot control. It is unrealistic<br />
and unnecessary to consider reorganiz<strong>in</strong>g <strong>the</strong> process to correct this defect.<br />
However, <strong>the</strong>re are actions that could be taken to provide analysts more control<br />
over external factors without significantly alter<strong>in</strong>g <strong>the</strong>ir roles. These<br />
actions would also reduce <strong>the</strong> amount of potential <strong>in</strong>fluence that one group<br />
could have over <strong>the</strong> entire process.<br />
First, analysts might be designated as reports or research analysts. The<br />
former would prepare products that address short-term tasks, such as writ<strong>in</strong>g<br />
for <strong>the</strong> PDB. As <strong>the</strong> process of collection and analysis is different for shortand<br />
long-term products, this might be a responsibility assigned primarily to<br />
more junior analysts. Research analysts might be those with more experience.<br />
Freed from <strong>the</strong> obligation to prepare short-term reports, senior analysts would<br />
be available for more <strong>in</strong>tense research efforts, such as those required for an<br />
NIE. In addition, cross-tra<strong>in</strong><strong>in</strong>g or experience <strong>in</strong> creat<strong>in</strong>g both products and<br />
<strong>the</strong> flexibility to switch from one process to ano<strong>the</strong>r would provide greater<br />
depth of personnel. If appropriate, movement to a long-term research position<br />
could be viewed as professional development.<br />
Second, personnel responsible for formatt<strong>in</strong>g and process<strong>in</strong>g raw data<br />
might be <strong>in</strong>cluded on accounts. Through association with a particular group,<br />
people <strong>in</strong> this role would have a reasonable idea of analysts’ requirements.<br />
This would allow <strong>the</strong> preselection and preparation of data, so that analysts<br />
could focus on “connect<strong>in</strong>g <strong>the</strong> dots.” The skills requirement for this role<br />
would be ak<strong>in</strong> to those of a research librarian.<br />
Third, tools to help <strong>the</strong> analyst identify, manage, and fuse relevant data<br />
could be identified and deployed. These tools, which need not be limited to<br />
those that are technology-based, should be used to support analysts’ labor<strong>in</strong>tensive<br />
tasks, <strong>the</strong>reby free<strong>in</strong>g <strong>the</strong>m to focus on <strong>the</strong> analysis of data.<br />
Employ alternative methods for exam<strong>in</strong><strong>in</strong>g work processes. Just as we used<br />
alternative methods to exam<strong>in</strong>e <strong>the</strong> I<strong>in</strong>telligence Cycle, and as managers press<br />
analysts to use alternative analyses <strong>in</strong> assess<strong>in</strong>g <strong>the</strong>ir targets, so should managers<br />
employ alternative methods for exam<strong>in</strong><strong>in</strong>g work processes. These methods<br />
56
TESTING THE INTELLIGENCE CYCLE<br />
should not simply test effectivenss; <strong>the</strong>y should also identify vulnerabilities and<br />
potential sources of o<strong>the</strong>r problems <strong>in</strong> <strong>the</strong> community’s analytical methods.<br />
57
PART III<br />
Potential Areas for Improvement<br />
59
CHAPTER FIVE<br />
Integrat<strong>in</strong>g Methodologists <strong>in</strong>to Teams of Experts 1<br />
<strong>Intelligence</strong> analysis, like o<strong>the</strong>r complex tasks, demands considerable<br />
expertise. It requires <strong>in</strong>dividuals who can recognize patterns <strong>in</strong> large data sets,<br />
solve complex problems, and make predictions about future behavior or<br />
events. To perform <strong>the</strong>se tasks successfully, analysts must dedicate years to<br />
research<strong>in</strong>g specific topics, processes, and geographic regions.<br />
Paradoxically, it is <strong>the</strong> specificity of expertise that makes expert forecasts<br />
unreliable. While experts outperform novices and mach<strong>in</strong>es <strong>in</strong> pattern recognition<br />
and problem solv<strong>in</strong>g, expert predictions of future behavior or events are<br />
seldom as accurate as Bayesian probabilities. 2 This is due, <strong>in</strong> part, to cognitive<br />
biases and process<strong>in</strong>g-time constra<strong>in</strong>ts and, <strong>in</strong> part, to <strong>the</strong> nature of expertise<br />
itself and <strong>the</strong> process by which one becomes an expert.<br />
Becom<strong>in</strong>g an Expert<br />
Expertise is commitment coupled with creativity. By this, I mean <strong>the</strong> commitment<br />
of time, energy, and resources to a relatively narrow field of study<br />
and <strong>the</strong> creative energy necessary to generate new knowledge <strong>in</strong> that field. It<br />
takes a great deal of time and regular exposure to a large number of cases to<br />
become an expert.<br />
1<br />
A version of this chapter orig<strong>in</strong>ally appeared as “Integrat<strong>in</strong>g Methodologists <strong>in</strong>to Teams of Substantive<br />
Experts <strong>in</strong> Studies <strong>in</strong> <strong>Intelligence</strong> 47, no. 1 (2003): 57–65.<br />
2<br />
Method for estimat<strong>in</strong>g <strong>the</strong> probability of a given outcome developed by Thomas Bayes (1702–<br />
61), an English ma<strong>the</strong>matician. See Thomas Bayes, “An Essay Toward Solv<strong>in</strong>g a Problem In <strong>the</strong><br />
Doctr<strong>in</strong>e of Chances.”<br />
61
CHAPTER FIVE<br />
Enter<strong>in</strong>g a field of study as a novice, an <strong>in</strong>dividual needs to learn <strong>the</strong> heuristics<br />
and constra<strong>in</strong>ts—that is, <strong>the</strong> guid<strong>in</strong>g pr<strong>in</strong>ciples and rules—of a given task<br />
<strong>in</strong> order to perform that task. Concurrently, <strong>the</strong> novice needs to be exposed to<br />
specific cases that test <strong>the</strong> reliability of such heuristics. Generally, novices<br />
f<strong>in</strong>d mentors to guide <strong>the</strong>m through <strong>the</strong> process of acquir<strong>in</strong>g new knowledge.<br />
A fairly simple example would be someone learn<strong>in</strong>g to play chess. The novice<br />
chess player seeks a mentor who can expla<strong>in</strong> <strong>the</strong> object of <strong>the</strong> game, <strong>the</strong> number<br />
of spaces, <strong>the</strong> names of <strong>the</strong> pieces, <strong>the</strong> function of each piece, how each<br />
piece is moved, and <strong>the</strong> necessary conditions for w<strong>in</strong>n<strong>in</strong>g or los<strong>in</strong>g a game.<br />
In time, and with much practice, <strong>the</strong> novice beg<strong>in</strong>s to recognize patterns of<br />
behavior with<strong>in</strong> cases and, thus, becomes a journeyman. With more practice<br />
and exposure to <strong>in</strong>creas<strong>in</strong>gly complex cases, <strong>the</strong> journeyman f<strong>in</strong>ds patterns not<br />
only with<strong>in</strong> but also among cases and, more important, learns that <strong>the</strong>se patterns<br />
often repeat <strong>the</strong>mselves. Throughout, <strong>the</strong> journeyman still ma<strong>in</strong>ta<strong>in</strong>s regular<br />
contact with a mentor to solve specific problems and to learn more<br />
complex strategies. Return<strong>in</strong>g to <strong>the</strong> example of <strong>the</strong> chess player, <strong>the</strong> <strong>in</strong>dividual<br />
beg<strong>in</strong>s to learn patterns of open<strong>in</strong>g moves, offensive and defensive strategies,<br />
and patterns of victory and defeat.<br />
The next stage beg<strong>in</strong>s when a journeyman makes and tests hypo<strong>the</strong>ses<br />
about future behavior based on past experiences. Once he creatively generates<br />
knowledge, ra<strong>the</strong>r than simply match<strong>in</strong>g patterns, he becomes an expert. At<br />
this po<strong>in</strong>t, he becomes responsible for his own knowledge and no longer needs<br />
a mentor. In <strong>the</strong> chess example, once a journeyman beg<strong>in</strong>s compet<strong>in</strong>g aga<strong>in</strong>st<br />
experts, makes predictions based on patterns, and tests those predictions<br />
aga<strong>in</strong>st actual behavior, he is generat<strong>in</strong>g new knowledge and a deeper understand<strong>in</strong>g<br />
of <strong>the</strong> game. He is creat<strong>in</strong>g his own cases ra<strong>the</strong>r than rely<strong>in</strong>g on <strong>the</strong><br />
cases of o<strong>the</strong>rs.<br />
The chess example <strong>in</strong> <strong>the</strong> preced<strong>in</strong>g paragraphs is a concise description of<br />
an apprenticeship model. Apprenticeship may seem to many a restrictive, oldfashioned<br />
mode of education, but it rema<strong>in</strong>s a standard method of tra<strong>in</strong><strong>in</strong>g for<br />
many complex tasks. In fact, academic doctoral programs are based on an<br />
apprenticeship model, as are such fields as law, music, eng<strong>in</strong>eer<strong>in</strong>g, and medic<strong>in</strong>e.<br />
Graduate students enter fields of study, f<strong>in</strong>d mentors, and beg<strong>in</strong> <strong>the</strong> long<br />
process of becom<strong>in</strong>g <strong>in</strong>dependent experts and generat<strong>in</strong>g new knowledge <strong>in</strong><br />
<strong>the</strong>ir respective doma<strong>in</strong>s.<br />
To some, play<strong>in</strong>g chess may appear ra<strong>the</strong>r trivial when compared, for<br />
example, with mak<strong>in</strong>g medical diagnoses, but both are highly complex tasks.<br />
Chess heuristics are well-def<strong>in</strong>ed, whereas medical diagnoses seem more open<br />
ended and variable. In both <strong>in</strong>stances, however, <strong>the</strong>re are tens of thousands of<br />
potential patterns. A research study discovered that chess masters had spent<br />
between 10,000 and 20,000 hours, or more than 10 years, study<strong>in</strong>g and play-<br />
62
INTEGRATING METHODOLOGISTS<br />
<strong>in</strong>g chess. On average, a chess master acquires 50,000 different chess patterns.<br />
3<br />
Similarly, a diagnostic radiologist spends eight years <strong>in</strong> full-time medical<br />
tra<strong>in</strong><strong>in</strong>g - four years of medical school and four years of residency—before<br />
be<strong>in</strong>g qualified to take a national board exam and beg<strong>in</strong> <strong>in</strong>dependent practice. 4<br />
Accord<strong>in</strong>g to a 1988 study, <strong>the</strong> average diagnostic radiology resident sees 40<br />
cases per day, or around 12,000 cases per year. 5 At <strong>the</strong> end of a residency, a<br />
diagnostic radiologist has acquired an average of 48,000 cases.<br />
Psychologists and cognitive scientists agree that <strong>the</strong> time it takes to become<br />
an expert depends on <strong>the</strong> complexity of <strong>the</strong> task and <strong>the</strong> number of cases, or<br />
patterns, to which an <strong>in</strong>dividual is exposed. The more complex <strong>the</strong> task, <strong>the</strong><br />
longer it takes to build expertise, or, more accurately, <strong>the</strong> longer it takes to<br />
experience a large number of cases or patterns.<br />
The Power of Expertise<br />
Experts are <strong>in</strong>dividuals with specialized knowledge suited to perform <strong>the</strong><br />
specific tasks for which <strong>the</strong>y are tra<strong>in</strong>ed, but that expertise does not necessarily<br />
transfer to o<strong>the</strong>r doma<strong>in</strong>s. 6 A master chess player cannot apply chess<br />
expertise <strong>in</strong> a game of poker; although both chess and poker are games, a<br />
chess master who has never played poker is a novice poker player. Similarly, a<br />
biochemist is not qualified to perform neurosurgery, even though both biochemists<br />
and neurosurgeons study human physiology. In o<strong>the</strong>r words, <strong>the</strong><br />
more complex a task, <strong>the</strong> more specialized and exclusive is <strong>the</strong> knowledge<br />
required to perform that task.<br />
Experts perceive mean<strong>in</strong>gful patterns <strong>in</strong> <strong>the</strong>ir doma<strong>in</strong>s better than do nonexperts.<br />
Where a novice perceives random or disconnected data po<strong>in</strong>ts, an<br />
expert connects regular patterns with<strong>in</strong> and among cases. This ability to identify<br />
patterns is not an <strong>in</strong>nate perceptual skill; ra<strong>the</strong>r, it reflects <strong>the</strong> organization<br />
of knowledge after exposure to and experience with thousands of cases. 7<br />
Experts have a deeper understand<strong>in</strong>g of <strong>the</strong>ir doma<strong>in</strong>s than do novices, and<br />
<strong>the</strong>y utilize higher-order pr<strong>in</strong>ciples to solve problems. 8 A novice, for example,<br />
might group objects toge<strong>the</strong>r by color or size, whereas an expert would group<br />
3<br />
W. Chase and H. Simon, “Perception <strong>in</strong> Chess.”<br />
4<br />
American College of Radiology. Personal communication, 2002.<br />
5<br />
A. Lesgold et al., “Expertise <strong>in</strong> a Complex Skill: Diagnos<strong>in</strong>g X-Ray Pictures.”<br />
6<br />
M. M<strong>in</strong>sky and S. Papert, Artificial <strong>Intelligence</strong>; J. Voss and T. Post, “On <strong>the</strong> Solv<strong>in</strong>g of Ill-<br />
Structured Problems.”<br />
7<br />
O. Ak<strong>in</strong>, Models of Architectural Knowledge; D. Egan and B. Schwartz, “Chunk<strong>in</strong>g <strong>in</strong> Recall of<br />
Symbolic Draw<strong>in</strong>gs”; K. McKei<strong>the</strong>n et al., “Knowledge Organization and Skill Differences <strong>in</strong><br />
Computer Programmers.”<br />
63
CHAPTER FIVE<br />
<strong>the</strong> same objects accord<strong>in</strong>g to <strong>the</strong>ir function or utility. Experts comprehend <strong>the</strong><br />
mean<strong>in</strong>g of data better than novices, and <strong>the</strong>y weigh variables with different<br />
criteria with<strong>in</strong> <strong>the</strong>ir doma<strong>in</strong>s better. Experts recognize variables that have <strong>the</strong><br />
largest <strong>in</strong>fluence on a particular problem and focus <strong>the</strong>ir attention on those<br />
variables.<br />
Experts have better doma<strong>in</strong>-specific short-term and long-term memory than<br />
do novices. 9 Moreover, experts perform tasks <strong>in</strong> <strong>the</strong>ir doma<strong>in</strong>s faster than novices<br />
and commit fewer errors while solv<strong>in</strong>g problems. 10 Interest<strong>in</strong>gly, experts<br />
also go about solv<strong>in</strong>g problems differently. At <strong>the</strong> beg<strong>in</strong>n<strong>in</strong>g of a task, experts<br />
spend more time th<strong>in</strong>k<strong>in</strong>g about a problem than do novices, who immediately<br />
seek to f<strong>in</strong>d a solution. 11 Experts use <strong>the</strong>ir knowledge of previous cases as context<br />
for creat<strong>in</strong>g mental models to solve given problems. 12<br />
Because <strong>the</strong>y are better at self-monitor<strong>in</strong>g than novices, experts are more<br />
aware of <strong>in</strong>stances where <strong>the</strong>y have committed errors or failed to understand a<br />
problem. 13 They check <strong>the</strong>ir solutions more often and recognize when <strong>the</strong>y<br />
are miss<strong>in</strong>g <strong>in</strong>formation necessary for solv<strong>in</strong>g a problem. 14 Experts are aware<br />
of <strong>the</strong> limits of <strong>the</strong>ir knowledge and apply <strong>the</strong>ir doma<strong>in</strong>’s heuristics to solve<br />
problems that fall outside of <strong>the</strong>ir experience base.<br />
The Paradox of Expertise<br />
The strengths of expertise can also be weaknesses. 15 Although one would<br />
expect experts to be good forecasters, <strong>the</strong>y are not particularly good at it.<br />
Researchers have been test<strong>in</strong>g <strong>the</strong> ability of experts to make forecasts s<strong>in</strong>ce<br />
<strong>the</strong> 1930s. 16 The performance of experts has been tested aga<strong>in</strong>st Bayesian<br />
probabilities to determ<strong>in</strong>e if <strong>the</strong>y are better at mak<strong>in</strong>g predictions than simple<br />
statistical models. Seventy years later, after more than 200 hundred experiments<br />
<strong>in</strong> different doma<strong>in</strong>s, it is clear that <strong>the</strong> answer is no. 17 Supplied with an<br />
equal amount of data about a particular case, Bayesian probability data are as<br />
8<br />
M. Chi, P. Feltovich, and R. Glaser, “Categorization and Representation of Physics Problems by<br />
Experts and Novices”; M. Weiser and J. Shertz, “Programm<strong>in</strong>g Problem Representation <strong>in</strong> Novice<br />
and Expert Programmers.”<br />
9<br />
W. Chase and K. Ericsson, “Skill and Work<strong>in</strong>g Memory.”<br />
10<br />
W. Chase, “Spatial Representations of Taxi Drivers.”<br />
11<br />
J. Paige and H. Simon, “Cognition Processes <strong>in</strong> Solv<strong>in</strong>g Algebra Word Problems.”<br />
12<br />
Voss and Post.<br />
13<br />
M. Chi, R. Glaser, and E. Rees, “Expertise <strong>in</strong> Problem Solv<strong>in</strong>g”; D. Simon and H. Simon,<br />
“Individual Differences <strong>in</strong> Solv<strong>in</strong>g Physics Problems.”<br />
14<br />
J. Lark<strong>in</strong>, “The Role of Problem Representation <strong>in</strong> Physics.”<br />
15<br />
C. Camerer and E. Johnson, “The Process-Performance Paradox <strong>in</strong> Expert Judgment.”<br />
16<br />
H. Reichenbach, Experience and Prediction; T. Sarb<strong>in</strong>, “A Contribution to <strong>the</strong> Study of Actuarial<br />
and Individual Methods of Prediction.”<br />
64
INTEGRATING METHODOLOGISTS<br />
good as, or better than, an expert at mak<strong>in</strong>g calls about <strong>the</strong> future. In fact, <strong>the</strong><br />
expert does not tend to outperform <strong>the</strong> actuarial table, even if given more specific<br />
case <strong>in</strong>formation than is available to <strong>the</strong> statistical model. 18<br />
There are few exceptions to <strong>the</strong>se research f<strong>in</strong>d<strong>in</strong>gs, but <strong>the</strong>se are <strong>in</strong>formative.<br />
When experts are given <strong>the</strong> results of <strong>the</strong> Bayesian probabilities, for<br />
example, <strong>the</strong>y tend to score as well as <strong>the</strong> statistical model if <strong>the</strong>y use <strong>the</strong> statistical<br />
<strong>in</strong>formation <strong>in</strong> mak<strong>in</strong>g <strong>the</strong>ir own predictions. 19 In addition, if experts<br />
have privileged <strong>in</strong>formation that is not reflected <strong>in</strong> <strong>the</strong> statistical table, <strong>the</strong>y<br />
will actually perform better than does <strong>the</strong> table. A classic example is <strong>the</strong> case<br />
of <strong>the</strong> judge’s broken leg. Judge X has gone to <strong>the</strong> <strong>the</strong>ater every Friday night<br />
for <strong>the</strong> past 10 years. Based on a Bayesian analysis, one would predict, with<br />
some certa<strong>in</strong>ty, that this Friday night would be no different. An expert knows,<br />
however, that <strong>the</strong> judge broke her leg Thursday afternoon and is expected to<br />
be <strong>in</strong> <strong>the</strong> hospital until Saturday. Know<strong>in</strong>g this key variable allows <strong>the</strong> expert<br />
to predict that <strong>the</strong> judge will not attend <strong>the</strong> <strong>the</strong>ater this Friday.<br />
Although hav<strong>in</strong>g a s<strong>in</strong>gle variable as <strong>the</strong> determ<strong>in</strong><strong>in</strong>g factor makes this case<br />
easy to grasp, analysis is seldom, if ever, this simple. Forecast<strong>in</strong>g is a complex,<br />
<strong>in</strong>terdiscipl<strong>in</strong>ary, dynamic, and multivariate task where<strong>in</strong> many variables<br />
<strong>in</strong>teract, weight and value change, and o<strong>the</strong>r variables are <strong>in</strong>troduced or omitted.<br />
Dur<strong>in</strong>g <strong>the</strong> past 30 years, researchers have categorized, experimented, and<br />
<strong>the</strong>orized about <strong>the</strong> cognitive aspects of forecast<strong>in</strong>g and have sought to<br />
expla<strong>in</strong> why experts are less accurate forecasters than statistical models.<br />
Despite such efforts, <strong>the</strong> literature shows little consensus regard<strong>in</strong>g <strong>the</strong> causes<br />
or manifestations of human bias. Some have argued that experts, like all<br />
humans, are <strong>in</strong>consistent when us<strong>in</strong>g mental models to make predictions. That<br />
is, <strong>the</strong> model an expert uses for predict<strong>in</strong>g X <strong>in</strong> one month is different from <strong>the</strong><br />
model used for predict<strong>in</strong>g X <strong>in</strong> a later month, although precisely <strong>the</strong> same case<br />
and same data set are used <strong>in</strong> both <strong>in</strong>stances. 20 A number of researchers po<strong>in</strong>t<br />
17<br />
R. Dawes, D. Faust, and P. Meehl, “Cl<strong>in</strong>ical Versus Actuarial Judgment”; W. Grove and P.<br />
Meehl, “Comparative Efficiency of Informal (Subjective, Impressionistic) and Formal (Mechanical,<br />
Algorithmic) Prediction Procedures.”<br />
18<br />
R. Dawes, “A Case Study of Graduate Admissions”; Grove and Meehl; H. Sacks, “Promises,<br />
Performance, and Pr<strong>in</strong>ciples”; T. Sarb<strong>in</strong>, “A Contribution to <strong>the</strong> Study of Actuarial and Individual<br />
Methods of Prediction”; J. Sawyer, “Measurement and Prediction, Cl<strong>in</strong>ical and Statistical”; W.<br />
Schofield and J. Garrard, “Longitud<strong>in</strong>al Study of Medical Students Selected for Admission to<br />
Medical School by Actuarial and Committee Methods.”<br />
19<br />
L. Goldberg, “Simple Models or Simple Processes?”; L. Goldberg, “Man versus Model of<br />
Man”; D. Leli and S. Filskov, “Cl<strong>in</strong>ical-Actuarial Detection of and Description of Bra<strong>in</strong> Impairment<br />
with <strong>the</strong> Wechsler-Bellevue Form I.”<br />
20<br />
J. Fries, et al., “Assessment of Radiologic Progression <strong>in</strong> Rheumatoid Arthritis.”<br />
65
CHAPTER FIVE<br />
to human biases to expla<strong>in</strong> unreliable expert predictions. 21 There is general<br />
agreement that two types of bias exist:<br />
• Pattern bias: look<strong>in</strong>g for evidence that confirms ra<strong>the</strong>r than rejects a<br />
hypo<strong>the</strong>sis and/or fill<strong>in</strong>g <strong>in</strong>—perhaps <strong>in</strong>advertently—miss<strong>in</strong>g data with<br />
data from previous experiences;<br />
• Heuristic bias: us<strong>in</strong>g <strong>in</strong>appropriate guidel<strong>in</strong>es or rules to make predictions.<br />
Paradoxically, <strong>the</strong> very method by which one becomes an expert expla<strong>in</strong>s<br />
why experts are much better than novices at describ<strong>in</strong>g, expla<strong>in</strong><strong>in</strong>g, perform<strong>in</strong>g<br />
tasks, and solv<strong>in</strong>g problems with<strong>in</strong> <strong>the</strong>ir doma<strong>in</strong>s but, with few exceptions,<br />
are worse at forecast<strong>in</strong>g than are Bayesian probabilities based on<br />
historical, statistical models. A given doma<strong>in</strong> has specific heuristics for perform<strong>in</strong>g<br />
tasks and solv<strong>in</strong>g problems, and <strong>the</strong>se rules are a large part of what<br />
makes up expertise. In addition, experts need to acquire and store tens of thousands<br />
of cases <strong>in</strong> order to recognize patterns, generate and test hypo<strong>the</strong>ses, and<br />
contribute to <strong>the</strong> collective knowledge with<strong>in</strong> <strong>the</strong>ir fields. In o<strong>the</strong>r words,<br />
becom<strong>in</strong>g an expert requires a significant number of years of view<strong>in</strong>g <strong>the</strong><br />
world through <strong>the</strong> lens of one specific doma<strong>in</strong>. This concentration gives <strong>the</strong><br />
expert <strong>the</strong> power to recognize patterns, perform tasks, and solve problems, but<br />
it also focuses <strong>the</strong> expert’s attention on one doma<strong>in</strong> to <strong>the</strong> exclusion of o<strong>the</strong>rs.<br />
It should come as little surprise, <strong>the</strong>n, that an expert would have difficulty<br />
identify<strong>in</strong>g and weigh<strong>in</strong>g variables <strong>in</strong> an <strong>in</strong>terdiscipl<strong>in</strong>ary task, such as forecast<strong>in</strong>g<br />
an adversary’s <strong>in</strong>tentions. Put differently, an expert may know his specific<br />
doma<strong>in</strong>, such as economics or leadership analysis, quite thoroughly, but<br />
that may still not permit him to div<strong>in</strong>e an adversary’s <strong>in</strong>tention, which <strong>the</strong><br />
adversary may not himself know.<br />
The Burden on <strong>Intelligence</strong> Analysts<br />
<strong>Intelligence</strong> analysis is an amalgam of a number of highly specialized<br />
doma<strong>in</strong>s. With<strong>in</strong> each, experts are tasked with assembl<strong>in</strong>g, analyz<strong>in</strong>g, assign<strong>in</strong>g<br />
mean<strong>in</strong>g to, and report<strong>in</strong>g on data, <strong>the</strong> goals be<strong>in</strong>g to describe an event or<br />
observation, solve a problem, or make a forecast. Experts who encounter a<br />
case outside <strong>the</strong>ir field repeat <strong>the</strong> steps <strong>the</strong>y <strong>in</strong>itially used to acquire <strong>the</strong>ir<br />
expertise. Thus, <strong>the</strong>y can try to make <strong>the</strong> new data fit a pattern previously<br />
acquired; recognize that <strong>the</strong> case falls outside <strong>the</strong>ir expertise and turn to <strong>the</strong>ir<br />
doma<strong>in</strong>’s heuristics to try to give mean<strong>in</strong>g to <strong>the</strong> data; acknowledge that <strong>the</strong><br />
21<br />
J. Evans, Bias <strong>in</strong> Human Reason<strong>in</strong>g; R. Heuer, Psychology of <strong>Intelligence</strong> Analysis; D. Kahneman,<br />
P. Slovic, and A. Tversky, Judgment Under Uncerta<strong>in</strong>ty; A. Tversky and D. Kahneman,<br />
“The Belief <strong>in</strong> <strong>the</strong> ‘Law of Small Numbers’.”<br />
66
INTEGRATING METHODOLOGISTS<br />
case still does not fit with <strong>the</strong>ir expertise and reject <strong>the</strong> data set as an anomaly;<br />
or consult o<strong>the</strong>r experts.<br />
An item of <strong>in</strong>formation, <strong>in</strong> and of itself, is not doma<strong>in</strong> specific. Imag<strong>in</strong>e<br />
economic data that reveal that a country is <strong>in</strong>vest<strong>in</strong>g <strong>in</strong> technological <strong>in</strong>frastructure,<br />
chemical supplies, and research and development. An economist<br />
might decide that <strong>the</strong> data fit an exist<strong>in</strong>g spend<strong>in</strong>g pattern and <strong>in</strong>tegrate <strong>the</strong>se<br />
facts with prior knowledge about a country’s economy. The same economist<br />
might decide that this is a new pattern that needs to be stored <strong>in</strong> long-term<br />
memory for some future use, or he might decide that <strong>the</strong> data are outliers of no<br />
consequence and may be ignored. F<strong>in</strong>ally, <strong>the</strong> economist might decide that <strong>the</strong><br />
data would be mean<strong>in</strong>gful to a chemist or biologist and, <strong>the</strong>refore, seek to collaborate<br />
with o<strong>the</strong>r specialists, who might reach different conclusions regard<strong>in</strong>g<br />
<strong>the</strong> data than would <strong>the</strong> economist.<br />
In this example, <strong>the</strong> economist is required to use his economic expertise <strong>in</strong><br />
all but <strong>the</strong> f<strong>in</strong>al option of consult<strong>in</strong>g o<strong>the</strong>r experts. In <strong>the</strong> decision to seek collaboration,<br />
<strong>the</strong> economist is expected to know that what appears to be new<br />
economic data may have value to a chemist or biologist, doma<strong>in</strong>s with which<br />
he may have no experience. In o<strong>the</strong>r words, <strong>the</strong> economist is expected to know<br />
that an expert <strong>in</strong> some o<strong>the</strong>r field might f<strong>in</strong>d mean<strong>in</strong>g <strong>in</strong> data that appear to be<br />
economic.<br />
Three disparate variables complicate <strong>the</strong> economist’s decisionmak<strong>in</strong>g:<br />
• Time context. This does not refer to <strong>the</strong> amount of time necessary to<br />
accomplish a task but ra<strong>the</strong>r to <strong>the</strong> limitations that come from be<strong>in</strong>g close<br />
to an event. The economist cannot say a priori that <strong>the</strong> new data set is <strong>the</strong><br />
critical data set for some future event. In “real time,” <strong>the</strong>y are simply data<br />
to be manipulated. It is only <strong>in</strong> retrospect, or <strong>in</strong> long-term memory, that<br />
<strong>the</strong> economist can fit <strong>the</strong> data <strong>in</strong>to a larger pattern, weigh <strong>the</strong>ir value, and<br />
assign <strong>the</strong>m mean<strong>in</strong>g.<br />
• Pattern bias. In this particular example, <strong>the</strong> data have to do with <strong>in</strong>frastructure<br />
<strong>in</strong>vestment, and <strong>the</strong> expert is an economist. Thus, it makes perfect<br />
sense to try to manipulate <strong>the</strong> new data with<strong>in</strong> <strong>the</strong> context of<br />
economics, recogniz<strong>in</strong>g, however, that <strong>the</strong>re may be o<strong>the</strong>r, more important<br />
angles.<br />
• Heuristic bias. The economist has spent a career becom<strong>in</strong>g familiar with<br />
and us<strong>in</strong>g <strong>the</strong> guid<strong>in</strong>g pr<strong>in</strong>ciples of economic analysis and, at best, has<br />
only a vague familiarity with o<strong>the</strong>r doma<strong>in</strong>s and <strong>the</strong>ir heuristics. An economist<br />
would not necessarily know that a chemist or biologist could identify<br />
what substance is be<strong>in</strong>g produced based on <strong>the</strong> types of equipment<br />
and supplies that are be<strong>in</strong>g purchased.<br />
67
CHAPTER FIVE<br />
This example does not describe a complex problem; most people would<br />
recognize that <strong>the</strong> data from this case might be of value to o<strong>the</strong>r doma<strong>in</strong>s. It is<br />
one isolated case, viewed retrospectively, which could potentially affect two<br />
o<strong>the</strong>r doma<strong>in</strong>s. But, what if <strong>the</strong> economist had to deal with 100 data sets per<br />
day? Now, multiply those 100 data sets by <strong>the</strong> number of doma<strong>in</strong>s potentially<br />
<strong>in</strong>terested <strong>in</strong> any given economic data set. F<strong>in</strong>ally, put all of this <strong>in</strong> <strong>the</strong> context<br />
of “real time.” The economic expert is now expected to ma<strong>in</strong>ta<strong>in</strong> expertise <strong>in</strong><br />
economics, which is a full-time endeavor, while simultaneously acquir<strong>in</strong>g<br />
some level of experience <strong>in</strong> every o<strong>the</strong>r doma<strong>in</strong>. Based on <strong>the</strong>se expectations,<br />
<strong>the</strong> knowledge requirements for effective collaboration quickly exceed <strong>the</strong><br />
capabilities of <strong>the</strong> <strong>in</strong>dividual expert.<br />
The expert is left deal<strong>in</strong>g with all of <strong>the</strong>se data through <strong>the</strong> lens of his own<br />
expertise. Let’s assume that he uses his doma<strong>in</strong> heuristics to <strong>in</strong>corporate <strong>the</strong><br />
data <strong>in</strong>to an exist<strong>in</strong>g pattern, store <strong>the</strong> data <strong>in</strong> long-term memory as a new pattern,<br />
or reject <strong>the</strong> data set as an outlier. In each of <strong>the</strong>se options, <strong>the</strong> data stop<br />
with <strong>the</strong> economist <strong>in</strong>stead of be<strong>in</strong>g shared with an expert <strong>in</strong> some o<strong>the</strong>r<br />
doma<strong>in</strong>. The fact that <strong>the</strong>se data are not shared <strong>the</strong>n becomes a potentially critical<br />
case of analytic error. 22<br />
In h<strong>in</strong>dsight, critics will say that <strong>the</strong> implications were obvious—that <strong>the</strong><br />
crisis could have been avoided if <strong>the</strong> data had been passed to one or ano<strong>the</strong>r<br />
specific expert. In “real time,” however, an expert often does not know which<br />
particular data set would have value for an expert <strong>in</strong> ano<strong>the</strong>r doma<strong>in</strong>.<br />
The Pros and Cons of Teams<br />
One obvious solution to <strong>the</strong> paradox of expertise is to assemble an <strong>in</strong>terdiscipl<strong>in</strong>ary<br />
team. Why not simply make all problem areas or country-specific<br />
data available to a team of experts from a variety of doma<strong>in</strong>s? This ought, at<br />
least, to reduce <strong>the</strong> pattern and heuristic biases <strong>in</strong>herent <strong>in</strong> rely<strong>in</strong>g on only one<br />
doma<strong>in</strong>. Ignor<strong>in</strong>g potential security issues, <strong>the</strong>re are practical problems with<br />
this approach. First, each expert would have to sift through large data sets to<br />
f<strong>in</strong>d data specific to his expertise. This would be <strong>in</strong>ord<strong>in</strong>ately time-consum<strong>in</strong>g<br />
and might not even be rout<strong>in</strong>ely possible, given <strong>the</strong> priority accorded gist<strong>in</strong>g<br />
and current report<strong>in</strong>g.<br />
Second, dur<strong>in</strong>g <strong>the</strong> act of scann<strong>in</strong>g large data sets, <strong>the</strong> expert <strong>in</strong>evitably<br />
would be look<strong>in</strong>g for data that fit with<strong>in</strong> his area of expertise. Imag<strong>in</strong>e a chemist<br />
who comes across data that show that a country is <strong>in</strong>vest<strong>in</strong>g <strong>in</strong> technologi-<br />
22<br />
L. Kirkpatrick, Capta<strong>in</strong>s Without Eyes: <strong>Intelligence</strong> Failures <strong>in</strong> World War II; F. Shiels, Preventable<br />
Disasters; J. Wirtz, The Tet Offensive: <strong>Intelligence</strong> Failure <strong>in</strong> War; R. Wohlstetter, Pearl<br />
Harbor.<br />
68
INTEGRATING METHODOLOGISTS<br />
cal <strong>in</strong>frastructure, chemical supplies, and research and development (<strong>the</strong> same<br />
data that <strong>the</strong> economist analyzed <strong>in</strong> <strong>the</strong> previous example). The chemist recognizes<br />
that <strong>the</strong>se are <strong>the</strong> <strong>in</strong>gredients necessary for a nation to produce a specific<br />
chemical agent, which could have a military application or could be benign.<br />
The chemist <strong>the</strong>n meshes <strong>the</strong> data with an exist<strong>in</strong>g pattern, stores <strong>the</strong> data as a<br />
new pattern, or ignores <strong>the</strong> data as an anomaly.<br />
The chemist, however, has no frame of reference regard<strong>in</strong>g spend<strong>in</strong>g trends<br />
<strong>in</strong> <strong>the</strong> country of <strong>in</strong>terest. He does not know if <strong>the</strong> <strong>in</strong>vestment <strong>in</strong> chemical supplies<br />
represents an <strong>in</strong>crease, a decrease, or a static spend<strong>in</strong>g pattern—answers<br />
<strong>the</strong> economist could supply immediately. There is no reason for <strong>the</strong> chemist to<br />
know if a country’s ability to produce this chemical agent is a new phenomenon.<br />
Perhaps <strong>the</strong> country <strong>in</strong> question has been produc<strong>in</strong>g <strong>the</strong> chemical agent<br />
for years, and <strong>the</strong>se data are part of some normal pattern of behavior.<br />
If this analytic exercise is to beg<strong>in</strong> to coalesce, nei<strong>the</strong>r expert must treat <strong>the</strong><br />
data set as an anomaly and both must report it as significant. In addition, each<br />
expert’s analysis of <strong>the</strong> data—an <strong>in</strong>crease <strong>in</strong> spend<strong>in</strong>g and <strong>the</strong> identification of<br />
a specific chemical agent—must be brought toge<strong>the</strong>r at some po<strong>in</strong>t. The problem<br />
is, at what po<strong>in</strong>t? Presumably, someone will get both of <strong>the</strong>se reports<br />
somewhere along <strong>the</strong> <strong>in</strong>telligence cha<strong>in</strong>. Of course, <strong>the</strong> <strong>in</strong>dividual who gets<br />
<strong>the</strong>se reports will be subject to <strong>the</strong> same three complicat<strong>in</strong>g variables<br />
described earlier—time context, pattern bias, and heuristic bias—and may not<br />
be able to syn<strong>the</strong>size <strong>the</strong> <strong>in</strong>formation. Thus, <strong>the</strong> burden of putt<strong>in</strong>g <strong>the</strong> pieces<br />
toge<strong>the</strong>r will merely have been shifted to someone else <strong>in</strong> <strong>the</strong> organization.<br />
In order to avoid shift<strong>in</strong>g <strong>the</strong> problem from one expert to ano<strong>the</strong>r, an actual<br />
collaborative team could be built. Why not explicitly put <strong>the</strong> economist and<br />
<strong>the</strong> chemist toge<strong>the</strong>r to work on analyz<strong>in</strong>g data? The utilitarian problems with<br />
this strategy are obvious: not all economic problems are chemical, and not all<br />
chemical problems are economic. Each expert would waste an <strong>in</strong>ord<strong>in</strong>ate<br />
amount of time. Perhaps one case <strong>in</strong> 100 would be applicable to both experts,<br />
but, dur<strong>in</strong>g <strong>the</strong> rest of <strong>the</strong> day, <strong>the</strong>y would drift back to <strong>the</strong>ir <strong>in</strong>dividual<br />
doma<strong>in</strong>s, <strong>in</strong> part, because that is what <strong>the</strong>y are best at and, <strong>in</strong> part, just to stay<br />
busy.<br />
Closer to <strong>the</strong> real world, <strong>the</strong> same example may also have social, political,<br />
historical, and cultural aspects. Despite an <strong>in</strong>crease <strong>in</strong> spend<strong>in</strong>g on a specific<br />
chemical agent, <strong>the</strong> country <strong>in</strong> question may not be <strong>in</strong>cl<strong>in</strong>ed to use it <strong>in</strong> a<br />
threaten<strong>in</strong>g way. For example, <strong>the</strong>re may be social data unavailable to <strong>the</strong><br />
economist or <strong>the</strong> chemist <strong>in</strong>dicat<strong>in</strong>g that <strong>the</strong> chemical agent will be used for a<br />
benign purpose. In order for collaboration to work, each team would have to<br />
have experts from many doma<strong>in</strong>s work<strong>in</strong>g toge<strong>the</strong>r on <strong>the</strong> same data set.<br />
Successful teams have very specific organizational and structural requirements.<br />
An effective team requires discrete and clearly stated goals that are<br />
69
CHAPTER FIVE<br />
shared by each team member. 23 Teams also require <strong>in</strong>terdependence and<br />
accountability, that is, <strong>the</strong> success of each <strong>in</strong>dividual depends on <strong>the</strong> success of<br />
<strong>the</strong> team as a whole as well as on <strong>the</strong> <strong>in</strong>dividual success of every o<strong>the</strong>r team<br />
member. 24<br />
Effective teams require cohesion, formal and <strong>in</strong>formal communication,<br />
cooperation, shared mental models, and similar knowledge structures. 25 Putt<strong>in</strong>g<br />
comb<strong>in</strong>ations such as this <strong>in</strong> place is not a trivial task. Creat<strong>in</strong>g shared<br />
mental models may be fairly easy with<strong>in</strong> an air crew or a tank crew, where an<br />
<strong>in</strong>dividual’s role is clearly identifiable as part of a clearly-def<strong>in</strong>ed, repetitive<br />
team effort, such as land<strong>in</strong>g a plane or acquir<strong>in</strong>g and fir<strong>in</strong>g on a target. It is<br />
more difficult with<strong>in</strong> an <strong>in</strong>telligence team, given <strong>the</strong> vague nature of <strong>the</strong> goals,<br />
<strong>the</strong> enormity of <strong>the</strong> task, and <strong>the</strong> diversity of <strong>in</strong>dividual expertise. Moreover,<br />
<strong>the</strong> larger <strong>the</strong> number of team members, <strong>the</strong> more difficult it is to generate<br />
cohesion, communication, and cooperation. Heterogeneity can also be a challenge;<br />
it has a positive effect on generat<strong>in</strong>g diverse viewpo<strong>in</strong>ts with<strong>in</strong> a team,<br />
but it requires more organizational structure than does a homogeneous team. 26<br />
Without specific processes, organiz<strong>in</strong>g pr<strong>in</strong>ciples, and operational structures,<br />
<strong>in</strong>terdiscipl<strong>in</strong>ary teams will quickly revert to be<strong>in</strong>g simply a room full of<br />
experts who ultimately drift back to <strong>the</strong>ir previous work patterns. That is, <strong>the</strong><br />
experts will not be a team at all; <strong>the</strong>y will be a group of experts <strong>in</strong>dividually<br />
work<strong>in</strong>g <strong>in</strong> some general problem space. 27<br />
23<br />
Dorw<strong>in</strong> Cartwright and Alv<strong>in</strong> Zander, Group Dynamics: Research and Theory; P. Fandt, W.<br />
Richardson, and H. Conner, “The Impact of Goal Sett<strong>in</strong>g on Team Simulation Experience”; J.<br />
Harvey and C. Boettger, “Improv<strong>in</strong>g Communication with<strong>in</strong> a Managerial Workgroup.”<br />
24<br />
M. Deutsch, “The Effects of Cooperation and Competition Upon Group Process”; D. Johnson<br />
and R. Johnson, “The Internal Dynamics of Cooperative Learn<strong>in</strong>g Groups”; D. Cartwright and A.<br />
Zander, Group Dynamics: Research and Theory; David Johnson and Roger Johnson, “The Internal<br />
Dynamics of Cooperative Learn<strong>in</strong>g Groups”; D. Johnson et al., “Effects of Cooperative, Competitive,<br />
and Individualistic Goal Structure on Achievement: A Meta-Analysis”; R. Slav<strong>in</strong>,<br />
“Research on Cooperative Learn<strong>in</strong>g”; R. Slav<strong>in</strong>, Cooperative Learn<strong>in</strong>g.<br />
25<br />
J. Cannon-Bowers, E. Salas, S. Converse, “Shared Mental Models <strong>in</strong> Expert Team Decision<br />
Mak<strong>in</strong>g”; L. Coch and J. French, “Overcom<strong>in</strong>g Resistance to Change”; M. Deutsch, “The Effects<br />
of Cooperation and Competition Upon Group Process”; L. Fest<strong>in</strong>ger, “Informal Social Communication”;<br />
D. Johnson et al., “The Impact of Positive Goal and Resource Interdependence on<br />
Achievement, Interaction, and Attitudes”; B. Mullen and C. Copper, “The Relation Between<br />
Group Cohesiveness and Performance: An Integration”; W. Nijhof and P. Kommers, “An Analysis<br />
of Cooperation <strong>in</strong> Relation to Cognitive Controversy”; J. Orasanu, “Shared Mental Models<br />
and Crew Performance”; S. Seashore, Group Cohesiveness <strong>in</strong> <strong>the</strong> Industrial Work-group.<br />
26<br />
T. Mills, “Power Relations <strong>in</strong> Three-Person Groups”; L. Molm, “L<strong>in</strong>k<strong>in</strong>g Power Structure and<br />
Power Use”; V. Nieva, E. Fleishman, and A. Rieck, Team Dimensions: Their Identity, Their Measurement,<br />
and Their Relationships; G. Simmel, The Sociology of Georg Simmel.<br />
27<br />
R. Johnston, Decision Mak<strong>in</strong>g and Performance Error <strong>in</strong> Teams: Research Results; J. Meister,<br />
“Individual Perceptions of Team Learn<strong>in</strong>g Experiences Us<strong>in</strong>g Video-Based or Virtual Reality<br />
Environments.”<br />
70
INTEGRATING METHODOLOGISTS<br />
Can Technology Help?<br />
There are potential technological alternatives to multifaceted teams. For<br />
example, an Electronic Performance Support System (EPSS) is a large database<br />
that is used <strong>in</strong> conjunction with expert systems, <strong>in</strong>telligent agents, and<br />
decision aids. 28 Although apply<strong>in</strong>g such a system to <strong>in</strong>telligence problems<br />
might be a useful goal, at present, <strong>the</strong> notion of an <strong>in</strong>tegrated EPSS for large<br />
complex data sets is more <strong>the</strong>ory than practice. 29 In addition to questions<br />
about <strong>the</strong> technological feasibility of such a system, <strong>the</strong>re are fundamental<br />
epistemological challenges. It is virtually <strong>in</strong>conceivable that a comprehensive<br />
computational system could bypass <strong>the</strong> three complicat<strong>in</strong>g variables of expertise<br />
described earlier.<br />
An EPSS, or any o<strong>the</strong>r computational solution, is designed, programmed,<br />
and implemented by a human expert from one doma<strong>in</strong> only, that of computer<br />
science. Historians will not design <strong>the</strong> “historical decision aid,” economists<br />
will not program <strong>the</strong> “economic <strong>in</strong>telligent agent,” chemists will not create <strong>the</strong><br />
“chemical agent expert system.” Computer scientists may consult with various<br />
experts dur<strong>in</strong>g <strong>the</strong> design phase of such a system, but, when it is time to sit<br />
down and write code, <strong>the</strong> programmer will follow <strong>the</strong> heuristics with which he<br />
is familiar. 30 In essence, one would be trad<strong>in</strong>g <strong>the</strong> heuristics of dozens of<br />
doma<strong>in</strong>s for those that govern computer science. This would reduce <strong>the</strong> problem<br />
of process<strong>in</strong>g time by simplify<strong>in</strong>g and l<strong>in</strong>k<strong>in</strong>g data, and it might reduce<br />
pattern bias. It would not reduce heuristic bias, however; if anyth<strong>in</strong>g, it might<br />
exaggerate it by reduc<strong>in</strong>g all data to a b<strong>in</strong>ary state. 31<br />
This skepticism is not simply a Luddite reaction to technology. Computational<br />
systems have had a remarkable, positive effect on process<strong>in</strong>g time, storage,<br />
and retrieval. They have also demonstrated utility <strong>in</strong> identify<strong>in</strong>g patterns<br />
with<strong>in</strong> narrowly def<strong>in</strong>ed doma<strong>in</strong>s. However, <strong>in</strong>telligence analysis requires <strong>the</strong><br />
expertise of so many diverse fields of study and is not someth<strong>in</strong>g a computational<br />
system handles well. Although an EPSS, or some o<strong>the</strong>r form of computational<br />
system, may be a useful tool for manipulat<strong>in</strong>g data, it is not a solution<br />
to <strong>the</strong> paradox of expertise.<br />
28<br />
An Expert System is a job-specific heuristic process that helps an expert narrow <strong>the</strong> range of<br />
available choices. An Intelligent Agent is an automated program (bot) with built-<strong>in</strong> heuristics used<br />
<strong>in</strong> Web searches. A Decision Aid is an expert system whose scope is limited to a particular task.<br />
29<br />
R. Johnston, “Electronic Performance Support Systems and Information Navigation.”<br />
30<br />
R. Johnston and J. Fletcher, A Meta-Analysis of <strong>the</strong> Effectiveness of Computer-Based Tra<strong>in</strong><strong>in</strong>g<br />
for Military Instruction.<br />
31<br />
J. Fletcher and R. Johnston, “Effectiveness and Cost Benefits of Computer-Based Decision<br />
Aids for Equipment Ma<strong>in</strong>tenance.”<br />
71
CHAPTER FIVE<br />
<strong>Analytic</strong> Methodologists<br />
Most doma<strong>in</strong>s have specialists who study <strong>the</strong> scientific process or research<br />
methods of <strong>the</strong>ir discipl<strong>in</strong>e. Instead of specializ<strong>in</strong>g <strong>in</strong> a specific substantive<br />
topic, <strong>the</strong>se experts specialize <strong>in</strong> master<strong>in</strong>g <strong>the</strong> research and analytic methods<br />
of <strong>the</strong>ir doma<strong>in</strong>. In <strong>the</strong> biological and medical fields, <strong>the</strong>se methodological<br />
specialists are epidemiologists. In education and public policy, <strong>the</strong>y are program<br />
evaluators. In o<strong>the</strong>r fields, <strong>the</strong>y are research methodologists or statisticians.<br />
Whatever <strong>the</strong> label, each field recognizes that it requires experts <strong>in</strong><br />
methodology who focus on deriv<strong>in</strong>g mean<strong>in</strong>g from data, recogniz<strong>in</strong>g patterns,<br />
and solv<strong>in</strong>g problems with<strong>in</strong> a doma<strong>in</strong> <strong>in</strong> order to ma<strong>in</strong>ta<strong>in</strong> and pass on <strong>the</strong><br />
doma<strong>in</strong>’s heuristics. They become <strong>in</strong>-house consultants—organiz<strong>in</strong>g agents—<br />
who work to identify research designs, methods for choos<strong>in</strong>g samples, and<br />
tools for data analysis.<br />
Because <strong>the</strong>y have a different perspective than do <strong>the</strong> experts <strong>in</strong> a doma<strong>in</strong>,<br />
methodologists are often called on by substantive experts to advise <strong>the</strong>m on a<br />
variety of process issues. On any given day, an epidemiologist, for example,<br />
may be asked to consult on studies of <strong>the</strong> effects of alcoholism or <strong>the</strong> spread of<br />
a virus on a community or to review a double-bl<strong>in</strong>d cl<strong>in</strong>ical trial of a new<br />
pharmaceutical product. In each case, <strong>the</strong> epidemiologist is not be<strong>in</strong>g asked<br />
about <strong>the</strong> content of <strong>the</strong> study; ra<strong>the</strong>r, he is be<strong>in</strong>g asked to comment on <strong>the</strong><br />
research methods and data analysis techniques used.<br />
Although well over 160 analytic methods are available to <strong>in</strong>telligence analyst,<br />
few methods specific to <strong>the</strong> doma<strong>in</strong> of <strong>in</strong>telligence analysis exist. 32 <strong>Intelligence</strong><br />
analysis has few specialists whose professional tra<strong>in</strong><strong>in</strong>g is <strong>in</strong> <strong>the</strong><br />
process of employ<strong>in</strong>g and unify<strong>in</strong>g <strong>the</strong> analytic practices with<strong>in</strong> <strong>the</strong> field. It is<br />
left to <strong>the</strong> <strong>in</strong>dividual analysts to know how to apply methods, select one<br />
method over ano<strong>the</strong>r, weigh disparate variables, and syn<strong>the</strong>size <strong>the</strong> results—<br />
<strong>the</strong> same analysts whose expertise is conf<strong>in</strong>ed to specific substantive areas and<br />
<strong>the</strong>ir own doma<strong>in</strong>s’ heuristics.<br />
Conclusion<br />
<strong>Intelligence</strong> agencies cont<strong>in</strong>ue to experiment with <strong>the</strong> right composition,<br />
structure, and organization of analytic teams. Yet, although <strong>the</strong>y budget significant<br />
resources for technological solutions, comparatively little is be<strong>in</strong>g<br />
32<br />
Exceptions <strong>in</strong>clude: S. Feder, “FACTIONS and Policon”; R. Heuer, Psychology of <strong>Intelligence</strong><br />
Analysis; R. Hopk<strong>in</strong>s, Warn<strong>in</strong>gs of Revolution: A Case Study of El Salvador; J. Lockwood and K.<br />
Lockwood, “The Lockwood <strong>Analytic</strong>al Method for Prediction (LAMP)”; J. Pierce, “Some Ma<strong>the</strong>matical<br />
Methods for <strong>Intelligence</strong> Analysis”; E. Sapp, “Decision Trees”; J. Zlotnick, “Bayes’<br />
Theorem for <strong>Intelligence</strong> Analysis.”<br />
72
INTEGRATING METHODOLOGISTS<br />
done to advance methodological science. Methodological improvements are<br />
left primarily to <strong>the</strong> <strong>in</strong>dividual doma<strong>in</strong>s, a practice that risks fall<strong>in</strong>g <strong>in</strong>to <strong>the</strong><br />
same paradoxical trap that currently exists. What is needed is an <strong>in</strong>telligencecentric<br />
approach to methodology that will <strong>in</strong>clude <strong>the</strong> methods and procedures<br />
of many doma<strong>in</strong>s and <strong>the</strong> development of heuristics and techniques unique to<br />
<strong>in</strong>telligence. In short, <strong>in</strong>telligence analysis needs its own analytic heuristics<br />
that are designed, developed, and tested by professional analytic methodologists.<br />
The desired outcome would be a comb<strong>in</strong>ed approach that <strong>in</strong>cludes formal<br />
<strong>the</strong>matic teams with structured organizational pr<strong>in</strong>ciples, technological systems<br />
designed with significant <strong>in</strong>put from doma<strong>in</strong> experts, and a cadre of analytic<br />
methodologists. These methodologists would act as <strong>in</strong>-house consultants<br />
for analytic teams, generate new methods specific to <strong>in</strong>telligence analysis,<br />
modify and improve exist<strong>in</strong>g methods of analysis, and promote <strong>the</strong> professionalization<br />
of <strong>the</strong> discipl<strong>in</strong>e of <strong>in</strong>telligence. Although, at first, develop<strong>in</strong>g a<br />
cadre of analytic methodologists would require us<strong>in</strong>g specialists from a variety<br />
of o<strong>the</strong>r doma<strong>in</strong>s and professional associations, <strong>in</strong> time, <strong>the</strong> discipl<strong>in</strong>e<br />
would mature <strong>in</strong>to its own subdiscipl<strong>in</strong>e with its own measures of validity and<br />
reliability.<br />
73
CHAPTER SIX<br />
The Question of Foreign <strong>Culture</strong>s: Combat<strong>in</strong>g<br />
Ethnocentrism <strong>in</strong> <strong>Intelligence</strong> Analysis<br />
The <strong>in</strong>telligence literature often cautions <strong>in</strong>telligence professionals to be<br />
wary of mirror imag<strong>in</strong>g. 1 Although <strong>the</strong> term is a misnomer (a mirror image is<br />
a reverse image), <strong>the</strong> concept is that <strong>in</strong>dividuals perceive foreigners—both<br />
friends and adversaries of <strong>the</strong> United States—as th<strong>in</strong>k<strong>in</strong>g <strong>the</strong> same way as<br />
Americans. 2 Individuals do, <strong>in</strong> fact, have a natural tendency to assume that<br />
o<strong>the</strong>rs th<strong>in</strong>k and perceive <strong>the</strong> world <strong>in</strong> <strong>the</strong> same way <strong>the</strong>y do. This type of projective<br />
identification, or ethnocentrism, is <strong>the</strong> consequence of a comb<strong>in</strong>ation<br />
of cognitive and cultural biases result<strong>in</strong>g from a lifetime of enculturation, culturally<br />
bound heuristics, and miss<strong>in</strong>g, or <strong>in</strong>adequate, <strong>in</strong>formation. 3<br />
Ethnocentrism is a phenomenon that operates on a conscious level, but it is<br />
difficult to recognize <strong>in</strong> oneself and equally difficult to counteract. In part, this<br />
is because, <strong>in</strong> cases of ethnocentric th<strong>in</strong>k<strong>in</strong>g, an <strong>in</strong>dividual does not recognize<br />
that important <strong>in</strong>formation is miss<strong>in</strong>g or, more important, that his worldview<br />
and problem-solv<strong>in</strong>g heuristics <strong>in</strong>terfere with <strong>the</strong> process of recogniz<strong>in</strong>g <strong>in</strong>formation<br />
that conflicts or refutes his assumptions.<br />
Take, for example, <strong>the</strong> proposition that o<strong>the</strong>rs do not th<strong>in</strong>k like Americans.<br />
It seems only <strong>in</strong>tuitive that o<strong>the</strong>r tribes, ethnic groups, nationalities, and states<br />
1<br />
Alexander Butterfield, The Accuracy of <strong>Intelligence</strong> Assessment; Richards J. Heuer, Jr., Psychology<br />
of <strong>Intelligence</strong> Analysis; Lisa Krizan, <strong>Intelligence</strong> Essentials for Everyone; J. R. Thompson,<br />
R. Hopf-Weichel, and R. Geiselman, The Cognitive Bases of <strong>Intelligence</strong> Analysis.<br />
2<br />
In this work, I use <strong>the</strong> broader term “ethnocentrism” to refer to <strong>the</strong> concept represented by mirror<br />
imag<strong>in</strong>g and projective identification .<br />
3<br />
In anthropology, ethnocentrism is <strong>the</strong> tendency to judge <strong>the</strong> customs of o<strong>the</strong>r societies by <strong>the</strong><br />
standards of one's own culture. This <strong>in</strong>cludes project<strong>in</strong>g one’s own cognition and norms onto o<strong>the</strong>rs.<br />
75
CHAPTER SIX<br />
have different histories, languages, customs, educational practices, and cultures<br />
and, <strong>the</strong>refore, must th<strong>in</strong>k differently from one ano<strong>the</strong>r.<br />
The problem, however, is that <strong>the</strong> cognitive process of understand<strong>in</strong>g or<br />
even recogniz<strong>in</strong>g that <strong>the</strong>re are cultural and cognitive differences is not <strong>in</strong>tuitive<br />
at all. Intuition is <strong>the</strong> act of immediate cognition, that is, perceiv<strong>in</strong>g someth<strong>in</strong>g<br />
directly through <strong>the</strong> use of culturally dependent heuristics and cognitive<br />
patterns accumulated through a lifetime without requir<strong>in</strong>g <strong>the</strong> use of rational<br />
or formal processes. This effort appears doomed to failure, because “try<strong>in</strong>g to<br />
th<strong>in</strong>k like <strong>the</strong>m” all too often results <strong>in</strong> apply<strong>in</strong>g <strong>the</strong> logic of one’s own culture<br />
and experience to try to understand <strong>the</strong> actions of o<strong>the</strong>rs, without know<strong>in</strong>g that<br />
one is us<strong>in</strong>g <strong>the</strong> logic of one’s own culture. This, however, does not have to be<br />
<strong>the</strong> case. Through acculturation and <strong>the</strong> use of specific strategies, tools, and<br />
techniques, it is possible to combat <strong>the</strong> effects of ethnocentrism without try<strong>in</strong>g<br />
to “th<strong>in</strong>k like <strong>the</strong>m.” This text <strong>in</strong>cludes two short case studies on failures to<br />
recognize ethnocentrism, both drawn from <strong>the</strong> author’s own experience and<br />
told from his perspective. These failures are <strong>the</strong>n exam<strong>in</strong>ed with <strong>the</strong> goal of<br />
develop<strong>in</strong>g strategies and techniques to combat ethnocentric bias.<br />
Case Study One: Tiananmen Square<br />
At <strong>the</strong> time of <strong>the</strong> prodemocracy protests of <strong>the</strong> Ch<strong>in</strong>ese students and, to a<br />
lesser extent, workers, between April and June of 1989, I too was a college<br />
student. I mention this because American college students and Ch<strong>in</strong>ese college<br />
students tend to perceive <strong>the</strong>mselves <strong>in</strong> very different ways, and <strong>the</strong>y are<br />
perceived by <strong>the</strong>ir societies as hav<strong>in</strong>g very different social roles. Ch<strong>in</strong>ese students<br />
perceive <strong>the</strong>mselves as hav<strong>in</strong>g moral authority, and <strong>the</strong>y are perceived<br />
as controll<strong>in</strong>g social capital and possess<strong>in</strong>g public status. There is a cultural<br />
norm <strong>in</strong> Ch<strong>in</strong>a that students, as <strong>the</strong> future elite, have a morally superior role <strong>in</strong><br />
society. I remember th<strong>in</strong>k<strong>in</strong>g at <strong>the</strong> time that, with <strong>the</strong> obvious exception of<br />
those <strong>in</strong> power, who risked los<strong>in</strong>g <strong>the</strong>ir privileged positions, any “rightm<strong>in</strong>ded”<br />
person <strong>in</strong> Ch<strong>in</strong>a would support democracy. A movement for democratic<br />
reform would liberalize <strong>the</strong> policies of a repressive regime, encourage<br />
personal freedom, and give <strong>the</strong> Ch<strong>in</strong>ese people a voice <strong>in</strong> <strong>the</strong>ir lives.<br />
When <strong>the</strong> university students went on strike and took over Tiananmen<br />
Square, <strong>the</strong> popular view <strong>in</strong> <strong>the</strong> United States, reflected <strong>in</strong> <strong>the</strong> US media, was<br />
that <strong>the</strong>y were college students protest<strong>in</strong>g for democratic reform. There were<br />
images of thousands of students rally<strong>in</strong>g and camp<strong>in</strong>g out on and around <strong>the</strong><br />
statue of <strong>the</strong> People’s Heroes. Throughout <strong>the</strong> square, banners and posters<br />
from universities supported democracy and freedom. The statue of <strong>the</strong> Goddess<br />
of Democracy erected by <strong>the</strong> demonstrators looked very much like our<br />
Statue of Liberty. Labor groups offered to jo<strong>in</strong> <strong>the</strong> students, people paraded <strong>in</strong><br />
front of <strong>the</strong> Great Hall of <strong>the</strong> People, and citizens donated blankets and food.<br />
76
COMBATING ETHNOCENTRISM<br />
Student leaders began a hunger strike to force a dialogue between <strong>the</strong> students<br />
and <strong>the</strong> government. All signs seemed clearly to po<strong>in</strong>t to a popular movement<br />
for democracy, for which <strong>the</strong>re was a groundswell of support.<br />
The Ch<strong>in</strong>ese government seemed hesitant or unsure. The People’s Liberation<br />
Army (PLA) was sent to surround <strong>the</strong> square, but citizens blocked <strong>the</strong>ir<br />
advance and tried to persuade <strong>the</strong> troops to be neutral. A curfew order was not<br />
obeyed; martial law was declared and ignored. Ano<strong>the</strong>r PLA move on Tiananmen<br />
Square was repelled. It appeared that <strong>the</strong> students had forced a stalemate<br />
and that <strong>the</strong>ir demands would be heard.<br />
At that po<strong>in</strong>t, my assumption was that <strong>the</strong> government was weakened and<br />
would be forced to respond to <strong>the</strong> protesters’ demands, at least to some degree.<br />
I anticipated a dialogue and concessions on both sides. Although I imag<strong>in</strong>ed<br />
<strong>the</strong> government was capable of resort<strong>in</strong>g to violence, I assumed that it would<br />
not. It seemed <strong>in</strong>conceivable that <strong>the</strong> citizens of Beij<strong>in</strong>g—10–12 million people—would<br />
not <strong>in</strong>tervene on behalf of <strong>the</strong> students. That many people could<br />
have overwhelmed <strong>the</strong> PLA had <strong>the</strong>y chosen to do so. I also assumed that <strong>the</strong><br />
soldiers of <strong>the</strong> PLA would be reluctant to fire on <strong>the</strong>ir own people, partly<br />
because <strong>the</strong> majority of both groups were from <strong>the</strong> same, dom<strong>in</strong>ant ethnic<br />
group of Ch<strong>in</strong>a, <strong>the</strong> Han, and, <strong>in</strong> part, because <strong>the</strong> soldiers represented a lower<br />
rung of Ch<strong>in</strong>ese society <strong>the</strong>n did <strong>the</strong> students. The notion of soldiers kill<strong>in</strong>g<br />
students would be an affront to <strong>the</strong> sensibilities of <strong>the</strong> Han, or so I thought. I<br />
was wrong.<br />
In <strong>the</strong> end, when <strong>the</strong> PLA carried out its orders to clear <strong>the</strong> square with<br />
force and end <strong>the</strong> protest, support for <strong>the</strong> protesters turned out to be relatively<br />
slight. The Ch<strong>in</strong>ese “middle class” never came to <strong>the</strong> students’ aid; <strong>the</strong> great<br />
majority of <strong>the</strong> Beij<strong>in</strong>g populace simply watched <strong>the</strong> events unfold. Moreover,<br />
it turned out that <strong>the</strong> labor groups participat<strong>in</strong>g <strong>in</strong> <strong>the</strong> demonstration were<br />
actually protest<strong>in</strong>g aga<strong>in</strong>st corporate corruption and <strong>the</strong> lack of job stability<br />
brought about by market reforms and not <strong>in</strong> support of <strong>the</strong> students’ demands<br />
for a loosen<strong>in</strong>g of restrictions on expression. What I perceived to be a groundswell<br />
of popular support for <strong>the</strong> students had been exaggerated and wishful<br />
th<strong>in</strong>k<strong>in</strong>g on my part.<br />
My failure to anticipate <strong>the</strong> way events would actual unfold <strong>in</strong> Tiananmen<br />
Square was tied to ethnocentric th<strong>in</strong>k<strong>in</strong>g and a lack of accurate and contextual<br />
<strong>in</strong>formation. Students <strong>in</strong> <strong>the</strong> United States are encouraged to be politically<br />
active, and <strong>the</strong>ir protests are often seen merely as m<strong>in</strong>or <strong>in</strong>conveniences that<br />
need to be endured. In Ch<strong>in</strong>a, however, <strong>the</strong> protest<strong>in</strong>g students were seen as a<br />
direct challenge to political authority and, much more so than <strong>in</strong> <strong>the</strong> United<br />
States, <strong>the</strong>ir actions were viewed as an outright conflict between <strong>the</strong> future<br />
elite and <strong>the</strong> current leadership. The protest itself was viewed as a violation of<br />
a taboo, upsett<strong>in</strong>g <strong>the</strong> cultural order and <strong>the</strong> stability of society.<br />
77
CHAPTER SIX<br />
As an observer, I missed <strong>the</strong> cultural context that was necessary to view <strong>the</strong><br />
events as an actual conflict and could not conv<strong>in</strong>ce myself that a violent solution<br />
was a possibility. I had discounted <strong>the</strong> hypo<strong>the</strong>sis that violence would<br />
occur, because I could not imag<strong>in</strong>e it occurr<strong>in</strong>g <strong>in</strong> <strong>the</strong> United States. This led<br />
me to discount raw data that would have refuted a hypo<strong>the</strong>sis that <strong>the</strong> two factions<br />
would reach a compromise. In addition, at that time, I had no formal<br />
ground<strong>in</strong>g <strong>in</strong> Ch<strong>in</strong>ese studies, nor had I been to Ch<strong>in</strong>a. Thus, I had not<br />
acquired <strong>in</strong>formation that would have helped me create a mean<strong>in</strong>gful context<br />
for <strong>the</strong> event.<br />
Years later, my wife and I were <strong>in</strong> Ch<strong>in</strong>a do<strong>in</strong>g ethnographic fieldwork on<br />
<strong>the</strong> socioeconomic effects of <strong>the</strong> spread of <strong>the</strong> English language and American<br />
culture <strong>in</strong> urban and rural Ch<strong>in</strong>a. 4 While <strong>the</strong>re, we spent a great deal of time<br />
talk<strong>in</strong>g with o<strong>the</strong>rs about <strong>the</strong> events of Tiananmen, and we decided to <strong>in</strong>clude<br />
<strong>in</strong> our research questions about <strong>the</strong> student protests, if for no o<strong>the</strong>r reason than<br />
to satisfy our own curiosity.<br />
What we found stood <strong>in</strong> contrast to media reports and <strong>the</strong> op<strong>in</strong>ions<br />
expressed by many pundits and scholars <strong>in</strong> <strong>the</strong> US and <strong>the</strong> West. After hundreds<br />
of <strong>in</strong>terviews with a wide variety of people <strong>in</strong> and around Beij<strong>in</strong>g, we<br />
found a consistent preoccupation among <strong>the</strong> “silent majority.” That was <strong>the</strong><br />
Cultural Revolution, which had affected all of <strong>the</strong> people we <strong>in</strong>terviewed.<br />
They had been participants, observers, or survivors, and, often, all three.<br />
In <strong>the</strong> mid-1960s, Mao Zedong sought to recapture power from reformm<strong>in</strong>ded<br />
opponents with<strong>in</strong> <strong>the</strong> Communist Party. Us<strong>in</strong>g radical party leaders as<br />
his <strong>in</strong>struments, he created <strong>the</strong> Red Guard, which was made up primarily of<br />
college students (although o<strong>the</strong>rs followed suit <strong>in</strong> time). The image of <strong>the</strong> Cultural<br />
Revolution was not simply <strong>the</strong> image of Mao; it was also <strong>the</strong> image of<br />
angry, violent, and powerful college students, who were <strong>the</strong> most visible proponents<br />
of <strong>the</strong> “Cult of Mao.” Accord<strong>in</strong>g to <strong>the</strong> people we <strong>in</strong>terviewed, it was<br />
<strong>the</strong> students who had chanted slogans, raised banners, paraded <strong>in</strong> public<br />
spaces, resisted older forms of social control, and seized power. With that<br />
power and <strong>the</strong> bless<strong>in</strong>gs of Mao, <strong>the</strong> youth and university students had committed<br />
many of <strong>the</strong> atrocities of <strong>the</strong> Cultural Revolution and plunged Ch<strong>in</strong>a<br />
<strong>in</strong>to a decade of chaos, dur<strong>in</strong>g which many <strong>in</strong>stitutions, <strong>in</strong>clud<strong>in</strong>g schools,<br />
were closed and many of <strong>the</strong> country’s cultural and historical artifacts were<br />
destroyed.<br />
4<br />
American anthropology is based on <strong>the</strong> ethnographic method and direct <strong>in</strong>teraction with <strong>the</strong> people<br />
who are be<strong>in</strong>g studied. This <strong>in</strong>teraction <strong>in</strong>cludes direct and participant observation and <strong>in</strong>terviews,<br />
or fieldwork, where one lives with <strong>the</strong> people be<strong>in</strong>g <strong>in</strong>vestigated. Cont<strong>in</strong>ental European<br />
schools of anthropology are not as obsessed with methodology and hands-on experience and tend<br />
to <strong>the</strong> more <strong>the</strong>oretical.<br />
78
COMBATING ETHNOCENTRISM<br />
At <strong>the</strong> height of <strong>the</strong> Cultural Revolution, any dissent was sufficient to br<strong>in</strong>g<br />
accusations of counterrevolutionary sympathies and to qualify one for “reeducation,”<br />
which could mean public denunciation, job loss, <strong>in</strong>carceration,<br />
forced labor, relocation, and even murder, torture, and rape. The traditional<br />
values of respect and honor were replaced with violence and terror, and <strong>the</strong><br />
historical social unit of <strong>the</strong> family had been disrupted and replaced with <strong>the</strong><br />
cult of Mao.<br />
For those who had lived through <strong>the</strong> Cultural Revolution, <strong>the</strong> student challenge<br />
to <strong>the</strong> government <strong>in</strong> Tiananmen <strong>in</strong> 1989 was also a challenge to social<br />
order and stability. The people we <strong>in</strong>terviewed remembered, correctly or not,<br />
that <strong>the</strong> faction of <strong>the</strong> Communist Party <strong>the</strong>n <strong>in</strong> power and <strong>the</strong> PLA had<br />
stopped <strong>the</strong> Red Guard and <strong>the</strong> Cultural Revolution, arrested its highest rank<strong>in</strong>g<br />
proponents and beneficiaries, <strong>the</strong> Gang of Four, and eventually restored<br />
order to <strong>the</strong> nation. The po<strong>in</strong>t of view of <strong>the</strong> people we <strong>in</strong>terviewed was that<br />
<strong>the</strong> PLA, despite <strong>the</strong> low social status of soldiers, had stopped <strong>the</strong> chaos.<br />
Although <strong>the</strong>y did not approve of kill<strong>in</strong>g students, <strong>the</strong> threat of ano<strong>the</strong>r cultural<br />
revolution, democratic or o<strong>the</strong>rwise, was more disturb<strong>in</strong>g to <strong>the</strong>m than<br />
<strong>the</strong> bloody climax <strong>in</strong> <strong>the</strong> square. Social order was <strong>the</strong> higher virtue.<br />
Tiananmen Square: Discussion<br />
We cut nature up, organize it <strong>in</strong>to concepts, and ascribe significances<br />
as we do, largely because we are parties to an agreement to<br />
organize it <strong>in</strong> this way—an agreement that holds throughout our<br />
speech community and is codified <strong>in</strong> <strong>the</strong> patterns of our language.<br />
Benjam<strong>in</strong> Whorf 5<br />
In 1987, a Ch<strong>in</strong>ese academic, M<strong>in</strong> Qi, performed <strong>the</strong> first national survey of<br />
Ch<strong>in</strong>ese political culture. 6 Respondents were asked, among o<strong>the</strong>r th<strong>in</strong>gs, to<br />
select statements that best described <strong>the</strong>ir understand<strong>in</strong>g of democracy. Of <strong>the</strong><br />
1,373 respondents, 6.6 percent responded that democracy meant that people<br />
could elect <strong>the</strong>ir political leaders and 3.4 percent that power was limited and<br />
divided. These replies tended to be from <strong>in</strong>dividuals under 25 years of age, <strong>in</strong><br />
college, and liv<strong>in</strong>g <strong>in</strong> urban centers.<br />
5<br />
Benjam<strong>in</strong> Whorf, along with fellow anthropologist Edward Sapir, developed <strong>the</strong> l<strong>in</strong>guistic relativity<br />
hypo<strong>the</strong>sis, assert<strong>in</strong>g that different speech communities had different patterns of thought.<br />
Although challenged by l<strong>in</strong>guist/philosopher Noam Chomsky and o<strong>the</strong>rs with <strong>the</strong> Universal<br />
Grammar hypo<strong>the</strong>sis, l<strong>in</strong>guistic relativity still has a significant amount of empirical research support.<br />
Benjam<strong>in</strong> Whorf, Language, Thought, and Reality.<br />
6<br />
M<strong>in</strong> Qi, Zhongguo Zhengzhi Wenhua [Ch<strong>in</strong>ese Political <strong>Culture</strong>]. Translation courtesy of a<br />
friend of <strong>the</strong> author who prefers to rema<strong>in</strong> anonymous.<br />
79
CHAPTER SIX<br />
In contrast, 25 percent responded that democracy was guided by <strong>the</strong> center<br />
(<strong>the</strong> party and <strong>the</strong> cadres), 19.5 percent that democracy meant that <strong>the</strong> government<br />
would solicit people’s op<strong>in</strong>ions (<strong>the</strong> party would ask people what <strong>the</strong>y<br />
thought), and 11 percent that democracy meant <strong>the</strong> government would make<br />
decisions for <strong>the</strong> people based on <strong>the</strong> people’s <strong>in</strong>terests but not <strong>in</strong>clud<strong>in</strong>g <strong>the</strong><br />
people’s direct vote. These three responses were more <strong>in</strong> l<strong>in</strong>e with <strong>the</strong>n-current<br />
party doctr<strong>in</strong>e and tended to be from <strong>in</strong>dividuals over 36 years of age liv<strong>in</strong>g <strong>in</strong><br />
both urban and rural sett<strong>in</strong>gs. This was <strong>the</strong> same demographic that experienced<br />
<strong>the</strong> Cultural Revolution.<br />
The election of representatives and <strong>the</strong> division and limitation of those representatives’<br />
power—what I would have considered to be two key aspects of<br />
democracy—were chosen by 10 percent of <strong>the</strong> sample, only slightly larger<br />
than <strong>the</strong> 6.3 percent of Ch<strong>in</strong>ese respondents who reported that <strong>the</strong>y didn’t<br />
know what <strong>the</strong> word “democracy” meant. My own perception of democracy<br />
fit with a young, urban, elite, college educated population, not with <strong>the</strong> majority<br />
of Ch<strong>in</strong>ese citizens.<br />
There was a very small sample of citizens <strong>in</strong> Tiananmen Square demand<strong>in</strong>g<br />
what looked and sounded like my American version of democracy. Yet, however<br />
much <strong>the</strong> students’ message resonated <strong>in</strong> <strong>the</strong> West, it did not do so <strong>in</strong><br />
Ch<strong>in</strong>a. My expectations notwithstand<strong>in</strong>g, <strong>the</strong>re was a cognitive disconnect<br />
between students and average citizens, which, along with <strong>the</strong> visceral semiotics<br />
of <strong>the</strong> Cultural Revolution, kept <strong>the</strong> two apart. 7 It was not just <strong>the</strong> message<br />
that had kept people <strong>in</strong> <strong>the</strong>ir homes dur<strong>in</strong>g <strong>the</strong> PLA siege on Tiananmen; it<br />
was also <strong>the</strong> messengers.<br />
The label “ethnocentrism” might be accurate, but it does not diagnose <strong>the</strong><br />
root of <strong>the</strong> problem. I did not use a variety of tools or techniques to question<br />
my underly<strong>in</strong>g assumptions and, <strong>the</strong>refore, I failed to make an accurate forecast.<br />
There were obvious statistical and analytic flaws. The former was pr<strong>in</strong>cipally<br />
a sampl<strong>in</strong>g error, both frame and selection bias (<strong>the</strong> students at<br />
Tiananmen did not represent <strong>the</strong> general population <strong>in</strong> Beij<strong>in</strong>g or Ch<strong>in</strong>a at<br />
large). More significant than simple technical or statistical flaws, however, my<br />
frame of reference and my assumptions about mean<strong>in</strong>gs, context, and values<br />
(or culture) misled me.<br />
The assumptions I made about <strong>the</strong> Tiananmen protests were products of my<br />
own enculturation, and I am not conv<strong>in</strong>ced that anyth<strong>in</strong>g short of <strong>the</strong> experience<br />
of analytic failure would have been sufficient for me to exam<strong>in</strong>e <strong>the</strong> process<br />
underp<strong>in</strong>n<strong>in</strong>g my reason<strong>in</strong>g. I never would have reexam<strong>in</strong>ed my mental<br />
mode without experienc<strong>in</strong>g failure. Failure is an event that is easily remem-<br />
7<br />
Semiosis is <strong>the</strong> production of cultural signifiers or signs and <strong>the</strong> cultural or contextual mean<strong>in</strong>g<br />
of those signs. This <strong>in</strong>cludes all modes of visual and auditory production.<br />
80
COMBATING ETHNOCENTRISM<br />
bered; it affects <strong>the</strong> ego and drives one to <strong>in</strong>vestigate errors and to adapt or<br />
change behavior based on those <strong>in</strong>vestigations. Failure is a learn<strong>in</strong>g event and<br />
results <strong>in</strong> a teachable moment. 8<br />
There seems to be little reason to perform a postmortem when events unfold<br />
as predicted. The natural assumption is that <strong>the</strong> mechanisms of analysis were<br />
valid, because <strong>the</strong> results of <strong>the</strong> analysis were accurate. The obvious danger is<br />
that this assumption discounts <strong>the</strong> possibility that one may be accurate purely<br />
by accident. Moreover, by focus<strong>in</strong>g only on failure, one risks sampl<strong>in</strong>g bias by<br />
only choos<strong>in</strong>g cases <strong>in</strong> which <strong>the</strong>re was error. The risk of ignor<strong>in</strong>g success is<br />
that potential lessons may go undiscovered. An alternative to rely<strong>in</strong>g on failure<br />
to challenge one’s assumptions is to create a standard practice of review<strong>in</strong>g<br />
each case regardless of outcome, pr<strong>in</strong>cipally through <strong>the</strong> use of a formal<br />
After Action Review (AAR).<br />
Case Study Two: The Red Team<br />
Recently, I was asked to serve on a newly formed red team with<strong>in</strong> <strong>the</strong><br />
Department of Defense. I agreed to participate, despite a number of serious<br />
concerns hav<strong>in</strong>g to do both with <strong>the</strong> nature and structure of red teams <strong>in</strong> general<br />
and with my own experience with ethnocentrism and its effects on analysis.<br />
These concerns are applicable not only to red teams, but also to any<br />
analyst put <strong>in</strong> <strong>the</strong> position of try<strong>in</strong>g to “th<strong>in</strong>k like <strong>the</strong>m.” 9<br />
This particular red team was part of a constructive/conceptual war game <strong>in</strong><br />
which <strong>the</strong>re were 11 participants, seven of whom had doctorates. Of <strong>the</strong> seven<br />
doctorates, three were psychologists, one was a historian, one was an economist,<br />
one was a political scientist, and one was an anthropologist. The o<strong>the</strong>r<br />
four participants had extensive military backgrounds. There were no physical<br />
scientists or eng<strong>in</strong>eers. N<strong>in</strong>e of <strong>the</strong> 11 participants were white males, one was<br />
a male born <strong>in</strong> <strong>the</strong> region of <strong>in</strong>terest, and one was a white female. All were<br />
middle class. Seven of <strong>the</strong> 11 were raised <strong>in</strong> nom<strong>in</strong>ally Christian homes and<br />
three <strong>in</strong> nom<strong>in</strong>ally Jewish homes. (I say nom<strong>in</strong>ally because it was not possible<br />
to determ<strong>in</strong>e <strong>the</strong>ir level of religious commitment dur<strong>in</strong>g this exercise.)<br />
I mention <strong>the</strong> demographics of <strong>the</strong> group because it was not representative<br />
of <strong>the</strong> adversary we were <strong>in</strong>tended to simulate. Although <strong>the</strong> group had<br />
8<br />
Charles Perrow, Normal Accidents.<br />
9<br />
Military red teams are meant to simulate <strong>the</strong> actions of an adversary <strong>in</strong> some type of war game or<br />
crisis simulation, usually with <strong>the</strong> goal of generat<strong>in</strong>g scenarios for tra<strong>in</strong><strong>in</strong>g and read<strong>in</strong>ess or for<br />
logistics and plann<strong>in</strong>g. These war games may be live, e.g., force-on-force simulations like those of<br />
<strong>the</strong> US Army Combat Tra<strong>in</strong><strong>in</strong>g Centers; virtual, as <strong>in</strong> flight simulators; or constructive, ei<strong>the</strong>r digital<br />
<strong>the</strong>ater-level simulations or purely conceptual games centered on strategic, tactical, or operational<br />
issues.<br />
81
CHAPTER SIX<br />
numerous doma<strong>in</strong> matter experts, very few had first-hand knowledge of <strong>the</strong><br />
region of <strong>in</strong>terest. Only one participant was from <strong>the</strong> area, had spent formative<br />
years <strong>the</strong>re, spoke <strong>the</strong> languages, and experienced <strong>the</strong> culture firsthand. As<br />
this group was assembled to simulate <strong>the</strong> behavior and decisionmak<strong>in</strong>g of a<br />
foreign adversary, this aspect was more important than it would have been for<br />
a substantive team develop<strong>in</strong>g threat assessments around a specific topic or<br />
target. Consequently, <strong>the</strong> scenarios developed by <strong>the</strong> red team often reflected<br />
an adversary whose behavior and decisionmak<strong>in</strong>g resembled those of educated,<br />
white, middle class Americans.<br />
The one member of <strong>the</strong> red team who had been born <strong>in</strong> and spent formative<br />
years <strong>in</strong> <strong>the</strong> region of <strong>in</strong>terest regularly stopped <strong>the</strong> scenario development process<br />
by say<strong>in</strong>g, “They wouldn’t do that” or “They don’t th<strong>in</strong>k that way.” On<br />
several occasions, he objected, “This scenario is way too complex” or “They<br />
wouldn’t use that tactic; it requires too much direct communication.” His<br />
objections were not usually based on military considerations; ra<strong>the</strong>r, <strong>the</strong>y were<br />
based on <strong>the</strong> cultural norms and mores of <strong>the</strong> adversary. He talked of k<strong>in</strong>ship<br />
relationships as a specific type of social network <strong>in</strong> <strong>the</strong> region and of <strong>the</strong> value<br />
of k<strong>in</strong>ship for understand<strong>in</strong>g <strong>the</strong> adversary’s <strong>in</strong>tentions. In short, he brought an<br />
ethnographic perspective to <strong>the</strong> exercise.<br />
Hav<strong>in</strong>g no personal or professional experience with this region or its cultures,<br />
I thought it appropriate to defer to his first-person experience. Ultimately,<br />
however, it proved difficult to conv<strong>in</strong>ce <strong>the</strong> group that this man’s<br />
cultural knowledge was, <strong>in</strong> fact, an area of specialized knowledge that needed<br />
to be factored <strong>in</strong>to each scenario. This difficulty was born out of ano<strong>the</strong>r type<br />
of ethnocentric bias.<br />
Invit<strong>in</strong>g an anthropologist to a red team exercise presupposes that <strong>the</strong> red<br />
team takes seriously <strong>the</strong> notion that cultural differences matter and that those<br />
cultural factors ought to be made explicit <strong>in</strong> <strong>the</strong> analytic process. The problem<br />
<strong>in</strong> this case was that <strong>the</strong> anthropologist was not an area expert for this region<br />
and its cultures, and <strong>the</strong> one area expert who was <strong>the</strong>re lacked <strong>the</strong> academic<br />
credentials to be taken seriously by <strong>the</strong> o<strong>the</strong>r members of <strong>the</strong> group. Had I<br />
been able to assert <strong>the</strong> same concepts that <strong>the</strong> o<strong>the</strong>r <strong>in</strong>dividual asserted, it<br />
would have had a certa<strong>in</strong> academic, or scientific, imprimatur because of my<br />
tra<strong>in</strong><strong>in</strong>g and experience. Because he lacked <strong>the</strong>se credentials, many of <strong>the</strong><br />
o<strong>the</strong>r <strong>in</strong>dividual’s <strong>in</strong>sights were lost, and <strong>the</strong> analytic product suffered as a<br />
result.<br />
The Red Team: Discussion<br />
I am reluctant to fault <strong>the</strong> organizers for <strong>the</strong> ethnocentric bias <strong>in</strong> <strong>the</strong> demographic<br />
composition of <strong>the</strong> red team. It is very difficult to assemble a truly<br />
82
COMBATING ETHNOCENTRISM<br />
representative red team. There is <strong>the</strong> obvious problem of security. Someone<br />
fully able to represent <strong>the</strong> adversary culturally would very likely be unable to<br />
obta<strong>in</strong> requisite clearances for participation <strong>in</strong> a classified red team exercise.<br />
In fact, even if it were possible to f<strong>in</strong>d someone both culturally representative<br />
and sympa<strong>the</strong>tic to <strong>the</strong> goals of <strong>the</strong> red team, such as <strong>the</strong> participant born <strong>in</strong><br />
<strong>the</strong> region, <strong>the</strong> conflicts triggered by that sympathy, cultural identity, and cultural<br />
allegiance could well lead to unforeseen cognitive biases that would be<br />
difficult to counteract. 10<br />
An alternative is to f<strong>in</strong>d an ethnic American citizen with similarities to <strong>the</strong><br />
people of <strong>the</strong> region of <strong>in</strong>terest, but simply f<strong>in</strong>d<strong>in</strong>g a US citizen with <strong>the</strong> same<br />
ethnicity as those of <strong>the</strong> region of <strong>in</strong>terest does not guarantee any special<br />
<strong>in</strong>sight <strong>in</strong>to <strong>the</strong>ir th<strong>in</strong>k<strong>in</strong>g. Ethnicity is not <strong>the</strong> same as shar<strong>in</strong>g culture or identity.<br />
Not all ethnic groups <strong>in</strong> <strong>the</strong> US are isolated and self-perpetuat<strong>in</strong>g. Many,<br />
<strong>in</strong> fact, put great effort <strong>in</strong>to try<strong>in</strong>g to assimilate <strong>in</strong>to <strong>the</strong> larger “American culture”<br />
by distanc<strong>in</strong>g <strong>the</strong>mselves from <strong>the</strong>ir culture of orig<strong>in</strong>. These people often<br />
struggle with <strong>the</strong>ir own concept of cultural identity and <strong>the</strong> broader issues of<br />
community affiliation. 11 Many immigrants and most first-generation offspr<strong>in</strong>g<br />
have already begun <strong>the</strong> process of acculturation. More strik<strong>in</strong>g, <strong>the</strong>ir offspr<strong>in</strong>g<br />
display a process of enculturation <strong>in</strong> <strong>the</strong> US by learn<strong>in</strong>g <strong>the</strong> language, attend<strong>in</strong>g<br />
<strong>the</strong> schools, assimilat<strong>in</strong>g local and national values, and establish<strong>in</strong>g ties to<br />
a diverse community outside of <strong>the</strong>ir own ethnic enclave. In fact, <strong>the</strong> children<br />
of recent immigrants share many of <strong>the</strong> same cognitive filters as those who are<br />
generations removed from migration. That said, <strong>the</strong>re are American citizens<br />
born <strong>in</strong> <strong>the</strong> region of <strong>in</strong>terest, like <strong>the</strong> member of <strong>the</strong> red team <strong>in</strong> which I participated,<br />
who do have <strong>in</strong>sight <strong>in</strong>to specific cultures, pr<strong>in</strong>cipally because <strong>the</strong>ir<br />
enculturation was affected by be<strong>in</strong>g born <strong>in</strong>, and liv<strong>in</strong>g <strong>in</strong>, a foreign region.<br />
The participant <strong>in</strong> that red team was a foreign-born American citizen, but<br />
foreign birth is not a necessary condition for enculturation. 12 Liv<strong>in</strong>g <strong>in</strong> a foreign<br />
region, speak<strong>in</strong>g <strong>the</strong> language, <strong>in</strong>teract<strong>in</strong>g with <strong>the</strong> people, develop<strong>in</strong>g<br />
community ties, and establish<strong>in</strong>g an identity with<strong>in</strong> that community are all<br />
part of <strong>the</strong> acculturation process and allow one to alter <strong>the</strong> cognitive filters<br />
through which one <strong>in</strong>terprets <strong>the</strong> world. Time spent on a US military base, <strong>in</strong> a<br />
US embassy, or <strong>in</strong> a Western hotel overseas does not lead to acculturation.<br />
10<br />
Philip Cushman, Construct<strong>in</strong>g <strong>the</strong> Self, Construct<strong>in</strong>g America; John Lucy, Language Diversity<br />
and Thought; Douglass Price-Williams, Explorations <strong>in</strong> Cross-Cultural Psychology; Marshall<br />
Segall, Cross-Cultural Psychology; Richard Shweder, Th<strong>in</strong>k<strong>in</strong>g Through <strong>Culture</strong>s; Yali Zou and<br />
Enrique Trueba, Ethnic Identity and Power; and Benjam<strong>in</strong> Whorf.<br />
11<br />
David Lev<strong>in</strong>son and Melv<strong>in</strong> Ember, American Immigrant <strong>Culture</strong>s. For raw data cover<strong>in</strong>g 186<br />
cultural groups s<strong>in</strong>ce 1937, <strong>in</strong>clud<strong>in</strong>g immigrants, see <strong>the</strong> Human Relations Area Files at Yale<br />
University.<br />
12<br />
Some anthropologists have argued that enculturation is specific to childhood but <strong>the</strong> evidence<br />
supports that it is a lifelong process. See Segall.<br />
83
CHAPTER SIX<br />
Quite <strong>the</strong> contrary, each of <strong>the</strong>se is a “virtual” America, an approximation of<br />
life <strong>in</strong> <strong>the</strong> United States on some foreign soil, and it is <strong>the</strong> time spent away<br />
from <strong>the</strong>se <strong>in</strong>stitutions that is important.<br />
The red team experience re<strong>in</strong>forced lessons I learned from my own analytic<br />
failures and biases. Watch<strong>in</strong>g <strong>the</strong> struggle between <strong>the</strong> man enculturated <strong>in</strong> <strong>the</strong><br />
region of <strong>in</strong>terest and <strong>the</strong> academic experts was a frustrat<strong>in</strong>g experience. It<br />
was clear that <strong>the</strong> experts would not, or could not, hear what he was say<strong>in</strong>g<br />
and that nei<strong>the</strong>r he nor I knew how to get <strong>the</strong> o<strong>the</strong>r experts to listen. I doubt<br />
this communication failure was <strong>the</strong> result of stubbornness or arrogance on<br />
anyone’s part. It seemed ra<strong>the</strong>r that <strong>the</strong> experts’ th<strong>in</strong>k<strong>in</strong>g naturally defaulted to<br />
<strong>the</strong>ir own cultural reference po<strong>in</strong>ts, which <strong>in</strong>terfered with his attempts to communicate<br />
his cultural knowledge.<br />
Specific cultural knowledge is a skill and <strong>the</strong> foundation for forecast<strong>in</strong>g <strong>the</strong><br />
behavior and decisionmak<strong>in</strong>g of foreign actors. Acquir<strong>in</strong>g cultural knowledge<br />
should be taken as seriously as learn<strong>in</strong>g any o<strong>the</strong>r facet of one’s analytic capabilities.<br />
Moreover, it is <strong>in</strong>cumbent on analysts to educate <strong>the</strong>ir own leadership<br />
and policymakers about <strong>the</strong> value and utility of cultural knowledge for <strong>in</strong>telligence<br />
analysis.<br />
Conclusion and Recommendations<br />
Ethnocentrism is a normal condition, and it results <strong>in</strong> analytic bias. The analytic<br />
community and <strong>in</strong>telligence researchers need to develop tools and techniques<br />
to combat analytic ethnocentrism. I believe that us<strong>in</strong>g cultural diversity<br />
as a strategy to combat ethnocentrism has much to recommend it. 13<br />
Security concerns may make it very difficult, if not impossible, to hire people<br />
who are genu<strong>in</strong>ely representative of a given culture. As an alternative to<br />
focus<strong>in</strong>g on hir<strong>in</strong>g practices, I recommend a formal cultural tra<strong>in</strong><strong>in</strong>g program<br />
to facilitate acculturation. The program would <strong>in</strong>clude language acquisition<br />
and a classroom segment centered on specific cultures, but it would go beyond<br />
<strong>the</strong>se by hav<strong>in</strong>g <strong>the</strong> students go to countries of <strong>in</strong>terest and <strong>in</strong>teract with <strong>the</strong><br />
13<br />
Some social action groups have appropriated <strong>the</strong> words “cultural diversity” from Levi-Strauss<br />
and <strong>the</strong> French school of structural anthropology as a rally<strong>in</strong>g cry to advance an agenda of equal<br />
access to resources and power. That is, <strong>the</strong> concept has been politicized, and, <strong>in</strong>vok<strong>in</strong>g <strong>the</strong> words<br />
“cultural diversity” <strong>in</strong> a public forum ensures that people will have some emotional reaction. This<br />
is not my <strong>in</strong>tention. The use of <strong>the</strong> words <strong>in</strong> this work is meant strictly <strong>in</strong> <strong>the</strong> technical sense, specifically,<br />
that is, to refer to <strong>in</strong>dividuals whose enculturation occurs among different cultures or<br />
<strong>in</strong>dividuals who have experienced acculturation. Acculturation is not specific to any one group,<br />
all people can and do experience acculturation to one degree or ano<strong>the</strong>r through cultural contact<br />
and cultural diffusion, def<strong>in</strong>ed as <strong>the</strong> spread<strong>in</strong>g of a cultural trait (e.g., material object, idea, or<br />
behavior pattern) from one society to ano<strong>the</strong>r without wholesale dislocation or migration. Moreover,<br />
acculturation can be accomplished purposefully through tra<strong>in</strong><strong>in</strong>g and fieldwork.<br />
84
COMBATING ETHNOCENTRISM<br />
people <strong>in</strong> <strong>the</strong>ir own sett<strong>in</strong>g and on <strong>the</strong>ir own terms. Students would be encouraged<br />
to <strong>in</strong>vestigate <strong>the</strong> rituals, norms, taboos, k<strong>in</strong>ship systems, and social networks<br />
of <strong>the</strong> cultures be<strong>in</strong>g studied. There would also be provision for<br />
cont<strong>in</strong>u<strong>in</strong>g on-l<strong>in</strong>e education and an on-l<strong>in</strong>e community of practice for mentor<strong>in</strong>g,<br />
problem solv<strong>in</strong>g, and peer-to-peer <strong>in</strong>teraction.<br />
In my view, a stand-alone tra<strong>in</strong><strong>in</strong>g program would be <strong>in</strong>sufficient to affect<br />
analytic processes without specific follow-on programs. Retention of tra<strong>in</strong><strong>in</strong>g<br />
requires repetition, problem solv<strong>in</strong>g, application, and evaluation. People must<br />
use what <strong>the</strong>y learn and <strong>the</strong>n determ<strong>in</strong>e if what <strong>the</strong>y have learned can improve<br />
<strong>the</strong> quality of <strong>the</strong>ir work. To this end, I recommend a formal After Action<br />
Review (AAR) process.<br />
The AAR is used by <strong>the</strong> US Army to capture lessons learned after a tra<strong>in</strong><strong>in</strong>g<br />
exercise or a live operation. Unlike conventional postmortems and traditional<br />
performance critiques, <strong>the</strong> AAR is used to evaluate successes as well as failures.<br />
Although failure generally receives more scrut<strong>in</strong>y and attention than success,<br />
an approach that only exam<strong>in</strong>es failure results <strong>in</strong> sampl<strong>in</strong>g error. If one<br />
only scrut<strong>in</strong>izes mistakes, o<strong>the</strong>rwise effective methods may be blamed for <strong>the</strong><br />
errors. That those techniques were successful <strong>in</strong> 99 out of 100 cases can go<br />
unnoticed, with <strong>the</strong> result that <strong>the</strong> failures receive disproportionate attention<br />
and bias <strong>the</strong> statistical results of <strong>the</strong> postmortem. The AAR was specifically<br />
designed to avoid this problem.<br />
The AAR process was <strong>in</strong>troduced <strong>in</strong> <strong>the</strong> mid 1970s, but it is based on <strong>the</strong><br />
oral history method of “after combat <strong>in</strong>terviews” employed by S.L.A. Marshall<br />
dur<strong>in</strong>g World War II, <strong>the</strong> Korean War, and <strong>the</strong> Vietnam War. As soon as<br />
possible after a battle, regardless of <strong>the</strong> outcome, Marshall would assemble<br />
soldiers who were <strong>in</strong>volved and, us<strong>in</strong>g a semistructured <strong>in</strong>terview technique,<br />
would engage <strong>the</strong>m <strong>in</strong> a group discussion about <strong>the</strong>ir <strong>in</strong>dividual and team roles<br />
and actions dur<strong>in</strong>g combat.<br />
The current AAR method also <strong>in</strong>cludes such objective data as tactics, logistics,<br />
kill ratios, time-to-task, accuracy-of-task, and operational outcomes. 14<br />
Informed by <strong>the</strong> objective data, a group discussion led by a facilitator tra<strong>in</strong>ed<br />
<strong>in</strong> <strong>the</strong> elicitation process ensues. The AAR, along with support<strong>in</strong>g documents,<br />
such as historical studies and relevant doctr<strong>in</strong>al materials, is <strong>the</strong>n stored <strong>in</strong> a<br />
knowledge repository at <strong>the</strong> US Army’s Center for Army Lessons Learned<br />
(CALL). 15<br />
With some customization, an AAR process and a lessons learned repository<br />
could be created for <strong>in</strong>telligence analysts. Although seem<strong>in</strong>gly time-consum<strong>in</strong>g<br />
and cumbersome, with tra<strong>in</strong><strong>in</strong>g and expert facilitators, <strong>the</strong> AAR process could be<br />
modified and streaml<strong>in</strong>ed for use by analysts at <strong>the</strong> end of a production cycle. As<br />
14<br />
John Morrison and Larry Meliza, Foundations of <strong>the</strong> After Action Review Process.<br />
85
CHAPTER SIX<br />
a practical matter, <strong>the</strong> process would be used mostly with longer works, such as<br />
assessments or estimates. The <strong>in</strong>telligence product, along with AAR notes, would<br />
<strong>the</strong>n be <strong>in</strong>corporated <strong>in</strong> a community knowledge repository. This knowledge<br />
repository would also help <strong>in</strong> <strong>the</strong> development and ref<strong>in</strong>ement of advanced analytic<br />
courses by provid<strong>in</strong>g course developers with basel<strong>in</strong>e analytic data. In short,<br />
<strong>the</strong> repository becomes a tool for cont<strong>in</strong>uous educational needs analysis and l<strong>in</strong>ks<br />
tra<strong>in</strong><strong>in</strong>g directly to <strong>the</strong> actual work practices of analysts. These data can be used<br />
as a test bed for research on <strong>the</strong> effectiveness of analytic methodology. In this<br />
way, <strong>the</strong> lessons learned are not lost to future generations of analysts.<br />
15<br />
See <strong>the</strong> US Army Center for Army Lessons Learned Web site, which has l<strong>in</strong>ks to numerous<br />
o<strong>the</strong>r repositories. Although each organization has customized <strong>the</strong> concept to meet its unique<br />
needs, all of <strong>the</strong> US military services, <strong>the</strong> National Aeronautics and Space Adm<strong>in</strong>istration, <strong>the</strong><br />
Department of Energy, <strong>the</strong> Environmental Protection Agency, <strong>the</strong> North Atlantic Treaty Organization,<br />
<strong>the</strong> United Nations, and <strong>the</strong> m<strong>in</strong>istries of defense of Australia and Canada, currently have<br />
Lessons Learned repositories.<br />
86
CHAPTER SEVEN<br />
Instructional Technology: Effectiveness and<br />
Implications for <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong><br />
J. D. Fletcher 1<br />
Rob Johnston<br />
The <strong>Intelligence</strong> <strong>Community</strong> has begun to <strong>in</strong>vest substantial resources <strong>in</strong><br />
<strong>the</strong> tra<strong>in</strong><strong>in</strong>g and education of its analysts. With <strong>the</strong> exception of a few<br />
advanced courses available through distance learn<strong>in</strong>g networks, this <strong>in</strong>struction<br />
is delivered us<strong>in</strong>g a conventional classroom model. This model possesses<br />
a number of <strong>in</strong>herent <strong>in</strong>efficiencies, <strong>in</strong>clud<strong>in</strong>g <strong>in</strong>consistent <strong>in</strong>struction, strict<br />
ties to time and place of <strong>in</strong>struction, large student-to-<strong>in</strong>structor ratios, and limited<br />
active participation by students due to class size and schedul<strong>in</strong>g.<br />
Research suggests that significant improvements can be achieved through<br />
<strong>the</strong> use of computer-based <strong>in</strong>structional technology. Accord<strong>in</strong>g to <strong>the</strong>se studies,<br />
this technology can <strong>in</strong>crease <strong>in</strong>structional effectiveness and reduce time<br />
needed to learn. It can achieve <strong>the</strong>se efficiencies, moreover, while both lower<strong>in</strong>g<br />
<strong>the</strong> cost of <strong>in</strong>struction and <strong>in</strong>creas<strong>in</strong>g its availability. 2 This chapter summarizes<br />
evidence on <strong>the</strong> promise of <strong>in</strong>structional technology for <strong>in</strong>telligence<br />
analysis tra<strong>in</strong><strong>in</strong>g.<br />
1<br />
Dr. J. D. Fletcher is a research staff member at <strong>the</strong> Institute for Defense Analyses, where he specializes<br />
<strong>in</strong> issues of manpower, personnel, and tra<strong>in</strong><strong>in</strong>g. He holds graduate degrees <strong>in</strong> computer<br />
science and educational psychology from Stanford University.<br />
2<br />
Because <strong>in</strong>structional technology makes few dist<strong>in</strong>ctions between formal education and professional<br />
tra<strong>in</strong><strong>in</strong>g, <strong>the</strong> term “<strong>in</strong>struction” will be used for both <strong>in</strong> this chapter.<br />
87
CHAPTER SEVEN<br />
Background<br />
The argument for <strong>the</strong> use of <strong>in</strong>structional technology usually beg<strong>in</strong>s with a<br />
comparative exam<strong>in</strong>ation of <strong>the</strong> effectiveness of classroom <strong>in</strong>struction and<br />
<strong>in</strong>dividual tutor<strong>in</strong>g. For <strong>in</strong>stance, <strong>the</strong> graph below illustrates <strong>the</strong> comb<strong>in</strong>ed f<strong>in</strong>d<strong>in</strong>gs<br />
of three dissertation studies that compared one-on-one tutor<strong>in</strong>g with oneon-many<br />
classroom <strong>in</strong>struction. 3<br />
It is not surpris<strong>in</strong>g that such comparisons would show that tutored students<br />
learned more than those taught <strong>in</strong> classrooms. What is surpris<strong>in</strong>g is <strong>the</strong> magnitude<br />
of <strong>the</strong> difference. Overall, as <strong>the</strong> figure shows, it was two standard devia-<br />
Individual Tutor<strong>in</strong>g Compared to Classroom Instruction<br />
tions. This f<strong>in</strong>d<strong>in</strong>g means, for example, that with <strong>in</strong>structional time held fairly<br />
constant one-on-one tutor<strong>in</strong>g raised <strong>the</strong> performance of 50th percentile students<br />
to that of 98th percentile students. These, and similar empirical research f<strong>in</strong>d<strong>in</strong>gs,<br />
suggest that differences between one-on-one tutor<strong>in</strong>g and typical classroom<br />
<strong>in</strong>struction are not only likely, but also very large.<br />
Why <strong>the</strong>n do we not provide <strong>the</strong>se benefits to all students? The answer is<br />
straightforward and obvious. With <strong>the</strong> exception of a few critical skills, such as<br />
aircraft pilot<strong>in</strong>g and surgery, we cannot afford it. One-on-one tutor<strong>in</strong>g has been<br />
described as an educational imperative and an economic impossibility. 4<br />
3<br />
Benjam<strong>in</strong> S. Bloom, “The 2 Sigma Problem: The Search for Methods of Group Instruction as<br />
Effective as One-to-One Tutor<strong>in</strong>g.” The dissertation studies were performed under Bloom’s direction.<br />
4<br />
M. Scriven, “Problems and Prospects for Individualization.”<br />
88
INSTRUCTIONAL TECHNOLOGY<br />
The success of one-on-one tutor<strong>in</strong>g may be expla<strong>in</strong>ed by two factors. First,<br />
measured <strong>in</strong> terms of questions asked and answered, tutors and <strong>the</strong>ir students<br />
engage <strong>in</strong> many more <strong>in</strong>structional <strong>in</strong>teractions per unit of time than is possible<br />
<strong>in</strong> a classroom. Second, one-on-one tutor<strong>in</strong>g can overcome <strong>the</strong> substantial<br />
spread of ability, measured by <strong>the</strong> time needed to reach m<strong>in</strong>imal proficiency,<br />
that is found <strong>in</strong> practically every classroom. Tutor<strong>in</strong>g reduces time-to-learn by<br />
adapt<strong>in</strong>g each <strong>in</strong>teraction to <strong>the</strong> needs of each student. Less time is spent on<br />
material <strong>the</strong> student has already learned, and more time is spent on material<br />
rema<strong>in</strong><strong>in</strong>g to be mastered.<br />
To <strong>in</strong>vestigate <strong>the</strong> <strong>in</strong>tensity of <strong>in</strong>structional <strong>in</strong>teractions, Art Graesser and<br />
Natalie Person compared question<strong>in</strong>g and answer<strong>in</strong>g <strong>in</strong> classrooms with those<br />
<strong>in</strong> tutorial sett<strong>in</strong>gs. 5 They found that classroom groups of students ask about<br />
three questions an hour and that any s<strong>in</strong>gle student <strong>in</strong> a classroom asks about<br />
0.11 questions per hour. In contrast, <strong>the</strong>y found that students <strong>in</strong> <strong>in</strong>dividual tutorial<br />
sessions asked 20–30 questions an hour and were required to answer 117–<br />
146 questions per hour. Reviews of <strong>the</strong> <strong>in</strong>tensity of <strong>in</strong>teraction that occurs <strong>in</strong><br />
technology-based <strong>in</strong>struction have found even more active student response<br />
levels. 6<br />
Differences <strong>in</strong> <strong>the</strong> time needed by <strong>in</strong>dividuals <strong>in</strong> any classroom to meet<br />
<strong>in</strong>structional objectives are also substantial. Studies on this issue have reported<br />
ratios vary<strong>in</strong>g from 1:3 to 1:7 <strong>in</strong> <strong>the</strong> times <strong>the</strong> fastest learners need to learn compared<br />
to <strong>the</strong> times needed by <strong>the</strong> slowest learners. Although <strong>the</strong>se differences<br />
may be due <strong>in</strong>itially to ability, <strong>the</strong>se studies suggest that such ability is quickly<br />
overtaken by prior knowledge of <strong>the</strong> subject matter. 7 This effect is particularly<br />
evident <strong>in</strong> <strong>in</strong>struction for post-secondary-school students, because prior knowledge<br />
rapidly <strong>in</strong>creases with age and experience. Technology-based <strong>in</strong>struction<br />
has long been recognized for its ability to adjust <strong>the</strong> pace of <strong>in</strong>struction to <strong>in</strong>dividual<br />
needs, advanc<strong>in</strong>g through <strong>in</strong>structional material as quickly or as slowly as<br />
required. The overall result has been substantial sav<strong>in</strong>gs <strong>in</strong> <strong>the</strong> time required to<br />
meet given <strong>in</strong>structional objectives. 8<br />
It should be emphasized that <strong>the</strong>se benefits are not achieved at <strong>the</strong> expense<br />
of <strong>in</strong>structional quality. Research has found that many <strong>in</strong>structional technologies<br />
have a positive impact on learn<strong>in</strong>g across a wide variety of student populations,<br />
sett<strong>in</strong>gs, and <strong>in</strong>structional subject matters. 9<br />
This research suggests that technology-based <strong>in</strong>struction results <strong>in</strong> substantial<br />
sav<strong>in</strong>gs of time and money. Studies have shown that <strong>the</strong> times saved aver-<br />
5<br />
Art Graesser and Natalie Person, “Question-Ask<strong>in</strong>g Dur<strong>in</strong>g Tutor<strong>in</strong>g.”<br />
6<br />
J. D. Fletcher, Technology, <strong>the</strong> Columbus Effect, and <strong>the</strong> Third Revolution <strong>in</strong> Learn<strong>in</strong>g.<br />
7<br />
Sigmund Tobias, “When Do Instructional Methods Make a Difference?”<br />
8<br />
J. D. Fletcher, “Evidence for Learn<strong>in</strong>g From Technology-Assisted Instruction.”<br />
9<br />
Ken Spencer, “Modes, Media and Methods: The Search for Educational Effectiveness.”<br />
89
CHAPTER SEVEN<br />
age about 30 percent, as seen <strong>in</strong> <strong>the</strong> table below. The reduction <strong>in</strong> overhead<br />
expenses averages 20–30 percent. 10 Research has shown that <strong>the</strong> cost ratios<br />
(calculated as <strong>the</strong> ratio of experimental <strong>in</strong>tervention costs over <strong>the</strong> costs of a<br />
control group) for <strong>in</strong>teractive multimedia technology (computer-based <strong>in</strong>struction<br />
with enhanced audio, graphics, and/or video; CD-ROM and DVD-based<br />
<strong>in</strong>struction; <strong>in</strong>teractive video, etc.) favor it over conventional <strong>in</strong>struction along<br />
with time sav<strong>in</strong>gs of about 31 percent. 11 Simulation of such systems as helicopters,<br />
tanks, and command-control systems for tra<strong>in</strong><strong>in</strong>g combat skills has<br />
also proven to be cost-effective. 12 The operational costs for simulation are, on<br />
average, 10 percent of <strong>the</strong> costs of us<strong>in</strong>g <strong>the</strong> actual systems to tra<strong>in</strong>. 13<br />
Time Sav<strong>in</strong>gs for Technology-Based Instruction<br />
Study (Reference) Number of Studies<br />
Reviewed<br />
Average Time Saved<br />
(Percent)<br />
Military Tra<strong>in</strong><strong>in</strong>g -<br />
13 54<br />
Orlansky<br />
Higher Education -<br />
8 31<br />
Fletcher<br />
Higher Education -<br />
17 34<br />
Kulik<br />
Adult Education - Kulik 15 24<br />
Meta-analysis Demonstrates <strong>the</strong> Effectiveness of Instructional<br />
Technology<br />
Researchers often use a meta-analytic approach to review and syn<strong>the</strong>size<br />
quantitative research studies on a variety of issues, <strong>in</strong>clud<strong>in</strong>g <strong>in</strong>structional<br />
effectiveness. 14 This method <strong>in</strong>volves a three-step process, which beg<strong>in</strong>s with<br />
<strong>the</strong> collection of studies relevant to <strong>the</strong> issue us<strong>in</strong>g clearly def<strong>in</strong>ed procedures<br />
that can be replicated. Next, a quantitative measure, “effect size,” is used to<br />
tabulate <strong>the</strong> outcomes of all <strong>the</strong> collected studies, <strong>in</strong>clud<strong>in</strong>g those with results<br />
that are not statistically significant. F<strong>in</strong>ally, statistical procedures are used to<br />
10<br />
Jesse Orlansky and Joseph Str<strong>in</strong>g, Cost-Effectiveness of Computer-Based Instruction <strong>in</strong> Military<br />
Tra<strong>in</strong><strong>in</strong>g; H. Solomon, Economic Issues <strong>in</strong> Cost-Effectiveness Analyses of Military Skill<br />
Tra<strong>in</strong><strong>in</strong>g; James Kulik, “Meta-<strong>Analytic</strong> Studies of F<strong>in</strong>d<strong>in</strong>gs on Computer-Based Instruction”; Rob<br />
Johnston, “The Effectiveness of Instructional Technology”; Ruth Phelps et al., “Effectiveness and<br />
Costs of Distance Education Us<strong>in</strong>g Computer-Mediated Communication”; J. D. Fletcher Effectiveness<br />
and Cost of Interactive Videodisc Instruction <strong>in</strong> Defense Tra<strong>in</strong><strong>in</strong>g and Education.<br />
11<br />
J. D. Fletcher, “Computer-Based Instruction: Costs and Effectiveness.”<br />
12<br />
Jesse Orlanksy et al., The Cost and Effectiveness of <strong>the</strong> Multi-Service Distributed Tra<strong>in</strong><strong>in</strong>g Testbed<br />
(MDT2) for Tra<strong>in</strong><strong>in</strong>g Close Air Support.<br />
13<br />
Jesse Orlansky et al., The Value of Simulation for Tra<strong>in</strong><strong>in</strong>g.<br />
14<br />
Gene Glass, “Primary, Secondary, and Meta-Analysis of Research.”<br />
90
INSTRUCTIONAL TECHNOLOGY<br />
syn<strong>the</strong>size <strong>the</strong> quantitative measures and describe <strong>the</strong> f<strong>in</strong>d<strong>in</strong>gs of <strong>the</strong> analysis.<br />
Meta-analysis appears to be especially suited for syn<strong>the</strong>siz<strong>in</strong>g <strong>the</strong> results of<br />
<strong>in</strong>structional research, and it has been widely used for this purpose s<strong>in</strong>ce its<br />
<strong>in</strong>troduction <strong>in</strong> 1976.<br />
Meta-analysis is still be<strong>in</strong>g developed as a technique, and some matters<br />
concern<strong>in</strong>g its use, notably <strong>the</strong> “file-drawer” problem and calculation of effect<br />
size, rema<strong>in</strong> unsettled. Chapter Twelve presents a more detailed explanation<br />
and <strong>the</strong>se considerations. Briefly, however, meta-analytic reviews of <strong>in</strong>structional<br />
technology effectiveness have found substantial results favor<strong>in</strong>g its use<br />
over traditional technologies of classroom <strong>in</strong>struction.<br />
Overall, effect sizes for post-secondary school <strong>in</strong>struction average about<br />
0.42, which is roughly equivalent to rais<strong>in</strong>g <strong>the</strong> achievement of 50th percentile<br />
students to that of 66th percentile students. 15 Reviews of more elaborate forms<br />
of <strong>in</strong>structional technology, such as those us<strong>in</strong>g applied artificial <strong>in</strong>telligent<br />
techniques, have found effect sizes <strong>in</strong> excess of 1.0, which is roughly equivalent<br />
to rais<strong>in</strong>g <strong>the</strong> achievement of 50th percentile students to that of <strong>the</strong> 84th<br />
percentile. 16 It seems reasonable to conclude that <strong>the</strong> reduced costs and<br />
reduced time to learn obta<strong>in</strong>ed <strong>in</strong> applications of <strong>in</strong>structional technology are<br />
not achieved at <strong>the</strong> expense of <strong>in</strong>structional effectiveness.<br />
Encourag<strong>in</strong>g as <strong>the</strong>se favorable results are, our ability to apply <strong>in</strong>structional<br />
technology efficiently may be <strong>in</strong> its <strong>in</strong>fancy. F<strong>in</strong>d<strong>in</strong>gs thus far have been based<br />
on <strong>in</strong>structional applications <strong>in</strong>tended to teach facts (e.g., What is <strong>the</strong> capital<br />
of Brazil? What is <strong>the</strong> Spanish word for chapel? Who was <strong>the</strong> first director of<br />
<strong>the</strong> Central <strong>Intelligence</strong> Agency?) concepts (e.g., What is a mass spectrometer<br />
used for? What is <strong>the</strong> difference between micro- and macro-economics? When<br />
must you use a torque wrench?), and procedures (e.g., How do you record a<br />
movie from television? How do you prepare a purchase requisition? How do<br />
you calibrate a radar repeater?). All <strong>in</strong>telligence analysts must possess a repertoire<br />
of facts, concepts, and procedures to perform <strong>the</strong>ir craft, and <strong>in</strong>structional<br />
technology holds great promise for <strong>in</strong>creas<strong>in</strong>g both <strong>the</strong> efficiency with which<br />
<strong>the</strong>y might develop this repertoire and <strong>the</strong>ir access to <strong>in</strong>structional resources<br />
for do<strong>in</strong>g so.<br />
However, <strong>the</strong> capabilities analysts may seek through <strong>in</strong>struction are likely<br />
to <strong>in</strong>clude more abstract, or “higher,” cognitive processes. For <strong>in</strong>stance, <strong>in</strong><br />
addition to learn<strong>in</strong>g a procedure, analysts may need <strong>the</strong> capability to recognize<br />
15<br />
Chen-L<strong>in</strong> Kulik., James Kulik and Barbara Shwalb, “Effectiveness of Computer-Based Adult<br />
Education: A Meta-Analysis”; Chen-L<strong>in</strong> Kulik and James Kulik, “Effectiveness of Computer-<br />
Based Education <strong>in</strong> Colleges”; Rob Johnston and J. D. Fletcher, A Meta-Analysis of <strong>the</strong> Effectiveness<br />
of Computer-Based Tra<strong>in</strong><strong>in</strong>g for Military Instruction; J. D. Fletcher, “Evidence for Learn<strong>in</strong>g<br />
from Technology-Assisted Instruction.”<br />
16<br />
Sherrie P. Gott, R. S. Kane, and Alan Lesgold , Tutor<strong>in</strong>g for Transfer of Technical Competence.<br />
91
CHAPTER SEVEN<br />
<strong>the</strong> procedure’s applicability <strong>in</strong> unfamiliar situations, modify it as needed, and<br />
use it to develop new approaches and procedures. Early on, Bloom discussed<br />
learn<strong>in</strong>g objectives as a hierarchy beg<strong>in</strong>n<strong>in</strong>g with knowledge at <strong>the</strong> most rudimentary<br />
level and ascend<strong>in</strong>g through comprehension, application, analysis,<br />
and syn<strong>the</strong>sis to evaluation. 17 Bloom’s is not <strong>the</strong> only such hierarchy to<br />
emerge from research on <strong>in</strong>structional design, but it seems to be <strong>the</strong> best<br />
known, and it describes as well as any <strong>the</strong> various levels of knowledge, skill,<br />
and ability to which learners may aspire.<br />
Current Research on Higher Cognitive Abilities<br />
Analysts have begun to discuss development of <strong>the</strong> higher cognitive abilities<br />
needed to deal with unanticipated and novel challenges. 18 Components of<br />
such “cognitive read<strong>in</strong>ess” may <strong>in</strong>clude:<br />
• Situation awareness—<strong>the</strong> ability to comprehend <strong>the</strong> relevant aspects of a<br />
situation and use this understand<strong>in</strong>g to choose reasonable courses of<br />
action. 19 Practice and feedback <strong>in</strong> complex, simulated environments have<br />
been shown to improve situation awareness.<br />
• Memory—<strong>the</strong> ability to recall and/or recognize patterns <strong>in</strong> a situation that<br />
lead to likely solutions. It may be supported by two underly<strong>in</strong>g <strong>the</strong>oretical<br />
mechanisms: encod<strong>in</strong>g specificity, 20 which stresses <strong>the</strong> importance of<br />
respond<strong>in</strong>g to relevant external and <strong>in</strong>ternal perceptual cues, and transferappropriate<br />
process<strong>in</strong>g, 21 which stresses <strong>the</strong> actions performed dur<strong>in</strong>g<br />
encod<strong>in</strong>g and retrieval. Some <strong>in</strong>structional techniques, such as overlearn<strong>in</strong>g,<br />
22 have been shown to enhance long-term retention. 23<br />
• Transfer—<strong>the</strong> ability to apply what is learned <strong>in</strong> one context to a different<br />
context. It can be perceived ei<strong>the</strong>r as <strong>the</strong> ability to select and apply procedural<br />
knowledge ga<strong>in</strong>ed <strong>in</strong> one context to ano<strong>the</strong>r (“low road” transfer) or<br />
as <strong>the</strong> ability to apply <strong>the</strong> pr<strong>in</strong>ciples abstracted from a set of contexts to<br />
ano<strong>the</strong>r (“high road” transfer). 24 Extensive practice, with feedback, will<br />
17<br />
Benjam<strong>in</strong>. S. Bloom, Taxonomy of Educational Objectives.<br />
18<br />
J. E. Morrison, and J. D. Fletcher, Cognitive Read<strong>in</strong>ess.<br />
19<br />
M. R. Endsley, “Design and Evaluation for Situation Awareness Enhancement.”<br />
20<br />
E. Tulv<strong>in</strong>g and D. M. Thomson, “Encod<strong>in</strong>g Specificity and Retrieval Processes <strong>in</strong> Episodic<br />
Memory.”<br />
21<br />
C. D. Morris, J. D. Bransford, and J. J. Franks, “Level of Process<strong>in</strong>g Versus Transfer-Appropriate<br />
Process<strong>in</strong>g.”<br />
22<br />
The use of specific problem-solv<strong>in</strong>g methods repetitively.<br />
23<br />
R. A. Wisher, M. A. Sabol, and J. A. Ellis Stay<strong>in</strong>g Sharp: Retention of Military Knowledge and<br />
Skills.<br />
92
INSTRUCTIONAL TECHNOLOGY<br />
enhance <strong>the</strong> former. Instruction <strong>in</strong> develop<strong>in</strong>g mental abstractions will<br />
enhance <strong>the</strong> latter.<br />
• Metacognition—<strong>the</strong> executive functions of thought, more specifically,<br />
those needed to monitor, assess, and regulate one’s own cognitive processes.<br />
25 Meta-cognitive skills can be enhanced by exercises designed to<br />
<strong>in</strong>crease awareness of self-regulatory processes. 26<br />
• Pattern Recognition—<strong>the</strong> ability to dist<strong>in</strong>guish <strong>the</strong> familiar from <strong>the</strong> unfamiliar.<br />
It may be accomplished by “template match<strong>in</strong>g,” which <strong>in</strong>volves<br />
compar<strong>in</strong>g reta<strong>in</strong>ed images with <strong>in</strong>com<strong>in</strong>g sensory impressions; or by<br />
“feature comparison,” which <strong>in</strong>volves recogniz<strong>in</strong>g and generaliz<strong>in</strong>g from<br />
dist<strong>in</strong>ctive features of a structure held <strong>in</strong> memory with <strong>in</strong>com<strong>in</strong>g sensory<br />
impressions. 27 Pattern recognition can be taught through a comb<strong>in</strong>ation of<br />
extensive practice, with feedback, and <strong>in</strong>struction <strong>in</strong> form<strong>in</strong>g abstractions.<br />
• Automaticity—processes that require only limited conscious attention. 28<br />
Automaticity can be taught by provid<strong>in</strong>g extensive practice, with feedback.<br />
• Problem Solv<strong>in</strong>g—<strong>the</strong> ability to analyze a situation and identify a goal or<br />
goals that flow from it, identify tasks and subtasks lead<strong>in</strong>g to <strong>the</strong> goal,<br />
develop a plan to achieve <strong>the</strong>m, and apply <strong>the</strong> resources needed to carry out<br />
<strong>the</strong> plan. Practice, with feedback, and overlearn<strong>in</strong>g can enhance problemsolv<strong>in</strong>g<br />
ability <strong>in</strong> many tasks. Techniques for problem solv<strong>in</strong>g can be successfully<br />
taught, as can <strong>the</strong> knowledge base needed to implement <strong>the</strong>m. 29<br />
• Decisionmak<strong>in</strong>g—a component of problem solv<strong>in</strong>g, but <strong>the</strong> emphasis <strong>in</strong><br />
decisionmak<strong>in</strong>g is on recogniz<strong>in</strong>g learned patterns, review<strong>in</strong>g courses of<br />
action, assess<strong>in</strong>g <strong>the</strong>ir impact, select<strong>in</strong>g one, and allocat<strong>in</strong>g resources to<br />
it. 30 Instruction <strong>in</strong> assess<strong>in</strong>g courses of action has been shown to improve<br />
decisionmak<strong>in</strong>g, but some aspects of successful decisionmak<strong>in</strong>g are more<br />
likely to be <strong>in</strong>born than tra<strong>in</strong>ed.<br />
• Mental Flexibility and Creativity—<strong>the</strong> ability to generate and modify<br />
courses of action rapidly <strong>in</strong> response to chang<strong>in</strong>g circumstances. 31 It<br />
24<br />
G. Salomon and D. N. Perk<strong>in</strong>s “Rocky Roads to Transfer: Reth<strong>in</strong>k<strong>in</strong>g Mechanisms of a<br />
Neglected Phenomenon.”<br />
25<br />
J. H. Flavell, “Metacognitive Aspects of Problem Solv<strong>in</strong>g.”<br />
26<br />
D. J. Hacker, Metacognition: Def<strong>in</strong>itions and Empirical Foundations [On-l<strong>in</strong>e Report].<br />
27<br />
M. H. Ashcraft, Fundamentals of Cognition.<br />
28<br />
R. M. Shiffr<strong>in</strong> and W. Schneider, W. “Controlled and Automatic Human Information Process<strong>in</strong>g:<br />
II. Perceptual Learn<strong>in</strong>g.”<br />
29<br />
J. R. Hayes, The Complete Problem Solver.<br />
30<br />
P. Slovic, S. Lichtenste<strong>in</strong>, and B. Fischoff, “Decision-mak<strong>in</strong>g.”<br />
31<br />
D. Klahr, & H. A. Simon, “What Have Psychologists (and O<strong>the</strong>rs) Discovered About <strong>the</strong> Process<br />
of Scientific Discovery?”<br />
93
CHAPTER SEVEN<br />
<strong>in</strong>cludes <strong>the</strong> ability to devise plans and actions that differ from and<br />
improve upon “school solutions.” Capabilities that widen <strong>the</strong> range of<br />
options can be taught, but higher levels of creativity are more likely to be<br />
<strong>in</strong>born than tra<strong>in</strong>ed.<br />
The above review suggests, first, that <strong>the</strong> creative processes needed by analysts<br />
can, to some extent, be broken down <strong>in</strong>to components, and second, that<br />
<strong>the</strong>se components can, aga<strong>in</strong> to some extent, be taught. Instructional technology<br />
can now substantially aid analysts <strong>in</strong> acquir<strong>in</strong>g <strong>the</strong> facts, concepts, and<br />
procedures needed to perform <strong>the</strong>ir craft. However, it must become <strong>in</strong>creas<strong>in</strong>gly<br />
“<strong>in</strong>telligent” if it is to compress <strong>the</strong> years of experience analysts now<br />
need to become proficient and help <strong>the</strong>m more rapidly acquire <strong>the</strong> advanced<br />
cognitive capabilities—those higher <strong>in</strong> Bloom’s hierarchy—that <strong>the</strong>y also<br />
need. To do this successfully, <strong>in</strong>struction must be tailored to <strong>the</strong> specific background,<br />
abilities, goals, and <strong>in</strong>terests of <strong>the</strong> <strong>in</strong>dividual student or user. Instructional<br />
technology must provide what has been called “articulate expertise.”<br />
Not only must it supply helpful and relevant guidance <strong>in</strong> <strong>the</strong>se more advanced<br />
levels of knowledge, skills, and abilities, it must do so <strong>in</strong> a way that learners<br />
and users with vary<strong>in</strong>g levels of knowledge and skill can understand.<br />
Discussion<br />
At this po<strong>in</strong>t, it may be worth review<strong>in</strong>g <strong>the</strong> capabilities provided by “non<strong>in</strong>telligent”<br />
<strong>in</strong>structional technology s<strong>in</strong>ce <strong>the</strong> 1950s. It has been able to: 32<br />
• accommodate <strong>the</strong> rate of progress of <strong>in</strong>dividual students, allow<strong>in</strong>g as<br />
much or as little time as each needs to reach <strong>in</strong>structional objectives;<br />
• tailor both <strong>the</strong> content and <strong>the</strong> sequence of <strong>in</strong>structional content to each<br />
student’s needs; 33<br />
• make <strong>the</strong> <strong>in</strong>struction easy or difficult, specific or abstract, applied or <strong>the</strong>oretical<br />
as necessary;<br />
• adjust to students’ most efficient learn<strong>in</strong>g styles (collaborative or <strong>in</strong>dividual,<br />
verbal or visual, etc.).<br />
Intelligent tutor<strong>in</strong>g systems are a different matter. They require quite specific<br />
capabilities that were first targeted <strong>in</strong> <strong>the</strong> 1960s. 34 Two key capabilities<br />
are that <strong>in</strong>telligent tutor<strong>in</strong>g systems must:<br />
32<br />
E. Galanter, Automatic Teach<strong>in</strong>g; R. C. Atk<strong>in</strong>son and H. A. Wilson, Computer-Assisted<br />
Instruction; P. Suppes and M. Morn<strong>in</strong>gstar, Computer-assisted Instruction at Stanford 1966-68; J.<br />
D. Fletcher and M. R. Rockway, “Computer-based Tra<strong>in</strong><strong>in</strong>g <strong>in</strong> <strong>the</strong> Military.”<br />
33<br />
J. S. Brown, R. R. Burton, and J. DeKleer, “Pedagogical, Natural Language and Knowledge<br />
Eng<strong>in</strong>eer<strong>in</strong>g <strong>in</strong> SOPHIE I, II, and III.”<br />
94
INSTRUCTIONAL TECHNOLOGY<br />
• allow ei<strong>the</strong>r <strong>the</strong> system or <strong>the</strong> student to ask open-ended questions and <strong>in</strong>itiate<br />
<strong>in</strong>structional, “mixed-<strong>in</strong>itiative” dialogue as needed or desired;<br />
• generate <strong>in</strong>structional material and <strong>in</strong>teractions on demand <strong>in</strong>stead of<br />
requir<strong>in</strong>g developers to foresee and store all <strong>the</strong> materials and <strong>in</strong>teractions<br />
needed to meet all possible eventualities.<br />
Mixed-<strong>in</strong>itiative dialogue requires a language for <strong>in</strong>formation retrieval,<br />
tools to assist decisionmak<strong>in</strong>g, and <strong>in</strong>struction that is shared by both <strong>the</strong> system<br />
and <strong>the</strong> student/user. The system must have <strong>the</strong> capability (referred to as<br />
“generative capability”) to devise, on demand, <strong>in</strong>teractions with students that<br />
do not rely on predicted and prestored formats. This capability <strong>in</strong>volves more<br />
than generat<strong>in</strong>g problems tailored to each student’s needs. It must also provide<br />
<strong>the</strong> <strong>in</strong>teractions and presentations that simulate one-on-one tutorial <strong>in</strong>struction,<br />
<strong>in</strong>clud<strong>in</strong>g coach<strong>in</strong>g, h<strong>in</strong>ts, and critiques of completed solutions.<br />
Cost conta<strong>in</strong>ment is one motivation for want<strong>in</strong>g to generate responses to all<br />
possible student states and actions <strong>in</strong>stead of attempt<strong>in</strong>g to anticipate and store<br />
<strong>the</strong>m. Ano<strong>the</strong>r arises from basic research on human learn<strong>in</strong>g, memory, perception,<br />
and cognition. As documented by Neisser among o<strong>the</strong>rs, dur<strong>in</strong>g <strong>the</strong><br />
1960s and 1970s, <strong>the</strong> emphasis <strong>in</strong> basic research on human behavior and on<br />
<strong>the</strong> way <strong>in</strong> which it is understood shifted from <strong>the</strong> strict logical positivism of<br />
behavioral psychology, which focused on directly observable actions, to consideration<br />
of <strong>the</strong> <strong>in</strong>ternal, cognitive processes that were needed to expla<strong>in</strong><br />
empirically observed behavioral phenomena and are assumed to mediate and<br />
enable human learn<strong>in</strong>g. 35<br />
The hallmark of this approach is <strong>the</strong> view that see<strong>in</strong>g, hear<strong>in</strong>g, and remember<strong>in</strong>g<br />
are all acts of construction, mak<strong>in</strong>g more or less use of <strong>the</strong> limited<br />
stimulus <strong>in</strong>formation provided by our perceptual capabilities. Constructivist<br />
approaches are <strong>the</strong> subject of much current and relevant discussion <strong>in</strong> <strong>in</strong>structional<br />
research circles, but <strong>the</strong>y are firmly grounded <strong>in</strong> <strong>the</strong> foundations of scientific<br />
psychology. 36 For <strong>in</strong>stance, <strong>in</strong> 1890, William James stated his General<br />
Law of Perception: “Whilst part of what we perceive comes through our<br />
senses from <strong>the</strong> object before us, ano<strong>the</strong>r part (and it may be <strong>the</strong> larger part)<br />
always comes out of our m<strong>in</strong>d.” 37<br />
In this sense, <strong>the</strong> generative capability sought by <strong>in</strong>telligent <strong>in</strong>structional<br />
systems is not merely someth<strong>in</strong>g nice to have. It is essential if we are to<br />
34<br />
J. R. Carbonell, “AI <strong>in</strong> CAI: An Artificial <strong>Intelligence</strong> Approach to Computer-Assisted Instruction”;<br />
J. D. Fletcher & M. R. Rockway.<br />
35<br />
U. Neisser, Cognitive Psychology.<br />
36<br />
For example, T. M. Duffy, and D. H. Jonassen, Constructivism and <strong>the</strong> Technology of Instruction;<br />
S. Tobias and L. T. Frase, “Educational psychology and tra<strong>in</strong><strong>in</strong>g.”<br />
37<br />
William James, Pr<strong>in</strong>ciples of Psychology: Volume I.<br />
95
CHAPTER SEVEN<br />
advance beyond <strong>the</strong> constra<strong>in</strong>ts of <strong>the</strong> prescribed, prebranched, programmed<br />
learn<strong>in</strong>g and ad hoc pr<strong>in</strong>ciples commonly used to design technology-based<br />
<strong>in</strong>struction. The long-term vision is that tra<strong>in</strong><strong>in</strong>g, education, and performance<br />
improvement will take <strong>the</strong> form of human-computer conversations.<br />
There has been progress toward this end. This conversational capability has<br />
been realized <strong>in</strong> systems that can discuss issues with students us<strong>in</strong>g a formal<br />
language, such as computer programm<strong>in</strong>g or propositional calculus. 38 More<br />
recent research suggests that significantly improved natural-language dialogue<br />
capabilities can be achieved by <strong>in</strong>structional technology. 39 Such an<br />
<strong>in</strong>teractive, generative capability is needed if we are to deal successfully with<br />
<strong>the</strong> extent, variety, and mutability of human cognition. Much can now be<br />
accomplished by <strong>in</strong>structional technology, but much more can be expected.<br />
Conclusion<br />
The research discussed above suggests that <strong>in</strong>structional technology can:<br />
• reduce costs of <strong>in</strong>struction;<br />
• <strong>in</strong>crease <strong>the</strong> accessibility of <strong>in</strong>struction;<br />
• <strong>in</strong>crease <strong>in</strong>structional effectiveness for analysts;<br />
• reduce <strong>the</strong> time analysts need to learn facts, concepts, and procedures;<br />
• track progress and ensure that all learners achieve <strong>in</strong>structional targets;<br />
• provide opportunities for help<strong>in</strong>g analysts to compress experience and<br />
achieve <strong>the</strong> higher cognitive levels of mastery demanded by <strong>the</strong>ir craft.<br />
In addition, <strong>the</strong> f<strong>in</strong>d<strong>in</strong>gs suggest a rule of “thirds.” This rule posits that <strong>the</strong><br />
present state-of-<strong>the</strong>-art <strong>in</strong> <strong>in</strong>structional technologies can reduce <strong>the</strong> cost of<br />
<strong>in</strong>struction by about a third and ei<strong>the</strong>r <strong>in</strong>crease achievement by about a third or<br />
decrease time to reach <strong>in</strong>structional objectives by a third. Eventually, <strong>in</strong>structional<br />
technology should provide a conversation between <strong>the</strong> analyst and <strong>the</strong><br />
technology that will tailor <strong>in</strong>struction <strong>in</strong> real time and on demand to <strong>the</strong> particular<br />
knowledge, skills, abilities, <strong>in</strong>terests, goals, and needs of each <strong>in</strong>dividual. This<br />
capability, now available <strong>in</strong> rudimentary forms, can be expected to improve and<br />
develop with time. Even <strong>in</strong> its current state of development, however, <strong>in</strong>structional<br />
technology deserves serious attention with<strong>in</strong> <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong>.<br />
38<br />
For example, BIP and EXCHECK, respectively. For <strong>the</strong> first, see A. Barr, M. Beard, and R. C.<br />
Atk<strong>in</strong>son, “A rationale and description of a CAI Program to teach <strong>the</strong> BASIC Programm<strong>in</strong>g Language”;<br />
for <strong>the</strong> second, see P. Suppes and M. Morn<strong>in</strong>gstar.<br />
39<br />
A. C. Graesser, M. A. Gernsbacher, and S. Goldman, Handbook of Discourse Processes.<br />
96
CHAPTER EIGHT<br />
Organizational <strong>Culture</strong>: Anticipatory Socialization and<br />
<strong>Intelligence</strong> Analysts<br />
Stephen H. Konya 1<br />
Rob Johnston<br />
I know it sounds silly, but I had this image of James Bond before I<br />
started work<strong>in</strong>g here. The truth is, I just sit <strong>in</strong> a cubicle, and I write<br />
reports.<br />
Every organization has a unique culture that is def<strong>in</strong>ed partly by its <strong>in</strong>dividual<br />
members and partly by its structure, history, and policies. For that culture<br />
to endure, it must be transmitted from current members to new members. This<br />
process, known as organizational socialization, is especially important <strong>in</strong><br />
organizations with strong, <strong>in</strong>sular cultures, as those with weak cultures have<br />
less to transmit and will tend to experience culture changes as members come<br />
and go.<br />
Although socialization beg<strong>in</strong>s prior to a person’s first day on <strong>the</strong> job and is<br />
a cont<strong>in</strong>uous process, it is experienced most <strong>in</strong>tensely by new employees. The<br />
cultural symbols acquired and <strong>in</strong>terpreted dur<strong>in</strong>g <strong>the</strong>ir <strong>in</strong>itial <strong>in</strong>teraction with<br />
<strong>the</strong> <strong>in</strong>stitution create potent and last<strong>in</strong>g impressions. 2 For <strong>the</strong>m, socialization<br />
1<br />
Stephen Konya is a Research Associate at <strong>the</strong> Institute for Defense Analyses, currently exam<strong>in</strong><strong>in</strong>g<br />
multimodal <strong>in</strong>terfaces for <strong>the</strong> dismounted for <strong>the</strong> DARPA/Army Future Combat Systems<br />
program. He holds an MS <strong>in</strong> <strong>in</strong>dustrial and organizational psychology from Rensselaer Polytechnic<br />
Institute.<br />
2<br />
Umberto Eco, A Theory of Semiotics; Clifford Geertz, The Interpretation of <strong>Culture</strong>s; Jacques<br />
Lacan, Ecrits; Ferd<strong>in</strong>and de Saussure, Course <strong>in</strong> General L<strong>in</strong>guistics.<br />
97
CHAPTER EIGHT<br />
is <strong>the</strong> process of learn<strong>in</strong>g <strong>the</strong> ropes; tra<strong>in</strong><strong>in</strong>g; and becom<strong>in</strong>g formally and<br />
<strong>in</strong>formally acqua<strong>in</strong>ted with what is actually of value with<strong>in</strong> <strong>the</strong> organization. 3<br />
It is also <strong>the</strong> time when one learns <strong>the</strong> organization’s norms and taboos and <strong>the</strong><br />
extent of its social capital. 4 In sum, formal and <strong>in</strong>formal socialization are types<br />
of control mechanism for ma<strong>in</strong>ta<strong>in</strong><strong>in</strong>g <strong>the</strong> norms, or status quo, with<strong>in</strong> any<br />
organization. 5<br />
Organizational Socialization<br />
Accord<strong>in</strong>g to Daniel Feldman, organizational socialization is “<strong>the</strong> process<br />
through which <strong>in</strong>dividuals are transformed from outsiders to participat<strong>in</strong>g,<br />
effective members of an organization.” 6 As shown <strong>in</strong> Figure 1, Feldman<br />
divides this process <strong>in</strong>to three stages: gett<strong>in</strong>g <strong>in</strong> (or anticipatory socialization),<br />
break<strong>in</strong>g <strong>in</strong> (or accommodation), and settl<strong>in</strong>g <strong>in</strong> (often referred to as role management).<br />
Dur<strong>in</strong>g <strong>the</strong> gett<strong>in</strong>g-<strong>in</strong> stage, potential employees try to acquire<br />
<strong>in</strong>formation about an organization from available sources, such as Web sites,<br />
professional journals, and corporate annual reports. The break<strong>in</strong>g-<strong>in</strong> stage<br />
<strong>in</strong>cludes orientation and learn<strong>in</strong>g organizational as well as job-related proce-<br />
Feldman’s three stages of organizational socialization.<br />
dures. The settl<strong>in</strong>g-<strong>in</strong> stage concludes when an <strong>in</strong>dividual atta<strong>in</strong>s full member<br />
status <strong>in</strong> <strong>the</strong> organization.<br />
While each of <strong>the</strong> three stages of socialization is important, <strong>the</strong> focus of this<br />
chapter is on <strong>the</strong> first, or anticipatory, stage. There are several reasons for this.<br />
Clearly, <strong>the</strong> expectations people develop about an organization <strong>the</strong>y are jo<strong>in</strong><strong>in</strong>g<br />
are important to a new recruit’s eventual satisfaction, retention, and performance.<br />
Moreover, because it can control several aspects of <strong>the</strong> recruitment<br />
process, this stage is often <strong>the</strong> easiest for an organization to change. This<br />
chapter will take both a descriptive and prescriptive approach to eas<strong>in</strong>g <strong>the</strong><br />
socialization of new employees.<br />
3<br />
William G. Tierney and Robert A. Rhoads, Faculty Socialization as Cultural Process.<br />
4<br />
See footnote 7 <strong>in</strong> Chapter Two.<br />
5<br />
John P. Wanous, Organizational Entry.<br />
6<br />
Daniel C. Feldman, “The Multiple Socialization of Organization Members.”<br />
98
ORGANIZATIONAL CULTURE<br />
Anticipatory Socialization<br />
Anticipatory socialization encompasses all of <strong>the</strong> learn<strong>in</strong>g that occurs prior<br />
to a recruit’s enter<strong>in</strong>g on duty. 7 At this stage, an <strong>in</strong>dividual forms expectations<br />
about <strong>the</strong> job and makes decisions about <strong>the</strong> suitability of fit between himself<br />
and <strong>the</strong> organization. What a person has heard about work<strong>in</strong>g for a particular<br />
organization, such as an <strong>in</strong>telligence agency, provides an idea of what to<br />
expect if hired. Conversely, <strong>in</strong>dividuals who do not believe <strong>the</strong>y would fit <strong>in</strong><br />
may decide not to apply.<br />
There are two variables that are particularly useful for track<strong>in</strong>g a potential<br />
employee’s progress through <strong>the</strong> anticipatory stage: The first is realism, or <strong>the</strong><br />
extent to which an <strong>in</strong>dividual acquires an accurate picture of daily life <strong>in</strong> <strong>the</strong><br />
organization. Realism is <strong>in</strong>fluenced by <strong>the</strong> level of success recruits achieve<br />
dur<strong>in</strong>g <strong>the</strong> <strong>in</strong>formation-shar<strong>in</strong>g and <strong>in</strong>formation-evaluation part of <strong>the</strong>ir<br />
recruitment. The second is congruence, or <strong>the</strong> extent to which <strong>the</strong> organization’s<br />
resources and <strong>the</strong> <strong>in</strong>dividual’s needs and skills are mutually satisfy<strong>in</strong>g.<br />
Congruence is <strong>in</strong>fluenced by <strong>the</strong> level of success an <strong>in</strong>dividual has achieved <strong>in</strong><br />
mak<strong>in</strong>g decisions about employment. Although it cannot directly <strong>in</strong>fluence<br />
congruence, which is an <strong>in</strong>herently personal experience, an organization can<br />
present relevant <strong>in</strong>formation <strong>in</strong> order to provide a realistic and accurate<br />
description of <strong>the</strong> work performed and <strong>the</strong> work environment.<br />
Organizations often use <strong>in</strong>terviews to beg<strong>in</strong> <strong>the</strong> socialization of new<br />
recruits. For example, an <strong>in</strong>terviewer will attempt to provide an accurate<br />
description of what to expect from <strong>the</strong> job and <strong>the</strong> organization, <strong>the</strong> purpose<br />
be<strong>in</strong>g to reduce <strong>the</strong> likelihood that a recruit will be disturbed by unanticipated<br />
situations. Interview<strong>in</strong>g is also used to determ<strong>in</strong>e <strong>the</strong> degree to which <strong>the</strong>re is a<br />
match between <strong>the</strong> values of potential recruits and <strong>the</strong> values of <strong>the</strong> organization.<br />
New recruits with personal values match<strong>in</strong>g those of <strong>the</strong> organization<br />
have been found to adjust to <strong>the</strong> organization’s culture more quickly than<br />
recruits with nonmatch<strong>in</strong>g values. 8<br />
Organizations also send cultural messages to new recruits dur<strong>in</strong>g <strong>in</strong>terviews.<br />
When <strong>the</strong>re are several rounds of <strong>in</strong>terviews with progressively senior<br />
members of <strong>the</strong> organization, for example, <strong>the</strong> message conveyed is that f<strong>in</strong>d<strong>in</strong>g<br />
<strong>the</strong> best person for <strong>the</strong> position is important. In contrast, hir<strong>in</strong>g for a parttime<br />
job at <strong>the</strong> lowest level of <strong>the</strong> organization is often accomplished quickly,<br />
to <strong>the</strong> extent that a person hav<strong>in</strong>g m<strong>in</strong>imally acceptable qualifications may<br />
7<br />
This stage is termed “pre-arrival” <strong>in</strong> Lyman W. Porter, Edward E. Lawler, and J. Richard Hackman,<br />
Behavior <strong>in</strong> Organizations.<br />
8<br />
Jerald Greenberg and Robert A. Baron, Behavior <strong>in</strong> Organizations: Understand<strong>in</strong>g <strong>the</strong> Human<br />
Side of Work.<br />
99
CHAPTER EIGHT<br />
often be hired on <strong>the</strong> spot. The cultural message <strong>in</strong> this case is that such<br />
employees are easily let <strong>in</strong> to and out of <strong>the</strong> organization.<br />
Ano<strong>the</strong>r, particularly pert<strong>in</strong>ent example is <strong>in</strong>telligence work, which requires<br />
that recruits undergo employment screen<strong>in</strong>gs unlike those found <strong>in</strong> most civilian<br />
jobs. Potential <strong>CIA</strong> analysts must submit to a thorough background <strong>in</strong>vestigation,<br />
a polygraph exam<strong>in</strong>ation, and f<strong>in</strong>ancial and credit reviews. Fur<strong>the</strong>r, a battery<br />
of psychological and medical exams must be passed prior to a formal<br />
employment offer. The timeframe for <strong>the</strong> background check elim<strong>in</strong>ates <strong>the</strong> possibility<br />
of a rapid hir<strong>in</strong>g decision. Even more important are <strong>the</strong> nonverbal messages<br />
sent to <strong>the</strong> recruit that this is a position of secrecy and high importance.<br />
Several sources of <strong>in</strong>formation contribute to beliefs about any organization.<br />
Friends or relatives who are already part of <strong>the</strong> organization might share <strong>the</strong>ir<br />
experiences with <strong>the</strong> person consider<strong>in</strong>g employment. Information might also<br />
be acquired from o<strong>the</strong>r sources, such as professional journals, magaz<strong>in</strong>es,<br />
newspaper articles, television, governmental and private Web sites, public<br />
statements or testimony, and annual reports. While <strong>the</strong>se sources of <strong>in</strong>formation<br />
about an organization are far from perfect (all may conta<strong>in</strong> positive and<br />
negative hyperbole), <strong>the</strong>y are still useful from <strong>the</strong> po<strong>in</strong>t of view of form<strong>in</strong>g<br />
prelim<strong>in</strong>ary ideas about what it might be like to work for that organization.<br />
Because competition for highly qualified employees is fierce, successful<br />
recruitment usually <strong>in</strong>volves a skillful comb<strong>in</strong>ation of salesmanship and diplomacy.<br />
Recruiters tend to describe <strong>the</strong>ir organizations <strong>in</strong> glow<strong>in</strong>g terms, gloss<strong>in</strong>g<br />
over <strong>in</strong>ternal problems and external threats, while emphasiz<strong>in</strong>g positive<br />
features. The result is that potential employees often receive unrealistically<br />
positive impressions of conditions prevail<strong>in</strong>g <strong>in</strong> a specific organization. When<br />
<strong>the</strong>y arrive on <strong>the</strong> job and f<strong>in</strong>d that <strong>the</strong>ir expectations are not met, <strong>the</strong>y experience<br />
disappo<strong>in</strong>tment, dissatisfaction, and even resentment that <strong>the</strong>y have been<br />
misled. In fact, research f<strong>in</strong>d<strong>in</strong>gs <strong>in</strong>dicate that <strong>the</strong> less employees’ job expectations<br />
are met, <strong>the</strong> less satisfied and committed <strong>the</strong>y are and <strong>the</strong> more likely<br />
<strong>the</strong>y are to th<strong>in</strong>k about quitt<strong>in</strong>g or actually to do so. 9<br />
These negative reactions are sometimes termed entry shock, referr<strong>in</strong>g to <strong>the</strong><br />
confusion and disorientation experienced by many newcomers to an organization.<br />
In order to avoid entry shock, it is important for organizations to provide<br />
job candidates with accurate <strong>in</strong>formation about <strong>the</strong> organization. Research<br />
supports <strong>the</strong> notion that people exposed to realistic job previews later report<br />
higher satisfaction and show lower turnover than those who receive glow<strong>in</strong>g,<br />
but often mislead<strong>in</strong>g, <strong>in</strong>formation about <strong>the</strong>ir companies. 10 Moreover, hav<strong>in</strong>g<br />
9<br />
John P. Wanous et al., “The Effects of Met Expectations on Newcomer Attitudes and Behavior:<br />
A Review and Meta-analysis.”<br />
10<br />
Bruce M. Megl<strong>in</strong>o et al., “Effects of Ralistic Job Previews: A Comparison Us<strong>in</strong>g an Enhancement<br />
and a Reduction Preview.”<br />
100
ORGANIZATIONAL CULTURE<br />
realistic expectations helps to ease <strong>the</strong> accommodation stage of <strong>the</strong> socialization<br />
process.<br />
Consequences of <strong>Culture</strong> Mismatch<br />
When I got here, I felt like a rabbit stuck <strong>in</strong> headlights. Now, I feel<br />
like a deer.<br />
It took me a while to figure out that this place runs more like a<br />
newspaper than a university.<br />
It’s pretty solitary work. I spend all day <strong>in</strong> my head. I really wasn’t<br />
expect<strong>in</strong>g that.<br />
There are several consequences of a cultural mismatch between an<br />
employee and an organization. Among <strong>the</strong>se consequences are culture shock,<br />
low job satisfaction, low employee morale, <strong>in</strong>creased absenteeism, <strong>in</strong>creased<br />
turnover, and <strong>in</strong>creased costs.<br />
<strong>Culture</strong> Shock. People often have to be confronted with different cultures<br />
before <strong>the</strong>y become conscious of <strong>the</strong>ir own culture. In fact, when people are<br />
faced with new cultures, it is not unusual for <strong>the</strong>m to become confused and<br />
disoriented, a phenomenon commonly referred to as culture shock.<br />
Beryl Hesketh and Stephen Bochner, among o<strong>the</strong>rs, have observed that <strong>the</strong><br />
process of adjust<strong>in</strong>g to ano<strong>the</strong>r culture generally follows a U-shaped curve. 11<br />
At first, people are optimistic about learn<strong>in</strong>g a new culture. This excitement is<br />
followed by frustration and confusion as <strong>the</strong>y struggle to learn <strong>the</strong> new culture.<br />
After six months or so with <strong>the</strong> organization, people adjust to <strong>the</strong>ir new<br />
cultures, become more accept<strong>in</strong>g of <strong>the</strong>m, and are more satisfied by <strong>the</strong>m. For<br />
those who enter a mismatched culture, <strong>the</strong> productivity issue is clear: <strong>the</strong> several<br />
months required to adjust and accept <strong>the</strong> new work style results <strong>in</strong> several<br />
months of even lower productivity than is obta<strong>in</strong>able with those who fit <strong>in</strong><br />
right away.<br />
Job Satisfaction. Job satisfaction is def<strong>in</strong>ed by one scholar as “people’s positive<br />
or negative feel<strong>in</strong>gs about <strong>the</strong>ir jobs.” 12 It is hardly surpris<strong>in</strong>g that dissatisfied<br />
employees may try to f<strong>in</strong>d ways of reduc<strong>in</strong>g <strong>the</strong>ir exposure to <strong>the</strong>ir jobs.<br />
This is especially significant when one considers that people spend roughly<br />
one-third of <strong>the</strong>ir lives at work.<br />
11<br />
Beryl Hesketh and Stephen Bochner, “Technological Change <strong>in</strong> a Multicultural Context: Implications<br />
for Tra<strong>in</strong><strong>in</strong>g and Career Plann<strong>in</strong>g”; Maddy Janssens, “Interculture Interaction: A Burden<br />
on International Managers?”<br />
12<br />
Edw<strong>in</strong> A. Locke, “The Nature and Causes of Job Satisfaction.”<br />
101
CHAPTER EIGHT<br />
Interest<strong>in</strong>gly, research suggests that <strong>the</strong> relationship between satisfaction<br />
and task performance, although positive, is not especially strong. 13 Thus,<br />
while job satisfaction may be important to <strong>the</strong> longevity of any <strong>in</strong>dividual<br />
career cycle, it is not a major factor <strong>in</strong> <strong>in</strong>dividual job performance. It does,<br />
however, <strong>in</strong>crease absenteeism, which has a negative effect on overall organizational<br />
productivity.<br />
Absenteeism and Turnover. Research <strong>in</strong>dicates that <strong>the</strong> lower an <strong>in</strong>dividual’s<br />
job satisfaction, <strong>the</strong> more likely he or she is to be absent from work. 14 As<br />
with job satisfaction and task performance, this relationship is modest but also<br />
statistically significant. An employee may even choose to leave an organization<br />
altoge<strong>the</strong>r. This voluntary resignation is measured as employee turnover<br />
and has fiscal consequences for both <strong>the</strong> <strong>in</strong>dividual and <strong>the</strong> organization.<br />
Fiscal Cost. Employee turnover is a critical cost element. The expense of<br />
recruit<strong>in</strong>g and tra<strong>in</strong><strong>in</strong>g new employees, along with lost productivity from<br />
vacant positions and overtime pay for replacement workers, <strong>in</strong>creases operat<strong>in</strong>g<br />
costs and also reduces employee organizational output.<br />
A 2002 study by <strong>the</strong> Employment Policy Foundation found that <strong>the</strong> estimated<br />
turnover cost is $12,506 per year per full-time vacancy for <strong>the</strong> average<br />
employee with total compensation (wages and benefits) of $50,025. 15 As <strong>the</strong><br />
average annual turnover benchmark with<strong>in</strong> <strong>the</strong> Fortune 500 is 23.8 percent,<br />
one can clearly see how critical it is for organizations to lessen <strong>the</strong> number of<br />
employees who leave voluntarily. Even unscheduled absences can be expensive—averag<strong>in</strong>g<br />
between $247 and $534 per employee, per day, accord<strong>in</strong>g to<br />
<strong>the</strong> same study.<br />
Anticipatory Socialization <strong>in</strong> <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong><br />
The secrecy is strange. I thought it would be romantic, but it turns<br />
out that it is just strange.<br />
I was sold on <strong>the</strong> cool factor. It’s still sort of cool, I guess.<br />
Accept<strong>in</strong>g a job with one of <strong>the</strong> 14 members of <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong><br />
differs from o<strong>the</strong>r professions <strong>in</strong> that it is difficult for new employees to have a<br />
clear and precise understand<strong>in</strong>g of <strong>the</strong> roles and responsibilities <strong>the</strong>y are about<br />
13<br />
The correlation is 0.17 accord<strong>in</strong>g to Michelle T. Iaffaldano and Paul M. Much<strong>in</strong>sky <strong>in</strong> <strong>the</strong>ir<br />
“Job Satisfaction and Job Performance: A Meta-Analysis.”<br />
14<br />
Lyman W. Porter et al., “Organizational Commitment, Job Satisfaction and Turnover Among<br />
Psychiatric Technicians.”<br />
15<br />
This number does not take <strong>in</strong>to account <strong>the</strong> additional costs with<strong>in</strong> <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong><br />
for background and security <strong>in</strong>vestigations.<br />
102
ORGANIZATIONAL CULTURE<br />
to assume. This is all <strong>the</strong> more pronounced because, for <strong>the</strong> most part, <strong>the</strong><br />
<strong>Intelligence</strong> <strong>Community</strong> organizations lack a civilian counterpart.<br />
Occasionally, <strong>the</strong> anticipatory socialization of people enter<strong>in</strong>g <strong>the</strong> <strong>in</strong>telligence<br />
analysis discipl<strong>in</strong>e will derive from accounts of current or former practitioners.<br />
More generally, however, a newcomer’s <strong>in</strong>itial impressions stem from <strong>the</strong> fictional<br />
media portrayals, which tend to emphasize <strong>the</strong> supposed glamour of operational<br />
tasks and pay little attention to <strong>the</strong> reality of research-based analytic<br />
work. The absence of hard knowledge about <strong>in</strong>telligence work is attributable, <strong>in</strong><br />
part, to <strong>the</strong> organizational secrecy of <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong> and, <strong>in</strong> part, to<br />
<strong>the</strong> actual socialization process that occurs after one has been accepted for<br />
employment and has passed <strong>the</strong> required background <strong>in</strong>vestigation.<br />
A newcomer’s experience is often contrary to <strong>in</strong>itial expectations. Employees<br />
are discouraged from talk<strong>in</strong>g about <strong>the</strong> specifics of <strong>the</strong>ir work outside of<br />
<strong>the</strong> organization or with those who have not been “cleared.” On an <strong>in</strong>dividual<br />
level, this experience translates <strong>in</strong>to professional culture shock and social isolation.<br />
Organizationally, an <strong>in</strong>tentionally closed system of this k<strong>in</strong>d has a<br />
number of potential performance-related consequences, among <strong>the</strong>m perpetuation<br />
of <strong>the</strong> exist<strong>in</strong>g organizational culture by hir<strong>in</strong>g familial legacies or those<br />
most likely to “fit <strong>in</strong>,” job dissatisfaction, low morale and consequent reduction<br />
<strong>in</strong> employee read<strong>in</strong>ess, <strong>in</strong>creased employee turnover, greater likelihood of<br />
“groupth<strong>in</strong>k,” and strong <strong>in</strong>ternal resistance to organizational change. 16<br />
S<strong>in</strong>ce <strong>the</strong> attacks of 11 September, <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong> has become<br />
more open about its role <strong>in</strong> government, its day-to-day work<strong>in</strong>g environment,<br />
and its employees’ functions and responsibilities. While this openness is an<br />
extension of an ongo<strong>in</strong>g trend toward public outreach—an example is <strong>the</strong><br />
<strong>CIA</strong>’s Officer-<strong>in</strong>-Residence program established <strong>in</strong> 1985—<strong>the</strong> community has<br />
accelerated this trend toward openness <strong>in</strong> an effort to help <strong>the</strong> public, and its<br />
representatives, understand <strong>the</strong> missions and value of <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong>.<br />
17<br />
This trend toward openness has improved employee retention by counteract<strong>in</strong>g<br />
<strong>the</strong> culture shock of mis<strong>in</strong>formed anticipatory socialization and resultant<br />
employee turnover. This trend also helps prepare <strong>the</strong> organization for <strong>the</strong><br />
<strong>in</strong>evitable changes to come by <strong>in</strong>creas<strong>in</strong>g <strong>the</strong> potential recruitment pool,<br />
expand<strong>in</strong>g <strong>the</strong> <strong>in</strong>tellectual diversity of its staff, and foster<strong>in</strong>g better relations<br />
with its broader constituency, <strong>the</strong> American public.<br />
16<br />
Irv<strong>in</strong>g Janis, Groupth<strong>in</strong>k.<br />
17<br />
See <strong>CIA</strong> Officer <strong>in</strong> Residence Program <strong>in</strong> Web Resources <strong>in</strong> bibliography.<br />
103
CHAPTER EIGHT<br />
Conclusion and Recommendations<br />
As noted, <strong>the</strong>re is someth<strong>in</strong>g of a disconnect between <strong>the</strong> largely fictionalized<br />
portrayal of <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong> <strong>in</strong> <strong>the</strong> popular media and <strong>the</strong><br />
actual experience of <strong>in</strong>telligence analysts. This disconnect can be exacerbated<br />
once a recruit is on <strong>the</strong> job and can lead to negative consequences and behaviors,<br />
such as organizational culture shock, employee dissatisfaction, and<br />
<strong>in</strong>creased employee absenteeism and turnover. This has an obvious effect on<br />
<strong>in</strong>dividual analysts, but it has a direct effect on <strong>the</strong> efficiency and effectiveness<br />
of <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong>.<br />
S<strong>in</strong>ce <strong>the</strong> September 2001 attacks, some members of <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong><br />
have acted to change <strong>the</strong> socialization process by provid<strong>in</strong>g accurate<br />
and realistic career <strong>in</strong>formation. One of <strong>the</strong> most widely used media for this is<br />
<strong>the</strong> Internet. For example, <strong>the</strong> Central <strong>Intelligence</strong> Agency’s (<strong>CIA</strong>) Web site<br />
conta<strong>in</strong>s a section on “Life at <strong>the</strong> <strong>CIA</strong>.” 18 This section conta<strong>in</strong>s <strong>in</strong>formation<br />
about <strong>the</strong> Agency and its culture, several analyst profiles and job descriptions<br />
written <strong>in</strong> <strong>the</strong> analyst’s own words, and <strong>in</strong>formation concern<strong>in</strong>g employee<br />
benefits and social and <strong>in</strong>tellectual diversity. Although <strong>the</strong> “Employment”<br />
section of <strong>the</strong> Federal Bureau of Investigation’s (FBI) Web site is less detailed<br />
than <strong>the</strong> “Life at <strong>the</strong> <strong>CIA</strong>” section of <strong>the</strong> <strong>CIA</strong> Web site, it does illustrate a typical<br />
first assignment. 19 In contrast, <strong>the</strong> “Careers” section of <strong>the</strong> Defense <strong>Intelligence</strong><br />
Agency’s (DIA) Web site conta<strong>in</strong>s detailed <strong>in</strong>formation on current job<br />
open<strong>in</strong>gs and <strong>the</strong> application process, but it provides no <strong>in</strong>formation about <strong>the</strong><br />
actual work of a DIA analyst. 20 Steps such as <strong>the</strong>se are encourag<strong>in</strong>g, but <strong>the</strong>y<br />
are still <strong>in</strong>sufficient. There are more active th<strong>in</strong>gs that can be done to facilitate<br />
<strong>the</strong> socialization of new employees.<br />
To beg<strong>in</strong>, <strong>Intelligence</strong> <strong>Community</strong> components should accept that what<br />
most people know about a job is often false and that it is <strong>in</strong>cumbent on <strong>the</strong><br />
organization and its recruiters to present accurate pictures and to work diligently<br />
to dispel myths. This will help to counteract <strong>the</strong> effects of culture<br />
shock. Instead of oversell<strong>in</strong>g a particular job or organization, recruiters should<br />
focus on facilitat<strong>in</strong>g <strong>the</strong> anticipatory socialization of potential employees by<br />
provid<strong>in</strong>g accurate <strong>in</strong>formation about <strong>the</strong> job and about <strong>the</strong> culture of <strong>the</strong> organization<br />
itself. Early <strong>in</strong> <strong>the</strong> selection process, applicants should be provided<br />
with realistic job previews, presented <strong>in</strong> ei<strong>the</strong>r written or oral form. Previews<br />
should conta<strong>in</strong> accurate <strong>in</strong>formation about <strong>the</strong> specific conditions with<strong>in</strong> an<br />
organization and <strong>the</strong> specific requirements of <strong>the</strong> job. Research has shown that<br />
provid<strong>in</strong>g accurate descriptions of tasks is important <strong>in</strong> <strong>in</strong>creas<strong>in</strong>g job commitment<br />
and job satisfaction, as well as decreas<strong>in</strong>g <strong>in</strong>itial turnover of new<br />
18<br />
See Central <strong>Intelligence</strong> Agency Web site <strong>in</strong> Web Resources.<br />
19<br />
See Federal Bureau of Investigation Web site <strong>in</strong> Web Resources<br />
20<br />
See Defense <strong>Intelligence</strong> Agency and US <strong>Intelligence</strong> <strong>Community</strong> Web sites <strong>in</strong> Web Resources.<br />
104
ORGANIZATIONAL CULTURE<br />
employees. 21 The job preview allows candidates to make an <strong>in</strong>formed decision<br />
to cont<strong>in</strong>ue with <strong>the</strong> recruitment process or to withdraw from it if <strong>the</strong>y feel <strong>the</strong><br />
job is not appropriate. Realistic previews also lower unrealistically high<br />
expectations. A particularly good example of such an effort can be found on<br />
<strong>the</strong> <strong>CIA</strong>’s Office of General Counsel Web site. This Web site <strong>in</strong>cludes a section<br />
titled “Misconceptions about work<strong>in</strong>g for <strong>the</strong> <strong>CIA</strong>,” which tries to dispel<br />
prejudices and biases about employment at <strong>the</strong> <strong>CIA</strong> by address<strong>in</strong>g <strong>the</strong>m <strong>in</strong> a<br />
straightforward manner. 22 In addition, <strong>the</strong> authors expla<strong>in</strong> <strong>the</strong> benefits of hav<strong>in</strong>g<br />
work experiences with <strong>the</strong> <strong>CIA</strong> for future employment endeavors <strong>in</strong> o<strong>the</strong>r<br />
areas.<br />
Interview screen<strong>in</strong>gs of applicants should be reviewed and improved where<br />
needed. Hir<strong>in</strong>g <strong>in</strong>terviews are not very effective predictors of job performance;<br />
even so, <strong>the</strong>re are ways to improve <strong>the</strong>ir reliability and validity.<br />
Numerous cognitive measurement <strong>in</strong>struments are available that help predict a<br />
match between an <strong>in</strong>dividual’s knowledge, skills, and abilities and specific<br />
behavioral, cognitive, and psychomotor tasks. 23 In addition, <strong>the</strong> use of structured<br />
<strong>in</strong>terview<strong>in</strong>g - pos<strong>in</strong>g <strong>the</strong> same questions to all applicants - is more<br />
effective than unstructured <strong>in</strong>terview<strong>in</strong>g. Structured <strong>in</strong>terviews allow for consistent<br />
comparisons among applicants. 24 Organizations should also consider<br />
us<strong>in</strong>g panel <strong>in</strong>terviews. Differences among <strong>in</strong>dividual <strong>in</strong>terviewers may result<br />
<strong>in</strong> <strong>in</strong>accurate judgment of an applicant, but <strong>the</strong> overall decision of a team of<br />
evaluators may improve reliability. 25<br />
The use of situational exercises should be <strong>in</strong>cluded <strong>in</strong> <strong>the</strong> recruitment process.<br />
These exercises usually consist of approximations of specific aspects of<br />
a job. They can be used to evaluate candidates’ job abilities and to provide<br />
candidates with simulated work tasks. The former can facilitate organizational<br />
evaluations of candidates’ performance on a job-related task; <strong>the</strong> latter may<br />
help candidates to decide whe<strong>the</strong>r <strong>the</strong> job would be a good match. 26<br />
A desirable additional step would be <strong>the</strong> creation and expansion of academic<br />
degree programs with a focus on <strong>in</strong>telligence and <strong>in</strong>telligence analysis.<br />
Fur<strong>the</strong>r, an enhanced effort to improve public awareness and understand<strong>in</strong>g of<br />
<strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong> through greater community outreach, <strong>in</strong>ternships,<br />
21<br />
Glenn M. McEvoy and Wayne F. Cascio, “Strategies for Reduc<strong>in</strong>g Employee Turnover: A<br />
Meta-analysis.”<br />
22<br />
See Central <strong>Intelligence</strong> Agency, Office of General Counsel Web site.<br />
23<br />
The Buros Institute of Mental Measurements tracks and reports <strong>the</strong> statistical validity and reliability<br />
of thousands of measurement <strong>in</strong>struments.<br />
24<br />
Richard D. Arvey and James E. Campion, “The Employment Interview: A Summary of Recent<br />
Research.”<br />
25<br />
P. L. Roth and James E. Campion, “An Analysis of <strong>the</strong> Predictive Power of <strong>the</strong> Panel Interview<br />
and Pre-Employment Tests.”<br />
26<br />
Wayne F. Cascio, Applied Psychology <strong>in</strong> Human Resource Management.<br />
105
CHAPTER EIGHT<br />
research fellowships, professional workshops, and academic forums will help<br />
to facilitate better employee relations by provid<strong>in</strong>g potential employees with a<br />
clearer perspective on what to expect after receiv<strong>in</strong>g <strong>the</strong>ir badge.<br />
106
CHAPTER NINE<br />
Recommendations<br />
The First Step: Recogniz<strong>in</strong>g A Fundamental Problem<br />
It is far too early <strong>in</strong> <strong>the</strong> research process to determ<strong>in</strong>e if any one organizational<br />
model for <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong> is more or less effective than any<br />
o<strong>the</strong>r, but I believe <strong>the</strong>re is a fundamental structural question that needs to be<br />
addressed at <strong>the</strong> outset. This is, <strong>in</strong> my view, that current report<strong>in</strong>g competes<br />
for time and resources with <strong>in</strong>dications and warn<strong>in</strong>g (I&W) <strong>in</strong>telligence. This<br />
emphasis is unlikely to change, for several reasons. First, current <strong>in</strong>telligence<br />
report<strong>in</strong>g results <strong>in</strong> significant “face-time” for <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong><br />
with policy makers, who, <strong>in</strong> turn, provide <strong>the</strong> resources that fund and support<br />
community activities. This is a significant contributor to <strong>the</strong> social capital that<br />
<strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong> commands.<br />
The second reason is that <strong>in</strong>-depth research of <strong>the</strong> k<strong>in</strong>d that contributes to<br />
I&W <strong>in</strong>telligence is a long-term <strong>in</strong>vestment whose payoff is often an abstraction.<br />
Not <strong>in</strong>frequently, successful warn<strong>in</strong>gs are taken for granted. Those that<br />
fail, however, may well <strong>in</strong>volve <strong>the</strong> community <strong>in</strong> public recrim<strong>in</strong>ations that<br />
cost <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong> significant social capital. In this sense, <strong>the</strong><br />
<strong>Intelligence</strong> <strong>Community</strong>’s focus on current report<strong>in</strong>g is understandable. The<br />
problem is that produc<strong>in</strong>g current <strong>in</strong>telligence tends to become an all-consum<strong>in</strong>g<br />
activity. The majority of analysts who participated <strong>in</strong> this study said that<br />
<strong>the</strong>ir time was spent on current report<strong>in</strong>g. Unfortunately, this does little to<br />
improve I&W <strong>in</strong>telligence, which requires long-term research, <strong>in</strong>-depth expertise,<br />
adoption of scientific methods, and cont<strong>in</strong>uous performance improvement.<br />
The return for <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong>, <strong>in</strong> terms of social capital,<br />
may be quite limited and even, as noted above, negative. Thus, <strong>the</strong> analytic<br />
area most <strong>in</strong> need of long-term <strong>in</strong>vestment often gets <strong>the</strong> least.<br />
107
CHAPTER NINE<br />
As <strong>the</strong> resources available to <strong>in</strong>telligence analysis are limited, it needs to be<br />
determ<strong>in</strong>ed if those resources are better spent on <strong>the</strong> report<strong>in</strong>g functions of <strong>the</strong><br />
<strong>Intelligence</strong> <strong>Community</strong> or on warn<strong>in</strong>g functions. It also needs to be determ<strong>in</strong>ed<br />
whe<strong>the</strong>r <strong>the</strong>se functions should be performed by <strong>the</strong> same analysts or if<br />
<strong>the</strong>y are two separate career tracks. To make this determ<strong>in</strong>ation, <strong>the</strong> <strong>Intelligence</strong><br />
<strong>Community</strong> will need to <strong>in</strong>vest <strong>in</strong> what I call a Performance Improvement<br />
Infrastructure as well as basic and applied analytic research.<br />
Performance Improvement Infrastructure<br />
The first step <strong>in</strong> improv<strong>in</strong>g job or task-specific performance is <strong>the</strong> establishment<br />
of a formal <strong>in</strong>frastructure designed explicitly to create an iterative performance<br />
improvement process. Such a process would <strong>in</strong>clude:<br />
• measur<strong>in</strong>g actual analytic performance to create basel<strong>in</strong>e data;<br />
• determ<strong>in</strong><strong>in</strong>g ideal analytic performance and standards;<br />
• compar<strong>in</strong>g actual performance with ideal performance;<br />
• identify<strong>in</strong>g performance gaps;<br />
• creat<strong>in</strong>g <strong>in</strong>terventions to improve analytic performance;<br />
• measur<strong>in</strong>g actual analytic performance to evaluate <strong>the</strong> effectiveness of<br />
<strong>in</strong>terventions.<br />
Several organizational, or <strong>in</strong>frastructure, assets should be developed to support<br />
this process. These should <strong>in</strong>clude:<br />
• basic and applied research programs;<br />
• knowledge repositories;<br />
• communities of practice;<br />
• development of performance improvement techniques.<br />
The performance improvement process would be repeated throughout <strong>the</strong><br />
life cycle of an organization <strong>in</strong> order to encourage cont<strong>in</strong>uous improvement.<br />
With <strong>the</strong> <strong>in</strong>frastructure and process <strong>in</strong> place, an organization would be capable<br />
of adapt<strong>in</strong>g to new or chang<strong>in</strong>g environmental conditions.<br />
Infrastructure Requirements<br />
Institutional changes, such as corporate reorganizations, are often enacted<br />
without a clear understand<strong>in</strong>g of <strong>the</strong>ir potential or actual impact. What is most<br />
often miss<strong>in</strong>g <strong>in</strong> such changes is a basic research plan or a systems approach<br />
108
RECOMMENDATIONS<br />
to determ<strong>in</strong>e and predict <strong>the</strong> effect on organizational performance. The same<br />
is true with <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong>. Although <strong>the</strong>re have been numerous<br />
proposals to reorganize <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong>—<strong>in</strong>clud<strong>in</strong>g those that<br />
resulted from <strong>the</strong> hear<strong>in</strong>gs of <strong>the</strong> Kean 9/11 commission—few have addressed<br />
<strong>the</strong> question of why one change would be any more effective than any o<strong>the</strong>r<br />
change. Merely assert<strong>in</strong>g, based on some a priori notion of effectiveness, that<br />
organizational scheme X is more effective than organizational scheme Y is<br />
<strong>in</strong>sufficient evidence. What is needed is a posteriori data, such as case studies,<br />
to support or refute <strong>the</strong> proposed change. 1<br />
Organizational Requirements. Many large organizations distribute performance<br />
improvement responsibilities throughout <strong>the</strong> organization at a supervisory<br />
or midlevel of management, but <strong>the</strong> group most often charged with<br />
collect<strong>in</strong>g and analyz<strong>in</strong>g performance data is <strong>the</strong> human resources department.<br />
This task generally <strong>in</strong>volves develop<strong>in</strong>g task-specific performance standards<br />
and metrics based on expert performance models and <strong>in</strong> accordance with corporate<br />
policy.<br />
The human resources department also becomes <strong>the</strong> central repository for<br />
pre-, periodic, and post-performance measurements. As this department generally<br />
has contact with employees throughout <strong>the</strong>ir careers, this is <strong>the</strong> most<br />
efficient way to manage, analyze, and <strong>in</strong>form senior leadership about aggregate<br />
changes <strong>in</strong> performance over time. Although data are collected at <strong>the</strong><br />
<strong>in</strong>dividual level, it is <strong>the</strong> aggregation of performance data that allows leadership<br />
to determ<strong>in</strong>e <strong>the</strong> effectiveness of any organizational change or job-related<br />
<strong>in</strong>tervention.<br />
Basel<strong>in</strong>e Data. Measur<strong>in</strong>g actual analytic performance is essential to <strong>the</strong><br />
establishment of a data driven performance <strong>in</strong>frastructure. The analysts <strong>in</strong> this<br />
study perceived <strong>the</strong>ir performance to be tied directly to <strong>the</strong> quantity of written<br />
products <strong>the</strong>y produced dur<strong>in</strong>g each review period. Count<strong>in</strong>g <strong>the</strong> number of<br />
analytic publications is one metric, of course, but it is hardly <strong>in</strong>dicative of analytic<br />
quality. Surgeons are a useful example of this problem.. They may<br />
count <strong>the</strong> number of patients <strong>the</strong>y treat, but this metric says more about system<br />
throughput and salesmanship than it does about surgical performance. Unlike<br />
<strong>the</strong> purely cognitive work of <strong>in</strong>telligence analysts, surgeons have <strong>the</strong> advantage<br />
of multiple physical outputs, which makes measurement an easier task.<br />
In particular, surgeons have patient outcomes, or morbidity and mortality<br />
ratios, which become a grounded end-state for all measurements. 2 O<strong>the</strong>r<br />
th<strong>in</strong>gs be<strong>in</strong>g equal, <strong>the</strong>se data <strong>the</strong>n ought to <strong>in</strong>form a prospective patient about<br />
where to take his or her bus<strong>in</strong>ess.<br />
1<br />
William Nolte, a deputy assistant director of central <strong>in</strong>telligence for analysis and production proposed<br />
such an idea <strong>in</strong> “Preserv<strong>in</strong>g Central <strong>Intelligence</strong>: Assessment and Evaluation <strong>in</strong> Support of<br />
<strong>the</strong> DCI ” <strong>in</strong> Studies <strong>in</strong> <strong>Intelligence</strong> 48, no. 3 (2004): 21–25.<br />
109
CHAPTER NINE<br />
For <strong>in</strong>telligence analysts, <strong>the</strong> question may be put as, “What is an analytic<br />
morbidity and mortality ratio?” The process of describ<strong>in</strong>g and identify<strong>in</strong>g<br />
morbidity and mortality, or error and failure <strong>in</strong> analytical terms, is a necessary<br />
step <strong>in</strong> identify<strong>in</strong>g mechanisms to develop, test, and implement performance<br />
improvement <strong>in</strong>terventions. There was little consensus among <strong>the</strong> participants<br />
<strong>in</strong> this study about what comprises failure, or even if failure was possible.<br />
There was greater consensus regard<strong>in</strong>g <strong>the</strong> nature of analytic error, which was<br />
generally thought to be a consequence of analytic <strong>in</strong>accuracy.<br />
Metrics. One could reasonably conclude that compounded errors lead to<br />
analytic failure. Conversely, one could conclude that failure is <strong>the</strong> result of<br />
analytic surprise, that its causes are different from <strong>the</strong> causes of error, and that<br />
it needs to be treated as a separate measurement. This subject is open to<br />
debate and will require fur<strong>the</strong>r research. It is still possible, however, to use<br />
both accuracy and surprise as metrics <strong>in</strong> evaluat<strong>in</strong>g analytic performance on a<br />
case-by-case basis.<br />
The advantage of an error and failure metric is that it is observable <strong>in</strong> a<br />
grounded state separate from <strong>the</strong> analytic process. Any analytic product can<br />
be reviewed to determ<strong>in</strong>e levels of accuracy, and any unexpected event can be<br />
traced back through analytic products to determ<strong>in</strong>e if <strong>the</strong>re was an <strong>in</strong>stance of<br />
surprise.<br />
Once levels of error and failure are calculated, along with measures of output,<br />
it is possible to determ<strong>in</strong>e expert levels of performance and to derive performance<br />
models based on successful processes. In any organization, <strong>the</strong>re<br />
will be those <strong>in</strong>dividuals with <strong>the</strong> greatest output—<strong>in</strong> this case, <strong>the</strong> greatest<br />
number of written products. There will also be <strong>in</strong>dividuals with <strong>the</strong> highest<br />
levels of accuracy—<strong>in</strong> this case, factual consistency. There will also be <strong>in</strong>dividuals<br />
who have <strong>the</strong> lowest <strong>in</strong>cident of surprise—<strong>in</strong> this case, those who generate<br />
<strong>the</strong> greatest number of potential scenarios and track and report<br />
probabilities most reliably. Us<strong>in</strong>g data-driven metrics means that expertise is<br />
not a function of tenure; ra<strong>the</strong>r, it is a function of performance.<br />
Once expert performers are identified, it is possible to capture <strong>the</strong>ir work<br />
processes and to develop performance models based on peak efficiency and<br />
effectiveness with<strong>in</strong> <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong>. Through <strong>the</strong> use of cognitive,<br />
behavioral, and l<strong>in</strong>guistic task analyses, ethnography, and controlled<br />
experiments, it is possible to generate process metrics to identify analytic<br />
methods that are more effective for specific tasks than o<strong>the</strong>r methods. This is<br />
not to say that <strong>the</strong>re is one analytic method that is <strong>the</strong> most effective for <strong>in</strong>tel-<br />
2<br />
Grounded Theory is <strong>the</strong> development of <strong>the</strong>oretical constructs that result from perform<strong>in</strong>g <strong>in</strong>terpretive<br />
analysis on qualitative data ra<strong>the</strong>r than rely<strong>in</strong>g on a priori <strong>in</strong>sights. The <strong>the</strong>ory is <strong>the</strong>n derived<br />
from some grounded data set. Barney Glaser and Anselm Strauss, Discovery of Grounded Theory;<br />
Barney Glaser, Theoretical Sensitivity; Barney Glaser, Basics of Grounded Theory Analysis.<br />
110
RECOMMENDATIONS<br />
ligence analysis; ra<strong>the</strong>r, each type of task will have an analytic method that is<br />
best suited to accomplish<strong>in</strong>g it <strong>in</strong> an efficient and effective manner.<br />
Develop<strong>in</strong>g <strong>the</strong>se metrics is no small task. It is a job that will require<br />
numerous researchers and research programs with<strong>in</strong>, or with access to, <strong>the</strong><br />
<strong>Intelligence</strong> <strong>Community</strong>. These programs will need formal relationships with<br />
human resource departments, analytic divisions, organizational leadership,<br />
and developers of tra<strong>in</strong><strong>in</strong>g and technology <strong>in</strong>terventions <strong>in</strong> order to have a<br />
positive effect on analytic performance.<br />
Research Programs<br />
The results of this research <strong>in</strong>dicate that <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong> needs<br />
to commit itself to performance research that is rigorous, valid (<strong>in</strong> that it measures<br />
what it proposes to measure), and replicable (<strong>in</strong> that <strong>the</strong> method is sufficiently<br />
transparent that anyone can repeat it). With<strong>in</strong> some <strong>in</strong>telligence<br />
organizations, this has been an ongo<strong>in</strong>g process. The problem is that most of<br />
<strong>the</strong> <strong>in</strong>ternal research has concentrated on historical case studies and <strong>the</strong> development<br />
of technological <strong>in</strong>novations. What is miss<strong>in</strong>g is focused study of<br />
human performance with<strong>in</strong> <strong>the</strong> analytic components of <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong>.<br />
Questions about <strong>the</strong> psychology and basic cognitive aptitude of <strong>in</strong>telligence<br />
analysts, <strong>the</strong> effectiveness of any analytic method, <strong>the</strong> effectiveness of<br />
tra<strong>in</strong><strong>in</strong>g <strong>in</strong>terventions, group processes versus <strong>in</strong>dividual processes, environmental<br />
conditions, and cultural-organizational effects need to be addressed.<br />
This effort will require commitment. Researchers will have to be brought<br />
<strong>in</strong>to <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong>, facilities will have to be dedicated to<br />
research<strong>in</strong>g analytic performance, expert analysts will have to give some percentage<br />
of <strong>the</strong>ir time to participat<strong>in</strong>g <strong>in</strong> research studies, managers and supervisors<br />
will have to dedicate time and resources to track<strong>in</strong>g analytic<br />
performance with<strong>in</strong> <strong>the</strong>ir departments, human resource staffs will have to dedicate<br />
time and resources to develop<strong>in</strong>g a performance repository, and <strong>the</strong>re<br />
will have to be formal <strong>in</strong>teraction between researchers and <strong>the</strong> community.<br />
<strong>Analytic</strong> Performance Research. In <strong>the</strong> previous section, I discussed <strong>the</strong><br />
need for analytic standards as part of <strong>the</strong> Performance Improvement Infrastructure.<br />
In terms of a research program, this will require, as a first step, <strong>the</strong><br />
collection of basel<strong>in</strong>e analytic performance data and a clear and measurable<br />
description of ideal analytic behavior. Next, <strong>the</strong>re should be a determ<strong>in</strong>ed<br />
effort by human performance researchers to develop, test, and validate analytic<br />
performance metrics and measurement systems. This will be a lengthy<br />
process. The accuracy and surprise measures suggested <strong>in</strong> this text require<br />
large historical and comparative data sets and are cumbersome and time consum<strong>in</strong>g<br />
to perform. Conduct<strong>in</strong>g behavioral, cognitive, and l<strong>in</strong>guistic task<br />
111
CHAPTER NINE<br />
analyses requires significant research expertise, ample time, and broad organizational<br />
access.<br />
In time, analytic performance research will become a highly specialized<br />
doma<strong>in</strong> and will require cont<strong>in</strong>uous organizational access not normally available<br />
to outsiders. It will become necessary for <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong> to<br />
establish <strong>in</strong>ternal or cooperative research centers <strong>in</strong> order to acquire <strong>the</strong><br />
research expertise necessary to analyze and effect performance improvement.<br />
There are numerous community outreach efforts on which <strong>the</strong>se centers<br />
can be built. Those efforts need to be expanded, however, and those programs<br />
need to <strong>in</strong>clude doma<strong>in</strong>s beyond <strong>the</strong> traditional relationship between <strong>the</strong> <strong>Intelligence</strong><br />
<strong>Community</strong> and political or geographic area experts. 3<br />
Institutional Memory. The results of this research program <strong>in</strong>dicate that<br />
<strong>the</strong>re is a loss of corporate knowledge <strong>in</strong> <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong> due to<br />
employee attrition and <strong>the</strong> lack of a central knowledge repository for captur<strong>in</strong>g<br />
“lessons learned.” A number of <strong>in</strong>dustries and government organizations,<br />
<strong>in</strong>clud<strong>in</strong>g <strong>the</strong> Departments of Defense and Energy and <strong>the</strong> National Aeronautics<br />
and Space Adm<strong>in</strong>istration, already ma<strong>in</strong>ta<strong>in</strong> centers for lessons learned as<br />
an <strong>in</strong>formation hub for its employees. 4<br />
These centers act as <strong>in</strong>formation repositories for successful and unsuccessful<br />
operations and <strong>in</strong>terventions. Their purpose is to reduce <strong>the</strong> amount of<br />
organizational redundancy and levels of error and failure by track<strong>in</strong>g, analyz<strong>in</strong>g,<br />
and report<strong>in</strong>g on after-action reviews and analytic outcome data. 5 The<br />
o<strong>the</strong>r primary function of <strong>the</strong>se repositories is to establish networks for communities<br />
of practice with<strong>in</strong> and among organizations.<br />
Networked communities of practice allow professionals to <strong>in</strong>teract, exchange<br />
methodological <strong>in</strong>formation, post and respond to <strong>in</strong>dividual case studies, and<br />
develop ad hoc teams of experts for specific problem solv<strong>in</strong>g tasks. With simple<br />
search tools, basic database software, and a simple network visualization <strong>in</strong>terface,<br />
any analyst <strong>in</strong> <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong> would be able to identify any<br />
o<strong>the</strong>r expert whose doma<strong>in</strong> specialty was needed to answer a specific question<br />
or solve a specific problem. Ano<strong>the</strong>r advantage of this model is <strong>the</strong> development<br />
of formal and <strong>in</strong>formal mentor<strong>in</strong>g with<strong>in</strong> <strong>the</strong> network. Any novice would<br />
be able to f<strong>in</strong>d an expert with<strong>in</strong> <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong> and establish a relationship<br />
that would be beneficial to both. With appropriate <strong>in</strong>centives, experts<br />
would be encouraged to contribute to <strong>the</strong> network and make available <strong>the</strong>ir time<br />
and expertise for <strong>the</strong> purpose of mentor<strong>in</strong>g.<br />
3<br />
An example of research<strong>in</strong>g <strong>the</strong> validity and reliability of metrics can be found <strong>in</strong> <strong>the</strong> Buros Mental<br />
Measurement Yearbook at <strong>the</strong> Buros Institute of Mental Measurements Web site.<br />
4<br />
See <strong>the</strong> US Army Center for Army Lessons Learned (CALL) Web site, which has l<strong>in</strong>ks to<br />
numerous o<strong>the</strong>r repositories.<br />
5<br />
See Chapter Six for a more detailed explanation of <strong>the</strong> After Action Review process.<br />
112
RECOMMENDATIONS<br />
<strong>Intelligence</strong> analysis, like o<strong>the</strong>r fields of science, is a cognitive process.<br />
Although tools and technologies may be available to assist cognitive processes,<br />
such as measurement devices for physical scientists, technology is ultimately<br />
merely a tool to be designed and developed us<strong>in</strong>g a human-centered<br />
approach. As such, any new technology needs to be a passive tool, employed<br />
by analysts to solve specific problems or answer specific questions, ra<strong>the</strong>r<br />
than a restrictive re<strong>in</strong>terpretation of cognition accord<strong>in</strong>g to <strong>the</strong> rules of b<strong>in</strong>ary<br />
computation and artificial <strong>in</strong>telligence <strong>the</strong>orists. 6<br />
<strong>Analytic</strong> Psychology and Cognition. As evidenced by <strong>the</strong> work of Richards<br />
Heuer and o<strong>the</strong>rs, <strong>the</strong>re is significant research to be conducted <strong>in</strong>to <strong>the</strong> cognitive<br />
mechanisms <strong>in</strong>volved <strong>in</strong> <strong>in</strong>telligence analysis. 7 Understand<strong>in</strong>g and def<strong>in</strong><strong>in</strong>g<br />
<strong>the</strong> heuristics used <strong>in</strong> perform<strong>in</strong>g <strong>in</strong>telligence analysis, as well as<br />
cognitive-load thresholds, multitask<strong>in</strong>g requirements, mechanisms that generate<br />
cognitive biases, and <strong>the</strong> utilization of pattern recognition strategies and<br />
anomaly detection methods are all areas that will prove fundamental to<br />
improv<strong>in</strong>g analytic performance.<br />
In addition to research<strong>in</strong>g basic cognitive functions and <strong>in</strong>telligence analysis,<br />
this area of research will be valuable for understand<strong>in</strong>g how external variables,<br />
such as time constra<strong>in</strong>ts and analytic production methods, affect <strong>the</strong><br />
cognitive process<strong>in</strong>g of <strong>in</strong>dividual analysts. Ano<strong>the</strong>r result will be <strong>the</strong> development<br />
of future employee screen<strong>in</strong>g and selection tools that will match <strong>the</strong><br />
specific cognitive requirements of <strong>in</strong>telligence analysis with each applicant’s<br />
<strong>in</strong>dividual knowledge, skills, and abilities.<br />
Analysts employ cognitive strategies that are time efficient <strong>in</strong> order to cope<br />
with <strong>the</strong> demands of produc<strong>in</strong>g daily written products, but such strategies are<br />
not necessarily <strong>the</strong> most effective analytic methods for <strong>in</strong>creas<strong>in</strong>g analytic<br />
accuracy and decreas<strong>in</strong>g <strong>the</strong> occurrence of analytic surprise. In fact, improv<strong>in</strong>g<br />
analytic accuracy and avoid<strong>in</strong>g surprise may require mutually exclusive<br />
analytic strategies. This l<strong>in</strong>e of <strong>in</strong>quiry will require basel<strong>in</strong>e performance data<br />
generated through <strong>the</strong> development of performance metrics and conducted <strong>in</strong><br />
conjunction with research <strong>in</strong> analytic methodology effectiveness. The results<br />
would <strong>the</strong>n be <strong>in</strong>tegrated <strong>in</strong>to a knowledge repository.<br />
These types of studies will require experimental psychologists and cognitive<br />
scientists work<strong>in</strong>g <strong>in</strong> controlled laboratory environments with consistent<br />
access to work<strong>in</strong>g professional analysts.<br />
6<br />
See Chapter Five for a more detailed description of <strong>the</strong> limitations of technological solutions.<br />
7<br />
Richards J. Heuer, Jr., Psychology of <strong>Intelligence</strong> Analysis; William Brei, Gett<strong>in</strong>g <strong>Intelligence</strong><br />
Right; Isaac Ben-Israel, “Philosophy and Methodology of <strong>Intelligence</strong>: The Logic of Estimate<br />
Process”; Klaus Knorr, Foreign <strong>Intelligence</strong> and <strong>the</strong> Social Sciences; Abraham Ben-Zvi, “The<br />
Study of Surprise Attacks.” See also Marjorie Cl<strong>in</strong>e, Carla Christiansen and Judith Fonta<strong>in</strong>e,<br />
Scholar’s Guide to <strong>Intelligence</strong> Literature.<br />
113
CHAPTER NINE<br />
Development and Validation of <strong>Analytic</strong> Methods. The <strong>Intelligence</strong> <strong>Community</strong><br />
rout<strong>in</strong>ely generates ad hoc methodologies to solve specific analytic<br />
problems or crises. However, once <strong>the</strong> problem has been solved, or <strong>the</strong> crisis<br />
averted, <strong>the</strong> new analytic method may or may not become part of <strong>the</strong> <strong>in</strong>stitutional<br />
memory. Often <strong>the</strong>se new methods are lost and need to be re-created to<br />
address <strong>the</strong> next problem or crisis. In addition, <strong>the</strong>se methods are seldom<br />
tested aga<strong>in</strong>st o<strong>the</strong>r compet<strong>in</strong>g analytic methods for validity or reliability. It is<br />
difficult for an analyst to know which analytic method to employ <strong>in</strong> a given<br />
situation or requirement.<br />
There are obvious <strong>in</strong>efficiencies <strong>in</strong> <strong>the</strong> current model. First, <strong>the</strong>re is <strong>the</strong> loss<br />
of corporate knowledge each time an <strong>in</strong>novative analytic method is generated<br />
and subsequently abandoned. Second, <strong>the</strong>re is no effectiveness test<strong>in</strong>g center<br />
where analytic methods can be compared for specific cases. Although <strong>the</strong>re<br />
are hundreds of analytic strategies, <strong>the</strong>re is no way to determ<strong>in</strong>e which strategy<br />
is <strong>the</strong> most effective for any particular problem set.<br />
The lack of an analytic methodology research agenda leads analysts to<br />
choose methods with which <strong>the</strong>y are most familiar or to choose those dictated<br />
by circumstance, such as deadl<strong>in</strong>es. Moreover, <strong>in</strong>stead of advanc<strong>in</strong>g <strong>the</strong> concept<br />
that <strong>in</strong>telligence analysis is science and needs to be engaged <strong>in</strong> like any<br />
o<strong>the</strong>r scientific discipl<strong>in</strong>e, <strong>the</strong> paucity of effectiveness data supports a deepseated<br />
community bias that analytic methods are idiosyncratic and, <strong>the</strong>refore,<br />
ak<strong>in</strong> to craft.<br />
The development of a research agenda for analytic methodology that is<br />
focused on collect<strong>in</strong>g effectiveness and validation data is <strong>the</strong> first step <strong>in</strong> mov<strong>in</strong>g<br />
<strong>in</strong>telligence analysis from a tradecraft model to a scientific model. This<br />
may be <strong>the</strong> most culturally difficult recommendation to implement: <strong>the</strong>re is<br />
cultural resistance to adopt<strong>in</strong>g a science-based model of <strong>in</strong>telligence analysis<br />
that is rooted <strong>in</strong> <strong>the</strong> traditions, norms, and values of <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong>.<br />
Ano<strong>the</strong>r difficult step will be to <strong>in</strong>troduce effectiveness data and correspond<strong>in</strong>g<br />
analytic methods to <strong>the</strong> community at large and to <strong>in</strong>corporate <strong>the</strong>se<br />
<strong>in</strong> future tra<strong>in</strong><strong>in</strong>g programs.<br />
Tra<strong>in</strong><strong>in</strong>g Effectiveness. Successful analysis demands group cohesion and<br />
<strong>the</strong> implementation of consistent, effective analytic methods with<strong>in</strong> <strong>the</strong> <strong>Intelligence</strong><br />
<strong>Community</strong>. The best way to achieve this is through formal basic and<br />
advanced tra<strong>in</strong><strong>in</strong>g programs. As noted earlier, several agencies with<strong>in</strong> <strong>the</strong><br />
community have <strong>in</strong>vested resources <strong>in</strong> formal tra<strong>in</strong><strong>in</strong>g programs, but <strong>the</strong>se<br />
programs are unique to each agency and are often miss<strong>in</strong>g evaluations of student<br />
performance. Although most formal courses <strong>in</strong>clude a written subjective<br />
evaluation of <strong>the</strong> <strong>in</strong>structor, as well as <strong>the</strong> student’s perception of <strong>the</strong> value of<br />
<strong>the</strong> course, <strong>the</strong> evaluation of student performance has yet to be formalized.<br />
114
RECOMMENDATIONS<br />
Without evaluat<strong>in</strong>g pre<strong>in</strong>tervention, or precourse, performance and follow<strong>in</strong>g<br />
that with a post<strong>in</strong>tervention evaluation, it is difficult to determ<strong>in</strong>e <strong>the</strong> effect that<br />
any tra<strong>in</strong><strong>in</strong>g <strong>in</strong>tervention will have on employee performance. In addition to<br />
formal measurements based on course objectives, it is important to collect performance<br />
data from managers and supervisors to evaluate <strong>the</strong> retention of tra<strong>in</strong><strong>in</strong>g<br />
and <strong>the</strong> impact that tra<strong>in</strong><strong>in</strong>g has had on actual day-to-day performance.<br />
Develop<strong>in</strong>g performance metrics will <strong>in</strong>form and advance <strong>the</strong> tra<strong>in</strong><strong>in</strong>g <strong>in</strong>terventions<br />
currently employed <strong>in</strong> <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong> and will determ<strong>in</strong>e<br />
<strong>the</strong> gap between ideal performance and actual performance. As such,<br />
<strong>the</strong> system is an iterative process of sett<strong>in</strong>g performance standards, measur<strong>in</strong>g<br />
actual performance, design<strong>in</strong>g tra<strong>in</strong><strong>in</strong>g <strong>in</strong>terventions to improve performance,<br />
and evaluat<strong>in</strong>g <strong>the</strong> effects of those <strong>in</strong>terventions on actual performance. The<br />
data derived from <strong>the</strong>se <strong>in</strong>terventions and measurements will <strong>the</strong>n contribute<br />
to <strong>the</strong> growth of <strong>the</strong> knowledge repository and streng<strong>the</strong>n <strong>the</strong> ties created<br />
through <strong>the</strong> communities of practice.<br />
Organizational <strong>Culture</strong> and Effectiveness. Identify<strong>in</strong>g exist<strong>in</strong>g organizational<br />
norms and taboos is <strong>the</strong> first step to creat<strong>in</strong>g an <strong>in</strong>ternal dialogue about<br />
<strong>the</strong> future of an organization and its place <strong>in</strong> a competitive environment. <strong>Culture</strong><br />
drives <strong>the</strong> operations of an organization, determ<strong>in</strong>es <strong>the</strong> people who are<br />
hired, enculturates new employees, establishes standards of behavior and systems<br />
of rewards, shapes an organization’s products, and determ<strong>in</strong>es <strong>the</strong> social<br />
capital that any organization may possess. In short, culture def<strong>in</strong>es an organization’s<br />
identity to itself and to o<strong>the</strong>rs.<br />
Understand<strong>in</strong>g <strong>the</strong> culture of <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong> and analyz<strong>in</strong>g <strong>the</strong><br />
effects of any performance <strong>in</strong>tervention on that culture contributes to <strong>the</strong> evaluation<br />
of <strong>in</strong>tervention effectiveness. Effective performance <strong>in</strong>terventions will<br />
have a positive effect on <strong>the</strong> organization’s culture and become <strong>the</strong>mselves<br />
measurement <strong>in</strong>struments.<br />
Develop<strong>in</strong>g cultural markers to track organizational change and performance<br />
improvement requires basel<strong>in</strong>e ethnographic data and <strong>the</strong> identification<br />
of key cultural <strong>in</strong>dicators. Once identified, cultural <strong>in</strong>dicators such as language<br />
use, norms, and taboos would be measured at regular <strong>in</strong>tervals and<br />
would serve as grounded data to determ<strong>in</strong>e levels of change with<strong>in</strong> <strong>the</strong> organization.<br />
This would permit <strong>in</strong>terventions to be modified before <strong>the</strong>y became<br />
ritualized with<strong>in</strong> <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong>.<br />
The Importance of Access<br />
The improvement of human performance often requires an organization to<br />
change its culture, and organizational leaders seldom possess sufficient power<br />
to mandate cultural change by edict. At best, management can <strong>in</strong>troduce<br />
115
CHAPTER NINE<br />
agents or agencies of change and manage <strong>the</strong>ir organization’s culture <strong>in</strong> <strong>the</strong><br />
same way <strong>the</strong>y manage physical and f<strong>in</strong>ancial resources. An organization’s<br />
culture shapes <strong>in</strong>dividual behavior by establish<strong>in</strong>g norms and taboos and, ultimately,<br />
determ<strong>in</strong>es <strong>the</strong> quality and character of an organization’s products.<br />
<strong>Culture</strong> and product are <strong>in</strong>separable, and one cannot be changed without<br />
affect<strong>in</strong>g <strong>the</strong> o<strong>the</strong>r. The choice confront<strong>in</strong>g any organization is to manage its<br />
<strong>in</strong>stitutional culture or to be managed by it.<br />
There is no s<strong>in</strong>gle path to carry<strong>in</strong>g out <strong>the</strong> research recommended <strong>in</strong> this<br />
work. It could be performed at a s<strong>in</strong>gle center or coord<strong>in</strong>ated through several<br />
specific centers; it could be purely <strong>in</strong>ternal to <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong>; a<br />
cooperative effort among <strong>the</strong> community, academe, and national laboratories;<br />
or some comb<strong>in</strong>ation of <strong>the</strong>se. What is most important to effective implementation<br />
is that <strong>the</strong>re be regular and open access among researchers and <strong>the</strong> <strong>Intelligence</strong><br />
<strong>Community</strong>. This may appear simple enough, but access equals trust,<br />
and trust is difficult to establish <strong>in</strong> any doma<strong>in</strong>. This is especially <strong>the</strong> case<br />
with<strong>in</strong> <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong>. The <strong>Intelligence</strong> <strong>Community</strong> needs to<br />
<strong>in</strong>crease its commitment to community outreach efforts. This study is one<br />
such effort.<br />
Dur<strong>in</strong>g <strong>the</strong> course of my research, <strong>the</strong> value of access and <strong>the</strong> premium <strong>the</strong><br />
community places on trust quickly became evident. At agency after agency,<br />
physical access restrictions, security clearances, forms, <strong>in</strong>terviews, phone<br />
calls, questions, vett<strong>in</strong>g, and more vett<strong>in</strong>g were all signs of <strong>the</strong> value, not of<br />
secrecy per se, but of trust and access. Without this sort of cooperation, this<br />
research would have been impossible, and this is an important lesson that<br />
ought to <strong>in</strong>form future research programs.<br />
116
PART IV<br />
Notes on Methodology<br />
117
CHAPTER TEN<br />
Survey Methodology<br />
This study <strong>in</strong>cluded 489 <strong>in</strong>terviews with <strong>in</strong>telligence professionals, academics,<br />
and researchers throughout <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong>. It also <strong>in</strong>volved<br />
participation <strong>in</strong> <strong>in</strong>telligence tra<strong>in</strong><strong>in</strong>g programs, workshops, and focus groups;<br />
direct observation of <strong>in</strong>telligence analysts perform<strong>in</strong>g <strong>the</strong>ir duties, and participant<br />
observation <strong>in</strong> a variety of analytic tasks. My access was not restricted to<br />
specific people, locations, or organizations. I was allowed to observe, <strong>in</strong>terview,<br />
and participate <strong>in</strong> whatever manner I thought would be most beneficial<br />
to <strong>the</strong> research project.<br />
Unlike o<strong>the</strong>r academic studies of <strong>the</strong> <strong>in</strong>telligence discipl<strong>in</strong>e (case studies or<br />
topic-specific postmortems, for example), this study was process oriented. It<br />
also differed from <strong>the</strong> work of Sherman Kent, Richards Heuer, and o<strong>the</strong>r <strong>in</strong>telligence<br />
professionals concerned with <strong>the</strong> process of <strong>in</strong>telligence analysis. 1<br />
Ra<strong>the</strong>r than hav<strong>in</strong>g an <strong>in</strong>telligence professional look<strong>in</strong>g out to <strong>the</strong> social and<br />
behavioral sciences, this study had a social scientist look<strong>in</strong>g <strong>in</strong> at <strong>the</strong> <strong>in</strong>telligence<br />
profession. Although some of <strong>the</strong> conclusions of this work may be similar<br />
to previous studies, <strong>the</strong> change <strong>in</strong> perspective has also led to some<br />
different f<strong>in</strong>d<strong>in</strong>gs.<br />
It is important to keep <strong>in</strong> m<strong>in</strong>d that cultural anthropology is a qualitative<br />
discipl<strong>in</strong>e and that, <strong>in</strong> general, its f<strong>in</strong>d<strong>in</strong>gs are descriptive and explanatory<br />
ra<strong>the</strong>r than <strong>in</strong>ferential or predictive. The use of ethnographic methods to<br />
describe a culture, <strong>the</strong> environment <strong>in</strong> which that culture operates, and <strong>the</strong><br />
work processes that culture has adopted is designed to generate testable <strong>the</strong>ory<br />
that can be <strong>in</strong>vestigated experimentally or quasi-experimentally us<strong>in</strong>g o<strong>the</strong>r<br />
1<br />
Sherman Kent, Strategic <strong>Intelligence</strong> for American World Policy; Richards J. Heuer, Jr., Psychology<br />
of <strong>Intelligence</strong> Analysis.<br />
119
CHAPTER TEN<br />
research methodologies. Additionally, ethnography is used to identify and<br />
describe <strong>the</strong> <strong>in</strong>fluence of different variables on cultural phenomena, aga<strong>in</strong><br />
with a focus on develop<strong>in</strong>g testable <strong>the</strong>ory. Unlike more quantitative discipl<strong>in</strong>es,<br />
cultural anthropology is not traditionally employed experimentally to<br />
test <strong>the</strong>ory or to generate predictive measures of statistical significance.<br />
The f<strong>in</strong>d<strong>in</strong>gs <strong>in</strong> this work describe <strong>the</strong> data collected dur<strong>in</strong>g this study, but<br />
<strong>the</strong>y do not <strong>in</strong>dicate <strong>the</strong> weight or general statistical effect of any one variable<br />
as opposed to any o<strong>the</strong>r variable. Although a s<strong>in</strong>gle variable might have more<br />
effect on <strong>the</strong> error or failure rate of <strong>in</strong>telligence analysis, fur<strong>the</strong>r quantitative<br />
research will be needed to determ<strong>in</strong>e those statistical values. Without additional<br />
quantitative support, it may not be possible to generalize from <strong>the</strong>se<br />
f<strong>in</strong>d<strong>in</strong>gs.<br />
Methodology<br />
This study used an applied anthropological methodology for <strong>the</strong> collection<br />
and analysis of qualitative data. 2 A traditional approach to ethnography, <strong>the</strong><br />
descriptive documentation of liv<strong>in</strong>g cultures, was modified for use <strong>in</strong> post<strong>in</strong>dustrial<br />
organizational sett<strong>in</strong>gs. 3 This method <strong>in</strong>cluded conduct<strong>in</strong>g <strong>in</strong>terviews,<br />
directly observ<strong>in</strong>g analysts perform<strong>in</strong>g <strong>the</strong>ir jobs, participat<strong>in</strong>g <strong>in</strong> analytic<br />
tasks and tra<strong>in</strong><strong>in</strong>g, and conduct<strong>in</strong>g focus groups. The sett<strong>in</strong>gs for this<br />
research <strong>in</strong>cluded <strong>the</strong> 14 members of <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong>, related<br />
government agencies, universities, th<strong>in</strong>k tanks, national laboratories, <strong>the</strong><br />
National Archives and related presidential libraries, and private and corporate<br />
locations.<br />
The background data were collected us<strong>in</strong>g a Q-sort literature review method,<br />
which is discussed <strong>in</strong> more detail <strong>in</strong> Chapters Three and Eleven. This procedure<br />
was followed by semi-structured <strong>in</strong>terviews, direct observation, participant<br />
observation, and focus groups. The Q-sort method was employed specifically<br />
because of its utility for develop<strong>in</strong>g taxonomic categories. 4<br />
The identity of <strong>the</strong> research participants will not be revealed. Participant<br />
responses and observational data ga<strong>the</strong>red dur<strong>in</strong>g <strong>the</strong> research process have<br />
been tabulated and made anonymous or aggregated accord<strong>in</strong>g to context and<br />
2<br />
Erve Chambers, Applied Anthropology: A Practical Guide; Alexander Erv<strong>in</strong>, Applied Anthropology:<br />
Tools and Perspectives for Contemporary Practice.<br />
3<br />
Russell Bernard, Research Methods <strong>in</strong> Anthropology: Qualitative and Quantitative Approaches;<br />
Robert Bogdan, Participant Observation <strong>in</strong> Organizational Sett<strong>in</strong>gs; Norman Denz<strong>in</strong> and Yvonna<br />
L<strong>in</strong>coln, Handbook of Qualitative Research; Jean Schensul and Margaret LeCompte, Ethnographer’s<br />
Toolkit. Vol. I - Vol. VII; James Spradley, Participant Observation; Robert Y<strong>in</strong>, Case<br />
Study Research: Design and Methods.<br />
4<br />
William Stephenson, The Study of Behavior: Q-Technique and its Methodology.<br />
120
SURVEY METHODOLOGY<br />
content and, thus, are not attributable to any specific <strong>in</strong>dividual. This is not<br />
simply <strong>the</strong> result of security procedures with<strong>in</strong> <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong>; it<br />
is also <strong>the</strong> professional obligation of every member of <strong>the</strong> American Anthropological<br />
Association, as stated <strong>in</strong> <strong>the</strong> American Anthropological Association<br />
Code of Ethics. 5<br />
The <strong>in</strong>terview technique employed <strong>in</strong> this study was semi-structured. Several<br />
specific questions about <strong>the</strong> participant’s perception of <strong>the</strong> nature of <strong>in</strong>telligence,<br />
<strong>the</strong> analytic process, <strong>the</strong> <strong>in</strong>telligence production cycle, and<br />
<strong>in</strong>telligence errors and failures were standard throughout <strong>the</strong> <strong>in</strong>terviews. O<strong>the</strong>r<br />
questions, specific to <strong>the</strong> <strong>in</strong>dividual’s job responsibilities, were tailored to<br />
each respondent. This method allowed for a more open-ended approach,<br />
which surveys and highly structured <strong>in</strong>terviews do not. The semi-structured<br />
method is more ak<strong>in</strong> to an open conversation (with consistent data collection<br />
constructs and prob<strong>in</strong>g questions) than to a formal <strong>in</strong>terview, which helps put<br />
<strong>the</strong> respondents at ease and makes <strong>the</strong> entire process seem somewhat less contrived.<br />
Access to <strong>in</strong>terview participants was made possible through <strong>the</strong> Center for<br />
<strong>the</strong> Study of <strong>Intelligence</strong>. Individuals at CSI <strong>in</strong>troduced me to <strong>the</strong>ir contacts<br />
throughout <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong>, <strong>in</strong>clud<strong>in</strong>g active and retired senior<br />
analysts, managers and senior leadership, as well as to academics and<br />
researchers. The various <strong>in</strong>telligence-tra<strong>in</strong><strong>in</strong>g centers put me <strong>in</strong> touch with<br />
new hires and novice analysts. Each <strong>in</strong>terviewee was asked to make recommendations<br />
and provide contact <strong>in</strong>formation for o<strong>the</strong>rs who might be <strong>in</strong>terested<br />
<strong>in</strong> participat<strong>in</strong>g <strong>in</strong> this research project. In addition, numerous<br />
<strong>in</strong>terviewees were approached without a formal or <strong>in</strong>formal <strong>in</strong>troduction from<br />
a previous participant. Only four of <strong>the</strong> 489 <strong>in</strong>dividuals contacted to date have<br />
decl<strong>in</strong>ed to participate <strong>in</strong> this study. This constitutes a participation rate of<br />
greater than 99 percent, which is unusually high for this type of research.<br />
Although a participation rate this high may be an artifact of <strong>the</strong> sampl<strong>in</strong>g<br />
method or of an organizational pressure to participate, it also may <strong>in</strong>dicate a<br />
general desire with<strong>in</strong> <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong> to support performance<br />
improvement research.<br />
Unlike random sampl<strong>in</strong>g, purposive sampl<strong>in</strong>g is an attempt to collect data<br />
from specific data sources. In anthropological studies, purposive sampl<strong>in</strong>g is<br />
regularly used to address specific issues and to answer specific questions.<br />
Normally, this approach requires f<strong>in</strong>d<strong>in</strong>g a “key <strong>in</strong>formant” or someone on <strong>the</strong><br />
<strong>in</strong>side of a specific culture who will become <strong>the</strong> researcher’s ally and access<br />
agent. In this particular study, <strong>the</strong> CSI staff acted as access agents to <strong>the</strong> <strong>Intelligence</strong><br />
<strong>Community</strong> at large.<br />
5<br />
American Anthropological Association, Code of Ethics of <strong>the</strong> American Anthropological Association.<br />
121
CHAPTER TEN<br />
Rely<strong>in</strong>g on such a “social network” sampl<strong>in</strong>g method for collect<strong>in</strong>g <strong>in</strong>terview<br />
data does pose potential statistical biases. 6 The likelihood that each new<br />
<strong>in</strong>terviewee was referred to me because of a friendly relationship with a previous<br />
<strong>in</strong>terviewee may mean that those references are “like m<strong>in</strong>ded” and not<br />
necessarily representative of <strong>the</strong> population of <strong>in</strong>telligence professionals. In<br />
order to counteract that bias, efforts were also made to enlist <strong>in</strong>dividuals without<br />
any social network-based <strong>in</strong>troduction. The “cold” contacts were <strong>in</strong>formed<br />
of <strong>the</strong> nature of <strong>the</strong> research project, its sponsorship, and its goals, given reference<br />
<strong>in</strong>formation for verification, and <strong>the</strong>n <strong>in</strong>vited to participate. The “cold”<br />
contact <strong>in</strong>terviewees were also asked to make recommendations and provide<br />
contact <strong>in</strong>formation for o<strong>the</strong>rs who might be <strong>in</strong>terested <strong>in</strong> participat<strong>in</strong>g <strong>in</strong> <strong>the</strong><br />
study.<br />
This strategy was used <strong>in</strong> an attempt to reduce <strong>the</strong> affects of sampl<strong>in</strong>g bias<br />
by generat<strong>in</strong>g parallel social network samples. The figure below is a visual<br />
representation of a parallel social-network sampl<strong>in</strong>g model. The central, or<br />
first-order, node on <strong>the</strong> left is a “cold” contact or unknown <strong>in</strong>dividual who recommends<br />
several second-order contacts, each represented as a node with<strong>in</strong> <strong>the</strong><br />
left box. The second-order “cold” contacts <strong>the</strong>n make additional recommendations<br />
for third-order contacts, and so on. The central (first-order) node on <strong>the</strong><br />
right is a “hot” contact or a known <strong>in</strong>dividual who recommends several second-order<br />
contacts, each represented as a node with<strong>in</strong> <strong>the</strong> right box. The second-order<br />
“hot” contacts<br />
Social Network Mapp<strong>in</strong>g<br />
<strong>the</strong>n make recommendations<br />
for third-order contacts,<br />
and so on.<br />
In many <strong>in</strong>stances, <strong>the</strong><br />
contacts from both social<br />
network samples overlapped<br />
or converged on<br />
specific <strong>in</strong>dividuals, as<br />
represented by <strong>the</strong> overlapped<br />
fourth-order nodes<br />
<strong>in</strong> <strong>the</strong> central column.<br />
There are several possible<br />
explanations for this convergence.<br />
It may <strong>in</strong>dicate<br />
that <strong>the</strong>re are a number of<br />
respected “thought leaders” <strong>in</strong> <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong> whom each contact<br />
believed I should <strong>in</strong>terview for this project, or <strong>the</strong> convergence of nodes<br />
6 Social network sampl<strong>in</strong>g is also known as “snowball” sampl<strong>in</strong>g <strong>in</strong> sociology<br />
and psychology.<br />
122
SURVEY METHODOLOGY<br />
might merely serve to emphasize <strong>the</strong> small size of <strong>Intelligence</strong> <strong>Community</strong>. In<br />
any case, this approach to sampl<strong>in</strong>g may help to ameliorate <strong>the</strong> sampl<strong>in</strong>g bias<br />
<strong>in</strong>herent <strong>in</strong> qualitative research.<br />
In addition to semi-structured <strong>in</strong>terviews, both direct and participant observation<br />
data collection methods were employed. The direct observation method<br />
<strong>in</strong>volved watch<strong>in</strong>g <strong>Intelligence</strong> <strong>Community</strong> analysts perform <strong>the</strong>ir tasks <strong>in</strong><br />
both actual and tra<strong>in</strong><strong>in</strong>g environments, record<strong>in</strong>g <strong>the</strong> physical and verbal <strong>in</strong>teractions<br />
<strong>the</strong>y had with one ano<strong>the</strong>r, and observ<strong>in</strong>g <strong>the</strong> steps used to create <strong>in</strong>telligence<br />
products. Direct observation occurred over <strong>the</strong> course of two years by<br />
observ<strong>in</strong>g 325 <strong>in</strong>dividual analysts and teams of analysts perform<strong>in</strong>g <strong>the</strong>ir specific<br />
tasks. The data collected from observ<strong>in</strong>g <strong>the</strong> 325 analysts were not<br />
<strong>in</strong>cluded <strong>in</strong> <strong>the</strong> semi-structured <strong>in</strong>terview data because I did not use <strong>the</strong> formal<br />
semi-structured <strong>in</strong>terview process to structure those <strong>in</strong>teractions. These observational<br />
data were recorded separately <strong>in</strong> field notes and used for triangulat<strong>in</strong>g<br />
<strong>the</strong> f<strong>in</strong>d<strong>in</strong>gs from <strong>the</strong> <strong>in</strong>terviews.<br />
The participant observation method is employed to give <strong>the</strong> researcher a<br />
“first-person” understand<strong>in</strong>g of <strong>the</strong> context and nuances associated with a task<br />
and <strong>the</strong> culture <strong>in</strong> which that task occurs. Although <strong>the</strong> researcher possesses<br />
only an approximation of <strong>the</strong> knowledge and understand<strong>in</strong>g of <strong>the</strong> actual practitioners<br />
of <strong>the</strong> task and <strong>the</strong>ir culture, this “first-person” perspective can lead<br />
<strong>the</strong> researcher to new <strong>in</strong>sights and new hypo<strong>the</strong>ses.<br />
Dur<strong>in</strong>g this study, <strong>the</strong> participant observation was conducted dur<strong>in</strong>g analytic<br />
production cycles, scenario development, and red cell exercises. This<br />
<strong>in</strong>cluded monitor<strong>in</strong>g my own analytic strategies, <strong>the</strong> analytic strategies of o<strong>the</strong>rs<br />
as diagramed or verbalized, <strong>the</strong> physical and verbal social <strong>in</strong>teractions<br />
among <strong>the</strong> participants, <strong>the</strong> environment <strong>in</strong> which <strong>the</strong> tasks occurred, and <strong>the</strong><br />
steps used to create a f<strong>in</strong>al <strong>in</strong>telligence product. These data, along with notes<br />
on social dynamics, taboos, and social power, were recorded <strong>in</strong> field notes and<br />
created a separate data source for triangulation.<br />
With modern anthropology, <strong>the</strong>se data normally would be captured on film,<br />
audiotape, or <strong>in</strong> some digital format. Due to <strong>the</strong> security requirements of <strong>the</strong><br />
<strong>Intelligence</strong> <strong>Community</strong>, however, <strong>the</strong> data were captured only <strong>in</strong> <strong>the</strong> written<br />
form of field notes. As is <strong>the</strong> case with <strong>the</strong> field notes, <strong>the</strong> identity of <strong>the</strong> <strong>in</strong>terview<br />
participants will not be disclosed. This is <strong>in</strong> keep<strong>in</strong>g with both <strong>the</strong> security<br />
practices of <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong> and <strong>the</strong> professional standards<br />
described <strong>in</strong> <strong>the</strong> American Anthropological Association Statement on <strong>the</strong><br />
Confidentiality of Field Notes. 7<br />
The data from <strong>the</strong> <strong>in</strong>terviews were analyzed us<strong>in</strong>g a method called <strong>in</strong>terpretational<br />
analysis. 8 This approach <strong>in</strong>cluded segment<strong>in</strong>g <strong>the</strong> <strong>in</strong>terview data <strong>in</strong>to<br />
7<br />
American Anthropological Association, Statement on <strong>the</strong> Confidentiality of Field Notes.<br />
123
CHAPTER TEN<br />
analytic units (or units of mean<strong>in</strong>g), develop<strong>in</strong>g categories, cod<strong>in</strong>g <strong>the</strong> analytic<br />
units <strong>in</strong>to content areas, and group<strong>in</strong>g <strong>the</strong> analytic units <strong>in</strong>to categories. From<br />
<strong>the</strong>se categories, general trends and specific <strong>in</strong>stances can be identified. As<br />
noted, <strong>the</strong> direct and participant observational data were analyzed separately<br />
<strong>in</strong> order to triangulate <strong>the</strong> f<strong>in</strong>d<strong>in</strong>gs from <strong>the</strong> <strong>in</strong>terview data. The purpose of<br />
us<strong>in</strong>g multiple data sources for triangulation is to uncover <strong>in</strong>ternal <strong>in</strong>consistencies<br />
<strong>in</strong> <strong>the</strong> data, to cross-check those <strong>in</strong>consistencies with <strong>the</strong> available literature,<br />
and to verify <strong>the</strong> content validity for each category.<br />
Demographics<br />
As of this writ<strong>in</strong>g, 489 semi-structured <strong>in</strong>terviews have been conducted with<br />
active and retired <strong>in</strong>telligence professionals, <strong>in</strong>telligence technology researchers,<br />
academics who teach <strong>the</strong> <strong>in</strong>telligence discipl<strong>in</strong>e or have published <strong>in</strong> it, and<br />
consumers of <strong>in</strong>tel-<br />
Distribution of Interviews by Doma<strong>in</strong> ligence products. 9<br />
Of <strong>the</strong> 489 <strong>in</strong>dividuals<br />
<strong>in</strong>terviewed,<br />
4%<br />
11% <strong>Intelligence</strong> 70-percent were<br />
Academics<br />
15%<br />
Professionals<br />
newly hired, active,<br />
or retired <strong>in</strong>telligence<br />
Technology Researchers<br />
profession-<br />
als; 15-percent were<br />
70%<br />
Consumers<br />
academics; 11-percent<br />
were <strong>in</strong>telligence<br />
technology<br />
researchers; and <strong>the</strong><br />
rema<strong>in</strong><strong>in</strong>g four percent were policy makers or senior consumers of <strong>in</strong>telligence<br />
products. The graph here shows <strong>the</strong> distribution of <strong>in</strong>terviews by percentage for<br />
each professional category.<br />
The table below lists each professional category and <strong>the</strong> correspond<strong>in</strong>g total<br />
number (N) of <strong>in</strong>dividuals <strong>in</strong>terviewed. The <strong>in</strong>telligence professional category<br />
is fur<strong>the</strong>r divided <strong>in</strong>to three sub-groups. The “novice” sub-group <strong>in</strong>cludes new<br />
hires and those with less than two years of experience. 10 The “active” sub-<br />
8<br />
Leonard Bickman and Debra Rog, Handbook of Applied Social Research Methods; Meredith<br />
Gall et al., Educational Research; Jonathan Gross, Measur<strong>in</strong>g <strong>Culture</strong>: A Paradigm for <strong>the</strong> Analysis<br />
of Social Organization; Ernest House, Evaluat<strong>in</strong>g with Validity; Jerome Kirk and Marc<br />
Miller, Reliability and Validity <strong>in</strong> Qualitative Research, Qualitative Research Methods, Volume 1;<br />
Delbert Miller, Handbook of Research Design and Social Measurement; Michael Patton, Qualitative<br />
Evaluation and Research Methods; Peter Rossi and Howard Freeman, Evaluation. A Systematic<br />
Approach.<br />
9<br />
Additional <strong>in</strong>terviews are be<strong>in</strong>g conducted.<br />
124
SURVEY METHODOLOGY<br />
group <strong>in</strong>cludes all those currently work<strong>in</strong>g <strong>in</strong> <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong><br />
with more than two years of experience. The “retired” sub-group <strong>in</strong>cludes<br />
those who have spent more than fifteen years <strong>in</strong> <strong>the</strong> <strong>in</strong>telligence profession<br />
and have s<strong>in</strong>ce gone on to ei<strong>the</strong>r full retirement or o<strong>the</strong>r organizations outside<br />
of <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong>.<br />
Of <strong>the</strong> 345 <strong>in</strong>telligence professionals <strong>in</strong>terviewed, 20 percent were novices,<br />
65 percent were active, and 15 percent were retired. The active and retired<br />
sub-groups <strong>in</strong>clude senior managers.<br />
Interview Categories and Numbers<br />
Category<br />
N<br />
<strong>Intelligence</strong> Professionals 345<br />
Novice (60)<br />
Active (233)<br />
Retired (52)<br />
Academics 73<br />
Technology Researchers 53<br />
Consumers 18<br />
Total Interviewed489<br />
In order to assure anonymity for <strong>the</strong> participants, I have created broader jobrelated<br />
functional categories and associated <strong>the</strong> number of <strong>in</strong>dividuals <strong>in</strong>terviewed<br />
with <strong>the</strong> broader categories ra<strong>the</strong>r than l<strong>in</strong>k<strong>in</strong>g <strong>the</strong>m to specific organizations<br />
with<strong>in</strong> <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong>. This is <strong>in</strong> contrast to aggregat<strong>in</strong>g <strong>the</strong><br />
agencies accord<strong>in</strong>g to each agency’s specific mission, process, or product.<br />
Although not an official member of <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong>, <strong>the</strong> Drug<br />
Enforcement Adm<strong>in</strong>istration is <strong>in</strong>cluded because of its <strong>in</strong>telligence function and<br />
resources. The table on <strong>the</strong> next page shows how I aggregated <strong>the</strong> agencies <strong>in</strong>to<br />
National-Technical, Defense, and Law Enforcement-Homeland Security categories<br />
accord<strong>in</strong>g to <strong>the</strong> professional functions of <strong>in</strong>terview participants.<br />
10<br />
The use of two years as a divide between novice and active is derived from <strong>the</strong> total amount of<br />
experience it is possible to ga<strong>in</strong> <strong>in</strong> that time. See <strong>the</strong> discussion of expertise <strong>in</strong> Chapter Five.<br />
125
CHAPTER TEN<br />
Agency Aggregation Accord<strong>in</strong>g to Interviewee Job-type<br />
National-Technical Defense Law Enforcement-<br />
Homeland Security<br />
Central <strong>Intelligence</strong> Defense <strong>Intelligence</strong> Department of<br />
Agency Agency Homeland Security<br />
National Security Army <strong>Intelligence</strong> Federal Bureau of<br />
Agency<br />
Investigation<br />
National Air Force <strong>Intelligence</strong> Department of Energy<br />
Reconnaissance Office<br />
National Geospatial Navy <strong>Intelligence</strong> Department of Treasury<br />
<strong>Intelligence</strong> Agency<br />
Department of State Mar<strong>in</strong>e Corps Drug Enforcement<br />
(INR) <strong>Intelligence</strong> Adm<strong>in</strong>istration<br />
The figure below shows <strong>the</strong> distribution of <strong>in</strong>telligence professionals <strong>in</strong>terviewed<br />
for this study accord<strong>in</strong>g to each broader functional category. Of <strong>the</strong><br />
345 <strong>in</strong>telligence professionals <strong>in</strong>terviewed, 214 work with<strong>in</strong> <strong>the</strong> National-<br />
Technical <strong>Intelligence</strong> category, 76 <strong>in</strong> <strong>the</strong> Defense <strong>Intelligence</strong> category, and<br />
55 <strong>in</strong> <strong>the</strong> Law Enforcement—Homeland Security category.<br />
126
CHAPTER ELEVEN<br />
Q-Sort Methodology<br />
Observations always <strong>in</strong>volve <strong>the</strong>ory.<br />
Edw<strong>in</strong> Hubble 1<br />
As described earlier, this work reflects triangulation of <strong>the</strong> data derived<br />
from <strong>the</strong> literature Q-sort, <strong>in</strong>terview responses, and observations. 2 The data<br />
<strong>in</strong>clude 489 <strong>in</strong>terviews, direct and participant observation of 325 analysts perform<strong>in</strong>g<br />
<strong>the</strong>ir jobs, participation <strong>in</strong> a variety of analytic tasks, and focus<br />
groups conducted to generate <strong>the</strong> taxonomy of variables that guided this study.<br />
The first Q-sort of <strong>the</strong> data was aggregated accord<strong>in</strong>g to <strong>the</strong> function of<br />
each <strong>in</strong>telligence organization, as listed <strong>in</strong> Table 1. The data were <strong>the</strong>n analyzed<br />
to determ<strong>in</strong>e response context accord<strong>in</strong>g to job type and to develop variable<br />
categories.<br />
The organizational Q-sort generated <strong>the</strong> broad variable group<strong>in</strong>gs used to<br />
create <strong>the</strong> second Q-sort parameters. The variable categories that emerged<br />
dur<strong>in</strong>g <strong>the</strong> <strong>in</strong>terpretive analysis of <strong>the</strong> first Q-sort of <strong>the</strong> data were compiled<br />
aga<strong>in</strong>, and a second Q-sort was performed based on those categories. The data<br />
was <strong>the</strong>n aggregated accord<strong>in</strong>g to categorical or variable group<strong>in</strong>gs of <strong>the</strong> second<br />
Q-sort, Table 2.<br />
The use of two separate Q-sort strategies generated <strong>the</strong> variables and <strong>the</strong>n<br />
de-contextualized <strong>the</strong> data <strong>in</strong> order to f<strong>in</strong>d consistent trends throughout <strong>the</strong><br />
<strong>Intelligence</strong> <strong>Community</strong>. That is, this strategy resulted <strong>in</strong> broad categories of<br />
f<strong>in</strong>d<strong>in</strong>gs that apply across many agencies. In those cases where <strong>in</strong>terview and<br />
1<br />
Edw<strong>in</strong> Hubble discovered <strong>the</strong> first evidence to support <strong>the</strong> Big Bang <strong>the</strong>ory that <strong>the</strong> universe is<br />
expand<strong>in</strong>g and that <strong>the</strong> Milky Way is not <strong>the</strong> only galaxy <strong>in</strong> <strong>the</strong> universe. He also developed <strong>the</strong><br />
Hubble Galaxy Classification System and Hubble’s Law (<strong>the</strong> far<strong>the</strong>r away a galaxy is from Earth,<br />
<strong>the</strong> faster its motion away from Earth). Edw<strong>in</strong> Hubble, The Realm of <strong>the</strong> Nebulae.<br />
2<br />
William Stephenson, The Study of Behavior: Q-Technique and its Methodology.<br />
127
CHAPTER ELEVEN<br />
observational data could have been sorted <strong>in</strong>to several categories, I based <strong>the</strong><br />
placement of <strong>the</strong> data on <strong>the</strong> question that generated <strong>the</strong> <strong>in</strong>terview response.<br />
Table 1. Q-Sort 1. Data Group<strong>in</strong>g Accord<strong>in</strong>g to Organizational Function.<br />
National – Technical Defense Law Enforcement –<br />
Homeland Security<br />
Central <strong>Intelligence</strong> Defense <strong>Intelligence</strong> Department of<br />
Agency Agency Homeland Security<br />
National Security Army <strong>Intelligence</strong> Federal Bureau of<br />
Agency<br />
Investigation<br />
National Air Force <strong>Intelligence</strong> Department of Energy<br />
Reconnaissance Office<br />
National Geospatial Navy <strong>Intelligence</strong> Department of <strong>the</strong><br />
<strong>Intelligence</strong> Agency<br />
Treasury<br />
Department of State Mar<strong>in</strong>e Corps Drug Enforcement<br />
(INR) <strong>Intelligence</strong> Adm<strong>in</strong>istration<br />
In several <strong>in</strong>stances throughout <strong>the</strong> text, <strong>the</strong> quotes that were used may well<br />
fit <strong>in</strong> a number of o<strong>the</strong>r categories. Once <strong>the</strong> data were sorted by variable, <strong>the</strong><br />
cod<strong>in</strong>g and context identifier notes were removed from all data <strong>in</strong> order to<br />
assure participant anonymity, <strong>in</strong> keep<strong>in</strong>g with <strong>the</strong> American Anthropological<br />
Association Code of Ethics, section III, A. 3<br />
Table 2. Q-Sort 2. Data Group<strong>in</strong>g Accord<strong>in</strong>g to Variable Categories.<br />
Time<br />
Constra<strong>in</strong>ts<br />
<strong>Analytic</strong><br />
Methods<br />
Organizational<br />
Norms<br />
<strong>Analytic</strong><br />
Identity<br />
<strong>Analytic</strong><br />
Tra<strong>in</strong><strong>in</strong>g<br />
Products Tradecraft Taboos Reportorial Formal<br />
Interactions Science Biases Academic Informal<br />
The quotes that appear throughout <strong>the</strong> text are exemplars from each variable<br />
category and <strong>in</strong>dicate trends found <strong>in</strong> <strong>the</strong> data-set. Although <strong>the</strong> exemplar<br />
quotes are not universal, nor are <strong>the</strong>y necessarily subject to generalization,<br />
<strong>the</strong>y do represent consistent f<strong>in</strong>d<strong>in</strong>gs from <strong>the</strong> <strong>in</strong>terview and observation data.<br />
Utiliz<strong>in</strong>g this approach to develop <strong>the</strong>ory is similar to <strong>the</strong> method <strong>in</strong> which<br />
grounded <strong>the</strong>ory is employed <strong>in</strong> sociology, specifically, us<strong>in</strong>g grounded data<br />
to generate <strong>the</strong>ory ra<strong>the</strong>r than us<strong>in</strong>g some a priori technique. The significant<br />
advantage to this approach is that <strong>the</strong> <strong>the</strong>ory is directly tied to data, provid<strong>in</strong>g<br />
it additional validity. Ano<strong>the</strong>r advantage is that <strong>the</strong> <strong>in</strong>dividuals who allowed<br />
me to <strong>in</strong>terview and observe <strong>the</strong>m are given some voice <strong>in</strong> <strong>the</strong> f<strong>in</strong>al product by<br />
way of direct quotes, which also provides some qualitative context.<br />
3<br />
American Anthropological Association, Code of Ethics of <strong>the</strong> American Anthropological Association.<br />
128
CHAPTER TWELVE<br />
The “File-Drawer” Problem and Calculation of Effect Size<br />
The file-drawer problem appears to have two causes: <strong>the</strong> reluctance of<br />
researchers to report <strong>the</strong>ir null results and <strong>the</strong> reluctance of professional journal<br />
editors to <strong>in</strong>clude studies whose results fail to reach statistical significance.<br />
Such studies rema<strong>in</strong> <strong>in</strong> <strong>the</strong> “file-drawers” of <strong>the</strong> researchers. How<br />
much would <strong>the</strong>se <strong>in</strong>accessible studies affect <strong>the</strong> results of our meta-analysis?<br />
The answer seems to be not much. 1<br />
Effect Size - Difference Between Two Means<br />
1<br />
Gene Glass, and Barry McGaw, “Choice of <strong>the</strong> Metric for Effect Size <strong>in</strong> Meta-Analysis”; Larry<br />
Hedges, “Estimation of Effect Size from a Series of Independent Experiments”; Larry Hedges and<br />
Ingram Olk<strong>in</strong>, “Vote-Count<strong>in</strong>g Methods <strong>in</strong> Research Syn<strong>the</strong>sis.”<br />
129
CHAPTER TWELVE<br />
Effect size is usually def<strong>in</strong>ed as <strong>the</strong> difference between <strong>the</strong> means of two<br />
groups divided by <strong>the</strong> standard deviation of <strong>the</strong> control group ⎛ Χ−Χc ⎞ .2<br />
⎜ ∆= e ⎟<br />
⎝ σ c<br />
⎠<br />
Effect sizes calculated <strong>in</strong> this way estimate <strong>the</strong> difference between two<br />
group means measured <strong>in</strong> control group standard deviations as seen <strong>in</strong> <strong>the</strong> figure<br />
above. Glass et al. suggest that <strong>the</strong> choice of <strong>the</strong> denom<strong>in</strong>ator is critical<br />
and that choices o<strong>the</strong>r than <strong>the</strong> control group standard deviation are defensible.<br />
3 However, <strong>the</strong>y endorse <strong>the</strong> standard choice of us<strong>in</strong>g <strong>the</strong> control group<br />
standard deviation.<br />
Alternatively, Hedges and Olk<strong>in</strong> show that, for every effect size, both <strong>the</strong><br />
bias and variance of its estimate are smaller when standard deviation is<br />
obta<strong>in</strong>ed by pool<strong>in</strong>g <strong>the</strong> sample variance of two groups <strong>in</strong>stead of us<strong>in</strong>g <strong>the</strong><br />
control group standard deviation by itself. 4 An effect size based on a pooled<br />
standard deviation estimates <strong>the</strong> difference between two group means measured<br />
<strong>in</strong> standard deviations estimated for <strong>the</strong> full population from which both<br />
⎛ Χe − Χc ⎞<br />
experimental and control groups are drawn: ⎜ g = ⎟ , 5 where<br />
⎝ S ⎠<br />
2 2 6<br />
S is <strong>the</strong> pooled standard deviation: (Ne − 1)(Se ) + (Nc − 1)(Sc )<br />
S = .<br />
e + N Nc − 2<br />
Most commentators suggest that effect sizes can be treated as descriptive<br />
statistics and entered <strong>in</strong>to standard tests for statistical significance. Hedges<br />
and Olk<strong>in</strong> have shown that <strong>the</strong> error variance around estimates of effect size is<br />
<strong>in</strong>versely proportional to <strong>the</strong> sample size of <strong>the</strong> studies from which <strong>the</strong> effect<br />
sizes are drawn. If <strong>the</strong> effect size <strong>in</strong> any review is drawn from studies employ<strong>in</strong>g<br />
widely different sample sizes, <strong>the</strong>n <strong>the</strong> heterogeneity of variance among<br />
effect sizes prohibits <strong>the</strong>ir use <strong>in</strong> conventional t-tests, analyses of variance,<br />
and o<strong>the</strong>r <strong>in</strong>ferential tests. This is <strong>the</strong> case <strong>in</strong> most of <strong>the</strong>se reviews; <strong>the</strong>refore,<br />
effect sizes reported <strong>in</strong> this study are treated only with descriptive statistics.<br />
The effect sizes for computer-based tra<strong>in</strong><strong>in</strong>g range from 0.20 to 0.46 depend<strong>in</strong>g<br />
on <strong>the</strong> population. 7 The effect size for distance <strong>in</strong>struction (television) is<br />
Control Mean, σc = Control<br />
Standard Deviation<br />
3<br />
Gene Glass, Barry McGaw and Mary Lee Smith, Meta-Analysis <strong>in</strong> Social Research.<br />
4<br />
Larry Hedges and Ingram Olk<strong>in</strong>, Statistical Methods for Meta-Analysis.<br />
5<br />
g=Hedge's Effect Size, S=Hedge's Pooled Standard Deviation<br />
6<br />
Ne=Number of experimental subjects, Nc=Number of control subjects, Se=Standard deviation<br />
of experimental group, Sc=Standard deviation of control group<br />
2<br />
∆=Glass's Effect Size, Χe = Experimental Mean, Χc =<br />
130
CHAPTER TWELVE<br />
0.15 and for <strong>in</strong>teractive videodiscs, <strong>the</strong> effect sizes range from 0.17 to 0.66<br />
depend<strong>in</strong>g on <strong>the</strong> population. 8 The effect size for flight simulation is 0.54 and<br />
<strong>the</strong> effect size for tutorials range from 0.25 to 0.41 depend<strong>in</strong>g on <strong>the</strong> presentation<br />
of <strong>the</strong> tutorial material. 9<br />
Although <strong>the</strong> effect sizes for <strong>in</strong>structional technology range from 0.15 to 0.66<br />
standard deviations, <strong>the</strong>y all report favorable f<strong>in</strong>d<strong>in</strong>gs when compared to conventional<br />
<strong>in</strong>struction.<br />
There are many possible<br />
explanations for<br />
<strong>the</strong> differences <strong>in</strong><br />
<strong>in</strong>structional technology<br />
effectiveness; it<br />
might be <strong>the</strong> result of<br />
population differences,<br />
system differences,<br />
<strong>in</strong>teractivity<br />
or <strong>in</strong>dividualization.<br />
From a purely utilitarian<br />
po<strong>in</strong>t of view,<br />
<strong>the</strong> reason may not<br />
be all that important.<br />
If, at <strong>the</strong> very least,<br />
us<strong>in</strong>g <strong>in</strong>structional<br />
technology forces <strong>the</strong><br />
producer to reth<strong>in</strong>k<br />
<strong>the</strong> content of <strong>the</strong><br />
course to match <strong>the</strong><br />
delivery system, <strong>the</strong>n<br />
revisit<strong>in</strong>g <strong>the</strong> pedagogy may be enough to produce <strong>the</strong> positive effect sizes. Whatever<br />
reason for <strong>the</strong> changes <strong>in</strong> effectiveness, <strong>the</strong> use of <strong>in</strong>structional technology<br />
saves <strong>in</strong>structional time, overhead costs, and results <strong>in</strong> a higher level of achievement<br />
for <strong>the</strong> students <strong>in</strong> a variety of doma<strong>in</strong>s.<br />
7<br />
The abbreviations <strong>in</strong> figure one: CBT=Computer Based Tra<strong>in</strong><strong>in</strong>g, DI=Distance Instruction,<br />
IVD=Interactive Video Disc, SIM=Simulation. More than 300 research studies were used to<br />
develop <strong>the</strong>se effect sizes, see Chen-L<strong>in</strong> Kulik., James Kulik and Barbara Shwalb, “Effectiveness<br />
of Computer-Based Adult Education: A Meta-Analysis”; Chen-L<strong>in</strong> Kulik and James Kulik,<br />
“Effectiveness of Computer-Based Education <strong>in</strong> Colleges”; Rob Johnston and J. Dexter Fletcher,<br />
A Meta-Analysis of <strong>the</strong> Effectiveness of Computer-Based Tra<strong>in</strong><strong>in</strong>g for Military Instruction.<br />
8<br />
Godw<strong>in</strong> Chu and Wilbur Schramm, Learn<strong>in</strong>g from Television; J. Dexter Fletcher Effectiveness<br />
and Cost of Interactive Videodisc Instruction <strong>in</strong> Defense Tra<strong>in</strong><strong>in</strong>g and Education; J. Dexter<br />
Fletcher, “Computer-Based Instruction: Costs and Effectiveness.”<br />
9<br />
R. T. Hays, J. W. Jacobs, C. Pr<strong>in</strong>ce and E. Salas, “Flight Simulator Tra<strong>in</strong><strong>in</strong>g Effectiveness: A Meta-<br />
Analysis”; Peter Cohen, James Kulik and Chen-L<strong>in</strong> Kulik, “Educational Outcomes of Tutor<strong>in</strong>g.”<br />
131
APPENDIX<br />
Selected Literature<br />
<strong>Intelligence</strong> Tools and Techniques<br />
Jerome K. Clauser and Sandra M. Weir, <strong>Intelligence</strong> Research Methodology:<br />
An Introduction to Techniques and Procedures for Conduct<strong>in</strong>g Research <strong>in</strong><br />
Defense <strong>Intelligence</strong> (Wash<strong>in</strong>gton, DC: US Defense <strong>Intelligence</strong> School,<br />
1976).<br />
Stanley Feder, “FACTIONS and Policon: New Ways to Analyze Politics,”<br />
(1987) <strong>in</strong> H. Bradford Westerfield, ed., Inside <strong>CIA</strong>’s Private World: Declassified<br />
Articles from <strong>the</strong> Agency’s Internal Journal (New Haven: Yale University<br />
Press, 1995).<br />
Craig Fleisher and Babette Bensoussan, Strategic and Competitive Analysis.<br />
Methods and Techniques for Analyz<strong>in</strong>g Bus<strong>in</strong>ess Competition (Upper Saddle<br />
River, NJ: Pearson Education, 2003).<br />
Leonard Fuld, The New Competitor <strong>Intelligence</strong>. The Complete Resource for<br />
F<strong>in</strong>d<strong>in</strong>g, Analyz<strong>in</strong>g, and Us<strong>in</strong>g Information About Your Competitors (New<br />
York: John Wiley & Sons, 1995).<br />
Ronald Garst, ed., A Handbook of <strong>Intelligence</strong> Analysis, 2nd ed (Wash<strong>in</strong>gton,<br />
DC: Defense <strong>Intelligence</strong> College, 1989).<br />
R. Hopk<strong>in</strong>s, Warn<strong>in</strong>gs of Revolution: A Case Study of El Salvador (Wash<strong>in</strong>gton,<br />
DC: Center for <strong>the</strong> Study of <strong>Intelligence</strong>, 1980) TR 80-100012.<br />
Morgan Jones, The Th<strong>in</strong>ker’s Toolkit: 14 Powerful Techniques for Problem<br />
Solv<strong>in</strong>g (New York, NY: Times Bus<strong>in</strong>ess, 1998).<br />
Larry Kahaner, Competitive <strong>Intelligence</strong>. How to Ga<strong>the</strong>r, Analyze, and Use<br />
Information to Move Your Bus<strong>in</strong>ess to <strong>the</strong> Top (New York: Touchstone,<br />
Simon and Schuster, 1996).<br />
133
APPENDIX<br />
Jonathan Lockwood and K. Lockwood, “The Lockwood <strong>Analytic</strong>al Method<br />
for Prediction (LAMP),” Defense <strong>Intelligence</strong> Journal 3, no. 2 (1994): 47–<br />
74.<br />
Douglas J. MacEach<strong>in</strong>, et al., The Tradecraft of Analysis: Challenge and<br />
Change <strong>in</strong> <strong>the</strong> <strong>CIA</strong> (Wash<strong>in</strong>gton, DC: Consortium for <strong>the</strong> Study of <strong>Intelligence</strong>,<br />
1994).<br />
Don McDowell, Strategic <strong>Intelligence</strong>: A Handbook for Practitioners, Managers<br />
and Users (Canberra, Australia: Istana Enterprises, 1998).<br />
Marilyn Peterson, Applications <strong>in</strong> Crim<strong>in</strong>al Analysis (Westport, CT: Greenwood<br />
Press, 1995).<br />
———, Bob Morehouse, and Richard Wright, eds. <strong>Intelligence</strong> 2000: Revis<strong>in</strong>g<br />
<strong>the</strong> Basic Elements (Sacramento CA: LEIU and IALEIA, 2000).<br />
John Pierce, “Some Ma<strong>the</strong>matical Methods for <strong>Intelligence</strong> Analysis,” Studies<br />
<strong>in</strong> <strong>Intelligence</strong> 21 (Summer 1977, declassified): 1–19.<br />
John Prescott and Stephen Miller, eds. Proven Strategies <strong>in</strong> Competitive <strong>Intelligence</strong><br />
(New York: John Wiley & Sons, 2001).<br />
Edw<strong>in</strong> Sapp, “Decision Trees,” Studies <strong>in</strong> <strong>Intelligence</strong> 18 (W<strong>in</strong>ter 1974,<br />
declassified): 45–57.<br />
David Schum, The Evidential Foundations of Probabilistic Reason<strong>in</strong>g (Evanston,<br />
IL: Northwestern University Press, 1994).<br />
Jack Zlotnick, “Bayes’ Theorem for <strong>Intelligence</strong> Analysis,” (1972), <strong>in</strong> H.<br />
Bradford Westerfield, ed. Inside <strong>CIA</strong>’s Private World: Declassified Articles<br />
from <strong>the</strong> Agency’s Internal Journal (New Haven: Yale University Press,<br />
1995).<br />
Cognitive Processes and <strong>Intelligence</strong><br />
Edgar Johnson, Effects of Data Source Reliability on Inference (Alexandria,<br />
VA: Army Research Institute for <strong>the</strong> Behavioral and Social Sciences, Technical<br />
Paper #251, 1974).<br />
Robert Mathams, “The <strong>Intelligence</strong> Analyst’s Notebook,” <strong>in</strong> Douglas Dearth<br />
and R. Thomas Goodden, eds., Strategic <strong>Intelligence</strong>: Theory and Application,<br />
2nd ed. (Wash<strong>in</strong>gton, DC: Jo<strong>in</strong>t Military <strong>Intelligence</strong> College, 1995).<br />
Avi Shlaim, “Failures <strong>in</strong> National <strong>Intelligence</strong> Estimates: The Case of <strong>the</strong> Yom<br />
Kippur War,” World Politics 28, no. 3 (1976): 348–80.<br />
Tools and Techniques as Cognitive Processes<br />
Bruce D. Berkowitz and Allan E. Goodman, Best Truth: <strong>Intelligence</strong> <strong>in</strong> <strong>the</strong><br />
Information Age (New Haven: Yale University Press, 2000).<br />
———, and Allan E. Goodman, Strategic <strong>Intelligence</strong> for American National<br />
Security (Pr<strong>in</strong>ceton: Pr<strong>in</strong>ceton University Press, 1989).<br />
134
APPENDIX<br />
Sherman Kent, “Words of Estimative Probability,” Sherman Kent and <strong>the</strong><br />
Board of National Estimates: Collected Essays (Wash<strong>in</strong>gton, DC: Center for<br />
<strong>the</strong> Study of <strong>Intelligence</strong>, 1994).<br />
Lisa Krizan, <strong>Intelligence</strong> Essentials for Everyone (Wash<strong>in</strong>gton, DC: Jo<strong>in</strong>t Military<br />
<strong>Intelligence</strong> College, 1999).<br />
<strong>Intelligence</strong> Analysis as Individual Cognitive Process<br />
Alexander Butterfield, The Accuracy of <strong>Intelligence</strong> Assessment: Bias, Perception,<br />
and Judgment <strong>in</strong> Analysis and Decision (Newport, RI: Naval War<br />
College, 1993).<br />
Richards J. Heuer, Jr., Psychology of <strong>Intelligence</strong> Analysis (Wash<strong>in</strong>gton, DC:<br />
Center for <strong>the</strong> Study of <strong>Intelligence</strong>, 1999).<br />
Robert Jervis, Perception and Misperception <strong>in</strong> International Politics (Pr<strong>in</strong>ceton:<br />
Pr<strong>in</strong>ceton University Press, 1977).<br />
Ephraim Kam, Surprise Attack: The Victim’s Perspective (Cambridge, MA:<br />
Harvard University Press, 1988).<br />
Gerald<strong>in</strong>e Krotow, The Impact of Cognitive Feedback on <strong>the</strong> Performance of<br />
<strong>Intelligence</strong> Analysts (Monterey, CA: Naval Postgraduate School, 1992).<br />
Mark Lowenthal, <strong>Intelligence</strong>: From Secrets to Policy (Wash<strong>in</strong>gton, DC: CQ,<br />
2000).<br />
David Moore, Creat<strong>in</strong>g <strong>Intelligence</strong>: Evidence and Inference <strong>in</strong> <strong>the</strong> Analysis<br />
Process. Unpublished Masters Thesis (Wash<strong>in</strong>gton DC: Jo<strong>in</strong>t Military <strong>Intelligence</strong><br />
College, 2002).<br />
J. R. Thompson, R. Hopf-Weichel, and R. Geiselman, The Cognitive Bases of<br />
<strong>Intelligence</strong> Analysis (Alexandria, VA: Army Research Institute, Research<br />
Report 1362, 1984).<br />
Error<br />
Daniela Busse and Chris Johnson “Us<strong>in</strong>g a Cognitive Theoretical Framework<br />
to Support Accident Analysis,” <strong>in</strong> Proceed<strong>in</strong>gs of <strong>the</strong> Work<strong>in</strong>g Group on<br />
Human Error, Safety, and System Development (Seattle, WA: International<br />
Federation for Information Process<strong>in</strong>g, 1998, 36–43).<br />
Peter Fishburn, Utility Theory for Decision Mak<strong>in</strong>g (New York: John Wiley,<br />
1970).<br />
Paul Fitts and Michael Posner, Human Performance (Belmont, CA: Brooks<br />
Cole, 1967).<br />
John Hammond, Ralph Keeney, and Howard Raiffa, “The Hidden Traps <strong>in</strong><br />
Decision Mak<strong>in</strong>g,” Harvard Bus<strong>in</strong>ess Review, Sept.– Oct. 1998: 47–58.<br />
135
APPENDIX<br />
Hede Helfrich, “Human Reliability from a Social-Psychological Perspective,”<br />
International Journal Human-Computer Studies 50 (1999): 193–212.<br />
Erik Hollnagel, “The Phenotype of Erroneous Actions,” International Journal<br />
of Man-Mach<strong>in</strong>e Studies 39 (1993): 1–32.<br />
Daniel Kahneman and Amos Tversky, “Prospect Theory: An Analysis of<br />
Decision Under Risk,” Econometrica 47, no. 2 (1979): 263–91.<br />
Duncan Luce and Howard Raiffa, “Utility Theory” <strong>in</strong> Paul Moser, ed. Rationality<br />
<strong>in</strong> Action: Contemporary Approaches (New York: Cambridge University<br />
Press, 1990).<br />
Judith Orasanu and Lynne Mart<strong>in</strong>, “Errors <strong>in</strong> Aviation Decision Mak<strong>in</strong>g: A<br />
Factor <strong>in</strong> Accidents and Incidents” <strong>in</strong> Proceed<strong>in</strong>gs of <strong>the</strong> Work<strong>in</strong>g Group on<br />
Human Error, Safety, and System Development (Seattle, WA: International<br />
Federation for Information Process<strong>in</strong>g, 1998, 100–107).<br />
James Reason, Human Error (New York: Cambridge University Press, 1990).<br />
John Senders and Neville Moray, Human Error: Cause, Prediction and<br />
Reduction (Hillsdale, NJ: Earlbaum, 1991).<br />
Neville Stanton and Sarah Stevenage, “Learn<strong>in</strong>g to Predict Human Error:<br />
Issues of Acceptability, Reliability and Validity,” Ergonomics 41, no. 11<br />
(1998): 1737–56.<br />
Language and Cognition<br />
Terry K. Au, “Ch<strong>in</strong>ese and English Counterfactuals: The Sapir-Whorf<br />
Hypo<strong>the</strong>sis Revisited,” Cognition 15 (1983): 155-187.<br />
Alfred Bloom, The L<strong>in</strong>guistic Shap<strong>in</strong>g of Thought: A Study <strong>in</strong> <strong>the</strong> Impact of<br />
Language on Th<strong>in</strong>k<strong>in</strong>g <strong>in</strong> Ch<strong>in</strong>a and <strong>the</strong> West (Hillsdale, NJ: Lawrence<br />
Erlbaum Associates, 1981).<br />
Lera Boroditsky, “Does Language Shape Thought? Mandar<strong>in</strong> and English<br />
Speakers’ Conceptions of Time,” Cognitive Psychology 43 (2001):1–22.<br />
Ernest Cassirer, “Le Langage et le Monde des Objets,” Journal de Psychologie<br />
Nomale et Pathologique 30, 1(4) (1933): 18–44.<br />
John Gumperz and Dell Hymes, Directions <strong>in</strong> Sociol<strong>in</strong>guistics: The Ethnography<br />
of Communication (Oxford: B. Blackwell, 1986).<br />
———, and Stephen Lev<strong>in</strong>son, Reth<strong>in</strong>k<strong>in</strong>g L<strong>in</strong>guistic Relativity (Cambridge:<br />
Cambridge University Press, 1996).<br />
Harry Hoijer, “The Relation of Language to <strong>Culture</strong>” <strong>in</strong> Alfred Kroeber, ed.<br />
Anthropology Today (Chicago: University of Chicago Press, 1953).<br />
Wilhelm von Humboldt “Uber die Verschiedenheit des Menschlichen Sprachbaues”<br />
<strong>in</strong> B. Bohr, ed. Gesammelte Schriften (Berl<strong>in</strong>: Verlag, 1907).<br />
Earl Hunt and Franca Agnoli, “The Whorfian Hypo<strong>the</strong>sis. A Cognitive Psychology<br />
Perspective,” Psychological Review 98, no. 3 (1991): 377–89.<br />
136
APPENDIX<br />
Dell Hymes, ed. Language <strong>in</strong> <strong>Culture</strong> and Society: A Reader <strong>in</strong> L<strong>in</strong>guistics<br />
and Anthropology (New York: Harper and Row, 1977).<br />
Penny Lee, The Whorf Theory Complex: A Critical Reconstruction (Amsterdam:<br />
Benjam<strong>in</strong>s, 1996).<br />
Lisa Liu, “Reason<strong>in</strong>g Counterfactually <strong>in</strong> Ch<strong>in</strong>ese: Are There any Obstacles?”<br />
Cognition 21 (1985): 239–70.<br />
John Lucy, Grammatical Categories and Cognition (Cambridge: Cambridge<br />
University Press, 1992) and Language Diversity and Thought (Cambridge:<br />
Cambridge University Press, 1992).<br />
George Miller and Philip Johnson-Laird, Language and Perception (Cambridge,<br />
MA: Harvard University Press, 1976).<br />
Richard Nisbett and Ara Norenzayan, “<strong>Culture</strong> and Cognition” <strong>in</strong> Douglas<br />
Med<strong>in</strong> and Hal Pashler, eds. Stevens’ Handbook of Experimental Psychology,<br />
3rd ed. (New York: John Wiley & Sons, 2002).<br />
Edward Sapir, Language (New York: Harcourt Brace, 1949).<br />
Jost Trier, Der deutsche Wortschatz im S<strong>in</strong>nbezirk des Verstandes (Heidelberg:<br />
Carl W<strong>in</strong>ter, 1931).<br />
Leo Weisgerber, Vom Weltbild der Deutschen Sprache (Dusseldorf: Padagogischer<br />
Verlag Schwann, 1953).<br />
Benjam<strong>in</strong> Whorf, Language, Thought, and Reality (New York: John Wiley,<br />
1956).<br />
137
BIBLIOGRAPHY<br />
Published Sources<br />
Ak<strong>in</strong>, Omer. Models of Architectural Knowledge. London: Pion, 1980.<br />
Allen, Charles, Comments of <strong>the</strong> Associate Director of Central <strong>Intelligence</strong> for<br />
Collection at a Public Sem<strong>in</strong>ar on <strong>Intelligence</strong> at Harvard University, spr<strong>in</strong>g<br />
2000. (.)<br />
Allison, Graham T. Essence of Decision: Expla<strong>in</strong><strong>in</strong>g <strong>the</strong> Cuban Missile Crisis.<br />
Boston: Little, Brown and Company, 1971.<br />
American Anthropological Association. Code of Ethics of <strong>the</strong> American<br />
Anthropological Association. Arl<strong>in</strong>gton, VA, 1998. (http://www.aaanet.org/<br />
committees/ethics/ethcode.htm.)<br />
———. Statement on <strong>the</strong> Confidentiality of Field Notes. Arl<strong>in</strong>gton, VA, 2003.<br />
(http://www.aaanet.org/stmts/fieldnotes.htm.)<br />
American College of Radiology. Written communications between staff and<br />
Johnston, 2002.<br />
Arvey, Richard D., and James E. Campion. “The Employment Interview: A<br />
Summary of Recent Research,” Personnel Psychology 35 (1982): 281–322.<br />
Ashcraft, M.H. Fundamentals of Cognition. New York: Addison Wesley<br />
Longman, 1998.<br />
Atk<strong>in</strong>son, R. C., and H. A. Wilson. Computer-Assisted Instruction: A Book of<br />
Read<strong>in</strong>gs, New York: Academic Press, 1969.<br />
Au, Terry K. “Ch<strong>in</strong>ese and English Counterfactuals: The Sapir-Whorf<br />
Hypo<strong>the</strong>sis Revisited,” Cognition 15 (1983): 155–87.<br />
Barr, A., M. Beard, and R. C. Atk<strong>in</strong>son. “A Rationale and Description of a<br />
CAI Program to Teach <strong>the</strong> BASIC Programm<strong>in</strong>g Language.” Instructional<br />
Science 4 (1975): 1–31.<br />
Bayes, Thomas. “An Essay Toward Solv<strong>in</strong>g a Problem <strong>in</strong> <strong>the</strong> Doctr<strong>in</strong>e of<br />
Chances,” Philosophical Transactions of <strong>the</strong> Royal Society of London 53:<br />
370–418. (http://plato.stanford.edu/entries/bayes-<strong>the</strong>orum/#Bib.)<br />
Ben-Israel, Isaac. “Philosophy and Methodology of <strong>Intelligence</strong>: The Logic of<br />
Estimate Process,” <strong>Intelligence</strong> and National Security 4, no. 4 (1989): 660–718.<br />
Ben-Zvi, Abraham. “The Study of Surprise Attacks,” British Journal of International<br />
Studies 5 (1979).<br />
139
BIBLIOGRAPHY<br />
Berkowitz, Bruce D., and Allan E. Goodman. Best Truth: <strong>Intelligence</strong> <strong>in</strong> <strong>the</strong><br />
Information Age. New Haven, CT: Yale University Press, 2000.<br />
———. Strategic <strong>Intelligence</strong> for American National Security. Pr<strong>in</strong>ceton, NJ:<br />
Pr<strong>in</strong>ceton University Press, 1989.<br />
Bernard, Russell. Research Methods <strong>in</strong> Anthropology: Qualitative and Quantitative<br />
Approaches. Thousand Oaks, CA: Sage, 1994.<br />
Betts, Richard K. “Policy-Makers and <strong>Intelligence</strong> Analysts: Love, Hate or<br />
Indifference,” <strong>Intelligence</strong> and National Security 3, no. 1 (January 1988):<br />
184–89.<br />
———. Surprise Attack. Wash<strong>in</strong>gton, DC: The Brook<strong>in</strong>gs Institution, 1982.<br />
Bickman, Leonard, and Debra Rog. Handbook of Applied Social Research<br />
Methods. Thousand Oaks, CA: Sage, 1997.<br />
Bloom, Alfred. The L<strong>in</strong>guistic Shap<strong>in</strong>g of Thought: A Study <strong>in</strong> <strong>the</strong> Impact of<br />
Language on Th<strong>in</strong>k<strong>in</strong>g <strong>in</strong> Ch<strong>in</strong>a and <strong>the</strong> West. Hillsdale, NJ: Lawrence<br />
Erlbaum Associates, 1981.<br />
Bloom, Benjam<strong>in</strong> S., ed. Taxonomy of Educational Objectives: The Classification<br />
of Educational Goals: Handbook I, Cognitive Doma<strong>in</strong>. New York:<br />
Longmans, Green, 1956.<br />
———. “The 2 Sigma Problem: The Search for Methods of Group Instruction<br />
as Effective as One-to-One Tutor<strong>in</strong>g,” Educational Researcher, 13 (1984):<br />
4–16.<br />
Boas, Franz. “An Anthropologist’s Credo,” The Nation 147 (1938): 201–4.<br />
Bogdan, Robert. Participant Observation <strong>in</strong> Organizational Sett<strong>in</strong>gs. Syracuse,<br />
NY: Syracuse University Press, 1972.<br />
Boroditsky, Lera. “Does Language Shape Thought? Mandar<strong>in</strong> and English<br />
Speakers’ Conceptions of Time,” Cognitive Psychology 43 (2001): 1–22.<br />
Bourdieu, Pierre. “The Forms of Capital” <strong>in</strong> John Richardson, ed. Handbook<br />
of Theory and Research for <strong>the</strong> Sociology of Education. New York: Greenwood<br />
Press, 1986.<br />
Brei, William. Gett<strong>in</strong>g <strong>Intelligence</strong> Right: The Power of Logical Procedure.<br />
Wash<strong>in</strong>gton, DC: Jo<strong>in</strong>t Military <strong>Intelligence</strong> College, 1996.<br />
Br<strong>in</strong>berg, David, and Joseph McGrath. Validity and <strong>the</strong> Research Process.<br />
Beverly Hills, CA: Sage, 1985.<br />
Brown, J.S., R.R. Burton, and J. DeKleer. “Pedagogical, Natural Language,<br />
and Knowledge Eng<strong>in</strong>eer<strong>in</strong>g <strong>in</strong> SOPHIE I, II, and III” <strong>in</strong> D. Sleeman and J.S.<br />
Brown, eds. Intelligent Tutor<strong>in</strong>g Systems, New York: Academic Press, 1982.<br />
Busse, Daniela, and Chris Johnson. “Us<strong>in</strong>g a Cognitive Theoretical Framework<br />
to Support Accident Analysis,” Proceed<strong>in</strong>gs of <strong>the</strong> Work<strong>in</strong>g Group on<br />
Human Error, Safety, and System Development. Seattle, WA: International<br />
Federation for Information Process<strong>in</strong>g, 1998, 36–43.<br />
Butterfield, Alexander. The Accuracy of <strong>Intelligence</strong> Assessment: Bias, Perception,<br />
and Judgment <strong>in</strong> Analysis and Decision. Newport, RI: Naval War<br />
College, 1993, AD-A266.<br />
140
BIBLIOGRAPHY<br />
Camerer, Col<strong>in</strong>, and Eric Johnson. “The Process-Performance Paradox <strong>in</strong><br />
Expert Judgment: How Can Experts Know so Much and Predict so Badly?”<br />
<strong>in</strong> K. Anders Ericsson and Jacqui Smith, eds. Toward a General Theory of<br />
Expertise: Prospects and Limit. Cambridge: Cambridge University Press,<br />
1991.<br />
Cannon-Bowers, Janis, Eduardo Salas, and S. Converse. “Shared Mental<br />
Models <strong>in</strong> Expert Team Decision Mak<strong>in</strong>g” <strong>in</strong> J. N. John Castellan, ed. Current<br />
Issues <strong>in</strong> Individual and Group Decision Mak<strong>in</strong>g. Hillsdale, NY:<br />
Lawrence Erlbaum Associates, 1983.<br />
Capra, Fritjof. “Criteria of Systems Th<strong>in</strong>k<strong>in</strong>g,” Futures, October 1985: 475–<br />
78.<br />
Carbonell, J. R. “AI <strong>in</strong> CAI: An Artificial <strong>Intelligence</strong> Approach to Computer-<br />
Assisted Instruction” <strong>in</strong> IEEE Transactions on Man-Mach<strong>in</strong>e Systems, 11<br />
(1970): 190–202; J. D. Fletcher and M. R. Rockway loc. cit.<br />
Carm<strong>in</strong>es, Edward, and Richard Zeller. Reliability and Validity Assessment.<br />
Newbury Park, CA: Sage, 1991.<br />
Cartwright, Dorw<strong>in</strong>, and Alv<strong>in</strong> Zander. Group Dynamics: Research and Theory.<br />
New York: Harper & Row, 1960.<br />
Cascio, Wayne F. Applied Psychology <strong>in</strong> Human Resource Management. 5th<br />
ed. Englewood Cliffs, NJ: Prentice Hall, 1998.<br />
Cassirer, Ernest. “Le Langage et le Monde des Objets,” Journal de Psychologie<br />
Nomale et Pathologique 30, 1(4) (1933): 18–44.<br />
Central <strong>Intelligence</strong> Agency. A Consumer’s Guide to <strong>Intelligence</strong>. Wash<strong>in</strong>gton,<br />
DC: Central <strong>Intelligence</strong> Agency, 1993.<br />
———. Factbook on <strong>Intelligence</strong> (http://www.cia.gov/cia/publications/facttell/<strong>in</strong>dex.html.)<br />
Chambers, Erve. Applied Anthropology: A Practical Guide. Prospect Heights,<br />
IL: Waveland Press, 1989.<br />
Chase, William G. “Spatial Representations of Taxi Drivers” <strong>in</strong> Don Rogers<br />
and John Slobada, eds. Acquisition of Symbolic Skills. New York: Plenum,<br />
1983.<br />
———, and Herbert Simon. “Perception <strong>in</strong> Chess,” Cognitive Psychology 4<br />
(1973): 55–81.<br />
———, and K. Anders Ericsson. “Skill and Work<strong>in</strong>g Memory” <strong>in</strong> Gordon<br />
Bower, ed. The Psychology of Learn<strong>in</strong>g and Motivation. New York: Academic<br />
Press, 1982.<br />
Chi, Michelene, Robert Glaser, and Ernest Rees. “Expertise <strong>in</strong> Problem Solv<strong>in</strong>g”<br />
<strong>in</strong> Robert Sternberg, ed. Advances <strong>in</strong> <strong>the</strong> Psychology of Human <strong>Intelligence</strong>.<br />
Hillsdale, NJ: Lawrence Erlbaum Associates, 1982.<br />
———, Paul Feltovich, and Robert Glaser. “Categorization and Representation<br />
of Physics Problems by Experts and Novices,” Cognitive Science 5<br />
(1981): 121–25.<br />
141
BIBLIOGRAPHY<br />
Chu, Godw<strong>in</strong>, and Wilbur Schramm. Learn<strong>in</strong>g from Television: What <strong>the</strong><br />
Research Says. Stanford, CA: Stanford University, Institute for Communication<br />
Research, 1968.<br />
Clauser, Jerome K., and Sandra M. Weir. <strong>Intelligence</strong> Research Methodology:<br />
An Introduction to Techniques and Procedures for Conduct<strong>in</strong>g Research <strong>in</strong><br />
Defense <strong>Intelligence</strong>. Wash<strong>in</strong>gton, DC: US Defense <strong>Intelligence</strong> School,<br />
1976.<br />
Cl<strong>in</strong>e, Marjorie, Carla Christiansen, and Judith Fonta<strong>in</strong>e. Scholar’s Guide to<br />
<strong>Intelligence</strong> Literature: Bibliography of <strong>the</strong> Russell J. Bowen Collection.<br />
Frederick, MD: University Publications of America, 1983.<br />
Coch, Lester, and John French. “Overcom<strong>in</strong>g Resistance to Change,” Dorw<strong>in</strong><br />
Cartwright and Alv<strong>in</strong> Zander, eds. Group Dynamics: Research and Theory.<br />
New York: Harper & Row, 1960.<br />
Cohen, Peter, James Kulik, and Chen-L<strong>in</strong> Kulik. “Educational Outcomes of<br />
Tutor<strong>in</strong>g: A Meta-Analysis of F<strong>in</strong>d<strong>in</strong>gs,” American Educational Research<br />
Journal 19 (1982): 237–48.<br />
Cushman, Philip. Construct<strong>in</strong>g <strong>the</strong> Self, Construct<strong>in</strong>g America. Read<strong>in</strong>g, MA:<br />
Addison-Wesley, 1995.<br />
Davis, Jack. “Combat<strong>in</strong>g M<strong>in</strong>dset,” Studies <strong>in</strong> <strong>Intelligence</strong> 35 (W<strong>in</strong>ter 1991):<br />
13–18.<br />
Dawes, Robyn. Written communications between Dawes and Johnston, 2002.<br />
———. “A Case Study of Graduate Admissions: Application of Three Pr<strong>in</strong>ciples<br />
of Human Decision Mak<strong>in</strong>g,” American Psychologist 26 (1971): 180–<br />
88.<br />
———, David Faust, and Paul Meehl. “Cl<strong>in</strong>ical Versus Actuarial Judgment,”<br />
Science 243 (1989): 1668–74.<br />
Dearth, Douglas, and R. Thomas Goodden, eds. Strategic <strong>Intelligence</strong>: Theory<br />
and Application, 2nd ed. Wash<strong>in</strong>gton, DC: Jo<strong>in</strong>t Military <strong>Intelligence</strong> College,<br />
1995.<br />
Denz<strong>in</strong>, Norman, and Yvonna L<strong>in</strong>coln, eds. Handbook of Qualitative<br />
Research. Thousand Oaks, CA: Sage, 1994.<br />
Deutsch, Morton. “The Effects of Cooperation and Competition Upon Group<br />
Process” <strong>in</strong> Dorw<strong>in</strong> Cartwright and Alv<strong>in</strong> Zander, eds. Group Dynamics:<br />
Research and Theory. New York: Harper & Row, 1960.<br />
Duffy, T. M., and D. H. Jonassen, eds. Constructivism and <strong>the</strong> Technology of<br />
Instruction: A Conversation. Hillsdale, NJ: Erlbaum, 1992.<br />
Eco, Umberto. A Theory of Semiotics. Bloom<strong>in</strong>gton, IN: Indiana University<br />
Press, 1979.<br />
Egan, D., and B. Schwartz. “Chunk<strong>in</strong>g <strong>in</strong> Recall of Symbolic Draw<strong>in</strong>gs,”<br />
Memory and Cognition 7 (1979): 149–58.<br />
Elk<strong>in</strong>s, Dan. An <strong>Intelligence</strong> Resource Manager’s Guide. Wash<strong>in</strong>gton, DC:<br />
Jo<strong>in</strong>t Military <strong>Intelligence</strong> College, 1997.<br />
142
BIBLIOGRAPHY<br />
Endsley, M.R. “Design and Evaluation for Situation Awareness Enhancement,”<br />
Proceed<strong>in</strong>gs of Human Factors Society 32nd Annual Meet<strong>in</strong>g, Santa<br />
Monica, CA: Human Factors Society, 1988.<br />
Erv<strong>in</strong>, Alexander. Applied Anthropology. Tools and Perspectives for Contemporary<br />
Practice. Boston: Allyn and Bacon, 2000.<br />
Evans, Jonathan St. B. T. Bias <strong>in</strong> Human Reason<strong>in</strong>g: Causes and Consequences.<br />
Hove, UK: Lawrence Erlbaum Associates, 1989.<br />
Fandt, Patricia, W. Richardson, and H. Conner. “The Impact of Goal Sett<strong>in</strong>g<br />
on Team Simulation Experience,” Simulation and Gam<strong>in</strong>g 21, no. 4 (1990):<br />
411–22.<br />
Feder, Stanley. “FACTIONS and Policon: New Ways to Analyze Politics,”<br />
(1987), <strong>in</strong> H. Bradford Westerfield, ed. Inside <strong>CIA</strong>’s Private World: Declassified<br />
Articles from <strong>the</strong> Agency’s Internal Journal. New Haven, CT: Yale<br />
University Press, 1995.<br />
Feldman, Daniel C. “The Multiple Socialization of Organization Members.”<br />
Academy of Management Review 6, no. 2 (1981): 309–18.<br />
Feldman, T<strong>in</strong>e, and Susan Assaf. Social Capital: Conceptual Frameworks and<br />
Empirical Evidence. An Annotated Bibliography. Social Capital Initiative<br />
Work<strong>in</strong>g Paper No. 5. Wash<strong>in</strong>gton, DC: The World Bank, 1999.<br />
Fest<strong>in</strong>ger, Leon. “Informal Social Communication” <strong>in</strong> Dorw<strong>in</strong> Cartwright and<br />
Alv<strong>in</strong> Zander, eds. Group Dynamics: Research and Theory. New York:<br />
Harper & Row, 1960.<br />
Fishburn, Peter. Utility Theory for Decision Mak<strong>in</strong>g. New York. John Wiley,<br />
1970.<br />
Fitts, Paul, and Michael Posner. Human Performance. Belmont, CA: Brooks<br />
Cole, 1967.<br />
Flavell, J.H. “Metacognitive Aspects of Problem Solv<strong>in</strong>g” <strong>in</strong> L. Resnick, ed.<br />
The Nature of <strong>Intelligence</strong>. Hillsdale, NJ: Lawrence Erlbaum Associates,<br />
1976.<br />
Fleisher, Craig, and Babette Bensoussan. Strategic and Competitive Analysis.<br />
Methods and Techniques for Analyz<strong>in</strong>g Bus<strong>in</strong>ess Competition. Upper Saddle<br />
River, NJ: Pearson Education, 2003.<br />
Fleishman, Edw<strong>in</strong>, and Marilyn Qua<strong>in</strong>tance. Taxonomies of Human Performance.<br />
New York: Academic Press, 1984.<br />
Fletcher, J. Dexter. “Computer-Based Instruction: Costs and Effectiveness” <strong>in</strong><br />
Andrew Sage, ed. Concise Encyclopedia of Information Process<strong>in</strong>g <strong>in</strong> Systems<br />
and Organizations. Elmsford, NY: Pergamon Press, 1990.<br />
———. Effectiveness and Cost of Interactive Videodisc Instruction <strong>in</strong> Defense<br />
Tra<strong>in</strong><strong>in</strong>g and Education. Alexandria, VA: Institute for Defense Analyses,<br />
1990, IDA P-2372.<br />
———. “Evidence for Learn<strong>in</strong>g From Technology-Assisted Instruction” <strong>in</strong><br />
Harold F. O'Neil, Jr. and Ray Perez, eds. Technology Applications <strong>in</strong> Education:<br />
A Learn<strong>in</strong>g View. Hillsdale, NJ: Lawrence Erlbaum Associates, 2003.<br />
143
BIBLIOGRAPHY<br />
———. Technology, <strong>the</strong> Columbus Effect, and <strong>the</strong> Third Revolution <strong>in</strong> Learn<strong>in</strong>g.<br />
Alexandria, VA: Institute for Defense Analyses, 2001, D-2562.<br />
———, and Rob Johnston. “Effectiveness and Cost Benefits of Computer-<br />
Based Decision Aids for Equipment Ma<strong>in</strong>tenance,” Computers <strong>in</strong> Human<br />
Behavior 18 (2002): 717–28.<br />
———, and M. R. Rockway. “Computer-based Tra<strong>in</strong><strong>in</strong>g <strong>in</strong> <strong>the</strong> Military” <strong>in</strong> J.<br />
A. Ellis, ed. Military Contributions to Instructional Technology. New York:<br />
Praeger Publishers, 1986.<br />
Folker, Robert D. <strong>Intelligence</strong> Analysis <strong>in</strong> Theater Jo<strong>in</strong>t <strong>Intelligence</strong> Centers:<br />
An Experiment <strong>in</strong> Apply<strong>in</strong>g Structured Methods. Wash<strong>in</strong>gton, DC: Jo<strong>in</strong>t Military<br />
<strong>Intelligence</strong> College, Occasional Paper #7, 2000.<br />
Ford, Harold. Estimative <strong>Intelligence</strong>: The Purposes and Problems of<br />
National <strong>Intelligence</strong> Estimat<strong>in</strong>g. Lanham, MD: University Press of America,<br />
1993.<br />
Freud, Sigmund. “A Philosophy of Life: Lecture 35,” New Introductory Lectures<br />
on Psycho-Analysis. London: Hogarth Press, 1933.<br />
Fries, J., et al. “Assessment of Radiologic Progression <strong>in</strong> Rheumatoid Arthritis:<br />
A Randomized, Controlled Trial,” Arthritis Rheum 29, no. 1 (1986): 1–9.<br />
Fuld, Leonard. The New Competitor <strong>Intelligence</strong>: The Complete Resource for<br />
F<strong>in</strong>d<strong>in</strong>g, Analyz<strong>in</strong>g, and Us<strong>in</strong>g Information About Your Competitors. New<br />
York: John Wiley & Sons, 1995.<br />
Galanter, E., ed. Automatic Teach<strong>in</strong>g: The State of <strong>the</strong> Art. New York: John<br />
Wiley & Sons, 1959.<br />
Gall, Meredith, Walter Borg, and Joyce Gall. Educational Research, Sixth<br />
Edition. White Pla<strong>in</strong>s, NY: Longman, 1996.<br />
Garst, Ronald, ed. A Handbook of <strong>Intelligence</strong> Analysis, 2nd ed. Wash<strong>in</strong>gton,<br />
DC: Defense <strong>Intelligence</strong> College, 1989.<br />
Geertz, Clifford. The Interpretation of <strong>Culture</strong>s. New York: Basic Books,<br />
1973.<br />
Giddens, Anthony. Modernity and Self-Identity. Stanford, CA: Stanford University<br />
Press, 1991.<br />
Glaser, Barney, and Anselm Strauss. Discovery of Grounded Theory: Strategies<br />
for Qualitative Research. Hawthorne, NY: Ald<strong>in</strong>e de Gruyter, 1967.<br />
———. Theoretical Sensitivity: Advances <strong>in</strong> <strong>the</strong> Methodology of Grounded<br />
Theory. Mill Valley, CA: Sociology Press, 1978.<br />
———. Basics of Grounded Theory Analysis: Emergence vs. Forc<strong>in</strong>g. Mill<br />
Valley, CA: Sociology Press, 1992.<br />
Glass, Gene. “Primary, Secondary, and Meta-Analysis of Research.” Educational<br />
Researcher 5 (1976): 3–8.<br />
———, and Barry McGaw. “Choice of <strong>the</strong> Metric for Effect Size <strong>in</strong> Meta-<br />
Analysis.” American Educational Research Journal 17 (1980): 325–7.<br />
———, Barry McGaw, and Mary Lee Smith. Meta-Analysis <strong>in</strong> Social<br />
Research. New York: Sage, 1981.<br />
144
BIBLIOGRAPHY<br />
Godson, Roy, ed. Compar<strong>in</strong>g Foreign <strong>Intelligence</strong>: The U.S., <strong>the</strong> USSR, <strong>the</strong><br />
U.K. and <strong>the</strong> Third World. Wash<strong>in</strong>gton, DC: Pergamon-Brassey, 1988.<br />
Goldberg, Lewis. “Simple Models or Simple Processes? Some Research on<br />
Cl<strong>in</strong>ical Judgments,” American Psychologist 23 (1968): 483–96.<br />
———. “Man versus Model of Man: A Rationale, Plus Some Evidence, for a<br />
Method of Improv<strong>in</strong>g on Cl<strong>in</strong>ical Inferences,” Psychological Bullet<strong>in</strong> 73<br />
(1970): 422–32.<br />
Goldgeier, James M. “Psychology and Security,” Security Studies 6, no. 4<br />
(1997): 137–66.<br />
Gott, Sherrie, P.R.S. Kane, and Alan Lesgold. Tutor<strong>in</strong>g for Transfer of Technical<br />
Competence. Brooks AFB, TX: Armstrong Laboratory, Human<br />
Resources Directorate, 1995, AL/HR-TP-1995-0002.<br />
Graesser, A.C., M.A. Gernsbacher, and S. Goldman, eds. Handbook of Discourse<br />
Processes, Mahwah, NJ: Lawrence Erlbaum Associates, 2003.<br />
———, and Natalie Person. “Question-Ask<strong>in</strong>g Dur<strong>in</strong>g Tutor<strong>in</strong>g.” American<br />
Educational Research Journal 31 (1994): 104–37.<br />
Greenberg, Jerald, and Robert A. Baron. Behavior <strong>in</strong> Organizations: The<br />
Human Side of Work. Upper Saddle River, NJ: Prentice-Hall, 2002.<br />
Grimes, Barbara, ed. Ethnologue, 14th ed. Dallas, TX: Summer Institute of<br />
L<strong>in</strong>guistics, 2000.<br />
Gross, Jonathan. Measur<strong>in</strong>g <strong>Culture</strong>: A Paradigm for <strong>the</strong> Analysis of Social<br />
Organization. New York: Columbia University Press, 1985.<br />
Grove, William, and Paul Meehl. “Comparative Efficiency of Informal (Subjective,<br />
Impressionistic) and Formal (Mechanical, Algorithmic) Prediction<br />
Procedures: The Cl<strong>in</strong>ical-Statistical Controversy,” Psychology, Public Policy,<br />
and Law 2, no. 2 (1996): 293–323.<br />
Gumperz, John, and Dell Hymes. Directions <strong>in</strong> Sociol<strong>in</strong>guistics: The Ethnography<br />
of Communication. Oxford: B. Blackwell, 1986.<br />
———, and Stephen Lev<strong>in</strong>son. Reth<strong>in</strong>k<strong>in</strong>g L<strong>in</strong>guistic Relativity. Cambridge:<br />
Cambridge University Press, 1996.<br />
Hacker, D.J. Metacognition: Def<strong>in</strong>itions and Empirical Foundations [On-l<strong>in</strong>e<br />
report]. Memphis, TN: The University of Memphis, 2001. Retrieved 30 May<br />
2001 from <strong>the</strong> World Wide Web. (http://www.psyc.memphis.edu/trg/<br />
meta.htm.)<br />
Hammond, John, Ralph Keeney, and Howard Raiffa. “The Hidden Traps <strong>in</strong><br />
Decision Mak<strong>in</strong>g.” Harvard Bus<strong>in</strong>ess Review, September–October 1998,<br />
47–58.<br />
Harvey, J., and C. Boettger. “Improv<strong>in</strong>g Communication With<strong>in</strong> a Managerial<br />
Workgroup,” Journal of Applied Behavioral Science 7 (1971): 164–74.<br />
Hayes, J.R. The Complete Problem Solver, Philadelphia: The Frankl<strong>in</strong> Institute,<br />
1981.<br />
Hays, R. T., et al. “Flight Simulator Tra<strong>in</strong><strong>in</strong>g Effectiveness: A Meta-Analysis.”<br />
Military Psychology 4, no. 2 (1992): 63–74.<br />
145
BIBLIOGRAPHY<br />
Hedges, Larry. “Estimation of Effect Size From a Series of Independent<br />
Experiments,” Psychological Bullet<strong>in</strong>, 7 (1982): 119–37.<br />
———, and Ingram Olk<strong>in</strong>. “Vote-Count<strong>in</strong>g Methods <strong>in</strong> Research Syn<strong>the</strong>sis,”<br />
Psychological Bullet<strong>in</strong> 88 (1980): 359–69.<br />
———, and Ingram Olk<strong>in</strong>. Statistical Methods for Meta-Analysis. New York:<br />
Academic Press, 1985.<br />
Helfrich, Hede. “Human Reliability From a Social-Psychological Perspective,”<br />
International Journal Human-Computer Studies 50 (1999): 193–212.<br />
Hesketh, Beryl, and Stephen Bochner. “Technological Change <strong>in</strong> a Multicultural<br />
Context: Implications for Tra<strong>in</strong><strong>in</strong>g and Career Plann<strong>in</strong>g” <strong>in</strong> Marv<strong>in</strong><br />
Dunnette and Leaetta Hough, eds. Handbook of Industrial and Organizational<br />
Psychology. Palo Alto, CA: Consult<strong>in</strong>g Psychologists Press, 1994.<br />
Heuer, Richards J., Jr. Psychology of <strong>Intelligence</strong> Analysis. Wash<strong>in</strong>gton, DC:<br />
Center for <strong>the</strong> Study of <strong>Intelligence</strong>, 1999.<br />
Hewitt, John P. Self and Society, 7th ed. Boston: Allyn and Bacon, 1997.<br />
H<strong>in</strong>chman, Lewis P., and Sandra K. H<strong>in</strong>chman, eds. Memory, Identity, <strong>Community</strong>.<br />
Albany, NY: State University of New York Press, 1997.<br />
Hoijer, Harry. “The Relation of Language to <strong>Culture</strong>” <strong>in</strong> Alfred Kroeber, ed.<br />
Anthropology Today. Chicago: University of Chicago Press, 1953.<br />
Hollnagel, Erik. “The Phenotype of Erroneous Actions.” International Journal<br />
of Man-Mach<strong>in</strong>e Studies 39 (1993): 1–32.<br />
Hopk<strong>in</strong>s, R. Warn<strong>in</strong>gs of Revolution: A Case Study of El Salvador. Wash<strong>in</strong>gton,<br />
DC: Center for <strong>the</strong> Study of <strong>Intelligence</strong>, 1980, TR 80-100012.<br />
House, Ernest. Evaluat<strong>in</strong>g with Validity. Beverly Hills, CA: Sage, 1980.<br />
Hubble, Edw<strong>in</strong>. The Realm of <strong>the</strong> Nebulae. New Haven, CT: Yale University<br />
Press, 1936.<br />
Hulnick, Arthur S. “The <strong>Intelligence</strong> Producer-Policy Consumer L<strong>in</strong>kage: A<br />
Theoretical Approach,” <strong>Intelligence</strong> and National Security 1, no. 2 (May<br />
1986): 212–33.<br />
Hunt, David. Complexity and Plann<strong>in</strong>g <strong>in</strong> <strong>the</strong> 21st Century: <strong>Intelligence</strong><br />
Requirements to Unlock <strong>the</strong> Mystery. Newport, RI: Naval War College,<br />
2000.<br />
Hunt, Earl, and Franca Agnoli. “The Whorfian Hypo<strong>the</strong>sis. A Cognitive Psychology<br />
Perspective,” Psychological Review 98, no. 3 (1991): 377–89.<br />
Hymes, Dell, ed. Language <strong>in</strong> <strong>Culture</strong> and Society: A Reader <strong>in</strong> L<strong>in</strong>guistics<br />
and Anthropology. New York: Harper and Row, 1977.<br />
Iaffaldano, Michelle T., and Paul M. Much<strong>in</strong>sky. “Job satisfaction and Job<br />
Performance: A Meta-Analysis,” Psychological Bullet<strong>in</strong> 97 (1985): 251–73.<br />
James, William. Pr<strong>in</strong>ciples of Psychology: Volume I. New York: Dover Press,<br />
1890/1950.<br />
Janis, Irv<strong>in</strong>g. Groupth<strong>in</strong>k, 2nd ed. Boston: Houghton Miffl<strong>in</strong>, 1982.<br />
Janssens, Maddy. “Interculture Interaction: A Burden on International Managers?”<br />
Journal of Organizational Behavior 16 (1995): 155–67.<br />
146
BIBLIOGRAPHY<br />
Jervis, Robert. Perception and Misperception <strong>in</strong> International Politics. Pr<strong>in</strong>ceton,<br />
NJ: Pr<strong>in</strong>ceton University Press, 1977.<br />
Johnson, David, and Roger Johnson. “The Internal Dynamics of Cooperative<br />
Learn<strong>in</strong>g Groups” <strong>in</strong> Richard Slav<strong>in</strong>, et al., eds. Learn<strong>in</strong>g to Cooperate,<br />
Cooperat<strong>in</strong>g to Learn. New York: Plenum, 1985.<br />
———, et al. “Effects of Cooperative, Competitive, and Individualistic Goal<br />
Structure on Achievement: A Meta-Analysis,” Psychological Bullet<strong>in</strong> 89,<br />
no. 1 (1981): 47–62.<br />
———, et al. “The Impact of Positive Goal and Resource Interdependence on<br />
Achievement, Interaction, and Attitudes,” Journal of General Psychology<br />
118, no. 4 (1996): 341–47.<br />
Johnson, Edgar. Effects of Data Source Reliability on Inference. Alexandria,<br />
VA: Army Research Institute for <strong>the</strong> Behavioral and Social Sciences, Technical<br />
Paper #251, 1974.<br />
Johnston, Rob. Decision Mak<strong>in</strong>g and Performance Error <strong>in</strong> Teams: Research<br />
Results. Arl<strong>in</strong>gton, VA: Defense Advanced Research Projects Agency, 1997.<br />
———. “The Effectiveness of Instructional Technology,” Proceed<strong>in</strong>gs of <strong>the</strong><br />
Virtual Reality <strong>in</strong> Medic<strong>in</strong>e and Developers’ Exposition. Cambridge, MA:<br />
VRS, 1995.<br />
———. “Electronic Performance Support Systems and Information Navigation,”<br />
Thread 2, no. 2 (1994): 5–7.<br />
———, and Sunil Bhoyrul. “A Prelim<strong>in</strong>ary Report of a Task Analysis of Laparoscopic<br />
Cholecystectomy Used to Assess <strong>the</strong> Effectiveness of a Virtual<br />
Reality Surgical Simulator,” Journal of Virtual Reality <strong>in</strong> Medic<strong>in</strong>e 2, no. 1<br />
(1996): 16–19.<br />
———, et al. “Assess<strong>in</strong>g a Virtual Reality Surgical Skills Simulator” <strong>in</strong><br />
Suzanne Weghorst, Hans Sieburg, and Karen Morgan, eds. Health Care <strong>in</strong><br />
<strong>the</strong> Information Age: Future Tools for Transform<strong>in</strong>g Medic<strong>in</strong>e. Wash<strong>in</strong>gton,<br />
DC: IOS Press, 1996.<br />
———, and J. Dexter Fletcher. A Meta-Analysis of <strong>the</strong> Effectiveness of Computer-Based<br />
Tra<strong>in</strong><strong>in</strong>g for Military Instruction. Alexandria, VA: Institute for<br />
Defense Analyses, 1998.<br />
———, J. Dexter Fletcher, and Sunil Bhoyrul. The Use of Virtual Reality to<br />
Measure Surgical Skill Levels. Alexandria, VA: Institute for Defense Analyses,<br />
1998.<br />
Jones, Morgan. The Th<strong>in</strong>ker’s Toolkit: 14 Powerful Techniques for Problem<br />
Solv<strong>in</strong>g. New York: Times Bus<strong>in</strong>ess, 1998.<br />
Jung, Carl. The Undiscovered Self. Translated by R.F.C. Hull. Pr<strong>in</strong>ceton, NJ:<br />
Pr<strong>in</strong>ceton University Press, 1990.<br />
Kahaner, Larry. Competitive <strong>Intelligence</strong>. How to Ga<strong>the</strong>r, Analyze, and Use<br />
Information to Move Your Bus<strong>in</strong>ess to <strong>the</strong> Top. New York: Touchstone,<br />
Simon and Schuster, 1996.<br />
147
BIBLIOGRAPHY<br />
Kahneman, Daniel, and Amos Tversky. “Prospect Theory: An Analysis of<br />
Decision Under Risk,” Econometrica 47, no. 2 (1979): 263–91.<br />
———, Paul Slovic, and Amos Tversky. Judgment Under Uncerta<strong>in</strong>ty: Heuristics<br />
and Biases. Cambridge: Cambridge University Press, 1982.<br />
Kam, Ephraim. Surprise Attack. The Victim’s Perspective. Cambridge: Harvard<br />
University Press, 1988.<br />
Kaufman, David L., Jr. Introduction to Systems Th<strong>in</strong>k<strong>in</strong>g. M<strong>in</strong>neapolis: Future<br />
Systems, 1978.<br />
Kaufman, Roger. “A Holistic Plann<strong>in</strong>g Model: A Systems Approach for<br />
Improv<strong>in</strong>g Organizational Effectiveness and Impact,” Performance and<br />
Instruction Journal, October 1983: 3–10.<br />
Kent, Sherman. Strategic <strong>Intelligence</strong> for American World Policy. Pr<strong>in</strong>ceton:<br />
Pr<strong>in</strong>ceton University Press, 1966.<br />
———. “Words of Estimative Probability,” Sherman Kent and <strong>the</strong> Board of<br />
National Estimates: Collected Essays. Wash<strong>in</strong>gton, DC: Center for <strong>the</strong> Study<br />
of <strong>Intelligence</strong>, 1994.<br />
Kirk, Jerome, and Marc Miller. Reliability and Validity <strong>in</strong> Qualitative<br />
Research. Qualitative Research Methods, Volume 1. Newbury Park, CA:<br />
Sage, 1986.<br />
Kirkpatrick, Lyman B., Jr. Capta<strong>in</strong>s Without Eyes: <strong>Intelligence</strong> Failures <strong>in</strong><br />
World War II. London: MacMillan Company, 1969.<br />
Klahr, D., and H.A. Simon. “What Have Psychologists (and O<strong>the</strong>rs) Discovered<br />
About <strong>the</strong> Process of Scientific Discovery?” Current Directions <strong>in</strong> Psychological<br />
Science 10.<br />
Kle<strong>in</strong>, Gary, Roberta Calderwood, and A. Cl<strong>in</strong>ton-Cirocco. Rapid Decision<br />
Mak<strong>in</strong>g on <strong>the</strong> Fire Ground. Yellow Spr<strong>in</strong>gs, OH: Kle<strong>in</strong> Associates, 1985,<br />
KA-TR-84-41-7 (prepared under contract MDA903-85-G-0099 for <strong>the</strong> US<br />
Army Research Institute, Alexandria, VA).<br />
Knight, Frank H. Risk, Uncerta<strong>in</strong>ty and Profit. Boston: Houghton Miffl<strong>in</strong>,<br />
1921.<br />
Knorr, Klaus. “Foreign <strong>Intelligence</strong> and <strong>the</strong> Social Sciences.” (Research<br />
monograph, Center of International Studies, Pr<strong>in</strong>ceton University, 1964.)<br />
Krizan, Lisa. <strong>Intelligence</strong> Essentials for Everyone. Wash<strong>in</strong>gton, DC: Jo<strong>in</strong>t<br />
Military <strong>Intelligence</strong> College, 1999.<br />
Krotow, Gerald<strong>in</strong>e. The Impact of Cognitive Feedback on <strong>the</strong> Performance of<br />
<strong>Intelligence</strong> Analysts. Monterey, CA: Naval Postgraduate School, 1992, AD-<br />
A252.<br />
Kuhn, Thomas. The Structure of Scientific Revolutions. Chicago: The University<br />
of Chicago Press, 1970.<br />
Kulik, Chen-L<strong>in</strong>, and James Kulik. “Effectiveness of Computer-Based Education<br />
<strong>in</strong> Colleges,” AEDS Journal, W<strong>in</strong>ter/Spr<strong>in</strong>g 1986: 81–108.<br />
148
BIBLIOGRAPHY<br />
———, and Barbara Shwalb, “Effectiveness of Computer-Based Adult Education:<br />
A Meta-Analysis,” Journal of Educational Comput<strong>in</strong>g Research 2,<br />
no. 2 (1986): 235–52.<br />
Kulik, James. “Meta-<strong>Analytic</strong> Studies of F<strong>in</strong>d<strong>in</strong>gs on Computer-Based<br />
Instruction” <strong>in</strong> Eva Baker and Harold O'Neil, eds. Technology Assessment <strong>in</strong><br />
Education and Tra<strong>in</strong><strong>in</strong>g. Hillsdale, NJ: LEA Publishers, 1994.<br />
Lacan, Jacques. Ecrits. Translated by Alan Sheridan. New York: W.W.<br />
Norton, 1977.<br />
Laqueur, Walter A. The Uses and Limits of <strong>Intelligence</strong>. New Brunswick, NJ:<br />
Transaction Publishers, 1993.<br />
Lark<strong>in</strong>, Jill. “The Role of Problem Representation <strong>in</strong> Physics” <strong>in</strong> Debra Gentner<br />
and Albert Stevens, eds. Mental Models. Hillsdale, NJ: Lawrence<br />
Erlbaum Associates, 1983.<br />
Lee, Penny. The Whorf Theory Complex: A Critical Reconstruction. Amsterdam:<br />
Benjam<strong>in</strong>s, 1996.<br />
Leli, D., and S. Filskov. “Cl<strong>in</strong>ical-Actuarial Detection of and Description of<br />
Bra<strong>in</strong> Impairment with <strong>the</strong> Wechsler-Bellevue Form I,” Journal of Cl<strong>in</strong>ical<br />
Psychology 37 (1981): 623–29.<br />
Lesgold, Alan, et al. “Expertise <strong>in</strong> a Complex Skill: Diagnos<strong>in</strong>g X-Ray Pictures”<br />
<strong>in</strong> Michelene Chi, Robert Glaser and Marshall Farr, eds. The Nature of<br />
Expertise. Hillsdale, NJ: Lawrence Erlbaum Associates, 1988.<br />
Levi-Strauss, Claude. Structural Anthropology. New York: The Pengu<strong>in</strong> Press,<br />
1968. (Orig<strong>in</strong>ally published <strong>in</strong> 1958.)<br />
Lev<strong>in</strong>e, George, ed. Constructions of <strong>the</strong> Self. Brunswick, NJ: Rutgers University<br />
Press, 1992.<br />
Lev<strong>in</strong>son, David, and Melv<strong>in</strong> Ember, eds. American Immigrant <strong>Culture</strong>s:<br />
Builders of a Nation. Vol. I and II. New York: Macmillan Reference, 1997.<br />
Litw<strong>in</strong>, Mark. “How to Measure Survey Reliability and Validity” <strong>in</strong> Arlene<br />
F<strong>in</strong>k, ed. The Survey Kit. Volume 7. Thousand Oaks, CA: Sage, 1995.<br />
Liu, Lisa. “Reason<strong>in</strong>g Counterfactually <strong>in</strong> Ch<strong>in</strong>ese: Are There Any Obstacles?”<br />
Cognition 21 (1985): 239–70.<br />
Locke, Edw<strong>in</strong> A. “The Nature and Causes of Job Satisfaction” <strong>in</strong> Marv<strong>in</strong> Dunnette,<br />
ed. Handbook of Industrial and Organizational Psychology. Chicago:<br />
Rand McNally, 1976.<br />
Lockwood, Jonathan, and K. Lockwood. “The Lockwood <strong>Analytic</strong>al Method<br />
for Prediction (LAMP),” Defense <strong>Intelligence</strong> Journal 3, no. 2 (1994): 47–<br />
74.<br />
Lowenthal, Mark. <strong>Intelligence</strong>: From Secrets to Policy. Wash<strong>in</strong>gton, DC: CQ,<br />
2000.<br />
Luce, Duncan, and Howard Raiffa. “Utility Theory.” Paul Moser, ed. Rationality<br />
<strong>in</strong> Action. Contemporary Approaches. New York. Cambridge University<br />
Press, 1990.<br />
149
BIBLIOGRAPHY<br />
Lucy, John. Grammatical Categories and Cognition. Cambridge: Cambridge<br />
University Press, 1992.<br />
———. Language Diversity and Thought. Cambridge: Cambridge University<br />
Press, 1992.<br />
MacEach<strong>in</strong>, Douglas J., et al. The Tradecraft of Analysis: Challenge and<br />
Change <strong>in</strong> <strong>the</strong> <strong>CIA</strong>. Wash<strong>in</strong>gton, DC: Consortium for <strong>the</strong> Study of <strong>Intelligence</strong>,<br />
1994.<br />
Marr<strong>in</strong>, Stephen. <strong>CIA</strong>’s Kent School. A Step <strong>in</strong> <strong>the</strong> Right Direction. Paper presented<br />
at <strong>the</strong> International Studies Association Annual Convention, New<br />
Orleans, LA, 2002. (http://www.isanet.org/noarchive/marr<strong>in</strong>.html.)<br />
———. “Improv<strong>in</strong>g <strong>CIA</strong> Analysis by Overcom<strong>in</strong>g Institutional Obstacles” <strong>in</strong><br />
Russell G. Swenson, ed. Br<strong>in</strong>g<strong>in</strong>g <strong>Intelligence</strong> About: Practitioners Reflect<br />
on Best Practices. Wash<strong>in</strong>gton, DC: Jo<strong>in</strong>t Military <strong>Intelligence</strong> College,<br />
2003.<br />
Mathams, Robert. “The <strong>Intelligence</strong> Analyst’s Notebook” <strong>in</strong> Douglas Dearth<br />
and R. Thomas Goodden, eds. Strategic <strong>Intelligence</strong>: Theory and Application,<br />
2nd ed. Wash<strong>in</strong>gton, DC: Jo<strong>in</strong>t Military <strong>Intelligence</strong> College, 1995.<br />
Mauss, Marcel. The Gift: The Form and Reason for Exchange <strong>in</strong> Archaic<br />
Societies. Translated by W. D. Halls. New York: W. W. Norton, 2000.<br />
Mayo, Elton. The Human Problems of an Industrial Civilization. New York:<br />
MacMillan, 1933.<br />
McDowell, Don. Strategic <strong>Intelligence</strong>: A Handbook for Practitioners, Managers<br />
and Users. Canberra, Australia: Istana Enterprises, 1998.<br />
McEvoy, Glenn M., and Wayne F. Cascio. “Strategies for Reduc<strong>in</strong>g Employee<br />
Turnover: A Meta-Analysis,” Journal of Applied Psychology 70, no. 2<br />
(1985): 342–53.<br />
McGuire, Hunter, et al. “Measur<strong>in</strong>g and Manag<strong>in</strong>g Quality of Surgery: Statistical<br />
vs. Incidental Approaches,” Archives of Surgery 127 (1992): 733–37.<br />
McKei<strong>the</strong>n, Ka<strong>the</strong>r<strong>in</strong>e, et al. “Knowledge Organization and Skill Differences<br />
<strong>in</strong> Computer Programmers,” Cognitive Psychology 13 (1981): 307–25.<br />
Mead, George. M<strong>in</strong>d, Self, and Society. Charles Morris, ed. Chicago: University<br />
of Chicago Press, 1967.<br />
Megl<strong>in</strong>o, Bruce M., et al. “Effects of Realistic Job Previews: A Comparison<br />
Us<strong>in</strong>g an Enhancement and a Reduction Preview.” Journal of Applied Psychology<br />
71 (1988): 259–66.<br />
Meister, David. Behavioral Analysis and Measurement Methods. New York:<br />
Wiley, 1985.<br />
———. Human Factors: Theory and Practice. New York: Wiley, 1971.<br />
Meister, Judith. “Individual Perceptions of Team Learn<strong>in</strong>g Experiences Us<strong>in</strong>g<br />
Video-Based or Virtual Reality Environments,” Dissertation Abstracts International,<br />
UMI No. 9965200, 2000.<br />
Miller, Delbert. Handbook of Research Design and Social Measurement.<br />
Newbury Park, CA: Sage, 1991.<br />
150
BIBLIOGRAPHY<br />
Miller, George, and Philip Johnson-Laird. Language and Perception. Cambridge:<br />
Harvard University Press, 1976.<br />
Mills, Theodore. “Power Relations <strong>in</strong> Three-Person Groups” <strong>in</strong> Dorw<strong>in</strong> Cartwright<br />
and Alv<strong>in</strong> Zander, Group Dynamics: Research and Theory. New<br />
York: Harper & Row, 1960.<br />
M<strong>in</strong> Qi, Zhongguo Zhengzhi Wenhua [Ch<strong>in</strong>ese Political <strong>Culture</strong>]. Kunm<strong>in</strong>g,<br />
Ch<strong>in</strong>a: Yunnan People’s Press, 1989.<br />
M<strong>in</strong>sky, Marv<strong>in</strong>, and Seymour Papert. Artificial <strong>Intelligence</strong>. Eugene: Oregon<br />
State System of Higher Education, 1974.<br />
Molm, L<strong>in</strong>da. “L<strong>in</strong>k<strong>in</strong>g Power Structure and Power Use” <strong>in</strong> Karen Cook, ed.<br />
Social Exchange Theory. Newbury Park, CA: Sage, 1987.<br />
Moore, David. “Creat<strong>in</strong>g <strong>Intelligence</strong>: Evidence and Inference <strong>in</strong> <strong>the</strong> Analysis<br />
Process.” Unpublished Masters Thesis. Wash<strong>in</strong>gton, DC: Jo<strong>in</strong>t Military<br />
<strong>Intelligence</strong> College, 2002.<br />
Morris, C.D., J.D. Bransford, and J.J. Franks. “Level of Process<strong>in</strong>g Versus<br />
Transfer-Appropriate Process<strong>in</strong>g,” Journal of Verbal Learn<strong>in</strong>g and Verbal<br />
Behavior 16 (1977).<br />
Morrison, J.E., and J.D. Fletcher. Cognitive Read<strong>in</strong>ess. Alexandria, VA: Institute<br />
for Defense Analyses, 2001, IDA Paper P-3735.<br />
———, and Larry Meliza. Foundations of <strong>the</strong> After-Action Review Process.<br />
Alexandria, VA: Institute for Defense Analyses, 1999.<br />
Mullen, Brian, and Carolyn Copper. “The Relation Between Group Cohesiveness<br />
and Performance: An Integration,” Psychological Bullet<strong>in</strong>, 115 (1994):<br />
210–27.<br />
National Aeronautics and Space Adm<strong>in</strong>istration. Astronaut Fact Book. NP-<br />
2000-09-008JSC. Wash<strong>in</strong>gton, DC: NASA, 2000.<br />
Neisser, U. Cognitive Psychology. New York: Appleton, Century Crofts, 1967.<br />
Nieva, V., E. Fleishman, and A. Rieck. Team Dimensions: Their Identity,<br />
Their Measurement, and Their Relationships, RN 85-12. Alexandria, VA:<br />
US Army Research Institute for <strong>the</strong> Behavioral and Social Sciences, 1985.<br />
Nijhof, W., and P. Kommers. “An Analysis of Cooperation <strong>in</strong> Relation to Cognitive<br />
Controversy” <strong>in</strong> Richard Slav<strong>in</strong>, Shlomo Sharan, Spencer Kagan, et<br />
al., eds. Learn<strong>in</strong>g to Cooperate, Cooperat<strong>in</strong>g to Learn. New York: Plenum,<br />
1985.<br />
Nisbett, Richard, and Ara Norenzayan. “<strong>Culture</strong> and Cognition” <strong>in</strong> Douglas<br />
Med<strong>in</strong> and Hal Pashler, eds. Stevens’ Handbook of Experimental Psychology,<br />
3rd ed., New York: John Wiley & Sons, 2002.<br />
Nolte, William. “Preserv<strong>in</strong>g Central <strong>Intelligence</strong>: Assessment and Evaluation<br />
<strong>in</strong> Support of <strong>the</strong> DCI, ” Studies <strong>in</strong> <strong>Intelligence</strong> 48, no. 3 (2005): 21–25.<br />
Orasanu, Judith. “Shared Mental Models and Crew Performance.” Paper presented<br />
at <strong>the</strong> 34th annual meet<strong>in</strong>g of <strong>the</strong> Human Factors Society, Orlando,<br />
FL, 1990.<br />
151
BIBLIOGRAPHY<br />
———, and Lynne Mart<strong>in</strong>. “Errors <strong>in</strong> Aviation Decision Mak<strong>in</strong>g: A Factor <strong>in</strong><br />
Accidents and Incidents,” Proceed<strong>in</strong>gs of <strong>the</strong> Work<strong>in</strong>g Group on Human<br />
Error, Safety, and System Development. Seattle, WA: International Federation<br />
for Information Process<strong>in</strong>g, 1998: 100–107.<br />
Orlansky, Jesse, and Joseph Str<strong>in</strong>g. Cost-Effectiveness of Computer-Based<br />
Instruction <strong>in</strong> Military Tra<strong>in</strong><strong>in</strong>g. Alexandria, VA: Institute for Defense Analyses,<br />
1979. IDA P-1375.<br />
———, et al. The Cost and Effectiveness of <strong>the</strong> Multi-Service Distributed<br />
Tra<strong>in</strong><strong>in</strong>g Testbed (MDT2) for Tra<strong>in</strong><strong>in</strong>g Close Air Support. Alexandria, VA:<br />
Institute for Defense Analyses, 1997.<br />
———. The Value of Simulation for Tra<strong>in</strong><strong>in</strong>g. Alexandria, VA: Institute for<br />
Defense Analyses, 1994. IDA P-2982.<br />
Paige, Jeffery, and Herbert Simon. “Cognition Processes <strong>in</strong> Solv<strong>in</strong>g Algebra<br />
Word Problems” <strong>in</strong> Benjam<strong>in</strong> Kle<strong>in</strong>muntz, ed. Problem Solv<strong>in</strong>g. New York:<br />
Wiley, 1966.<br />
Patton, Michael. Qualitative Evaluation and Research Methods, 2nd ed. Beverly<br />
Hills, CA: Sage, 1990.<br />
Perrow, Charles. Normal Accidents: Liv<strong>in</strong>g with High Risk Technologies. Pr<strong>in</strong>ceton,<br />
NJ: Pr<strong>in</strong>ceton University Press, 1999.<br />
———. Written communications between Perrow and Johnston, 2002.<br />
Peterson, Marilyn. Applications <strong>in</strong> Crim<strong>in</strong>al Analysis. Westport, CT: Greenwood<br />
Press, 1995.<br />
———, Bob Morehouse, and Richard Wright, eds. <strong>Intelligence</strong> 2000: Revis<strong>in</strong>g<br />
<strong>the</strong> Basic Elements. Sacramento, CA: LEIU and IALEIA, 2000.<br />
Phelps, Ruth, et al. “Effectiveness and Costs of Distance Education Us<strong>in</strong>g<br />
Computer-Mediated Communication,” American Journal of Distance Education<br />
5, no. 3 (1991): 7–19.<br />
Pierce, John. “Some Ma<strong>the</strong>matical Methods for <strong>Intelligence</strong> Analysis,” Studies<br />
<strong>in</strong> <strong>Intelligence</strong> 21 (Summer 1977) (declassified): 1–19.<br />
Popper, Karl. The Logic of Scientific Discovery. London: Hutch<strong>in</strong>son, 1959.<br />
———. Conjectures and Refutations: The Growth of Scientific Knowledge.<br />
New York: Routledge, 2002 repr<strong>in</strong>t.<br />
Porter, Lyman W., Edward E. Lawler, and J. Richard Hackman. Behavior <strong>in</strong><br />
Organizations. New York: McGraw-Hill, 1975.<br />
———, et al. “Organizational Commitment, Job Satisfaction and Turnover<br />
Among Psychiatric Technicians,” Journal of Applied Psychology 59 (1974):<br />
603-609.<br />
Prescott, John, and Stephen Miller, eds. Proven Strategies <strong>in</strong> Competitive<br />
<strong>Intelligence</strong>. New York: John Wiley & Sons, 2001.<br />
Price-Williams, Douglass. Explorations <strong>in</strong> Cross-Cultural Psychology. San<br />
Francisco: Chandler and Sharp, 1975.<br />
Putnam, Robert. Bowl<strong>in</strong>g Alone: The Collapse and Revival of American <strong>Community</strong>.<br />
New York: Simon and Schuster, 2000.<br />
152
BIBLIOGRAPHY<br />
———. “The Prosperous <strong>Community</strong>: Social Capital and Public Life,” The<br />
American Prospect 4, no. 13 (1993): 35–42.<br />
Quirk, John, et al. The Central <strong>Intelligence</strong> Agency: A Photographic History.<br />
Guilford, CT: Foreign <strong>Intelligence</strong> Press, 1986.<br />
Reason, James. Human Error. New York: Cambridge University Press, 1990.<br />
Reichenbach, Hans. Experience and Prediction. Chicago: University of Chicago<br />
Press, 1938.<br />
Richelson, Jeffrey. The U.S. <strong>Intelligence</strong> <strong>Community</strong>, 4th ed. Boulder, CO:<br />
Westview Press, 1999.<br />
Roethlisberger, Fritz J., and William J. Dickson. Management and <strong>the</strong> Worker:<br />
An Account of a Research Program Conducted by <strong>the</strong> Western Electric Company,<br />
Hawthorne Works, Chicago. Boston: Harvard University Press, 1939.<br />
Rosenberg, Marc J. “Performance Technology: Work<strong>in</strong>g <strong>the</strong> System,” Tra<strong>in</strong><strong>in</strong>g,<br />
January 1990: 5–10.<br />
Rossi, Peter, and Howard Freeman. Evaluation. A Systematic Approach, 5th<br />
ed. Newbury Park, CA: Sage, 1993.<br />
Roth, P. L. ,and James E. Campion. “An Analysis of <strong>the</strong> Predictive Power of<br />
<strong>the</strong> Panel Interview and Pre-Employment Tests,” Journal of Occupational<br />
and Organizational Psychology 65 (1992): 51-60.<br />
Sacks, H. “Promises, Performance, and Pr<strong>in</strong>ciples: An Empirical Study of<br />
Parole Decision-Mak<strong>in</strong>g <strong>in</strong> Connecticut,” Connecticut Law Review 9 (1977):<br />
349–422.<br />
Salomon, G., and D.N. Perk<strong>in</strong>s. “Rocky Roads to Transfer: Reth<strong>in</strong>k<strong>in</strong>g Mechanisms<br />
of a Neglected Phenomenon,” Educational Psychologist 24 (1998).<br />
Sapir, Edward. Language. New York: Harcourt Brace, 1949.<br />
Sapp, Edw<strong>in</strong>. “Decision Trees,” Studies <strong>in</strong> <strong>Intelligence</strong> 18 (W<strong>in</strong>ter 1974)<br />
(declassified): 45–57.<br />
Sarb<strong>in</strong>, Theodore. “A Contribution to <strong>the</strong> Study of Actuarial and Individual<br />
Methods of Prediction,” American Journal of Sociology 48 (1943): 593–602.<br />
Saussure, Ferd<strong>in</strong>and. Course <strong>in</strong> General L<strong>in</strong>guistics. Translated by W. Bask<strong>in</strong>.<br />
New York: McGraw-Hill, 1959.<br />
Sawyer, J. “Measurement and Prediction, Cl<strong>in</strong>ical and Statistical,” Psychological<br />
Bullet<strong>in</strong> 66 (1966): 178–200.<br />
Schensul, Jean, and Margaret LeCompte, eds. Ethnographer’s Toolkit. Vol. I–<br />
Vol. VII. Walnut Creek, CA: Alta Mira Press, 1999.<br />
Schofield, W., and J. Garrard. “Longitud<strong>in</strong>al Study of Medical Students<br />
Selected for Admission to Medical School by Actuarial and Committee<br />
Methods,” British Journal of Medical Education 9 (1975): 86–90.<br />
Schum, David. The Evidential Foundations of Probabilistic Reason<strong>in</strong>g. Evanston,<br />
IL: Northwestern University Press, 1994.<br />
Scriven, M. “Problems and Prospects for Individualization” <strong>in</strong> H. Talmage,<br />
ed. Systems of Individualized Education. Berkley, CA: McCutchan, 1975.<br />
153
BIBLIOGRAPHY<br />
Seashore, S. Group Cohesiveness <strong>in</strong> <strong>the</strong> Industrial Workgroup. Ann Arbor:<br />
University of Michigan Press, 1954.<br />
Segall, Marshall. Cross-Cultural Psychology: Human Behavior <strong>in</strong> Global<br />
Perspective. Monterey, CA: Brooks/Cole, 1979.<br />
Senders, John, and Neville Moray. Human Error: Cause, Prediction and<br />
Reduction. Hillsdale, NJ: Lawrence Earlbaum Associates, 1991.<br />
Shiels, Frederick L. Preventable Disasters: Why Governments Fail. Savage,<br />
MD: Rowman and Littlefield Publishers, 1991.<br />
Shiffr<strong>in</strong>, R.M. and W. Schneider. “Controlled and Automatic Human Information<br />
Process<strong>in</strong>g: II. Perceptual Learn<strong>in</strong>g,” Psychological Review 84 (1977).<br />
Shlaim, Avi. “Failures <strong>in</strong> National <strong>Intelligence</strong> Estimates: The Case of <strong>the</strong><br />
Yom Kippur War,” World Politics 28, no. 3 (1976): 348–80.<br />
Shweder, Richard. Th<strong>in</strong>k<strong>in</strong>g Through <strong>Culture</strong>s. Cambridge, MA: Harvard<br />
University Press, 1991.<br />
Simmel, Georg. The Sociology of Georg Simmel. Translated by K. Wolff.<br />
Glencoe, IL: Free Press, 1950.<br />
Simon, D., and Herbert Simon. “Individual Differences <strong>in</strong> Solv<strong>in</strong>g Physics<br />
Problems” <strong>in</strong> Robert Siegler, ed. Children’s Th<strong>in</strong>k<strong>in</strong>g: What Develops? Hillsdale,<br />
NJ: Lawrence Erlbaum Associates, 1978.<br />
Slav<strong>in</strong>, Robert. “Research on Cooperative Learn<strong>in</strong>g: Consensus and Controversy,”<br />
Educational Leadership 47, no. 4 (1989): 52–55.<br />
———. Cooperative Learn<strong>in</strong>g. New York: Longman, 1983.<br />
Slovic, P., S. Lichtenste<strong>in</strong>, and B. Fischoff. “Decision-Mak<strong>in</strong>g” <strong>in</strong> R.C. Atk<strong>in</strong>son<br />
et al. eds. Steven’s Handbook of Experimental Psychology (2nd ed.), Volume<br />
2: Learn<strong>in</strong>g and Cognition, New York: John Wiley & Sons, 1988.<br />
Solomon, H. Economic Issues <strong>in</strong> Cost-Effectiveness Analyses of Military Skill<br />
Tra<strong>in</strong><strong>in</strong>g. Alexandria, VA: Institute for Defense Analyses, 1986, IDA P-<br />
1897.<br />
Spencer, Herbert. The Study of Sociology. New York: D. Appleton. 1874.<br />
Spencer, Ken. “Modes, Media and Methods: The Search for Educational<br />
Effectiveness,” British Journal of Educational Technology 22, no. 1 (1991):<br />
12–22.<br />
Spradley, James. Participant Observation. New York: R<strong>in</strong>ehart and W<strong>in</strong>ston,<br />
1980.<br />
Stanton, Neville, and Sarah Stevenage. “Learn<strong>in</strong>g to Predict Human Error:<br />
Issues of Acceptability, Reliability and Validity,” Ergonomics, 41, no. 11<br />
(1998): 1737–56.<br />
Stephenson, William. The Study of Behavior: Q-Technique and its Methodology.<br />
Chicago: University of Chicago Press, 1953.<br />
Suppes, P., and M. Morn<strong>in</strong>gstar. Computer-Assisted Instruction at Stanford,<br />
1966-68: Data, Models, and Evaluation of <strong>the</strong> Arithmetic Programs. New<br />
York: Academic Press, 1972.<br />
154
BIBLIOGRAPHY<br />
Thompson, J. R., R. Hopf-Weichel, and R. Geiselman. The Cognitive Bases of<br />
<strong>Intelligence</strong> Analysis. Alexandria, VA: Army Research Institute, Research<br />
Report 1362, 1984, AD-A146.<br />
Tierney, William G., and Robert A. Rhoads. Faculty Socialization as Cultural<br />
Process: A Mirror of Institutional Commitment. Wash<strong>in</strong>gton, DC: ASHE-<br />
ERIC Higher Education Report No. 6, 1993.<br />
Tobias, Sigmund. “When Do Instructional Methods Make a Difference?” Educational<br />
Researcher 11, no. 4 (1982): 4–9.<br />
———, and L. T. Frase. “Educational Psychology and Tra<strong>in</strong><strong>in</strong>g” <strong>in</strong> S. Tobias<br />
and J. D. Fletcher, eds. Tra<strong>in</strong><strong>in</strong>g and Retra<strong>in</strong><strong>in</strong>g: A Handbook for Bus<strong>in</strong>ess,<br />
Industry, Government, and <strong>the</strong> Military. New York: Macmillan Library Reference,<br />
2001.<br />
Treverton, Gregory F. Reshap<strong>in</strong>g National <strong>Intelligence</strong> <strong>in</strong> an Age of Information.<br />
Cambridge: Cambridge University Press, 2001.<br />
Trier, Jost. Der deutsche Wortschatz im S<strong>in</strong>nbezirk des Verstandes. Heidelberg:<br />
Carl W<strong>in</strong>ter, 1931.<br />
Trochim, William. The Research Methods Knowledge Base. C<strong>in</strong>c<strong>in</strong>nati, OH:<br />
Atomic Dog, 2001.<br />
Tulv<strong>in</strong>g, E., and D.M. Thomson. “Encod<strong>in</strong>g Specificity and Retrieval Processes<br />
<strong>in</strong> Episodic Memory,” Psychological Review 80 (1973).<br />
Tversky, Amos, and Daniel Kahneman. “The Belief <strong>in</strong> <strong>the</strong> ‘Law of Small<br />
Numbers,’” Psychological Bullet<strong>in</strong> 76 (1971): 105–10.<br />
———. “Judgment Under Uncerta<strong>in</strong>ty: Heuristics and Biases,” Science 185<br />
(1974): 1124–31.<br />
US Jo<strong>in</strong>t Forces Command. Department of Defense Dictionary of Military<br />
and Associated Terms. USJFCOM Jo<strong>in</strong>t Publication 1-02. Amended through<br />
June 5, 2003. (http://www.dtic.mil/doctr<strong>in</strong>e/jel/doddict/<strong>in</strong>dex.html.)<br />
von Humboldt, Wilhelm. “Uber die Verschiedenheit des Menschlichen<br />
Sprachbaues” <strong>in</strong> B. Bohr, ed. Gesammelte Schriften. Berl<strong>in</strong>: Verlag, 1907.<br />
Voss, James, and Timothy Post. “On <strong>the</strong> Solv<strong>in</strong>g of Ill-Structured Problems”<br />
<strong>in</strong> Michelene Chi, Robert Glaser and Marshall Farr, eds. The Nature of<br />
Expertise. Hillsdale, NJ: Lawrence Erlbaum Associates, 1988.<br />
Vygotsky, Lev. M<strong>in</strong>d and Society, Cambridge: Harvard University Press,<br />
1930.<br />
Wanous, John P. Organizational Entry: Recruitment, Selection, Orientation,<br />
and Socialization. Read<strong>in</strong>g, MA: Addison-Wesley, 1992.<br />
———, et al. “The Effects of Met Expectations on Newcomer Attitudes and<br />
Behavior: A Review and Meta-Analysis,” Journal of Applied Psychology 77<br />
(1992): 288–97.<br />
Warner, Michael. “Wanted: A Def<strong>in</strong>ition of ‘<strong>Intelligence</strong>,’” Studies <strong>in</strong> <strong>Intelligence</strong><br />
46, no. 3 (2002):15–22.<br />
155
BIBLIOGRAPHY<br />
Weiser, M., and J. Shertz. “Programm<strong>in</strong>g Problem Representation <strong>in</strong> Novice<br />
and Expert Programmers,” Instructional Journal of Man-Mach<strong>in</strong>e Studies 14<br />
(1983): 391-96.<br />
Weisgerber, Leo. Vom Weltbild der Deutschen Sprache. Dusseldorf: Padagogischer<br />
Verlag Schwann, 1953.<br />
White, Robert. Task Analysis Methods: Review and Development of Techniques<br />
for Analyz<strong>in</strong>g Mental Workload <strong>in</strong> Multiple Task Situations. St. Louis,<br />
MO: McDonnell Douglas Corporation, 1971, MDC J5291.<br />
Whorf, Benjam<strong>in</strong>. Language, Thought, and Reality. New York: John Wiley,<br />
1956.<br />
Wirtz, James J. The Tet Offensive: <strong>Intelligence</strong> Failure <strong>in</strong> War. Ithaca, NY:<br />
Cornell University Press, 1991.<br />
Wisher, R.A., M.A. Sabol, and J.A. Ellis. Stay<strong>in</strong>g Sharp: Retention of Military<br />
Knowledge and Skills. Alexandria, VA: US Army Research Institute for <strong>the</strong><br />
Behavioral and Social Sciences, 1999, ARI Special Report 39. (.)<br />
Wohlstetter, Roberta. Pearl Harbor: Warn<strong>in</strong>g and Decision. Stanford, CA:<br />
Stanford University Press, 1962.<br />
Wolf, Eric. Pathways of Power. Build<strong>in</strong>g an Anthropology of <strong>the</strong> Modern<br />
World. Berkeley: University of California Press, 2001.<br />
Y<strong>in</strong>, Robert. Case Study Research: Design and Methods 2nd ed. Thousand<br />
Oaks, CA: Sage, 1994.<br />
Ziemke, Carol<strong>in</strong>e, Philippe Loustaunau, and Amy Alrich. Strategic Personality<br />
and <strong>the</strong> Effectiveness of Nuclear Deterrence. Wash<strong>in</strong>gton, DC: Institute<br />
for Defense Analyses, 2000, D-2537.<br />
Zlotnick, Jack. “Bayes’ Theorem for <strong>Intelligence</strong> Analysis” (1972) <strong>in</strong> H.<br />
Bradford Westerfield, ed. Inside <strong>CIA</strong>’s Private World: Declassified Articles<br />
from <strong>the</strong> Agency’s Internal Journal. New Haven, CT: Yale University Press,<br />
1995.<br />
Zou, Yali, and Enrique Trueba. Ethnic Identity and Power. Albany: State University<br />
of New York, 1998.<br />
156
BIBLIOGRAPHY<br />
Web resources<br />
Buros Institute of Mental Measurements (http://www.unl.edu/buros/).<br />
Central <strong>Intelligence</strong> Agency ma<strong>in</strong> website (http://www.cia.gov).<br />
Central <strong>Intelligence</strong> Agency Office of General Council (http://www.cia.gov/<br />
ogc/best.htm).<br />
Central <strong>Intelligence</strong> Agency Officer <strong>in</strong> Residence Program (http://<br />
www.cia.gov//csi/officer.html).<br />
Defense <strong>Intelligence</strong> Agency (http://www.dia.mil).<br />
Federal Bureau of Investigation (http://www.fbijobs.com or http://<br />
www.fbi.gov).<br />
Human Relations Area Files. Yale University. (http://www.yale.edu/hraf/).<br />
National Aeronautics and Space Adm<strong>in</strong>istration-Johnson Space Center career<br />
astronaut biography. (http://www.jsc.nasa.gov/Bios/astrobio<br />
_activemgmt.html).<br />
National Imagery and Mapp<strong>in</strong>g Agency (http://www.nima.mil/).<br />
National Reconnaissance Office (http://www.nro.gov/).<br />
National Security Agency (http://www.nsa.gov/).<br />
US Air Force (http://aia.lackland.af.mil/aia/).<br />
US Army (http://usaic.hua.army.mil/<strong>in</strong>dex.htm).<br />
U.S. Army Center for Army Lessons Learned (http://call.army.mil/).<br />
US Coast Guard (http://www.uscg.mil/USCG.shtm).<br />
US Department of Energy (http://www.energy.gov/eng<strong>in</strong>e/content.do).<br />
US Department of Homeland Security (http://www.dhs.gov/dhspublic/).<br />
US Department of State (http://www.state.gov/s/<strong>in</strong>r/).<br />
US Department of <strong>the</strong> Treasury (http://www.treasury.gov/).<br />
US <strong>Intelligence</strong> <strong>Community</strong> (http://www.<strong>in</strong>telligence.gov/<strong>in</strong>dex.shtml).<br />
US Mar<strong>in</strong>e Corps (http://www.usmc.mil/).<br />
US Navy (http://www.nmic.navy.mil/).<br />
157
AFTERWORD<br />
Joseph Hayes 1<br />
The unexam<strong>in</strong>ed life is not worth liv<strong>in</strong>g<br />
Socrates.<br />
A very popular error: hav<strong>in</strong>g <strong>the</strong> courage of one’s convictions;<br />
ra<strong>the</strong>r, it is a matter of hav<strong>in</strong>g <strong>the</strong> courage for an attack on one’s<br />
convictions!<br />
Nietzsche<br />
Rob Johnston has written a superb book, a study of <strong>in</strong>telligence as it is actually<br />
practiced. Rob’s book is alive with specific, practical recommendations<br />
about how <strong>the</strong> practice of <strong>in</strong>telligence could be made better. The literature of<br />
<strong>in</strong>telligence is overwhelm<strong>in</strong>gly devoted ei<strong>the</strong>r to studies which, however rigorous<br />
<strong>in</strong> <strong>the</strong>ir academic structure, fail to convey <strong>the</strong> humanness of <strong>the</strong> enterprise<br />
or to books and articles, too often self congratulatory or self promot<strong>in</strong>g,<br />
which are little more than assemblages of enterta<strong>in</strong><strong>in</strong>g anecdotes. Rob’s study<br />
deserves a place of honor on <strong>the</strong> very small bookshelf reserved for analytically<br />
sound, deeply <strong>in</strong>sightful works on <strong>the</strong> conduct of <strong>in</strong>telligence. Any serious<br />
discussion of reform or significant change <strong>in</strong> <strong>the</strong> ways <strong>in</strong> which US <strong>in</strong>telligence<br />
is organized, structured, and carried out will need to take this book as a<br />
start<strong>in</strong>g po<strong>in</strong>t.<br />
Rob’s work bears witness to <strong>the</strong> imag<strong>in</strong>ation and commitment to excellence<br />
on <strong>the</strong> part of <strong>the</strong> senior <strong>in</strong>telligence officials who made it possible for a cultural<br />
anthropologist to carry out his field work <strong>in</strong> <strong>the</strong> secret, sometimes hermetically<br />
1<br />
Joseph Hayes is a retired senior officer of <strong>the</strong> Central <strong>Intelligence</strong> Agency Directorate of Operations.<br />
He served more than 30 years <strong>in</strong> <strong>the</strong> clandest<strong>in</strong>e service.<br />
159
sealed, world of <strong>in</strong>telligence. Rob’s work also bears witness to <strong>the</strong> tremendous<br />
passion among <strong>in</strong>telligence professionals to understand better what <strong>the</strong>y do, why<br />
<strong>the</strong>y do it, and how <strong>the</strong>ir work could be improved. There is, <strong>in</strong> this community, a<br />
palpable desire to do better.<br />
S<strong>in</strong>ce <strong>the</strong> tragedy of 9/11 and <strong>the</strong> bitter controversies surround<strong>in</strong>g Iraqi WMD,<br />
<strong>the</strong> world of <strong>in</strong>telligence analysis has been scrut<strong>in</strong>ized with an <strong>in</strong>tensity of almost<br />
unprecedented dimensions. The focus of scrut<strong>in</strong>y, however, has been on <strong>the</strong><br />
results, not on <strong>the</strong> process by which <strong>the</strong> results were produced and certa<strong>in</strong>ly not<br />
on <strong>the</strong> largely anonymous corps of civil servants whose work was at <strong>the</strong> heart of<br />
<strong>the</strong> issue. Those people, and how and why <strong>the</strong>y do what <strong>the</strong>y do, are at <strong>the</strong> heart<br />
of Rob’s important study. If we are ever to make <strong>the</strong> improvements that must be<br />
made <strong>in</strong> <strong>the</strong> quality of our <strong>in</strong>telligence work, <strong>the</strong>n we must beg<strong>in</strong> with a more<br />
mature and nuanced understand<strong>in</strong>g of who actually does <strong>the</strong> work, how, and why.<br />
There is a context with<strong>in</strong> which <strong>the</strong> work is done, a def<strong>in</strong>ite culture with values,<br />
traditions, and procedures that help shape important outcomes.<br />
Rob has characterized a world that I f<strong>in</strong>d all too familiar. It is a world <strong>in</strong> which<br />
rewards and <strong>in</strong>centives are weighted heavily <strong>in</strong> favor of fill<strong>in</strong>g <strong>in</strong> gaps <strong>in</strong> conventional<br />
knowledge ra<strong>the</strong>r than <strong>in</strong> lead<strong>in</strong>g <strong>the</strong> way to alternative po<strong>in</strong>ts of view. It is<br />
a world <strong>in</strong> which confirm<strong>in</strong>g evidence is welcome and rewarded and disconfirmatory<br />
evidence is, at best, unwelcome and, at worst, discounted. It is a world <strong>in</strong><br />
which <strong>the</strong> legitimate and often necessary resort to secrecy has served, all too<br />
often, to limit debate and discussion. It is a world <strong>in</strong> which <strong>the</strong> most fundamentally<br />
important questions—what if and why not—are too often seen as distractions<br />
and not as <strong>in</strong>vitations to reth<strong>in</strong>k basic premises and assumptions.<br />
Much of <strong>the</strong> recent discussion concern<strong>in</strong>g <strong>the</strong> performance of <strong>in</strong>telligence<br />
organizations has been conducted <strong>in</strong> almost mechanical terms. “Connect<strong>in</strong>g<br />
<strong>the</strong> dots,” “m<strong>in</strong><strong>in</strong>g <strong>the</strong> nuggets” are phrases offered as a way of understand<strong>in</strong>g<br />
<strong>the</strong> exquisitely subtle, complex bus<strong>in</strong>ess of mak<strong>in</strong>g <strong>in</strong>telligence judgments.<br />
As Rob Johnston’s book makes abundantly clear, this is first and foremost a<br />
human enterprise. All of <strong>the</strong> <strong>in</strong>tellectual power, biases, and paradigms that<br />
<strong>in</strong>form <strong>the</strong> th<strong>in</strong>k<strong>in</strong>g of <strong>the</strong> people who actually do <strong>the</strong> work need to be understood<br />
<strong>in</strong> <strong>the</strong> organizational context <strong>in</strong> which <strong>the</strong>y do <strong>the</strong>ir work. In <strong>the</strong> f<strong>in</strong>est<br />
tradition of anthropological field research, Rob has observed, collected data<br />
rigorously, reflected with deep <strong>in</strong>sight upon it, and produced a study both<br />
sophisticated and extremely useful.<br />
I worked myself for more than 30 years <strong>in</strong> <strong>the</strong> clandest<strong>in</strong>e operations area of<br />
<strong>CIA</strong>, a part of <strong>the</strong> <strong>Intelligence</strong> <strong>Community</strong> that calls out for <strong>the</strong> same k<strong>in</strong>d of<br />
understand<strong>in</strong>g, professional, and constructive scrut<strong>in</strong>y this book has devoted to<br />
<strong>the</strong> analytic realm. My fervent hope is that <strong>the</strong> human <strong>in</strong>telligence service will<br />
benefit from <strong>the</strong> same k<strong>in</strong>d of rigor and constructive understand<strong>in</strong>g <strong>the</strong> analytic<br />
side has now experienced. My real hope is that Rob is available for <strong>the</strong> job.<br />
160
The Author<br />
Dr. Rob Johnston is an ethnographer who specializes <strong>in</strong> <strong>the</strong> cultural anthropology<br />
of work. He has been a research staff member at <strong>the</strong> Institute for<br />
Defense Analyses and a Director of Central <strong>Intelligence</strong> Postdoctoral<br />
Research Fellow at <strong>the</strong> Central <strong>Intelligence</strong> Agency’s Center for <strong>the</strong> Study of<br />
<strong>Intelligence</strong>, where he is now a member of <strong>the</strong> staff.<br />
Dr. Johnston is a Fellow of <strong>the</strong> Royal Anthropological Institute, <strong>the</strong> Society<br />
for Applied Anthropology, and <strong>the</strong> Inter-University Sem<strong>in</strong>ar on Armed Forces<br />
and Society.<br />
161