Aircraft survivability

Material Information

Aircraft survivability
Place of Publication:
Arlington, VA
Joint Aircraft Survivability Program Office (JASPO)
Publication Date:
Copyright Date:
Three times a year


Subjects / Keywords:
Aeronautics -- Safety measures -- Periodicals -- United States ( lcsh )
Aeronautics -- Safety measures ( fast )
United States ( fast )
Periodicals. ( fast )
newspaper ( marcgt )
serial ( sobekcm )
periodical ( marcgt )
Periodicals ( fast )


Dates or Sequential Designation:
Began with 1998.

Record Information

Source Institution:
University of Florida
Holding Location:
University of Florida
Rights Management:
This item is a work of the U.S. federal government and not subject to copyright pursuant to 17 U.S.C. §105.
Resource Identifier:
656541464 ( OCLC )
TL553.5 ( lcc )

UFDC Membership

Digital Military Collection


This item is only available as the following downloads:

Full Text


Aircraft Survivability Fall 2001 2 A i rc r aft Survivability is published three times a year by the Joint Technical Coordinating Group on Aircraft Survivability (JT C G / A S ) The JTCG/AS is chartered by the Joint Aeronautical Commanders Group. J T CG/AS Central Office 1 2 13 Jefferson Davis Highway C r y stal Gateway #4, Suite 1103 Arlington, VA 22202 P h o n e : 70 3 6 07 3 5 09 D S N : 3 2 7 3 5 09 h t t p : / / j t c g j c t e. j c s. m i l : 9 101 V i e w s and comments are welcome and m a y be addressed to the Editor, Dale B. Atkinson, at E-mail: d b a @ s t a r p o w e r. n e t Phone: 70 3 4 51 3 011 Fax: 70 3 4 51 4 2 7 8 Mailing list additions, deletions, and/or changes may be directed to 46 OG/OGM/OL-A C / S U R V I A C ATTN: Linda Ry a n 2 7 00 D Street, Bldg. 1661 W r i g h t P atterson AFB, OH 45433-7605 P h o n e : 9 3 7 2 5 5 4 8 4 0 D S N : 7 8 5 4 8 4 0 F A X : 9 3 7 2 5 5 9 6 7 3 E-mail: C r eative Dir e c t o r Christina P. McNemar S U R V I A C Satellite Office 3 1 9 0 Fairview Park Drive Falls Church, VA 22 0 4 2 P h o n e : 70 3 2 8 9 5 4 6 4 E-mail: m c n e m a r c h r i s t i n a @ b a h c o m Cover and Newsletter Design Christina P. McNemar About the cover Heavy reliance on models and simulations has resulted in an ever-increasing empha sis on the ability to demonstrate their credibility Contents Guest NotesSome Parting Thoughts on Survivability Modeling and Simulation 3 by M r D a vid H. H a l l P ersonal Thoughts to our Colleagues in the M&S Survivability Community 4 by Mr.James F. O B r yon Assuring M&S Credibility for Defense Acquisition and T&E 6 by Colonel W.F o r rest Cr a i n Modeling and Simulation in Operational Test and Evaluation 10 by Dr. M a r ion L.Williams and Dr.Frank Gray Air Force M&S Policy 12 by Lt Col Gerry Smither and Lt Col Skip Langbehn Pioneers of SurvivabilityDavid H. Hall 14 by Mr.Dale B. A t k i n s o n Survivability, Lethality and Effectiveness Whats in a Word? 18 by Dr.Paul H.Deitz The Need for Speed 22 by Jon S. O g g D i r ector ASC/EN MANPADS Analysis Methodology Development 24 by Mr. A l e x G.K u r tz and Dr.Ronald L. H i n r ichsen M&S Credibility Workshop Assuring M&S Credibility for Defense A cquisition and T&E Survivability Lethality and System Effectiveness 26 by Mr. D a vid H.Hall National Summit on U.S. Defense Policy: A cquisition, Research, Test and Evaluation 30 by Mr.Tr a c y She p p a r d M&S Workshop 200 2 31 Calendar of Events 32


Aircraft Survivability Fall 2001 3 Guest Notes M ark Twain said that Adam, the first man, w a s really in a great position when it came to public speaking (or writing in a new s l e t t e r ) because he knew that when he said something really good nobody else had said it before him. So it is with some trepidation that I put fingers to keyboard to try and say something notew o r t h y about where I think surv i v ability models and simulations (M&S) need to go from here, at the risk of repeating what somebody else m a y have said better already. But after a 33-year career doing modeling, simulation, and weapons sy s t e m s a n a l ys i s I do have a few things left to say. I'd really just like to talk about three thingsnew M&S, M&S credib i l i t y and old M&S. First, I'd like to say to the developers of new M&S architectures and new M&S (like JMASS and AJEM) that if you build it, they probably won't come. Just because you can get it to work, doesn't mean anybody else will use it. It may be a great innovation, but there's a fairly long transition period between building a new M&S tool and getting somebody else to use it on a regular basis. If you wonder why almost nobody is using the JMASS models yet, it's partly because analysts have yet to be convinced that it's in their best interest to spend the time and money required to switch from ESAMS, TRAP or RADGUNS. The same is true of AJEMusers have to be s h o wn that there are capabilities in AJEM that they can't get in JSEM, SHAZAM, or COV A R T And, they have to be convinced that AJEM will be a stable, useable tool that works for their application. Convincing them is going to take time and a lot of effort. So that means that the J T CG/AS and the JTCG/ME are going to have to plan support for existing tools for some significant period of t i m e or they risk dropping support for the tools that e ve r y one is using before there are viable replacements. A d d i t i o n a l l y they need to plan for more training in new M&S, like AJEM, to get people to accept and use them. Secondly, at some point DoD is going to come to it's senses and realize that while everything it does relies on M&S, almost nobody can demonstrate that they work correctly. That's not to say that M&S ha v e no credibili tyit's just that in a lot of cases there is no document ed evidence showing where a model works and where it does not. People who develop M&S go to a lot of effort to convince themselves that their model works correctly, but they almost never write down what they did so that they can convince somebody else! And the pressure of producing new software on schedule and within budget almost alw ay s guarantees that documenta tion, verification, and validation (V&V) fall off the table. So, new software develop ments are no better V&Vd than legacy tools. The JTCG/AS can best serve their DoD and industry customers by putting more funding and effort into V&V and document ing the survivability M&S being used by the community. The ECAT is a good example of what can be done (although it needs to be expanded), as are the continuing efforts b y JASA to provide standard A c c r e d i t a t i o n Support Packages (ASP) for all SURVIA C models and simulations And lastly, at the risk of sounding like a curmudgeon, just because something is old doesn't mean it's not any good. A legacy tool is one that has been used and honed over the years into a thing of beauty, and a j o y forev e r Well, maybe not forev e r But the s u r v i v ability M&S that are in SURVIAC today h a ve been improved, verified, validated, and used over many years by many users for a wide variety of applications. They're not perfect, but they're getting the job done, and they're being constantly enhanced and i m p r o ved by many people. So don't be in a hurry to replace them just for the sake of getting something new or something that's supposedly easier to maintain. You may wish later that you had left well enough alone. And that's about it. I've enjoyed w o r k i n g with the JTCG/AS over the last 15 years or so, and I hope to keep seeing everybody in a new incarnation starting next y e a r Since Dave is retiring in January 2 0 02 we asked him to give us his parting t h oughts and suggestions. E d i t o r Some Parting Thoughts on Sur v i v a b i l i t y Modeling and Simulation by Mr. David H. Hall


station. This input station consisted of a small 8 x 8 foot w i n d o wless room, equipped with a seat and a boxy vintage IBM electric typewriter, which could take commands typed in by the students trying out the teaching machine. It was exciting (and, at times, exasperating) as the fledgling software was written, corrected, improved and applied to help teach student volunteers how to read and write German, take court s t e n o g r a p h y, and perform elementary statistics. My early exposure to the practical application of M&S quickly showed me that it is, indeed a tw o w ay street. Computers and the models that make them serve us do some things well, some things not well, and other things not at all. Computers don't get tired, bored or show other signs of emotion, although I have heard people try to attribute emotions to them on occasion. They can r e t r i e v e and manipulate data extremely well. They are also unforgiving. Use a zero instead of the letter o, or use a comma instead of a period and they almost seem proud to tell you that y o u v e made a mistake (again). In a word, they're a toolnot the end. O v er the past couple of decades, I have watched as the Department of Defense (DoD) has made multiple attempts to manage its growing M&S inv e s t m e n t s by promulgating policies regarding the development and use of M&S. In 1994, a major step in this direction w a s the formation of the Defense Modeling and Simulation Office (DMSO) to serve as the DoD lead in assuring that our M&S plans, practices and investments are w e l l founded and executed. These responsibilities are found in the DoD 5000 Directiv e s Regulations and I n s t r u c t i o n s The Department has also published a v a r i ety of other documents relating to M&S including the DoD M&S Master Plan, Simulation-Based A c q u i s i t i o n a n d other similar documents. The problem today is that, while there are regulat i o n s directiv e s charters and guidelines addressing DoD M&S, few, if any, are being taken seriously or put into p r a c t i c e This fact, again, was recently made abundantly clear as I asked the over 200 attendees of the M&S Conference hosted in February 2 0 0 1 in Reno, Nevada by the JTCG/AS to take a piece of paper and write four senI 'm delighted that this issue of A i rc ra f t S u r v i v a b i l i t y has been devoted to the significant issue of modeling and simulation (M&S). Very few topics conjure up as much emotion and interest as that of M&Sits role, mission, funding, management, realism, configuration control, and future. From the very beginning of my professional c a r e e r computers and simulations have been constant companions. My first exposure to them was in the Biophysics Department at IBM's Thomas Watson Research Center, Y o r k t o wn, New York in the early 60's, while still in undergraduate school. It was a privilege to work with the professionals there, on what was to become the world's first computerbased teaching machine. As I recall, an old IBM 650 and 704 w e r e s o m e h o w hardwired together with an input Aircraft Survivability Fall 2001 4 Personal Thoughts to our Colleagues in the M&S S u r vivability Community by Mr. James F. OBryon


tences summarizing these regulations. Only two or three would even venture a public answ e r After all has been said and done in DoD M&S, much more has been said than done. While policy and procedures are needed and have been promulgated, I see little indication that they have been much more than a wish, an intention, a desire, but certainly not a reality or ev e n a moving force. Another data point that brought this into focus ev e n more was a recent M&S survey which the DDO T & E / L F T office completed and published earlier this y e a r We examined M&S in support of acquisition programs to try to assess the state-of-affairs for a representative cross-section of current defense acquisition programs (22 to be exact). We were able to brief every Service A c q u i s i t i o n E x e c u t i v e, the DDR&E, and the USD(AT&L) on the r e s u l t s From one came the observation, SB A Simulation Based Acquisition? That's just a slogan; it's a bumper sticker. There were other similar comments from those briefed reflecting the unanimous diagnosis that a less than healthy state of M&S management, funding, and application presently exists in DoD. We all know that to get a remedy, we must first admit that we're sick, and then identify what that sickness is. The survey and other testimonials provide ample proof that these directives and regulations have gone unheeded for the most part and there's much work to be done before DoD complies with its own M&S directiv e s Let me list a few candid comments made at the Reno M&S conference worth noting OSD is such a fragmented organization that you can find any opinion you want, maybe youll even find a good one. Working with military instructions is like building a sauna out of ice cubes. All models are wrong, some are useful. E ve r y one wants commonality of models, but they want it in their way. There's no such thing as validating a model. Validation is just a failed attempt to falsify a model. You might argue with a couple of these points, but its clear that much remains to be done. What are some of the M&S challenges facing the Pentagon today? Let me share my own personal feelings on the subject Implement those DoD directives relating to M&S already on the books. Organize an M&S management enterprise (e g consortium) within DoD to directly support the program manager acquisition community across the board with DMSO in lead and adequate resources to do the job. Gain early substantial funding from program managers to invest in M&S enterprise either by mandate or by major revision in PM incentive structure. Implement a Microsoft-windows type of configuration control across M&S used in support of DoD. Reconcile the current dichotomy in modeling supporting the training/readiness community versus the modeling supporting the materiel acquisition community. Make M&S deliverables a requirement for each acquisition program to the maximum extent possible. Stop looking at M&S as a replacement for realistic testing. T&E and M&S are partn e r s not competitors. Stop advertising that M&S will initially s a ve money. For a period of time, in fact, it will actually cost more to address M&S shortfalls since realistic T&E and realistic M&S must both be done to address current VV&A issues. Expand application of M&S to also assist in experimental design. Build into M&S by means of comment c a r d s monitor screen warnings and paper output, the shortfalls and assump tions intrinsic in the M&S being used as w ell as information on the configuration control POC. H a ving made the above-mentioned proposals for the broader M&S community, what challenges face the surviv a b i l i t y / v u l n e r a b i l i ty/lethality/M&S communities? Let me home in on sev e r a l We must develop and apply an M&S archit e c t u r e in cooperation with DMSO, which permits a meaningful and workable analytical bridge between platform state (lev e l of damage and mission degradation) and combat effectiv e n e s s We must avoid using these models and simulations as promotional gimmicks. We must assure that the most widely used models supporting the Survivability comAircraft Survivability Fall 2001 5 c o n t i n ued on page 9


oped specifically for Crusader, which is an excellent example of how simulation reuse can save time and reduce costs. Extensive use of simulations allowed the Crusader team to re-baseline the system in a matter of weeks when they had to trim more than 40 percent of its weight to meet the Arm y s future lighter force requirem e n t s H o w e ve r if program managers cant locate the M&S tools they need, and if after they locate them they dont h a ve confidence in them to provide credible answ e r s then this capability is of no v a l u e The challenge for the M&S community in supporting the Departments acquisition process is to make M&S resources accessible and, most importantly, credible. To appreciate the Departments changing view of the importance of M&S to the acquisition process, take a look at the current DoD 5000 series of acquisition documents and compare them to what they replaced. The use of the words modeling and simulation have gone from a mere mention in a subparagraph, to M&S being emphasized up front in concept exploration, and planned for and used throughout the life cycle of T o d ay s complex weapon systems and systems of systems cannot be dev e l o p e d and evaluated through hardw a r e o n l y test and evaluation (T&E). The reasons are m a n y, but really come down to a cost comparison of hardware prototypes and hardw a r e testing in the appropriate environments. Modeling and simulation (M&S) can help reduce those costs and other risks. The DoD Simulation Based Acquisition init i a t i v e promotes the use of simulations throughout the life cycle of every weapon system, and among all weapon system dev e l o p ment programs. There are tremendous time and dollar savings to be had, and perhaps zero environmental impacts and safety worries if we can wring out systems and certify their performance using M&S before we ever bend metal in production. During the Arm y s dev e l opment of the Crusader cannon artillery system, for example, the program used more than 1 5 0 simulations. None of them were dev e l Aircraft Survivability Fall 2001 6 Assuring M&S Credibility for Defense Acquisition and T&E by Colonel W. F o r r est Cr a i n


weapon sy s t e m s Dr. Delores Etter, former Deputy Under Secretary of Defense for Science and T e c h n o l o g y s a w an increasing appetite for M&S tools as adv a n c e d t e c h n o l o g i e s both Government and commercial, promise significant increases in M&S capabilities. She also noted that many barriers still exist in M&S (e.g., human b e h a vior representation, 4-D presentations) and supporting technologies (e.g., software intensive sy s t e m s processing power and bandwidth), but emphasized that DoD must continue to address those issues. The Defense Modeling and Simulation Office (DMSO), as the Departments focal point for M&S, has focused its attention in three key areas that are benefiting not only the acquisition community, but the training, analysis and experimentation communities as w e l l The first area is architecture. The DMSO-dev e l o p e d High Level Architecture (HLA) is the accepted standard for distributed simulations in the DoD. Further, it is becoming an industry and international standard as well. It has been accepted by the North Atlantic T r e a t y Organization (NATO); was adopted as an international standard by the Object Management Group (OMG) in N o vember 1998; and cleared the final hurdle to becoming an Institute of Electrical and Electronic Engineers (IEEE) standard in September 2000. The functionality of the HLA is based on the premise that no one simulation can satisfy all uses and users. It allows an individual simulation or set of simulations developed for one purpose to be used for another purpose by federating them. The c o l l a b o r a t i v e environments established at the Air F o r c e Research Laboratory at Wright Patterson AFB, Ohio, and the Arm y s Night Vision Laboratory at Fort Belv o i r Virginia, are good examples of using the HLA in an acquisition process. Further, the HLA complements the Test and Training Enabling Architecture (TENA) in purp o s e design, development, and implementat i o n The second area of DMSO emphasis is environmental representation. DMSO inv e s t m e n t s in Integrated Natural Environment (INE) projects have lowered barriers to the use of M&S. INE projects include the Master Environmental Library (MEL), a one-stop shop for discovery and ordering data sources in all environmental domains; the Synthetic Environment Data Representation and Interchange Specification (SEDRIS) that allows standard data interchange and unambiguous data representation; and the Environment Scenario Generator (ESG) that offers tailored scenario generation and just-in-time-production ordering. What are the benefits? You can build tailored, physically consistent scenarios to achieve test plan objectiv e s You can generate environmental conditions that otherwise would be difficult to come by, such as those in denied areas of the world, or that w o u l d require risk of injury or damage to the environment. And, you can operate at slower or faster than real time to focus on critical sy s t e m and/or environmental components. F i n a l l y the DMSO focuses on its role in d e v eloping DoD M&S policy, especially with respect to verification, validation and accreditation (VV&A). VV&A is the critical process for gaining assurance that a selected model or simulation can produce credible results. With community inv o l v ement, the DMSO led the d e v elopment of DoD Instruction (DoDI) 5 0 0 0 6 1, DoD M&S Verification, V a l i d a t i o n and Accreditation, which established common terminology and defined high-level roles and responsibilities across the Department. The DMSO also worked with the M&S community to develop the Web-based DoD Recommended Practices Guide (RPG), which defines the underlying philosophy, principles, and methodologies recommended for use in DoD VV&A efforts. You can review the RPG online at h t t p : / / w w w m s i a c d m s o m i l / v v a / While Department-wide application of those recommended practices will help assure the Aircraft Survivability Fall 2001 7 V isit the VV&A Recommended Practices Guide online at c o n t i n ued on page 8


Colonel W. Forrest Crain, U.S. Army, has served as the Director of the Defense Modeling and Simulation Office since March 1, 2000. He received his commission from the United States Military Academy in 1975. He is currently pursuing a doctorate in Information Technology at George Mason University. He has served in a variety of combat arms positions throughout his career. Before joining the DMSO he served as Chief of Strategic Plans for the Multi National Division (North) in Bosnia. Colonel Crain turned the directorship of the DMSO over to Navy Captain Michael G. Lilienthal on September 4 and retired from active duty on September 7. Captain Lilienthal is now the Director of the Defense Modeling and Simulation Office (DMSO), Alexandria, Va. Captain Lilienthal is a triple "Domer" of the University of Notre Dame. He received a B.A. degree in Psychology with a minor in Physics in 1973; followed by a M.A. degree in Experimental Psychology in 1975; and a Ph.D. in Experimental Psychology in 1978. He was commissioned in 1978. Captain Lilienthal was board certified as a Professional Ergonomist (No. 462) in 1994. Dr. Lilienthal is a fellow of the Aerospace Human Factors Association and the Aerospace Medical Association. He is also a member of the Human Factors and Ergonomics Society Editors Note: This article is a recap of Colonel Crain s presentation at the February 12-15 conference on A ssuring Modeling and Simulation (M&S) Credibility for Defense A cquisition and T&E in Reno, Nevada. credibility of M&S, each of the Military services and DoD Components are responsible for d e v eloping VV&A policies and guidance to suit their organizational needs. The level of use of M&S by DoD and industry is a paradigm shiftand the paradigm continues to shift as technology offers us b i g g e r b e t t e r f a s t e r M&S is a proven, coste f f e c t i v e tool for getting the acquisition job done cheaper, quicker, safer and cleaner, but taking full advantage of it will require the continued commitment, cooperation, education, and evolution of the M&S and acquisition c o m m u n i t i e s The DMSO can promote the confident use of M&S for acquisition through the establishment of standards that encourage interoperability and reuse, through the development of environmental representation resources that reduce T&E costs and risks, and through the promulgation of Departmental policies and practices that guide the development of credible models and simulations, but in the end assuring the credibility of M&S for Defense acquisition and T&E is a task that has to be worked at all lev e l s Aircraft Survivability Fall 2001 8 c o n t i n ued from page 7 Tom Wasmund from the Naval Surface Warfare Center at Dahlgren, Virginia, retired on 30 September 2 0 0 1 after over 40 years of g o vernment and military service. Tom graduated from the University of Washington with a BSEE in 1959, and started his career as a Na v y pilot by entering flight training later that y e a r He also received a BSAE and a MSAE from the N a val Postgraduate School in 1966 and 1967, r e s p e c t i ve l y Among his other contributions, Tom has been the driving force behind the d e v elopment of the A d v anced Joint E f f e c t i v eness Model (AJEM) that just entered service this y e a r He has been a long time member and chairman of various vulnerability and lethality committees for both the JTCG/AS and the JTCG for Munitions E f f e c t i v eness (JTCG/ME), and has been the AJEM Project Manager since 1993. AJEM, sponsored by both the J T CG/AS and JTCG/ME, is a lethality/vulnerability/end game model meant to account for all types of w a r h e a d and projectile kill mechanism effects against all types of air targets in a single model. Tom has been cited by the Chairmen of both organizations for his outstanding leadership and perseverance in successfully carrying this project through to the release of AJEM 1.0 in July of this y e a r We wish Tom the very best in his retirement. AJEM Guru Retir e s


Aircraft Survivability Fall 2001 9 capabilities and issues, and improving the computer modeling of weapons system lethality and vulner a b i l i t y I believe str o n g ly that Live Fire Testing should be continued. The DoD needs to reclaim its leadership in M&S. Recall that the very first computer and simulation w a s built for the U.S. Army under a contract with the U n i v ersity of P e n n sy l v ania for the purpose of producing ballistic tables. Furthermore, the first image generated by a computer was also performed by the Army, that of a simple line drawing of an XM-1 tank, some 30 years ago at the Ballistic Research Laboratories (now ARL) at Aberdeen Proving Ground. While the private sector community has transformed the world of computers the ability of DoD to effectively manage its M&S has not kept pace, resulting in w a s t e duplication of effort, and misuse of M&S in applications for which they have not been v a l i d a t e d W e v e come a long way from the simple teaching machine of the early 60s and the simplistic two dimensional black and white computer drawing of an Army v e h i c l e and for that, were all grateful. W e v e gotten to the point where our computer-generated output looks close enough to reality that we could mistake it for the real thing. And therein lies the problem. I think too m a n y in our community have extrapolated this leadingedge computer-aided design (CAD) capability to one of assuming that the physics of vulnerability/lethality/surv i v ability of those computer representations is equally understood. Nothing could be further from the truth. Colonel Forrest Crain, DMSOs Director recently stated that real progress in M&S will be determined by demonstrated benefits, not policy directives or rhetoric. The JTCG/AS, working in close cooperation with the Joint Technical Coordinating Group for Munitions E f f e c t i v eness (JTCG/ME), is uniquely positioned to provide that leadership within the surviv a b i l i t y / v u l n e r a b i l i ty/lethality communities. I invite all readers of this publication to join us in this effort. I welcome your feedback. I may be reached at jobry o n @ d o t e Mr. OBryon serves as Deputy Director, Operational Test and Evaluation in the Office of the Secretary of Defense and has directed the Live Fire Test Program since 1986. He also has oversight responsibility for the JTCG/AS and JTCG/ME programs of the DoD. He is a graduate of The K i n g s College, George Washington University and Massachusetts Institute of Technology munity are promptly VV&Ad down to their source codes, correcting errors where possible and publishing cautions, assumptions and shortcomings explicitly as an integral part of the M&S output. We must migrate to best-of-breed models and cease supporting models which do not reflect the state-of-the-art for a giv e n application. We must maintain effective and current configuration control of M&S under our p u r v i e w We must realistically account for secondary damage mechanisms and cascading d a m a g e We must migrate from empirically-based M&S to physics-based vulnerability/lethality models. We must fully support the program management community without losing our o b j e c t i v i t y We must educate DoD and the private sector on M&S regulations, requirements, c a p a b i l i t i e s databases and methodologies available and formalize a forum to ensure that this occurs on a regular basis. We must develop meaningful risk/costbenefit methodologies which can support wise investment decisions and surviv a b i l i ty design trades. S u r v i v ability and lethality M&S and testing continue to be strongly supported by the Administration. The following response to a question on June 22, 2 0 0 1 from Senator Strom Thurmond by the Honorable Michael W y n n e Principal Deputy Undersecretary of Defense for Acquisition, Technology and Logistics says it w e l l Live Fire Testing is an important and integr a l part of the Department's weapon system test and evaluation pr o c e s s providing timely and accur a t e assessments of system survivability, vulner a b i l i t y and munitions lethality. Live Fire Testing also provides insights into methods of reducing the vulnerability and improving the lethality of weapons and weapon platf o r m s assessing battle damage r e p a i r Personal Thoughts to our Colleagues... c o n t i n ued from page 5


M&S. Certainly M&S can increase the efficiency of the design and development process, create products that work better after fewer prototypes, and thus save time and money through fewer re-tests. Operational testers should be able to take advantage of these same simulat i o n s How e ve r with few exceptions, modeling and the integration of models into simulations are fragmented, under funded, and pretty much the exclusive domain of d e v elopment contractors. The following is our view of h o w M&S should fit into OT&E and what we think is necessary to make this happen. First the Basics OT&E is defined in Title 10, U.S. Code, as the field test, under realistic combat conditions, of any item of (or key component of) w e a p o n s equipment, or munitions for the purpose of determining the effectiv e n e s s and suitability of the w e a p o n s equipment, or munitions for use in combat by typical military users; and the evaluation of the results of such test. Title 10 also s a ys that OT&E does not include an operational assessment based exclusively on (a) computer modeling [or] (b) simulation. None of this precludes appropriate uses of M&S in support of OT&E. In fact, DoD 5000.2R stipulates that test planning shall consider M&S, and test results shall also be used to dev e l o p and improve models and simulations. It further states when actual testing is not possible to support an operational assessment, such assessments may utilize computer modeling and/or hardware in the loop simulations (preferably with real operators in the loop). DOT&E policy also permits using M&S to ev a l uate test results. So our formal direction is to use M&S for planning better tests and producing better ev a l u a tions of test results. Not to replace tests. M&S may have its biggest impact in planning and predicting operational tests. During planning, simulations can help identify more profitable areas for testing, define data and instrumentation requirements, and provide a venue for running factor-screening experiments. Test predictions can significantly improve our test process by making sure that we understand what to look for and to help us recognize when events depart from Y ears ago, Clay Thomas, one of the Air F o r c e s wise old men, was quoted as s a ying Artificial Intelligence has great potentialit always will have. Clay, kno w n for his subtle statements, would probably have said the same thing about modeling and simulation (M&S). Over the past few y e a r s many expectations for modeling and simulation (M&S) have outpaced capabilities. This has certainly been the case with respect to Operational Test and Evaluation (O T & E ) a p p l i c a t i o n s Like many Department of Defense testers, we are receiving increased pressure to reduce test costs by substituting Aircraft Survivability Fall 2001 10 Modeling and Simulation in Operational Test and Evaluation by Dr. Marion L. Williams and Dr. Frank Gr a y


the expected. The trick is to do this at a cost that doesnt exceed benefits. Our experience has been that simulations needed for test planning or test prediction are rarely available in a form usable by operational testers. Usually models d e v eloped to support acquisition are internal products of the dev e l o p e r The models are not documented, are not a deliv e r a b l e and the results of specific studies using the models are not in a form releasable to external agenc i e s We have tried to develop models for our own O T & E u s e or to significantly modify existing models. But the results have been very high costs ($10+ million for one program) with minimal benefits. We either started too l a t e or ran out of moneyor both. On a more positive note, we have had some success in using models for operational assessments. Air-to-air missile developers routinely use modeling for sy s t e m design and development, and operational testers have been able to take advantage of this investment. A recent example is the AIM-9X, where the program office and contractor included the testers from the beginning, incorporating functionality for testers and faithfully including their validation requirements. Models used to predict test results have been more valuable when there is feedback from the test into the model to correlate the results, correct model inaccurac i e s or modify incorrect assumptions. But we have learned that tests must be planned with such correlation in mind. It has been very difficult to correlate models and tests after the fact. In one particular effort designed to correlate model and test results, we were never surprised when the model of the test item was used, but were always surprised when the actual test item w a s used. There were always factors in the test that were not accounted for in the model. Once we learned of the problem, the model could be corrected. M&S in OT&E will become cost-effective tools only when we learn to routinely leverage on M&S inv e s t ments and efforts by dev e l o p e r s There should be an integrated M&S plan for each acquisition program outlining the requirements for each application, and with a coordinated modeling effort. Models used by contractors for design should be made available for other applic a t i o n s including OT&E. There needs to be a plan for comparing test data with model data throughout the acquisition cycle so that the model-test-model process is continuous from initial development through operational testing and employment. In many instances there is no convenient way for this to happen except in an ad hoc manner. Use of models in direct support of OT&E is an important topic, but largely undefined. We k n o w that models can support testing, but our mission is to put the real system in the real environment with real operators to make sure it can accomplish its operational task. We fully understand that even a field test of a system is a simulation of the true operational environment, but it provides the only opportunity to make sure that the system works not only as designed, but accomplishes the intended mission. The truest source of OT&E data is from a real test item in a test environment. Sometimes an acceptable substitute is the real test item in a simulated environment. For example, a realistic electromagnetic environment is often impossible to create outdoors. So we use facilities such as anechoic chambers to stimulate a real test item with a dense electronic environAircraft Survivability Fall 2001 11 c o n t i n ued on page 1 7


g u i d a n c e As Figure 1 shows, how e ve r the AFIs dev e l oped under this policy did not have a consistent theme or focus. Some AFIs were M&S-centric, such as those on VV&A, Model Management, and Data Standards while the AFI on Acquisition was functionally oriented. Extending across these AFIs were crosscutting functional areas such as acquisition, test, intelligence, and logistics. Each AFI was first drafted within AF/XOC and then coordinated across the Air F o r c e After years of effort, only the AFIs on VV&A and Exercises & Wargames made it through the coordination cycle successfully and w e r e published. U n f o r t u n a t e l y the AFI on VV&A was successfully coordinated and published only after it was w a t e r e d d o wn and broadened to the point where it gave only vague guidance making it practically useless. Why didnt this approach to AFIs work? Before answering this question, we need to address VV&A and its links to M&S in the decision process. V&VHow Much Is Enough? Models and simulations provide information as one of many inputs to a decision-maker. Thus, models and simulations are merely tools used in a process designed to achieve a result. Figure 2 outlines this decision-making process from an M&S perspective. An informed decision is made by the decisionmaker based on alternatives presented by the trained a n a l y st who has done a careful analy s i s grounded in accredited models and simulations carefully chosen from a strong base of possibilities that have been v e r ified and validated for that purpose. Drawing conclusions (the decision) is the responsibility of the d e c i s i o n m a k e r The purpose of the analysis is to provide critical insights and present a cogent summary of results to the decision-maker. From an M&S-centric perspective, the analy s i s depends on accredited information provided by a suite of verified and validated models and simulations. T h e r e f o r e the experienced decision-maker is critically dependent on V&V. In reality, the decision-making process rarely works as shown in Figure 2. Figure 3 shows what really happens. The decisionmaker makes an informed decisionPERIOD! A dramatic shift has occurred in the d e v elopment of Air Force Modeling and Simulation (M&S) policy. This article contrasts the old and new ways of d e v eloping policy and addresses V e r i f i c a t i o n Validation and Accreditation (VV&A) as a specific topic. Headquarters Air F o r c e Directorate of Command & Control (AF/XOC) is the Air Staff organization responsible for developing Air Force M&S policy. This policy is implemented through directives and instructions. Secretary of the Air Force for Acquisition, Information Dominance Directorate (SAF/AQI) is the organization on the staff of the Secretary of the Air Force responsible for M&S policy for acquisition. Because of this tie, the two organizations work closely together. M&S PolicyThe Old W a y Figure 1 depicts how Air Force M&S policy used to be structured. Air Force Policy Directive (AFPD) 16-10 provided general guidance and o b j e c t i v es and served as an umbrella for Air Force Instructions (AFIs) that provided specific Aircraft Survivability Fall 2001 12 Air Force M&S Policy by Lt Col Gerry Smither and Lt Col Skip Langbehn Figure 1. Air Force M&S PolicyPrevious Structur e


E v erything else below the line may or may not influence the decision-maker and the decision made. The tenuous chain that links M&S and V&V to the decision-maker is easily brokenwith good reason. As General Shaud, the former Chief of Staff, Supreme Headquarters Allied Powers Europe, said, process is no substitute for careful thought. Results from M&S are simply one of many inputs to the decision-maker. This begs the question how much V&V of the models and simulations is enough? The M&S party line says that VV&A is all about risk reduction, not for the M&S comm u n i t y but for the decision-maker. How e ve r the reality is that decisions, even bad ones, will be made regardless of the quality or quantity of V&V. When a bad decision happens there is an inevitable search for the guilty p a r t y V&V wont be the reason for the bad decision, but overlooking it makes M&S an easy, obvious target. Once M&S is found guilty, the search stops. VV&A is used to develop a clear understanding of the strengths and weaknesses of the tools being used. U n f o r t u n a t e l y all the V&V in the world will not stop a bad analy s i s sidetrack a bad decision, or save a troubled program. Given all of this, why do we continue to focus much of our attention, much of our funding, and much of our effort on the V&V of models and simulations when V&V is, by any understanding, only a small part of the story? The answer is that its easier to focus on the tools and toys rather than on the people and processes. To compound the problem, attempting to write a single policy to guide V&V policy almost guarantees a one size fits all mindset. As Figure 4 illustrates, a one-size V&V does NOT fit all. In fact, theres a spectrum of V&V that ranges from relatively useful to practically useless. One line of the diagram indicates the relative usefulness of V&V, while the other line indicates the relative situational c o m p l e x i t y The left side of the diagram is characterized by few systems and generally one v e r sus one assessments that are detailed but show l o w situational complexity. V&V dominates in this realm and tends to be very objective and empirically v e r i f i a b l e Examples can be found in the M&S done by the Test and Ev a l u a t i o n community where detailed engineering modeling demands rigorous V&V. The right side of the figure represents a chaotically complex situation, characterized by thousands of systems and generally force-onforce assessments that are empirically unv e r i f i a b l e In this situation, theories abound and facts are few. Accreditation dominates in this realm and V&V tends to be very subjective. Warfighting decisions fall into this category Aircraft Survivability Fall 2001 13 c o n t i n ued on page 1 6 Figure 2 Decision-Making ProcessThe Theor y Figure 3. Decision-Making ProcessThe Reality Figure 4. The Spectrum of V&V


enhancement features for Marine Corps helicopters against air-to-air threats. In 1979 he became head of the Ordnance Systems Evaluation Branch where he supervised 10 weapons systems analysts performing design a n a l y ses of anti-air ordnance for a variety of w e a p o n s sy s t e m s including Phoenix, Standard Missile, Sparrow, AMRAAM, AIAAM, and other systems in exploratory d e v elopment. Dave was introduced to the JT C G / A S when he attended a meeting of the V u l n e r a b i l i t y A n a l y sis Subgroup at W r i g h t P atterson Air Force Base. D a ve also represented the Division in the JTCG/ME as well as the on The Technical Cooperation Program ( T T CP) fuze and analysis working groups. TTCP is a c o o p e r a t i v e effort with the nations of Australia, New Zealand, Canada and the United Kingdom. In 1984 Dave became the Chief Analyst of the S u r v i v ability Division, providing technical oversight of all division projects, and is also the Branch Head of the t w o analysis Branches in the Division, supervising up to 30 scientists, engineers, and analy s t s The Surviv a b i l i t y Division is responsible for developing and demonstrating air vehicle survivability technology, and tracking the maturation of that technology into the fleet. The analysis branches are responsible for evaluating requirements and technologies in the disciplines of air combat surv i v ability and anti-air missile lethality. W e a p o n s S y stems supported by Dave and his people include the F-14, F/A-18, V-22, A-12, AX, A-6, AV-8B, H-60, H-1, H46, H-53, T o m a h a wk, as well as others. Since 1987, Dave has served as the Chairman of the J T CG/AS Methodology Subgroup (now the Surviv a b i l i t y Assessment Subgroup) coordinating the dev e l o p m e n t of survivability methodology across the Services. Dav e s outstanding methodology expertise and friendly and c o o p e r a t i v e manner served him and the JTCG/AS w e l l over the y e a r s The Methodology Subgroup has done a lot with relatively little funding, under rapidly changing Service and OSD M&S initiativ e s and changing times in the acquisition world in general. From 1992 to 1996, he also served as the Joint Project Manager for the OSD sponsored Susceptibility Model Assessment and Range Test (SMART) project. SMART developed and demonstrated joint model and D ave Hall, who has been a mainstay of the JTCG/AS, has decided to retire in January 2 0 0 2 and we want to take this opportunity to recognize Dave for his long service and many contributions to the J T CG/AS, JTCG/ME, SURVIAC, and the ov e r a l l s u r v i v ability community over many y e a r s D a ve grew up in Lakewood, California and r e c e i v ed a B.S. and M.A. in Mathematics from California State University in 1968 and 1972, r e s p e c t i ve l y Dave worked his way through college at Disneyland, first selling ice cream and then picking up trash. Dave met his wife, T e r r i his freshman year in golf class and they w e r e married in June 1968. A week later, Da v e went to work in the Fuze Department of the Naval W e a p o n s Center that was originally located in Corona, California. He worked as a mathematician evaluating fuze concepts for Standard Missile the ZUNI rocket, a non-nuclear hard point defense missile, and other related s y stems. In 1972, he became an operations research ana lyst in the Weapons Analysis Division where he wrote a major portion of the Exploratory D e v elopment Requirements Document on S&T initiatives for Navy airborne weapons sy stems and conducted analyses to support the identified requirements. He worked on a major pro ject to develop and implement a methodology to evaluate advanced w e a p o n s system design concepts, tactics, and force structure that inv o l v ed working on a daily basis with operational Navy pilots. He also conducted a Lanchester analysis of the Y o m Kipper War that introduced him to the survivability area. In 1977, Da v e went to work for John Morrow in the Survivability and Lethality Division, and has worked in the survivability and lethality area ever since. He initially managed a project to develop a methodology for evaluating alternative surviv a b i l i t y Aircraft Survivability Winter 2000 14 by Mr. Dale B. A t k i n s o n


simulation verification, validation, accreditation, and configuration management processes now being used by the Joint Strike Fighter (JSF) and many other organizations and programs. This effort resulted in the establishment of the Joint Accreditation Support A c t i v i t y ( J ASA), which provides M&S support services to many DoD system acquisition programs, such as the JSF, the F / A1 8 E / F AIM-9X, ESSM, RAM, and others. In 1997 he r e c e i v ed the National Defense Industrial Associations S u r v i v ability Leadership Award for planning and executing the SMART project that developed a rational c o s t e f f e c t i v e process for verifying, validating, and accrediting (VV&A) models that support acquisition d e c i s i o n s . Dave said that the SMART project was the most interesting major project he was inv o l v ed with over the duration of his career, and he really appreciated the opportunity to work with an outstanding team of professionals, from both Government and industry. D a ve also said that the JASA team that developed from the SMART project has been a joy to work with and has p r o vided invaluable support to a number of acquisition p r o g r a m s During this time, Da v e also represented the U.S. on the TTCP Panel for Weapons Evaluation where he served as the U.S. Mission Effectiveness Focus Officer and led a key technical area investigating cooperativ e analysis techniques for the evaluation of advanced w eapons s y stems technology initiatives. Da v e has also been the Navy JTCG/AS member of the SURVIA C T echnical Coordinating Group (TCG), composed of both JTCG/AS and JTCG/ME members. The TCG pro vides technical o v ersight of SURVIAC. During 1998 and 1999, Dave served as the Modeling and Analysis lead for the JSF Survivability Integrated Product Team (IPT) under the JSF Joint Program Offices Systems Engineering Directorate. He w as responsible for coordinating survivability method ology and analysis used in support of s y stem design and evaluating alternate technology options. Da v e is also the chairman of the Na v al Air Warfare Center Weapons Division (NAWCWD) Analysis R e s o u r c e s Science and Technology (S&T) Network. The Network coordinates analytical resources and technologies across the various sites of the Center and administrates discretionary funds for impro v ements in analytical capabilities for evaluating S&T initiatives Da v e is also as a member of the NDIA Combat Survivability Division (CSD) Executive Board as well as the Chairman of their Strategic Planning Committee. Da v e has taught the analysis, modeling, and simula tion part of Professor Balls survivability short course since 1985, and continues to do so. Da v e has received numerous a w ards o v er the y ears, including being selected as a Fellow of the Na v al Weapons Center in 1993 and the Michelson Laboratory A w ard in June 2001. Da v e is very proud of his wife and family His wife, Terri, served as the CEO of Desert Area Resources and Training (DART), a nonprofit agency that serves developmentally dis abled adults and children. Terri worked there 27 years and upon leaving recently, DART announced that they are naming a new build ing after her. RADM Johnson, the NA W CWD C o m m a n d e r also presented her with a plaque for her service to the Navy and the community. Their two children, Julie and Jason, are both grown and ha v e successful careers of their own. Da v e and Terri mo v ed to the coast in Carlsbad, California in July of this year, where Terri has accepted a position as Director of an agency serving the homeless Da v e is commuting to China Lake until he retires and then plans to become a consultant upon his retirement. On behalf of the JTCG/AS and the surviv ability community as a whole, I would like to a c k n o wledge Dave as someone who has made a difference and express our apprecia tion to him for doing such an outstanding job o v er the years. We wish him well in his new career as a consultant. M r Dale B. Atkinson may be reached at dba@ s t a r p o w e r. n e t Aircraft Survivability Winter 2000 15


General Sandstrom of AF/XOC, one author of this paper offered the following choice w e v e been almost spectacularly unsuccessful at writing, staffing, and getting coherent policy out to the Air F o r c e . Either the civilians and military...both past and present, are un w i l l i n g untrained, unable, uneducated, underachievers too ignorant to properly write and staff a package, or w e r e going about this wrong. F o r t u n a t e l y Brigadier General Sandstrom chose the latter reason and agreed with the conclusion that, w e v e been trying to write broad guidance for folks who need specifics. As a result, we came up with a new approach to dev e l op M&S policy by using Integrated Product T e a m s ( I P T s). By rotating Figure 1 by ninety degrees, the focus is now on the functional areas and how M&S should be used in a supporting role rather than writing M&S policy for the sake of M&S. AFPD 16-10 now acts as the umbrella for a series of functionally oriented AFIs. We formed a small, dedicated IPT with a rotating co-chair to d e v elop the AFIs. One chairperson was drawn from the functional lead while the other co-chair is from A F / X OC. AFI 16-1 0 0 2, M&S Support to Acquisition, w a s the first AFI developed. Our effort was chaired by S A F / A QI and co-chaired by AF/XOCA. As the focus shifts to a different functional area, a different IPT member is designated as the chairperson, but the core members of the team remain the same. This structure allows the IPT to create AFIs that are linked, address a specific functional problem, and incorporate crosscutting M&S i s s u e s Finally, having the same IPT members has the added benefit of avoiding stovepiped approaches. Without a community or functional area focus we lose support for M&S. Broader issues like VV&A and data standards lose meaning and support when separated from the end user. Experience shows we can either write ineffective M&S AFIs that attempt to be all things to all people or write tailored AFIs that address specific functional area needs. This AFI team is designed with cross-functional representation with, as a minimum, r e p r e s e n t a t i v es from logistics, test & evaluation, acquisition and intelligence. The IPT drafts each AFI by identifying problems in the particular functional area being addressed that the use of M&S could help resolve. AFI 16-1 0 0 2, M&S Support to Acquisition, was published in June 2000. AFI 16-1 0 0 3, M&S Support to T e s t and Evaluation, is now in coordination and publication is planned for the Fall timeframe of 2 0 0 1. The third AFI in the series, AFI 16-1004, Threat M&S Validation, is n o w in final draft form and ready for the first round of as the situational complexity increases, the rela t i v e importance of V&V decreases. The bottom line is that V&V, while useful, is not the key player in command decisions. Perhaps the term "VV&A" should be recoined as "A V&V" to remind us of the spectrum of V&V that exists and the relative importance of Accreditation and Verification & Validation. As author Steven Covey reminds us in his book, The Seven Habits of Highly Effective P e o p l e begin with the end in mind. Although VV&A represents the chronological order of ev e n t s this terminology generally c o n ve y s the notion that Accreditation is an after-thought and that a one size fits all V&V e x i s t s For planning purposes, the Accreditation decision and process must be considered first. The key to successful V&V is a clear understanding of the decision being m a d e the decision-maker, and the relative importance of the information being pro v i d e d by the M&S. Only then can the V&V of the models and simulations be properly scoped and planned. With this discussion of VV&A as background, were back to our original question, w h y didnt this approach to AFIs work? Ev e n though the current AFI on VV&A was successfully staffed and approved, it was practically useless because the guidance it provided w a s too broad. In a briefing to (then) Brigadier Aircraft Survivability Fall 2001 16 c o n t i n ued from page 1 3 Figure 5. Air Force M&S PolicyNew Structur e


ment. But we dont want to generate operational test results with a model of the test item. Operational testers are in the front of the pack in attempting to use M&S to improve testing. We want to do it where it makes sense, and where models to support those efforts are already av a i l a b l e We are not, how e ve r planning to replace operational testing with simulation, to use M&S in areas where it doesnt make sense, or to use inadequate models. Dr. Marion L. Williams is the chief scientist, Headquarters Air Force Operational Test and Evaluation Center, Kirkland Air Force Base, New Mexico. The center is responsible for testing mor e than 250 major programs at 20 different loca tions worldwide. Serving in this role since 1980, he provides technical direction and guidance to the AFOTEC commander, the headquarters staff, the AFOTEC detachments and operating loca t i o n s and various external organizations to ensure the scientific proficiency of Air Force Operational Test and Evaluation. Dr. Williams r eceived his B.S. in Aeronautical Engineering from Texas A&M University in 1956, his M.S. in Mechanical Engineering from the University of New Mexico in 1971, and his Doctorate Degree in Industrial Engineering and Management (Operations Research), from Oklahoma State University. Dr. Williams is a Fellow of the MOR S and recipient of the prestigious Vance R. Wanner Memorial Award for contributions to military operations research. Dr. Frank B. Gray is the Deputy Technical D i re c t o r Air F o r ce Operational Test and Evaluation Center, Kirkland Air Force Base, New Mexico. He is Responsible for technical oversight of F-22, Joint Strike Fighter, and other tests con ducted by the Special Test Directorate. Dr. Gray has been involved with test and evaluation in the Air Force since 1978. He received his B.S. and M.S. in Aeronautical Engineering from Ohio State University, and his Ph.D. in Industrial Engineering and Experimental Statistics from New Mexico State University Modeling and Simulation in Operational Test and Evaluation c o n t i n ued from page 1 1 coordination. This functional approach allows M&S issues like VV&A to be addressed within the functional A F I s allowing us to cancel the separate AFI on VV&A. S u m m a r y The Air Force is taking a functional approach to the d e v elopment of M&S policy. Good policy helps people in the field do their job b e t t e r Dont write broad guidance for folks who need specifics. Modeling and simulation is only a small part of the s t o r y Its important to understand where it fits in the overall big picture. Dont focus on the tools and toys rather than on the people and processes. VV&A, while useful, is not the key player in command decisions. There is no one size fits all when it comes to V&V, but rather a spectrum of V&V depending on the situational complexity. V&V should not be the tail that wags the dog. Its important to separate the V&V from the A. A more appropriate term for VV&A is A V&V to emphasize that Accreditation should determine both the scope of the M&S effort and the V&V r e q u i r e d Lieutenant Colonel Gerry Smither was a co-chair of the Integrated Product Team that developed AFI 16-1002, M&S Support to Acquisition. Until recently, Lt Col Smither was the Air Force Program Element Monitor for Modeling and Simulation working in the office of the Secretary of the Air F orce (Acquisition), Information Dominance Directorate S AF/AQI. This office is responsible for acquisition modeling and simulation policy within the Air Force and acquisition oversight of modeling and simulation programs. Lt Col Smither is now with the Single Integrated Air Pictur e (SIAP) System Engineering Task Force Lieutenant Colonel Skip Langbehn was a co-chair of the Integrated Product Team that developed AFI 16-1002, M&S Support to Acquisition. Until recently, Lt Col Langbehn was a member of the Air Staff as Chief, Analysis Development B r anch, MS&A Pr o g r ams Division, Dir e c t o r ate of Command and Control (AF/XOC). This office is responsi ble for Air Force Modeling and Simulation policy and pr ograms. Lt Col Langbehn is now with the Joint Staff in J 8 / Fo r ces Division. He may be reached at skip. Aircraft Survivability Fall 2001 17


What field observables are comparable to the standard S/L/E model metrics? During the onset of live-fire testing and modeling in the 1980s, the answers to these questions werent so obvious to the ballistic vulnerability workers in the A r m y. The initial live-fire (LF) tests against the M113 and then the Bradley resulted in a multitude of contentious issues concerning methodologies for choosing munitions/hit points, the value (if any) of models, and the comparability of models and tests. During the period of the Abrams live fire program, some new strategies w e r e d e ve l o p e d 2 At the foundation of those strategies has been an evolving attempt to systematize and structure the elements of S/L/E in order to clarify meanings, to establish clear causal (i.e., time-forward) event chains, and illuminate comparable model-test metrics. 3 5 Figure 1 illustrates one such effort in that process. In the middle of the figure are listed six specific levels or metric classes that are typically encountered in an endto-end weapon analy s i s Level -1 embodies the characteristics of weapon detection and identification. Level 0] represents the weapon conditions at warhead launch. Level 1 represents the characteristics of a weapon at the moment it begins to interact ( e .g., hit or detonate) at the target. L e v el 2 represents the state of the target components (i.e., killed, notkilled) after interaction with the threat. Level 3 represents the related platform capability states (e.g., ability to move, communicate, sense, e n g a g e replenish) of the platform. L e v el 4 represents the utility state of the platform (e.g., ability to service usefully a particular mission). I ssues of surviv a b i l i t y lethality and effect i v eness (S/L/E) have been around a long t i m e In fact it can be argued that Operations Research (OR) matured as a discipline during World War II. 1 In a field that has been subject to over 60 years of measurement, modeling, and analysis one might assume that associated terms, methods and phrases should be clear and precise to all practitioners of the art. But this is not necessarily the case. Surprised? Well then, contemplate the following questions W h y do all S/L/E metrics seem to fall in the range [0.0 < x < 1.0]? Most estimates in S/L/E are posed in terms of probabilities. What are the sources of r a n d o m n e s s ? W h y are single (S/L/E event) outcomes usually described as probabilities? What dimensions, if any, are associated with S/L/E metrics? Aircraft Survivability Fall 2001 18 S u rv i v a b i l i t y Lethality and Ef f e c t i v e n e s s W h a t s in a W o rd ? by Dr. Paul H. Deitz Figure 1. Levels Relevant to Survivability, Lethality, and Effectiveness. Lethality and Vulnerability ar e often used to pursue different agendas. Typically, (Weapon) Lethality (left side of diagram) is analyzed begin ning with the weapons launch at Level 0. Thus warhead hit dispersion at the target is factored in. To esti mate complete weapons Effectiveness, weapon target detection and identification must be included as well. By contrast, Vulnerability (right side of diagram) normally starts with an assumed hit at Level 1. Susceptibility covers all of the prior factors leading up to a hit. Survivability provides the overall accounting.


From the Target (or Platform) Orientation shown on the right of Figure 1, the issue of Vulnerability assumes the initiation of warhead-target interaction (Level 1) and generally proceeds through an estimate of target (or platform) utility. The key events leading up to w a r h e a d interaction with a target can be wrapped in the term S u s c e p t i b i l i t y The processes of Susceptibility and Vulnerability when combined end-to-end can be described as Surviv a b i l i t y By contrast, from the Threat-Weapon Orientation s h o wn on the left of Figure 1, Lethality includes not only issues of target engagement (Level 1) through target utility (Level 4), but the warhead delivery as well; hence the addition of Level 0. For an overall estimate of E f f e c t i ve n e s s many analyses also include weapon detection-identification; hence the addition of Level These levels describe a complex set of events as a sequence of physical states that occur just as one phy s i cal or engineering process ends and another begins. Thus the output of one process forms the input for the n e x t To complete this paradigm, we require a set of operators to connect the physical states. In Figure 2, we repeat the description of the Levels or phy s i c a l e n g i n e e r i n g states on the left-hand side. In the center of the figure, we represent each Level by an ellipse populated by v a r i ous state vectors (shown as bullets ). We use the notation O n n + 1 to denote a mapping from Level n to Lev e l n+1. The operator abstraction, O n n + 1 represents a way to describe actions covered in one of the following: P h ysics-based tests, typically of the O 1 2 o p e r a t o r Engineering-based tests, typically of the O 2 3 o p e r a t o r Operational tests, that specifically focus on the O 3 4 operator D e v elopmental tests, that examine any of the mappings between Level 1 and Level 3 Mission training, that focuses on the relationships b e t w een Level 3 and Level 4 M&S by means of which abstractions are dev e l o p e d to represent the physical, engineering, or operational activities represented by each of the O p e r a t o r s It is this last property, that a dual is formed betw e e n actual physical, engineering, operational states and test activities (operators) on the one hand, and M&S phy s i cal, engineering, operational states and simulated operators on the other, that holds the possibility for model validation. Clearly validation is possible only if the model abstractions, as described by the Lev e l s and the physical measurements, gathered via experimentation, are fully equivalent. Here are a few observations about the Lev e l s H o w e v er many levels one chooses to model, the mappings are generally complex and stochastic The metrics of Levels -1 through 3 are p h ysically based and measurable The metrics associated with Level 4, E f f e c t i ve n e s s are generally not measurable The development of Probability Density Functions (PDFs) for each metric at each L e v el is crucial The results at a given level are due to all that preceded (i.e., the factorial of parameter growth problem) There are no state vectors at any Lev e l described by a probability. How e ve r PDFs can be estimated at any Level by computing multiple state vectors and observing fractional occurrences of specific outc o m e s G i v en this level of problem decomposition, it is easy to understand the challenges LFT brought to the M&S domain. Consider a vulnerability test in which the initial conditions h a ve been established. This sets the conditions at Level 1, right-hand side in Figure 1. This is sometimes described as the end-game analy s i s From the moment when a warhead strikes a target (or a detonation occurs in proximity), the following operator sequences must occur to result in a credible Level 4 utility estimate O 1 2 Mapping: Beginning with the initial threat mechanism, e.g. fragment(s), jets, blast, all relevant deriv a t i v e mechanisms must be inferred as well (e.g. fire, smoke, shock). These threats must then be juxtaposed against all the target components with which they can interact. Based then on the character and magnitude of the threats and the susceptibility of each component, a decision must be made as to whether a kill has occurred. The difficulty of this estimate is increased when multiple damage mechanisms are present and where no single threat is capable of a kill, but significant damage aggregation can occur, none-the-less. Aircraft Survivability Fall 2001 19


complex interplay among the possible combinations of MoPs and particular MoEs. Most legacy V/L models have tended to blur the physical and engineering processes that begin at Level 1 and end at Level 4. The development of physical and engineering abstractions for each operator represents a complex challenge; if these transformations are prematurely combined, the chances for a credible VV&A process are greatly diminished. The collapsing of the operators b e t w een Levels 1 and 4 is an example of what can be called lumped-parameter modeling. By this kind of approximation, physical and engineering complexity is oversimplified and there is not adequate or accurate parameterization of the determining v a r i a b l e s Thus, key intermediate observables are not estimated and final o b s e r v ables are insufficient to validate model results. Is all of this simply notional? A c t u a l l y no! The vulnerability structure illustrated in Figure 2 was used to construct the direct-fire vulnerability model SQuASH in O 2 3 M a p p i n g : In this step, the components judged to be killed by the O1,2 Mapping process along with those properly performing are transformed to a series of platform performance metrics. Examples might include main armament rate-of-fire, number of Gs turning rate, the number of feet per minute r a t e o f c l i m b The process can be performed a number of ways; in some cases engineering models exist for this purpose. Often fault trees are used for this estimation. O 3 4 M a p p i n g : This operator takes measurable performance at Level 3 and transforms M e a s u r e s o f P erformance (MoPs) to M e a s u r e s o f E f f e c t i v eness (MoEs) at Level 4. This transformation, couched in mission perf o r m a n c e may be the most difficult to frame completely and objectiv e l y Clearly there is a Aircraft Survivability Fall 2001 20 Figure 2. A Mapping Abstraction. The ellipses in the middle column represent mathematical spaces. The bullets () contained within the spaces represent vectors. The ar r ows represent mapping operators that take a vector at one Level and perform a mapping to a vector at the next lower Level. On the left, the descriptors for the various Levels are given; on the right, the actions of the various operators. Note the single connect ed path of ar r ows from Level to Level 4 represents a single sample in an end-to-end connected process. As shown here, the initial conditions at Level could be drawn stochastically or input deterministically Since many of the succeeding operators (O 1 0 to O 3,4 ) are stochastic, the results from Level 0 through Level 4 usually represent a set of connected stochastic metrics in outcome space.


support of the Army Abrams Live-Fire program. Later the f r a m e w ork was used to generate a new vulnerability environment appropriate to direct-fire and indirect-fire ground and anti-air targets. 6 It was also used to improve greatly the context for the estimation of personnel casua l t i e s 7 including a context for operational effectiv e n e s s These ideas emerged as well in indirect-fire code to support vulnerability estimates against logistical and tactical t a r g e t s 8 1 0 And this same code formed the foundation for the V/L portion of the A d v anced Joint Effectiv e n e s s Model (AJEM). 1 1 1 2 F i n a l l y in pursuit of the important DoD mandate to perform code VV&A, the model-measure dual described a b o ve has contributed to improved methods to compare intermediate and final vulnerability metrics. 1 3 1 5 One of the most important strategies to emerge has been the practice of developing probability density functions (PDFs) for each class of model metric and then comparing particular test-derived measurements against the dist r i b u t i o n s Due to sample restrictions, single outcomes m a y serve only to reject model algorithms; how e v er as sample sizes increase, confidence in model predictions can be built. Earlier we asked, Whats in a Word? Over the y e a r s some of the words used by the community to represent various metrics and processes have not necessarily been applied in the same way. The constructs of Figures 1 and 2 were developed to provide better precision and more informed meaning. Armed with these observ a t i o n s it is possible to build more rigorous abstractions, executable c o d e s and credible VV&A Strategies. Dr. Paul H. Deitz is the Technical Director for the U.S. Army Materiel Systems Analysis Activity. In 1964 he received his B.A. in Physic from Gettysburg College, P ennsylvania. He earned his M.S. in Engineering and his Ph.D. in Electrical Engineering from University of W ashington, Seattle in 1971 and 1973, respectively. He is the Recipient of the first annual Arthur L. Stein Memorial Cup of Excellence in Live-Fire Test and Evaluation, present ed on 15 January 1997 by the Director, Operational Test & Evaluation, Office of the Secretary of Defense, . for con tributions as a national leader in the modeling and simula tion of ballistic Live-Fire phenomena." E n d n o t e s 1. Klopcic, J. Terrence and Reed, Harry L., Ed., Historical P erspectives on Vulnerability/Lethality Analysis US Arm y R esearch Laboratory, ARL-SR-90, April 1999. 2. Deitz, Paul H. and Ozolins, Aivars Computer Simulations of the Abrams Live-Fire Field Testing, Proceedings of the XXVII Annual Meeting of the Army Operations Research Symposium, 12-13 October, 1988. 3. Deitz, Paul H., A V/L Taxonomy for Analyzing Ballistic Live-Fire Events Proceedings of the 46th Annual Bomb & Warhead Technical Symposium, 13-15 May 1996. 4. Deitz, Paul H. and Starks, Michael W., The Generation, Use, and Misuse of PKs in V ulnerability/ Lethality Analyses Proceedings of the 8th Annual TARDEC Symposium, 2527 March 1997. 5. Deitz, Paul H., P arsing SMAR T : What are the Pieces and How Do They Fit Together?, Proceedings of the 1999 Fall Simulation Interoperability Workshop, s, September 1217, 1999. 6. Hanes, Phillip J., Henry, Scott L., Moss, Gary S., Murra y Karen R., and Winner, Wendy A., Modular UNIX-Based Vulnerability Estimation Suite (MUVES) Analysts Guide US Arm y Ballistic Research Laboratory Memorandum R eport BRL-MR-3954, December 1991. 7. Frew, Kellye C. and Killion, Ellen M., User's Manual for Operational Requirements-based Casualty Assessments (ORCA) Softwar e System-Alpha+ Version Applied Research Associates, Inc., July 1996. 8. Juarascio, Stephanie S., Hunt, Christopher A., Steelman, Bambi S., Hunt, Elaine M., H. F erry, Michael T., and Hunt, Jim E., Pre-Shot Predictions of the Army Tactical Missile System (A TA CMS) Block IA With the M74 Antipersonnel Antimateriel Bomblet (APAM) (U) US Army Research Laboratory ARL-MR 396, April 1998. 9. Juarascio, Stephanie S. and Keithley William E., Experimental Testing of the V ulnerability of the HIND-D to M74 Bomblet Damage Assessment (U), US Army Research Laboratory ARL-TR-2276, October 2000. 1 0.Mergler, Paige R. and Steelman Bambi S., Ammunition Vulnerability Study in Support of ATA CMS Block IA Follow-on Evaluation Effort f or Milestone III, US Army Research Laboratory ARL-TR-2 0 23, July 1999. 1 1. W asmund, Thomas L., AJEM: Advanced Joint Effectiveness Model, Aircraft Survivability JTCG/AS, Summer 2000, p. 24. 12. A dvanced Joint Effectiveness Model (AJEM), http://www 13. W ebb, David W., T ests for Consistency of V ulnerability Models US Army Ballistic R esearch Laboratory BRL-TR-3 03 0, August 1989. 14.Baker, William E., Saucier, Richard, Muehl, Theodore M., and Grote, Ricky L., Comparison of MUVES-SQuASH With Bradley Fighting Vehicle Live-Fire Results US Arm y R esearch Laboratory ARL-TR-1846, No v ember 1998. 15.Steelman, Bambi S., Hunt, Christopher A., Juarascio, Stephanie S. and Mergler, Paige R., P ostshot Analysis of the Army Tactical Missile System (A TA CMS) Block IA With the M74 Antipersonnel Antimateriel (APAM) Bomblet (U), US Army Research Laboratory ARL-TR-2004, June 1999. Aircraft Survivability Fall 2001 21


vided increased flexibility across acquisition, albeit not without some attendant collateral damage for programs that have attempted to push the reform to the limit. Y e t in the aggregate, the reform movement has had a net p o s i t i v e impact on the way we acquire and sustain weapon systems for our ultimate customersthe W a r f i g h t e r s As for the counterpoint, place yourself in the W a r f i g h t e r s seat. Has reform placed more capable (better) systems into their hands quicker (faster) and at less cost (cheaper)? To the contrary! We actually have significantly increased the time and cost of placing new systems into their inventories in addition to driving up the cost of maintaining these systems once fielded. So, from their viewpoint, acquisition reform does not appear to h a ve yielded the return on investment proclaimed by those charged with delivering. In fact, from a 5 0 0 0 0 foot view it appears to have had the opposite effect. This point was amplified by General Speedy Martin, C o m m a n d e r U.S. Air Forces in Europe, in his address at this y e a r s National Aerospace Systems T e c h n o l o g y Conference where he stated the enemy is time. Time, is used in the context of both providing capability to the field and integrating information to enable near-realtime targeting. Our partial solution to the former is to introduce the concept of spiral development or ev o l u tionary acquisition. In short, deliver incremental capability sooner. The concept is not new but yet another adaptation from the commercial sector, PCs being a model. Provide a rudimentary capability at a reasonable price with ample room for expansion and add capability through hardware and software over time, at an added cost. The Joint Strike Fighter is an example of this model applied to weapon system procurement. For $22 billion (TY$) and 7.5 years the Warfighter gets a basic stealthy air-mud capability. Another 3+ years and $3 billion (TY$), you get the system the Warfighter needs. Is this b e t t e r faster, cheaper? I submit not. So, what is the acquisition communitys mission or role? To continue our tradition of providing unparalleled technology in the form of weapon systems to our customer?a given. Focusing on the future, our pioneering (make a difference) mission is to take the lead H as Acquisition Reform been a success or a failure over the past decade? I will argue it has been both, depending upon ones perspective. From the v a n t a g e point of the acquisition community it is v i e w ed, overall, as positive. Under the banner of acquisition reform we have seen some significant efficiencies in contracting, both in the business sense of the word as well as industrial base and work force. We have made progress in moving to a more streamlined business model adopting and adapting best practices from the commercial sector. Further, we have d e v eloped long term relationships with our industry partners both in acquiring w e a p o n systems and sustaining them while mo v i n g from an oversight to insight role in conducting our business. DoD is no longer leading in the d e v elopment of key technology areas, especially electronics, but rather leveraging off of the continuous refreshment of commercial technology brought about by the boom in the rapid and seemingly endless market thirst for b e t t e r faster, cheaper. In addition, the focus or infatuation on reducing waste has driven a national Lean movement that is serving to propel industry, and to a lesser extent G o vernment towards improving efficiencies and productivity. This, in turn, is driving further consolidation or contraction of both the G o vernment and industry industrial base resulting in few e r but more capable, production and sustainment facilities. Another dimension to this reform movement is nearly a 50 percent reduction in the human capital inv o l v ed in acquisition over the past decade. The military specifications/standards (specs/stds) crusade of the early 9 0 s brought about by then Defense Secretary P e r r y proclaimed that Government specs/stds w o u l d be the exception. This government standards reduction and the myriad of other G o vernment streamlining initiatives has proAircraft Survivability Fall 2001 22 by Mr. Jon S. Ogg


in moving the acquisition community into the position of being the supplier of choice rather than default. This will entail a new look at how we provide capability while reducing cycle time and attendant costs. Our major thrust is to bring Modeling and Simulation (M&S) into the 21st century as THE enabling tool for transforming a physical model test centric acquisition process into a virtual model-simulation centric business, under the banner of Simulation Based A c q u i s i t i o n ( S B A), or more appropriately Simulation Enabled Acquisition (SEA). I believe we are on the verge of a revolution in the development process brought about through confluence of advancements in M&S fidelity and stakeholder acceptability, coupled with the rising costs and inability to test the sy s t e m s o f s ystems in a r e p r e s e n t a t i v e environment. Combine this state with the emerging revolution in air warfare with the introduction of Uninhabited Combat Air Vehicles (UCAVs), where M&S is the principal tool for deriving capability based r e q u i r e m e n t s designing, developing and verifying the system, and you have an environment ripe for change. The recent success enjoyed by the Predator in fla w l e s s l y executing the Hellfire missile integration is a shining example of the power of M&S in cutting an otherwise 1218 month effort down to three. And its only the beginning...a changing SEA state with a need for speed in delivering capability as the catalyst. What is in the future for the Air Force in general and the acquisition community in particular? My money is on an acceleration of Unmanned Air V e h i c l e s / Uninhabited Combat Aerial Vehicles (UA V s / U C A Vs) at the expense of conventional sy s t e m s more Science and Technology investment in key enabling technologies that complement UCAVs to include advancements in I n t e l l i g e n c e Surv e i l l a n c e and Reconnaissance (ISR), w e a p o n s and the next giant leap hydrocarbon fueled hypersonic propulsion, a prerequisite for making routine access to space affordable and providing for a standoff near real-time targeting capability (Missilier concept). Also, I see a return to the 60s/early 70s where we demonstrated the realm of the possible through flight of demos (X-), a Back to the Future sequel. And who is best suited to champion these mov e m e n t s ? Those who have the focus, attitude and perseverance to make a difference. Is it not the 21 s t century where dreams of y e s t e r y ear are the realities of tomorro w ? Mr. Jon S. Ogg is the Director, Engineering and T echnical Management Directorate, Aeronautical Systems Center, Air Force Materiel Command, W right-Patterson AFB. He received his B.S in aeronautical and astronautical engineering, his M.S. in aeromechanical engineering, and com pleted the coursework for a PhD in engineering mechanics from Ohio State Univer s i t y and another M.S. in business management fr o m Massachusetts Institute of Technology. He is rec ognized as the Air Force's leading authority on integrity programs for propulsion and power sys tems. He has spent 25 years of service in acquisi tion and has been involved in every phase of a system's lifecycle including serving in the capacity of chief engineer for the F-22 program for nearly a decade. He helped pioneer the current integrat ed product process development and product team approach in place today across acquisition. He may be reached at Aircraft Survivability Fall 2001 23


gy to predict target penetration and damage. FA T E P E N a n a l y sis algorithms are comprised of analytical/empirical engineering type, terminal ballistic penetration equations designed to provide very fast run times for production-type analysis runs. The finite element analysis (FEA) uses high fidelity physics-based structural a n a l y sis algorithms which account for the material densities and non-linearities as well as failure strengths and/or strains of both the MANPADS and target. An explicit time integration scheme is used to solve the equations of motion of the bodies as they make contact, interact, fail, and move. The Naval Surface Warfare Center, Dahlgren, Virginia, through its contractor Applied Research Associates of D e n ve r Colorado, is responsible for the FATEPEN mode l s / a l g o r i t h m s The FATEPEN models are based as much as possible on fundamental principles of mechanics together with assumptions regarding the primary loading and response mechanisms as determined by experiment or first principle code calculations. Under the current effort, new algorithms are being developed to adapt the existing penetration models to partial impact geometries and to add the capability for time resolv e d m u l t i b o d y penetration and damage analy s i s both of which are needed to improve body-on-body impact damage predictions. M an Portable Air Defense Sy s t e m s ( M A N P ADS) have become a prevalent threat to both military and civilian aircraft. In recent conflicts, it has been p r o ven that aircraft have survived MANP A D S e n c o u n t e r s Some MANPADS missiles also failed to detonate on or within the aircraft. The s u r v i v ability/vulnerability aircraft analy s i s community is beginning to understand the critical issues relating to the impacts of MANPADS missiles with aircraft, how e ve r the community still needs a validated set of analy s i s tools to handle this threat. In recent years a series of aggressive multi-year programs have been initiated to address these v o i d s These programs have incorporated parallel efforts that integrate first principle, high-fidelity, nonlinear structural analysis codes, test data, and analytical/empirical penetration equations to a d v ance the state-of-the-art in vulnerability a n a l y sis techniques and understanding of airc r a f t M A N P ADS encounters. This article presents a snapshot of the MANP A D S methodology development projects. The main objective of these efforts is to a d v ance aircraft vulnerability assessment and p r e d i c t i v e methodologies for missile encount e r s Specific objectives are to Apply advanced finite element/finite difference structural analysis codes to the body-on-body penetration problem and analytically predict missile v e l o c i t y missile position, penetration depth, degradation of aircraft structure, and missile kinetic energy as a function of time and D e v elop algorithms to adapt the Fast Air Target Encounter Penetration (FA T E P E N ) code for use in predicting MANPADS missile encounters with aircraft structures. Figure 1 illustrates two distinct, mutually supporting parallel code development efforts. The tri-Service A d v anced Joint Effectiv e n e s s Model (AJEM) uses the FATEPEN methodoloAircraft Survivability Fall 2001 24 M A N P ADS Analysis Methodology Development by Mr. Alex G. Kurtz and Dr. Ronald L. Hinrichsen Figure 1. Interaction of MANPADS F A TEPEN and Finite Element Analysis (FEA) Development Methodology


The 46th Test Wing, W r i g h t P atterson AFB Ohio, through its contractor, University of Illinois (UI) at Urbana-Champaign, is responsible for the body-onbody finite element analysis impact studies. They have built/obtained finite element and computer aided design (CAD) aircraft models and fabricated one FEA M A N P ADS missile. The MANPADS missile finite element model was constructed in detail and is comprised of discrete sections of an actual missile (seeker, w a r h e a d guidance and control, and motor). The MANPADS missile model also contains detailed data on section geomet r i e s exterior dimensions, joint construction, joint strength, component construction, material properties, mass properties, and rocket motor case strength. Figure 2 presents a collage of finite element predictions of M A N P ADS impacts on various targets. In each frame of the figure, the Von Mises stress contours resulting from a MANPADS impacting the target are shown. Testing is critical to credible modeling and simulations (M&S). Joint Live Fire (JLF) is not chartered to conduct validation and verification (V&V) of the analy s i s codes; how e ve r when opportunities were presented, the M A N P ADS analysis development programs have augmented JLF tests to extract very specific data. This took the form of camera placement/speed, additional strain g a u g e s additional blast gauges, and additional accelerometers specifically placed to augment recent or future analy s i s Data were used to verify both missile breakup and aircraft damage. Another way the programs are conducting incremental V&V, is to run pre-test predictions for future MANPADS tests. F o l l o wing the tests, the code developers and test engineers meet to discuss t e s t / a n a l y sis results, anomalies, and data v o i d s To ensure credible MANPADS modeling and simulation methodology dev e l o p m e n t 46th Test Wing, UI, NSWC, and ARA collaborate together to ensure that the simulations, multiple tests and analysis programs are completely integrated. Although the codes lack warhead and blast a l g o r i t h m s at project completion, the vulnerability analyst will be able to use FEA procedures and higher fidelity phy s i c s b a s e d body-on-body algorithms to perform detailed pre-test predictions, conduct post-test analy s e s and investigate potential vulnerability reduction techniques. He/she will also be able to use updated FATEPEN engineering algorithms to acquire initial MANPADS body-on-body aircraft vulnerability estimates, do vulnerability production runs, and conduct design tradeoff studies. M r Alex Kurtz received his B.S. in A e ro n a u t i c a l / As t r onautical Engineering fr o m Ohio State University. He is a research and test engineer for the 46 th T est Wing, Wright-Patterson AFB, Ohio. He has been an Aircraft Survivability Specialist for 14 year s working on various air craft vulnerability reduction programs to include: A C-130U Gunship, F-16, C-130J, C-5, C-17, F11 1, and F-22. He is currently the Chairman of the Aircraft & Crew Protection Committee for the JTCG/AS. He may be reached at 937.255.630 2 extension 250 or Dr. Ronald L. Hinrichsen received his B.S. in A eronautical Engineering from the University of Arizona and his M.S. and Ph.D. in Aeronautical Engineering from the Air F o r ce Institute of T echnology. He is a Senior Research Scientist employed by the National Center f o r S u p e r computing Applications, University of Illinois at Urbana/Champaign (NCSA/UIUC). He has over 25 years of experience in the teach ing, development, and application of composite materials, solid mechanics, structural dynamics aeroacoustics, and computer aided design. He may be reached at 937.904.5137 or Aircraft Survivability Fall 2001 25 Figure 2. Collage of Finite Element Predictions of MANPADS Impacts on Various Targets


To further explore the conclusions of the DO T & E s t u d y the Modeling and Simulation Workshop (held in February 2 0 0 1 in Reno, Nevada) identified the unique problems faced by acquisition programs using survivability and lethality M&S. The Workshop also explored potential solutions to these problems. One major conclusion recognized the need for better integration of acquisition program M&S requirements into OSD and Service policies, guidance, and initiativ e s The Workshop was constructed around presentations and panel sessions by OSD and Service decision-makers, acquisition program managers, and T&E professionals. These included DoD, Industry, academia, and the International acquisition community. The program w a s divided into three sessions: OSD View of M&S Use in Acquisition; Acquisition Program M&S Challenges and P e r s p e c t i v es; and Test and Evaluation Community M&S Challenges and P e r s p e c t i ve s Mr. James F. OBry o n (Deputy Director, Operational Test and Ev a l u a t i o n / L i v e Fire Testing, Office of the Secretary of Defense) gave the keynote A d d r e s s Briefings in the second session included representat i v es of Air, Sea, and Land and international acquisition programs using M&S to support their requirements. The third session presenters represented Service operational testing organizations, intelligence organizations, and r e p r e s e n t a t i v es of test and evaluation facilities. Each da y s session ended with a panel (comprised of the da y s presenters) discussion focused on W o r k s h o p o b j e c t i ve s The panels addressed both open mike and structured questions. On Monday afternoon, the Workshop was preceded by the JASA tutorial entitled Essentials of Simulation Credibility. The Workshop was specifically structured to dev e l o p a consensus and ensure that Workshop products properly reflected the views of the majority of the particip a n t s A summary of the previous da y s discussions, conclusions and observations was presented at start of each da y s session. The discussion of that summary each d a y was aimed at developing consensus, with significant audience participation (both during and after the daily summaries). Strong minority opinions were captured S u r v i va b i l i t y lethality and mission effect i v eness Models and Simulations (M&S) are a key element of support to acquisition program milestones. Hea v y reliance on these M&S has resulted in an ev e r increasing emphasis on the ability to demonstrate their credibility. At least one recent study has shown, how e ve r that DoD and Service M&S initiatives provide for neither long-term support of these critical M&S, nor for assessment of their credibility. The Director, Operational Test and Evaluation (DO T & E ) commissioned a study in 1999, conducted by Hicks & Associates Inc. (Modeling & Simulation Survey Briefing dated January 2000), on M&S use in acquisition. The study s h o wed that while an acquisition program manager (PM) must invest early in M&S to make the best use of those M&S, a number of factors hamper his or her ability to fully benefit from that investment. The study found t h a t The Services vary in their approaches to M&S management Industry is the predominant d e ve l o p e r / o wner of the M&S used for acquisition There is often uncertainty about the pedigree of M&S used The typical PM does not view DoD-wide M&S investments as cost -or schedule e f f e c t i v e As a result, the suitability and credibility of M&S used to support acquisition programs m a y be unknown or assumed. The DO T & E study recommended, among other actions, that OSD should direct the implementation of a process to identify and satisfy M&S requirements for acquisition programs, and review and clarify roles and functions of DoD M&S o r g a n i z a t i o n s Aircraft Survivability Fall 2001 26 M&S Credibility W o r k s h o p Assuring M&S Credibility for Defense Acquisition and T&E Sur v i v a b i l i t y Lethality and System Ef f e c t i v e n e s s 1215 Febr u a r y 2001, Reno, Nevada by Mr. David H. Hall


and noted. These daily summaries (including the third d ay s summary, which was not discussed at the Workshop) were sent to all participants via E-mail after the Workshop for review and comment. The numerous comments and change requests were summarized and incorporated, with strong minority opinions recorded in the final summary. This process guaranteed that the results and conclusions of the Workshop reflect a true consensus of opinion from the 200 plus workshop part i c i p a n t s In addition, post-Workshop comments reflect some comments from participants management that were not in attendance. C o n c l u s i o n s S u r v i va b i l i t y lethality and system effectiveness M&S are key elements of support to acquisition programs and their demonstrated credibility is essential to these prog r a m s But DoD and Service M&S initiatives do not provide for long-term assurance and maintenance of credibility of these M&S used in acquisition. A c q u i s i t i o n programs and the T&E community are by and large required to forge their own paths to credible integration of M&S. The Workshop results, independent studies, and J A S A s direct experience working for acquisition programs support these conclusions. M&S and VV&A policies and technical guidance have been made available to the acquisition community through the Service M&S Offices and DMSO. The bottom line conclusion of the three-day workshop w a s that meaningful implementation of these policies is hampered by several factors that combine to lessen their potential benefits Inconsistent M&S and VV&A definitions and requirements across Services Inconsistent standards applied to M&S softw a r e d e ve l o p m e n t S e v erely constrained program resources Policy flexibility that permits a minimalist approach to VV&A Lack of an OSD-level focal point for M&S credibility in acquisition Defense Acquisition Board (DAB) process that does not include any reviews of M&S credibility Lack of any impacts for not following policies on M&S credibility Confusion over responsibilities for M&S credibility generated by Acquisition R e f o r m In fact, the combination of factors listed a b o ve makes it quite possible for an acquisition program to comply with individual Service and DoD M&S and VV&A policies to accredit a simulation, and still not have a robustly evaluated, objectively credible simulat i o n Put another way, accredited simulations are not necessarily credible simulations. M&S and VV&A policy compliance is a necess a r y but not sufficient, condition to demonstrate the credibility of a simulation objectiv e l y and robustly. The general assumption at OSD that M&S credibility has been assured by the time the program reaches a DAB review needs to be revisited. It was clear from the Workshop briefings and panel discussions that Service M&S Offices and DMSO do not want the role of policing compliance with M&S and VV&A policy, and they do not have the resources to take on such a role. The M&S credibility buck appears to stop at the (OT) community, which articulated the greatest caution about M&S use in acquisition of the three communities represented at the Workshop (OSD, Programs and T&E). It was observed, how e ve r that it is too late in a program to uncover M&S problems or limitations in the OT phase of an acquisition program. Furthermore, the OT community has e v en more resource constraints than do the acquisition programs themselv e s Thus, the OT community cannot be a wholly effective w atchdog for M&S credibility in acquisition. F i n a l l y the Modeling And Simulation T e s t and Evaluation Reform (MASTER) concept briefed by Mr. OBryon showed promise of being able to address some of the technical issues brought out at the Workshop. How e ve r there was serious concern expressed by many Workshop participants about MASTER issues, such as how to fund the effort and how to deal with acquisition program control of M&S. R e c o m m e n d a t i o n s The briefings, panel discussions and daily summaries from the Reno M&S Workshop all seem to coalesce around the conclusion that Aircraft Survivability Fall 2001 27


On the basis of the active feedback and continuing interest in the outputs of this workshop from many of the participants, there seems to be at least prima facie evidence of the validity of these concerns at the w o r k i n g l e v el. We recommend that OSD take short-term action to establish the validity and the magnitude of the issues suggested by the Workshop. It is further recommended that OSD consider long-term options for addressing both the technical and the political problems raised by these issues. Two high-payoff follow-on actions in the near term a r e D e v elop an implementation plan for MASTER D e v elop data on how acquisition programs have accredited M&S in the past. First, since the JTCG/AS and the JTCG/ME co-sponsored the Workshop under DOT&E auspices, we recommend that these two groups spearhead the development of a concept description and implementation plan for MASTER. The goal of this effort would be to generate consensus in the acquisition community about what MASTER should look like in theory, and how it might work in practice. This would address the concern expressed by Workshop participants that MASTER looks good on paper but that the devil is in the d e t a i l s. Second, we recommend that a short, focused study of previous acquisition-related M&S Accreditation R e p o r t s and V&V efforts be conducted. The purpose of this effort would be to determine whether DoD and Service M&S policies were being implemented in a manner that actually provides objective evidence of M&S credibility for acquisition applications. It would also attempt to determine whether similar standards of M&S credibility w e r e being applied across similar programs for similar types of M&S. This study would address comments from some Workshop participants that, while there may be loopholes in DoD and Service M&S and VV&A policies, they are not being widely used to circumvent the need for demonstrated M&S credibility in acquisition decision making. This effort would also extend the data generated by the 1999 DOT&E M&S study. The results of both of these efforts would be briefed back to the OSD principals who participated in the Workshop and who sponsored the effort. The results would then be briefed to the larger acquisition community at the next M&S workshop, March 4, 2 0 0 2 in Reno, Nevada. More information is on this upcoming conference is available at h t t p : / / w w w n a w c w p n s n a v y m i l / ~ j a s a / there is no cause and effect relationship b e t w een following M&S and VV&A policy and technical guidance, and having confidence in M&S credibility for acquisition applications. It seems clear that if OSD is to reap the benefits implicit in DoD and Service M&S policies, then the VV&A carrot for acquisition Program Managers needs to be made bigger, and the OSD stick must be more effective. In practical terms, acquisition programs need access to M&S technical support services that go beyond what the Service M&S Offices and DMSO can pro v i d e Acquisition programs need experienced assistance with meaningful implementation of M&S and VV&A policies within the constraints of their particular risk and budget profiles. On the other hand, OSD needs to know that M&S and VV&A policies are actually having an objective impact on the credibility of M&S used in acquisition. Some potential steps that OSD might take to make the VV&A carrot bigger rev o l v e around providing acquisition programs with more practical help Program-specific technical assistance geared toward meaningful implementation of VV&A policy and technical guida n c e at a level consistent with application risk and program resource constraints Resources to develop and archive VV&A documentation for common use M&S Training in common, cost-effective approaches to VV&A for acquisition applications of M&S so they can do it on their o w n On the other hand, a significant step to making the M&S policy stick big enough to attract the attention of acquisition Program Managers would be to require acquisition programs to present M&S Accreditation Cases for independent review well before the D A B This would ensure that simulations that adhere to M&S and VV&A policy are, in fact, demonstrably credible for their intended use. This concept is consistent with Dr. Ganslers memo of December 2000 on Independent Expert R e v i e w s of Software Intensive Sy s t e m A c q u i s i t i o n s . It is also consistent with some international approaches to M&S evaluation in an Acquisition Reform environment. Aircraft Survivability Fall 2001 28


Aircraft Survivability Fall 2001 29 March 48 Silver Legacy Reno, Nev a d a Like Workshop 2 0 0 1, we anticipate participation from a wide spectrum of the M&S community, both Government and industry. W orkshop 2 0 0 2 will be a logical sequel to the successful 2 0 0 1 Workshop and will address the recommendations and conclusions of the follow-on actions presented in the article written by Mr. David Hall. The primary goal of the workshop is to enhance the way we do business in surviv a b i l i t y vulnerability, lethality with the employment of M&S. Reaching out across DoD for broad participation in the planning and execution of this workshop a Workshop Executive Advisory Committee (WEAC pronounced we-ack) has been created. The WEAC is currently defining the workshop specific topics and issues in support of the primary goal with the following objectiv e s D e v elop consensus on problems, issues and obstacles to credible integration of M&S into acquisition programs and T&E. D e v elop suggestions for better integration of acquisition program M&S needs into OSD and Service policies, guidance and i n i t i a t i ve s Identify potential solutions, approaches and inv e s t m e n t strategies for M&S in acquisition programs and T&E. In order to achieve the workshop objectiv e s we have planned to promote more audience participation by creating working groups to define solutions to stated issues and set aside more time for panel discussions and summaries. The WEAC currently consists of Simone Youngblood (DMSO) Mike Weisenbach (JT C G / A S ) Bob Cook (USAF) Debra Ridgeway (USA) Frank Gray (AFO T E C ) John Haug (A T E C ) Brian Hall (OPTEVFOR) Jim Sebolka (Wash. Inst.of Technologyin support of OSD) Doug Fraedrich (NRL) Hans Mair (ID A ) Tracy Sheppard (Univ. of T e x a s Austinin support of OSD) K a t h y Russell (J A S A ) Ron Ketcham, Chairman (J A S A ) An informational brochure and Call for Abstracts was released in October. The tentative agenda is as follow s M o n d a yMarch 4 Early R e g i s t r a t i o n VV&A Tutorial E v ening Mixer T u e s d ay T h u r s d a y Daily Key Note Speakers P r e s e n t a t i o n s Working Groups Panel Discussions Daily Summaries Sponsored by the JTCG/AS in cooperation with DOT&E/LFT and coordinated by the Joint A c c r e d i t a t i o n Support Activity (JASA) and the Workshop Executive A d v i s o r y Committee (WEA C ) For more information contact Ron Ketcham 7 6 0 9 3 9 2 3 6 3 K e t c h a m R L @ n a v a i r. n av y .mil or Kathy R u s s e l l 7 6 0 9 3 9 4 90 8 R u s s e l l Ka 2 @ n a v a i r. n av y .mil. Also keep checking the JASA W e b site for continual updates. h t t p : / / w w w. n a w c w p n s. n av y. m i l / ~ j a s a /


Fire Testing, the Honorable Philip Co y l e former D i r e c t o r Operational Test and Evaluation, Dr. Frank Fernandez, former Director of the Defense A d va n c e d Research Projects Agency, and providing an industry pers p e c t i v e, Mr. George Muellner, Vice President and General Manager, Phantom W o r k s The Boeing C o m p a n y. The business of addressing the issues to be addressed in the White Paper began with the keynote presentations and continued to the end of the event. Panel sessions were convened to address each of the conference topics: National Missile Defense, national security, international cooperation and the global marketplace, the role of g o vernment laboratories in national defense, commercial products in national defense, defense procurement, emerging threats, and test, evaluation and acquisition. In addition to these panel sessions, a Town Hall meeting, a trademark of NDIAs Test and Evaluation confere n c e s provided an opportunity for session chairs and the audience to discuss candidly their thoughts, concerns and proposals for correcting the deficiencies discussed during the panel sessions. Out of this discussion emerged the key points to be discussed and addressed in the conference White P a p e r F o l l o wing is a partial list of those points Attracting younger workers into the defense i n d u s t r y I m p r o ving business/finances of industry versus continuing cut-throat competition Contactor participation in operational testing for l e a r n i n g Adequacy of test funding A business model for Defense Industry investments in Research and Dev e l o p m e n t G o vernment review of Industry Research and D e ve l o p m e n t Impact, or lack thereof, of recent A c q u i s i t i o n Reform changes Testing Commercial-of-the-Shelf products to military standards. The White Paper in its entirety, once appropriately staffed through the Department leadership, will be made available to the conference attendees via the folT he annual NDIA-sponsored National Test and Evaluation Conference w a s held in Long Beach, California this y e a r With the change in A d m i n i s t r a t i o n s it was appropriate that this y e a r s event be a National Summit on U.S. Defense P o l i c y : Acquisition, Research, Test and Ev a l u a t i o n Panel sessions covered a range of topics critical to the state of defense within the Unites States and its allies. The list of speakers addressing the conference was as diverse as the topics t h e m s e l ve s The conference began with five tutorial sessions covering topics such as acquisition reform, defense policy, and the use of nonlethal weapons in the Middle East. The latter tutorial was presented by Colonel Amir Ellenbogen of the Israeli Defense Forces and was a thought provoking and shocking examination of the current state of affairs in the Middle East. Needless to say, a lively discussion followed this timely and important presentation. The tutorials addressing acquisition and defense policy were given by Gov e r n m e n t and industry representatives alike and brought to light both the advances made in acquisition reform as well as the fact that there is much y e t to do. And with that charge, the conference began in earnest the following morning with an address to the conference attendees by Congressman Stephen Horn (R C A ) Congressman Horn provided a videotaped statement during which he asked the attendees to take seriously the conference objective of p r o viding a white paper of recommendations to the Bush Administration addressing issues ranging from National Missile Defense, the state of readiness, and international cooperation and the role of the private sector. In addition to Congressman Horn, keynote presentations were given by the Conference Chairman, Mr. James OBryon, Deputy D i r e c t o r Operational Test and Ev a l u a t i o n / L i v e Aircraft Survivability Fall 2001 30 National Summit on U.S. Defense Policy Acquisition, Research, Test and Evaluation by Mr. Tracy Sheppard


l o wing Web sites: w w w d o t e o s d m i l w w w n d i a o r g and w w w c s a u t e x a s e d u In addition to the business of the conference, a highlight of the event was a presentation of the annual NDIA awards for outstanding achievement in Test and E v aluation. This y e a r s recipients included The Walter W. Hollis Award for Lifetime A c h i e ve m e n t in Defense Test and Evaluation was presented to the Honorable Philip E. Co y l e former Director, Operational Test and Evaluation. This award, first presented to Mr. Walter Hollis, Deputy Under Secretary of the Army for Operations Research, recognizes those individuals who throughout their careers, have demonstrated exceptional leadership and have made significant contributions to the Defense community. And thus ended another successful NDIA Test and Evaluation conference. Senior leaders from Defense and industry, as well as the hundreds of attendees representing every facet of the defense community, came together, spoke c a n d i d l y and made this event a complete succ e s s Congressman Horns charge to take this task seriously was met. A wealth of important and timely information came forth as a result of this conference. While only a fraction of that information is presented here, a more thorough review of the event can be obtained from a CD-ROM containing the conference p r o c e e d i n g s To receive a copy of the CD, please contact Ms. Debi Denney at debi_denn e y @ i a t u t e x a s. e d u As indicated throughout the conference, there is much yet to do. A topic critical to the changing test and evaluation and defense acquisition process, is the role of dev e l o p m e n tal testing. This will be the theme of the 2 0 0 2 NDIA National Test and Evaluation conference to be held February 25 through March 1, 2 0 0 2 in Savannah, Georgia. Please join us there. Tr acy Sheppard is the Technical Director of the W ashington Office of the Center for Strategic Analysis, University of Texas (UT) at Austin. Prior to joining the research faculty of UT, T r acy served for over 15 years within DoD, first as a Marine and then in positions at Aberdeen Proving Ground and within the office of the Deputy Director for OT&E/LFT in the Pentagon. T r acy received his AS and Bachelor of Electrical Engineering degrees from the Johns Hopkins University in Baltimor e Maryland. Aircraft Survivability Fall 2001 31 M i l i t a r y C i v i l i a n C o n t r a c t o r A r m yCapt John Eggert Glenn McPherson Charles R a m s d e l l N av y LCDR Michael DodickLuis Cortes Robert R o s a d o Air F o r c e Lt Col David Nelson James K e i t h Charles T r i s k a D o D CDR Michael Stanton William Colson D r Anil Joglekar Mr. Philip E. Coyle, III was presented the Walter W Hollis Award for Lifetime Achievement in Defense T&E.


COMMANDER NA V AL AIR SYSTEMS COMMAND (4.1.8 J) 47123 BUSE ROAD PA TUXENT RIVER, MD 20670-1547 Official Business PRSRT STD U.S. POSTAGE P AID P AX RIVER MD Permit No 22 calendar of events 58 Monter e y CA A i r craft Survivability Symposium 2001 h t t p : / / w w w. n d i a o r g 2629 Orlando, FL I n t e r s e rv i c e / I n d u s t r y Training, Simulation and Education Conference (I/ITSEC) Contact: 2729 Gaithersburg, MD National Military Sensing Symposium Closing the Information Loop with the W a rf i g h t e r Contact: IRIA, 734.994.1200 2729 Nellis AFB, NV Brawler/ESAMs User Group Meeting Contact: Paul Jeng, 937.421.2712 DEC 68 Charlottesville, VA BLUEMAX, ALARM, RADGUNS User G r oup Meeting Contact: Paul Jeng, 937.431.2712 NOV JAN Information for inclusion in the Calendar of Events may be sent to SURVIAC, Washington Satellite Office Attn: Christina McNemar 3190 Fairview Park Drive, 9 th Floor Falls Church, VA 22042 PHONE: 703.289.5464 F AX: 703.289. 5 4 6 7 1417 Reno, NV 40th AIAA Aerospace Sciences Meeting Exhibit h t t p : / / w w w. a i a a o r g FEB 2528 Savannah, GA National Test and Training Confer e n c e Contact: