<%BANNER%>

Determination of Mission Success Probabilty in Air Force Flying Missions by Bayesian Belief Networks and Operational Ris...

Permanent Link: http://ufdc.ufl.edu/UFE0043523/00001

Material Information

Title: Determination of Mission Success Probabilty in Air Force Flying Missions by Bayesian Belief Networks and Operational Risk Matrices
Physical Description: 1 online resource (66 p.)
Language: english
Creator: Cao, Emmanuel B
Publisher: University of Florida
Place of Publication: Gainesville, Fla.
Publication Date: 2011

Subjects

Subjects / Keywords: air -- bayesian -- belief -- force -- matricies -- networks -- risk
Industrial and Systems Engineering -- Dissertations, Academic -- UF
Genre: Industrial and Systems Engineering thesis, M.S.
bibliography   ( marcgt )
theses   ( marcgt )
government publication (state, provincial, terriorial, dependent)   ( marcgt )
born-digital   ( sobekcm )
Electronic Thesis or Dissertation

Notes

Abstract: Continually changing technologies, especially in the field of software, allow us to enhance the manner of business and increase efficiency while decreasing risk. This the nature of Systems Engineering and in this thesis I will use two core programs, NeticaT and Design Expert®, to identify calculable risk for flying operations in an Air Force squadron. The current standard of risk identification in United States Air Force flying squadrons is with ORM, or Operational Risk Management, and essentially comes in the form of a worksheet where users identify the risky elements on a mission and produce an overall risk rating. I will use this tool in this thesis as the primary source of data to estimate a Mission Success Probability (MSP), a potentially invaluable deliverable to military leaders. If successful, the user will have a specific gauge as to how his mission will unfold with supporting data to make significant decisions, such as continuing or halting a mission with specific risks present. In this thesis, I will discuss the use of Bayesian mathematics, the basis for the NeticaT software that I used, and its utility to military decision making specifically in the realm of aviation. Like many Systems Engineering applications, the purpose of this idea is that it improves on existing procedures and technologies to deliver more specific results. It draws upon multiple sources to create automated capabilities that were nonexistent before. Instead of only having an indicator of when ORM levels are "Low, Medium, High, or Extreme," one can now predict or have exact probabilities based on historical data. Based on this Mission Success Probability, or MSP, users can now become precisely aware of how certain risks affect their mission resulting in safer and more effective missions. Upon completion of this case study, I validated the use of the Mission Success Probability calculator in NeticaT by measuring strong correlation effects in Design Expert® between predicted MSPs and actual Overall Mission Effectiveness (OME) percentages. In addition to quantifying the magnitude actual risks that occur during a typical flying mission, I confirmed multiple advantages my system has over the current risk assessment matrix.
General Note: In the series University of Florida Digital Collections.
General Note: Includes vita.
Bibliography: Includes bibliographical references.
Source of Description: Description based on online resource; title from PDF title page.
Source of Description: This bibliographic record is available under the Creative Commons CC0 public domain dedication. The University of Florida Libraries, as creator of this bibliographic record, has waived all rights to it worldwide under copyright law, including all related and neighboring rights, to the extent allowed by law.
Statement of Responsibility: by Emmanuel B Cao.
Thesis: Thesis (M.S.)--University of Florida, 2011.
Local: Adviser: Boginski, Vladimir L.
Electronic Access: RESTRICTED TO UF STUDENTS, STAFF, FACULTY, AND ON-CAMPUS USE UNTIL 2013-12-31

Record Information

Source Institution: UFRGP
Rights Management: Applicable rights reserved.
Classification: lcc - LD1780 2011
System ID: UFE0043523:00001

Permanent Link: http://ufdc.ufl.edu/UFE0043523/00001

Material Information

Title: Determination of Mission Success Probabilty in Air Force Flying Missions by Bayesian Belief Networks and Operational Risk Matrices
Physical Description: 1 online resource (66 p.)
Language: english
Creator: Cao, Emmanuel B
Publisher: University of Florida
Place of Publication: Gainesville, Fla.
Publication Date: 2011

Subjects

Subjects / Keywords: air -- bayesian -- belief -- force -- matricies -- networks -- risk
Industrial and Systems Engineering -- Dissertations, Academic -- UF
Genre: Industrial and Systems Engineering thesis, M.S.
bibliography   ( marcgt )
theses   ( marcgt )
government publication (state, provincial, terriorial, dependent)   ( marcgt )
born-digital   ( sobekcm )
Electronic Thesis or Dissertation

Notes

Abstract: Continually changing technologies, especially in the field of software, allow us to enhance the manner of business and increase efficiency while decreasing risk. This the nature of Systems Engineering and in this thesis I will use two core programs, NeticaT and Design Expert®, to identify calculable risk for flying operations in an Air Force squadron. The current standard of risk identification in United States Air Force flying squadrons is with ORM, or Operational Risk Management, and essentially comes in the form of a worksheet where users identify the risky elements on a mission and produce an overall risk rating. I will use this tool in this thesis as the primary source of data to estimate a Mission Success Probability (MSP), a potentially invaluable deliverable to military leaders. If successful, the user will have a specific gauge as to how his mission will unfold with supporting data to make significant decisions, such as continuing or halting a mission with specific risks present. In this thesis, I will discuss the use of Bayesian mathematics, the basis for the NeticaT software that I used, and its utility to military decision making specifically in the realm of aviation. Like many Systems Engineering applications, the purpose of this idea is that it improves on existing procedures and technologies to deliver more specific results. It draws upon multiple sources to create automated capabilities that were nonexistent before. Instead of only having an indicator of when ORM levels are "Low, Medium, High, or Extreme," one can now predict or have exact probabilities based on historical data. Based on this Mission Success Probability, or MSP, users can now become precisely aware of how certain risks affect their mission resulting in safer and more effective missions. Upon completion of this case study, I validated the use of the Mission Success Probability calculator in NeticaT by measuring strong correlation effects in Design Expert® between predicted MSPs and actual Overall Mission Effectiveness (OME) percentages. In addition to quantifying the magnitude actual risks that occur during a typical flying mission, I confirmed multiple advantages my system has over the current risk assessment matrix.
General Note: In the series University of Florida Digital Collections.
General Note: Includes vita.
Bibliography: Includes bibliographical references.
Source of Description: Description based on online resource; title from PDF title page.
Source of Description: This bibliographic record is available under the Creative Commons CC0 public domain dedication. The University of Florida Libraries, as creator of this bibliographic record, has waived all rights to it worldwide under copyright law, including all related and neighboring rights, to the extent allowed by law.
Statement of Responsibility: by Emmanuel B Cao.
Thesis: Thesis (M.S.)--University of Florida, 2011.
Local: Adviser: Boginski, Vladimir L.
Electronic Access: RESTRICTED TO UF STUDENTS, STAFF, FACULTY, AND ON-CAMPUS USE UNTIL 2013-12-31

Record Information

Source Institution: UFRGP
Rights Management: Applicable rights reserved.
Classification: lcc - LD1780 2011
System ID: UFE0043523:00001


This item has the following downloads:


Full Text

PAGE 1

1 DETERMINATI ON OF MISSION SUCCESS PROBABILT Y IN AIR FORCE FLYING MISSIONS BY BAYESIAN BELIEF NETWORKS AND OPERATIONAL RISK MATRICES By EMMANUEL BRIAN CAO THESIS PRESENTED TO THE GRADUATE SCHOOL OF THE UNIVERSITY OF FLORIDA IN PARTIAL FU LFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTERS OF SCIENCE UNIVERSITY OF FLORIDA 2011

PAGE 2

2 2011 Emmanuel Brian Cao

PAGE 3

3 I would like to dedicate this thesis to Dr. Henry Pfister, who passed while I was a student at UF. Not on ly did his teachings about Bayesian Belief networks inspire the core of my thesis, but he also provided me en couragement and validation to pursue System Engineering studies

PAGE 4

4 ACKNOWLEDGEMENTS I would like to thank Dr. Vladimir Boginski for his effor ts in guiding me during my work for this thesis, as well as Judi Shivers who assisted me throughout my time at the UF REEF. I owe many thanks to my friends who guided me in my academic studies, particularly my friends in the 36 th Electronic Warfare Squadro n at Eglin AFB, without wh om I would have never began my m my father Khai, my mother Bernadette, and my sister Alice for their continual love and support.

PAGE 5

5 TABLE OF CONTENTS p age ACKNOWLEDGEMENTS ................................ ................................ ................................ ............. 4 LIST OF TABLES ................................ ................................ ................................ ........................... 7 LIST OF FIGURES ................................ ................................ ................................ ......................... 8 ABSTRACT ... 9 CHAPTER 1 OPERATI ONAL RISK MANAGEMENT (ORM) ................................ ................................ 11 1 1 ORM Background ................................ ................................ ................................ ............ 11 1 2 ORM Utility to Air Force Flying Units ................................ ................................ ............ 12 1 3 Advantages and Disadvantages of Current ORM Methods ................................ ............. 14 2 BAYESIAN BELIEF MODELS ................................ ................................ ............................ 18 2 1 A Bayesian Principles ................................ ................................ ................................ ...... 18 2 2 Bayesian Belief Networks ................................ ................................ ................................ 19 2 4 Bayesian Application to ORM Modeling ................................ ................................ ......... 21 2 5 Netica ....... .. .......................................................................... .................................... 21 3 CONVERSION OF ORM WORKSHEETS INTO MSP CALCULATOR ........................... 24 3 1 The Approach ....... 24 3 2 Design Expert ................................ ................................ ................................ ................ 25 3 3 Enumerati ng ORM Worksheets ................................ ................................ ....................... 29 3 4 ................................ ................................ ............. 32 3 5 Data Collection ................................ ................................ ................................ ................. 32 4 RESULTS ....... 35 4 1 Models and Computational Results ..... 39 4 2 Significant Factors ................................ ................................ ................................ ............ 35 5 DISCUSSION ................................ ................................ ................................ ......................... 38 5 1 Lessons Learned ................................ ................................ ................................ ............... 38 5 2 Recommendations ................................ ................................ ................................ ............ 40 5 3 Areas for Continued Research ................................ ................................ .......................... 43 6 CONCLUSION ................................ ................................ ................................ ....................... 49 APPENDIX A FIGURES ................................ ................................ ................................ ................................ 50 B TABLES ................................ ................................ ................................ ................................ 57

PAGE 6

6 LIST OF REFERE NCES ................................ ................................ ................................ ............... 6 4 BIOGRAPHICAL SKETCH ................................ ................................ ................................ ......... 66

PAGE 7

7 LIST OF TABLES Table page 5 1 ................................ .......................... 46 5 4 A ction ................................ ................................ .... 48 B 1 ORM Data with Overall Mission Effectiveness, May June 2010 ................................ ........ 57 B 2 ORM Data with Mission Success Probability and Overall Mission E ffectiveness July August 2010 ................................ ................................ ................................ ................. 59 B 3 ................................ ................................ ....................... 61 B 4 ANOVA Table for Thesis Results with Maintenance Category Added ................................ 63

PAGE 8

8 LIST OF FIGURES Figure page 1 1 ................................ ................................ 17 1 2 ORM Matrix Excerpt ................................ ................................ ................................ ......... 17 1 3 ORM Approval Thresholds ................................ ................................ ................................ 17 1 4 Mitigation Examples ................................ ................................ ................................ .......... 17 2 1 Sample Bayesian Belief Network ................................ ................................ ...................... 23 2 2 Sample Troops Belief Network ................................ ................................ ......................... 23 3 1 Design Interaction Plot ................................ ................................ ................................ ...... 34 3 2 Design Conditional Probability Table ................................ ................................ ............... 34 4 1 Predicted Mission Success Probability compared with Overall Mission Effective ness ................................ ................................ ................................ ...................... 36 4 2 Normal Probability Plot of Model Results ................................ ................................ ......... 36 4 3 List of Factors with their Magnitude Effect on MSP, no separate Mainte nance Effect .... 37 4 4 List of Factors with their Magnitude Effect on MSP, Maintenance Effect included ........ 37 5 1 Altering child node exa mple ................................ ................................ .............................. 46 5 2 ................................ ................................ ......... 47 5 3 ............... 47 A 1 9 th SOS Operational Risk Management Form ................................ ................................ ... 50 A 2 130 th RQS Home Station Risk Assessment Form ................................ .............................. 51 A 3 AFSOC Form 40 ................................ ................................ ................................ ................ 52 A 4 ................................ .............. 53 A 5 ................................ ...................... 55

PAGE 9

9 Abstract of the Thesis Presented to the Gradua te School of the University of Florida in Partial Fulfillment of the Requirements for the Degree of Master of Science DETERMINATION OF MISSION SUCCESS PROBABILTY IN AIR FORCE FLYING MISSIONS BY BAYESIAN BELIEF NETWORKS AND OPERATIONAL RISK MATRICES By Emmanuel Brian Cao December 201 1 Chair: Vladimir Boginski Major: Industrial and Systems Engineering Continually changing technologies, especially in the field of software, allow us to enhance the manner of business and increase efficiency while decreasing risk. This the nature of S ystems E ngineering an d in this thesis I will use two core programs, Netica and Design Expert to identify calculable ris k for flying operations in an Air Force squadron. The current standard of risk identification in United States Air Force flying squadrons is with ORM, or Oper ational Risk Management, and essentially comes in the form of a worksheet where users identify the risky elements o n a mission and produce a n overall risk rating I will use this tool in this thesis as the primar y source of data to estimate a Mission Success Probability (MSP) a potentially invaluable deliverable to military leaders. If successful, the user will have a specific gauge as to how his mission will unfold with supporting data to make significant decisions, such as continuing or halting a mission with specific risks present. In this thesis, I wil l discuss the use of Bayesian mathematics, the basis for the Netica software that I used, and its utility to military decision making specifically in the realm of aviation. Like many S ystems Engineering a pplications the purpose of this idea is that it improves on existing procedures and technologies to deliver more spec ific results. It draws upon multiple sources to create automated capabilities

PAGE 10

10 Medium, High, or Extreme historical data Based on this Mission Success Probability, or MSP, users can now become precisely aware of how certain risks affect their mission resulting in safer and more effective missions. Upon completion of this case study, I validated the use of t he Mission Success Probability calculator in Netica by measuring strong correlation effects in Design Expert between predicted MSPs and actual Overall Mission Effectiveness (OME) percentages. In addition to quantifying the magnitude actual risks that occ ur during a typical flying mission, I confirmed multiple advantages my system has over the current risk assessment matrix.

PAGE 11

11 CHAPTER 1 OPERATIONAL RISK MAN AGEMENT (ORM) 1 1 ORM Background O perational Risk Management is a widely used, regulatory enforced process used by all branches of the Armed Forces to specifically identify risks of a certain operation and to mitigate such risks. This is usually accomplished before an operation as the name implies, such as a combat mission or operational test. In the Air Force specifically, ORM is regulated by Air Force Instruction 90 902 and 90 9, Operational Risk Management. According to this AFI, ORM is making process to systematically evaluate possible courses of action, identify risks and benefits, and determine the best course of action for any given situation ovides the process and tools to understanding at risk behavior on both on and off duty the Air Force defines ORM as an all inclusive effort that occurs before, during, and after any operation and functions to accomplish the mission in an effective and predictable manner, while keeping safety paramount. Air Force leaders must be able to achieve consistent and standardized results from the components they command [8]. The intended effect is for the ORM users to produce ways to mitigate or control the identified risk levels. These measures can be as simple as a discussion about safety awareness before an extended weekend, or complex as a worksheet detailing all the risky aspects that may adversely affect a flying mission. Air Force ORM is d ivided into 3 levels [10]: BASIC : Everyday, simple risk management measures such as wearing a safety belt

PAGE 12

12 OPERATIONAL : Job function related risk management procedures such as cancelling a risky mission. STRATEGIC: Staff or organization level decision maki ng managing risk, such as employing a safer tactic during a military campaign. An example of this occurred during Operation Storm when Air Force strike packages changed their attack profiles to a higher altitude due to unnecessary losses on low level attac k profiles. 1 2 ORM Utility to Air Force Flying Units For the purposes of this thesis, we will mainly be dealing the Operational and Strategic level of ORM as it applies to flying units in the U. S. Air Force. All Ai r Force aviators are trained to counter operational risk with prescribed solutions which h ave proven effective. Figure 1 1 [15] is the 6 Step ORM Process by which all Air Force personnel are mandated to follow. These mitigation methods can often r educe risk and in many cases totally negate the risk. For many flying squadrons, this process is manifested in a traditional ORM worksheet which captures all the possible risks on a mission and presents a score for the purposes of awa reness. When the traditional ORM sheet is filled out, the user makes his or her inputs based on experience and garnered knowledge from fellow aviators or instructions. The user is always an experienced aviator, with at least 6 years of aviation and approxi m ately 9 10 years being the mean [4]. This equates to at least 1,000 hours of flying experience per user and a worksheet that is often completed with a high degree of consistency [1] Figure 1 2 is a snapshot of an actual ORM Matrix filled out for risks pe

PAGE 13

13 T here is much room for subjectivity in these matrices as flying experiences differ from user to user and the opinions or feelings concerning the risk of a mission may change accordingly, causing non standardized results. For example an aviator who had a near fatal experience with bird collisions in his or her flying history may consistently heighten the risk level of the BASH (Bird Aircraft Strike Hazard) as opposed to a user that has never had a bird incident and consistently gives This is an inexact science, as aircrew members subjectively quantify the level of risk for a mission based only 4 levels of rating and, although they have a higher awareness of risk factors, the final risk level d etermination is judged based on opinion and not factual evidence, leaving much to be desired. In addition a squadron may choose to set a threshold of say, before it requires the flying crew to notify or ask permission from leader ship to proceed with the mission. In terms of ORM deliverables, the process stops after the aircrew briefing but the principles are still practiced during the mission. In other words, once the aircrew determines the risk level, they discuss it as a group, weigh it against alternatives and mitigating factors, and then remain vigilant for risk related factors throughout the mission. At the very least, aircrews will be aware of the inherent risks of the mission and will be more vigilant than before. If this l the military chain of command would be notified in order to make this determination for the crew. ORM Regulatory guidance prohibits acceptance of unnecessary risk, although in wartime conditions the tolerances may differ [13]. Colonel James Stanley from Headquarters Air Combat Command provides an excellent example of flying ORM from his excerpt as published in the Combat Edge magazine:

PAGE 14

14 The ACC staff was reviewing a proposal for F 16s to pull som e of our Iceland alert commitment. Historically, this alert had been tasked to F 4... and now F 15... aircraft, both of which are two engine aircraft. Now a proposal to use F 16s was up for consideration. During review of the proposal, the following issues were identified and researched: (1) the F 16 has only one engine, (2) prevailing weather and crosswind were valid concerns, and (3) alternate airfields would be difficult to reach due to the range of the F 16. It was the classic opportunity for ORM to wor k its magic! The hazards were identified as weather, range, and single engine operations. The risk was assessed for the F 16 to perform this mission in the demanding Iceland environment. Analysis revealed that no risk control measures were available or rea listic. The decision was made not to task the F 16 [10] 1 3 Advantages and Disadvantages of Current ORM Methods The data points used in this thesis will be recovered from ORM worksheets used by the MC 130P aircrews. The MC 130P aircrews are heavily t asked and combat tested aviators, and there are two main types of ORM worksheet used within this aircrew community on the ir training missions th SOS Operational Risk 1) and the othe th RQS Home Station Risk 2). Both contain their share of advantages and disadvantages. The is divided into 5 subsections that identify major areas of risk to flying miss subsection that is affected, For example, if multiple crew members were experiencing personal issues before a fl ight, they would annotate this under section The Aircraft Commander would then assess the severity of the risky factor and determine whether the subsection should be rate igh or Extreme over all risk assessment is then made based on the summation of the subsections and like the subsections is giv rating. If the ORM level were elevated, the crew would make a determination to decide if the benef its of the mission outweigh the risks. For example, if a

PAGE 15

15 may elect to continue based on the number of training events he can get done. However, this may not be true in a combat mission, where a may be too risky based on the increased number of threats and decreased amount of mitigating factors. There are 3 main thresholds for the overall risk assessment. The first one requires Assistant Director of Operations (ADO) or Director of Opera tions (DO) approval if the mission ommander (in charge of multiple units of a single aircraft platform) and the third requires G roup C omma nder (a leader in charge of multiple units of different aircraft his is illustrated in Figure 1 3 A positive aspect of this system is that it allows user oversight in instances where simply adding the risk scores would be inaccurate. An example would be a mission where all of the factors in the but the user may have and give the if he feels the combined magnitude of the f actors has a greater effect on an inexperienced aircrew. This would address shortfalls in standardized computations that do not recognize the intangible factors. Before the final risk assessment is made, the top risks of the mission are identified as well as the mea ns to mitigate them. For example, in the worksheet in figure, Troops are rated as medium because of the lack of proficiency in the crew members; however there are instructors on board to provide supervision. This, along with other mitigating f However, this subjective system of scalar rating has a good num ber of drawbacks as well. (1) As previously stated, the overall results based on the same individual subjections and factors may vary fro m user to user and variance is undesirable in military systems which are constructed

PAGE 16

16 to have as much predictability as possible. (2) The ratings are not exact as the final ratings are measured on a scale. There is no exact science to this figure, and when there are means of delivering exact probabilities we should strive to achieve them. The other method is the additive method which is used for the th RQS Home Station The form is categorized in the same manner as with the differe nce being that each individual factor is given a numerical score with different factors given weights depending on their significance, and then collectively added to provide a final score. The final score is then used to determine a risk level and similar to the form the different score thresholds correlate to different approval levels. These are the main disadvantages of this system lies in t he additive computation of t he risk score because it disregards the conditional nature of many of the variables. In this worksheet, all of the variable s are given a standard weight on a scoring scale. For example, other words, this means at its most extreme effec t, holds twice or substantiated from any historical data but more of a subjective rating. This leaves room to be desired for a modeling system that takes int o account not only the In addition, there is a need for the standardization of factor weights, whereas in previous instances, users subjectively determined weights to assig n to different sections and decided upon the risk level on their own. The flaw here is the standard of risk may change from one user to the next and may deliver inexact results. Therefore, this standardization should be based on actual data rather than individual opinion with user oversight.

PAGE 17

17 Figure 1 Figure 1 2. ORM Matrix Excerpt Figure 1 3. ORM Approval Thresholds Figure 1 4. Mitigation Examples 1. Identify the Hazards 2. Assess the Risks 3. Analyze Risk Control Measures 4. Make Control Decisions 5. Implement Risk Controls 6. Supervise and Review

PAGE 18

18 CHAPTER 2 BAYESIAN BELIEF MODE LS 2 1 A Bayesian Pri nciples A Bayesian Modeling of the ORM worksheet addresses all of these drawbacks addressed in the previous section and retains the benefits. In order to model relationships from concurrent events with probabilities based on historical data, Bayesian Belie f Networks are the best tool to produce within these constraints. Bayesian Belief Networks were first introduced to me in Dr. network exercises and research on the Netic Systems Engineer. In addition to modeling the components of a system in a graphical format, Bayesian Belief Networks proved the following tools for a Systems Engineer [14]. They are (1) the definition of variables, (2) the definition of causal relationships, (3) the identification of significant factors, (4) the evaluation of effect on other nodes, and (5) it incorporates historical data. At the core of modeling algorithms is the conditional nature of the Bayesian calculation, which computes the probabilities of the multiple related events. The law of conditional probability, which is the basis for Bayesian software programs, is the work Rev. Thomas Bayes and more specifically his discovery in the 1700s of a basic law of probability [6]. This is expressed by where P(a) is the probability of even a happening and (a|b) is the probability of a occurring given that event b has occurred. For example, say that 50% of people in car accidents are seriously

PAGE 19

19 h urt. Also, 1 in 50,000 people get in accidents and 1 in 20 people have been seriously hurt. What is the probability that a person with a serious injury was involved in a car accident? P(Accident|Seriously Hurt Person) = P(Seriously Hurt Person| Accident) x P(Accidents) P(Seriously Hurt Person) = 0.5 x (1/50,000) (1/20) =.0002 Meaning if someone has a serious injury, it had a .02% of being from a car accident. Another way to express this equation that better fits the context of this thesis is where the hypothesis H is updated based on the additional evidence E and past experience. This is called posterior probability expressed by P(H|E,c). The term (P(H|c) is the probability of H given c alone. P(E|H ,c) is the probability of evidence assuming the hypothesis H and historical information is true. P(E|c) is the probability of evidence given c is true. This term is independent of H and can be regarded as a normalizing or scaling factor. This rule will be instrumental for the purposes of this thesis, as it will be used to compute multi variable relationships in an efficient and accurate manner which would have been a long and arduous process if done by hand [19]. 2 2 Bayesian Belief Networks A Bay esian Belief Network models causes and effects in a particular system and delivers probabilities of system success based on the probabilities of its components. At the basis of the relationships between component probabilities and system calculations is Ba yesian Mathematics

PAGE 20

20 as explained in the previous section. These types of networks are particularly useful in situations where all data is unavailable or decisions must be made with uncertainty. A Bayesian Belief Network graphically models the probabilistic relationships and presents the results in an intuitive manner [19]. We can see in the following sample diagram that each of the variables in the system are represented by a node, which internally contains the different states. In Figure 2 1 the top nod es may just be dual states, where either the sky is sunny or cloudy. Raining is dependent on the Cloudy node, the grass can be wet or dry, and the sprinkler can be on or off. The nodes are connected with the arrows pointing in the line of influence, and i n this example they are used to determine whether there will be wet grass or not. Each state inside of a node has conditional probabilities, which are determined by the parent nodes that connect to it. In this example, the model can be used to determine a case such as the lawn is wet, and if it was more likely to be caused by the rain or the sprinkler. It may be a result of the cloudiness which causes rain, or the sun, which may cause the homeowner to turn on his sprinkler. Historical information about the relationships between the nodes is used to determine whether the node is in one state or another are contained within each node in tables called Conditional Probability Tables (CPTs). These tables represent the probabilistic relationships between the state s of the connected nodes. For this example, they would be used to answer questions like: If the lawn is wet, is more likely from the rain or sprinkler? How likely is it that I will water my lawn on a cloudy day?

PAGE 21

21 2 3 Bayesian Application to ORM Modeling Th is is the core of my thesis, as the goal is to make measurements about a system based on many sub factors which have varying degrees of effect based on other individual factors. For example, lified if preflight fatigue is present, meaning that if someone is tired before a mission they will be exponentially more fatigued during the mission due to the increased number of external physical hindrances (i.e. reduction in oxygen, increased temperatu re). As we can see the current rating individualizes increase phy sical strain on night time visual flying). This current system places the responsibility on the user to draw relationships between the factors and accurately rate them. A Bayesian Model would alleviate this responsibility, as long as the model correctly draws the relationships and uses an accurate weighting system. Using the previous example in Figure 2 Unlike the scalar and additive models, the B ayesian model incorporates all of the individual factors together instead of separately, which is a more accurate reflection of their relationships in actualit y. 2 N functions in the same format as other probabilistic modeling software in that it graphically depicts the belief networks into junction trees for fast probabilistic reasoning. It functions by

PAGE 22

22 capturing the relationships among a set of joint probabilistic variables and displaying them in terms of nodes (the variables) and links. The links are causal uncertain influences that determine posterior probabilities based on prior distribu tions by using Bayesian Belief updates from the new findings. The Netica algorithm has the following characteristics. Assumes conditional probabilities as independent Prior distributions are dirichlet functions Requires a large number of cases Network starts in state of ignorance Nodes where case has a value and parental valu es begin with conditional probabilities if modified Learning is sequential and not iterative Automatic process that allows for missing values In addition, it has a feature that allows the user to define the probabilistic relationships by equations. It can factor time delays, updating disconnected links and has facilities for easy discre tization of continuous networks [14]

PAGE 23

23 Figure 2 1 Sample Bayesian Belief Network Figure 2 2 Sample Troops Belief Network

PAGE 24

24 CHAPTER 3 CONVERSION OF ORM WORKSHEETS INTO MSP CALCULATOR 3 1 The Approach As stated before, this thesis propose s to deliver a Mission Success P robability (MSP) level based on data from the ORM worksheets and calculated through Netica Combining these two tools, Netica will complement the vagueness and uncertainty the ORM worksheets presents. Netica will precisely gauge the odds of mission success based on historical data and will identify the main sources of risk. In order to determine the exact weights of all the fact ors used in the final MSP calculation, I will enlist the use of Design Expert (DE) a D esign of Experiments (DO E ) software tool used measure statistical information about a test. The entering arguments to design a model in DE are the enumerated ratings in the ORM worksheet as well as the Overall Mission Effectiveness for the same mission found in the Air Force Special Operations Command Form 40 (Figure A 3) The AF SOC Form 40 is essentially a mission summary that describes the final outcome of training mi ssions and most importantly, it provides a final Overall Mission Effectiveness or OME at the bottom. This number quantifies the portion of the original goals that was actually accomplished during the mission and must be completed at the end of the missio n by the same user that filled out the ORM worksheet at the beginning of the mission The data set to design the model will be taken from 31 MC 130P training missions from Eglin AFB, FL over the period from May 2010 to June 2010 (Table B 1 ) and the data s et to test the hypothesis will be taken from 30 flights from July 2010 to August 2010 by the same unit (Table B 2 ) This comparison will hopefully yield a successful correlation and the end product will be a validated electronic version of the ORM workshe ets, where a user would input the significant sources of risk in a mission and the final deliverable would the probability of

PAGE 25

25 mission success. Ideally, this number would be the used to justify the validity of the mission and a threshold would be set if the re would be a need for further approval. Once this final probability is calculated, I will perform an Analysis of Variance (ANOVA) using the Design Expert software on the results and will be able to decipher the most significant factors affecting this s core. In addition, the ANOVA will also give information about correlation factors and the accuracy of my hypothesis. 3 2 Design Expert Design Expert is DOE based software that designs the structure of an experiment with an exponential amount of data poi nts, and interprets the data in a variety of user friendly formats. D.E. was introduced to me the Design of Experiments class as a very valuable tool in designing lengthy experiments with multiple factors and testing for statistical significance of model f actors Design Expert Deliverables Other than producing standard statistical data about an experiment such as Sum of Squares and Degrees of Freedom, Design Expert gives an in depth analysis of process factors and mixture components, and also identifies critical factors and optimal conditions in a trial. This information is delivered in a variety of visuals, the most important one being the Ana lysis of Variance (ANOVA) T able [18] ANOVA tables display the following key statistics about the correlation fa ctor of a model: F VALUE Test for comparing term variance with residual (error) variance. If the variances are close to the same, the ratio will be close to one and it is less likely that the term has a significant effect on the response. It is calculated by term Mean Square divided by Residual Mean Square.

PAGE 26

26 P ROB > F Probability of seeing the observed F value if the null hypothesis is true (there is no factor effect). Small probability values call for rejection of the null hypothesis. The probability equa ls the proportion of the area under the curve of the F distribution that lies beyond the observed F value. The F distribution itself is determined by the degrees of freedom associated with the variances being compared. In plain English, if the Prob>F value is very small (less than 0.05) then the individual terms in the model have a significant effect on the response.) MEAN SQUARE Estimation of model variance, calculated by the term sum of squares divided by term degrees of freedom ADJUSTED R SQU ARED A measure of the amount of variation around the mean explained by the model, adjusted for the number of terms in the model. The adjusted R squared decreases as the number of terms in the model increases if those additional ter ms e model [18] PREDICTED R SQUARED A measure of the amount of variation in new data explained by the model. The predicted R squared and the adjusted R squared should be within 0.20 of each other. Otherwise there may be a problem with either the data or the model. Look for outliers, consider transformations, or consider a different order polynomial. ADEQUATE PRECISION This is a signal to noise ratio. It compares the range of the predicted values at the design points to the average prediction error. Ra tios greater than 4 indicat e adequate model discrimination [18] p = number of model parameters (including intercept (b 0) and any block coefficients) = residual MS from ANOVA table n = number of experiments [18]

PAGE 27

27 These are key statistics in determining the correlation factor of a system, and I will primarily use them to determine the significance of model factors in this thesis as well as the accuracy of our calculated MSP. Design Expert Model Types In addition to displaying all of the sta tistical information graphically in 2 and 3 D formats, D.E. also allows the user to adjust the factor settings to attain the optimal output. The following are the four types of design D.E. can model our test: FACTORIAL DESIGNS : Identifies vital factors aff ecting test; determines number of data points required for an accurate analysis but cannot analyze a prior test with limited data RESPONSE SURFACE MODEL (RSM): helps quantify the relationship between one or more measured responses and the vital input fact ors. This also allows for a historical model, meaning that it can be used for an accomplished test with a limited amount of data points MIXTURE DESIGN TECHNIQUES : Offers different configurations for optimal formulation COMBINED DESIGNS : Combines process variables, mixture components, and categorical factors in one design The model that best fits our thesis design is the RSM Historical Model, because it allows for the use of data points from previously accomplished te sts, while allowing up to 50 factors rating as 1, for all 34 factors. This is illustrated in Ta ble B 1 where the entire ORM worksheet in converted numerically on a n Excel spreadsheet.

PAGE 28

28 On the first trial run, there were errors with the entire design as the ANOVA tabl e returned a p value of meaning that no factors were statisti cally significant. In addition, the model had a Predicted R squared value of 0.0851. As stated above, this negative value implies that the overall mean was a better predictor of the response from the current model. This made sense, because in retrospect there were many factors (34) without much variability in their individual ranges (1 and 2 scores only) to make any serious statistical difference. Simply put, the high level of variability with this model caused it to be ineffective. Therefore I adjusted the model eliminating excess parent nodes and keeping only the key chil d nodes SP In addition, for reasons of statistical significance which will be addressed later in the thesis, I ad account for mission cancellation due to the aircraft status. Now with only 4 factors, the ANOVA table returned much more viable results as seen below. The p value was <.0001, meaning that the model terms were significant and in this case, they were and The F value was 1 57.01 which is strong and implies the model is significant Finally, the Adequate Precision measurement was 31.424 which exceeds the standard of 4 for a good signal to noise ratio. All of these measurements confirm that the model we used followed a normal distribution with acceptable levels of variance, enough to make determinations about the model terms. Figure 3 1 is a n interaction graph of the significant factors in the model, with 12 of the three factors as well as a 100 % mean MSP. In r egre ssion analysis seen in Figur e 3 1 below Design Expert computed the fo llowing regression equation to predict our model in coded terms and identifying the relative significance of the factors by comparing the factor coefficients.

PAGE 29

29 I used these coefficients to determine each of the subcategories (Mission, Terrain, Troops, and Maintenance) relative impact on the MSP and produced the following Conditional Probability Table (Fig ure 3 2) to calculate the MSP: 3 3 Enumerating ORM Worksheets variabl es, if any. A. Mission This portion represents the risk presented by the mission profile and these are the individual factors explained: MISSION PRIORITY. T he necessity of the mission based on the number of units supported and the command level from which t he order originated DECONFLICTION. C onnotes the number of aircraft that are a part of the same mission. This adds to the complexity if it causes a congested airspace. COMPLEXITY. T he level of risk is increased due to the addition of more mission events, including Helicopter Air Refueling (HAR), airdrops, or nighttime low level navigation. Affected by Mission Priority and Deconfliction FAMILIARITY. A ddresses the frequency of the mission events and many users are accustomed to the profile. PROFILE. H igh lights the any extraordinary events that do not normally occur on a regular mission, with higher scores given to planned, complicated maneuvers such as High Altitude Airdrops. Affected by Deconfliction and Familiarity SUPPORTED FORCES. Highlights any risk that may arise from unfamiliarity with other units participating in the same mission. ENEMY. Another source of risk to mission completion is the presence of the enemy. Enemy is the opposing force to mission completion, and seeks to increase the factor o f risk as much as possible. It is a major consideration in the MC 130P mission, and the mission profile is often shaped around enemy tactics. Because of

PAGE 30

30 INTELLI GENCE. T he strength of our intelligence and our own ability to locate the enemy and our knowledge of effective tactics to defeat the enemy PROBABILITY OF DETECTION. PROBA BILITY OF ENGAGEMENT E in areas of high vulnerability. Affected by the Probability of detection. PROBABILITY OF DEFEATING THE THREAT the threat and make it a non factor. Affecte d by the Probability of engagement B. Terrain Terrain denotes the external factors that the user generally does not have any control over. This category divides the mission temporally by the order in which the events occur. DEGRADED AIRCRAFT. T he condition of the aircraft START/TAXI/TAKEOFF Highlights any possible issues during the very start of the mission, such as taking off with a heavy aircraft in a short runway. Affected by Degraded Aircraft ENROUTE Any mechanical risks encountered on the high lev el portions of the flight, such as the inability to pressurize the aircraft. Affected by Performance OBJECTIVE. Any risks present in the area of focus for the mission. Affected by Enroute RECOVERY/DIVERT Any mechanical hazards anticipated in the termi nal portion of the mission, such as a gear malfunction. Affected by Objective WAIVERS. Risk inherent in higher level approval for non standard practices PERFORMANCE. The level of aircraft reliability. This is dependent on condition in which Maintenance troops can maintain the Aircraft (FMC Fully Mission Capable, PMC Partially Mission Capable, NMC Not Mission Capable). Affected by Waivers

PAGE 31

31 ILLUMINATION. Level of brightness. At night, this plays a role in Low Level Navigation and Terrain Following. WEAT HER Any risk based on forecasted adverse weather. Affected by Illumination BASH Bird Aircraft Strike Hazard amount of birds in the air C. Troops This is the most controllable category of all. It pertains to the factors concerning the human conditions o f the user. The aircrew is usually a mix of 8 personnel comprised of 5 positions, so there a variety of personal issues that may affect the mission. This category has a significance well. PREFLIGHT FATIGUE Rates the users physical state and rest quality INFLIGHT FATIGUE R ates the environmental conditions that may wear on the user. Affected by Preflight Fatigue WAIVERS A gain, any risk inherent to higher level a pproval for non standard procedures pertaining to physical conditions (waiving authorized rest time to launch a crew early) HUMAN FACTORS. R isk from personal distractions. Affected by Waivers TIME: T his is another manageable factor as the user controls t he timeline for the mission and the majority of the time. In the case that real world conditions call for a compressed timeline (i.e. a priority alert mission or consecutive flying schedules), this may be a factor, as limited time available results in a gr eater chance of routine flying procedures being violated. Compressed timelines, although not always causal, have been usually found to be influencing factors in major aircraft accidents, as evidenced by the SOHO Spacecraft Accident [7]. Because of this, I Execution and Preparation are parent nodes. PLANNING. T ime available in the planning process. The greater the amount of planning time, the higher probability for mission success

PAGE 32

32 PREPARATION : T he level of user preparedness prior to mission execution. Affected by Planning EXECUTION : The balance between the time available and the amount of mission 3 4 Use of Netica to build ORM Model These factors listed above were modeled in the same way they ar e presented in the ORM worksheets, with the individual factors as parent nodes for the individual category they belong to, now listed as child nodes. In some cases, these individual factors are parent nodes for other individual factors as described in Sect Finally, these Categorical child nodes become the parent nodes for the Mission Success Probability (MSP ) child node. The MSP node contains the conditional probability table from figure, which is used in calculating the MSP. The initial Netica model is Figure A 3 and the final Netica model can be seen in Figure A 4. 3 5 Data Collection The experiment bega n as previously stated by transferring the prescribed settings from the ORM worksheets onto the Netica belief network in the parent nodes. After a few trial s, it was necessary to make a standardized rule concerning the nodes in order to maximize the use o f the Bayesian Calculations All Parent Nodes should be filled out first Child nodes should be filled out only if there is a rating for that node other than Probability Table This allows for the user to see the effect of the parent node on the child node, with the option to override the belief network in areas the user sees fit.

PAGE 33

33 This addresses a drawback of the network that was addressed earlier in Section 1 3 Advantages and Disadvantages of Current ORM Methods Categorical No des and, obviously, the Mission Success Probability Node, should not be changed and the Bayesian calculation should be used. It also became obvious in the trial runs that there needed to be a change in the Netica model with respect to the weighting scale given to the maintenance factor, which includes the Maintenance error s account for 25.8 % of mission failure, and in a calculation where the maintenance factor is normalized along with the errain subse ction; the resulting mission success percentage only shows a difference of 2.2% at the most extreme levels. The most significant source of error occurred in situations where there were high and extreme levels of risk in the maintenance subsection. This has such a poor correlation factor that Design Expert determines that there are no significant terms in this model and that the overall mean is a better determinant of the score. In other words, the error caused by the missions due to maintenance is large en ough to render the Netica model statistically useless. In order to correct for this, I separated the maintenance subsection from the which returned much improved results Instead of having a statistically insignificant model, the correlation facto r now had an F Test value of 157.01 which shows that there is strong correlation and that we can use this Netica design to model risk Therefore, we will use 4 overall categories (Troops, Maintenance, Terrain, and Mission ) to determine mission success as opposed to the 5 categories presented in the earlier part of the thesis.

PAGE 34

34 Figure 3 1. Design Interaction Plot Figure 3 2. Design Conditional Probability Table

PAGE 35

35 Mission Effectiveness (Operations) = +166.45833 14.71230 Mission 29.86111 Troops 28.83929 Maintenance + 9.22619 Mission Troops 9.09722 Mission Maintenance +5.15873 Troops Maintenance Figure 3 3. Design Regression Analysis taken from Anova table C HAPTER4 RESULTS 4 1 Models and Computational Results Once these adjustments are made, the results improved to their best possible correlation factor. In Table B 1 is the list of all the individual factors and their rating for each run, along w ith the MSP and actual Overall Mission Effectiveness (OME) According to the ANOVA table (Table B 4 ) the F value of 219.18 implies that the model is significant and the p value is < .0001, meaning that there is less than a 0.01% chance the F value was th e result of noise The 34.429 This implies there is good correlation, because APRs of less than 4 imply there is too much noise or variance. In addition, the normal probability plot in Figure 4 2 show s there is a normal distribution of the MAP with no significant outliers. Overall, these results show that the model built in Netica shows successful correlation to actual mission results. 4 2 Significant Factors Here is the list of significant factors ordered by most significant to the least significant measured by the difference of n the MSP at its full effec t ) Figure 4 3 represents the magnitude order prior to

PAGE 36

36 al locating a separate section for maintenance, and Figure 4 4 represents the order after the change. Figure 4 1. Predicted Mission Su ccess Probability compared with Overall Mission Effectiveness Figure 4 2. Normal Probability Plot of Model Results

PAGE 37

37 1. Troops 22.00% 10. Qualification/ Proficiency 6.60% 19. Prepa 1.70% T 28. Illumination 0.30% 2. Mission 21.0 0% 11. Prob. Of Defeating Threat 6.30% 20. Objective 1.30% T 28. Prob. Of Detection 0.30% 3.Inflight Fatigue 19.20% 12. Terrain 5.60% T 21. BASH 1.00% T 30. Performance 0.20% 4. Human Factors 15.00% 13. Prob. Of Engagement 5.60% T 21. Execution 1.00% T 30. Start/Taxi/TO 0.20% 5. Enemy 12.10% 14. Priority 3.70% 23. Enroute 0.70% 32. Waivers (AC) 0.10% 6. Time 11.10% 15. Supporting/ Supported Forces 3.00% 2 4. Familiarity 0.60% 33. Degraded Aircraft 0.10% 7. Preflight Fatigu 11.00% 16. Weather 2.90% 25. Intelligence 50% 34. Planning 0.02% 8. Profile 9.20% 17. Recovery/ Divert 2.20% T 26. Deconfliction 0.40% 9. Waivers (Personnel) 8.00% 1.80% T 26. Type 0.40% Figur e 4 3 List of Factors with their Magnitude Effect on MSP, no separate Maintenance Effect Figur e 4 4 List of Factors with t heir Magnitude Effect on MSP, Maintenance Effect included 1. Troops 34.1% 13. Profile 7.6% 25. BASH .. 1.7% 2. Maintenance 32.9% 14. Weather 5.2% 26. Execution 1.4% 3. Start/Taxi/Takeoff 29.6% 15. Enemy 4.6% 27. Familiarity 0.8% 4. Inflight Fatigue 27.4% 16. Preparation 3.8% T 28. Deconflictio 0.6% 5. Degraded Air craft 26.2% 17. Objective 3.6% T 28. Illumination 0.6% 6. Human Factors 21.4% T 18. Enroute 3.2% T 28. Type 0.6% 7. Mission 18.9% T 18. Priority 3.2% 31. Probability of Defeating Threat 8. Time 17.4% T 18. Recovery/ Divert 3.2% 32. Intelligence 0.3% 9. Preflight Fatigue 13.7% 21. Performance 3.1% T 33. Planning 0.2% 10. Terrain 11.4% 22. Supporting/ Supported Forces 2.6% T 33. Probability of En gagement 0.2% 11. Waivers (Troops) 10.9% 23. Waivers 2.5% 35. Probability of Detection 0.3% 12. Qualification/ Proficiency 9.0% 24. Complexity 2.4%

PAGE 38

38 CHAPTER 5 DISCUSSION 5 1 Lessons Learned Maintenance The Maintenance factor requires its own category. The original proposal called for the maintenance factor to be a subset of the terrain cate gory. However, upon the initial trial, I found that the most signi aintenance node in terms of frequency and magnitude so much so that it rendered the Netica model statistically insignificant As detailed, I made adjustments and corrected the discrepancy allowing the model to have significant factors with an F value of 157.01 whereas there were no significant factors before. The e ffect of Human Fac tors From Figure 4 4 we see that the human element, has the most significant effect on our MSP with a factor of 34.1 %. There are several reasons to explain this. As stated previously, this is the most controllable factor in aviation a nd has the most variability in system effect. nature is not so easily changed [3] The ability to accept and mitigate a certain amount of risk while successfully completing a m ission lies with the user, so all of the elements affecting this ability, such as physical condition, personal distractions, or overall proficiency, should have the most bearing. In other words, category. This relationship is well documented in civilian aviation as well in terms of accident rates As we can see in Table 5 1 the human elem ent is causal for 71.9% of all US aircraft accidents in 2007.

PAGE 39

39 The effect of random events Most statistical test models are rarely ever given a 100% reliability rate. Although, ultra reliability and 6 sigma methodology can bring us close, eventually the re will always be an outlier data point that decreases this reliability rate. These random events are certainly a factor in the MSP Netica model, as none of the nodes in their natural state ever have a 100% probability rating. This presents an issue because by nature Bayesian calculations will continually decrease a if the factor pr obabilities are fe wer than 100%, which they all are We see this in our Netica less of an impact the more displaced the node is from the actual MSP node. This results in a MSP of 8 8.5 % with all nodes are set to their n atural state, and if all our controllable nodes are set to 91.0 %. Although, t his presents a disparity with missions with actual OME s of 100% which occur quite often (48.4% of data set missions) it is telling of the effect of random events not covered in the ORM worksheet. This means that random events not accounted for when the ORM worksheet is completed have an 11.5 % (100% 88.5%) average effect on all missions covered in this data set and 9.0 % (100% 91%) when all controllable factors are minimized This is the result of unforeseen factors appear ing during the course of the flight. mission, an unforeseen tasking may be issued to the crew which would normally have raised the edium to takeoff This tasking may have affected the OME and the n factors into the overall effect of all of these unforeseen random events. Users of the Netica worksheet should recognize this when looking at the MSP and understand that the MSP is limited to fewer than 100% due to the overall average effect of random events in the conditional probability calculation.

PAGE 40

40 5 2 Recommendations For these reasons, I sup port my original thesis to make use of the Bayesian Belief Network approaches in order to determine risk level of a flying mission over the traditional ORM Matrix. To recap, these are the following improvements on the standard ORM Matrix Maintenance The influence of the maintenance factor has been well documented in this thesis. The bottom line is that it affects 25.8 % of the missions in the sample group and is the direct cause of 100% of the cancelled missions (0% Overall Mission Effectiveness ), resulting in an overall 32.9 % significance factor from Figure 4 4 In this thesis, the magnitude was recognized and addressed by dedicating a separate category the Netica model being functional with a strong correlation factor, it also drastically changed the order of individual factor significance in the model, as seen in the difference between Tables 4 1 and 4 2 Figure 4 3 represent s the Netica model without a separat category and related factors are prioritized in the bottom third of all factors, but in the current usable model ( Figur e 4 4 ) they are in the top third with the the second highest rating ( 32.9 %) in table. The traditional ORM worksheet only provide aintenance Netica the aintenance factor. For example, in the traditional matrix the user may Start/Taxi/Takeoff (M ainte meaning that a cataclysmic mechanical incident, such as a gear malfunction, will probably happen in the initial

PAGE 41

41 0.1 % max imum change ( Figur e 4 3 ). This would be issues and have ultimately resulted complete mission cancellation. The Netica model with a decreasing the MSP by 29.6 % (Figure 4 4 ) in the same example, alerting the user that the probability o f mission failure is very high. I dentification of the most significant f actors In the traditional ORM matrix, the user must subjectively identify the top risks for the mission. Although this is usually done easily in less risky missions in scenarios where risk is more frequent, this is less easily quanti fiable. In these missions where there are multiple significant factors, it is important to prioritize the top risks as there is often only a limited amount of time that can be dedicated to mitigation and only the most important factors can be addressed If we follow the order of factor significance shown in table, we will have a general guideline as to the order in which should prioritize present risks. An example would be occurs quite often in the combat theater for missions that involve casualty evacuatio n. Due to the urgent natur e of the mission, aircrews will for address a certain amount of these risks, the order in which they should emphasize their miti gation techniques according to Figur e 4 4 is Threshol ds Much like the 9 th SOS Operational Risk Management worksheet, this design should contain thresholds as indicators to notify leadership when the MSP reaches a certain level. From the model analysis in ANOVA ( Table B 4 ), the mean is 75.00 with a standard deviation of 8.1.

PAGE 42

42 Using this, we determine the following threshold much in the same manner a grading scale is made. 1. > 83.1 %: No notification necessary 2. 75.0 83.1%: Assistant Director of Operations/Director of Operations Approval 3. 66.8 74.9%: Squadron Commander Approval 4. <66.8%: Group Commander Approval As a quick reference, this scale has good fidelity as 100% of t he failed missions have MSPs in the 4 th category and 8 6.7 % (1 3 /15) of the missions with a 100% OME have MSPs in the first category. User Control of the worksheet still remains the same as a traditional ORM Matrix It is an often repeated mantra of that the responsibility of the outcome of the mission falls solely upon the operator. There is a limit to the role of technology in a mission and the user ultimately accepts the risk. This is also the case with the Netica Belief Network, where the user has the f inal decision to determine the risk, and the tool functions only as a reference tool. The Netica Belief network allows users to adjust any node as they see fit, including the child nodes if they do not agree with the calculated results. For example, the u Preflight Human Factors 15.0 % effect. The user may feel that this may be too low of an effect since all of these factors compounde However, if these child nodes are changed they will be annotated as such when the child node itself turns to a purple color ( Fig ure 5 1). D oing so would

PAGE 43

43 not be gene rally advisable unless the conditions warrant, because it would negate the Bayesian calculation from the parent nodes. 5 3 Areas for Continued Research Maintenance In this thesis, I propose that a separate proba bility matrix or belief n etwork to be de dicated to the aintenance factor and completed by maintenance personnel, with operational review This includes direct input from maintenance personnel and possibly an entire belief network dedicated to the maint aintenan ce belief network would be constructed with the same framework of the final one are presented in this network. ORM worksheet as an indicator of Maintena nce Troops physical well being, which as stated before has a significant bearing on task completion rate. The goal would be for the M aintenance Bayesian Model to furnish a n overall capability rating based on the confidence levels of the subcategories, and this rating would then be used in the Mission Bayesian Model as the maintenance factor. An example Netica network is seen in Figure 5 2. This would better complete the strength of the overall mission success probability, by ensuring higher fidelity in the most significant variable A study should be conducted much like this one, into the effectiveness and ac curacy of the network. This would better explain the most significant factor is mission completion rate. Some hypotheses to test that may improve maintenance effectiveness would be

PAGE 44

44 How much does human fatigue/maintenance rest affect job completion rate? H ow much do external factors (i.e. time lines, operational pressure, leadership pressure) affect job completion rate? How much does experience affect the job completion rate of time? These are just some of the issues the Netica network would provide evidence for possible improvements in the maintenance process. M itigation Factor In Section 5 1 Lessons Learned we saw that even at the lowest levels of risk in the Netica design, th e MSP at its highest was 91.0 % which presents a 9.0 % average disparity with 15 missions in our data set with a 100% OME I believe that for the missions in which the aircrew completed the mission with a higher OME than the Netica calculated MSP, the aircrew practiced good Risk Management techniques as prescribed i n Section 1 2 ORM Utility to Air Force Flying Units and negated the amount of risk present inherent in the mission. This is a good point for research, as to the quantification of the Risk Mitigation techniques. For rating reducing the overall MSP by 27 .4% with all other factors constant ( Figure 4 4 ), the factor can be reduced down to 15.0% if there is cold water and extra crew members on board to relieve the fatigued crew. This data can be used to qu alitatively rate different mitigation techniques for corresponding risks. d ata points A basic text of the Bayesian Belief network is it produces outcome likelihoods based on incomplete data. This

PAGE 45

45 with little to no frequency. Because of this infrequency, there is a lack of data to s upport many of scenarios, and so belief probabilities must be used to best estimate th e effect. For exa H risk scenarios in any of the sample, nor have I ever seen this type of risk rating would probably constitute a sky full of birds essentially forming an impassable layer in the sky. Therefore, we have to best guess the effect of this type of setting to calculate the MSP. Perhaps research into the safety reports of aircraft mishaps may reveal more negative effects of A good example of th is type of data source is the NALL Report [3] which reports annual aircraft accident data, categorized by the source of risk leading to the accident. Figure 5 3 is a breakdown of aircrafts mishaps by different portions of a flight containing varying levels of risk. In addition, t he a flaw in of ac tual experiences with the enemy so much of this network is determined by belief. To increase fidelity in this area, more research is necessary into incidents where actual enemy contact has occu rred or scenarios with a narios can also be performed and gauged in a simulator as well. Of course, much of this information would be controlled and classified so this would be a research topic to be implemented only in a closed or wartime environment, withholding results to the public. The research would involve much historical extrapolation possibly from previous wars/conflicts where actual enemy contact has occurred. Figure 5 4 gives us a comparison of threat environment, as compared to aircraft lost in Operations Iraqi and Enduring Freedom, which are classified and

PAGE 46

46 OIF/OEF, and although there are numerous other var iables that would play into the final differing threat environments. Table 5 Major Cause All Accidents Fa tal Accidents Mechanical 219 (15.8%) 19 (7.5%) Other or Unknown 170 (12.3%) 42 (16.7%) Pilot related 996 (71.9%) 191 (75.8%) Figure 5 1. Altering child node example

PAGE 47

47 Figure 5 Figure 5 47 47 14 4 2 25 1 21 12 5 44 2 2 0 20 40 60 Mechanical Failures Fuel Management Preflight and Taxi Cruise Descent/Approach Landing Pilot incapacitation Fatal Amateur Built Accidents Total Amateur Built Accidents

PAGE 48

48 Table 5 4. Hostile Combat Action Losses to Vietnam OEF/OIF Vietnam OEF/OI F Vietnam OEF/OIF (Attack and Observation) ( Cargo and Utility ) ( Total ) Losses 757 35 1,309 35 2,066 70 Fatalities 644 33 2,421 112 3,065 145 Fatality/Loss Ratio 0.85 0.94 1.85 3.2 1.48 2.07 Flight Hours 2,927,13 0 1,310,619 9,777,753 1,705,654 12,704,883 3,026,483 Combat Loss Rate 25.86 2.67 13.39 2.05 16.26 2.31 Combat Fatality Rate 22 2.52 24.76 6.57 24.12 4.79

PAGE 49

49 CHAPTER 6 C ONCLUSION In this thesis, I have created a mission success probability calculator based on a Bayesian Belief Network that incorporates mission factors from a traditional U.S. Air Force ORM (Ope rational Risk Management) Worksheet. This worksheet coupled with their associated mission compl etion summaries provided me with enough historical data from a 2 month flying period to complete the probability tables for each factor. I began the thesis by explaining my reasoning behind the use of ORM worksheets, the Netica program, and my development of the Belief Network. After a few trial runs, I found flaws in the network and made adjustments until it produced a satisfactory result. in order to make the model statist ically significan t. I used Design Expert software to determine the weighting system for each factor in the model based on historical data from the initial data set, and also to determine the validity of the results. The results showed a mean 75.0% with 8.1 % standard devia tion and F value of 219.18 correlation factor. These results not only validated my original thesis but multiple learning points surface as well, the most important one being the significance of the human and maintenance factor. From these learning points a s well as other questions I encountered during the experimental process, I found areas for continual research that would provide insight into the nature of probability analysis and its application to flying operations. This thesis provided a contribution t owards the improvements to an operational system in the realm of military aviation and validated the results with System Engineering Tools

PAGE 50

50 APPENDIX A FIGURES Figure A 1. 9 th SOS Operational Risk Management Form

PAGE 51

51 Figure A 2. 130 th RQS Home Station Risk Assessment Form

PAGE 52

52 Figure A 3. AFSOC Form 40

PAGE 53

53 Figure A

PAGE 54

54 Figure A 4. C ontinued

PAGE 55

55 Figure A

PAGE 56

56 Figure A 5. C ontinued

PAGE 57

5 7 APPENDIX B TABLES Table B 1. ORM Data with Overall Mission Effectiveness, May June 2010

PAGE 58

58 Table B 1. C ontinued

PAGE 59

59 Table B 2. ORM Data with Mission Success Probability and Overall Mission Effectiveness, July August 2010

PAGE 60

60 Table B 2. Continued

PAGE 61

61 Table B 3 ANOVA Table for Netica Model Source Sum of Squares df Mean Square Value p value (Prob > F) Model 40852.8 6 6808.8 157.01 < 0.0001 significa nt A Mission 90.62 1 90.62 2.09 0.1612 C Troops 16.57 1 16.57 0.38 0.5424 D Mx 1676.31 1 1676.31 38.66 < 0.0001 AC 87.2 1 87.2 2.01 0.169 AD 353.98 1 353.98 8.16 0.0087 CD 153.58 1 153.58 3.54 0.072 Pure Error 1040.77 24 43.37 Cor Total 41893.6 30 Std. Dev. 6.59 R Squared 0.9752 Mean 72.58 Adj R Squared 0.9689 C.V. % 9.07 Pred R Squared N/A PRESS N/A Adeq Precisio n 31.424 Case(s) with leverage of 1.0000: Pred R Squared and PRESS statistic not defined Factor Estimate Coefficient df Error Standar d Low 95% CI High 95% CI VIF Intercept 15.97 1 15.29 15.58 47.53 A Mission 21.58 1 14.93 52.4 9.23 13.56 C Troops 9.15 1 14.81 21.41 39.71 15.22 D Mx 58.03 1 9.33 77.29 38.77 31.91 AC 20.76 1 14.64 9.46 50.97 18.88 AD 20.47 1 7.16 35.26 5.68 17.29 CD 11.61 1 6 .17 1.12 24.34 11.47 Table B 3. Continued

PAGE 62

62 Final Equation in Terms of Coded Factors: Mission Effectivenes s (Operations) = 15.97 21.58 A 9.15 C 58.03 D 20.76 A C 20.47 A D 11.61 C D Final Equation in Terms of Actual Factors: Mission Effec tivenes s (Operations) = 166.458 14.712 Mission 29.861 Troops 28.839 Mx 9.22619 Mission Troops 9.0972 Mission Mx 5.15873 Troops Mx

PAGE 63

63 Table B 4. ANOVA Table for Thesis Results wit h Maintenance Category Added Source Sum of Squares df Mean Square F Value p value (Prob>F) Model 42566.8 3 14188.9 219.18 0.0001 significant A A 115.53 1 115.53 1.78 0.1932 A2 1042.05 1 1042.05 16.1 0.0005 A3 1295.41 1 1295.41 20.01 0.0001 Res idual 1683.17 26 64.74 Lack of Fit 1483.17 22 67.42 1.35 0.4262 Pure Error 200 4 50 Cor Total 44250 29 Std. Dev. 8.05 R Squared 0.962 Mean 75 Adj R Squared 0.9576 C.V. % 10.73 Pred R Squared 0.9535 PRESS 2059.12 Adeq P recision 34.429 Coefficient Factor Estimate Standard df 95% CI Error 95% CI Low High VIF Intercept 0.39 1 5.85 11.63 12.41 A A 97.86 1 73.26 248.45 52.72 168.6 A2 749.85 1 186.9 365.67 1134.03 830.61 A3 594.57 1 132.92 867.79 321.3 6 283.7 Final Equation in Terms of Coded Factors: OME = 0.39 97.86 A 749.85 A 2 594.57 A 3 Final Equation in Terms of Actual Factors: OME = 1442.682 67.6258 A 1.0 1343 A 2 4.76E 03 A 3

PAGE 64

64 LIST OF REFERENCES [1] AIR FORCE INSTRUCTION 11 2HC 130 Aircrew [2] AIR FORCE INSTRUCTION 90 901 MANAGEMENT 1 APRIL 2000 [3] 2008 NALL REPOR T 25 Mar 2009 [4] D. B Pannell. AIR FORCE SPECIAL OPERATIONS COMMAND RATED EXPERIENCE DECLINE: EVOLUTION AND SOLUTIONS. 22 November 2006 [5] D. C. Montgomery, Design and Analysis of Experiments 7 th edition. 2009 [6] H.L. Pfister. Bayesian Belief Networks, Netic a Software. 11 Nov 2008 [7] [8] Airpower Journal Winter 1993 [9] K.D. Anthony Introduction to Causal Modeling, B ayesian Theory and Major Bayesian Modeling Tools for the Intelligence Analyst 27 Nov 2006 [10] Combat Edge May 1999 [11] M W. Maier and Eberhardt Rechtin. The Art of Systems Architectin g 2 Sept 2001 [12] Aircraft Survivability 26 July 2010 [13] Management ? [14] Netica Version 1.05. March 15, 1997 [15] ORM University. Step ORM http://www.seco.noaa.gov/Safety/ORM/ORMUCBT%201_0/fundamentals/chapter 2/chapter.html [16] R. L. Featherstone. DETERMINATION OF CRITICAL FACTORS IN UNMANNED CASUALTY EVACUATION IN THE DISTRIBUTED ENVIRONMENT. June 2009

PAGE 65

65 [17] S.S. Therrien. A BAYESIAN MODE L TO INCORPORATE HUMAN June 2002 [18] Stat Ease Software Overview with Masthead Software Overview. 10 Sep 2010 [19] Bnet.builder About Bayesian Belief Networks. 2004

PAGE 66

66 BIOGRAPHICAL SKETCH Emmanuel Bria n Cao was born in 1981 to Khai Cao and Bernadette An in Tustin, CA. He spent his early childhood in Santa Ana, CA before moving in 1988 to Oceanside, CA, a suburb north of San Diego. Raised by Vietnamese immigrants, Emmanuel learned at a young age the valu e of preparing and sacrificing for the future. Emmanuel then attended St. Augustine High School in San Diego where he played tennis, participated in many leadership c lubs, and graduated with Magna c um Laude Honors in 1999. After high school, Emmanuel chose the University of California, Berkeley as his college. Pursuing a medical school scholarship, Emmanuel joined the Air Force Reserve Officer Training Corps but, like many college students, studies. Inspired by his fellow AFROTC cadets into a love for aerospace and prompted by the Sept. 11, 2001 tragedy, Emmanuel decided to take a more proactive role in the military as an aviator. Emmanuel was chosen to be an U.S. Air Force Navigator and afte r graduating with a degree in Physical Sciences in 2005. His first operational assignment was to the 9 th Special Operations Squadron in Eglin Air Force Base, Flori da as a Navigator on the MC 130P Combat Shadow Aircraft. There, Emmanuel 140 combat missions in Operations Iraqi Freedom and Enduring Freedom and accruing over 18 00 hours on the aircraft. While at Eglin AFB, Emmanuel was inspired by his friends to continue his education, this time pursuing a Master Degree in Systems Engineering at the University of Florida. Emmanuel is currently stationed at the 130 th Rescue Squadr on, Moffett es include running, golf, trave ling, and he has one sibling, Alice.