Citation
An Analysis of risk in the aerospace industry

Material Information

Title:
An Analysis of risk in the aerospace industry
Creator:
Pinney, William Emery, 1941- ( Dissertant )
Wilmot, William V. ( Thesis advisor )
Blodgett, R. H. ( Reviewer )
Braswell, R. N. ( Reviewer )
James, J. H. ( Reviewer )
Langham, M. R. ( Reviewer )
Place of Publication:
Gainesville, Fla.
Publisher:
University of Florida
Publication Date:
Copyright Date:
1967
Language:
English
Physical Description:
ix, 117 leaves : illus. ; 28 cm.

Subjects

Subjects / Keywords:
Analytical estimating ( jstor )
Contract incentives ( jstor )
Corporations ( jstor )
Cost analysis ( jstor )
Cost estimates ( jstor )
Cost plus contracts ( jstor )
Estimation methods ( jstor )
Financial risk ( jstor )
Insurance risks ( jstor )
Point estimators ( jstor )
Aerospace industries -- United States ( lcsh )
Decision making -- Mathematical models ( lcsh )
Dissertations, Academic -- Management and Business Law -- UF ( lcsh )
Management and Business Law thesis Ph. D ( lcsh )
Risk -- United States ( lcsh )
Genre:
bibliography ( marcgt )
non-fiction ( marcgt )

Notes

Thesis:
Thesis -- University of Florida.
Bibliography:
Bibliography: leaves 109-114.
Additional Physical Form:
Also available on World Wide Web
General Note:
Manuscript copy.
General Note:
Vita.

Record Information

Source Institution:
University of Florida
Holding Location:
University of Florida
Rights Management:
Copyright [name of dissertation author]. Permission granted to the University of Florida to digitize, archive and distribute this item for non-profit research and educational purposes. Any reuse of this item in excess of fair use or other copyright exemptions requires permission of the copyright holder.
Resource Identifier:
022373583 ( AlephBibNum )
13692680 ( OCLC )
ACZ5393 ( NOTIS )

Downloads

This item has the following downloads:


Full Text













AN ANALYSIS OF RISK( IN THE AEROSPACE
INDUSTRY












By
WILLIAM EMERY PINNEY













A DISSERTATION PRESENTED TO THE GRADUATE COUNCIL OF
THE UNIVERSITY OF FLORIDA
IN PARTIAL FULFIZLLMENT OF THE REQUIREMlENTS FOR THE
DEGREE OF DOCTOR OF PHILOSOPHY










UNIVERSITY OF FLORIDA
December, 1967










PREFACE


A serious problem confronting program planners in the

aerospace industry is that of forecasting the probabilities

of success and failure of programs during the initial devel-

opment stages. Many programs have final cost and schedule

overruns amounting to more than double the initial estimates,

and in some cases these overruns necessitate complete can-

cellation of projects. A methodology for assessing the

technological difficulty of advanced programs in terms of

the probability of success within stated budgetary and

schedule constraints is needed. Such a procedure would per-

mit early evaluation in more realistic terms; encourage the

setting of more readily attainable goals for critical pro-

grams; and permit substantial savings in national resources

by singling out high-risk, low-priority projects for early
cancellation.

The primary purpose of this study is the derivation of

a procedure for the measurement of program risk. Past

attempts at such a procedure are studied and evaluated. The

inputs required and the variables of greatest usefulness are

selected and the methodology is developed. Suggestions for






improvement of the method and possible areas of application

are then treated. Finally, a sample program is used to ex-

ercise the model developed in the study and to highlight its

strengths and weaknesses.

The author wishes to express his appreciation to the

many persons who helped in the preparation of this report.

Specific thanks go to T. E. Brents, Jr., W. J. Bailey III,

and M. L. Williamson, Jr., for their assistance in the

development of the methodology and in researching the

available literature; to R. A. Gorrell and Dr. C. B. Moore

for supervision and direction; to my wife, June, for encour-

agement and understanding; to Judy Robinson Day and Pameula

Crockett for handling the typing and editing; to Dr. J. L.

Wortham fo'r proofing the original roughs; and particularly to

the supervisory committee: Dr. W. V. Wilmot, Jr., Dr. R. H.

Blodgett, Dr. W. O. Ash, Dr. R. N. Braswell, Dr. J. H. James,

and Dr. M. R. Langham.


iii





AN ANALYSIS OF RISK

IN THE AEROSPACE INDUSTRY




Chapter I The Measurement of Risk 1

Introduction 1

Types of Risk in Aerospace Contracting 3

The Need for Quantitative Risk Measurement 8

Past Attempts at Risk Measurement 10

Introduction 10
State of the Art Measurement 12
Quantitative Risk Assessment 15
Summary 20


Chapter II State of the Art Advance (SOA-A)
Measurement 24

Introduction 24

Development 26

Summary 34


Chapter III A Risk Index Methodology_ 37

Selection of Parameters 37

SOA-AT (ST) 37
Performance 37
Program Cost .39
Time 40
Other Considerations 40

Reasons for Subjective Assessment 41

Risk Index Procedure 42





Page

Chapter III A Risk Index Methodology (Continued)

Sensitivity Analysis 50

Summary and Conclusion 52


Chapter IV Refinements 56

Revision of the Present Model 56

Possible Measures 59

Point Estimate of the Probability of Success 60
Point Estimate with Standard Deviation 60
Attainment Profile 61
Financial or Economic Risk Function 62
Other Possible Measures 62

Problem Areas 63

Data Accumulation 63
Bias 64
Distribution Forms 65
Nonuniiform SOA-A Measures 66
Cost-Time Tradeoffs 66

Summary 68


Chapter V Extension to Decision Making 70

Purpose 70

Management Decisions 70

Decision M~aking .70
Game Theory and Utility 73

Attainment Profile 74

Contracting and Risk 81








Chapter VI Risk Index Application 90

introduction 90

Specifications 90

~and ( Development 91
SSOA-A Measurement 99

Sensitivity Analysis 102

Conclusion 104


References 109





LIST OF TABLES


Table Page
Number Title Number

1 Development Cost and Time Variance Factors
in Twelve Weapon Programs 14

2 Summary of Major Risk Studies 23

3 Design and Performance Specifications 91

4 Initial Cost Estimated by Cost Category 92

5 Optimistic, Modal, Pessimistic, and Esti-
mated Mean Program Cost Estimates 93

6 Optimistic, Modal, Pessimistic, and Esti-
mated Mean Program Schedule Estimates 94

7 Derivation of ai 100

8 Derivation of S' 101

9 Sensitivity Analysis for FT and ~T103


vii





LIST OF FIGURES


Figure Page
Number Title Number

1 Risk-End Item Matrix 19 '

2 Risk Curves 21

3 Theoretical Limit Curve 29

4 Hypothetical Limit for ST 32

5 State of the Art Advance Procedure 35

6 Frequency Distribution from Cost Estimates 44

7 Frequency Distribution from Time Estimates 45

8 Cumulative Frequency Distribution from Cost
or Time Distribution 47

9 Risk Index Methodology 54

10 Cumulative Probability Measures of Program
Success for Dollars and Time Using NAM 57

11 Time-Cost Tradeoff Curve 67

12 Attainment Profiles 76

13 Attainment Profile Analysis 78

14 Attainment Profile Procedure 80

15 Contract Type Vs. Program Risk 86

16 Cumulative Distribution Function Risk Vs.
RDT&rE Budget 95

17 Cumulative Distribution Function Risk Vs.
Total Program Budget 96

18 Cumulative Distribution Function Risk Vs.
Program Length 97


viii






Figure Page
Number Title Number

19 Cumulative Distribution Function Risk Vs.
Total Program Length 98

20 Relationships Among Schedule, Budget, and
Risk 105










CHAPTER I.

THE MEASUREMENT OF RISK


Introduction


It is very difficult to force the aerospace industry in

the United States into the generalized models used in eco-

nomic analysis. The demand is monopsonistic, with the Federal

Government accounting for over 90 percent of the purchases

from many aerospace firms; however, some competition does

exist among the various branches of the military and between

the Department of Defense and the National Aeronautics and

Space Administration.

The supply side presents an even more difficult analy-

tical problem. Investigation reveals that the top five com-

panies receive less than one-fourth of the total dollar

value of prime contracts; the top 25 contractors receive

approximately one-half; and the top.100 receive less than

three-fourths of the total (1). These figures are indica-

tive of the large number of firms in the industry. Entry is

restricted, however, by the extremely large capital require-

ments. In a typical program, the customer will issue a Re-

quest fo-i Proposal (RF~P) or a Request for Quotation (RFQ) to


- 1-




- 2 -


a selected group of contractors. While it is possible for a

contractor who was not asked to bid to submit an unsolicited

proposal, the normal procedure is for the bidding to be re-

stricted to those firms selected by the Department of D~e-

fenSE (DOD).*

Among these fewJ firms a strong competition begins which

wJill eventually culminate in one of them being selected as

prime system contractor. Since only one contract is usually

awarded for a given weapon system, the competition must take

place before the order is placed. Although a losing bidder

may occasionally become a subcontractor to the prime con-

tractor for a subassembly of the system, success lies in

winning the prime contract.

A complicating factor is the prevalence of non-price

competition. For example, Boeing submitted a significantly

lower final bid on the TFX (F-111) program than did General

Dynamics; but the latter wJon the contract on the basis that

Secretary of Defense M4cNamara considered their cost figures

"more realistic." This points up still another problem

area the fact that firms will deliberately underquote

costs in a proposal for the research and development program



*:A. notable exception to this rule is Macnonald Aircraft
Corporation's highly successful F-4 aircraft, whichi resulted
from an unsolicited proposal.




- 3-


and purposely take a loss in order to secure the business,

safe in the knowledge that these losses can be recouped by

overstating costs in the production program which will

follow.* Practices such as these imply that there is a

large element of risk in the aerospace industry. Perhaps

the first sort of risk which comes to mind is the risk that

the firm will not secure the contract. This is a competi-

tive risk which must be borne by every firm in the market-

place economy and will not be treated in this discussion.

The questions addressed here are the more basic ones of

whether the firm can afford to bid at all on a program;

after the decision to bid, what is the lowest realistic

figure to bid; and, after the contract has been secured,

what are the chances of successful completion of the pro-

gram. The remainder of this study will be devoted to an

investigation of these aspects of aerospace contracting.


Types of Risk in Aerospace Contracting

The most familiar treatment of the subject of risk is

probably Frank H. Knight's Risk, Uncertainty, and Profit (2).




*Similar results may be observed in some cases, even
though the contractor submits his bids in good faith,
through the operation of what has been referred to as an
"optimistic bias" on the part of estimators. This phenome-
non will~be mentioned again in Chapter IV.




r:


Knight differentiates between risk, wch~i~ch he defines as ani

event which has a known a priori probability of occurrence,

and uncertainty, which is the occurrence of totally unpre-

dictable events. If a contingency is insurable, it is risk;

if not, it is uncertainty. Knight's thesis is that since

risk: is predictable it is not a rationale for (economic)

profit but that the bearing of uncertainty is a legitimate

argalment for profits. In the aerospace industry, the

government assumes all. or a portion of this uncertainty w~hen

it negotiates a cost-plus- fixed- fee or a cost-plus- ince~ntive-

fee contract.

There are other types of "risk" involved in the opera-

tion of anl aerospace firm. The stockholder feels that he

assumes a type of risk by purchasing shares of the company's

stock. The firm assumes a type of riske when it invests the

stockholders' funds in fixed plant and equipment .0r1 in'basice

research or in development of a product for which the poten-

tial customer has not: yet issued a firm contract. Riskt is

assumed by both parties when a contract is signed: normally

a product of acceptable quality must be produced by a dead-

line and within prescribed cost: limits. If the contractor

does not deliver, he runs the risk of financial loss and

loss of good name wJith the customer; the customer runs the

risk of being unable to meet the military requirements of




- 5 -


the nation adequately if the contractor is late or the

quality is substandard.

Another concept, attainment risk, is the uncertainty of

obtain _ing an acceptable production result in terms of pre-

state 1 specifications." It is possible to break attainment

risk into several component risks. Schedule risk, for exam-

ple, :s the uncertainty of completing a project before a

prescribed deadline; cost risk is the uncertainty of com-

pletinZ a project: within predetermined budgetary restraints;

performance risk, or quality risk, reflects the uncertainty

of meeting the physical performance and reliability specifi-

cations set: forth in the contract. The recent trend towiard

incentive contracts represents an attempt by the government

to reduce attainment risk by inducing contractors to meet or

surpass contractual goals. Hagen (4) concludes that fixed-

fee contracts are never Pareto-optimal and can, in a wide

range of cases, be replaced by incentive-fee contracts,

which will increase the utility of both parties.

There are other significant forms of risk that can be

generally classed as prestige risk~s. The success or failure



-:General El~ectric's Risk Appraisal of Progrrams System
(~RPS) defines risk in this way: "Risk ... can be defined
as the probability that the wlork being done will miss the
triple target of cost, delivery schedule, or technical per-
formance." (3)








of United States space projects is a determinant of public

opinion in many parts of the worTld. Since some nations rely

heavily on the purchase of American military machinery for

their defensive capabilities, the success or failure of

AEmerican programs can have international implications.

Thus, the failure or cancellation of a major project could

materially affect United States prestige abroad.

Prestige riski can also be an important factor within

the industry since a firm~'s reputation can be a significant

asset or liability. Sales to the public are dependent in

part on company prestige. Even if the company sells exclu-

sively to the government, prestige or reputation may be the

deciding factor in contract aw~ards. The "Department of De-

fense Evaluation of the Performance of Major Contractors"

(5) is a publication which quantifies company prestige in

the eyes of the major customer of the aerospace industry.

This rating can be an important determinant of a company's

position in the industry. The reputation of a company can

also affect hiring policies and ot~her- non-sales areas. Pres-

tige riskc can therefore be a significant determinant in pro-

gram evaluation.

These three major types of riske are highly interrelated.,

The failure of a contractor to produce a product of accept-

able quality can involve economic and prestige losses to the








company. The customer can also incur economic and prestige

losses by being forced to pay more than expected for the

product or by being forced to release an inferior system to

allied countries. The allocation of riski between the twio

parties wi;ll be determined primarily by the type of con-

tract. Thee Federal Government h~as assumed much of the risk

in research and development programs. Recent years have

seen a trend toward the incentive type contract for programs

which will produce a physical piece of hardware, which in~di-

cates an implicit recognition of the risks associated with

aerospace programs and a desire to share the risks between

the contractor and the customer. Hagen shows that the in-

centive contract can result in greater utility for both con-

tractor and customer through more equitable risk distribu-

tion-.

This brief discussion suggests that the ability to

measure risk would increase the ability of the program plan-

ner to choose the specific contractual terms which would

result in the best possible allocation of the risk asso-

ciated with a program. In the sections which follow~,

specific benefits of a risk measurement will be suggested

and some of the attempts which others have made at risk

measurement will be reviewed.




- 8 -


The Need for Quantitative Risk Measurement


Aft-.r considerable study and discussion whith experi-

enced aerospace planners, it was decided that some type of

attainment risk measure would be the best possible device

for evaluating the various types of risk which are associ-

atLed withh aerospace programs. It could explicitly consider

financial attainment, quality or performance attainment, and

time or schedule attainment; it implicitly considers pros-

tige risk by assuming that it is a function of those types

of risk( which are considered explicitly.

A. risk measure which could assign a more specific risk

classijf-ication to a project could be used to reevaluate pro-

gram cost and performance in more than general terms. A

major portion of the $4-90 million loss sustained by General

Dynamics on its 880-990 commercial jet program could have

been saved if adequate assessment of the budget, schedule,

and performance risks of the program had been made at the

outset, or at any of several points during the program. For

an excellent discussion of this program and the reasons for

the losses, see "Howi a Gr~~eat Corporatijon Got Out of Control"

(6) and "GD: The Hard Road Back~ from the B~rink~" (7).

Another case in point is the Skcyboltl missile program.

When thlis program ws~n cancelled in 19)63, th~e program costs




- 9 -


had risen from 1960 estimates of $;893 million to almost $3

billion, anld the program was far behind schedule (8). Over

$150 million was considered to be an unsalvageable loss (9).

If Skybolt had been reevaluated by an accurate risk measure

while still under feasibility analysis, the project could

have been altered considerably. The congressional contro-

versy could have occurred at the time of the initial appro-

priation and not after millions of dollars had been wasted.

The Skiybolt program could have been cancelled, post~-

poned, or accepted under revised cost and schedule esti-

mates; if the project were accepted, funding levels would

have been more realistic. If the cancellation had come

early in the development, a considerable amount of time and

money would have been saved; and Great Britain, whlich had

planned to purchase the missile system, could have redefined

its military program at a more convenient time. This is

just one example of a situation where failure to adjust cost

and schedule estimates with full consideration of the risk~

involved led eventually to cancellation of the program.

Both government and industry have wasted valuable re-

sources when projects were cancelled or extensively rede-

fined late in the program. Consequently, both government

and industry would benefit from a risk measurement which

could quantify program risks during the early phases of




-10


development. Recent years have seen real advances in the

accuracy of estimating techniques. Unfortunately, there has

been an even greater increase in the requirement for still

more sophisticated techniques and better estimating proce-

dures. Better estimating methods incorporating risk adjust-

ing procedures are becoming even more important in modern

contracting and planning in the aerospace industry.


Past Attemplts at Risk MIeasurement


Introduction

Since the concept of risk and its role in decision

making in the aerospace industry has been treated before,

this section will be a survey of previous studies of risk in

aerospace programs. An extensive search of the current

l~iter~ature has disclosed very little documentation of con-

cepts or methodologies for the generation of a risk~ index.

Several studies, however, have considered some form of risk

or uncertainty. Among these are A~chian (10); Miarshall and

Meckling (11); Peck and Scherer (12); Polski, Clausen, and

Paige (13); Summers (14); and Terrell (1,5). The purpose of

this section will be to relate pertinent conclusions and

simnil~arities among the above-mecntioned studies to appropri-

ate areas investigated in this report.




7"
LI


.In 1950, A. A.. A~ch~ian of RANDG Corporation published a

study of the var-iabillity of actual costs froml cost esti.-

mates. The reliability of estimates from four sources was

studied. The four types of estimates with the average error

for the trade journal. data studied were as follows:

(1) Engineer-ing Estimates 2:25%

(2) Cost Estimat~or's Estimates +23%/

(3) 'Public Engineers' Construction Cost Estimators 1s6%

(4) Contractors' Bid Estimates +~21%

Alchian concluded that the difference between the estimate

and the realized true cost for any device of an improved

quality would usually be larger than any of the above devii-

ations. The study dealt only with the unreliability of costi

estimates, and no technique was presented to obtain better

cost estimates.

The most extensive analysis of risk in weapons acquisi-

tion wzas made by Peck, and Scherer. They observed that gen-

eralcly weapons program decisions involve fours basic elements:

the potential enemy threat, cost, the state of the art, and

time. Uncertainties in estimating thie last three items -

the technical feasibility, the development time, and the

development and production costs are the primary con-

tributing factors of program risk.

























































*Refernce 12, page 303.


- 12


Although many factors were conIsideredd qualitatively, no

analytical solutions were derived for estimating any aspects

of program success. Peck~ and Schierer recognize the imm~ense

complex:ity of qruant~itative risk analysis by saying:

The existence of these significant uncer-
tainties plainly complicates the use of quanti-
tative analysis in program decisions. Indeed,
the complications from uncertaintyT are so great
that our entire approach of applying an optimi-
zation model to the program decision can be
questioned.*


State of the Art Measurement

A single facet of this complexity is reflected in the

technological advance inherent in a given program. For a

comparative analysis of programs to be effective, it is

essential that the technological complexities be compared,

The methodology employed by Peck and Scherer for evalu-

ating technological difficulty was simply a within-sample

ranking of paired programs (nine programs being used) by

expert opinion. The criterion used was the selection of

that weapon system from each pair which represented the more

ambitious state of the art advance (SOA-A), considering the

time period during which it: was developed. The ranking was

then arranged on a scale from zero to one hundred units to







compare the relative degrees of difficulty or technological.

advance. This methodology (the ranking concept') has limited

applicability when one tries to measure the technological

advance of a newi program which is not~ functionally compar-

able to any member of the sample. The main problem is de-

termining a realistic technique for quantitative measurement

of technological advance.

One measure of program success often used is the rela-

tive size of overruns* in budgets and schedules encountered

at completion. In the analysis of 12 programs, Peck and

Scherer determined an average cost overrun of 3.23 and a

corresponding development time overrun factor of 1,36.

Table 1 shows the individual results for the 12 programs.

In their study of 22 Air Force weapon system. develop-

ments, Marshall and Mleckl~ing noticed a similar trend in thle

production cost overruns. This trend wias still evident evren

after the latest cost estimates were deflated by a price

index and adjusted for output quantity changes. Cargo and

tanker aircraft had the smallest average, 1.2; fighters had



"Overrun is the ratio of final actual to initial esti-
mate.

~This overrun is considerably larger than the one found
by Alch~ian. The probable reason is that the Peck and
Schlerer data contained only large programs, while some of
Alchian's data were from smaller, less ambitious projects.








Table 1

DE'VELOPMENT COST AND TIME?'r VARIANCE FACTORS
IN 12 W~jEAPONS PR\OGRAM,~S'



Program Development Development
Cost Cost Factor~ Time Factorb


A 4.0 1.0
B 3.5 2.3
C 5.0 1.9
D 2.0 n.a.
E: n.a. 0.7
F 7.0 1.8
G 3.0 1.3
H 2.0 1..0
I_ 2.4 1.3
J 2.5 1.3
Kt 0.7 1.0
L, 3.0 1.4

Average 3.2 1.36


I.r 1 -


~from Peck and Scherer (12), page 429.

Development factors are ratios of final actuals
to initial estimates.


an overrun of 1.7; bombers, 3.0; and missiles, 5.2. The 22

programs were then grouped by experts according to degree of

technological advance. The average production cost overruns

for various levels of advance were as follows:

low~ advance 1.35
medium advance 1.75
high advance 4.30

This trend of increasing. cost overrun with increased

technical advance supports similar findings by Pecke and




- 15 -


Scherer. For the above programs, Marshall and M~eckling de-

termined average cost and schedule overruns of 3.01 and 1,28

re sp ec tively. I n the development of mi-~litary weapons, there

is a tendency to place more emphasis on meeting the perform-

ance capability even if this means sacrificing cost and time

schedules. It is noteworthy that in most cases the cost

overrun has the larger ratio.

Marshall and M4eckling do not present a methodology for

predicting the final outcome of a program. However, the

above results indicate that the probability of program over-

runs increases directly though not necessarily proportion-

ally with the amount of technological advance reqluiredl by

the program.

Peck and Scheirer conclude that the unpredictability of

a weapon system program manifests itself primarily in the

cost dimension due to the premium placed on performance and

the tradeoffs among performance, time, and resources.


O~u ntitative Risk- Assessment

It becomes readily apparent that there exist many

forces which affect the outcome of a program, some of which

interact in a complex- array of tradeoffs, as in th~e case of

scheduling and funding. There are also the parameters which

are ph~ysi~cally unmeasurable but still qualitatively compared

by expert opinion in the appropriate fields.




- 16 -


The first well-known analytical study involving uncer-

tainty was performed by Summers. This was a statistical

study to adjust original cost estimates by an analytic func-

tion such that the revised value would be closer to the actual

outcome. Based on 22 weapon systems and 68 cost estimates,

Summers' study found these three variables to be most impor-

tant:

1. time the estimate is made in relation to the
development program

2. degree of technological advance required

3. length of the development period.

The result of Summers' analysis is an exponential function,

In F =aO-alt + a2At + a3A + aqA2 + a5L+ a6T + u,*

which can also be expressed in the equivalent form,

F = Kealt ea2tA eagA eaqA2 ea5L ea6T.~

where the following parameters are defined:

K,a = constants,

t = a fraction representing the time elapsed in
the program when the estimate was made,

A = a numerical measure representing the techno-
logical advance in the program,

L = the length of the development period,

T = the calendar year, and

u,v = error terms.


#Reference 14, page 33.




- 17


Once again the problem of measuring the inherent tech-

nological advance of the program arises. The technique used

in evaluating A was a wi~jjth~in-sample ranking determined by

RANTD engineers. For reasons given previously, difficulties

occur in consistently applying this measurement scheme to

new programs.

In estimating techniques that utilize data of past pro-

grams, Summers concludes

...it must be emphasized that the crudeness
of the data precluded the possibility of obtain-
ing anything like a precise description of cost
estimating errors in the years following World
War3L IIC... WhJ'at is claimed is that the methods
employed in this study constitute a reasonable
w7ay of looking hard and close at the information
available. The merit of the formulation offered
here lies in its explicit method of derivation
and in the possibility it offers for future re-
finement with mo-re complete data and with experi-
ence of newJ kinds of systems and new product-ion
techniques ."

To recapitulate, the results of Summers' study are- simni-

lar to those of other investigations reported in this sec-

tion. These results may be sumrmarized as follows:

The use of a subjective ranking by experienced
engineers to evaluate the state of the art ad-
vance of the program.

The conclusion that the average cost overrun in-
creases with technological difficulty and de-
creases w~ith respect to how close the program is
to completion wh~en the estimate is made.


'Reference 14, page 11.

























































~Refernce 13, page 1.


- 8


In 1964t, aIt the AIAA~- Annual Meeting, ~Polski, Clausen,

and Paigo presented a clear and w~ell-defined methodology for

reducing risk on research and development (R&D) programs.

The purpose of such a meithodology is as followtis:

Such a system has benefits for both customers
and contractor-s because an increase in knowledge
of the risks leads to more sound decisions con-
cerning them and general improvements in program
management plans, methods, and communications.
Greater knowledge of risks may also lead to ultij_-
mate improvements in such areas as:
Source Selection
Program Definition
R&D Program Value

Four ranges of risk (1) high, (2) moderate, (3) minor,

and (4) l~ow a-re used in the analysis. A risk-end item

matrix is generated along with suggestions and costs for

lowering the risk in the defined areas. (See Frigure 1.)

This is the first well-defined methodology for utilizing the

knowledge of a wi7de variety of people involved in a program.

This implies that a risk-cost tradeoff matrix could be found.

Te~rrll attempted to unify the areas that contributed

most to uncertainty wijth a more quantitative assessment of

risk for programs in the early phases of definition. A

methodology is presented which quantitatively compares the

technological advance of different programs. This






















C Iaillfo a T 2

Mormllmrry7vu 2 2




2dinr IT Itinw u




fa ,werivu~lu~or/lwe III -l 1r


74;?w~9~n~rrarrrwrr( -J I r I -I

C~sno /cw ..s -saI~IT -1 -a -.s





O // _1/J//f.Ioe 2






I~inalar!!M)& _r1D I -r =




#117IX/#V/C .3W I -1 1~


Fl I I /A G~w ntriful = z I

k "NJ"rYrN Skius 5 I J 1
Q) ~ N~urnIrd ZAX/? I T I -






3/JA7I~YNY~lN -ru rac-r~

IJrf lrNY Jarade _

sayx~wr Dinvadn a ncisr T
JIMi)YVR//V}#JNa ////7;I

startlnr si r


I
C'
O
U

hi
U
rle


3












r


~L a, ~t
vlkO
~ ~r
(nF~~L~C~
cn 3 a
F~ Esl a
3 F~ vl ~
t 'r( m
bDT~ 3 Q)U
rl a) OkO
=dC~~Z


t:
~I

vl
E
Q)

tn
h
tn


- 19-




- 20 -


measurement begins as the ratio of the proposed development

item to the best available (developed) item and is then ad-

justed by use of a theoretical limit assessment curve. This

curve takes into consideration the relationship of the de-

sired performance capability to the theoretically possible

performance levels. A procedure is developed for applying

standard statistical techniques to the performance character-

istics for more than ten programs in order to generate a

family of attainment curves. These curves predict the prob-

ability of attaining performance, reliability, and schedule

with corresponding tradeoffs involving cost. (See Figure 2.)

The projects included in Terrell's sample showed an

average cost overrun of 2.5 and a schedule overrun of 1.2.

These values fall into the general ranges of values deter-

mined by previous researchers.

In conclusion, with the attainment prediction equations,

a specific technique for evaluating the degree of difficulty

in a program, and considerations involving the theoretical

limit, Terrell's study is the most advanced attempt at the

evaluation of a risk index.


Summary

This chapter indicates something of the variety of con-

cepts which have been analyzed under the name "risk." The




- 21 -


gDb





b0


,Ys/ ,ume/7;&~t~ 9


~S/2/~7;7na~Y3S S~




22 -


need for a quantitative procedure for the measure~en~t of

attainment risk~- is established, and a survey of the w~ork to

date is presented.

A summ~ary of the contributions of the principal studies

in thEr area of risk~ is given in Tabl~e 2., The evolutionary

chara-:ter of the development and the sources of some of the

conce ~ts presented in this study are indicated, Included

are th-e primary areas of interest, functional relationships

developed, and a brief summary of the important conclusions

of eachi study.






















'C;


d




Z

(G
i~U
UY.
Y1
OC
UO



PiO
rO '*~

3UO
m ~i ~i
r,'CIP
olol~
E~vl
nvl
>. o
cco~
1U
vl
UU
rtU1
U~UO
O~U

10
u
~olu~
sr~ca
~ale~~
v, O
U Y1 Zv,
u,~~o
oo R
UUVI
UCVI
rr~lE

ZIU~U
UCa)d
uo>
~UO


-
------


r(
n
u
rr
u
v
d
LI
O
U
m

x
~oc,
Fi
~iir,


~ i rv
OU~~C~
(OcC)nl,
~o~ii L,
r~ 1J vl ~r
~ ~ ar c vl
CPCrTi
XO r( 01
UF=U(LI
~rrm~
~ u t~-~
r~C 7
UOIYIY~
~ u o. ~O
a~ ~ , r; cm
u
r:tu vl c: o
70(1-1~
040
IU U r(U II.
o rlu ,, E -~
E m c o oi
~)dr(C)Ucu
U~C) U
~ ~lu001
7r( r(U C
Ol~brr
mumv~olru
~ r: o _lu
XUCO ~-~U
C~r(rrl~C
re )IUUC
E o u ar ~ at
~~~cEu~
~C~ o~o


I -I I s I


0 0















V]0 v, MO



Cc GI 03 r-- 0 U


HF., 0 cc a 1 0t





ZWag of' u "t w


ZO i ~ r 298 U .1,
















C c



-19 91 I 9 ,


- 23 -


C1
O CI
U L1

~i~~C

vl r,
7 >.
C

rr
X 01
C U
~CC
U C)

EC7i

UUU?
v 7
X vl
pt ~
V1 01
~rJT:
ool-,
~
uuv:

N r( d
rl V1 W
~Evl a~
Om~ r:
E~ L1 C1 01
01 ec ,~ c
ug~



Y~d d
vl rl~ P:
*I ~1
~LUly
VIOU
01rl C V1
O~vlCI ~~
~ ~n ~C
eL~olu
U~Cd~l,
~o ~ oltu o
o~le~r:
~~rolu


O
'y
3
rj D
01
f r
re u
I; vr

O
4
a

O
v,

O II
cr, rrlr

.e
hi EU
rl
L: ~I
E.
o x
~ vlu
to c
ar ~
DI >rg
C rOU
tr ~~
ccm o
$~ c
c~rc:
rl oE3

YINC~C
~ 000
U(U~ d
LU O U V1 L)
E u~ru-~
~~c~oEol
UOZIUI~I-O
V1UtrrUl)gl
OU rjv~U
L~UICIOIC
UUI rl
Y1 C'UtCl(p
O i.rr ~ O U
U~O~~G rT,


r(001001W

FJ CC O
W QH r


~CII
p, S
UrJ
or ~E
uc=
t.d ~
~ UO
> CI Crr

rj (D U3

tuu
rr. c rl Fi
UC)C

U 3 ~ r
o v~ri
ler
o o u
C
r C ~IX
U O >~
O rr 0~1
U U ~~
U C ri~
~cE
U 21 dV1
dV1 Y.
"

bC d rC ril
J:
n m~vl
n ~o~o
N UCUU
/01 m ~ ~a
~ e~Ec:
u ~ o ~ ~ e
c Ll~utao
d V) mEI~Fi
OIUOIOIOI
C Ir C)
1 lu ct C c
~ ~c o E o~ vl
~ol a~ol~p
u~xoo~u
>duccrum
07m~UCFi
c~-~$ rl
oc~~c~c~p~
IJ~ooolu~
dU>O
m~w~I:


)r
EI U
O
C
F:
a
Y. vl
O C)
V1
D. Y1
L re

uce
Cvl
PI -ri i
~Lin

v~C:
rl , ~
r15

UO
Olr.u
U

v.ctu
rCP.
~CiLI
tia~c
?n;
o u

d0

21-~E


~clb~c~
Coioi
u>~o
~c~oa,
Lid Ln (U

uvllu
coloca
013 U
VIUCIC.
a'~L~i
~d3
ClrOVIC)
ULU~
O ~O













CHAPTER II

STATE OF THE ART ADVANCE (SOA-A) MEASUREMENT


Introduction


In almost all of the work which has been done on the

problem of risk,* the researchers have concluded that the

most important single parameter in the assessment of Risk is

the advancement in the so-called state of the art, or the

increase in the level of technology which is required to

accomplish the project under consideration. Since this

study deals primarily with research and development ptro-

grams, it is intuitively evident that projects which require

greater technological advances will generally involve

greater risks. Therefore, if a Risk index is to be devel-

oped, one should first address the problem of deriving a

methodology for the assessment of the state of the art ad-

vance (SOA-A) associated with a project. It is important

that the system developed be logically consistent and that




*kThroughout the development which followJs, the capital-
ized "Risk" will refer to the specific concept of attainment
risk as discussed in the first chapter of this study, with
the.10wqr case "risk" referring to the more general concept.


-24 -




-25-


it be sufficiently general to be applicable to a wide range

of program types while at the same timez requiring a com-mon

methodology and thereby assuring a measure of consistency in

SOA-A measurements made by different analysts on different

projects.

Peckc and Scherer indicate that wJhat they call "State of'

the Art Index" measures "the technical achievement and inno-

vation required to accomplish a quality (performance and

reliability) objective."' They place a project on a scale

which runs from zero to one hundred and assign it a nu-mber

relative to the other (historical) projects already placed

upon the same scale. This approach is based upon the ex-

perience and judgment of those in the field and may be re-

ferred to as a completely intuitive approach. An alternate

approach would be one in which only quantitative factors are

considered; experience, judgment, and intuition would not

play an important role in the synthesis of a value for the

SOAI-A of a project. In the first place, sufficient data

are not available to permit this type of analysis; in the

second, even w~ith perfect data i~t would still possess cer.

tain shortcomings since what this type of analysis gains in

objectivity, it forfeits in the loss of the invaluable

qualities of experience and judgment.

~Reference 12, page 289.




-26 -


While the approach presented here will not be com-

pletely ofojective, in that it makes some use of subjective

or intuitive judgments, it attempts to force the analyst to

quantify at a lower level and to follow a set procedure for

determining the SOA-A of a project. This may be called a

subjective-analytic approach.





Given an advanced project proposal, the most important

performance criteria (e.g., speed, weight, accuracy, thrust,

and range) are listed in order of decreasing importance

(xl... xn) and each criterion is given a weight which indi-

cates its importance to the success of the program relative

to the othe-r criteria (al...an)* Each successive w~eight. is

less than or equal to the previous one, and all are relative

to the most important criterion (xl)*

al~ ~ ~ = 1;a- *a2 i+1

Factors which should be analyzed in the evaluation of

the ai include expected development cost, expected develop-

ment time, and importance to the success of the project.

Normally, a subsystem which is expected to account for

50 percent of th~e total project cost wrlll be more heavily

wegh~6t~ed in determining a Risk measure than w~ill one which

is estimated at: 1.0 percent of the total. Similar reasoning




-27


leads to the same conclusion about articles with_3 diffc-erent

expected development times. The third factor depends upon

the relative value the customers places upon the various fea-

tures of the project. There are a few major design features

which are of extreme importance in a project. These are thf-e

reasons the project was deemed necessary in the first place,

In addition, there will be other correlative or subordinate

"extra" features of the "nice to have" variety. In a

squeeze to produce an acceptable product by a deadline,

these secondary measures may be compromised much more

readily than the primary design features. The experienced

analyst can pinpoint these distinctions and use this infor-

mation to assist in the ranking of criteria and assigning of

the ai'

The criteria having been selected and weighted relative

to each other, the next step is to determine the SOA-A rep-

resented by each of the criteria on the l~ist. Let the SOA-Ah

for xi be designated by Si, where Si is defined to be the

ratio of the desired performance level to the performance

capability of the most advanced currently available (devel-

oped) model.

Desired performance level
Si current demonstrated capability

In cases where an increase in performance results in a




- 28-


decrease (e.g., weight, volume, resolution), simply invert

the ratio to obtain Si.

At this point in the development, a set of numbers has

been computed that indicates the relative magnitudes of the

performance increases that will be required by the proposed

system. Now it becomes desirable to adjust these perform-

ance ratios by taking theoretical limits into consideration.

As a technological breakthrough occurs, new performance

capabilities are realized; but a given level of technologi-

cal capability can be exploited only so far. The principle

of diminishing marginal returns begins to operate; and in-

creasingly greater amounts of time, money, and other re-

sources are required for additional performance advances.*

The modelsaccounts for this fact by adjusting the Si for

theoretical limit considerations. An adjusted value for S.

(S ) can be .obtained by referring to the theoretical limit

curve of Figure 3.~ The curve is used by estimating the


*xAt the 13th annual meeting of the American Astronauti-
cal Society, Edward N. Hall presented a paper on An Econom-
ical Approach to Space Transportation (16), in which he
demonstrates an exponential relationship between the ratio
of actual final program cost to initial estimates of program
cost and the change (or advance) in the SOA which is sought.
This supports the author's choice of the exponential form
for the I curve of Figure 3.

~The curve in Figure 3 was postulated by the author af-
ter analyzing Terrell's curve and discussing the subject at
great length with experienced aerospace planners. It should
be determined empirically as more data become available.




-29 -


.10










.01


0 25507510

Pecn ofTertia iit--.4



Fiur


THEORETICAL LIMIT CURVE




- 30 -


degree to which the present level of technological development

has been exploited and the approximate percent of the theore-

tical limit which has been attained. Entering this value on

the abscissa, the corresponding value of ri can be found on

the ordinate. If is a dimensionless number which indicates the

technological difficulty of the project. This 7 curve is

determined empirically by assigning Sivlustcieran

historical programs and determining the corresponding values

for the Yi. The curve may be revised from time to time. It

may also be necessary to define different curves for different

classes of programs.

The alueof i is obtained by multiplying Si by Tt i'



Thus, a set of values for the Si has been established which

takes theoretical limit considerations into account. Finally,

the individual Si measures are combined in a weighted summa-

tion in order to arrive at a measure of the SOA-A of the total

system (ST). Mathematically:


ST : Bi S
i=1

Obviously, the series ST will continue to increase as

the value of n increases. It therefore becomes necessary to

demonstrate that in practice ST will tend to approach a limit-

ing valu~e. Study of past programs reveals that values for Si




-31 -


tend to lie between one and five; and, in most areas, a value

of three would indicate a large increase.*; It is difficult

to imagine an aircraft project which would require an increase

in speed of more than twice the present capability or one

which would require a threefold increase in engine thrust.

Since Si is the product of Si and~i, its value can be

made large by inserting a large value for either Si or 17i*

If Ti is large, the technological difficulty is great; and
the relative amounts of time and dollars required will norm-

ally be large; therefore, one should expect a high value for

a Since large values for Ti indicate reduced cost-effective-

ness, it would not be expected that unimportant criteria would

be designed to require values approaching the theoretical

limit. Onpthe other hand, it is possible to have large values

of Si accompanied by small values ofti (for example, before

a technological breakthrough has been exploited). Any

criterion which requires a large performance increase should

rank high in importance to the project and will therefore

carry a relatively high ai'



*Notable exceptions to this idea are found in the
fields of electronics and computers. Recent advances, such
as integrated circuits, have given rise to quantum increases
in speed and decreases in weight and volume. ~The assumed
relationships among high values for Si,17i, Si, and ai tend
to minimize the existence of these exceptions as an o jection
to the hypothesis that ST approaches a limiting value.





- 32 -


I---~---i-~~-~: 1-I~~-~-~~ I

I- 1-

~ ~ i~-I~-i I :ii; I ii;5t t r I i i /
--- I---~-I- tTI 1 ~- i....i.. '.....i.,
1 I - ~
s~ ~
L~J;.;~--------(--'-
i'" I P il~j;i :1
i- ~I
''
-I
n'r-i I i~iri i
--i-::-~- -I-r- I
C ''-'"
r;
i
iI;--I
-r;
i: r i--
1 (
i" :I i 11, iii ::~: I
;1. ."i
fi~'ttlil I-; iiil I I ---- C:rr
i; i _: I:
11 I: r: !i-l ;i 1~
; b?:_~.;illlllll~-.l
I
r ~ .1 ciIn:~ '',-t-i
Si .. I; ;: fiir r: i
-,;--
"
I
r -I
v,
2i~C~e, (iTii:- I
i;ti i_/ 1 !Tli~
it I~
rr,--
-Ir,"t --(i-
I i! i i
'ili ; L.1 Ir-i I I-;-f I;iif
I iii -:
li- '.t-L~1 L!~~I~-. ~LIT -TIX-~X
~( II;T
-t~ift?;' ~I
::i:I:
i-;cT
~. j. i -;
1/
LIS: !r 1 i II
: -,r
II I- I i- r-
i-":- i:.TI? --1-i-i P./.a~f~LL~e-,._1
I' : 'i"i!i "ii

2 4 6 8 20 12

n -- --~



Figure I,

IIYPOTMETICAL IIII'IJ.I rV~
li~f~l ~nn ST




-33 -


As indicated above, if either Si or'C~i is large, the

product will tend to be large also. Since large values of

either S. or r. tend to be associated with large ai values,

it may be concluded that Si will be positively correlated

with ai. Following this line of thought, then as i increases,

ai, either Qi or S and therefore Si will decrease, yielding

a relationship between ST and n which has the form of Figure

4, where ST is the hypothesized limiting value of ST. Analy-

sis of over twenty real and hypothetical programs, ranging

from the very simple to the extremely ambitious in techno-

logical complexity, indicates that the expected range of

values for ST is

0 & S^ f 1.2.

In gd'n~eral, it is the intention of the author that

the value of n should be less than ten. Beyond this value,

the relationships among the a. are virtually impossible to

assess with any degree of accuracy. The procedure is designed

to permit comparisons among similar programs, and it is the

intention of the author that only the major performance

measures be included in the list. It is expected that n will

tend to have a value of four to six in most instances. Since

the methodology is applicable to a wide range of sorts and

sizes of projects, it can be used to evaluate a complete

system, then be reapplied to the more important subsystems.




34 -


Summary


A review of the methodology for generating the state

of the art advance (SOA-A) for a project would be instructive

at this point. The steps to be followed (see Figure 5) are

these:

(1) Select the performance characteristics which

are most important to the system and rank

these from most to least important.

(2) Assign numbers (a ) to each of these criteria,

indicating their relative importance to the

overall proj ect. al = 1; ai-1 2 ai2 ai+1 *

(3) Calculate the S. as the ratio of expected

performance to the best currently available

model performance, inverting the ratio in

cases where improved performance results in

a small numerical value for the index.

(4) Estimate the percent of the theoretical

limit which the desired performance ap-

proaches. Using the Theoretical Limit Curve,

find the appropriate values for T~..

(5) Adjust the S. to account for theoretical limit

considerations. S. = T.S.










____II_


.__-_3


Ranking by Rela-
tive Importance
and Assignment
Weights (ai)


Determ~iination of
Best Available
Performance


Calculation of
Ratios of De-
sired Performance
to Best Available
(Si)


~IC_ ~~1__1


I__ __


Adjustment for
Theoret~ical L~imiti
Considerations
(s6


W~eightedl
Symnati~on of'
Si to Obtain
SOA-A for Program


-35 -


Program
Performance
Requirements


Selection or
Performance
Criteria (X:i)


Dat~a from
C their
Programs


Theoretical
JLimit:
Derivations


Historical
Data


Risk Index
Determination


Figure 5


STATE OF THE ART ADVANCE PROCEDURE




- 36 -


(6) Combine the Si in a weighted summation to obtain

a measure of the overall SOA-A for the total

proj ect.

ST i Si
i=1

This procedure permits the use of subjective judgment

on the part of the analyst in assigning the weights to the

criteria and in the selection of the criteria themselves.

It requires quantification at a level below the total system

level, however, and thus forces the analyst to examine

critically the actual makeup of the final composite SOA-A

measure. This procedure combines the stronger features of

both the completely intuitive approach and the wholly

analytical method, resulting in a balanced analysis, which

will tend to give more consistent SOA-A measurements in

the long run.













CHAPTER III

A RISK INDEX METHODOLOGY


Selection of Parameters


SOA-AT (ST)

In the preceding chapter, it was pointed out that the

state of the art advance required by a project is the most

important single factor in the determination of an overall

Risk index. All other factors held constant, the Risk

associated with a program would increase if the SOA-A in-

creases. cl~n the absence of any information about funding

and time schedules, the SOA-A would be a reasonable measure

of the inherent Risk of a program. This will be one of the

factors used to develop the Risk index.


Performance

Performance attainment is a vital measure of program

success. Detailed consideration was given this variable

during the course of this study. In the final analysis, it

was decided that performance would not be used explicitly in

the procedure. The reasons for this decision were these:


-37 -




- 38 -


(1) ST measures the inherent technological difficulty

of meeting performance goals. To also include a

measure of the probability of meeting these goals

would be a form of double counting.

(2) Historical data refer primarily to programs which

have met (or substantially completed) the stated

performance goals.

(3) A program is not considered to be finished until

the performance criteria have been met to the satis-

faction of the procurer.

(4) W~hile there are some instances in which projects

have been declared to be completed when not all of

the performance goals were met, these are the

exceptions, and the programs were in all likeli-

hood not considered successful by either contractor

or purchasing agency.

(5) If a project is completed before the time and cost

constraints have been encountered, the contractor

would tend to add additional capability, but only

to the point of using up the remaining resources.

Since many projects overrun these resource contraints,

it seems improbable that contractors would volun-

tarily exceed the required performance goals, and

that the overruns would be small.




- 39-


(6) There does not appear to be a systematic method

for measuring the cost and time overruns for a

program which is cancelled.

In light of these facts, it was decided to consider the

meeting of the performance goals (or substantial compliance)

to be a constant factor at 100 percent of the contract spec-

ifications and that this would be a benchmark from which the

other variables would be measured. In other words, the total

program cost (actual) is the cost up to the point when a

satisfactory product has been delivered to the customer,

and this date will mark the completion of the project in

time, as well.


Program Cost

Program cost is a tangible, measurable resource con-

straint in any program. It is a readily available item of

data on historical programs and is an objectively measurable

quantity. Virtually every attempt at risk measurement has

included cost as a factor. Any discussion of Risk must

ultimately be expressed in terms of dollars in order to be

useful to the decision maker, since any discussion of Risk

by managers, program planners, or legislators will have

economic overtones. Therefore, cost is one of the variables

which will be used to determine the Risk index.




- 40-


Time

To Government planners, considerations of time are

second in importance only to those of cost. To the military

commander, time is the overriding consideration, since the

ultimate wceapon delivered a day (or an hour!) too late can

be totally useless. In planning for the defense of the

nation, the meeting of delivery schedules is vital. Since

the 1950's, this country has been engaged in a vigorous

competition to overcome and eliminate various "gaps" (missile

gap, booster gap) and to out-perform the Soviets in other

ways (the "Moon Race"). In each instance, time was the next:

most important criterion after performance and reliability,

and cost goals were repeatedly revised upward in order to

meet and surpass time objectives. Time has therefore been

chosen as the third variable which w~ill be utilized in the

derivation of an index of Risk.


Other Considerations

As has been indicated above, some previous studies have

used measures other than the three selected here. Since the

approach used here will be to build a set of logical rela-

tionships, the three variables which best describe the pro-

gram have been chosen. In many of the previous studies cited,

the approach used was to fit hypothesized functional rela-

tions.hips to historical data by use of a least squares




- 41 -


routine. While this approach gives the best estimates over

the sample, it does not, in general, give the best estimates

for programs not included in the sample. Since this study

attempts to develop a system based on logic rather than

empirical fit, these three basic factors were selected.

It should be emphasized that almost all of the prior

studies relied upon these three as inputs, even though in

many instances other measures were added to give a better

empirical fit to the data. In the first issue of Aerospace

Management, General Electric Company's Missile and Space

Division presents a summary of their Program Appraisal and

Revision (PAR) system which lists the most important

variables as "Technical Performance, Schedule Performance,

and Cost Performance"(17).


Reasons for Subjective Assessment


The chapter of this report dealing with the method

for determining the SOA-A of a project indicated the

principal advantages realized by a system which permits the

experienced analyst to make use of his training and to

bring his expert judgement to bear upon the problem. It

was indicated there that, while data collection for a purely

mechanical system would be a virtually impossible task,

even the availability of perfect data would not make such




- 42 -


an approach attractive since it precludes the use of the in-

valuable qualities of experience and judgment on the part of

the senior analyst. Consequently, the development of the

Risk index begins with the generation of two sets of estimates-

one for program cost, the other for development time. It is.

assumed here that these estimates are derived using the best

currently available cost and schedule estimating procedures,

and that they are consistent with the other estimates which

would normally be made for the program.


Risk Index Procedure


The development of the index is begun in a manner similar

to the PERT methodology. Given the performance requirements

of a program, a set of three estimates is prepared for each

of the input variables (dollars and time) indicating the ex-

pected values of resources required to meet the performance

goals:

(1) Low an optimistic value which has a probability
of only 1 percent of actually being met, even if
all goes well and no major delays or cost increases
occur.

(2) High a pessimistic value which has only a 1
percent chance of not being met.

(3) Modal an estimate of the most likely value, taking
into consideration past projects of a similar
nature.




- 43 -


Using these three estimates, probability (of program

success) distributions can be constructed for development

time and cost. Following the PERT analogy, the Beta

distribution wias selected.* Figure 6 and Figure 7 are re-

presentative of possible forms for such distributions. The

preponderance of opinion seems to indicate that the right-

skewed form of Figure 7 is the more likely one for both

schedule and funding; the possibility does exist, however,

that some projects may be better described by the skewed

left or even the symmetrical distribution. This possibility

is one of the reasons for the selection of the versatile

Beta distribution.

Superimposed on these distributions, the actual values

of the contractual (or tentative) values for time and funding

(to and $o) are plotted. The shaded areas under the curves
to the left of these resource restrictions represent estimates

of the probabilities of development 'time and cost success for

the program ( Vt and $) ; that is, the probabilities that the

project will be completed within the assumed budgetary and

schedule restraints.





*;For a thorough discussion of the PERT assumptions and
their implications, see K. R. McCrimmon and C. A. Ryavec,
An Analytical Study of the PERT Assumptions (18).


































































~------ X~u~nba~L~I


:.i
..
_._._I~ .i~--~_I


--! I


-i
-r


- 64 --


--i--
--"
--~1--r-

--t


i.__/
-~--~------"
.. ..1











-~-i --I









_-~VI
-- --~~---- I
"--~LLI~:_i.

_~.~___ci -I
i :i i--i- i


I _


i._l-L_ i-
1 .:il : i------- -i--

-i-, '--!--
I_ ,
i I -i_
i.
1_1. -T-~"-~
I r


-'----
r.` I i .~ .1 .;.

i r- j -H-tf
r?


!---i -i--:


Il-sT~
I ,


_I." i,.


i--*+ _.i
.i i.
II- i~_i~i j - r-- I i

ti i~-l~-t--



r I
L~ 1- .1
i-
r. iii


,

i'
s I~I_
-,--.--
Li ISj~-~-i----I- i cl~
i 1 ;.-ci..!
i.i..i~ .~1 i_.;.
1-
ti-,-
i
r ----

I
:I- .. i..i r-I--j

:-C_-C---
.~.. i
"'


i~~


]_




;-I
L-i.~


.~_~_I
.1_1


I~
i~Z
t-ti-
ii.i
-i--l-"
i


/-i r-i-?
1 ~i ii i
r
j i




- 4,5


I--:---1 -i- --C-i- i I ~---:---; -
j:: r ------- --- :-i I
----I -,- i~.--i- ~-~-----~ ~--i~~--
~1.1 .I. C:. I on
_I-_l-_~--j~__~i .. /_ .~_T I c
.i-?-i--i--i- I---'-i : II rl 1 I i-
I:-----I : -- r-i 1~~~.~ 1L 1_ i
.._~~_._~1_... _..~J~ __
-.-I i:.:I.. ---:-'
I T~ -f El ti;---l-;1~_1 i
!-i--
I
--~ -- -~ -'~-"-I t~l_.~I ---.'"
..... .~i.~
Iri : i--:-i cn
__i~ i... i...:...i i--i i ---~ o\
o
---'--'- ~------~- -:-: .....:.. L.
--- r--i--f- ~1 i ~7 Tfj -';
L. ., ) : .i' .1,. :.. I.: 1 111 ji_( ~
._I..'... i. .'.... ~_-. i..~i. I .1. i..?~ T
.i~...i...~ W
I
: i A
~J t-;
i 1-r;
I- i ,- i i~i v r-:
I /1
-1--- w;
]-;l--l-i y -1 I ---i--i--! t E-l T~I
i-----1 1 _ii. U
.I 1_.:. c
(u ~n
i i 0 E4
ii~ i ~i_ I. .1 a,
iL .Ii. 3 P
o c~
__ri. !__1 t r I -::i::(j a rll
---i- -I ----i- - a a
----:1 :-, I o
.--l----t "
iii _,. i.i-'- -~ :- i !
i-
0'
------~--~-\1: -i _I ~ .L J\ ._-II I_ ]j`_~2 w
i- i- Er-(
i\
--,--- ;T~-T
i...... ~~I--L j I .~
: 1 I~ r
o
rl

I--r:-t-
i ~: i.. __C__I, f. ~
~C::::~ . _1__1.__
i j

Q----- X~rranba~izl




- 46 -


In most cases the modal estimate will be either the

best available estimate or the contractual figure, since the

person performing the analysis w;ill base his estimates upon

information from the group which generated the contractual

figures. Even though thle value for $o or to may fall on or

near the modal value of the frequency distribution, the

shaded area (the value of ~$or @t) will be a function of

the distribution form, and wll therefore differ from one

distribution (and project) to the next. It is often of

interest: to determine the effect which a change in funding

or time from the expected values will1 have upon the proba-

bility of program success. As will be showJn presently, a

sensitivity analysis permits calculations of this sort.

If a sensitivity analysis is to be attempted or if there

are several alternative values for $o or t the generation

of a cumulative distribution function from the frequency

distribution (see Figure 8) permits a much faster analysis

while at the same time makcingg the model more easily under-

stood. Fromn suich a function the value of ~ o: l/t can be

read directly fr-om t~he ordinate for any value of $o or to.

The probability of twoo independent events both occuring

is the product of the probabilities that each will occur.

P(both A and B) = P(A) P(B)

While the events considered here (schedule success and budget




- 47 -


LIow 10% Modal ,$o 90% High


Development~ Time or Cost: >--


Low 10% $o 90% High
or
t,
Development: Timre or Cost >---

F~igure 8


CUMUYLTjATiIVEr FREQUENCY DISTRIBUTION'I) F'ROMl COST ORl TI'ZE DIESTRIB3UTION




- 48-


success) are clearly not independent, the present state of

development in Risk analysis is sufficiently crude to permit

this simplifying assumption in the analysis. Among others,

Terrell made a similar assumption with reasonable success.

Relying upon past empirical evidence as a partial justi-

fication for the simplifying assumption, the product of the

probability of cost success (Y ) and the probability of time

success (}ft) shall be used to represent the unadjusted prob-

ability of success for the total program ( WU '

YU = $v v
The unadjusted probability of failure for the project ( U)

is the complement of~VU*

U, = 1 U, = 1 /t $a
It was indicated in Chapter II that the value of the

SOA-A measure (ST) would normally fall into the range
0 L ST L 1.2 .Since it is intended that the estimate of

the experts shall be the base figure which is adjusted for

SOA-A, the unadjusted probability of failure will be multiplied

by 1 + ST in order to adjust the probability of failure in

light of the expected technological difficulty of the project.

The existence of an "optimistic bias" in defense contractor

estimates has been discussed in a previous section of this

study. This final adjustment attempts to account for such

bias by increasing the probability of failure by an amount




- 49


which is dependent upon the technological advance sought.

This would appear logical since if no technological advance

is sought, the estimates should be accurate as given, while

if a large advance is sought, the probability of failure can

more than double.

Once the Risk index has been generated, it becomes de-

sirable to put it into its most meaningful and useful form.

The concept of Risk can be thought of in two ways: one of

which is negative (pi the probability of program failure),

and the other positive (f/T, the probability of program suc-

cess). In general, a statement of the sort "The probability

that this system can be produced for X dollars in Y months is

0.63," would be preferred to the corresponding statement of

the probability of failure. Obviously, the two measures are

complementary; that is, "T = 1- z = 1-(1- /b)(1+S ) .

If the value of f'T is negative (for example, if3V or

t~ = 0, 3ig = -ST), it should be considered to be zero. Since
no advanced program can have a negative value for ST' ~T is

always positive. This places the following limits on the

indices: 0 fT & C1.0; 0 5: ff 6 1.0 The Risk index is

designed to support, not replace, the decision maker. Like

any tool, it should be used with common sense and good judg-

ment. A small value for?'~ or7Y should normally eliminate

a project from consideration, regardless Af its 7'f value.




-50 -


Sensitivity Analysis

If the Success index ( FT) and the Risk index ( ~T) are
expressed in terms of the original inputs, it can be seen

that, while the indices may be imperfect, they do respond
in a quite reasonable manner to changes in the input para-
meters.

) 7 = ,1-(1- 3(g 3L?)(1 + ST) IT = (1- 37 ff )(1 + ST)= 1-~T

0 I 3'rT 5 1.0 ; 0 IL pT I 1.0 ; 0 f ST 1.2
Consider the change in y'z for the following:

(1) An increase in $o:

An increase in $o leads to an increase in ff


This increase in ff leads to a decrease in ~T


A decrease in PJT implies a corresponding increase

in 3LrT


(2) A decrease in to:
The decrease in to leads to a decrease in 31r


which in turn leads to an increase in (dT or a
corresponding decrease in }'tT




-51 -


(3) An increase in ST:

If a change in configuration or in one of the

system requirements occurs wJhich causes ST to

increase, this will cause ~T to increase andff

to decrease by the same amount.

+ ST*+n(1 t p$)(1 + ST) = dT dV
From the magnitudes of the changes in T brought about

by changes in the resource constraints ($o and to), the

analyst can estimate 3 T and 3 'T.Thsetitsmy
$, Soe esiae toy
be valid over greater or lesser ranges, depending upon the

location of the points chosen along the cumulative distribu-

tion function of Figure 8. As a rule of thumb, the middle

third is usually fairly linear while the ends are definitely

nonlinear. If he can then detennine the dollar value of a

schedule decrease in the program (for example, if he is told

that the Air Force will pay an additional $2 million for a

three-month speed up in the time table), he may determine the

resultant impact: upon program Risk of these resource constraint

changes. If it is desired to determined the impact on Risk of

a three-month reduction in program length and a $2 million

budget increase, he must find

T~dY T,~ Ato =~ -3 onhs T d$, = + $2 million
If the Risk is reduced by decreasing time and increasing

money by a corresponding amount, this indicates that the




- 52 -


schedule mnay profiiitably be reduced. WJhile th~is analysis wi~ll

not nec~essarily give- the op~tina~l. mix; ofE t~imel and fundling for-

a project:, ~it wi~ll, ndic-atief the properly di~rectiocn fotr :I~impove--

ment. Efficient strategies can be determined b~y varying

thne values ofr $0 and! to uniil thez dollar eqyuiv:l~ent of a

schedule change produes an equilib~rium6 of the form


+I nt, 'F a $ .

Thi~s strategy i~ e-fficien7t because the amo~unt, th;e custocmer- is

w~illing to increase the funding for a decrease in schedule

andl the. time extenzlsion he- is wrZilln t~o grant: foril a dccreanse

in cost- are nrot suff~icient- to reduce the~ Ri:sk~ associatedl wijth

the program.

Th~e values wh~ich cr.e obtraijned fTor the. sens~-'itivi~ties of

'4 to changes in resour-ce restrictions v1..i hve mecanin, onlyS

ove:' rlim-ited~ range., If sigenif~ica~nt change's aref CconSII:ltemplated

se~nsit-ivities should be rczomputed for thbe prLoposed values








thle selec:i~on~ of input: parametersj- an~d thes decision to use a

pa~rtal~ly su.bjct~ivc evaiua~tion o= t~he R~ick: in)de::. Trhe st-eps

followedrr~ in the~lr deve.7il~opmentr of t~he :inxdexi (,rea Ts.Figure 9) were7~1:

(1.) Clvenr- the-? padrici:?:~l.nce rqi-Cl.Sr7?rements ofC a Syste.,

genratc-ce low, hligh,and mo"-dall varlueCs for : alndl $,




- 53


(2) Uis~ing t.hese~ esti rt-sl c-,ntiruct~ the probabjilit

(of~ p-c ~. suzccuse dl:stribut~",ions forT $ ndt

(3) On: t:hcese disl'tibutilons, p~oi: $ ; ~id t-o, and calc-

latre ilri ar~s toc t--he letLi ofj iLtheser con~stint~L~.lsS


( t-') f (Uilti edR~i:y, th~e c~umua!.tive" distri.-
but'io;. fu~nct7~ion of FTiigure S nay bei generated andC~);

th~e ;?-lioj:;.i oili ies rd di._ ,,y f-crom CLn co i. ~rut*.)

(4:) Calculate thre unadijusiret p-obai7-'ility ofi success


(5) djus theprobbiliy o failuvac for state of el.~


are advance ,=( /UT



take2 the ce~.:p a -inot of f:*T, wfiich1 is the piOrobabi.lty

of total prc_ onK SucFCess,

'/~1 = 1 (1 $i $(1+

(7) To0 apply thect index,, genleractce the sensitivities.

11/', and


(8) Dte-rmi:ne efficient- all.oor:tion wh7er~e




The index~ derived here~ is nost: an ui.lt-im~te rindcex i~n any

s enls e. I~t- is anl attem-npt- to derive~\7 a 1 ogica:~ lly cons8iS ten~T

no~thodol.o 3y fo:- Ccenerating 2 R~isl: info',a:. vbich7 hars so:.10i ;e-3l



Cefore t1~ _inliiC can be~ usedc -ao cniae











Distributions of
> T~ime and D~ollar
Attain~ment Probabil~ities







Probabilities of
> Time and Dollar
Success (0, and #4)







Probability of
SProgram Success (#Tr)
or Failure (Q07)







Sensitivity
Analysis and
Program~ Eval~uationl







Figure 9


Performance
Requirements


Values for
Funding Level ($0)
and
Development Schedule (to)


Value for
SOA4-AT


RISK INDEX METHODOLOGY




-55 -


it must be validated by application to real problems. Al-

though it is as yet an untried tool, the methodology developed

here represents a step forward toward a measure which has

physical meaning (probability of program success or failure) .

Additional research and a broader, more accurate objective

data base will refine or revise the technique suggested here

into an even more practicable and reliable tool.













CHAPTER IV
REFINEMENTS


Revision of the Present Model


In cases where the program is sufficiently well defined,

alternate methods for the generation of the cumulative dis-

tribution form may provide a more solid base for the deriva-

tion of the Risk index. One such method involves the use of

concepts developed by Faucett, Henry, and Wilson (19) in

their Network Analysis Model (NAM). Their model requires that

a Program Evaluation and Review Technique (PERT) type network-

be constructed and time or dollar estimates made for each

activity. A computerized model then makes many passes

through the network in a random manner and generates a cumu-

lative distribution function for the probability of completion

versus time or funding. If the program is defined in suffi-

cient detail to permit this type analysis, the resulting

distribution may give a more accurate picture of the relation-

ship which actually exists between cumulative probability of

success and the levels of funding and schedule. (See Figure 10.)

A second method which may find application in reducing

subjectivity is the technique known as the Graphical Evaluation


-56 -
























o ?o



OE

w---- <00
-rO O

5-








5 O co


OrO





E-4v


F~0


-57




-58 -


and Review Technique (GERT)(20). While NAM is a simulation,

GERT is an analytical technique in which topological analysis

permits the generation of the moments of the distribution

which describes the network representing the program. If the

form of the distribution can be recognized from these moments,

then the actual distribution can be generated from the

moments. Further study is necessary in order to determine

the applicability of this method to the Risk index method-

010gy, but the approach appears to have promise.

Another technique, which may be used on projects which

are not defined in great detail, as well as the more clearly

detailed cases, is the one presented by Sobel (21). He shows

that while the PERT technique assumes a scaled Beta distribu-

tion, the ,specification of one additional piece of informa-

tion (the 80 percent central range) by the analyst permits

much greater freedom in the form of the frequency distribu-

tion and in general results in distributions which more

accurately describe the actual function. Computer programs

are developed which analyze and quantify the uncertainty of

cost estimates. While Sobel's analysis is stated only in

terms of cost, it could be applied equally well to time dis-

tributions. Additional effort in this area would enhance the

validity of the Risk index developed here.




- 59-


Two recently developed decision making techniques may

lend themselves to incorporation into the sections of the

Risk procedure which require subjective analysis. These are

DELPHI*, developed by the RAND Corporation, and PATTERN

developed by Honeywell's Military Products Group. The two

methods are very similar in that they both center around

reevaluation of decisions upon receipt of additional infor-

mation (the opinions of other experts on the same problem).

The basic difference is that DELPHI forbids discussion between

the participants, while PATTERN encourages it. Both methods

claim to have demonstrated that in most cases consensus answers

can be reached after several iterations. Techniques such as

these may find application in the subjective portions of the

Risk index~procedure.


Possible Measures


This segment of the study lists some other potential

refinements of the Risk index which would be useful in the


*Developed under United States Government contract, this
technique derives its name from "Project DELPHI", which began
in the early 1950's at RAND. The initial use was to deter-
mine vulnerability of the United States to nuclear attack.
It has since been refined and is being promoted as a manage-
ment planning tool.

Planning Assistance through Technical Evaluation of
Relevance Numbers. Developed by Honeyw~ell to aid in ranking
proposals in terms of relative attractiveness for future
business.




- 60-


assessment of program proposals, along with the principal

advantages and some of the problems associated with imple-

menting each. This list is not intended to be complete, but

rather to indicate the wide variety of possible approaches 'to

the problem of risk.


Point Estimate of the Probability of Success

The measure developed in this study may be classed as

a point estimate of the probability of success. Use of a

logically consistent methodology assures some degree of uni-

formity among values assigned by different analysts and per-

mits extension to decision making and assignment of tangible

meaning to the index. The principal disadvantages of this

sort of measure lie in the difficulty of generating a set of

data in the proper form, the fact that some of the assumptions

necessary to the derivation of the index may not hold strictly

true, and the inability of the analyst to assign a confidence

measure to his estimate of the probability of success.


Point Estimate with Standard Deviation

The point estimate with a measure of dispersion is an

approach which was considered at some length during this investi-

gation. The inclusion of a dispersion measure would permit the

analyst to indicate a confidence band about the value he obtains




- 61 -


for the index* and would allow him to express a confidence

level for the data from which the estimates were derived.

The main difficulty arises from the fact that the form of

the distribution of 7Y' is unknown and must be assumed. The

inability to derive a logically sound methodology for measur-

ing the standard deviation prevented its inclusion in the

method developed here.


Attainment Profile

The attainment profile as a measure of risk is discussed

in a following section. In essence, it evaluates the proba-

bilities of attaining various percentages of the desired

schedule, budget, and performance goals. An advantage of

this approach is that by use of a profilee adjustment function"

the profile may be updated over time. This permits analysis

of a more dynamic nature than is possible with other types of

risk index. Disadvantages are the difficulty encountered in

the determination of the form of the initial profile and the

fact that the entire profile must be updated for any change

in inputs or specifications.~



*;This could take the form of a probability statement:
P(.82 I )2, I .96) = 99%

Subsequent to the work done in this area in the course
of this study, Terrell (22,23) independently synthesized an
approach similar to the attainment profile for a study of
the effects of schedule compression upon program risk.




- 62 -


Financial or Economic Risk Function

The financial or economic risk function represents a

particularly useful tool for decision makers, since the

decision to attempt a project ultimately hinges upon economic

considerations.> This method would generate a risk function

in terms of the impact risk has upon the firm from an economic

point of view. It relieves the analyst of the necessity of

making a "two-step" study, that is, first assessing the risk

and then applying this measure to financial considerations.

The principal disadvantage stems from the fact that it

requires simultaneous consideration of several aspects of a

many-faceted problem, and the resulting difficulty necessi-

tates the use of even more sophisticated analytical techniques.


Other Possible Measures

There are many other possible types of risk indices

which may be developed, depending upon the particular defi-

nition of risk one chooses. One such general type of index

which was considered in the course of this investigation

is the idea that risk may be a variance (of program length

or funding or product reliability, for example). At least



*xThere have been instances in which military values
became the overriding objectives, but even under these con-
ditions, the decision of how to implement the desired system
becomes one of economics.




- 63 -


one other study (24) speaks of risk as being the variance

of factors which are not included in the estimating equations.

In general, the term "risk" will mean different things to

different people in different contexts, and the type of

measure which best describes risk is a function of the

particular definition employed. For a discussion of other

measures of risk, see Some New Approaches to Risk by Byrne

et al. (25).


Problem Areas


Data Accumulation

One major problem, regardless of the approach used in

the risk analysis, is the unavailability of accurate data

in usable form. The form of government data can obscure

the actual contractor costs; there is little usable data

exchanged between firms because of the competitive nature

of the industry; firms are reluctant to provide data which

indicate that they have overrun cost and/or time estimates;

both security regulations and proprietary motives restrict

this flow of data.

One possible solution to this problem is the generation

of a data base by conducting a series of controlled experi-

ments in which care is exercised to assure that accurate

cost, schedule, and performance data are accumulated in a




-64 -


usable format. This could be carried out within a single

firm or (if precautions were taken to protect the parties

involved) among firms. Obviously, all firms involved might

benefit from the larger data base available as a result of'

such cooperation. Certainly the availability of a better

file of historical data would improve the accuracy of any

risk index.


Bias

The problem of bias is one which appears at every phase

of risk analysis. Several studies speaker of the "optimistic

bias" on the part of the prospective contractor in submitting

estimates. This may be separated into deliberate bias and

unintentional bias. The deliberate bias is injected in

order to underbid competitors and obtain a contract, knowing

that additions will permit the tentative figure to be raised

to a more realistic one after the program has begun. Any

point in the analysis which permits or requires judgment on

the part of the analyst or estimator invites bias of a personal

judgment nature. While seemingly undesirable at first glance,

this type of bias may well be a desirable feature if the

analysis makes use of the experience of senior analysts.

Another type of bias is the bias in a statistical sense

which is present in the selection of the sample of data

points from among the total universe of all aerospace programs.




-65 -


The small sample size, and the fact of nonrandomized selec-

tion of points, preclude even the measurement of the bias.

This sort of bias has been unavoidable in the past, since

the data which were available in usable form were limited.

Hopefully, this situation will improve over time as the

need for data more applicable to risk analysis becomes

recognized throughout the industry.


Distribution Forms

At various places in the analysis of Risk, it becomes

necessary to assume forms for statistical distributions.

In most instances the lack of data makes justification of

the assumption difficult. A case in point is the assumption

of the beta distribution to describe the distribution of

the probabilities of completing a program within certain

resource constraints. The primary justification given was th~at

a previous study (PERT) had assumed this distribution form

for a similar parameter with good results. One reason for

the inability to generate a measure of dispersion to accompany

the Risk index of this study was the inability to justify

the choice of distribution forms. The only solution to this

problem is the accumulation of sufficient data to enable

analysts to determine the actual forms. Lacking this data

bank, one may only rely upon logic and pragmatic results for

justification of assumed distributions.




- 66 -


Nonuniform SOA-A Measures

At present there appears to be no methodology available

for measuring SOA advances that does not involve a substan-

tial amount of judgment or subjective measurement. While

the use of subjective judgment has the advantage of permitting

the experienced analyst to utilize the knowledge acquired

over time, it does preclude the possibility of specifying a

single number wJhich would represent the SOA-A for a program.

At the present time there is an array of SOA-A measures rather

than a single number; there is, in general, no consensus among

analysts as to the "true" value. There are advantages and

disadvantages to this arrangement, but the problem does

exist. Chapter II contains a more thorough discussion of

this problem.


Cost-Time Tradeoffs

In the'analysis presented in this study, as in prior

studies, it was necessary to make the simplifying assumption

that cost and development time are independent parameters.

In fact, the economist is aware that a tradeoff relationship

exists between these inputs similar to that depicted in

Figure 11. In general, the function will be different for

each program and will change over time during the program.

The tradeoffs may be expressed at a point or over a small







segment of the curve at a given time, but these are sensitive

to political, social, and military attitudes and thus may

change rapidly and radically. As more sophisticated tech-

niques are brought to bear upon the problem, the cost-time

tradeoff function can be defined and periodically updated

to permit the development of a more refined risk index.





























Cost ($)

Figure 11'
TIMIE-COST TRADEOFF CURVE
(Reference 12, page 510)


- 67-




- 68-


Summary


This chapter has been devoted to a discussion of some

possible refinements in the Risk index developed in Chapter

III. Several techniques which can be of value in reducing

subjectivity in certain areas of the study (NAM, GERT, DELPHI,

PATTERN, and Sobel's approach) are presented. The difficulty

associated with the use of NAM or GERT arises from the ne-

cessity of having the program defined to a level of detail

which is unrealistic for most advanced programs. The main

barrier to use of the iterative judgment techniques is that

these require carefully structured batteries of questions,

the cooperation of several experienced analysts, and greater

amounts of time than may usually be available. Further analy-

sis may reveal that a modification of one of these general

approaches will prove fruitful.

In analysis of the Risk index of Chapter III, several

types of risk measurement indices which would be of value in

evaluating aerospace program proposals are treated. Some of

these are essentially refinements of the model developed in

this study, while others represent different ways of looking

at risk and the application of different sorts of analytical

tools. A final section outlines some of the problems which







face the analyst in atrt~empting to measure the riske associated

w~ith a particular aerospace. program.












CHLAYPTE V
EXTENSIONS TO DECISION MAKINlG


Pur ose


The purpose of this chapter is to indicate a means for

the application of th~e Risk index to thie problem solving or

decision making process. First, the various environments

in whJlich decisions are made are presented; then there follows

a brief discussion of the elements of game theory with ex-

tensions to utility theory and the application of the Risk

index in both of these areas; finally, a. section is devoted

to explanation of various types of contracts and the re-

lationshi~ps which may exist between R~isk~ and contract type,


Manageme7Sntt7 Decisions


Decision M3aking


Economists usually distinguish three broad categories

or classes of decisions: decisions under conditions of

certainty, decisions under conditions of uncertainty*', and


*For3 a good discussion of thre difference between uncor-
taiint~y and r-iskc,! see K~night (2).


-70





7i


decisions under risk.*~ The class-ificatiions are dependents

upon the ,confidence the decision maker mnay place in the data

whlich he used. as a basis for the decision.

One of the clearest ways to differentiate among these

three situations is to consider a hypothetical aerospace

contractor. He has the ca~pabil.ity of building either long

range military aircraft (bombers) or inter-contrinental balSlis-

tic missiles (ICBM's) but not: both, and finds that he must

decide which ofr these twJo product: lines he will produce. Due

to the long lead times required, he cannot wait until the

defense department issues a request for proposals on the newi

systems to be ordered but must decide now upon hi~s course of

action for several years to come.

If the company's president has a son-in-law who is

Under Secretary of Defense and hias access to the information

that the Secretary of Defense has decided to rely exclusively

upon procurement of IfCBM's for the nation's defense in the

coming years, the company can decide with certainty (or as


*The termi "risk" as used in this section does not refer
speciFically to the same concept as the term "Risk" used in
previous sections. Although the connotations are similar, the
termn "risk~" will be used to refer tIo the general concept and
the capitalized "Risck" to tihe more narrowsly defined concept
of this report








close to certainty as is possible in the defense industry).

If the president w7ere not so fortunate in his daughter's choice

of suitors, thie firm w-~ould find itself operating under. cond~i

tions of uncertainty having no idea which course of action

it should pursue. In the absence of any :information about

the relative likelihood of the possible defense postures to

be assumned by the defense department, the game th,2eorists

suggest that one possible wayT to makec the decision is to

assume equal prob~abilit~ies of occurrence for each and flip a

coln.

There is, hlowiever, a middle ground between secure know~-

ledge and complete bew-iilderment on the part of the decision

mak~er. This area w~ill be referred to as decision making under

conditions of risk. In the example, the firin could move from

conditions of uncertainty to those of risk by searching the

available literature, hiring people from the defense depart-

ment, performing analyses of the conditions wh~-ich wjouzld

determine the future defense posture (eog., analysics of the

enemy threat in the future), and assigning probabilities of

occurrence to the possible alternatives. Having done this,

the decision maker is said to be deciding under conditions

of: risk.




-73


To0 summarize., if I-'co ciesio n12i makr nowS "the con8equlCai-

ces of the alt.er.nativee cour~ses of' acion open to him at thea

time of th~e decIison1, he decideS uinder: ce ~tacint-y; if h~e

k~now:s the possible ou;tcomes 3Tnd canl assis: a se~t of pr~obac-

bilities to thke:1ll, he 6 ides ulnder. riisk~; if he~ cannot: assign~r

p-robr=biliti~es to the, possibleC out-coI~ nsor if he~ d~oes not k *;

w7hat: oT-n-c6"?0 1 :-7 occur, he diecidea s under conditioons of un-~-

certainty, By ass~ini-Lng a? probab~ility of success o0 pr~oposedd

poc ,this re-por.t atte pts~ to eil:f ble the~ decision mi-;zaker

in t-he. aerospace industry to move fjlrom decisions mraki~ng under

conditions of uncerta-cinty to decision n asking unide riFsk.,




One arPea i~n which th'e Risk~ indlex m~ay fin-d diirect appli-

ccation a~s a? dccision maki,,g tool. is game theory. TIhis is

a wirjdely known tool for quantitai-ive decision Ir~lkin3 which,

in reent years, has come into w~ide-spread use as a method

fo r making business decisions. Since this reportr at~emipiTS

to enable th~e decision maker to move from the area of un-

certainty to that of risk~, it is perhaps instructive to

examine this qur-ntjitative deeci.sion ur.king tool.


~f~or~ a thoro:_ugh c"naly-sis of this sub~jeCt, Lthle readerT
is rleferred~ to Chlernoff a~nc d Hses: (26) Luce andcl ~RaiZff (27),
S1cala (28); Van; :T3uman2 ad ;\co~~rganster~n (29), cind Uill~iams
( 3 ) .




- 74 -


In moving into the area of decision making under condi-

tions of risk, the analyst finds it necessary to assign pro-

bablities of occurrence to the various possible outcomes of

a proposed pr.ogram. One method of accomplishing this is by

dlirect: substitution of tihe Risk indexx of thlis study ( '~l) for.

the pr~obability th~at th~e project- will be successful. A more

sophisticated analysis would also consider the value of com-

pleting 50 or 75 percent of the project (usually this would

result in a negative payoff or loss) and t-echniques similar

to the attainment profile could be applied to generate the

probabilities of occurrence of the respective events. The

same sort of analysis can be applied to a utility function,

once that function has been generated.


Attainment Pr-ofile*


The discussion to this point has considered Risk in terms

of the probability of completing 100 percent of the program

goals. A more thorough analysis would consider the entire

life of a project and the probability of completing various

percentages of the total program requirements within the


*cMuch of the initial wJo-rk onr the attailltnmen profile con-
cepti was done by W. J. Bailey III of General Dynamics' Fort:
WlortLh Division in an unpublished p~aper. The author gratefully
acktnowledges M~r. Bailey's assistance in the preparation of
this report.




-75 -


given resource restrictions. A technique which attempts this

sort of analysis is the attainment profile.

Programs with equal probabilities of completing 100 per-

cent of the performance goals can have significantly different

profiles; conversely, projects with similar profiles may differ

with respect to their probabilities of 100 percent completion.

Consider the profiles of Figure 12. Profiles I and II each

have JUT values of 10 percent; yet the probability of completing

90 percent of project I is only 15 percent. Project I could

therefore entail much greater risk than project II. Project

III has a mruch larger IlT than project II, but it may actually

involve more risk.

It is possible to derive a profit* function which states

profit earned by the contractor as a function of the percent

of project goals completed. If such a function were defined

and superimposed on the attainment profile, the expected value

of profit and the probability of achieving any given level of

profit (or loss) can be generated. Such values could be used

as inputs to financial planning models or game type decision

models, and should generally tend to result in a more thorough

analysis than the simpler Risk index developed in Chapter III.


*The term "profiit"~ as used in this section refers to
accounting profit (income less cost) rather than to economic
profit.













































































Fig~ure 1.2


ATTAI'CINMENT PRrOFILES


T *


I


I __ CI


; ---t
j.T/ i


____


90 100
% Completed --c





I


- 76 -


)i



h
U
I-(
r:
P
(II
P
O
k
PI











f
h
U
rl
r(
p
rJ
.f)
O
k
PI


:0.1 Probability of


L. comp~let~ing 100% of

the-; prjc

















I-completing 100% ~, rb~l~ of
-t--, :he project, but
i:much less risk tihan

project 1


% Completed ------c-


% Completed --------


III


'i i
-i


----p

..~i....i.~
i__


0.4 Frobability of
completing 100% of
the project, but
miay have mrore r-isk~
than project II9


c L. i i~T
I I I i
iTi
i ; ;_.._b.
i ;-1







: i -:i


'"1
i..i.
L
_tL!_1..
-1 /-


:. i i. i




-77' -


It is also theoretically possible to derive a utility

function for the firm and superimpose it on the attainment

profile. While this is a theoretical possibility, the prac-

tical realization of such a curve is doubtful. Profit functions

relative to percent of completion are very difficult to define,

and utility is not linear with profit and can change over time

with shifts in government policy, tax legislation, corporate

size and image considerations, and other highly subjective

factors. In short, the development of a utility function for

use with the attainment profile is not practical at the present

time.

Assuming a linear profit function,* Figure 13 indicates

some of the uses of the attainment profile concept as an ana-

lytical toQl. I1T is the probability of completing 100 percent

of the goals, and P is the dollar value of the profit which is

expected if all requirements are met within the limits set.

Point B is the breakeven point; if this fraction of the re-

quirements are met, the firm will neither gain nor lose money.

P(B) is the probability of attaining this level of completion.

Point M marks the 70 percent completion level. Point L shows

the expected amount of loss for this amount of completion, and

P(M) is the probability that this level can be realized.


*kOver the relevant ranges of completion, this assumption
is not unreasonable. Incentive contracts purposely define a
linear return in the area of the contractual figures.


























~ -I 1-






- -i -i-




~ oM


- 78 -


$;Profit: -- -


$Loss --c--


__ (







t.i-li I- ~~~-----ll-_I
i .


:---i- liri 11_1










I ~
4"L:.~_EL~,-.':-_-'~;:
r"i -I i -- ~ i
;; --: -I



I .~ .-
c\

--
---'
Ir~i- i-


i

1
-J :I ....? ..
:,:i.)-i- --
cr : i
-~-I i-


t




:
I


CO

E-


-i~~i r-1. re -fI









I




-LT_ -- -


i, I- (




- 79 -


The construction of an attainment profile is not a

well_-defi~ned process at present, Onse possible approach is

to rely upon the expertise of the analyst; another is to

use an approachr similar to the Risk index method to quantify

the 1robability ofr success at several levels of completion

and connect the points thus generated to for~m the profile;

a thi :d aLpproach1 wQould utilize historical data on similar

proje-its; a fourth method would be some combination of the

first three approaches.

The flow diagram of Figure 14 displays a procedure for

devellopi~ng and! applying the attainment profile concept in

aerospace planning. The program paramezters are input: into

a NAMT, or GERT model. The outputs from the mnodel are used

to generate the initial atitainmlent distribution. This

initial profile is then adjusted by t-he evaluation of

subjective factors and historical data. The entire profile

can then be periodically updated or revised as conditions

change over time.

In summary, this section has discussed the concept of

the attainment profile; its possibilities, shortcomings,

and some of the difficulties associated with its development

are suggested. As a concept which exhibits great potential

as a decision making tool, its value is obvious; as a





.- 0


*H aco
Or G


0 h~rl

0 O
X r-4, O!
*rl oC


~-3 J.-,
al
u
v,
';r C
r, i,
?j cJ
(J dJ
C I~
5 CC


co a




A r


A


W
3
n
w
u
o
E4;
P-r


d
k Pt
~jg
bD j~
rl
lu
E;
I-i]

H


C o

0r 0 l


k C
bnO
0 0


O -
13 Co
0L~r


A


OcJ


C ON *-rl
rJE6 C *8l e-(O!C

c) CO0
f-o *r
0>* *


C ,C

30


to
k ci
P0




- 81 -


practically attareinable measure for aerospace programs, its

future is less certain. As further research gives greater

insight into t~he na7turfe of risk and its consequences to

program success, the attainm~en~t profile may become a powierful

manage alent tool.


Contracting and Risk*


Ti e area of contract provisions wiias originally investi-

gated T it~h the intent of determining how these provisions

reflected or measured! contractor efficiency. It: was hoped

that such an investigation wJould generate factors that could

be used to debias or normalize cost data for individual

firms' e~ffciency or lack of it. This original line of

investigation wJas not productive. The analysis of contract

provisions, however, did shed light on the relationship

between cont~ractl provisions and Risk. Even more imrpo~rtant,

the investigation suggested the possible application of a

Riskc index as an aid in establishing contract types and

contract provisions. Currently the techniques used in con-

tract procurement are still being developed and contract

terms are dete-rmined largely on the basis of rules of thumnb


*cThe author gratefully ack~now\ledges the assistance given
by H~r, Morr~is L. W~Silliamson, Jr. of the Fort: H~ortih Division
of Genreral DnJrami~-cs in the preparation of- this section.








and judgmnc~t. AFSrCM 7 ~ stC ineigHa5ne

Procedures (31) r~epresent-~ls an a~ttempit byr the De~p rece"nt of

Defense to systentize the~:- system:;is cdev.e!ri.lopmen and contracting

Pr-ocedur-e. As this sysi-emn and otherc7s s~imliarss in naturzle ar~e

developed, the contr-~act-ing proedu~ire i~ll. become~ mo"re~- well

defined.

Selection of~ contract pr-ovisions Is essent-iall.y a~ po-~

cess by wh:i~ch flinanciali or cost- r~iske is a~llocat~edr betweelcn

the contracting agncy and the contractor.. It should b :

kept~ in minl-d t-hat- a pr-oject's sche~~dule r~isk andc pIerfL-or.'nce:

risk cannot be a~llocated;; the- cont~~\~;racing agncy alway-js

bear~s ulti:jr:at~e :ver~ponslr::.rib~iliy for these risks. Th;e con-

traczting agen~cyj, bour.vrc;, can miim-ize~F thoseCC risk-s by estab~i-

lishing c~ontiractc provicion~s which stimulat~ri e thl~e contru'ctoros

to, mee~t schedule and p:rfcsrurance regairements.~l:?-ts

Thi~e following basic cl2ass~ification s of contracts types

were obtained froma M;ooe (32):

I. Cost
a. CP~FF = Cost; plus ~fixed fee
b. FFR13-E = Fixedi pr~i~ce rocdetencijnable
(retroactive)

II. Incentive
a. FFI% =- 'Fixed pri~ce incent-i~ve includingg
performane ~c incentive
b. CPIF:~ = C~ost plus incentiZ:;ve fee~c

d. FF2,1? Fiv f~i:,c pri-ce .rtc;ich (withou'.)j.




- 83


More explicitly, the following conditions characterize cost

and incentive type contracts:

CPFF: Established on an estimate of total cost
that the contractor will incur plus a fixed
fee (usually a percentage of total cost).
Provisions state that the fee does not
change although costs may increase.

FPR-E: Provides for a fixed price for certain
periods of time, but provides for retro-
active price redetermination.

FPI: A negotiated target cost and target profit
is determined. A price ceiling and a for-
mula for adjusting the final price and
profit are also defined. (Ceilings and
formulae provide the incentive for con-
tractors to reduce costs and to share in
the savings.)

CPIF: An estimated target cost and a target fee
is determined. A4 formula for fee adjust-
ment (proportional to under/over run) is
set. Similar to FPI except there is no
separate price ceiling on a CPIF contract.

FPR-A: Price is fixed initially for some interval
of time and then reset at intervals for each
future period. Similar to a series of FFP
contracts for each interval.

FFP: Provides for delivery of a product or ser-
vice for a price specified in the contract.

CPAF: (Cost plus award fee) The contractor is
awarded a fee which is based upon his per-
formance in managing the program and meet-
ing the requirements as they evolve. This
type contract is not mentioned by Moore,
but has been used recently in contracts
which involve advanced technology and re-
quire frequent adjustment of specifications.

Within the framework of the various contract types,

efficiency in procurement and contracting is partially




- 84 -


determined by risks and incentives. More specifically, the

incentive rates are determined to some degree by the inherent

technological uncertainty involved in the particular program.

When the initial assessment of technological uncertainty is'

high, management often attempts to prolong contract negotia-

tions. This prevents prematurely fixed target prices, which

are often unrealistic and lead to cost overruns.

In most advanced programs, technical uncertainty de-

creases with program progress. The contractor being forced

into relatively early commitments must anticipate future

areas of difficulty and incorporate these uncertainties into

the contract essentials (i.e., varying the profit rates,

clauses for future renegotiations). The inclusion of terms

such as these decreases the inherent financial risk to the

contractor. Therefore, one of the main purposes of the

contract is to serve as an allocater of financial risk be-

twleen the contractor and the procurer.

The firm fixed price contract places maximum financial

and technological risks upon the contractor and therefore

induces a maximum incentive for cost control. The fixed

price contract provides a minimum cost Risk for the customer

since the maximum amount to be paid is fixed during negotia-

tions. This contract type is used for low SOA-A type pro-

grams or when reasonably definite design or performance




85-


specifications are available. When the cost estimating uncer-

tainties are not minimal or when uncertainties surrounding

the contract performance cannot be evaluated, consideration

is given to other contract types. For example, by the addi-

tion of escalation clauses, the contractor's financial risk

in the fixed price contract is reduced. As one progresses to

the fixed price with incentive clauses, the total program

risk is distributed between the procurer and contractor who

share the responsibility for costs greater or less than the

original estimate.

The cost plus incentive fee contract is negotiated when

it is highly probable that the development is feasible and

the desired performance objectives have been determined.

This type of contract does not change the technical uncer-

tainty connected with a program, but it may indirectly in-

crease the probability of succeeding. This is accomplished

by providing limited incentives (and penalties) and thus

stimulating the contractor to be more efficient in his

efforts and to manage the contract effectively. This type

of contract may be best suited to programs of moderate

complexity and technological advance.

The cost plus a fixed fee contract is a cost reimburse-

ment contract with a fixed fee for the contractor independent

of the management's ability to control costs. This implies




- 86 -


the financial risk to the contractor is almost zero. Hence,

this type of contract has been used in high complexity pro-

grams where the SOA-A is great, the level of effort required

is unknown, and the relative probability of success is un-

determined.

Figure 15 illustrates the general relationship between

the particular contract type desired by the contractor and

the program's technological risk.





CPFF


P1 Incentive
B Types


a With
toEscalation


FFP

High Moderate Low

Program Risk ( T)

Figure 15

CONTRACT TYPE VERSUS PROGRAM RISK




- 87-


Although contract form alone does not explain the final

performance of a program, it is an important factor in de-

termining the size of cost and schedule over/under runs.

Although the population of DOD contract procurement

actions per year numbers in the thousands, most of the

samples investigated by recent researchers have been rela-

tively small and stratified in type. Because of this non-

random selection, inferences drawn cannot be justified in

the statistical sense. However, some general trends from

past programs can be obtained from the studies by Moore,

Marshall and Meckling, and Peck and Scherer.

The study by Moore concludes that on the cost type

contracts, the average profit rate ranges from 6.3 percent

to 6.8 percent. On the other hand profit rates of incentive

contracts are higher by a sizeable amount, averaging around

8 percent profit. The rationale for the difference is that

there is a higher financial risk attached to the incentive

contracts and that a higher reward must be offered to induce

the contractor to be efficient.

For example, from the sample of 228 incentive contracts

investigated by Moore, one finds that on the average the

apparent efficiency probabilities for an under/over run are:

0.74 for a cost underrun
0.50 that the cost underrun is 0 10 percent
0.69 thiat results will be within +- 10 percent of the
target costs




- 88 -


Care must be taken in the interpretation of the above

results. Although there is indicated a 0.74 probability of

a cost underrun, one should not conclude that the procure-

ment of an incentive contract for a program will indicate

that program's success with the above stated probability.

It must be kept in mind that the type contract negotiated

is usually indicative of program complexity and uncertainty,

and that the process of contracting is a complex procedure.

As pointed out by Moore, "The basic rationale for renegotia-

tion is the existence of uncertainty in defining what is

being contracted for and uncertainty in defining and measuring

performance. "*

In another perspective, Marshall and Meckling in their

research came to the following general conclusions:

(1) "Early estimates of important parameters are
usually quite inaccurate. They are inaccurate
in two respects. First, such estimates are
strongly biased toward overoptimism. Second,
aside from the bias, the errors in estimates
evidence a substantial variation. That is,
even if estimates were multiplied by an
appropriate standard factor to eliminate the
bias, a non-negligible source of error re-
mains.

(2) The accuracy of estimates is a function of the
stage of development, i.e., estimates improve
as development of the item progresses. This
also means that estimates for development pro-
jects representing only modest advances tend
to be better than for more ambitious projects."


.*rRef. 27, p. 120 .

Ref. 11, p. 1.




- 89-


In the earliest stages of research and exploratory

development, there are little significant data available.

Here, use of a CPFF or general cost contract may be neces-

sary because of inability to analyze the magnitude of tech-

nical or performance uncertainty. However, once more

definitive data for a particular program are available,

other contract types with associated incentive provisions

are possible. The Risk index facilities establishment of

these provisions. Risk measurements of the prospective

program plus Risk measurements and outcomes for previous

programs will aid in determining whether cost plus incentive

fee, fixed price incentive, or some other form is most

appropriate. In addition, with the use of an index and a

game theory formulation, the establishment of incentive

provisions will be placed on a firmer foundation. With the

index, the probabilities of meeting cost, schedule, and

performance objectives contained in the provisions may be

assessed in a consistent quantitative manner.













CHAPTER VI:

RISK INDEX APPLICATION


Introduction


One way to promote a more thorough understanding of

the Risk index methodology is through the use of an example.

Due to the classified nature of much of the cost and per-

formance data on government aerospace programs, it is not

possible to present an actual program as an example of the

use of the Risk index. It was, therefore, deemed necessary

to generate a hypothetical example program in order to

demonstrate the use of the method. It should be pointed

out that for the purposes of this analysis the question of

whether or not the firm receives the contract will not be

considered. The aim is rather to evaluate the feasibility

of bidding on the program under various combinations of

schedule and budget.


Specifications


The design and performance specifications shown in

Table 3 for a jet fighter aircraft will be taken as the


-90 -




Full Text

PAGE 1

AN ANALYSIS OF RISK IN THE AEROSPACE INDUSTRY By WILLIAM EMERY PINNEY A DISSERTATION PRESENTED TO THE GRADUATE COUNCIL OF THE UNIVERSITY OF FLORIDA IN PAETIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY UNIVERSITY OF FLORIDA December, 1967

PAGE 2

UNIVERSITY OF FLORIDA 3 1262 08552 3800

PAGE 3

PREFACE A serious problem confronting program planners in the ' aerospace industry is that of forecasting the probabilities of success and failure of programs during the initial development stages. Many programs have final cost and schedule overruns amounting to more than double the initial estimates, and in some cases these overruns necessitate complete cancellation of projects. A methodology for assessing the technological difficulty of advanced programs in terms of the probability of success within stated budgetary and schedule constraints is needed. Such a procedure would permit early evaluation in more realistic terms; encourage the setting of more readily attainable goals for critical programs; and permit substantial savings in national resources by singling out high-risk, low-priority projects for early cancellation. The primary purpose of this study is the derivation of a procedure for the measurement of program risk. Past attempts at such a procedure are studied and evaluated. The inputs required and the variables of greatest usefulness are selected and the methodology is developed. Suggestions for ii

PAGE 4

AN ANALYSIS OF RISK IN THE AEROSPACE INDUSTRY Page Chapter I The Measurement o£ Risk 1 Introduction 1 Types of Risk in Aerospace Contracting 3 The Need for Quantitative Risk Measurement 8 Past Attempts at Risk Measurement 10 Introduction 10 State of the Art Measurement 12 Quantitative Risk Assessment 15 Summary 20 Chapter II State of the Art Advance (SOA-A) Measurement 24 Introduction 24 Development 26 Summary 34 Chapter III A Risk Index Methodology 37 Selection of Parameters 37 SOA-A-j (S,j) 37 Performance 37 Program Cost 39 Time 40 Other Considerations 40 Reasons for Subjective Assessment 41 Risk Index Procedure 42 IV

PAGE 5

improvement of the method and possible areas of application are then treated. Finally, a sample program is used to exercise the model developed in the study and to highlight its strengths and weaknesses. The author wishes to express his appreciation to the many persons who helped in the preparation of this report. Specific thanks go to T. E. Brents, Jr., W. J. Bailey III, and M. L. Williamson, Jr., for their assistance in the development of the methodology and in researching the available literature; to R. A. Gorrell and Dr. C. B. Moore for supervision and direction; to my wife, June, for encouragement and understanding; to Judy Robinson Day and Pameula Crockett for handling the typing and editing; to Dr. J. L. Worthara fo*r proofing the original roughs; and particularly to the supervisory committee: Dr. W. V. Wilmot, Jr., Dr. R. H. Blodgett, Dr. W. 0. Ash, Dr. R. N. Braswell, Dr. J. H. James, and Dr. M. R. Langham. Ill

PAGE 6

Page Chapter III A Risk Index Methodology (Continued ) Sensitivity Analysis 50 Sxirnmary and Conclusion 52 Chapter IV Refinements 56 Revision of the Present Model 56 Possible Measures 59 Point Estimate of the Probability of Success 60 Point Estimate with Standard Deviation 60 Attainment Profile 61 Financial or Economic Risk Function 62 Other Possible Measures 62 Problem Areas 63 Data Accumulation 63 Bias 64 Distribution Forms 65 Nonuniform SOA-A Measures 66 Cost-Time Tradeoffs 66 Summary 68 Chapter V Extension to Decision Making 70 Purpose 70 Management Decisions 70 Decision Making 70 Game Theory and Utility 73 Attainment Profile 74 Contracting and Risk 81

PAGE 7

Page Chapter VI Risk Index Application 90 I introduction 90 Specifications 90 yr and Y^ Development 91 • SOA-A Measurement 99 Sensitivity Analysis 102 Conclusion 104 References 109 vi

PAGE 8

LIST OF TABLES Table Page Number Title Number 1 Development Cost and Time Variance Factors in Twelve Weapon Programs 14 2 Sxjmmary of Major Risk Studies 23 3 Design and Performance Specifications 91 4 Initial Cost Estimated by Cost Category 92 5 Optimistic, Modal, Pessimistic, and Estimated Mean Program Cost Estimates 93 6 Optimistic, Modal, Pessimistic, and Estimated Mean Program Schedule Estimates 94 7 Derivation of a^ 100 8 Derivation of s! 101 9 Sensitivity Analysis for ^^^ and /^m 103 vxi

PAGE 9

LIST OF FIGURES Figure Nximber 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Title Risk-End Item Matrix Risk Curves Theoretical Limit Curve Hypothetical Limit for S,p State of the Art Advance Procedure Frequency Distribution from Cost Estimates Frequency Distribution from Time Estimates Cxamulative Frequency Distribution from Cost or Time Distribution Risk Index Methodology Cumulative Probability Measures of Program Success for Dollars and Time Using NAM Time-Cost Tradeoff Curve Attainment Profiles Attainment Profile Analysis Attainment Profile Procedure Contract Type Vs. Program Risk Cumulative Distribution Function Risk Vs, RDT6cE Budget Cumulative Distribution Function Risk Vs. Total Program Budget Cxiraulative Distribution Function Risk Vs. Program Length Page Number 1921 29 32 35 44 45 47 54 57 67 76 78 80 86 95 96 97 Vlll

PAGE 10

Figure Page Number Title Number 19 Cumulative Distribution Function Risk Vs. Total Program Length / 98 20 Relationships Among Schedule, Budget, and Risk 105 xx

PAGE 11

CHAPTER I THE MEASUREMENT OF RISK Introduction It is very difficult to force the aerospace industry in the United States into the generalized models used in economic analysis. The demand is monopsonistic , with the Federal Government accounting for over 90 percent of the purchases from many aerospace firms; however, some competition does exist among the various branches of the military and betv^een the Department of Defense and the National Aeronautics and Space Administration. The supply side presents an even more difficult analytical problem. Investigation reveals that the top five companies receive less than one-fourth of the total dollar value of prime contracts; the top 25 contractors receive approximately one-half; and the top . 100 receive less than three-fourths of the total (1) . These figures are indicative of the large number of firms in the industry. Entry is restricted, however, by the extremely large capital requirements. In a typical program, the customer will issue a Request for Proposal (PvFP) or a Request for Quotation (RFQ) to 1 -

PAGE 12

2 a selected group of contractors. VJhile it is possible for a contractor who vas not asked to bid to submit an unsolicited proposal, the normal procedure is for the bidding to be restricted to those firms selected b}'the Department of Defense (DOD) .Among these few firms a strong competition begins which will .-ventually culminate in one of them being selected as prime system contractor. Since only one contract is usually awarded for a given weapon system, the competition must take place before the order is placed. Although a losing bidder may occasionally become a subcontractor to the prime contractor for a subassembly of the system, success lies in winning the prime contract. A complicating factor is the prevalence of nonprice competition. For eicample, Boeing submitted a significantly lower final bid on the TFX (F-lll) program than did General Dynamics; but the latter won the contract on the basis that Secretary of Defense McNamara considered their cost figures "more realistic." This points up still another problem area the fact that firms will deliberately underquote costs in a proposal for the research and development program '•"A notable exception to this rule is MacDonald Aircraft Corporation's highly successful F-4 aircraft, v;hich resulted from an unsolicited proposal.

PAGE 13

3 and purposely take a loss in order to secure the business, safe in the knowledge that these losses can be recouped byoverstating costs in the production program which will follow.* Practices such as these imply that there is a large element of risk in the aerospace industry. Perhaps the first sort of risk which comes to mind is the risk that the firm will not secure the contract. This is a competitive risk which must be borne by every firm in the marketplace economy and will not be treated in this discussion. The questions addressed here are the more basic ones of whether the firm can afford to bid at all on a program; after the decision to bid, what is the lowest realistic figure to bid; and, after the contract has been secured, what are t;he chances of successful completion of the program. The remainder of this study will be devoted to an investigation of these aspects of aerospace contracting. Types of Risk in Aerospace Contracting The most familiar treatment of the subject of risk is probably Frank H. Knight's Risk, Uncertainty, and Profit (2). *Similar results may be observed in some cases, even though the contractor submits his bids in good faith, through the operation of what has been referred to as an "optimistic bias" on the part of estimators. This phenomenon will , be mentioned again in Chapter IV.

PAGE 14

4 Knight differentiates betvjeen risk, vi'hich. he defines as an event which has a known a prio ri probability of occurrence , and uncertainty, which is the occurrence of total!}' unpredictable events. If a contingency is insurable, it is risk; if not, it is uncertainty. Knight's thesis is that since risk is predictable it is not a rationale for (economic) profit but that the bearing of uncertainty is a legitimate argument for profits. In the aerospace industry, the government assumes all or a portion of this uncertainty when it negotiates a cost-plusfixedfee or a cost-plusincentivefee contract. There are other types of "risk" involved in the operation of an aerospace firm. The stockholder fee'J.s that he assumes a type of risk by purchasing shares of the company's stock. The firm assumes a type of risk when it invests the stockholders' funds in fixed plant and equipment or in "basic research or in development of a product for which the potential customer has not yet issued a firm contract. Risk is assumed by both parties when a contract is signed: normally a product of acceptable quality must be produced by a deadline and V7ithin prescr5.bed cost limits. If tlie contractor does not deliver, he runs the risk of financial loss and loss of good name V7ith the customer; the customer runs the risk of being unable to meet the military requirements of

PAGE 15

5 the nation adequately if the contractor is late or the quality is substandard, Another concept, attainment risk, is the uncertainty of obtai'.ing an acceptable production result in terms of prestate 1 specifications.'* It is possible to break attainment risk ;'nto several component risks. Schedule risk, for example, : s the uncertainty of completing a project before a prescribed deadline; cost risk is the uncertainty of completing a project V7ithin predetermined budgetary restraints; performance risk, or quality risk, reflects the uncertainty of meeting the physical performance and reliability specifications set forth in the contract. The recent trend toward incentive contracts represents an attempt by the government to reduce attainment risk by inducing contractors to meet or surpass contractual goals. Hagen (4) concludes that fixedfee contracts are never Pare tooptimal and can, in a wide range of cases, be replaced by incentivefee contracts, V7hich V7ill increase the utility of both parties. There are other significant forms of risk that can be generally classed as prestige risks. The success or failure ^General Electric 's Risk Appraisal of Programs System (RAPS) defines risk in this v/ay: "Risk ... can be defined as the probability that the work being done will miss the triple target of cost, delivery schedule, or technical performance." (3)

PAGE 16

6 of United States space projects is a determinant of public opinion in many p^irts of the v7orld. Since some nations rely heavily on the purchase of American military machinery for their defensive capabilities, the success or failure of American programs can have international ivaplications. Thus J the failure or cancellation of a major project could materially affect United States prestige abroad. Prestige risk can also be an important factor V7ithin the industry since a firia's reputation can be a significant asset or liability. Sales to the public are dependent in part on company prestige. Even if the company sells exclusively to the government, prestige or reputation may be the deciding factor in contract av.'ards. The "Department of Defense Evaluation of the Performance of Major Contractors" (5) is a publication which quantifies company prestige in the eyes of the major customer of the aerospace industry. This rating can be an important determinant of a company's position in the industry. The reputation of a company can also affect hiring policies and other nonsales areas. Prestige risk can therefore be a significant determinant in program evaluation. These three major types of risk are highly interrelated. The failure of a contractor to produce a product of acceptable quality can involve economic and prestige losses to the

PAGE 17

_ 7 company. The customer can also incur economic and prestige ' losses by being forced to pay more than expected for the product or by being forced to release an inferior system to allied countries. The allocation of risk betv.-'cen the two parties will be determined primarily by the type of contract. The Federal Government has assumed much of the risk in research and development programs. Pvecent years have seen a trend toward the incentive type contract for programs V7hich will produce a physical piece of hardware, which indicates an implicit recognition of the risks associated with aerospace programs and a desire to share the risks between the contractor and the customer. Hagen shows that the incentive contract can result in greater utility for both contractor and customer through more equitable risk distribution. This brief discussion suggests that the ability to measure risk v/ould increase the ability of the program planner to choose the specific contractual terms which V70uld result in the best possible allocation of the risk associated with a program. In the sections which follow, specific benefits of a risk measurement will be suggested and some of the attempts v.-hich others have made at risk measurement V7ill be reviewed.

PAGE 18

8 T he Need for Quantitative R isk Me asurement If After considerable study and discussion with experienced aerospace planners, it was decided that soms: type of attainment risk measure would be the best possible device for evaluating the various types of risk which are associated with aerospace programs. It could explicitly consider financial attainment, quality or performance attainment, and time or schedule attainment; it implicitly considers prestige risk by assuming that it is a function of those types of risk which are considered explicitly. A risk measure which could assign a more specific risk classification to a project could be used to reevaluate program cost and performance in more than general terms. A major portion of the $490 million loss sustained by General Dynamics on its 880-990 commercial jet program could have been saved if adequate assessment of the budget, schedule, and performance risks of the program had been made at the outset, or at any of several points during the program. For an excellent discussion of this program and the reasons for the losses, see "Hov/ a Great Corporation Got Out of Control" (6) and "GD : The Hard Road Back from the Brink" (7). Another case in point is the Skybolt missile program. VJhen this program was cancelled in 1963, the program costs

PAGE 19

. 9 had risen from 1960 estimates of $893 million to almost $3 billion, p.nd the program V7as far behind schedule (8) . Over $150 million was considered to be an unsalvageable loss (9) . If Skybolt had been reevaluated by an accurate risk measure v.-'hile still under feasibility analysis, the project could have been altered considerably. The congressional controversy could have occurred at the tirae of the initial appropriation and not after millions of dollars had been v/asted. The Skybolt program could have been cancelled, postponed, or accepted under revised cost and schedule estimates; if the project were accepted, funding levels would have been more realistic. If the cancellation had come early in the development, a considerable amount of time and money would have been saved; and Great Britain, v.liich had planned to purchase the missile system, could have redefined its iTiilitary program at a more convenient time. This is just one example of a situation where failure to adjust cost and schedule estimates with full consideration of the risk involved led eventually to cancellation of the program. Both government and industry have wasted valuable resources when projects were cancelled or extensively redefined late in the program. Consequently, both governm^ent and industry would benefit from a risk measurement which could quantify program risks during the early phases of

PAGE 20

10 development. Recent years have seen real advances in the accuracy of estimating techniques. Unfortunately, there ha; been an even greater increase in the requireip.ent for still more sophisticated techniqiies and better estimating procedures. Better estimating methods incorporating risk adjusting procedures are becoming even more important in modern contracting and planning in the aerospace industry. Past A ttempts at Risk Measurement Introduction Since the concept of risk and its role in decision making in the aerospace industry has been treated before, this section will be a survey of previous studies of risk in aerospace programs. An extensive search of the current 3.iterature has disclosed very little documentation of concepts or methodologies for the generation of a risk index. Several studies, however, have considered some form of risk or uncertainty. Among these are Alchian (10) ; Marshall and Heckling (11); Peck and Scherer (12); Polski, Clausen, and Paige (13) ; Summers (14) ; and Terrell (15) . The purpose of this section V7ill be to relate pertinent conclusions and similarities among the above-mentioned studies to appropriate areas investigated in this report.

PAGE 21

11 . In 1950, A. A. Alchian of RAKD Corporation published a study of the variabilit}'' of actual costs frora cost estiA mates. The reliability of estiraates froni four sources V7as studied. The four types of estimates with the average error for the trade journal data studied were as follows: (1) Engineering Estimates ±25% (2) Cost Estimator's Estimates +23% (3) Public Engineers' Construction Cost Estimators ±167. (4) Contractors' Bid Estimates ±21% Alchian concluded that the difference between the estimate and the realized true cost for an}' device of an improved quality v7ould usually be larger than any of the above deviations. The study dealt only v'ith the unreliability of cost estimates, and no technique V7as presented to obtain better cost estimates. The most extensive analysis of risk in weapons acquisition V7as made by Peck and Scherer. They observed that generally weapons program decisions involve four basic elements: the potential enemy threat, cost, the state of the art, and time. Uncertainties in estimating the last three items the technical feasibility, the development time, and the development and production costs are the primary contributing factors of program risk.

PAGE 22

12 Although irf;ny factors were considered qualitatively, no analytical solutions V7ere derived for estiraating any aspects of program success. Peck and Scherer recognize the immense complexity of quantitative risk analysis by saying: The existence of these significant uncertainties plai.nly complicates the use of quantitative analysis in program decisions. Indeed, the complications from uncertainty are so great that our entire approach of applying an optimization model to the program decision can be questioned.* State of the Art Measurement A single facet of this complexity is reflected in the technological advance inherent in a given program. For a comparative analysis of programs to be effective, it is essential that the technological complexities be compared. The methodology employed by Peck and Scherer for evaluating technological difficulty V7as simply a v/ithinsample ranking of paired programs (nine programs being used) by expert opinion. The criterion used was the selection of that weapon system from, each pair which represented the more ambitious state of the art advance (SOA-A) , considering the time period during which it V7as developed. The ranking v^as then arranged on a scale from zero to one hundred units to ^'Reference 12, page 303.

PAGE 23

~ i: compare the relative degrees of difficulty or technological advance. This methodology (the ranking concept) has limited applicability when one tries to measure the technological advance of a new program which is not functionally comparable to any member of the sample. The main problem, is determining a realistic technique for qtiantitative measurement of technological advance. One measure of program success often used is the relative size of overruns* in budgets and schedules encountered at completion. In the analysis of 12 programs, Peck and Scherer determined an average cost overrun of 3.2^'^ and a corresponding development time overrun factor of 1.36. Table 1 shov7s the individual results for the 12 programs. In their study of 22 Air Force weapon system developments, Marshall and Heckling noticed a similar trend in the production cost overruns. This trend was still evident even after the latest cost estimates V7ere deflated by a price index and adjusted for output quantity changes. Cargo and tanker aircraft had the sm.allest average, 1.2; fighters had ^Overrun is the ratio of final actual to initial estimate. ^''^This overrun is considerably larger than the one found by Alchian. The probable reason is that the Peck and Scherer data contained only large programs, while som.e of Alchian 's data were from smaller, less ambitious projects.

PAGE 24

14 Table 1 DEVELOPMENT COST AND TBIE VARIANCE FACTORS ^ IN 12 WEAPONS PROGRAMS''' Program Cost

PAGE 25

15 -• Scherer. For the above programs, Marshall and Heckling determined average cost and schedule overruns of 3.01 and 1.28 respectively. In the development of military weapons, there is a tendency to place more emphasis on meeting the performance capability even if this means sacrificing cost and time schedules. It is noteworthy that in most cases the cost overrun has the larger ratio. Marshall and Heckling do not present a m.ethodology for predicting the final outcome of a program. Hov7ever, the above results indicate that the probability of program overruns increases directly though not necessarily proportionally V7ith the amount of technological advance required by the progrc'm. Peck and Scherer conclude that the unpredictability of a weapon system program manifests itself primarily in the cost dimension due to the premium placed on performance and the tradeoffs among performance, time, and resources. Qua ntit ative Risk Assessment It becomes readily apparent that there exist many forces which affect the outcome of a program, some of which interact in a complex array of tradeoffs, as in the case of scheduling and funding. There are also the parameters which are physically unmeasurable but still qualitatively compared by expert opinion in the appropriate fields.

PAGE 26

<> 16 The first well-known analytical study involving uncertainty was performed by Summers. This was a statistical study to adjust original cost estimates by an analytic function such that the revised value would be closer to the actual outcome. Based on 22 weapon systems and 68 cost estimates, Summers' study found these three variables to be most important : 1. time the estimate is made in relation to the development program 2. degree of technological advance required 3. length of the development period. The result of Summers' analysis is an exponential function, In F = a^ + a^t + a2At + a3A -h a^k'^ + a3L + a^T + u,-v which can also be expressed in the equivalent form, F = Ke^lt ^a2tA ^asA ga4A2 ^35! ^361 ^^.. where the following parameters are defined: K,a = constants, t = a fraction representing the time elapsed in the program when the estimate was made, A = a numerical measure representing the technological advance in the program, L = the length of the development period, T = the calendar year, and u,v = error terms. ^Reference 14, page 33,

PAGE 27

17 Once again the problem of measuring the inherent technological advance of the program arises. The technique used in evaluating A was a vjithinsample ranking determined by RAND engineers. For reasons given previously, difficulties occur in consistently applying this measurem.ent scheme to new programs . In estimating techniques that utilize data of past programs. Summers concludes ...it must be emphasized that the crudeness of the data precluded the possibility of obtaining anything like a precise description of cost estimating errors in the years following VJorld War II..., What is claimed is that the methods employed in this study constitute a reasonable V7ay of looking hard and close at the information available. The merit of the formulation offered here lies in its explicit method of derivation and in the possibility it offers for future refinement V7ith miore complete data and V7ith experience of new kinds of systems and new production techniques.'' To recapitulate, the results of Summers' study aresimilar to those of other investigations reported in this section. These results may be summarized as follov7s : • The use of a subjective ranking by experienced engineers to evaluate the state of the art advance of the program. • The conclusion that the average cost overrun increases with technological difficulty and decreases with respect to how close the program is to completion when the estimate is made. ''Reference 14, page 11.

PAGE 28

18 In 1964, at the AIAA Annual Meeting, Polski, Clausen, and Paige presented a clear and well-defined methodology for reducing risk on research and development (R&D) programs. The purpose of such a methodology is as follov7s: Such a system has benefits for both custoraers and contractors because an increase in knowledge of tVie risks leads to more sound decisions concerning them and general improvements in program management plans, methods, and communications. Greater knowledge of risks may also lead to ultimate improvements in such areas as: • Source Selection • Program Definition • R&D Program Value" Four ranges of risk (1) high, (2) moderate, (3) minor, and (4) low are used in the analysis. A risk-end iteui matrix is generated along with suggestions and costs for lowering the risk in the defined areas. (See Figure 1.) This is the first V7elldefined methodology for utilizing the knowledge of a wide variety of people involved in a program. This iiTiplies that a risk-cost tradeoff matrix could be found. Terrell attempted to unify the areas that contributed most to uncertainty v/ith a more quantitative assessment of risk for programs in the early phases of definition. A methodology is presented which quantitatively compares the technological advance of different programs. This "'Reference 13, page 1.

PAGE 29

19 W3 I c O -U CJ CO 0) >^H U « i-< 3 O C 5-< 4J c ;-< 3 4J O CO y-i 3 en

PAGE 30

20 measurement begins as the ratio of the proposed development item to the best available (developed) item and is then adjusted by use of a theoretical limit assessment curve. This curve takes into consideration the relationship of the desired performance capability to the theoretically possible • performance levels. A procedure is developed for applying standard statistical techniques to the performance characteristics for more than ten programs in order to generate a family of attainment curves. These curves predict the probability of attaining performance, reliability, and schedule with corresponding tradeoffs involving cost. (See Figure 2.) The projects included in Terrell's sample showed an average cost overrun of 2,5 and a schedule overrun of 1.2. These values fall into the general ranges of values determined by previous researchers. In conclusion, with the attainm.ent prediction equations, a specific technique for evaluating the degree of difficulty in a program, and considerations involving the theoretical limit, Terrell's study is the most advanced attempt at the evaluation of a risk index. Summary This chapter indicates something of the variety of concepts which have been analyzed under the name "risk." The

PAGE 31

21 )i$i^Ainisvn3^'^/f) R u CO in 3} o 0) (1) 0) PS >tSI>f30m%'Od>}3J-^/fi Ja ^9l^3?n(33H09~^^

PAGE 32

-. 22 need for a quantitative procedure for the measurement of attainment risk is established, and a survey of the v^ork to date is presented. A summary of the contributions of the principal studies in th J area of risk is given in Table 2. The evolutionary chara -ter of the development and the sources of some of the conce" ts presented in this study are indicated. Included are the primary areas of interest, functional relationships developed, and a brief summary of the important conclusions of each study.

PAGE 33

23 -

PAGE 34

CHAPTER II STATE OF THE ART ADVANCE (SOA-A) MEASOREMENT Introduction In almost all of the work which has been done on the problem of risk,* the researchers have concluded that the most important single parameter in the assessment of Risk is the advancement in the so-called state of the art, or the increase in the level of technology which is required to accomplish the project under consideration. Since this study deals primarily with research and development program.s, it is intuitively evident that projects which require greater technological advances will generally involve greater risks. Therefore, if a Risk index is to be developed, one should first address the problem of deriving a methodology for the assessment of the state of the art advance (SOA-A) associated with a project. It is important that the system developed be logically consistent and that ^Throughout the development which follows, the capitalized "Risk" will refer to the specific concept of attainment risk as discussed in the first chapter of this study, with the low^r case "risk" referring to the more general concept. 24 -

PAGE 35

it be suf fic5.ently general to be applicable to a wide range of program types V7hile at the same time requiring a common methodology and thereby assuring a measure of consistency in SOA-A measurements made by different analysts on different projects . Peck and Scherer indicate that what they call "State of the Art Index" measures "the technical achievement and innovation required to accomplish a quality (performance and reliability) objective."" They place a project on a scale which runs from zero to one hundred and assign it a number relative to the other (historical) projects already placed upon the sawe scale. This approach is based upon the experience and judgment of those in the field and may be referred to as a completely intuitive approach. An alternate approach would be one in V7hich only quantitative factors are considered; experience, judgment, and intuition would not play an im.portant role in the synthesis of a value for the SOA-A of a project. In the first place, sufficient data are not available to permit this type of analysis; in the second, even with perfect data it v7ould still possess certain shortcomings since V7hat this type of analysis gains in objectivity, it forfeits in the loss of the invaluable qualities of experience and judgment. "'Reference 12, page 289.

PAGE 36

26 While the approach presented here will not be completely objective, in that it makes some use of subjective or intuitive judgments, it attempts to force the analyst to quantify at a lov/er level and to follow a set procedure for determining the SOA-A of a project. This may be called a subjective-analytic approach . Deve lopment Given an advanced project proposal, the most important performance criteria (e.g., speed, weight, accuracy, thrust, and range) are listed in order of decreasing importance (xi...Xj^), and each criterion is given a v/eight which indicates its importance to the success of the program relative to the other criteria (ai...aj^). Each successive weight is less than or equal to the previous one, and all are relative to the most important criterion (xj^) . «1 =^ 1' -i-1 -^i^i-1-1 Factors which should be analyzed in the evaluation of the aj^ include expected development cost, expected development time, and importance to the success of the project. Normally, a subsystem which is expected to account for 50 percent of the total project cost will be m.ore heavily weighted in determining a Risk measure than will one vjhich is estimated at 10 percent of the total. Similar reasoning

PAGE 37

27 leads to the same conclusion about articles v.'ith different expected development times. The third factor depends upon the relative value the customer places upon the various features of the project. There are a few major design features v;hich are of extreme importance in a project. These are the reasons the project was deemed necessary in the first place „ In addition, there vjill be other correlative or subordinate "extra" features of the "nice to have" variety. In a squeeze to produce an acceptable product by a deadline, these secondary measures may be compromised much more readily than the primary design features. The experienced analyst can pinpoint these distinctions and use this information to assist in the ranking of criteria and assigning of the aj^. The criteria having been selected and v/eighted relative to each other, the next step is to determine the SOA~A represented by each of the criteria on the list. Let the SOA~A for X£ be designated by Sj^, where Sj^ is defined to be the ratio of the desired performance level to the performance capability of the most advanced currently available (developed) model. Sa = desi red perfor mance level ^ current demonstrated capability In cases where an increase in performance results in a

PAGE 38

28 decrease (e.g., weight, volume, resolution), simply invert the ratio to obtain S., At this point in the development, a set of numbers has been computed that indicates the relative magnitudes of the performance increases that will be required by the proposed system. Now it becomes desirable to adjust these performance ratios by taking theoretical limits into consideration. As a technological breakthrough occurs, new performance capabilities are realized; but a given level of technological capability can be exploited only so far. The principle of diminishing marginal returns begins to operate; and increasingly greater amounts of time, money, and other resources are required for additional performance advances.* The models-accounts for this fact by adjusting the S^ for theoretical limit considerations. An adjusted value for S^ (S.) can be obtained by referring to the theoretical limit curve of Figure 3. The curve is used by estimating the *At the 13th annual meeting of the American Astronautical Society, Edv/ard N. Hall presented a paper on An Economical Approach to Space Transportation (16) , in which he demonstrates an exponential relationship between the ratio of actual final program cost to initial estimates of program cost and the change (or advance) in the SOA which is sought. This supports the author's choice of the exponential form for the T curve of Figure 3. ^'^The curve in Figure 3 was postulated by the author after analyzing Terrell's curve and discussing the subject at great length with experienced aerospace planners. It should be determined empirically as more data become available.

PAGE 39

29 .10 .08 .06 04. ^ .03 .02. ,01 25 50 75 100 Percent of Theoretical Limit >Figure 3 THEORETICAL LIMIT CURVE

PAGE 40

30 degree to which the present level of technological development has been exploited and the approximate percent of the theoretical limit which has been attained. Entering this value on the abscissa, the corresponding value of T^ can be found oh the ordinate. IT is a dimensionless number which indicates the technological difficulty of the project. This T curve is determined empirically by assigning S^ values to criteria in historical programs and determining the corresponding values for the V-. The curve may be revised from time to time. It may also be necessary to define different curves for different classes of programs. The value of S. is obtained by multiplying S^ by T^. I Thus, a set of values for the S. has been established which takes theoretical limit considerations into account. Finally, t the individual S^ measures are combined in a weighted summation in order to arrive at a measure of the SOA-A of the total system (SJ) . Mathematically: i=l Obviously, the series S-j. will continue to increase as the value of n increases. It therefore becomes necessary to demonstrate that in practice S^ will tend to approach a limiting value. Study of past programs reveals that values for S^

PAGE 41

31 ^tend to lie between one and five; and, in most areas, a value of three would indicate a large increase.* It is difficult to imagine an aircraft project v/hich would require an increase in speed of more than twice the present capability or one which would require a threefold increase in engine thrust. Since S. is the product of S. and T, its value can be 1 *^ 11' made large by inserting a large value for either S. or Tj^. If T. is large, the technological difficulty is great; and the relative amounts of time and dollars required will normally be large; therefore, one should expect a high value for a.. Since large values for T. indicate reduced cost-effectiveness, it would not be expected that unimportant criteria would be designed to require values approaching the theoretical limit. On.the other hand, it is possible to have large values of Saccompanied by small values of T(for example, before a technological breakthrough has been exploited) . Any criterion which requires a large performance increase should rank high in importance to the project and will therefore carry a relatively high a. . *Notable exceptions to this idea are found in the fields of electronics and computers. Recent advances, such as integrated circuits, have given rise to quantum increases in speed and decreases in weight and volume. The assumed relationships among high values for S-i_,T'i, S^, and atend to minimize the existence of these exceptions as an objection to the hypothesis that Sy approaches a limiting value.

PAGE 42

32 aiiis s^ ^^^^smm^ m:-\ IIP CO — ^ 2 n Figure 4 HYPOTHETICAL LIMIT FOR Sr

PAGE 43

33 As indicated above, if either S^ or V is large, the product will tend to be large also. Since large values of either S. orT. tend to be associated with large a^values, 11 ^ it may be concluded that S. will be positively correlated with a.. Following this line of thought, then as i increases. t a., either T. or S., and therefore S^ will decrease, yielding a relationship between Sm and n which has the form of Figure A, where SZ is the hypothesized limiting value of S„. Analysis of over twenty real and hypothetical programs, ranging from the very simple to the extremely ambitious in technological complexity, indicates that the expected range of values for S,^ is i S* ^ 1.2 . In g^eral, it is the intention of the author that the value of n should be less than ten. Beyond this value, the relationships among the a. are virtually impossible to assess with any degree of accuracy. The procedure is designad to permit comparisons among similar programs, and it is the intention of the author that only the major performance measures be included in the list. It is expected that n will tend to have a value of four to six in most instances. Since the methodology is applicable to a wide range of sorts and sizes of projects, it can be used to evaluate a complete system, then be reapplied to the more important subsystems.

PAGE 44

34 Summary A review of the methodology for generating the state of the art advance (SOA-A) for a project would be instructive at this point. The steps to be followed (see Figure 5) are these: (1) Select the performance characteristics which are most important to the system and rank these from most to least important. (2) Assign numbers (a.) to each of these criteria, indicating their relative importance to the overall project, a-, = 1; ^±-i^ ^i~ ^i+1 (3) Calculate the S. as the ratio of expected performance to the best currently available model performance, inverting the ratio in cases where improved performance results in a small numerical value for the index. (4) Estimate the percent of the theoretical limit which the desired performance approaches. Using the Theoretical Limit Curve, find the appropriate values for T. . (5) Adjust the S. to account for theoretical limit considerations. S. = ^j^S.

PAGE 45

35 Prograin Performance Requirements Selection of Performance Criteria (X^) Data from C ther Pre grams Determination of Best Available Perforiaance Theoretical Lirait Derivations Ranking by Relative Importance and Assignment VJeights (aj[)

PAGE 46

36 (6) Combine the S^ in a weighted summation to obtain a measure of the overall SOA-A for the total project. S . l a, s! 1 = 1 This procedure permits the use of subjective judgment on the part of the analyst in assigning the weights to the criteria and in the selection of the criteria themselves. It requires quantification at a level below the total system level, however, and thus forces the analyst to examine critically the actual makeup of the final composite SOA-A measure. This procedure combines the stronger features of both the completely intuitive approach and the wholly analytical method, resulting in a balanced analysis, which will tend to give more consistent SOA-A measurements in the long run.

PAGE 47

CHAPTER III A RISK INDEX METHODOLOGY Selection of Parameters SOA-At (S^) In the preceding chapter, it was pointed out that the state of the art advance required by a project is the most important single factor in the determination of an overall Risk index. All other factors held constant, the Risk associated with a program would increase if the SOA-A increases. <^In the absence of any information about funding and time schedules, the SOA-A would be a reasonable measure of the inherent Risk of a program. This will be one of the factors used to develop the Risk index. Performance Performance attainment is a vital measure of program success. Detailed consideration was given this variable during the course of this study. In the final analysis, it was decided that performance would not be used explicitly in the procedure. The reasons for this decision were these: 37 -

PAGE 48

38 (1) S^ measures the inherent technological difficulty of meeting performance goals. To also include a measure of the probability of meeting these goals would be a form of double counting. (2) Historical data refer primarily to programs which have met (or substantially completed) the stated performance goals . (3) A program is not considered to be finished until the performance criteria have been met to the satisfaction of the procurer. (4) \7hile there are some instances in which projects have been declared to be completed when not all of the performance goals were met, these are the exceptions, and the programs were in all likelihood not considered successful by either contractor or purchasing agency. (5) If a project is completed before the time and cost constraints have been encountered, the contractor would tend to add additional capability, but only to the point of using up the remaining resources. Since many projects overrun these resource contraints, it seems improbable that contractors would voluntarily exceed the required performance goals, and that the overruns would be small.

PAGE 49

= 39 (6) There does not appear to be a systematic method for measuring the cost and time overruns for a program which is cancelled. In light of these facts, it was decided to consider the meeting of the performance goals (or substantial compliance) to be a constant factor at 100 percent of the contract specifications and that this would be a benchmark from which the other variables would be measured. In other words, the total program cost (actual) is the cost up to the point when a satisfactory product has been delivered to the customer, and this date will mark the completion of the project in time, as well. Program Cos t Program cost is a tangible, measurable resource constraint in any program. It is a readily available item of data on historical programs and is an objectively measurable quantity. Virtually every attempt at risk measurement has included cost as a factor. Any discussion of Risk must ultimately be expressed in terms of dollars in order to be useful to the decision maker, since any discussion of Risk by managers, program planners, or legislators will have economic overtones. Therefore, cost is one of the variables which will be used to determine the Risk index.

PAGE 50

40 Time To Government planners, considerations of time are second in importance only to those of cost. To the military commander, time is the overriding consideration, since the • ultimate x^eapon delivered a day (or an hour I) too late can be totally useless. In planning for the defense of the nation, the meeting of delivery schedules is vital. Since the 1950 's, this country has been engaged in a vigorous competition to overcome and eliminate various "gaps" (missile gap, booster gap) and to out-perform the Soviets in other ways (the "Moon Race") . In each instance, time was the next most important criterion after performance and reliability, and cost goals were repeatedly revised upward in order to meet and surpass time objectives. Time has therefore been chosen as the third variable which will be utilized in the derivation of an index of Risk. Other Considerations As has been indicated above, some previous studies have used measures other than the three selected here. Since the approach used here will be to build a set of logical relationships, the three variables which best describe the program have been chosen. In many of the previous studies cited, the approach used was to fit hypothesized functional relationships to historical data by use of a least squares

PAGE 51

41 routine. While this approach gives the best estimates over the sample, it does not, in general, give the best estimates for programs not included in the sample. Since this study attempts to develop a system based on logic rather than empirical fit, these three basic factors were selected. It should be emphasized that almost all of the prior studies relied upon these three as inputs, even though in many instances other measures were added to give a better empirical fit to the data. In the first issue of Aerospace Management , General Electric Company's Missile and Space Division presents a summary of their Program Appraisal and Revision (PAR) system which lists the most important variables as "Technical Performance, Schedule Performance, and Cost Performance" (17) . Reasons for Subjective Assessment The chapter of this report dealing with the method for determining the SOA-A of a project indicated the principal advantages realized by a system which permits the experienced analyst to make use of his training and to bring his expert judgement to bear upon the problem. It was indicated there that, while data collection for a purely mechanical system would be a virtually impossible task, even the availability of perfect data would not make such

PAGE 52

42 an approach attractive since it precludes the use of the invaluable qualities of experience and judgment on the part of the senior analyst. Consequently, the development of the Risk index begins with the generation of two sets of estimates one for program cost, the other for development time. It is assumed here that these estimates are derived using the best currently available cost and schedule estimating procedures, and that they are consistent with the other estimates which would normally be made for the program. Risk Index Procedure The development of the index is begun in a manner similar to the PERT methodology. Given the perform.ance requirements of a program, a set of three estimates is prepared for each of the input variables (dollars and time) indicating the expected values of resources required to meet the performance goals: * (1) Low an optimistic value which has a probability of only 1 percent of actually being met, even if all goes well and no major delays or cost increases occur. (2) High a pessimistic value which has only a 1 percent chance of not being met. (3) Modal an estimate of the most likely value, taking into consideration past projects of a similar nature.

PAGE 53

43 Using these three estimates, probability (of program success) distributions can be constructed for development time and cost. Following the PERT analogy, the Beta distribution was selected." Figure 6 and Figure 7 are representative of possible forms for such distributions. The preponderance of opinion seems to indicate that the rightskex^/ed form of Figure 7 is the more likely one for both schedule and funding; the possibility does exist, however, that some projects may be better described by the skewed left or even the symmetrical distribution. This possibility is one of the reasons for the selection of the versatile Beta distribution. Superimposed on these distributions, the actual values of the contractual (or tentative) values for time and funding (t and $ ) are plotted. The shaded areas under the curves o o to the left of these resource restrictions represent estimates of the probabilities of development 'time and cost success for the program ( ]^jand |^g) ; that is, the probabilities that the project will be completed within the assumed budgetary and schedule restraints. *For a thorough discussion of the PERT assumptions and their implications, see Ko R. McCrimraon and C. A. Ryavec, An Analytical Study of the PEPvT Assumptions (18) .

PAGE 54

44 A) 10 o o u to o u CM VD •H CO H H CO P4 H CO o CJ 53 O M ra H CO M u Xouonbsa^

PAGE 55

45 60 •H o 6^ o 12 O I CO H

PAGE 56

46 In most cases the modal estimate will be either the best available estimate or the contractual figure, since the person performing the analysis will base his estimates upon information from the group v/hich generated the contractual figures o Even though the value for $^ or t may fall on or near the modal value of the frequency distribution, the shaded area (the value of }l/^ ox ]l/ ,^) v/ill be a functrion of the distribution form, and vrill therefore differ from one distribution (and project) to the next. It is often of interest to determine the effect V7hich a change in funding or time from the expected values vjill have upon the probability of program success. As will be shoxv^n presently, a sensitivity analysis permits calculations of this sort. If a sensitivity anal^'^sis is to be attempted or if there are several alternative values for $ or t , the generation o o of a cumulative distribution function from the frequency distribution (see Figure 8) permits a much faster analysis while at the same time making the model more easily understood. From such a function the value of W^ or V/^ can be read directly from the ordinate for any value of $ or t^. The probability of two independent events both occuring is the product of the probabilities that each will occur. P(both A and B) = P(A) • F(B) While the events considered here (schedule success and budget

PAGE 57

47 A o a 3 XT (U 5-1 Low 10% Modal $o 907o High or to Development Tiine or Cost -— — > A 1 -^ >^ o c •H to I .0 .9 .8 .6 .4 .2 .1 : j . , , •

PAGE 58

48 success) are clearly not independent, the present state of development in Risk analysis is sufficiently crude to permit this simplifying assumption in the analysis. Among others, Terrell made a similar assumption with reasonable success. Relying upon past empirical evidence as a partial justification for the simplifying assumption, the product of the probability of cost success (]^$) and the probability of time success ( ]^^) shall be used to represent the unadjusted probability of success for the total program (V^-n) * The unadjusted probability of failure for the project (^ yd is the complement of )^jj. It was indicated in Chapter II that the value of the SOA-A measure (S„) would normally fall into the range ^ Srp ^ 1.2 . Since it is intended that the estimate of the experts shall be the base figure which is adjusted for SOA-A, the unadjusted probability of failure will be multiplied by 1 + Srr, in order to adjust the probability of failure in light of the expected technological difficulty of the project. The existence of an "optimistic bias" in defense contractor estimates has been discussed in a previous section of this study. This final adjustment attempts to account for such bias by increasing the probability of failure by an amount

PAGE 59

49 which is dependent upon the technological advance sought. This would appear logical since if no technological advance is sought, the estimates should be accurate as given, while if a large advance is sought, the probability of failure can more than double. Once the Risk index has been generated, it becomes desirable to put it into its most meaningful and useful form. The concept of Risk can be thought of in two ways: one of which is negative (;Z^^, the probability of program failure), and the other positive (^rr,, the probability of program success). In general, a statement of the sort "The probability that this system can be produced for X dollars in Y months is 0.63," would be preferred to the corresponding statement of the probability of failure. Obviously, the two measures are complementary; that is, }^^ = 1 ;2^,j, = 1 -(1?^y) (1+S^) . If the value ofjKj, is negative (for example, if^^ or yr = 0^ j^ = -Srp), it should be considered to be zero. Since no advanced program can have a negative value for S^, ^ ^ is always positive. This places the following limits on the indices: £ ^^ ^ 1.0; £: /^ ^ 1.0 . The Risk index is designed to support, not replace, the decision maker. Like any tool, it should be used with common sense and good judgment. A small value for >^ '^^^$ should normally eliminate a project from consideration, regardless f its y^ value.

PAGE 60

50 Sensitivity Analysis If the Success index C^™) and the Risk index ( ^rp) are expressed in terms of the original inputs, it can be seen that, while the indices may be imperfect, they do respond in a quite reasonable manner to changes in the input parameters, Jr^ = 1-(1f^ f^) (1 + S^) ^^ = (1j^^ ^$) (1 + S^)= 1-^^ ^ ]^^ ^ 1.0 ; :^ ^^ i^ 1.0 ; £ S^ :^ 1.2 Consider the change in ]^ for the following: (1) An increase in $ : An increase in $ leads to an increase in f^^ o ? This increase in 1^ ^ leads to a decrease in S + ^;^^^ -A (1 ^^^-J^^Xl + S^) = -A)Z5^ . A decrease in S implies a corresponding increase in ^T • (2) A decrease in t : The decrease in t leads to a decrease in 2^^ o ' c which in turn leads to an increase in p or a corresponding decrease in ^

PAGE 61

51 (3) An increase in S„: I If a change in configuration or in one of the system requirements occurs which causes S„ to increase, this will cause ^ to increase and)^ to decrease by the same amount. +As^-^+A(i ]^^r^)a + s^) = +^^^ = -A]^ From the magnitudes of the changes in „ brought about by changes in the resource constraints ($ and t ), the analyst can estimate \^J^ and '\ i ^ . These estimates may c) $o ^ to be valid over greater or lesser ranges, depending upon the location of the points chosen along the cumulative distribution function of Figure 8. As a rule of thumb, the middle third is usually fairly linear while the ends are definitely nonlinear. If he can then determine the dollar value of a schedule decrease in the program (for example, if he is told that the Air Force will pay an additional $2 million for a three-month speed up in the time table) , he may determine the resultant impact upon program Risk of these resource constraint changes. l£ it is desired to determined the impact on Risk of a three-month reduction in program length and a $2 million budget increase, he must find /i^T=4rT|4t^ = .3„„„,hs ^^^T I ^^^ , + ^2 million " If the Risk is reduced by decreasing time and increasing money by a corresponding amount, this indicates that the

PAGE 62

52 " schedule may profitably be reduced. While thlr. aij.alysis will not necessarily give the optimal mix of time and funding for a project, it v/ill 3,ndicnte the proper direction for improvement. Efficient strategies can be determined l.^y varying the values of $ and t^ until the dollar equivalent of a schedule change produces an equilibriujn of tho form This strategy ir efficient because the amount the customer is V7illing to increase the funding for a decrease in schedule and the time extension he is willing to grant for a decrease ill cost are not sufficient to reduce the Risk associated V73 th the program. The values which are obtained foi* tlie savi-itivities of y^'r, to changes in resource restrictions \'ill have meaning oo.ly over limited ranges. If significant changes are contemplated, sensitivities should be recoraputed for the pioposed values of $^ and ty. S u mma r y an d C on c 1 u s ijDn This chapter presents a discussion of the reasons behind the selection of input parameters and the decision to use a partially subjective evaluation of the hisk inde::. The steps followed in the development of the index (sc.-: Figure 9) were; (.1) Given the performance requirements of a system, generate lov?, high, and modal values for t and $.

PAGE 63

53 u (2) Using thesa estimates, ccxistruct the probability (oi: program succesr) disto-'ibutions for $ aiid t. (3) Cu theae distributions, plot $y and t^^ and calculate the areas' to the left of these constraints ( y^A and l^j.) . (Ultimately, the cumulative distribution function of Figure 8 ma}' be generated and the probabilities read directly from the ordinate.) (4) Calculate the unadjusted probability of success (5) Adjust the probability of failure for state of the art advance ^^, (1 ~ ;^]j) (+ ^-p) • (6) To sta-:e the Risk dn more p'^citive an take the complement of p ^^ V7hich is the probability of total program success. yr^ 1 " (1 •^f\r^){i -h s^) (7) To apply the indeX; generate the sensitivities. Ar-i At and A]^v o -4$ o (8) Determine efficient allocation where A'j^T -1A t. ^n TA o The index derived here is not an ultimate index iii any sense. It is an attempt to derive e lor:,ically consistent methodology for generating a Risk i-ade^r vhich has some real meaning a-nrl can find realistic useful:. oto in decision i.vnking Before the inde:-; can be used V7ith any degree of confidence.

PAGE 64

54 Performance Requirements Distributions of Time and Dollcir A 1 1 a inme n t Probabilities

PAGE 65

55 it must be validated by application to real problems. Although it is as yet an untried tool, the methodology developed here represents a step forward toward a measure which has physical meaning (probability of program success or failure) . Additional research and a broader, more accurate objective data base will refine or revise the technique suggested here into an even more practicable and reliable tool.

PAGE 66

. CHAPTER IV REFINEMENTS Revision of the Present Model In cases where the program is sufficiently well defined, alternate methods for the generation of the cumulative distribution form may provide a more solid base for the derivation of the Risk index. One such method involves the use of concepts developed by Faucett, Henry, and Wilson (19) in their Netv>7ork Analysis Model (NAM) . Their model requires that a Program Evaluation and Review Technique (PERT) type networkbe constructed and time or dollar estimates made for each activity. A computerized model then makes many passes through the network in a random manner and generates a cumulative distribution function for the probability of completion versus time or funding. If the program is defined in sufficient detail to permit this type analysis, the resulting distribution may give a more accurate picture of the relationship which actually exists between cumulative probability of success and the levels of funding and schedule. (See Figure 10.) A second method which may find application in reducing subjectivity is the technique known as the Graphical Evaluation 56 -

PAGE 67

57 1 '

PAGE 68

58 and Review Technique (GERT) (20) . While NAM is a simulation, GERT is an analytical technique in which topological analysis permits the generation of the moments of the distribution which describes the network representing the program. If the form of the distribution can be recognized from these moments, then the actual distribution can be generated from the moments. Further study is necessary in order to determine the applicability of this method to the Risk index methodology, but the approach appears to have promise. Another technique, which may be used on projects which are not defined in great detail, as well as the more clearly detailed cases, is the one presented by Sobel (21). He shows that while the PERT technique assumes a scaled Beta distribution, the ^specification of one additional piece of information (the 80 percent central range) by the analyst permits much greater freedom in the form of the frequency distribution and in general results in distributions which more accurately describe the actual function. Computer programs are developed which analyze and quantify the uncertainty of cost estimates. While Sobel 's analysis is stated only in terms of cost, it could be applied equally well to time distributions. Additional effort in this area would enhance the validity of the Risk index developed here.

PAGE 69

59 Two recently developed decision making techniques may lend themselves to incorporation into the sections of the Risk procedure which require subjective analysis. These are DELPHI*, developed by the RAND Corporation, and PATTERN^'^, " developed by Honeywell's Military Products Group. The two methods are very similar in that they both center around reevaluation of decisions upon receipt of additional information (the opinions of other experts on the same problem). The basic difference is that DELPHI forbids discussion between the participants, while PATTERN encourages it. Both methods claim to have demonstrated that in most cases consensus answers can be reached after several iterations. Techniques such as these may find application in the subjective portions of the Risk index,-procedure. Possible Measures This segment of the study lists some other potential refinements of the Risk index which would be useful in the *Developed under United States Government contract, this technique derives its name from "Project DELPHI", which'began in the early 1950 's at RAND. The initial use was to determine vulnerability of the United States to nuclear attack. It has since been refined and is being promoted as a management planning tool. Planning Assistance through Technical Evaluation of Relevance Numbers. Developed by Honeyv/ell to aid in ranking proposals in terms of relative attractiveness for future business .

PAGE 70

60 assessment of program proposals, along with the principal advantages and some of the problems associated with implementing each. This list is not intended to be complete, but rather to indicate the wide variety of possible approaches to the problem of risk. Point Estimate of the Probability of Success The measure developed in this study may be classed as a point estimate of the probability of success. Use of a logically consistent methodology assures some degree of uniformity among values assigned by different analysts and permits extension to decision making and assignment of tangible meaning to the index. The principal disadvantages of this sort of measure lie in the difficulty of generating a set of data in the proper form, the fact that some of the assumptions necessary to the derivation of the index may not hold strictly true, and the inability of the analyst to assign a confidence measure to his estimate of the probability of success. Point Estimate with Standard Deviation The point estimate with a measure of dispersion is an approach which was considered at some length during this investigation. The inclusion of a dispersion measure would permit the analyst to indicate a confidence band about the value he obtains

PAGE 71

61 for the index''^ and would allow him to express a confidence level for the data from which the estimates were derived. The main difficulty arises from the fact that the form of the distribution of ]^ is unknoxm and must be assumed. The inability to derive a logically sound methodology for measuring the standard deviation prevented its inclusion in the method developed here. Attainment Profile The attainment profile as a measure of risk is discussed in a following section. In essence, it evaluates the probabilities of attaining various percentages of the desired schedule, budget, and performance goals. An advantage of this approach is that by use of a "profile adjustment function' the profile may be updated over time. This permits analysis of a more dynamic nature than is possible with other types of risk index. Disadvantages are the difficulty encountered in the determination of the form of the initial profile and the fact that the entire profile must be updated for any change in inputs or specifications.* *This could take the form of a probability statement: P(.82 ^ Ft ^ -96) = 997o Subsequent to the work done in this area in the course of this study, Terrell (22,23) independently synthesized an approach similar to the attainment profile for a study of the effects of schedule compression upon program risk.

PAGE 72

62 Financial or Economic Risk Function The financial or economic risk function represents a particularly useful tool for decision makers, since the decision to attempt a project ultimately hinges upon economic considerations.* This method would generate a risk function in terms of the impact risk has upon the firm from an economic point of view. It relieves the analyst of the necessity of making a "two-step" study, that is, first assessing the risk and then applying this measure to financial considerations. The principal disadvantage stems from the fact that it requires simultaneous consideration of several aspects of a many-faceted problem, and the resulting difficulty necessitates the use of even more sophisticated analytical techniques. Other Possible Measures There are many other possible types of risk indices which may be developed, depending upon the particular definition of risk one chooses. One such general type of index which was considered in the course of this investigation is the idea that risk may be a variance (of program length or funding or product reliability, for example). At least *There have been instances in which military values became the overriding objectives, but even under these conditions, the decision of how to implement the desired system becomes one of economics.

PAGE 73

63 one other study (24) speaks of risk as being the variance of factors which are not included in the estimating equations. In general, the term "risk" will mean different things to different people in different contexts, and the type of measure which best describes risk is a function of the particular definition employed. For a discussion of other measures of risk, see Some New Approaches to Risk by Byrne et al. (25). Problem Areas Data Accumulation One major problem, regardless of the approach used in the risk analysis, is the unavailability of accurate data in usable foirm. The form of government data can obscure the actual contractor costs; there is little usable data exchanged between firms because of the competitive nature of the industry; firms are reluctant to provide data which indicate that they have overrun cost and/or time estimates; both security regulations and proprietary motives restrict this flow of data. One possible solution to this problem is the generation of a data base by conducting a series of controlled experiments in which care is exercised to assure that accurate cost, schedule, and performance data are accumulated in a

PAGE 74

64 usable format. This could be carried out within a single firm or (if precautions were taken to protect the parties involved) among firms. Obviously, all firms involved might benefit from the larger data base available as a result of such cooperation. Certainly the availability of a better file of historical data would improve the accuracy of any risk index. Bias The problem of bias is one which appears at every phase of risk analysis. Several studies speak of the "optimistic bias" on the part of the prospective contractor in submitting estimates. This may be separated into deliberate bias and unintentional bias. The deliberate bias is injected in order to underbid competitors and obtain a contract, knowing that additions will permit the tentative figure to be raised to a more realistic one after the program has begun. Any point in the analysis which permits or requires judgment on the part of the analyst or estimator invites bias of a personal judgment nature. VJhile seemingly undesirable at first glance, this type of bias may well be a desirable feature if the analysis makes use of the experience of senior analysts. Another type of bias is the bias in a statistical sense which is present in the selection of the sample of data points from among the total universe of all aerospace programs.

PAGE 75

65 The small sample size, and the fact of nonrandomized selection of points, preclude even the measurement of the bias. This sort of bias has been unavoidable in the past, since the data which were available in usable form were limited. ' Hopefully, this situation will improve over time as the need for data more applicable to risk analysis becomes recognized throughout the industry. Distribution Forms At various places in the analysis of Risk, it becomes necessary to assume forms for statistical distributions. In most instances the lack of data makes justification of the assumption difficult. A case in point is the assumption of the beta distribution to describe the distribution of the probabilities of completing a program within certain resource constraints. The primary justification given was that a previous study (PERT) had assumed this distribution form for a similar parameter with good results. One reason for the inability to generate a measure of dispersion to accompany the Risk index of this study was the inability to justify the choice of distribution forms. The only solution to this problem is the accumulation of sufficient data to enable analysts to determine the actual forms. Lacking this data bank, one may oniiy rely upon logic and pragmatic results for justification of assumed distributions.

PAGE 76

66 Nonuniform SOA-A Measures At present there appears to be no methodology available for measuring SOA advances that does not involve a substantial amount of judgment or subjective measurement. While the use of subjective judgment has the advantage of permitting the experienced analyst to utilize the knowledge acquired over time, it does preclude the possibility of specifying a single nxomber which would represent the SOA-A for a program. At the present time there is an array of SOA-A measures rather than a single number; there is, in general, no consensus among analysts as to the "true" value. There are advantages and disadvantages to this arrangement, but the problem does exist. Chapter II contains a more thorough discussion of this problem. Cost-Time Tradeoffs In the analysis presented in this study, as in prior studies, it was necessary to make the simplifying assumption that cost and development time are independent parameters. In fact, the economist is aware that a tradeoff relationship exists between these inputs similar to that depicted in Figure 11. In general, the function will be different for each program and will change over time during the program. The tradeoffs may be expressed at a point or over a small

PAGE 77

o 67 segment of the curve at a given time, but these are sensitive to political, social, and military attitudes and thus may change rapidly and radically. As more sophisticated techniques are brought to bear upon the problem, the cost-time " tradeoff function can be defined and periodically updated to permit the development of a more refined risk index. E Cost ($) Figure 11 TIME -COST TRADEOFF CURVE (Reference 12, page 510)

PAGE 78

68 Summary This chapter has been devoted to a discussion of some possible refinements in the Risk index developed in Chapter III. Several techniques which can be of value in reducing subjectivity in certain areas of the study (NAM, GERT, DELPHI, PATTERN, and Sobel's approach) are presented. The difficulty associated with the use of NAM or GERT arises from the necessity of having the program defined to a level of detail which is unrealistic for most advanced programs. The main barrier to use of the iterative judgment techniques is that these require carefully structured batteries of questions, the cooperation of several experienced analysts, and greater amounts of time than may usually be available. Further analysis may reveal that a modification of one of these general approaches will prove fruitful. In analysis of the Risk index of Chapter III, several types of risk measurement indices which would be of value in evaluating aerospace program proposals are treated. Some of these are essentially refinements of the m.odel developed in this study, while others represent different ways of looking at risk and the application of different sorts of analytical tools. A final section outlines some of the problems which

PAGE 79

69 face the analyst in attempting to measure the risk associated* \;ith a particular aerospace program.

PAGE 80

CHAPTER V EXTEKSIOi.^ TO DECISION MAKING Pur pose The purpose of this chapter is to indicate a means for the application of the Risk index to the problem solving or decision making process. First, the various environments in v^hich decisions are made are presented; then there follov7S a brief discussion of the elements of game theory with extensions to utility theory and the application of the Risk index in both of these areas; finally, a section is devoted to explanation of various types of contracts and the relationships which may exist betv7een Risk and contract type. Management peci s ions Decision Making • Economists usually distinguish three broad categories or classes of decisions: decisions under conditions of certainty, decisions under conditions of uncertainty^ , and "For a good discussion of the difference between uncertainty and risk, see Knight (2) . 70 -

PAGE 81

71 decisions under risk»" The classifications are dependent upon the confidence the decision maker may place in the data which he used as a basis for the decision. One of the clearest V7a5''s to differentiate among these three situations is to consider a hypothetical aerospace contractor. He has the capability of buildiiig either long range military aircraft (bombers) or intercontinental ballistic missiles (ICBM's), but not both, and finds that he must decide which of these tV70 product lines he will produce. Due to the long lead times required, he cannot wait until the defense department issues a request for proposals on the nev; systems to be ordered but must decide now upon his course of action for several years to come. If the company's president has a son-in-law vAio is Under Secretary of Defense and has access to the information that the Secretary of Defense has decided to rely exclusively upon procurement of ICBM's for the nation's defense in the coming years, the company can decide with certainty (or as "The term "risk" as used in this section does not refer specifically to the same concept as the term "Risk" used in previous sections. Although the connotations are similar, the term "risk" will be used to refer to the general concept and the capitalized "Risk" to the more narrowly defined concept of this report o

PAGE 82

72 close to certainty as is possible in the defense industry) . If the president were not so fortunate in his daughter's choice of suitors, the firm v7ould find itself operating vmder conditions of uncertainty having no idea V7hich course of action it she ild pursue. In the absence of any information about the relative likelihoods of the possible defense postures to be ass.med by the defense department, the game theorists suggest that one possible wa}'' to make the decision is to assume equal probabilities of occurrence for each and fl5.p a coin. There is, hoxi/ever, a middle ground between secure knov7ledge and complete bev7ilderment on the part of the decision maker. This area will be referred to as decision making under conditions of risk. In the example, the firm could move from conditions of uncertainty to those of risk by searching the available literature, hiring people from the defense department, performing analyses of the conditions which would determine the future defense posture (eog., analysis of the enemy threat in the future) , and. assigning probabilities of occurrence to the possible alternatives. Having done this, the decision maker is said to be deciding under conditions of risk.

PAGE 83

~ 73 To summarize, if the decision maker knows the consequences of the o.lternative courses of action open to him at the time of the decision, he decides under certainty; if he knov7S the possible outcomes and can assign a set of probabilities to them, he decides under risk; if he ca.nnot assigi"> probabilities to the possible outcomes or if he does not know what outcomes may occurj he decides under conditions of uncertainty. By assigning a probability of success to proposed programs, this report attempts to enable the decision maker in the aerospace industry to move from decision making under conditions of uncertainty to decision making under risk. Ga sie Theory ^':" and IJtili.ty One area in V7hich the Risk index may find direct application as a decision makiiig tool is game theory. This is a widely known tool for quantitative decision inaking which, in recent years, has come into wide-spread use as a method for m.aking business decisions. Since this report attempts to enable the decision maker to move from the area of uncertainty to that of risk, it is perhaps instructive to examine this quantitative decision m'.king tool. "For a thorough analysis of this subject, the reader is referred to Chernoff and Moses (26), Luce and Raiffa (27), Swalm (28), Voii Neumann and Morgans tern (29), end Williams (30).

PAGE 84

74 111 moving into the area of decision making under conditioas of risk, the analyst finds it necessary to assign probabilities of occurrence to the various possible outcoraes of a proposed program. One method of accomplishing this is by direct: substitution of the Risk index of this study ((^'p) for the piobability that the project will be successful, A more sophisiiicated analysis V70uld also consider the value of completing 50 or 75 percent of the project (usually this V7ould result in a negative payoff or loss) and techniques similar to the attainment profile could be applied to generate the probabilities of occurrence of the respective events. The same sort of analysis can be applied to a utility function, once that function has been generated. Attai nment P r o file'''-' The discussion to this point has considered Risk in terms of the probability of completing 100 percexit of the program goals, A more thorough analysis would consider the entire life of a project and the probability of completing various percentages of the total program requirements within the *Much of the initial work on the attainment profile concept was done by VJ. J. Bailey III of General Dynamics' Fort Worth Division ivi an unpublished paper. The author gratefully atcknowledges Mr. Bailey's assistance in the preparation of this report.

PAGE 85

75 given resource restrictions. A technique which attempts this sort of analysis is the attainment profile. Programs with equal probabilities of completing 100 percent of the performance goals can have significantly different profiles; conversely, projects with similar profiles may differ with respect to their probabilities of 100 percent completion. Consider the profiles of Figure 12. Profiles I and II each have y/^ values of 10 percent; yet the probability of completing 90 percent of project I is only 15 percent. Project I could therefore entail much greater risk than project II. Project III has a m.uch larger j!/^ than project II, but it may actually involve more risk. It is possible to derive a profit'-^ function which states profit earned by the contractor as a function of the percent of project goals completed. If such a function were defined and superimposed on the attainment profile, the expected value of profit and the probability of achieving any given level of profit (or loss) can be generated. Such values could be used as inputs to financial planning models or game type decision models, and should generally tend to result in a more thorough analysis than the simpler Risk index developed in Chapter III. *The term "profit" as used in this section refers to accounting profit (income less cost) rather than to economic profit.

PAGE 86

76 4-1 •H .-I •rl to o 0.1 Probability of corapleting 100% of the project 7t. Coriipleted 90 100 0.1 Probability of completing 100% of the project, but much less risk than project 1 % Completed 90 100 III 0.4 Probability of completing 1007o of the project, but may have more risk than project II _LL|4 "/o Completed 90 100 Figure 12 ATT A IN14F,NT PROF ILE S

PAGE 87

77 It is also theoretically possible to derive a utility function for the firm and superimpose it on the attainment profile. While this is a theoretical possibility, the practical realization of such a curve is doubtful. Profit functions relative to percent of completion are very difficult to define, and utility is not linear with profit and can change over time with shifts in government policy, tax legislation, corporate size and image considerations, and other highly subjective factors o In short, the development of a utility function for use with the attainment profile is not practical at the present time , Assuming a linear profit function,* Figure 13 indicates some of the uses of the attainment profile concept as an analytical tool. J^^ is the probability of completing 100 percent of the goals, and P is the dollar value of the profit which is expected if all requirements are met within the limits set. Point B is the breakeven point; if this fraction of the requirements are met, the firm will neither gain nor lose money. P(B) is the probability of attaining this level of completion. Point M marks the 70 percent completion level. Point L shows the expected amount of loss for this amount of completion, and P(M) is the probability that this level can be realized. *Over the relevant ranges of completion, this assumption is not unreasonable. Incentive contracts purposely define a linear return in the area of the contractual figures

PAGE 88

78 $Profit «$Lc P4 o ^|:^:F4?i£:fertt|^l^:fei|:LF;;|rH-;hHJ^ii^ ::"p:::i|v;:[":; pa Cm CO o •r-l CO w M o Pi H P n H ^nTTTqeqcaa

PAGE 89

79 The construction of an attainment profile is not a well-defined process at presento One possible approach is to rely upon the expertise of the analyst; another is to use an approach similar to the Risk index method to quantify the 1 robability of success at several levels of co.npletion and connect the points thus generated to form the profile; a third approach would utilize historical data on similar projects; a fourth method x^ould be some combination of the first three approaches. The flow diagran of Figure 14 displays a procedure for developing and applying the attainment profile concept in aerospace planning. The program parameters are input into a NAM or GERT model. The outputs from the model are used to gen-jrate the initial attainment distribution. This initial profile is then adjusted by the evaluation of subjective factors and historical data. The entire profile can then be periodically updated or revised as conditions change over time. In summary, this section has discussed the concept of the attainment profile; its possibilities, shortcomings, and some of the difficulties associated with its development are suggested. As a concept which exhibits great potential as a decision making tool, its value is obvious; as a

PAGE 90

80 c 4J C'J 4J M e :3 ^ C .n •r—) •,-! 'H CJ JJ .U C 4J W ^^
PAGE 91

81 practically attainable maasure for aerospace programs, its future is less certain. As further research gives greater insight into the nature of risk and its consequences to program success, the attainment profile may becorae a pov7arful manage aent tool. Contracting and Risk"^' Tie area of contract provisions vras originally investigated X ith the intent of determining how these provisions reflected or m.easured contractor efficiency. It was hoped that suoh an investigation would generate factors that could be used to debias or normalize cost data for individual firms* efficiency or lack of it. This original line of investigation V7as not productive. The analysis of contract provisions, hov7ever, did shed light on the relationship bettv^een contract provisions and Risk. Even more important, the investigation suggested the possible application of a Risk index as an aid in establishing contract types and contract provisions. Currently the techniques used in contract procurement are still being developed and contract terms are determined largely on the basis of rules of thumb *The author gratefully acknowledges the assistance given by Mro Morris L. Williamson, Jr. of the Fort Worth Division of General D^'nam/ics in the preparation of this section.

PAGE 92

o.-c and judgment . AFS_CM 375_Sy s tera yZr^glp.e.eri.Vi^ Managp.me n L Procedur es (31) represents an attoiupt by the Departnient of Defense to systeuiize the systems developracnt avx\ contracting procedure. As this system and others sprailar in nature are developed, the contracting procedure v/ill become more well defined. Selection of conti^act provisions is essentia.lly c process by which financial or cost risk is allocated betv7een the contracting aL,e'ncy and the contractor. it should be kept in miiid that a project's schedule risk and performance risk cannot be allocated; the contracting agency always bears ultinate responsibility for these risks. The contracting agency, however, can minimize these risks by establishing contract provisions v/hich stimulate the contractor to meet schedule and perfcnnance requirements. The follov.'ing basic classifications of contract types were obtained from Moore (32) : I. Cost a. CPFF =Cost plus fixed fee b. FPR-E Fixed price rede terminable (retroactive) II. Incentive a. FPI Fixed price iv»ceni:ive (including pcrform^ance incentives) b. CPIF -Cost plus inccuitive fee c. FPri-A ' Fixed price redeterminable (prcospective) d. FF? -Fir;.' fixed price with (\vithout) escalation

PAGE 93

83 More explicityly, the following conditions characterize cost and incentive type contracts: CPFF: Established on an estimate of total cost that the contractor will incur plus a fixed fee (usually a percentage of total cost) . Provisions state that the fee does not change although costs may increase. FPR-E: Provides for a fixed price for certain periods of time, but provides for retroactive price redetermination. FPI CPIF: A negotiated target cost and target profit is determined. A price ceiling and a formula for adjusting the final price and profit are also defined. (Ceilings and formulae provide the incentive for contractors to reduce costs and to share in the savings.) An estimated target cost and a target fee is determined. A formula for fee adjustment (proportional to under/over run) is set. Similar to FPI except there is no separate price ceiling on a CPIF contract. FPR-A: Price is fixed initially for some interval of time and then reset at intervals for each future period. Similar to a series of FFP contracts for each interval, FFP: Provides for delivery of a product or service for a price specified in the contract. CPAF: (Cost plus award fee) The contractor is awarded a fee which is based upon his performance in managing the program and meeting the requirements as they evolve. This type contract is not mentioned by Moore, but has been used recently in contracts which involve advanced technology and require frequent adjustment of specifications. Within the fraraev7ork of the various contract types, efficiency in procurement and contracting is partially

PAGE 94

84 determined by risks and incentives. More specifically, the incentive rates are determined to some degree by the inherent technological uncertainty involved in the particular program. When the initial assessment of technological uncertainty is high, management often attempts to prolong contract negotiations. This prevents prematurely fixed target prices, which are often unrealistic and lead to cost overruns. In most advanced programs, technical uncertainty decreases with program progress. The contractor being forced into relatively early commitments must anticipate future areas of difficulty and incorporate these uncertainties into the contract essentials (i.e., varying the profit rates, clauses for future renegotiations) . The inclusion of terms such as these decreases the inherent financial risk to the contractor. Therefore, one of the m.ain purposes of the contract is to serve as an allocater of financial risk between the contractor and the procurer. The firm fixed price contract places maximum financial and technological risks upon the contractor and therefore induces a maximum incentive for cost control. The fixed price contract provides a minimum cost Risk for the customer since the maximum amount to be paid is fixed during negotiations. This contract type is used for low SOA-A type programs or when reasonably definite design or performance

PAGE 95

85 specifications are available. When the cost estimating uncertainties are not minimal or when uncertainties surrounding the contract performance cannot be evaluated, consideration is given to other contract types. For example, by the addition of escalation clauses, the contractor's financial risk in the fixed price contract is reduced. As one progresses to the fixed price with incentive clauses, the total program risk is distributed betvjeen the procurer and contractor who share the responsibility for costs greater or less than the original estimate. The cost plus incentive fee contract is negotiated when it is highly probable that the development is feasible and the desired performance objectives have been determined. This type of contract does not change the technical uncertainty connected with a program, but it may indirectly increase the probability of succeeding. This is accomplished by providing limited incentives (and penalties) and thus stimulating the contractor to be more efficient in his efforts and to manage the contract effectively. This type of contract may be best suited to programs of moderate complexity and technological advance. The cost plus a fixed fee contract is a cost reimbursement contract with a fixed fee for the contractor independent of the management's ability to control costs. This implies

PAGE 96

86 the financial risk to the contractor is almost zero. Hence, this type of contract has been used in high complexity programs where the SOA-A is great, the level of effort required is unknovm, and the relative probability of success is un" determined. Figure 15 illustrates the general relationship betwe the particular contract type desired by the contractor and the program's technological risk. en 0) E-" 4J O u o o CPFF Incentive Types With Escalation FFP High Moderate Low Program Risk ( j^„) Figure 15 CONTRACT TYPE VERSUS PROGRAM RISK

PAGE 97

87 Although contract form alone does not explain the final performance of a program, it is an important factor in determining the size of cost and schedule over/under runs. Although the population of DOD contract procurement actions per year numbers in the thousands, most of the samples investigated by recent researchers have been relatively small and stratified in type. Because of this nonrandom selection, inferences drawn cannot be justified in the statistical sense. However, some general trends from past programs can be obtained from the studies by Moore, Marshall and Meckling, and Peck and Scherer. The study by Moore concludes that on the cost type contracts, the average profit rate ranges from 6.3 percent to 6.8 pergent. On the other hand profit rates of incentive contracts are higher by a sizeable amount, averaging around 8 percent profit. The rationale for the difference is that there is a higher financial risk attached to the incentive contracts and that a higher reward must be offered to induce the contractor to be efficient. For example, from the sample of 228 incentive contracts investigated by Moore, one finds that on the average the apparent efficiency probabilities for an under/over run are: « 0.74 for a cost underrun 0.50 that the cost underrun is 10 percent 0.69 that results will be within + 10 percent of the target costs

PAGE 98

88 Care must be taken in the interpretation of the above results. Although there is indicated a 0.74 probability of a cost underrun, one should not conclude that the procurement of an incentive contract for a program will indicate " that program's success with the above stated probability. It must be kept in mind that the type contract negotiated is usually indicative of program complexity and uncertainty, and that the process of contracting is a complex procedure. As pointed out by Moore, "The basic rationale for renegotiation is the existence of uncertainty in defining what is being contracted for and uncertainty in defining and measuring performance. "* In another perspective, Marshall and Meckling in their research came to the follov/ing general conclusions: (1) "Early estimates of important parameters are usually quite inaccurate. They are inaccurate in two respects. First, such estimates are strongly biased toward overoptimism. Second, aside from the bias, the errors in estimates evidence a substantial variation. That is, even if estimates were multiplied by an appropriate standard factor to eliminate the bias, a non-negligible source of error remains . (2) The accuracy of estimates is a function of the stage of development, i.e., estimates improve as development of the item progresses. This also means that estimates for development projects representing only modest advances tend to be better than for more ambitious projects."*^ *Ref. 27, p. 120 'Ref. 11, p. 1.

PAGE 99

89 In the earliest stages of research and exploratory development, there are little significant data available. Here, use of a CPFF or general cost contract may be necessary because of inability to analyze the magnitude of technical or performance uncertainty. However, once more definitive data for a particular program are available, other contract types with associated incentive provisions are possible. The Risk index facilitites establishment of these provisions. Risk measurements of the prospective program plus Risk measurements and outcomes for previous programs will aid in determining whether cost plus incentive fee, fixed price incentive, or some other form is most appropriate. In addition, with the use of an index and a game theory formulation, the establishment of incentive provisions will be placed on a firmer foundation. With the index, the probabilities of meeting cost, schedule, and performance objectives contained in the provisions may be assessed in a consistent quantitative manner.

PAGE 100

CHAPTER VI • RISK INDEX APPLICATION Introduction One way to promote a more thorough understanding of the Risk index methodology is through the use of an example, Due to the classified nature of much of the cost and performance data on government aerospace programs, it is not possible to present an actual program as an example of the use of the Risk index. It was, therefore, deemed necessary to generate a hypothetical example program in order to demonstrate the use of the method. It should be pointed out that for the purposes of this analysis the question of whether or not the firm receives the contract will not be considered. The aim is rather to evaluate the feasibility of bidding on the program under various combinations of schedule and budget. Specifications The design and performance specifications shown in Table 3 for a jet fighter aircraft will be taken as the 90 -

PAGE 101

91 definition of the program. In addition to these specifications, the avionics systems must include terrain -following radar; the capability for multiple launch and tracking of the latest, most advanced missiles; infrared optical system; arid provisions for all-weather, day-night operation. The Department of Defense indicates that it expects that if the development program is successful, there will probably be an order of approximately 1000 units. Table 3 DESIGN AND PERFORMANCE SPECIFICATIONS Maximum Speed at Optimal Altitude Maximum Takeoff Weight Gross Weight (Empty) Maximum Thrust at Sea Level (total) Combat Radius (Supersonic) Ferry Range Number of Engines Wing Span Length Y^^ and ]//g Development Mach 3, 1720 knots 60,000# 30,000# 50,000# 1000 miles 4000 miles 2 50' 60' The advanced projects cost analysis group of the company develops estimates for the costs of the program. These are presented in Table 4.

PAGE 102

92 Table A INITIAL COST ESTIMATES BY COST CATEGORY Cost Item

PAGE 103

93 -

PAGE 104

94 Here the program will be considered in two ways: first, the RDT&E program separately; second, if there appears to be a reasonable chance of successfully developing an acceptable aircraft, the whole program (both RDT&E and Production) will be considered as a package. The data from Table 5 and the PERT assumptions (X = g ) were used to generate the cumulative distributi functions of Figures 16 and 17 for the RDT&E and total programs, respectively. Similarly, the time estimates of Table 6 were used to determine the cumulative time distributions of Figures 18 and 19. on Table 6 OPTIMISTIC, MODAL, PESSIMISTIC, AND ESTIMATED MEAN PROGRAM SCHEDULE ESTIMATES Program

PAGE 105

95 w c o t-{ l-l l-l pq d •H d p4 vo PQ H O PS > CO M O M H O O M H t3 P3 M Cd H M Q > O '^

PAGE 106

96 CO CNJ oH O FQ

PAGE 107

97 o to d 0) CO »^ W) o u CM w <^ H P 00 l-l 0) H O W O O H O CO > CO M O o M H t3 pq M P5 H W > 5 — I — O O o 00 o VO — I — o o CM O X '/'

PAGE 108

f)J ^a r— 1

PAGE 109

99 of areas under the appropriate probability density function. The probabi?.ity density functions v/ere determined by trial and error: Knowing the mode and the 1 percent end points, a preliminary curve is dra\m; the areas to the left and right of th-: mean are determined and the tentative curve is adjusted in such a way as to make these areas equal. If the method suggef ted by Sobel and discussed in Chapter IV of this study v-ere used, the cumulative curve could be sketched direct ".y, since the abscissa values are knovm for the 1, 10, 50, 90 and 99 percent ordinate values. This observation prompts the recommendation that the Risk index methodology could 1: 3 made simpler, faster, and easier to use by the addition of a requirement for the 80 percent central estimate. From tr;2 cumulative distributions of Figures 16, 17, 18 and 19, the probabilities of dollar and time success (W^ and W ) can be determined for any given values of $ and t o o * SOA-A Measurement The remaining input required for the generation of the Risk index (0.^,) is the state of the art advance (Srp) required by the project. The first step in the determination of S^ is a listing of the critical subsystems (X^) and assigning of the relative weights (a^.) . This procedure is shovm in Table 7.

PAGE 110

100 Table 7 DERIVATION OF a, Subsystem

PAGE 111

101 CO •r-l o o CO m o o in o o

PAGE 112

o -102 To determine S;i; is is necessary to sura the products RDT&E: 4 St = Z a^S • = (.065)(.7) + (.0585)(1) + (.035)(.45) i=l + (.018)(.08) = .1213 TOTAL 4 PROGRAM: S = ^ a S ' = (.065) (.8) + (.0585) (1) J1=1 " ^ + (.035)(.55) + (.018)(.12) = .1320 Sensitivity Analysis At this point the only additional inputs required to determine the Risk index are the values to be used for $ and t^. The unadjusted probabilities of success for the two programs in terms of funding and schedule may be determined by referring to Figures 16, 17, 18 and 19. Table 9 summarizes the values for the Risk and Success indices for hypothetical values of $^ and t^. If it is assumed that the firm will not undertake any project which has a probability of success of less than 0.50, and that the DOD has indicated the amount of time which will be allowed for the completion of the program, a distribution may be generated which will indicate the minimum bid the firm can submit and the probability of success which accompanies any higher level of funding. For example, consider

PAGE 113

103 H PS o CO I-l CO >^ H M H M W iZ M to [-1

PAGE 114

104 the RDT6cE program described above. If the project is outlined as a five-year study contract and the firm is asked to bid, the distribution of Figure 20 could be generated to describe the chances of success. Figure 20 shows that with a 60-month deadline, the minimum possible bid for a 50 percent chance of success is $1,76 Billion. Given 70 months the bid could drop to $1.65 Billion; if the schedule is compressed to 50 months, the minimum bid is $2.62 Billion. Figure 20 can be interpreted in another way: given $^ = $3 billion, the probability of success for the three values of t can be read directly (t^ = 90 months, frj, = .92; t^ = 60 months, f = .86; t^ = 50 months, j^T = .55) . Conclusion This chapter has sho\m the application of the Risk index derived in this study to a hypothetical aerospace program. Starting with the performance specifications, the time and budget were assumed and from these the distributions for f and]^^ were generated. The specifications were used as a base for the SOA-A measurement (S^) , and the assumption of various values for $^ and t^ led to the statement of the corresponding values for)/^^. The sensitivity of ^^ to changes in $Q and t^ was indicated, and a graphical technique was

PAGE 115

105 M H O W

PAGE 116

105 presented v/hich permits analysis of the minimum bid figure for a given program length and a stated probctbility of program success. Exercise of the model shows up several problem areas. The first is that the method suggested by Sobel would reduce the effort required by the analyst and perhaps increase the accuracy of the algorithm. It V7as noted that a significant amount of judgment is involved in the measurement of the SOA-A. At present it seerns unfeasible as v/ell as undesirable to eliminate this subjective element, since lack of a data bank forces reliance upon the experience and expertise of the individual analyst. A major weakness is the definition of theTcurve. The curve of Figure 3 vjas the product of extensive review of the m.eager store of available data; lengthy conversations with experienced analysts; thorough investigation of a similar curve derived by Terrell, and the assumptions behind it; and much thoughtful consideration by the author. It is felt that until such time as repeated use of the Risk algoritlira generates a data bank of historical data to permit the definition of a revised T curve, the curve of this report represents as good an estimate of the actual relationship as can be developed. In conclusion, the Risk index developed here is not presented as an ultimate measure, but rather as a tentative

PAGE 117

107 step in a direction V7liich shov7S promisee In the final analysis, it will be the revision and modification of the technique developed here and above all its usefulness in program planning and project management v/hich \7ill be the ultimate measure of its value. *

PAGE 118

REFERENCES

PAGE 119

REFERENCES 1. One Plundred Companies and Their Subsidiary Corporations Listed According to Net Value of Military Prime Contract Awards, Washington, D.C.: U.S. Government Printing Office, Office of the Secretary of Defense, Directorate for Statistical Services, 1965, 1. 2. Knight, F. H. Risk, Uncertainty, and Profit , Boston: Houghton Mifflin, 1921. 3. "Managing Risks for More Effective Program Control," Aerospace Management , Volxnie I, Number 1, Valley Forge: General Electric Company, Missiles and Suace Division (Spring, 1966) , 17. ' 4. Hagen, 0. "Risk Aversion and Incentive Contracting," Economic Record , Sydney, Australia: University of New South Wales (Septem.ber, 1966), 416-429. 5. Guide to the Evaluation of the Performance of Major Development Contractors , Washington, D.C.: U.S. Government Printing Office, U.S. Department of Defense, 1964, 1. 6. "How a Great Corporation Got Out of Control," Fortune Magazine (January, 1962), 64-69, and (February, 1962), 120-122. 7. "GD: The Hard Road Back from the Brink," Business Week (December 7, 1963), 144-146. 8. Appropriations for 1964 , Part 2, Washington, D.C. : U.S. Government Printing Office, U.S. Department of Defense, 1963, 557. 9. "Skybolt," Market Intelligence Reports , "Rockets, Missiles, and Spacecraft," Greenwich, Connecticut: DMS, Inc. (October, 1963) , 2. 109

PAGE 120

110 10. Alchian, A. A. Reliability of Cost Estimates Some Evidence , Santa Monica: The RAND Corporation, 1950. 11. Marshall, A. W. and Meckling, W. H. Predictability of the Costs, Time, and Success of Development , Santa Monica: The RAND Corporation, 1959. 12. Peck, M. J. and Scherer, F. M. The Weapons Acquisition Process An Economic Analysis , Cambridge: Harvard Business School, 1962. 13. Polski, J. R. ; Clausen, I. M. ; and Paige, H. W. Risk Appraisal of Programs Systems (R/.PS) , Paper No. 64-406, Valley Forge: General Electric Company, 1964. 14. Summers, Robert. Cost Estimates as Predictors of Actual Weapon Costs: A Study of Major Hardware Articles , Santa Monica: The RAND Corporation, 1965. 15. Terrell, J. B. Development Risk (Master's Thesis), Dallas: Southern Methodist University, 1964. 16. Hall, Edward N. An Econom.ical Approach to Space Transportation , AASd7-133, paper presented at the thirteenth Annual Meeting of the Am.erican Astronautical Society (May 3, 1967) . 17. "Contractor Performance Evaluation System," Aerospace Management , Volume I, Number 1, Valley Forge: General Electric Company, Missiles and Space Division (Spring, 1966) . 18. McCrimmon, K. R. and Ryavec, C. A. An Analytical Study of the PERT Assum-ptions , Santa Monica: The RAND Corporation, 1962. 19. Faucett, W. M. ; Henry, G. C. ; and Wilson, R. K. Network Analysis Model , EER-FW-368, Fort Worth: General Dynamics Corporation, 1964. 20. Pritsker, A. A. B. C-SRT: Graphical Evaluation and Review Technioue , Santa Monica: The RAND Corporation, 1966. 21. Sobel, S. A Com.puterized Technique to Express Uncertainty in Advanced System Cost Estimates , Washington, D.C.: U.S. Department of Commerce Clearinghouse, 1965.

PAGE 121

Ill 22. Terrell, J. B. Program Acceleration Study, Final Report,* FZM-473o, Section G and Appendix E, Fort'l'/orth: General Dynamics Corporation, 1966 (Secret). 23. Terrell, J. B. and V/ickstrora, C. A. Risk Ana 1 y sis for S chedule Corap res s ion , Mil-0-141, Fort Uorth: General Dynaraics Corporation (December, 1966), 24. Mc Knight, J. S., et al. Lannch Veliicle Systems Ccs'c Model, FZM-4070j Fort VJorth: General Djaiamics Corporation^ 1964o 2.5. Byrne, R.„ , et al. S ome Me v7_Aporoaches to Risk, Pittsburg: Carnegie Institute of Technology, Graduate School of Industrial Administration, 1967. 26. Chernoff, H. and Moses, L. E. El emen tary Dec is ion Theory, New York: John Wiley Sc Sons, 1960. 27. Luce, R. D. and Raiffa, H. Games and DRcisions_ In-^ troductiou and Critical S urvey , New York: John Wiley &'"s'onc,'" 1957. 23. Sv/alm, R. 0„ "Utility Theory Insights into Risk .Taking." Harva'cd Business Pveview (November, 1966), 123-136. 29. Von Nermiann, J. and Morgans tern, 0. Theorj of &emes_ and Econ omic Behavior, Princeton: Princeton University Pi'ess', 1953. 30. Williams, J. D. The Compleat Strate^ysj:, New York: McGraw-Hill, 1954. 31. £^/stems_Enjy^neering Managcm^ent_J^i^c£dur^ AFSCM 375-5, Washington, D.C.: IJ. S. Government Printing Office, Air J"orcG Systf^ms Command, 1966. 32. Moore, F. T. MJ.litary__Procurement and Coatracting : An_ Econc>mic_ Ana lysis, Santa Monica: The P^ND Corporation, 1962.

PAGE 122

112 ADDITIONAL REFERENCES Books Davidson, D.; Suppes, P.; and Siegel, S. Decision Making , An Experitnental Approach , Stanford: Stanford University Press, 1957. Halff, John F. Study Guide and Cases for the Third Edition of Principles of Management by Koontz and O'Donnell , New York: McGraw-Hill, 1964. Hitch, C. J. and McKean, R. N. The Economics of Defense in the Nuclear Age , Cambridge: Harvard University Press, 1960. Keithly, E. M. Manual of Style for the Preparation of Papers and Reports , Cincinnati: South-Western Publishing Company, 1963. Kendall, M. G. Advanced Theory of Statistics , Volume II, London: Griffin, 1951. Koefod, P. E. The Writing Requirements for Graduate Degrees , New York: Prentiss-Hall, 1964. Morris, W. T. The Analysis of Managem.ent Decisions , Homewood, Illinois: Irwin, 1964. Savage, L. J. The Foundations of Statistics , New York: John Wiley & Sons, 1954. Scherer, F. M. The Weapons Acquisition Process Economic Incentives , Cambridge: Harvard Business School, 1964. Simon, H. A. Administrative Behavior , New York: McMillian, 1957. Snedecor, George W. Statistical Methods , Ames, Iowa: The Iov7a State Press, 1956.

PAGE 123

113 Turabian, Kate L. A Manual for Writers of Terra Papers, Theses, and Dissertations , Chicago: The University of Chicago Press, 1955. Wilks , S. S. Mathematical Statistics , Princeton: Princeton University Press, 1943. Papers and Periodicals Alchian, A. A. Reliability of Progress Curves in Airframe Production , Santa Monica: The RAND Corporation, 1962. Austin, R. N. , et al. A Study of Manned Mars Exploration in the Unfavorable Time Period (1975-1985) , FZM-4039-3, Fort Worth: General D5?naraics Corporation, 1964. Bagby, F. L., et al. A Feasibility Study of Techniques for Measuring and Predicting the State of the Ar'i , Dayton: Air Research and Development Command, 1959. Bickner, R. S. The Changing Relationship Between the Air Force and the Aerospace Industry , Santa Monica: The RAND Corporation, 1964. Brunner, E. D. The Cost of Basic Research Effort: Air Force Experience, 1954-1954 , Santa Monica: The RAND Corporation, 1965. Churchman, C. W. and Ackoff, R. L. "An Approximate Measure of Value," Journal of the Operations Research Society of America , Volume 2, Number 2, 1954, 183. Gordon, J. T. and Helmer, Olaf. Report on a Long Range Forecasting Study , Santa Monica: The RAND Corporation, 1964. Haynes , Grady Lee. Quantification and Utilization of Subjectively Determined Data in the Construction of Mathematical Models (Master's Thesis), College Station: Texas ASM University, 1966. Large, J. P., editor. Ccncepts and Procedures of Cost A.nalysis , Santa Monica: The RAND Corporation, 1963. NASA PERT and Comparison Cost System Handbook , Washington, D.C.: National Aarcn.u.utics ax"id Space Administration, 1962. «

PAGE 124

114 Novick, David. "What Do VJe Mean by Research and Development?" California Management "Review , Volume II, Number 3 (Spring, 1960) , 44. Pounders, Cedric Jack. The Choice Among Alternative Weapons and Weapons Systems: An Economic Analytical Treatment (Ph.D. Thesis) , Dallas: Southern Methodist University, 1965. Roberts, E. B. and Sloat, J. B. Effects of Incentive Con tracts in Research and Developmtent : A Prelimdnary Research Report , Cambridge: Sloan School of Managem.ent, Reprint No. 705, 1966. Samson, D. G. and Coble, H. F. Aircraft Pianninp; and Cost Model , ERR-Fl-J-456, Fort Worth: General Dynamics Corporation, 1965. Zabroxvski, S. K,, et al. Industrial R&D Funds in Relation to Other Economic Variables , Washington, D.C,: National Science Foundation, 1964.

PAGE 125

BIOGRAPHICAL SKETCH

PAGE 126

BIOGRAPHICAL SKETCH William Emery Pinney was born September 16, 1941, at Pensacola, Florida. He was graduated from Pensacola High School in June, 1959, and entered the University of Florida in September of that year. On April 19, 1964, he received the Bachelor of Science degree in Electrical Engineering at the University of Florida. In August, 1965, he received the degree of Master of Business Administration and immediately began work toward the degree of Doctor of Philosophy in Economics and Business Administration. He completed his course work and passed his qualifying examinations for admission to candidacy in August, 1966. His major area comprises the fields of Management, Economic Theory, and Statistics; the minor field is Operations Research. From September, 1966, to the present time he has been employed as Senior Operations Analyst at the Fort Worth Division of General Dynamics Corporation, On April 26, 1966, he was married to Linda June White. Their first child, Laura Brooke, was born January 19, 1967. He is treasurer of the Fort Worth Economics Society, a member lib -

PAGE 127

ii.7 of the American Astrona-aLical Society and th^^ National ManagSM.eiit Association, and a part time nicinber of the graduate economics faculty in the evening college at Texas Christian University.

PAGE 128

This dissertation was prepared under the direction of the chairman of the candidate's supervisory committee and has been approved by all members of that committee. It was submitted to the Dean of the College of Business Administration and to the Graduate Council, and was approved as partial fulfillment of the requirements for the degree of Doctor of Philosophy. December 19, 1967 Dean, College of Business Administration Supervisory Committee: Chairman, \J. V. Wilraot^ Jr. R (^A^o/ H. patnes i H. patn mTrT Lang ham;; Dean, Graduate School . N. BrasVell X^

PAGE 130

5376 ,.")