A methodological study of various follow-up procedures for community college vocational programs


Material Information

A methodological study of various follow-up procedures for community college vocational programs
Physical Description:
viii, 155 leaves : ; 28 cm.
Noriega, Penny Proctor, 1953-
Publication Date:


Subjects / Keywords:
Educational surveys -- Methodology   ( lcsh )
Vocational education   ( lcsh )
College graduates -- Employment   ( lcsh )
bibliography   ( marcgt )
theses   ( marcgt )
non-fiction   ( marcgt )


Thesis--University of Florida.
Includes bibliographical references (leaves 149-154).
Statement of Responsibility:
by Penny Proctor Noriega.
General Note:
General Note:

Record Information

Source Institution:
University of Florida
Rights Management:
All applicable rights reserved by the source institution and holding location.
Resource Identifier:
aleph - 000097768
notis - AAL3209
oclc - 06609871
lcc - LB1028 .N67 1979x
System ID:

This item is only available as the following downloads:

Full Text







I would like to thank my committee, Dr. Stanley Su, Dr. James Hale,

and Dr. John Nickens, for their help and support during the writing of

this dissertation. I would like to extend special thanks to my chair-

man, Dr. John Nickens, for his continued encouragement throughout my

graduate career.

Personal thanks go to Dr. Debbie Hasselo and Dr. J. Nick Ehringer

for their assistance and exchange of moral support as we worked toward

dissertation completion. My appreciation also goes to Dr. Rosanne Gmuer

for her encouragement, understanding, and confidence. I would also like

to thank Carole Alonso for her diligence in typing this dissertation in

spite of the obstacles.

My gratitude also extends to my parents, Robert and Ann Proctor,

and my brother, Gary, for their sustaining belief in my abilities. Much

love and appreciation goes to my husband, Brian, for his advice, assis-

tance, and cooperation throughout this project.



LIST OF TABLES . . ... v

ABSTRACT . .. . vi


Statement of the Problem. . . 6
Delimitations . ... 6
Justification of the Study .............. 7
Assumptions. . . ... .. 8
Definitions of Terms . . 9
Procedures . ... ... ... 11
Organization of the Research Report. . .. 11


Introduction . . ... 13
Mail Questionnaire . ... 17
Telephone Survey . .... 58
Personal Interview .................. 72
Computerized Data Retrieval System . 86


Mail Questionnaire . . 95
Telephone Survey . .... 99
Personal Interview .. .. .. .. .. .101
Computerized Data Retrieval System . .104
Vocational Education Follow-up Studies ... .105
Follow-up Study of Vocational Program Graduates. .106
Follow-up Study of Vocational Graduates' Employers .. .120


Comparisons. . .... .130
Evaluation .. .. ... ..... .. ..136
Mail Questionnaire . . .. 137
Telephone Survey . . .. .138
Personal Interview . .. .139
Computerized Data Retrieval System ........ .139
Further Research . .. .. 140


REFERENCES . . .. .. .49




Table 1 Cost Estimates for Mail Questionnaires .. 22

Table 2 Comparison of Major Study Costs Using Different
Monetary Inducements. . 24

Table 3 Costs for Mail Questionnaires and Personal
Interviews . . .. 26

Table 4 Christison's Cost Effectiveness Study Results. .. 27

Table 5 Cost Estimates for Telephone Interviews. ... 62

Table 6 Cost Estimates for Five Hundred Interviews 76

Table 7 A Comparison of a Mail Questionnaire and a
Computerized Data Retrieval System Study 88

Table 8 Performance Characteristic Rating for Personal
Interviews (Face-to-Face), Mail Questionnaires,
and Telephone Interviews . ... 92

Table 9 Analysis of the Four Methods of Follow-up Based
on Five Criteria . . 96

Table 10 Comparison of Cost for Different Sample Sizes. 107

Table 11 Ratings of the Four Methods of Follow-up Across
the Five Qualitative Criteria for Vocational
Graduates Follow-up. . 119

Table 12 Ratings of the Four Methods of Follow-up Across
the Five Qualitative Criteria for the Employer
Follow-up. . . 128

Abstract of Dissertation Presented to the Graduate Council
of the University of Florida in Partial Fulfillment of
the Requirements for the Degree of
Doctor of Philosophy


Penny Proctor Noriega

December 1979

Chairman: John M. Nickens
Major Department: Educational Administrative and Supervision

The problem of this study was to compare and evaluate selected

methods of conducting follow-up studies of graduates of community college

vocational education. Four methods were chosen for in-depth analysis:

mail questionnaires, telephone surveys, personal interviews, and computer-

ized data retrieval systems. Related literature was searched for criteria

on which to compare the methods. Five criteria were chosen: costs, re-

sponse rate, length of time, personnel required, and quality of data.

The four methods were then analyzed based on the five criteria. The rela-

tionships between the four methods of follow-up were discovered and reported.

Interrelationships between the five criteria and the four methods of follow-

up were also found. The advantages and disadvantages of each method were

analyzed and the subsequent strengths and weaknesses of the methods explored.

Consideration was given to each method's ability to collect follow-up data.


Theanalysis produced suggestions for optimal use of each method.

Specific methodological analyses were performed and recommendations

made for two common vocational education follow-up needs, studies of

vocational education employed graduates and employers of these graduates.

The results indicated that the mail questionnaire method of follow-

up was advantageous in obtaining follow-up information concerning soci-

ally sensitive subjects. This method was also found to be the most ef-

fective in costs and time when dealing with large sample sizes. A major

disadvantage of this method was low response rates. Various techniques

for improving response rates were reported in the Review of Related


The telephone survey was found to be advantageous when performing

a small, spontaneous study. Costs, time limits, personnel requirements,

and sample size were found to be interrelated in the telephone survey.

The telephone survey was found to be an acceptable compromise between the

mail questionnaire and the personal interview.

The personal interview was found to produce high costs but also pro-

cured the highest quality data in relation to complexity and depth of

answers. Sample size, geographic dispersion, costs, personnel required,

and length of time were found to be interrelated in the personal inter-


The computerized data retrieval system was found to have many ad-

vantages: low costs, short time period, and few personnel requirements.

However, the type of data that could be collected was limited, and the

geographic location of respondents was restricted.

Overall results indicated that for the purpose of follow-up voca-

tional graduates, either a mail questionnaire or a telephone survey

would be preferable. The computerized data retrieval system could be

utilized for limited data or as a validation procedure. For a follow-up

of employers of graduates of vocational education, the personal inter-

view method was found to be preferable to the other three methods for

in-depth evaluation. If more general information were desired, the tele-

phone survey and the mail questionnaire could be utilized.

A table was developed from the analysis listing the four methods of

follow-up and the five criteria. The strengths and weaknesses of each

method were reported for each criterion including sample size and geo-

graphic dispersion relationships.





During the 1970's an increasing emphasis on accountability was

imposed on educators by legislators and the general public. "Social

unrest and dissatisfaction with the social institutions--of which

the educational system is only one--have led to the politics of con-

frontation by which results, rather than promises, are demanded"

(Morphet, Johns, & Reller, 1974, p. 542). Legislation intended to ac-

complish accountability has become common as legislators attempt to

determine the results which are being achieved in relationship to

moneys appropriated for education (Morphet et al., 1974).

Educational accountability is not easily explained or measured.

Williams and Snyder (1974) explained accountability in the following


Accountability includes evaluation of educational processes
and outcomes in terms of stated objectives and communication
of findings to educators and the public. The findings from
evaluation projects, of course, also provide a base for modify-
ing educational objectives and programs. (p. 40)

Morphet et al. (1974, p. 545) stated that one of the necessary condi-

tions in accountability is that "goals and objectives must be stated

inmeasureable terms." Gleazer (1973, p. 110), when president of the

American Association of Junior Colleges (AAJC), stated that accounta-

bility is available to vocational programs in the forthright terms of


getting jobs for the people trained, preferably jobs related to their

training. Brantner (1975) stated that it is how these vocational edu-

cation graduates fare in the world of work that is the proper measure

of accountability. Kaiserchat (1972) reported that the design of

follow-up studies may well be able to answer any existing questions of


Follow-up, usually considered one aspect of an evaluation system,

is designed to obtain data, information, or opinions about or from

former participants in the educational process (Headrick, 1977, p. 2).

Data frequently sought include length of time between graduation and

employment, field of employment, type of work performed, salary, and

promotion history. Student evaluations of their programs and training

received while attending the college, and employer opinions on the

level of training exhibited by their employees who are graduates from

the college can also be collected. Information collected through fol-

low-up studies is used to measure the effectiveness of courses, and is

essential to systemized evaluation of vocational programs (Headrick,

1977, p. 2).

Employability is an element of vocational education accountabil-

ity. Many community college leaders are required to supply much of the

information needed to answer the questions raised by accountability.

Unfortunately, the more information desired, the more difficult and

complex becomes the process of obtaining the information. The trade-

off in follow-up studies is between the amount of information collected,

and the cost and time that can be afforded by an institution for the

acquisition process. The more information desired, the more complex

and difficult the information acquisition techniques, necessitating

higher costs in institutional resources. In many instances, the acqui-

sition of information deemed necessary is an impossible task due to the

lack of cooperation on the part of the population or sample being fol-


Gold and Morris (1977) stated the following observation after

working with follow-up studies. "It is almost universally agreed that

the task of obtaining follow-up information that is reliable, valid,

and useful is a difficult one" (p. 7).

Some of the major limitations associated with the traditional

methods of follow-up have been high costs, extended length of time for

completion, extensive use of college personnel, low response rate to

questionnaires, and uncertain reliability of information (Baugh, 1976;

Krueck, 1975; Miner, 1977; Paul, 1976). Some of these limitations

are found wholly or partially in every follow-up study.

In 1973, the Virginia Department of Community Colleges began a

national review of follow-up studies made by community college person-

nel of former vocational education students. It was found that the

great majority of community college leaders were not satisfied with the

follow-up methods they found available (Williams & Snyder, 1974, p. 40).

Krisham Paul (1976) reported that researchers "still have not developed

adequate measures which would be acceptable to most administrators of

vocational education" (p. 1). Therefore, there is an obvious need for

better and more feasible methods of follow-up.

The traditional methods of follow-up have been the mail question-

naire and the personal interview. A mail questionnaire study involves

mailing a list of written questions to potential respondents for them

to answer and return by mail. A personal interview study involves a

person acting as an interviewer, usually going to the potential re-

spondents, verbally asking the questions, and recording the answers on

an answer sheet. Jackson and Rothney (1961) made the following state-

ment concerning the need for validation studies on the traditional

research methods employed for follow-up:

The merits, weaknesses, and biases of both the mail question-
naire and the interview have been discussed frequently, but
there has been little research designed to assess the compara-
tive contributions to the two procedures for follow-up purposes.
(p. 569)

Krisham Paul (1976) agreed that data collection is primarily geared

to questionnaire surveys and interviews, and pointed out they both suf-

fer from the problems of nonresponse and inadequate sampling.

The methods of follow-up have been expanded to include two re-

cently employed methods. The first is the telephone survey about which

little comparative research has been conducted. The second is the com-

puterized data retrieval system, first utilized by the Florida Commun-

ity/Junior College Inter-institutional Research Council (IRC) in 1975.

The telephone survey consists of interviewers calling on the tele-

phone and verbally asking prospective respondents a series of questions.

The respondents answer the questions verbally and the interviewer fills

in the answers on an answer sheet. This eliminates travel expenses

but can bias the sample since not every prospective respondent has a

telephone (Weiss & Hatry, 1971, p. 23).

The computerized data retrieval system utilized a new data source

for education, the Department of Commerce data file. This method was

used by the IRC to supply information for State of Florida government

reports. The IRC obtained a data tape of employment information from

the Florida Department of Commerce located in Tallahassee, Florida.

The data tape contained pertinent information about salary, weeks

worked, and industry code, all classified by social security number


The IRC next obtained data tapes from its community college mem-

bers and the universities of the state of Florida. These data tapes

had student information also classified by ssn. The data tapes were

then merged by matching ssn. Colleges who sent the ssn of their voca-

tional programs' graduates on their data tapes had the numbers matched

to the numbers on the Department of Commerce data tape, and the data

corresponding to the matched ssn was extracted from the Department of

Commerce tape. The data were then organized and analyzed to perform a

vocational education follow-up study.

The research validating the various methods of follow-up is ex-

tremely limited. Only a small portion of the research in the area of

follow-up surveys has dealt with comparisons of methodology and proce-

dures of follow-up methods, and those studies appear to have serious

limitations for utilization (Krueck, 1975, p. 4). Therefore, there

appears to be a need for a study which will compare the various meth-

odologies of follow-up, including techniques used to overcome their

limitations, to evaluate the usefulness of each method for the benefit

of those who wish to employ follow-up studies for the betterment of

vocational programs.

Statement of the Problem

The problem of this study was to compare and evaluate selected

methods of conducting follow-up studies of graduates of vocational edu-

cation. These methods included the mail questionnaire, the telephone

survey, the personal interview, and the computerized data retrieval

system. This study used five criteria to compare the four methodolo-

gies previously listed. The advantages and disadvantages of each

method were identified and evaluated for appropriate use in conduct-

ing follow-up information research. The following five criteria formed

the basis of comparison:

1. Cost. The monetary expenditures associated with the methods

used to perform a follow-up study.

2. Response rate. The ratio of completed responses to the total

number of possible responses, expressed in percentages.

3. Length of time. Estimates of length of time involved in the

methods used to perform follow-up studies data collection.

4. Personnel required. The amount of people and their necessary

skills and training required to perform the methods of follow-up

studies data collection.

5. Quality of data. The accuracy and completeness of the data



The review of related literature for this study was limited to

studies reported in the Educational Resources Information Center (ERIC)

and the Education Index for the years 1970 to 1978, and available pub-

lications in the libraries of the University of Florida and the Univer-

sity of South Florida.

It was not the intent of this study to review the basic funda-

mentals of follow-up methodologies. Rather, the most recent strategies

used in follow-up studies were reviewed to discover the best use of

each strategy. Thus, the details of instrument development, prepara-

tion, and evaluation procedures were not studied.

Justification of the Study

The addition of efficient and more complete methods of research

is an advancement in scientific methods. New and better methods of

performing research are always in demand. The gathering of reliable

and valid information is a prevalent problem associated with vocational

education follow-up studies. In addition to being complex, acquiring

reliable and valid information has also been expensive.

Follow-up studies are a type of survey research which are used to

obtain information from or about former students of educational institu-

tions. According to Berdie and Anderson (1978) there is a need for

systematic study of survey research methods so they can be used cor-

rectly and efficiently. This study systematically compared various

methods of follow-up and thereby supplemented the literature regarding

the utility of follow-up methods for acquiring information essential

to community college vocational programs. This study synthesized in-

formation concerning the traditional methods of follow-up and the more

recently applied methods. They were evaluated and compared to discover

the extent to which the more recent methods eliminated or minimized

some of the problems associated with traditional methods of follow-up.

Besides conducting a comparison among follow-up methods, this

study added to the literature about two recent developments in follow-up

methodologies. One recent method about which limited information is

available is the telephone survey. The telephone survey has been em-

ployed in a limited capacity and has experienced limited exposure in

the literature. Comparisons among the telephone survey and the other

methods in the literature are rare.

Another recently employed method with limited literature is the

computerized data retrieval system. This method includes a new source

of data for educational researchers, the state Department of Commerce

data tapes. Since the computerized data retrieval system has only re-

cently been utilized, very little research has been reported regarding

its usefulness in educational follow-up studies. This study evaluated

the extent of the usefulness of the computerized data retrieval system

and its potential as a method of follow-up for community college voca-

tional programs.

These new follow-up methods could provide information vital to

community college vocational programs in the areas of government report-

ing, self studies, and program improvement. Improved methods of infor-

mation collected could also assist in the advancement of knowledge

concerning educational benefits accruing to all students.

An important assumption for this study was that the five criteria

chosen by this researcher from a review of the literature were valid

for evaluation and comparison of follow-up methods for community col-

lege vocational education use.

Definition of Terms

Accountability. The concept that education should be evaluated in

terms of stated objectives and goals.

Community college. A two year institution of higher education

with programs designed to meet the post secondary education needs of a


Computerized data retrieval system. The process of using a compu-

ter program to locate necessary stored data, matching related data sets

from several sources, and organizing the data into usable information.

Data tape. A magnetic tape that has data stored in the form of

magnetic charges, usually containing a multitude of individual records.

Florida Department of Commerce tape. A data tape on which the

Florida Department of Commerce has stored employment data of all em-

ployed persons in the state of Florida.

Follow-up study. A study conducted to observe the subsequent ex-

periences of students after terminating their enrollment in an educa-

tional institution.

Graduate. A recipient of either an Associate of Science degree,

Associate of Arts degree, or a certificate of completion from a commun-

ity college.

Interviewer bias. The respondent's responses are affected by some

characteristic or attitude of the interviewer.

Mail questionnaire. A questionnaire mailed out to a population

or a sample of a population with a list of questions for people to an-

swer and mail back.

Monetary incentive or inducement. Offering respondents money to

respond to a mail questionnaire.

Personal interview. An interview conducted on a one-to-one corre-

spondence with the interviewer in the presence of the person being

interviewed. The interviewer asks the questions and records the an-


Response. A completed questionnaire or interview. Also, a match

in the computerized data retrieval system.

Response bias. Those who do not respond to a questionnaire or

interview may differ in nonrandom ways from those who do respond, there-

fore biasing the results.

Telephone survey. An interview conducted over the telephone

rather than in the presence of the person being interviewed. An inter-

viewer calls and asks a series of questions. The respondents answer

the questionnaire verbally over the phone, and the interviewer records

the responses.

Sample. The utilization of only a portion of a population to

study the entire population. The sample must be selected in such a way

as to meet certain criteria, in order to obtain results generalizable

to the total population.

State University System. The collection of nine public universi-

ties of the state of Florida.

Vocational education program. An educational program designed to

prepare the student for direct entrance into employment, usually at the

semi-professional or technical level. The program at a community col-

lege is typically one or two years in length.


The procedures for this study involved three major phases. The

first phase was a review of the related literature of follow-up studies.

The second phase was data organization. The third phase was an evalu-

ation of the various methods of follow-up.

Review of Related Literature

The review of related literature involved the study of four methods

of follow-up: the mail questionnaire, the telephone survey, the per-

sonal interview, and the computerized data retrieval system. A brief

description of each method was reported. Information concerning costs,

response rate, length of time, personnel required, and quality of data,

relating to the four methods was obtained and reported. Suggestions

for improvement of follow-up studies presented in the literature were

also reported.

Data Organization

Once the review of literature was completed, the required informa-

tion concerning the five evaluative criteria was organized to facili-

tate comparison among the various follow-up methods.


The costs, response rate, length of time, personnel required, and

quality of data were compared across the four follow-up methods. Each

method was then evaluated in terms of strengths and weaknesses. Recom-

mendations were made regarding the most beneficial use of each method

for vocational education follow-up.

Organization of the Research Report

This study is reported in four chapters. The first chapter in-

cludes the background and rationale of the study, statement of the pro-

blem, delimitations, justification of the study, assumptions, defini-

tions of terms, and procedures. Chapter two presents the review of the

related literature, which includes a qualitative review of the four

methods of follow-up and a review of literature concerning the five

evaluative criteria to be associated with each method. Chapter three

continues a discussion of the four methods of follow-up and a compari-

son of the four follow-up methods relative to the five evaluative cri-

teria. Chapter four draws conclusions for follow-up studies as well as

implications for further research.




A review of the literature demonstrated that educational surveys

and follow-up studies in particular are a recent development in higher

education. "The American survey of higher education, which is largely

a product of the twentieth century, is only one section of the broader

field of educational surveys in general" (Eells, 1937, p. 15). This

statement was made by Walter Crosby Eells, professor of education at

Stanford University, in his 1937 book, Surveys of American Higher Edu-

cation. The book was written as a comprehensive study and evaluation

of the survey reports that had been published up to that time. At no

point did the author mention follow-up studies, for at that time, no

studies existed.

Junior and community colleges were created early in the twentieth

century and began to expand to include vocational education at the post

secondary level during the 1920's and through the 1940's (Hillway, 1958,

p. 6). It was in this area of education that the scope of evaluation

was first broadened to include the follow-up of program completors

(Headrick, 1977, p. 2). The follow-up study has continued to develop

and expand along with vocational education and junior and community

colleges, whose growth increased tremendously during the years 1960

through 1975 (Jennings, 1975, p. 29). With the era of accountability

in the 1970's, follow-up studies were increasingly used for justifica-

tion for public funds spent on vocational programs (Headrick, 1977,

p. 3; Williams & Snyder, 1974, p. 40).

Accountability includes evaluation of educational processes and

outcomes in terms of stated educational objectives, and the communica-

tion of these findings to educators and the public (Williams & Snyder,

1974, p. 40). Because vocational education programs are relatively

new in higher education, it is considerably more difficult to develop

an accountability strategy for them than for traditional college pro-

grams, which evaluate themselves in terms of degrees produced and aca-

demic honors (Williams & Snyder, 1974, p. 40).

The need to gather accountability information for vocational edu-

cation programs was filled by the follow-up study. Originally, follow-

up procedures were employed in vocational education as a segregated

process, with little or no regard for integration with formal evaluation

or management plans (Headrick, 1977, p. 2). This restrictive approach

toward follow-up procedures has been in rapid decline as educators

become increasingly aware of the need for comprehensive management and

systemized evaluation programs (Headrick, 1977, p. 2).

McKinney and Oglesby (1971) stated a very general definition which

covers all types of follow-up studies. They stated that a follow-up

study was "a procedure for accumulating pertinent data from or about

individuals after they have had similar or comparable experiences" (p. 7).

There are various reasons for performing a follow-up study and collect-

ing pertinent information. Headrick (1977) listed four purposes for

performing a follow-up study:

1. To improve programs;

2. To provide a basis for accountability;

3. To aid in planning; and

4. To aid in decision making. (p. 3)

The Southern Regional Education Board (SREB) agreed with Headrick

that a college conducts a follow-up study for a variety of reasons.

Basically, however, according to the SREB, the main reason a college

or university conducts a follow-up study is to assess how well it has

met its objectives (SREB, 1978, p. 1). How well the objectives are

being met, which is the effectiveness of an educational program, is

important, but Brantner (1975) pointed out that before these can be

measured, the results must be observable.

The first objective of the follow-up study is to report the re-

sults of the educational programs by observing how the graduates fare

in the world of work. Once the results have been reported, then the

evaluation can be performed on the information gathered. Evaluation

should be done only on true and accurate results. Therein lies the

importance of the follow-up study, to gather reliable, accurate, and

pertinent information about the results of educational programs, and

evaluate and report on these results.

Follow-up studies for vocational programs are usually quite clear

on the type of information they wish to gather because vocational edu-

cation is well defined. Vocational education is any program whose major

goal is that the student shall attain gainful employment after a course

of study requiring no longer than two years for completion (Cohen, 1971,

p. 137). With the major goal of vocational education so clearly de-

fined as gainful employment, most vocational follow-up studies concen-

trate on employment information of their graduates.

Krisham Paul (1976) listed three questions for which most voca-

tional follow-up studies might seek answers.

1. What happens to the youths and adults who, after completion
of their training, try to find work rather than continue

2. Are there significant differences between the job-related
experiences of persons who complete vocational education
programs and those who do not?

3. Do the vocational education programs justify their costs?
(p. 6)

Most of the vocational follow-up studies that are reported in the

literature have focused on the first question concerning the experiences

of graduates who seek employment. Information on comparison of voca-

tional graduates to nongraduates is difficult to collect and the ques-

tion of vocational programs justifying their costs involves a value


The SREB stated that traditional follow-up procedures make the

assumption that the graduate's own perception of how well the institu-

tion has fulfilled its objectives is one way of assessing the attain-

ment of program objectives (SREB, 1978, p. 2). Other methods of assess-

ment that can be used are scores on graduate record exams and competency

tests. Nevertheless, the major focus of most vocational programs'

follow-up studies stresses the opinion of the graduates and concentrates

on gathering information from them (SREB, 1978, p. 2).

There are three methods of follow-up customarily used and all of

them concentrate on gathering information from graduates. These are

the mail questionnaire, the telephone survey, and the personal inter-

view (Denton, Ash, Brown, Dowell, Gutcher, Hoyle & Swinney, 1977).

These methods all depend on a question and answer format. The questions

depend on the type of information desired and the purpose of the follow-

up study. The Southern Regional Education Board has published a manual

of procedures and common core questions for follow-up studies of voca-

tional programs of two year institutions (1978). The manual includes

criteria for the selection of follow-up questions and recommends that

an institution add questions that are tailored to its own needs while

using the suggested core questions for inter-institutional comparisons

(see appendix).

The most frequently used method of follow-up is the mail question-

naire. It will be explored and reported on first, followed by the

telephone survey and the personal interview. Last to be reported on

will be the computerized data retrieval system.

Mail Questionnaire
Champion and Sear (1969) reported that the mail questionnaire has

been used for more than 40 years (p. 335). It has been used extensively

in many areas of social research and has been the most frequent method

employed with educational follow-up studies (McKinney & Oglesby, 1971,

p. 9).

The mail questionnaire procedure involves the mailing of a written

list of questions, called a questionnaire, to potential respondents.

The respondents then answer the questions by writing on the question-

naire, and mail it back.

The mail questionnaire has many advantages over the other survey

methods. It is generally agreed that the mail questionnaire is the

least expensive method of conducting a survey of a large sample or

population (Berdie & Anderson, 1974, p. 17). Compared with the second

most popular method of data collection, the personal interview, the

mail questionnaire has the following three major advantages:

1. If its questions aim at situations or decisions which con-
cern the entire household, rather than one individual,
then the mail questionnaire will offer the advantage of
opportunity for consultation within the family; the personal
interviewer rarely has a chance to obtain a "family inter-

2. More important, unavailability is sometimes a serious handi-
cap for the personal interviewer.

3. Third, its impersonality often prompts different shading to
the replies than those obtained when the respondent is part
of the psychological situation of being questioned by another
person. For some types of questions, the mail responses are
probably more specifically candid. (Franzen & Lazarsfeld,
1945, p. 294).

Benson (1946) stated that "in mail surveys there is no chance of

an interviewer intentionally or unintentionally, biasing the answers"

(p. 234). These are reasons why the mail questionnaire has become the

most widely employed method of follow-up. But, as with the other

methods of information acquisition, the mail questionnaire has its dis-


The mail questionnaire survey procedure will be reviewed using the

five variables previously chosen to compare and evaluate the various

follow-up methods. These variables are costs, response rate, length

of time, personnel required, and quality of data. A review of related

literature found a substantial amount of written material concerning

certain variables, while other variables were scarcely mentioned. There-

fore, information on each variable needed for later comparison is re-

ported, with those variables being of major interest to past researchers

having a greater quantity of research to report.


Costs for mail questionnaires were difficult to isolate and compare

across studies. Different studies included different types of costs, in-

curred in different years, and utilizing different techniques. The va-

riety of methods utilized to increase response rate made comparison of

costs across studies very complicated. Therefore, costs will first be

reported for various individual questionnaire studies, followed by com-

parison studies and cost effectiveness studies.

The SREB reported that costs of follow-up surveys vary widely, de-

pending primarily on whether or not staff costs were included. The

National Center for Higher Education Management Systems (NCHEMS) esti-

mated materials and mailing costs range from 17q to 26t per student con-

tact (as contrasted with student response) for nonprofit bulk and first

class rate mailing, respectively. This included only one mailing (SREB,

1978, p. 23).

The University of North Carolina (Chapel Hill) reported costs of

35.6t and 41.5 per student contact for surveys performed in 1976 and

1977, respectively. These amounts included only one mailing, first

class, as well as staff costs in processing, survey preparation, and

mailing. They did not include costs for analyzing responses and pre-

paring a report (SREB, 1978, pp. 23-24).

The University of Illinois reported costs of $1.86 and $1.50 per

student contact of 1972 and 1973 graduates, respectively. These costs

included three mailings and costs for staff, computer time, evaluation

and reporting (SREB, 1978, p. 24).

Navarro College, a two year institution in Texas, monitored follow-

up questionnaires and costs for six institutions while helping in the

development of the Texas Information System. The college reported that

staff costs accounted for 70% of total follow-up costs. For mailed

follow-up questionnaires, with response processed by computer, the per

student cost was calculated to be $3.27, including 70% staff costs.

Three and four mailings were included in the follow-up studies (SREB,

1978, p. 24).

Student contact was based on the number of questionnaires delivered.

Response rate refers to the percentage of questionnaires returned, a more

practical figure to use when evaluating costs. Increasing response rate

is important to researchers and the cost rise according to the attempts

and techniques used to increase the response rate. Therefore, costs

should be associated with response rate to show a more accurate indica-

tion of how much can be spent to increase the return of questionnaires.

Hochstim and Athanasopoulos (1970) reported a mail questionnaire

study performed in 1965. The total costs for the study were $37,000

for 5,630 returned questionnaires, a 70% response rate. The cost per

response was calculated by dividing the total cost by the number of

returned questionnaires to obtain a cost of $6.55 per questionnaire re-

turned. Since researchers are clearly interested in returned question-

naires, cost related to returned questionnaires is a more meaningful

figure than cost per student contact. The latter does not take into

consideration the response rate of the study.

Weiss and Hatry in their book, An Introduction to Sample Surveys for

Government Managers (1971), listed some cost ranges based on survey ex-

periences, and included other cost information that could be synthesized.

They broke down the costs to the various components of a mail question-

naire and listed them as seen in Table 1. These cost estimates were

based on 1971 cost figures. Both a second mailing of the questionnaire

and telephone calls to nonrespondents were used to increase response rate.

Also included was a sampling of nonrespondents to test for response bias,

since just over 50% response rate was achieved. Their estimate for the

cost of the mailed questionnaire was $8.10 per response.

Krueck (1975) reported a study where he compared the results of

three alternative methods of follow-up with the purpose of examining

the rate of response, costs, and nature of the response. Krueck divided

the costs into two sections, fixed costs and treatment costs. Fixed

costs were costs that applied commonly to all three treatments such as

printing and processing. Treatment costs varied depending on the pro-

cedure employed.

In Krueck's study, the mail questionnaires' initial response rate

was 37.4% of all mail questionnaires. A telephone call to remind po-

tential respondents increased the response to 42.7%. A second tele-

phone reminder brought an additional 7.3% for a total of 50.6% response.

A total of 164 questionnaires was initially mailed out at a cost of

16t each, including a self-addressed, stamped return envelope. Seven-

teen additional questionnaires were mailed after telephone reminder

Table 1
Cost Estimates For Mail Questionnaire

1. Overall planning and direction

2. Preparing listing

3. Selecting sample, addressing labels

4. Developing questionnaire (including confer-
ences and 30 pretest interviews)

5. Printing questionnaires, cover letter, return
envelopes, supplies

6. Interviewing
a. Telephone follow-ups to nonreturners
(approximately 1,200)

b. Interviews with 50 nonreturners (to check
whether they differ from questionnaire re-
(1) by telephone (approximately 30)
(2) in person (approximately 20)

7. Editing, coding, punching, tabulating about
1,000 returned questionnaires

8. Telephoning

9. Mailing, including one mail follow-up to non-
returners (approximately 1,500)

10. Analyzing data and report writing

11. Reproducing report

Average cost of the estimated
1,050 responses

For 2,000 ques-
tionnaires mailed

$ 1,500











$ 8,475


Note. From An Introduction to Sample Surveys for Government Mana-

gers, by C. H. Weiss & H. P. Hatry. Washington, D. C.; Urban Institute,

1971, p. 46.

calls. Thus, $27.60 was expended for the mailing of questionnaires.

The work involved 46 manhours for a total cost of $165.60. Therefore,

the cost per response was approximately $2.00. Krueck (1975) found

this procedure to be the least expensive of the three procedures.

Dillman (1972) performed large mail surveys in 1970 and 1971. The

costs were approximately the same both years. Direct costs for postage,

paper and printing of the questionnaires, envelopes, and cover letters,

amounted to 68U per completed questionnaire. Labor costs were figured

at the rate of $2.00 per hour and added to direct costs bringing the

total cost per response to $1.46 (p. 257).

Wotruba (1966) investigated the effects of monetary inducement and

no inducement on the percentage and completeness of responses to a

mailed questionnaire. He performed a pretest to ascertain how many

questionnaries needed to be mailed with 25t included, a promise of 50,

or no monetary inducement, to obtain a return of 750 questionnaires.

The pretest results demonstrated that with a 250 monetary inducement,

1,875 questionnaires would have to be sent out to obtain 750 completed

questionnaires. With a promise of 50t, 3,750 questionnaires would

need to be sent out, and if no monetary inducement was included, 4,167

questionnaires would need to be sent out. Wotruba's results showed

that the approach using the 25t inducement would incur the least ex-

pense when completeness of responses (all questions answered) was im-

portant. If completeness was not important, the approach with no

inducement was least expensive. Table 2 presents a comparison of costs

between the three inducement alternatives. Second mailings to compen-

sate for incomplete questionnaires were included.

S- 24
L L 000 0 o C n L LO -W C0 n LI
E o0 o oCo iLn n L o Co
Lim cr L tl 0 a lO Cr- X 00rO -O- X 0C) 0
3 C) 0 o CM in m- C (v i cn oo
S .0 co m Or- LO i-
C <30 0 .0 4-
9- i-X 0

20 0

w 0 *+)
4 ( -0: -C C

E *r- Um U" LO LO m o 0M r., r- r~% r% Ln LO 3
S a) a) 4
l0 000 C0o f% CMMM / 0LO0 0n
L Ll 0 o -f- CM Ul *
Io U- r -- C% M o ,- I- r- CM un
Sf 0 CM N" C. a C
-- IC < r*. C ..- i-


0 c


S r->. In ^
a) +41 Ln COOOC 0 0 00 .0-U- C O* a)
4- c ri. cMLno- CD r-, t.Or-LOl cnd- sl-
4- a .
*r I OLC) CO ;-CJ0; X O k.Ln X 3 cc; Xm x CV m
0M tru 0 Ln co Oi a% C\j c C:
N c 00 dLn cO 00 0) C
mCT) a- CM r o
a) c CM -. -*

-I -

0 *

V)) C
0 ) 0 -

0C 0. r
4-) a) U) (U
(/ L/ + a4 -)
C) a) La) )
L C r- a

So -+j 0= 0
C u in 2a) n D en
.. in t- -a C
4- -f 0a) 3S-
0 oo h- c* r E 2 S.
C O R. QS-- L La )
o U- (D a 4.3 4.3 |
( C O -0) 0 CC aa)
*i-cu r 5n *SL E a (V re i- s- "
s. Cd) y-.r- r-. ca 0) -fyt--.. c Q. 'U '.0
r_ C 4C) + *- C:-P U) 0 E 413 to
S0 C4-J 0 ECL 0 Si 0l L) i- ( U 14-) U C -
0 Cda a) 41 m.'4.J 0
0 f. Ca) (a "a r +- a
4J 4- +J3 o m 4- +3 -0i = S
+.p_-r_ ) c O: C i C: E S-
Sto -r- .V) 4-)- W CL S- S- 0 'U
a)2 E cU 4- E Li ea 2 00 a)c
U 0t- ) 4-Ca) r4- (D ca a) C5 4-4- LL. in
S- 0 L S-3L O C f S- S- U a
CU+4 +3 J QUJ 4 3) 4 + 3 3 c) CL U,4.3 4 f+4J-0 4-)a (V+J4) 4- 'a 4+4+
>> 0 ( cu c Ul cC c ct 0n m w )o
a S- 0S OU C 0 0 4 C
u co U 4-0 00 0 O
+J C, .. *r *. Z +4
VU)) a-a4- 01 4) a a)
(D 4-) (a U .- +j > o
CLu Ln 4+ L.3 (A 4+) 44
Xa)0O O a0 0 00 0
LUI = ) I-Q-CL I-c- CS)

Ronald Baugh (1976) performed a follow-up cost study which com-

pared the cost of mail questionnaires to personal interviews, as well

as manual versus computer analysis of follow-up results. The results

of Baugh's study can be seen in Table 3. A comparison showed the per-

sonal interview costs to be 3 1/2 times more expensive. The cost for

a mail questionnaire with computer processing was $8.20 per response

and for manual processing was $4.85 per response.

Baugh (1976) concluded that there must be some optimum number of

responses where manual processing is less expensive than computer pro-

cessing, however; above the optimum number of responses, computer pro-

cessing would be less expensive than manual (p. 10). He also reported

that mailing bulk rate did not save money due to the use of personnel

to sort and bundle the bulk mail.

Christison (1977) performed a cost effectiveness study for follow-

up questionnaires with various methods to increase response rate. He

concluded that a reminder phone call, a second copy of the cover letter

and questionnaire, and prior contact by phone before mailing the in-

strument were all cost effective. High cost and high returns were as-

sociated with offering several dollars as monetary reward. Least

expensive, but with low return, was a single mailing of the cover let-

ter and questionnaire with no further contacts. The costs associated

with Christison's study are listed in Table 4.

Erdos (1974) stated that low overall costs result in more flexi-

bility per dollar spent for the mail questionnaire. The combined costs

of mailing, postage, and incentives offered are usually less than hir-

ing staff for personal contacts. Because of this difference in expense,

Table 3

Costs for Mail Questionnaires and Personal Interviews

Mail Questionnaire

Computer processing

Manual processing


Attempts Responses

14,234 7,780

227 125

Total Cost per

Attempts Responses

4.482 8.201

2.672 4.852

Personal interview

Computer processing 271 271 28.672 28.672

Manual processing 0 0 0 0

Note. From "Follow-up Cost Study" by R. C. Baugh, Forth Worth,

Texas, Tarrant County Junior College, 1976, p. 26.








































01 u-
1 (
r0 0
(0 o





-o 0


S r-
S r:

CL 0
I-3 0
01 r
C *W-

-c en
Q. s
01 *i
'- >
01 10
I- Q

0o C co co UL co r
Mv Ln IC Ll) k0 ko rO

CM (> o o o

SLY LO 0 0 0 0C

00 C0 0 0 0 0

co (NI 1.0 C a 0 <
; (v, C;00

LO uO) L L LO n n

Cej C(N C.J C"j C\J C\M C\j

4 -


















one could afford to produce a large number of completed questionnaires

by using the mail method (p. 2-91).

When looking at costs of mail questionnaires in relation to the per-

sonal interview, most researchers agreed that the mail questionnaire costs

less than the personal interview (Benson, 1946, p. 234). Berdie and

Anderson (1974) concurred with the above statement. They stated, "the

expense of printing questionnaires and distributing them to large numbers

of people is considerably less than that of interviewing similar numbers

of people" (p. 17). Gordon (1975) agreed, summarizing the advantage of

the mail questionnaire as being its economy (p. 77).

Response Rate

Kerlinger (1973) stated that the mail questionnaire has serious draw-

backs, the primary defect being lack of response, or not enough question-

naires being completed and returned. Most researchers criticized the

mail questionnaire basically for the low response frequently encountered

and for the ease of misinterpretation. Mildred Parten (1950) went so far

as to describe mail questionnaires as almost invariably open to question

(p. 400). As stated by Hackler and Bourgette (1973) the problem with re-

sponse rates was that those who did not respond might differ in nonrandom

ways from those who did respond (p. 276). This brings response bias into

the questionnaire results. Investigators who have supported the notion

of bias produced by nonresponse were listed by Leslie (1972) as Rothney

and Mooren, 1952, Bennett and Hill, 1964, Anastasiow, 1964, Norman, 1948,

Suchman and McCandless, 1940, and Pucel, Nelson, and Wheeler, 1971. Leslie

(1972) questioned these researchers and their common view that low response

necessarily biases the results.

According to Leslie (1972), the studies that have dominated the

theory of response bias are based primarily on the late 1940's and

early 1950's studies. He concluded:

The evidence of bias in these studies, however, was in most
cases the demonstration that demographic traits of the re-
spondents, differed from non--or late respondents. Differ-
ences between respondents and non--or late respondents, ac-
cording to answers to substantive questions posed, were not
usually shown. (p. 327)

Leslie (1972) developed the following generalization to counter-

act the prevalent view that response bias severely limited the useful-

ness of surveys and questionnaires which did not have 100% return:

1. There is ample evidence that response rate bias may occur
in mail surveys. However, much of the available evidence re-
veals only differences between respondents and nonrespondents
or late respondents in terms of such independent variables as
sex, geography, age, etc. It is often assumed that these dif-
ferences lead to differences between respondents and nonrespon-
dents on the dependent variables, i.e., the questions under study.

2. When populations surveyed are homogeneous (having a common
group identity), minor differences on independent variables be-
tween respondents and nonrespondents or late respondents may oc-
cur, but differences as to dependent variables are unlikely.
(p. 328)

Leslie's (1972) conclusion was that researchers surveying issues

directly related to homogeneous groups should not be overly concerned

about the percentage of questionnaire returns, just that enough re-

sponses are gained to meet statistical assumptions. Benson (1946)

said something similar to Leslie's conclusion. Benson stated:

If the segment of the population to be surveyed is homogeneous,
the differences in tastes or opinions within that group are
much more restricted. Therefore, in a mail survey, results
for returns, from a small proportion of such a group would tend
to be more like the whole than is the case when tastes or opin-
ions within the group to be sampled are very different. (p. 237)

Many researchers still consider response bias the primary weakness

of the mail questionnaire. Wallace (1954) stated:

There is present in the response to any mail questionnaire an
important number of persons who reply to most of the
mail questionnaires they receive. These persons have higher
education and, very probably, certain other distinctive charac-
teristics. (p. 51)

Berdie and Anderson (1974) reported that many researchers have

found that nonrespondents are usually the least educated members of the

sample (p. 52). Baur (1947-48) compared early to late respondents and

found that the less educated and the married respondents were underre-

presented in the early returns, and overrepresented in the last returns.

Hochstim and Athanasopoulos (1970) performed a study on mail ques-

tionnaire response bias by using personal interviews for follow-up on non-

respondents. They found a slight response bias (p. 74). Personal con-

tact to retrieve the questionnaires by personally interviewing the non-

respondents tended to improve estimates associated with low socioeconomic

studies, since this stratum was least likely to respond by mail. It

also introduced some not-at-home bias characteristic of personal inter-

view surveys. This caused the final sample to be slightly overweighted

with large families and women.

Hochstim and Athanasopoulos (1970) found the largest difference

between the mailed response and total response after personal contact

to nonresponses occurred in two categories:

1. Group activities -- people who reported no organizational
membership and those who indicated no political activity were
less likely than others to respond by mail.

2. Socioeconomic characteristics -- a similar tendency was
shown by people with low education, job security, and with-
out provision for medical care. (p. 77)

Hochstim and Anthanasopoulos (1970) made the following tentative


Surveys conducted entirely by mail can achieve a good repre-
sentation of the universe. When parameters are not known,
careful evaluation should be made of differences in response
to substantive data by various socioeconomic and other rele-
vant groups in the sample to see whether a bias in the total
return would exist if such groups were over -- or underre-
ported. (p. 81)

Other studies have suggested that response bias is not slight but

can be substantial. Weiss and Hatry (1971) reported that people with

extreme views are more apt to respond to questionnaires than people

with moderate views. They also reported that older and less educated

people as well as women respond less frequently to mail questionnaires.

These differences between respondents and nonrespondents would certainly

cause a response bias. Most researchers still regard the low response

rate of questionnaires as the major drawback of the mail questionnaire

in survey and follow-up research.

To avoid the problems of response bias, the Advertising Research

Foundation recommended an 80% or better response rate for mail question-

naire surveys (Erdos, 1974, p. 2-102). They stated that this would

bring the rate of nonresponse into line with the rate of substitution

in well-conducted personal interview studies (p. 2-102). To be confi-

dent of an unbaised representation, Kerlinger (1973) stated that "every

effort should be made to obtain returns of at least 80 to 90 percent or

more, and lacking such returns, to learn something of the characteristics

of the nonrespondents" (p. 414).

The SREB suggested a check for nonresponse bias when response is

not as high as desired. The SREB reported several ways in which

nonresponse bias may be determined without tremendous expenditure of re-


The most direct check is to select a limited random sample
among the non-respondents for telephone interviews in order
to obtain responses from some non-respondents and then to
determine whether their responses follow the same pattern as
from the respondents. Often in such telephone interviewing,
the length of call can be reduced by asking selected questions
instead of the entire series of the survey form. A statistical
test (such as the chi-square test) is then used to determine
whether there is a significant difference between the survey
responses from respondents and non-respondents. A "sequential
non-respondent sampling" telephone technique, that takes the
place of a statistical test, has been used in Maryland in con-
junction with a state-wide survey of community college students.
This sequential sample technique essentially sets an arbitrary
range of tolerance variation around the results obtained on
any one question from the respondents. Cumulative results
from the responses to a question from telephoned non-respon-
dents are computed until this response result stabilizes (per-
haps after 30-50 calls). If the stabilized cumulative response
result from the non-respondents falls within the variation
tolerance previously established, then the respondents' and
non-respondents' responses are considered essentially the
same. (p. 28)

Another way for checking for nonresponse bias was to use demographic


If the demographic characteristics for respondents and the
total population in the survey are essentially the same, then
the inference can be made that the responses are representa-
tive of the total population, at least on the basis of their
demographic characteristics. In order to check for demo-
graphic differences between respondents and the total popu-
lation, demographic averages on various characteristics are
computed for the respondents, and then compared to those
for the total population. A chi-square test is applied to
check whether the resulting differences are significant.
(pp. 29-30)

If the demographic characteristics of the respondents and the total

population differ, further checks should be made using substantive

questions compared by sex, race, grade point average, or other vari-


The last way to check for response bias suggested by the SREB was

to check for differences between early and late respondents.

The inference is sometimes made that since non-respondents
might have more in common with late respondents than with
early respondents, therefore, if the responses from late
respondents significantly differ from those given by early
respondents, this might indicate non-respondents' bias. In
order to use this method to check for non-response bias it
is necessary to number responses sequentially as they arrive.
Survey results might then be compared for those graduates
that were among the first one-third to reply. If responses
are not significant different, the assumption is made that
the non-respondents would also not be significantly differ-
ent. (pp. 30-31)

Baur (1947-48) performed a study using a questionnaire to survey

veterans on their plans for education and training after leaving the

service. To analyze the amount of response bias, his sample was

divided into five groups according to the number of days that elapsed

between the original mailing date and receipt of the questionnaire.

Baur found that the greatest bias was introduced by differences in

interest in the subject of the questionnaire. Those without definite

plans for education or training were underrepresented in the early

returns and overrepresented in the late returns. Therefore, he con-

cluded that the smaller the proportion of returns in a mail questicn-

naire on a subject of interest to only a part of the sample, the

greater the bias attributed to differences in the subject. Baur also

found that the less educated and the married respondents were under-

estimated in the early returns and overrepresented in the late returns.

Nielsen, Moos, and Lee (1978) performed an extensive mail question-

naire follow-up study on college students from their freshman to senior

year. They found significant differences between early and late re-

spondents on nearly a dozen variables. They stated that their results

indicated that researchers cannot account for follow-up nonresponse

bias by making statistical adjustments according to data available at

initial testing. They suggested more study needs to be done to develop

categories of nonresponders who may be motivated to cooperate by differ-

ent types of follow-up techniques.

Many researchers with low response rates preferred to make extra

efforts to have more questionnaires returned to increase the accuracy

of their study, but these extra efforts increased the costs of follow-

up studies (Weiss & Hatry, 1971, p. 23). Hochstim and Athanasopoulos

(1970) performed a mail questionnaire study in which the initially re-

turned questionnaire cost $6.55 per questionnaire. A personal inter-

view follow-up of nonrespondents to the mailed questionnaire increased

the rate of response, but the additional questionnaires filled out

during personal interviews cost $13.64 each.

Bower and Renkiewicz (1977) reported that extra efforts did in-

crease response rates in mail questionnaires and encouraged researchers

to use them. Researchers have investigated a wide range of variables

in trying to increase the response rate for mail questionnaires. Among

the variables were monetary incentive, first class versus third class

mailing, stamped versus business reply envelopes, advance letters, per-

sonal versus nonpersonal wording of the cover letter, timing of the

questionnaire and follow-up mailings, length of the questionnaire, re-

peated mailings, and intrinsic interest.

Monetary incentive. Hackler and Bourgette (1973) applied a theory of

dissonance to mail questionnaire response. They theorized that sending


a dollar to potential respondents along with a questionnaire would

cause the respondents to feel cheap if they kept the dollar and did

not return the questionnaire, therefore causing internal dissonance.

According to Hackler and Bourgette, the dollar was not a great enough

reward to justify the mildly improper behavior of not returning the

questionnaire. Therefore, to reduce the dissonance felt by the re-

spondents by not returning the questionnaire or the dollar, the respond-

ents would return the questionnaire.

Hackler and Bourgette (1973) sent one group a dollar with the

questionnaire and the other group received the questionnaire without

the dollar. They expected those who received the dollar would exper-

ience dissonance, and to reduce their dissonance, would return the

questionnaire. Their results confirmed their expectations. The dollar

group returned 71% of the questionnaires compared with 39% return with

the nondollar group. When the nondollar group was later sent a dollar,

their response increased.

Huck and Gleason (1974) also performed an experiment utilizing

monetary incentive to increase response rate. They used four differ-

ent groups. The first group received a quarter on the first mail out,

the second group received a quarter on the second mail out, the third

group received a quarter on the third mail out, and the fourth group

was the control group, receiving no quarter with three mail outs. Huck

and Gleason's results were that on the first mail out the group receiv-

ing the quarter had a return of 78% with the other three groups only

receiving a return of 50%. The difference was significant. After the

second mailing, with the second group receiving the quarter, both the

first and second groups were significantly higher in responses than the

third and fourth groups. The same was true for the third mail out.

Therefore, Huck and Gleason recommended sending a monetary incentive

with the second mail out to get the highest response for the least cost.

Wotruba (1966) performed a study to investigate the effects of two

types of monetary reward or inducements, and no inducement, on the per-

centage and completeness of response to a mail questionnaire. His re-

sults were that sending a monetary inducement with the questionnaire

produced a better return of the completed questionnaire. There was no

significant difference between the promise to send a monetary induce-

ment and no inducement at all. Wotruba concluded that the value of an

immediate inducement seemed to be stronger than the monetary value of

a delayed inducement, as long as the immediate inducement was not un-

reasonably small. He also concluded that when completeness was consid-

ered, the desired number of responses was obtained with least overall

cost using the immediate inducement. If completeness of the question-

naire was disregarded, the no inducement approach was least expensive.

Kephart and Bressler (1958) reported that the inclusion of a

quarter with a questionnaire could augment or even create another re-

sponse bias among the receivers. They suggested that before researchers

include a monetary inducement into their survey, they should consider

that it is quite possible that people who are influenced by a quarter

or by any other type of inducement may differ from those who are not

so influenced.

Wiseman (1973) performed a study involving four variables. His

results were that a 10t monetary incentive and a post card reminder

improved response rate by 8%. Wiseman also offered to return survey

results but found this had a slightly negative effect on response. The

overall results were that respondents given a 10 monetary incentive,

a post card reminder, and a stamped return envelope demonstrated the

highest response.

First class versus third class mailing. Gullahorn and Gullahorn

(1963) noted that first class mailing of a questionnaire elicited a

somewhat higher return than third class mailing (p. 294). Warwick and

Lininger (1975) agreed that higher returns were noted when question-

naires were sent out first class mail rather than third class mail

(p. 131). The SREB supported this finding: "First class mail is more

personal and also has the advantage that the post office will forward

such mail to a forwarding address, which is not done for bulk rate

mailing" (p. 22). But, the SREB stated, bulk rates cost considerably

less than first class rates. The SREB made the following recommenda-


On a bulk rate mailing envelope, stamp the message, "Return
Postage Guaranteed." If the addressee is not at the address,
the letter will be returned to the sender with the reason for
nondelivery. The sender pays a 45 charge for this service.
Re-address the letter and send it out first class this time,
and now the post office will forward it to a forwarding
address, if the post office has one. (pp. 22-23)

Return envelopes. In regard to return envelopes, Gullahorn and

Gullahorn (1963) reported that more questionnaires with stamped return

envelopes were returned than with business reply envelopes. In Wise-

man's 1973 study, the stamped return envelopes produced a 13% increase

in response rate. Champion and Sear (1969) found that response rates

were greater when regular stamped envelopes and special delivery postage

were used. Selgas (1977), using cover letters, stamped addressed

return envelopes, and two mailings obtained a 70% response rate with

his mail questionnaire. Warwick and Lininger (1975) also reported that

stamped, addressed, return envelopes, rather than conventional business

reply envelopes, were one of the more effective devices to increase

response rate (p. 131).

Advance letter. McKinney and Oglesby (1971) suggested the use of

an advance letter to be mailed out before the questionnaire. This ad-

vance letter or "alert card" informed the student that an important

questionnaire was soon to arrive and their cooperation in filling it

out and returning it was vital.

Ford (1967) performed a study observing the effects of an advance

letter on response rates, quickness of return, completeness of the

responses, and the accuracy of the responses. His study demonstrated

a significant increase in the rate of return with the use of an advance

letter, but did not find a significant effect with the other three

factors. Erdos (1974) also reported that the use of an advance notice

sent prior to the questionnaire improved returns of the questionnaire

(p. 2-100).

Parsons and Medford (1972) stated that "previous studies of ad-

vance notice in mail questionnaire surveys have shown significant im-

provement in response rates" (p. 258). They suggested that when the

sample is drawn from a fairly homogeneous population, the advance

notice does not always improve the response rate. They conducted two

experiments where one group of a homogeneous sample received advanced

notice of the upcoming questionnaire and the other group of the sample

received no advance notice. They found in both experiments there was

no significant increase in response with the advance notice group.

They suggested their findings indicated that advance notice was not

necessary when the sample was from a fairly homogeneous population.

Cover letter. An important part of most mail questionnaires was

the accompanying cover letter. Berdie and Anderson (1974) stated that

the cover letter should be short and to the point (p. 59). According

to the SREB (1978), the letter set the tone for the message from the

institution to the students. The president's signature on the cover

letter increased its importance in the view of the student. The cover

letter should also stress the confidentiality of the student's re-

sponse and emphasize the value of the individual student's evaluation

of the institution. Francel (1966), a professional survey conductor,

stated the most important feature of a cover letter should be its

friendly, personal tone. He also reported an action phrase, such as

"Won't you do it now while you're thinking about it" and stressing the

importance of the individual respondent's opinion would have a positive

effect on the response rate.

Various studies have found that a personally worded and typed

cover letter did elicit a higher response rate. Edwin Carpenter (1974-

75) did a study involving different levels of personalization. He

came to the conclusion that reducing personalization would reduce re-

sponse. If the response to the questionnaire was high, in tnis case

70% or better, then the difference in response rate would not be

substantial, but at lower levels of response rate, the difference would

be substantial.

Arnold Linsky's (1965) experiments consisted of four factors and

their effect on response rate. He found that personalization and empha-

sizing the importance of the respondent (the person who receives the

questionnaire) had a significant positive effect on the response rate.

Emphasizing social utility and an appeal to help the researchers had no

effect on the response rate. Champion and Sear (1969) confirmed Linsky's

results in that they found that response rate would be greater when the

respondent's importance to the study was emphasized.

Frazier and Bird (1958) tested the use of a handwritten postscript

for increasing response rate. The postscript read "P.S. We need your

help in this report. Could you please send it in promptly?" They found

a 27% increase in response with the addition of the handwritten postscript

(p. 187).

Other researchers suggested that personalizing a cover letter does

not have as much effect on response as it has been given credit. Andreasen

(1970) stated that most researchers assume the effect of personalization

is positive on response rate. He reported that many cover letters are

careful to emphasize anonymity for respondents. A personalized cover

letter could negate the advantage of anonymity of a mail questionnaire.

Andreasen did a study using a questionnaire sent out to lottery winners

under the assumption that they would prefer anonymity because of the

harassment from sales agents. He varied three levels of personalized

cover letters with the questionnaire. His findings were inconclusive,

though the most personalized, handwritten form of correspondence was

somewhat less effective in generating response. He found it took a

significant dose of personalization to secure a moderately significant

effect on response rate. Therefore, he concluded that the costs of per-

sonalization usually are not justified by their benefits and adds that

in some studies, and for some respondents, the expenditures for person-

alization may have the opposite effect than the one desired.

Timing. Timing the questionnaire arrival could also have an ef-

fect on the response rate. The SREB (1978) reported that a questionnaire

timed to reach students around the middle of November had the advantage

of perhaps finding them at their parents' homes, which might correspond

to the permanent address of the student on college records. "Some insti-

tutions have found the response rate increased when a mailing was timed

for the Thanksgiving holiday" (p. 19).

Reminder postcards have frequently been employed to increase re-

sponse rate. Nichols and Meyer (1966) performed a study with various

time schedules for sending out the reminder postcards. These postcards

included a "thank you" to those who had promptly returned the question-

naire and a request to those who had yet to return the questionnaire

to do so. Nichols and Meyer found that a thank you and reminder post-

card, sent three days after the questionnaire mailing was the most ef-

fective time for increased response. The best overall response came

from using both the three day postcard reminder and a later (45 days)

second mailing of the questionnaire.

Length of the questionnaire. The general opinion on the length

of the questionnaire has been the shorter, the better (Erdos, 1974;

Filion, 1975; Parten, 1950; Robin, 1965). This assumption was based on

the theory that respondents would be more willing to fill out a

questionnaire that would take less of their time than a longer, more

time consuming one. According to Erdos (1974) a questionnaire that be-

came long, six pages or more, should be administered by a well trained

interviewer who can maintain the respondent's interest (p. 2-92). He

further stated that when the mail questionnaire becomes too long, the

percentage of response falls off sharply.

Sletto's (1940) findings suggested the length of the questionnaire

was not an important variable in questionnaire response rate. Clausen

and Ford (1947) further challenged the widespread belief that a shorter

questionnaire would elicit a higher response rate. They showed that a

multi-subject questionnaire obtained a larger response in its initial

mailing than a shorter, single subject questionnaire. They also re-

ported that one or two pages of supplementary questions did not affect

responses to a three to six page questionnaire.

Champion and Sear (1969) varied the length of a questionnaire in

their follow-up study after they noted that other researchers reported

that generally shorter questionnaires were returned more frequently.

They performed a study utilizing questionnaires of three different

lengths. Their results were that the longer questionnaires were returned

most frequently, both in the initial and the second mailing. These

results differed considerably from accepted beliefs about mail ques-

tionnaires. The SREB stated that there is a consensus among experts

on surveying, that shorter questionnaires yield higher response rates

than longer questionnaires (p. 12). The Champion and Sear (1969)

results and the Clausen and Ford (1947) results indicated that the

issue of questionnaire length was more complex than first realized and

deserves further study.

Dillman (1972) took his questionnaire and had it and the cover

letter photographically reduced and put in booklet form to make the

questionnaire appear smaller. He performed two studies, one in 1970

and one in 1971, and obtained a response rate of 75% for each study

(p. 255). The appearance and form of the questionnaire should also be

considered for size and respondent's interpretation of the size,

according to Dillman (1972). Erdos (1974) stated that the layout of

a questionnaire should be designed to avoid a crowded appearance (p. 2-


Repeated mailing. The SREB (1978) reported that all institutions

have found that repeated mailings of questionnaires did improve re-

sponse rates:

The added cost of extra mailing, however, must be weighed
against the improvement in response rates. A minimum of
two mailings is universally recommended by those with ex-
perience. Responses continue to rise after two mailings,
but at diminishing rates. (SREB, 1978, pp. 19-20)

Some studies use a reminder postcard as the second mail out, but the

SREB suggested that each mailing include a questionnaire. A suggested

time limit for returning the questionnaire should also be included, with

ten days to two weeks being suggested by the SREB. Selgas (1977) in his

follow-up study, included two mailings of the questionnaire with a

cover letter and a stamped addressed return envelope. He separated the

mailings by one month and obtained a 70% response rate to deliverable

questionnaires. Cox (1966) reported that previous recommendations were

to schedule one month for a two wave mailing. But, if projected time

was critical, then a two wave mailing could be conducted in a two week

period without too much loss of response since a majority of the returns

arrive soon after the mailing.

McKinney and Oglesby (1971) provided a list of recommendations for

performing follow-up studies which stressed repeated mailings. Once the

reason for the follow-up questionnaire was determined, the population to

be studied should be chosen. The next most important part of any follow-

up study was selecting the information gathering instrument. A useful

procedure for developing a follow-up questionnaire can be found in McKin-

ney and Oglesby's 1971 article, as well as in the Follow-up Surveys of

College Graduates by the SREB (1978). McKinney and Oglesby list recom-

mended steps for performing a follow-up study. They started with what

they call an "alert card," which was an advance letter informing the stu-

dent of the upcoming questionnaire and its importance. The student was

urged to complete and return the questionnaire. Following is the order

suggested by McKinney and Oglesby for performing a follow-up study using

a mail questionnaire. They suggested the steps be performed at one week


First mailing "Alert card."
Second mailing Follow-up instrument, cover letter,
and return evelope, stamped and ad-
Third mailing First thank you reminder card.
Fourth mailing Second request follow-up instrument
(second cover letter, and return en-
velope stamped and addressed).
Fifth mailing Second thank you reminder card. (p. 16)

Interest. Warwick and Lininger (1975) reported from their own obser-

vations that the greater the intrinsic interest of the subject matter to

the respondent's personal interest, the better the chances of a high re-

sponse rate (p. 132). Berdie and Anderson (1974) reported that offering

the respondent the results of the study can sometimes stimulate interest

and response (pp. 58-59).

Baur (1947-48) reported that interest in the subject of the ques-

tionnaire can increase response bias. He found that when comparing

early to late respondents, those interested in the subject returned

their questionnaires early while those not as interested returned theirs


Monetary incentive, first class versus third class mailing, stamped

versus business reply envelopes, advance letter, personal versus non-

personal wording of the cover letter, timing of questionnaire and fol-

low-up mailings, length of the questionnaire, repeated mailings, and

intrinsic interest are all different tactics for increasing response

rate. Berdie and Anderson suggested the use of varying tactics for dif-

ferent samples and studies to maximize response rates (1974, p. 53).

Cost Effectiveness

Rossmannand Astin (1974) performed a study where they compared the

different tactics and techniques for increasing response rate to the

costs of performing those techniques. In their study of cost effective-

ness, they investigated the cost in comparison to the response rate of

first class versus non-profit outgoing postage, business reply versus

live stamp return, matched insertion in envelopes versus window envelopes,

auto typed letters versus personalized letters, one mailing versus two

mailings (with and without a reminder postcard), and types of non-profit


The results of theRossmann and Astin study (1974) generally held

that the least expensive methods of performing a mail questionnaire

follow-up were the most cost effective. In comparing first class post-

age to non-profit postage, the results showed that using first class

postage caused an increase in response rate that varied from 0.2% to 3%.

The cost per response, however, favored non-profit postage with the sav-

ings varying from 8 to 18U. In the comparison of business reply versus

live stamp return, they found that live stamp increased response rate by

no more than 2%, but costs were considerably more, varying from 144 to

234 higher per response.

Their matched insertion versus window envelopes comparison found the

additional cost of matched insertion was approximately $15.00 per thou-

sand questionnaires. The results favored the window envelope with re-

sponse rate increasing slightly for window envelopes and cost per response

between the two techniques being similar.

The use of auto typed letters, with a personally addressed question-

naire included in the second wave,resulted in the highest rate of return,

but also had the highest cost. The 3% increased response rate hardly

seemed to justify the increased cost of 59t per response, according to

Rossmann and Astin (1974).

A postcard reminder, but no second mailing of the questionnaire de-

creased costs, but the response rate decreased 10% below the use of a

second mailing of the questionnaire. The 10% difference in response rate

was deemed, by Rossmann and Astin (1974), worth the extra cost for the

second mailing of the questionnaire. No postcard reminder with no second

mailing of the questionnaire had the lowest cost, but also the lowest re-

sponse rate. It was observed that a postcard reminder between the first

and second mailing made little difference in the results. Therefore, a

second mailing of the questionnaire, without a postcard reminder, seemed

to provide the maximum benefit to response rate without unnecessary addi-

tions to costs.

Rossmann and Astin (1974) also used three types of nonprofit per-

mits: printed, precanceled stamps, and meter. The data showed the

difference among the three methods was small, with a slightly higher

response rate produced with precanceled stamps.

The overall results of the Rossmann and Astin (1974) study were

that the use of nonprofit outgoing postage, window envelopes, and

business reply return postage resulted in considerably lower costs per

contact and per respondent, and did not lead to significant lower re-

sponse rates. The use of a second mailing of the questionnaire proved

to be an effective technique for increasing the response rate while

the use of reminder postcards and various types of nonprofit postage

had little impact.

Length of Time

Dillman (1978) reported that mail questionnaires are usually

locked into a definite time schedule of printing, folding, stuffing,

and mailing dates; and the different methods for increasing response

rate can affect the length of time involved in a study (p. 67). Exact

time limits are not available. Parten (1950) reported that mailing

costs are low compared with the transportation, as well as time costs,

associated with studies which require a field staff. The expensive

and time consuming task of training a staff of investigators is absent

with the mail questionnaire (p. 94). Time involved in performing a

follow-up study is associated with costs since the longer a staff mem-

ber is on the payroll, the higher the costs. The lower costs of mail

questionnaires could be associated with less time involved needed to

perform a follow-up study.

Jackson and Rothney (1961) stated the economic advantage of the

mail questionnaire was in saving cost and time (p. 571). Kephart and

Bressler (1958) stated the reason the mail questionnaire has come to

be favored by social scientists is for the expediency of the study

(p. 123). According to Kephart and Bressler, information can be pro-

duced more quickly by the mail questionnaire than by interview methods

(p. 123).

Berdie and Anderson (1974) stated the use of mail questionnaires

could facilitate collecting data from an extremely large sample in a

short period of time (pp. 17-18). However, they continued, "with small

sample sizes, questionnaire administration may be more time consuming

than interviewing each subject" (p. 18).

The length of time for the development of a mail questionnaire

does vary. The development of an instrument and planning the follow-

up study depends on the situation and people involved, and will not be

dealt with here. The time involved with the actual performance of the

study will be considered the length of time of the follow-up study.

The start of a mail questionnaire follow-up study shall be when

the first mail out of the questionnaire takes place. According to the

National Opinion Research Center, six weeks to three months are re-

quired from the start of a follow-up study until the data are ready for

final form. Two to three weeks are required before the last completed

questionnaire is received, and from one week to a month is needed to

tabulate figures and compute percentages (Parten, 1950, p. 61).

Nichols and Meyer (1966) studied the effect of a third day post-

card reminder and a 16th day postcard reminder on response rate.

By the 16th day, 58% of the group receiving the third day postcard re-

minder had returned their questionnaire, but only 37% of the group not

receiving the third day postcard had returned their questionnaire.

After 45 days, there was no difference between the two groups. The re-

sults suggested that a third day postcard elicited the highest rate of

return in the shortest period of time.

Dillman (1972) included a one week postcard, a three week letter,

and a seven week mailing of a second questionnaire in his follow-up

study. The response time after this final mailing would extend the

length of this study to approximately ten weeks, after which would

come the evaluation. Dillman (1978) listed eight weeks as the time

needed to conduct the implementation of his total design method of

conducting mail questionnaires (p. 67).

According to some authors, the time involved with the actual

questionnaire distribution and collection can be shortened. Stanton

(1939) noted that 90% of the total possible returns to a mail ques-

tionnaire would be returned within three weeks following the first

mailing of the questionnaire. Francel (1966) reported that it is well

known that in mail questionnaires, the majority of replies tend to

come in during the first week after mailing. The remaining come in

at a rapidly diminishing rate over the succeeding three or four weeks

(p. 89).

Boyd and Westfall (1964) reported that a minimum of two weeks was

usually necessary before the bulk of the returns from one mailing were

in. Scott (1961) reported that most researchers seem to assume that

the best policy was to wait until returns have almost stopped before
sending the second follow-up (p. 166).

Cox (1966) performed a follow-up study to investigate response

patterns with mail questionnaires. He reported that the approach of

waiting for the returns to slack off could cause unnecessary rejection

of mail surveys as a research method when time was very short (p. 392).

He reported that previous studies of response patterns to mail ques-

tionnaires have resulted in recommendations that one month be used for

a two wave questionnaire (p. 396).

Clausen and Ford (1940) reported that mailing intervals may be cut

down by perhaps one half. A second mailing can be made before the

first returns have finished coming in. The drawback is that some ques-

tionnaires will go needlessly to those who are in the process of re-


The data from Robinson and Agisim's study (1951) showed that re-

sponse is low during the first six days, but there is an increase in

cumulative returns on the seventh day. This seems to be a standard

response pattern. According to Cox (1966), these data suggest that it

would be unwise to employ a mailing interval of less than seven days.

Cumulative returns in the Robinson and Agisim study exceeded 90% on

the 14th day. Cox (1966) concluded that the optimum mailing interval

therefore should be between the 7th and 14th day after the previous


Gray (1957) found that response patterns for the second mailing

of a questionnaire are almost identical to the pattern of the first

mailing. Therefore, according to Cox (1966), the same mailing inter-

val can be used for all mailings. Cox went further to point out that

if there were no significant differences between the replies to the

first and second mailings, there would be no need to mail out the

questionnaire a third time (p. 395). He stated that the total time re-

quired to conduct a two wave questionnaire in the event of a short time

frame, would be approximately two weeks.

Erdos (1974) supported the two weeks time period. He stated "by

sending out a single mailing by air, a nationwide survey of a large

sample distribution all over the United States can be completed in two

weeks" (p. 2-91).

Mooren and Rothney (1956) performed an experiment to observe if

personalized forms of mail questionnaires would produce faster responses

than general nonpersonal forms. They found no significant difference

among the personal and general groups of their experiment in terms of

speed of response (p. 412).

In spite of researchers' attempts to decrease the length of time

required to perform a mail survey, the time still depends on various

factors, such as size of the study, the response rate desired, and the

pattern of response. Researchers have condensed the time to a minimum

of two weeks for a two wave questionnaire (Cox, 1966).

Personnel Required

The personnel involved with a follow-up study depends on the size

of the study and the resources available. Many educational institu-

tions use their everyday staff to perform a follow-up study. Basically,

a mail questionnaire requires clerical help in the areas of typing,

printing, and putting the follow-up instrument in the mail envelope.

When the questionnaires are mailed back, they must be organized and

tabulated, using clerical help or keypunchers. A programmer may be

needed to program the statistical analysis before the results are ready

to be evaluated and reported.

Parten (1950) reported that mail questionnaire studies are less

expensive than other methods of follow-up because there is no large

staff to train. More staff may be needed if there are plans to collect

data from the people who do not answer the questionnaire.

Dillman's (1972) follow-up studies consisted of mailing out 4,500

questionnaires each. He used 1,215 labor hours for each study to per-

form the data collection process. In this instance, Dillman used a

substantial amount of labor but was performing a follow-up study on a

relatively large sample.

Franzen and Lazarsfeld (1945) reported one of the advantages of the

mail questionnaire was that it required few personnel. They stated

"the mail can carry a questionnaire into any territory and at the same

time can make the collecting of information a one-man job" (p. 293).

Benson (1946) also reported that widely scattered geographic coverage

can be obtained more easily and less expensively through mail question-

naires than through personal interviews. Erdos (1974) stated that mail

questionnaires can be conducted from one office with built in checks

and controls for every phase of the operation (p. 2-91). Dillman (1978)

reported that the personnel requirements for the mail questionnaire

were far less than those required for either personal interviews or

telephone surveys with equal sample size, and the skills required were

mostly clerical (p. 67).

As Franzen and Lazarsfeld stated, a mail questionnaire can be per-

formed by one person, even though most studies do employ some clerical

help. The mail questionnaire can get by on a small staff or a large

staff, depending on the size of the sample, the budget, and the labor

resources available.

Krueck (1975) disagreed with the prevalent opinion that question-

naires required fewer human resources and manhours than personal con-

tact methods. He found in his follow-up methods comparison study, that

12.5 clerical manhours were required for labeling and stuffing envelopes,

checking returned questionnaires, and assimilating information obtained

through the use of telephone reminder calls (p. 22). He stated that

his finding "should dispel any belief that fewer human resources are re-

quired in a mailed questionnaire unless one considers repeated mailings"

(p. 22).

Quality of Data

The quality of the data had two main aspects in this study. First

to be considered were the results of the study. Were the results based

on true and accurate data? Second to be considered were the data. Were

the data from which the results are taken complete? For the results of

any follow-up study to be useful, they must be accurate, and the data

must be complete. Otherwise, any conclusions drawn from the results will

not be dependable.

Accuracy of information which is returned on a questionnaire can be

difficult to verify. Parten (1950) supported questionnaire information

because she believed the respondent may answer questions more frankly

by mail since anonymity is frequently used. She also stated that since

the questionnaires were standardized, that eliminated possible personal

interviewer suggestiveness or bias (p. 94). On the negative side, Parten

(1950) suggested that since the respondent completes the questionnaire

without the aid of a personal interviewer, misinterpretation of ques-

tions is possible (p. 95).

Franzen and Lazarsfeld (1945) reported an item in favor of mail

questionnaires in the area of accuracy. They compared the mail ques-

tionnaire'to the personal interview on the basis of respondents educa-

tional level. They found that apparently, when not faced by an inter-

viewer, incomplete college and high school records were more willingly

divulged (p. 311). From this finding, Franzen and Lazarsfeld concluded

the mail questionnaire elicited more accurate responses in the area of

reporting educational level. Benson (1946) agreed that mail question-

naires elicited more accurate data. He stated that it is conceivable

that on certain subjects answers might be more reliable than those

that could be obtained by an interview. Hochstim (1967) also concluded

that respondents were more honest in mail questionnaires than the

other methods of gathering data. Knudson, Pope, and Irish (1967)

stated that research comparing questionnaires to interviews were rare,

though a few studies have indicated that the questionnaire may be the

more successful in gaining valid information.

Although the above researchers have contended that the question-

naire elicits accurate data, other researchers emphasized that it is

difficult to check the accuracy of the responses on a mail question-

naire. Parten (1950) stated, "checks on the honesty and reliability

of returns are difficult to devise when the personal interviewer does

not see and size up the informant" (p. 96).

Obtaining completed questionnaires has been another problem with

mail questionnaires. Hackler and Bourgette (1973) reported that mail

questionnaires are usually more appropriate for collecting sensitive

information (p. 276). But many researchers also felt that personal

interviews, rather than mail questionnaires, elicited more complete

answers. Jackson and Rothney (1961) performed a study utilizing both

mail questionnaires and interviews. They concluded "the interviews

elicited significantly more complete answers than the mail question-

naires" (p. 571). They also concluded the interview method drew forth

more extensive responses to questions than the mail questionnaire.

Krueck (1975) explored the differences in the nature of responses

elicited by each method of conducting a follow-up study. He found

questions concerning satisfaction with instruction, counseling, and open-

ended questions had a much higher frequency of nonresponse on a mail

questionnaire. He also reported that the mail questionnaire elicited

a greater number of negative responses when compared to the personal

forms of data collection (p. 19).

Wotruba (1966) found that sending a monetary inducement with the

questionnaire produced not only a better return, but also produced more

completed questions. He concluded that when completeness of the ques-

tionnaire information was vital, immediate monetary inducement was bene-


Mooren and Rothney (1956) performed a study to discover whether per-

sonalization would produce more complete responses. They found no signi-

ficant difference between personalized and general forms of questicn-

naires in terms of quantity of response (p. 412). Parten (1950)


summed up the advantages of the mail questionnaire as follows:

1. If mailed questionnaires are used, it is possible to cover
a wider geographical area and to reach a much larger popula-
tion with given funds than could be accomplished by personal
interviews with each informant. This lower cost applies pri-
marily if personal follow-ups are not made.

2. Mailing costs are relatively low compared with the trans-
portation and costs for a field staff.

3. The expensive and time-consuming task of training a staff
of investigators is eliminated. This assumes, of course, that
a large staff will not be needed to collect the schedule data
from the people who do not answer the questionnaire.

4. The informant may answer questions more frankly by mail
since anonymity is assured. On the other hand, some respond-
dents may hesitate to put their ideas in writing for fear
that their schedules may be identified even though unsigned.
Actually, it is questionable whether anonymity is either an
advantage or a disadvantage.

5. The questionnaire may reach groups who are more or less
protected from solicitors and investigators. In high-rent
apartment houses or private homes where servants protect the
occupants from solicitors and other doorbell ringers, for
example, it is often difficult for investigators to gain ad-
mittance. Of course, the mere fact that the mail is received
does not guarantee that it will not be filed in the waste-
basket by a secretary or even by the addressee as soon as he
glances at the heading.

6. Personal antagonism to investigators which may lead to a
refusal to give the desired information is avoided.

7. If time is not an important consideration and if the
sample extends over a wide area, the cost of securing practi-
cally complete returns is probably lower than in the personal
interview method.

8. The questions are standardized, whereas in the personal
interview the investigator may alter them or suggest answers.

9. The questionnaire can be answered at the convenience of
the respondent. This gives him time to deliberate on each
point, and if necessary, to look up information needed to fill
in the items. However, he may consult other members of his
household, so his reply may be more representative of the
Family's point of view than of his own.

10. It is claimed that the mail questionnaire brings many

more returns from the man of the house than does the
telephone or personal interview method.

11. Where the persons to be reached are located in
widely scattered areas of cities and are a mobil ele-
ment of the population, it may be easier to locate
them by mail (registered or special delivery) than by
other methods. (pp. 94-95)

Parten (1950) also listed the disadvantages of the mail out question-


1. The people who return questionnaires are not repre-
sentative of the groups to whom the schedules are sent.
This limitation is sufficiently great to outweigh almost
all the advantages listed above.

2. The returns from mailed questionnaires sent to the
general public are usually very low, often ranging from
about 10 to 20 percent. The percentage of returns varies
greatly, however, with different schedules and informants.
One survey of M.D.'s in New York State received about 50
percent returns without follow-ups. By continued effort
the returns may be increased considerably over the first
responses. The techniques for increasing mail returns
are dealt with in Chapter XI (in the Parten Book).

3. Since the informant fills in the data on the question-
naire without the assistance of an investigator, he may
misinterpret questions, omit essential items, or send in
material which cannot be put in form for tabulation, thus
making it necessary to discard many of the questionnaires.

4. The questions used must be simple and practically
self-explanatory, since no training can be given the infor-
mant on their meaning and on how to fill out the schedule.

5. In most studies, the questionnaire must be relatively
brief if high returns are to be obtained.

6. If the sample is to be unbiased, it is necessary to
supplement the mailed returns with information obtained by
personal interviews with the nonrespondents.

7. Checks on the honesty and reliability of returns are
difficult to devise when the personal interviewer does not
see and size up the informant.

8. it is practically impossible to return unsatisfactory
or incomplete schedules to the informant for correction.


9. Because most people would rather talk than write,
questionnaires must be made very interesting to induce

10. An up-to-date address list of potential survey in-
formants is difficult to find.

11. Mail returns from the last third of the respondents
come in slowly; hence the mail survey must be spread over
a relatively long period, if a high percentage of returns
is to be secured.

12. Many questions which might antagonize the respondent
cannot be included on the mail questionnaires but can be
asked in personal interviews when the informant gradually
can be led around to the subject. (pp. 95-96)

Telephone Survey

Telephone surveys consist of interviews taking place over the tele-

phone rather than in person. Dillman (1978) reported that research litera-

ture on telephone surveys was as thin as that for the mail questionnaire

was thick (p. 9). The telephone survey was used in the Literary Digest

debacle of 1936. The Digest pollsters predicted a landslide victory for

Landon over Roosevelt in the presidential race. Using the telephone

caused a clear social class bias since only 35% of households in the

United States had telephones (p. 9). Therefore, the poll predicted in-

correctly and the use of the telephone as a survey method was generally

used only as a complement to other survey methods (p. 9).

As the percentage of households having telephones increased, so

did the feasibility of the telephone survey. Dillman (1978) reported

that the use of the telephone survey had mushroomed in the 1970's

(p. 10). Payne (1974) reported that interviews by telephone were satis-

factory, efficient, and economical for a wide range of survey purposes

(p. 2-105). Warwick and Lininger (1975) agreed. They stated that

researchers conducting follow-up studies have turned increasingly to the

telephone survey (p. 130). They reported the telephone survey had many

of the advantages of the personal interview, yet it was considerably

less expensive. It overcame the problem of mounting refusal rates in

some areas where strangers coming into the hume were unacceptable.

Some respondents who would not allow an interviewer in their home were

perfectly willing to talk with the same individual on the telephone

(p. 130).

Warwick and Lininger (1975) also reported the limitations they

associated with the telephone survey. In their opinion, the greatest

drawback was the difficulty of establishing the same rapport that was

possible in a personal interview situation. With the telephone sur-

vey, there was a lack of communication through facial expression, ap-

pearance and sheer physical presence. Other limitations reported by

Warwick and Lininger included the respondent's ability to hang up.

As a result, telephone surveys were usually shorter and less demanding

than personal interviews. The asking of sensitive questions was also

limited because it increased the risk that the respondent would hang


In Payne's (1974) opinion, the main drawback of the telephone

survey, at least historically, was the fact that not everyone had a

telephone (p. 2-105). With well over 95% of all households obtaining

telephones, the chance of response bias due to lower income groups

not possessing a telephone was reduced (Dillman, 1978, p. 9). But

those in higher income groups who decided to have unlisted telephone

numbers could again induce response bias if the telephone directory

was the information pool from which the sample was drawn (Payne, 1974,

p. 2-107). Also the problem existed where higher income homes had

more telephones, and possibly a better than average chance of being in-

cluded in the survey. An indirect sampling technique which employed

the local telephone exchange and random numbers for the last four

digits could be utilized (Payne, 1974, p. 2-119). For further suggest-

ions on sampling techniques to be used in telephone surveys, see

Payne, "Data Collection Techniques: Telephone Surveys," in the Hand-

book of Marketing Research (Payne, 1974). For the purpose of this

paper, it will be assumed that sampling is not a problem, that the re-

searcher has access to the telephone numbers needed to perform a study.

McKinney and Oglesby (1971) also reported on the effectiveness of

sampling when used for a telephone survey. They stated that while the

telephone interview is- limited to the amount of information which can

be obtained, it may be the only workable procedure for some segments

of the population (p. 9).

The telephone survey technique will be reviewed using the five

variables previously chosen to compare and evaluate the various follow-

up methods. The five variables are: costs, response rate, length of

time, personnel required, and quality of data.


Denton et al., (1977) reported that a telephone survey required

a large monetary investment to locate and actually contact potential

respondents by phone. Comparing the telephone survey to the personal

interview, Parten (1950) reported that the telephone survey could pro-

vide a tremendous savings in transportation costs (p. 86). Weiss and

Hatry (1971) agreed that by utilizing the telephone survey, travel was

eliminated and costs reduced (p. 23). They estimated that for 500

telephone interviews, the average cost per interview would be $17.00

(see Table 5). They included detailed costs for preparation. Consider-

ing only actual interviewing costs, the costs for the study were $9.33

per response.

Thomas Krueck (1975) performed a comparison of three methods of

conducting a follow-up study. He reported that as conducted in his

study, the telephone survey required no immediate out-of-pocket outlay

of funds (p. 22). The costs were involved with the manpower needed to

make the telephone calls. He employed research assistants and paid

them $5.00 an hour. He calculated that 55 hours and 12 minutes were

devoted to telephone calling, producing a total cost of $276.00. The

response to the telephone survey was 108 out of 164 possible responses,

or a 66% response rate. Therefore, the cost per response was approx-

imately $2.55.

Payne (1974) agreed that an advantage of the telephone survey

over the personal interview was in the area of cost savings. He stated:

Generally, the shorter the actual interview time relative
to other time factors, the greater the economy of the
telephone method seldom do the data collection
costs of the telephone survey exceed two-thirds of the com-
parable costs of the personal survey. (p. 2-110)

Payne continued on to state that differences in costs between

telephone and mail surveys depend on the response rates.

Depending primarily on mail response rates, one or the other
may prove more economical, since the mailing cost per return
for a 10 percent mail response is essentially five times
that for a 50 percent mail response. (p. 2-110)

Table 5
Cost Estimates For Telephone Interviews

For 500 tele-
phone interviews
1. Overall planning and direction

2. Preparing listing $ 500

3. Selecting sample 300

4. Developing interview questionnaire (including
conferences, 20 pretest personal interviews,
10 pretest telephone interviews) 600

5. Reproducing interview questionnaires, training
materials, manuals, supplies 650

6. Interviewing costs
a. Recruitment 200
b. Training (20 interviewers for 15 hours,
plus salary of trainer) 1,000
c. Interviewing time 1,900
d. Supervision 800
e. Supplementary in-person interviews for
50 persons without telephones or not answering 1,000

7. Editing, coding, punching, tabulating 800

8. Telephoning 500

9. Mailing 10

10. Analyzing data and report writing 150

11. Reproducing report 100

Total $8,510
Average cost per interview $17.00

Note. From An Introduction to Sample Surveys for Government Man-

agers, by C. H. Weiss & H. P. Hatry. Washington, D. C.: Urban Insti-

tute, 1977, p. 47.

Response Rate

According to Parten (1950) the response rate on the telephone sur-

vey depended on whether or not recalls were made on "no answers" and

"busy signals" (p. 89). She reported that only about 1 1/2% refused

the calls, and about 1/3 were incomplete due to no answers or busy

signals. Therefore, approximately 2/3 of the calls made resulted in

completed interviews on the first call.

The response rate to a telephone survey was believed to depend on

the length of the survey, but studies listed by Dillman (1978, p. 55)

found only 2% to 4% of the respondents refused to complete a telephone

survey lasting from 20 to 30 minutes. Dillman (1978) reported that it

was not known how long interviews could be before cut offs become a

major problem, but the evidence clearly suggested that once people

were on the telephone, the length of the interview did not appear to

affect the response rate (p. 55).

Michael Quantz (1977) performed a telephone survey of community

college career students who graduated between 1973 and 1976. Telephone

interviewers, people actually in the program the respondents had gradu-

ated from, attempted 228 contacts. If the graduate was not available,

the desired information was obtained from a relative or co-worker. The

interviewers completed 191 of 228 possible contacts. The response rate

for this study was 84%.

Krueck's 1975 study performed a comparison of the three methods

of conducting a follow-up study. He reported that the telephone survey

methods had a higher response rate than the personal interview or the

mail questionnaire. The response rate in his study was approximately

66%. Following were the reasons listed by Krueck for any lack of re-

sponse to the telephone survey: 11 were either a wrong number or a

disconnected telephone, 19 had moved out of the local telephone dis-

trict, 14 could not be followed-up, and 1 refused to respond (p. 17).

Warwick and Lininger (1975) reported that perhaps the greatest

success with the telephone survey was seen when it was used as a

follow-up to a personal interview. A study using a personal interview

with multiple follow-up using a telephone call: obtained a response

rate of 97.5% (p. 131).

Length of Time

Parten (1950) stated that the telephone survey brought about a

savings in time as well as in costs, and it could be used when a study

needed to be conducted quickly and inexpensively. Parten also noted a

large portion of a population of a community could be covered in a

short time due to the fact a few telephone callers could make several

hundred calls per day (p. 86).

In Krueck's (1975) study a total of 303 phone calls were made to

obtain the response of 108 former students. The actual interviews

taken over the telephone required approximately 15 minutes each for a

total of 27 hours. The preliminary calls averaged 4 minutes each for

a total of 13 hours. Therefore, the actual number of minutes required

to obtain each response was 22 minutes (p. 23).

Payne (1974) stressed the need for interesting questions on inter-

esting subjects in telephone surveys (p. 2-108). It was much easier to

hang up when bored, than to ask an interviewer to leave. Although it

was once thought that telephone interviews must be brief, telephone

interviews have gone on for as long as 25 to 30 minutes (Payne, 1975,

p. 2-108). Payne gave the warning that, "a careful look might better

be taken at the questionnaire to judge whether its inordinate length is

justified" (p. 2-108).

The time involved to perform a follow-up telephone survey varied

not only according to the length of the interview, but with the number

of people placing the calls, and the number of recalls made. Parten

(1950) reported that the telephone survey was still the quickest of

the three main follow-up research techniques (p. 91). Parten also re-

ported that if the interview itself was brief, then an interviewer

could complete about 30 calls per hour (p. 91).

Payne (1974) reported that more interviews could be done by tele-

phone than in person within a limited time period, and more hours of

the day could be productive (p. 2-110). Dillman (1978) stated that as

a general rule, "telephone surveys are the fastest to complete" (p. 67).

Denton et al., (1977) alternately suggested that the telephone

survey could involve a great deal of time if it was necessary to locate

the potential respondents' phone numbers. In a situation which would

require substantial amounts of time to locate respondents and gather

telephone numbers, then the telephone survey would not necessarily be

the quickest method of a follow-up study.

Personnel Required

The central people involved in a telephone survey are the people

who make the telephone calls. Almost all the other factors, such as

cost, time involved, and even response rate, depended on the people

making the telephone calls. According to Parten (1950, p. 90), if good

interviewers were employed, the number of refusals would be low. She also

reported that it was easy to train and supervise interviewers since they

could work all in one room directly beside the supervisor (p. 91).

In Quantz's (1977) study, the telephone interviewers were faculty

members from the different program areas. They contacted the student

who had graduated from their particular program. No special training

was required.

Dillman (1978) reported that personnel requirements for telephone

surveys were less than for personal interviews, but more than for mail

questionnaires (p. 66). Close supervision meant the skills and train-

ing needed could be developed as on the job training. In college and

university settings, students with flexible schedules and training in

research methodology could be used.

The size of the sample or population determines the number of

telephone interviewers required. Other people involved with the follow-

up study, such as statistical and keypunch operators, varied according

to the size of the sample, not the method of follow-up employed. With

the telephone survey, one or two telephone interviewers could investi-

gate a large portion of the population in a short time because several

hundred calls could be completed per day (Parten, 1950, p. 86).

Dillman (1978) reported that with a telephone survey:

Relatively few interviewers can conduct a large number of
interviews in a short period of time, in sharp contrast
to the rate of two or three a day often expected for face
to face interviews. (p. 66)

Quality of Data

To discover the quality of data, two considerations must be taken

into account; the accuracy and truth of the data, and the completeness

of the data.

Already mentioned was the belief that a telephone survey must be

short, nondemanding and void of sensitive questions to get respondents

to respond with accurate data (Warwick and Lininger, 1975). This limited

the type of data that could be collected and therefore could reduce the

ability of the researchers to accurately ascertain information on certain


Payne (1974) disagreed. He reported that respondents may reply with

greater candor to personal questions asked on the telephone than to the

same questions asked in person (pp. 2-122).

A major problem connected with the telephone survey for collecting

accurate and true data was improper sampling. Weiss and Hatry (1971)

reported that since not everyone had a telephone, there was sampling

bias with studies utilizing telephones. Lower income people have fewer

telephones and some people have unlisted telephone numbers. According

to Weiss and Hatry (1971), a recent investigation showed that people

with unlisted telephone numbers were primarily middle income and dispro-

portionately black (pp. 23-24). Therefore, researchers employing the

telephone survey should consider the intended respondents, to ascertain

whether or not a telephone survey will produce sampling bias.

Another problem with telephone surveys, according to Warwick and

Lininger (1975), was that telephone surveys were sometimes used for mag-

azines and other lead-in sales presentations as well as other types of

phone solicitation (p. 131). Parten (1950) stated that if the respondent

was annoyed by the phone call, he or she may deliberately give misinfor-

mation (p. 87). Without: the interviewer actually facing the respondent

to observe his or her reactions, the interviewer must accept the answers

without question (Parten, 1950, p. 87).

According to Parten (1950), other limitations that existed when

gathering data using a telephone survey was the lack of privacy on party

lines, housewives answered the phone more often than other members of

the family, and misinformation was hard to check in short telephone calls

(pp. 89-92).

One problem Payne (1974) reported was that open-ended questions might

not be answered in as great detail over the telephone as in person

(p. 2-108). Although the answers could be shorter, much of the extra

verbiage was repetition or elaboration. Therefore, telephone surveys

usually provided complete answers even if less words were used.

When considering completeness of data, Parten (1950) reported that

the telephone interview must be short, so only a few brief items could

be investigated (p. 87). Therefore, certain topics which would require

extensive information would not be able to gain complete data from a

telephone interview.

Krueck (1975) found that negative comments were inhibited in the

two forms of personal contact, the telephone survey and the personal in-

terview. He also found a lack of consistency in negative answers when

comparing mail questionnaires to the telephone survey. He suggested

further research was needed with regard to the effect on personal in-

volvement in willingness to report in a negative manner (p. 19).

Hochstim (1967) reported that the personal interview elicited the

highest number of socially desirable answers, the mail questionnaire

elicited the least number of socially desirable answers, the telephone

survey fell between the two. Dillman (1978) stated that it seemed that

the transfer of the interviewer from the scene to the telephone decreased

the probability of socially desirable responses (p. 62).

Payne (1974) reported that the telephone interview was midway be-

tween the zero interviewer effect of the mail questionnaire, and the pro-

nounced interviewer effect in the personal interview. According to

Payne (1974), as little as possible of interviewer bias should go into

the recorded responses. To accomplish this, questions should be asked

in a standardized manner (p. 2-111). Dillman (1978) reported that the

distortion from interviewer bias would be easier to control in a tele-

phone interview than a personal interview with the opportunity for closer

supervision (p. 69). He also concluded that the telephone interview was

the best method in terms of the ability to control contamination from

other people (p. 65).

In a telephone interview, completeness of data depended on a com-

pleted phone interview. To accomplish this, researchers nave suggested

short interviews, avoiding sensitive questions, and employing well-

trained and competent interviewers to "sell" the respondent on the im-

portance of the interview (Parten, 1950, pp. 86-93; Warwick & Lininger,

1975, pp. 130-131).

Parten listed the following advantages of the telephone survey as

a summary of the follow-up technique:

1. The telephone interview is the quickest of the survey
techniques. Interviewers can complete about thirty calls
per hour if the calls are brief. It is especially adapted
to surveys of radio programs where a great many interviews
must be made during a given program.

2. The refusal rate is usually low among people who are
reached by phone.

3. The coincidental method can be used thus eliminating
the memory factor.

4. It is easy to train and supervise interviewers since
they can work in one room directly beside the supervisor.

5. The approach and questions are easy to standardize
from one interviewer to another.

6. The cost per completed interview is low for the
sample covered.

7. The geographic distribution of the sample can be
easily controlled. An address listing of numbers is
usually available and can be used for drawing the sample.

8. For studies of middle- and high-income groups, the
telephone interview may be satisfactory because most
of them have phones.

9. Interviews may be scattered over a wide area within
a city without adding to the cost.

10. As compared with a mail questionnaire, the telephone
survey is preferable because it usually costs less per
return. Returns are higher on first solicitation, and
they can be more effectively controlled from the point of
neighborhood distribution. (pp. 91-92)

Parten listed the following disadvantages of the telephone survey:

1. As a sample of the general population, telephone
subscribers are not representative. So unless the
telephone interview is supplemented by a method that
covers nonsubscribers, it should not be used. Less
hang half of all homes in towns over 2500 had telephones.

2. Detailed data cannot be gathered by this method
because the informants soon become annoyed or impatient.
If the schedule is too lengthy, the informant may either
hand up or give unreliable answers.

3. When observation of the situation is an important
element, the telephone interview is not useful. If the
interviewer is supposed to evaluate the answers as to
trustworthiness, he has very little to go on in a short
telephone conversation.

4. Information about the respondent must be limited to
one or two facts. Such items as age, nationality, in-
come, etc., are difficult to secure by telephone.

5. Attitude scales must be used with caution. Also,
opinions are less likely to be given freely since the
informant cannot be certain of the credentials of the
person calling.

6. Since rural telephone ownership was low, the tele-
phone interview is not useful in such areas. Also,
because rural rates are higher than urban, the cost of
telephone inquiries is greater than in cities.

7. The brevity of the introduction and questions does
not give the informant much time to orient himself to
the subject matter of the survey. Reactions requiring
careful thought such as criticisms of various products,
suggestions as to new uses of products, appeals, etc. --
should not be obtained by this technique.

8. The telephone situation neither encourages the re-
spondent to amplify his replies nor gives the inter-
viewer much time to jot down the comments. A face-to-
face interview is more conducive to a considered response.

9. The task of checking on no-answers, wrong numbers,
busy signals, etc., is time-consuming but must be done
if the sample is to be representative of telephone

10. It is difficult to secure privacy on party lines.

11. The time may come when the telephone technique will
be used by so many groups tnat informants will develop
an antagonism to all telephone inquiries.

12. The surveyor must be careful not to antagonize in-
formants by phoning too early or too late in the day.
One well-known survey agency makes it a policy never to
call before 8:30 A.M. or after 10:30 P.M.

13. Misinformation is hard to detect and check in
short inquiries. (pp. 92-93)

In his 1975 study, Krueck stated the results of his study supported the

idea that more attention should be paid to telephone interviewing. It

had produced a higher response rate than the mail questionnaire or the

personal interview with only slightly higher costs than the mail ques-

tionnaire and much lower costs than the personal interview.

Personal Interview

According to Warwick and Lininger (1975), a personal interview is

a form of verbal interaction designed to obtain information satisfying

the objectives of a particular study. "The interviewing process is

decidedly interactive, involving communication between the interviewer

and respondent" (Warwick & Lininger, 1975, p. 182). The standardized

interview, which was. designed to collect precisely the same categories

of information from a number of respondents so that the answer of all

respondents could be classified and compared, is the type of interview

which will be referred to in this paper as a "personal interview" (Gordon,

1975, p. 60). Three purposes for the personal interview were suggested

by Froelich and Darley (1952). These were supplementing information,

verifying information, and observation of characteristics or mannerisms.

Mayer (1974) reported that personal interviews were certainly the

most common way of gathering data in marketing research. He described

the personal interview as the most flexible method of data collection.

According to Mayer:

The interviewer's physical presence enables the researcher
to use various forms of interaction between the interviewer
and the respondent. The most common interaction is that
the interviewer asks questions of the respondent, following
a structured format or questionnaire, and records the re-
spondent's answers on the questionnaire. (p. 2-82)

Warwick and Lininger (1975) reported "the success of the personal

interview depends on the quantity and quality of the information ex-

changed between the interviewer and the respondent" (p. 182). They

stated four interacting parties or conditions affect the flow of infor-

mation; the interviewer, the respondent, the topic, and the immediate

interviewing situation. The interviewer, facing the respondent, must

communicate the question, motivate the respondent to cooperate, probe

for additional information, and record the information gathered from

the respondent. The respondent must provide answers which are complete

and truthful. The topic should stimulate the interest of the respondent

without arousing anxiety, hostility, or fear. The interviewing situa-

tion must consider such conditions as the time and place of the conver-

sation, the presence of any "third parties," and the similarities be-

tween the interviewer and the respondent (p. 183).

Mayer (1974) reported that the personal interview assumes the least

about the respondent (p. 2-85). The human interaction element brought

variability to the personal interview follow-up method. The personal

interview will be reviewed using the five variables previously chosen to

compare and evaluate the various follow-up methods. These five variables

are: costs, response rate, length of time, personnel required, and

quality of data.


It was generally agreed that conducting personal interviews was an

expensive method of gathering follow-up information. Since an inter-

viewer must be physically present, personal interviews were more expen-

sive than other methods of follow-up (Mayer, 1974, p. 2-85). Sudman

(1967) reported that from 30% to 40% of the interviewer's time was spent

interviewing. Mayer (1974) reported:

The interviewer spends time in learning the specific re-
quirements of the survey, traveling to the designated areas,
locating the required households, identifying the desired
respondent, asking questions, recording the answers, and sub-
mitting the completed work. Many of the attempts to contact
potential respondents may be unsuccessful. (p. 2-86)

Travel costs, including transportation expenses, interviewer time, and

the training of interviewers increased the cost of personal interviews.

Jackson and Rothney (1961) reported that for every dollar spent

for their mailed questionnaire study, approximately 60 dollars were

spent on the personal interview study. In this instance, the interview

method cost 60 times as much as the mail questionnaire method.

Krueck (1975) analyzed the costs in a comparison study of the three

methods of conducting follow-up methods. The personal interview method

included the travel which was converted to a dollar figure at a rate of 10

per mile. Krueck's study was fortunate in that 60% of the interviews

were conducted at the study sight, requiring no travel for the interviewer,

and thereby saving expenses. For the off location interviews requiring

travel, the average distance traveled one way was seven miles. Thus a

minimum of 284 miles were logged in conducting interviews for a cost of

$30.20. Clerical help and other manpower costs brought the total cost

of the personal interview study to $426.50. When the response rate was

considered, the cost per response was $5.86. Krueck concluded that the

personal interview follow-up method was the least cost effective method

when compared to the telephone survey and the mailed questionnaire.

Weiss and Hatry (1971) reported that in the early 1970's, profes-

sional interviewers were earning from $2.00 to $3.00 an hour. Interviewers

usually received pay not only for interviewing time, but for time spent

in training, reading instruction, traveling, locating and contacting re-

spondents, and preparing the data after the interview (p. 20). Further

costs associated with the personal interview method, according to Weiss

and Hatry (1971), were the time and money spent for recruiting and se-

lecting the interviewers, and for trainers and supervisors. Weiss and

Hatry listed the following factors as having an effect on interviewing


--Productivity (number of interviews completed per hour),
depending on the interviewers' ability, experience and

--Time spent for call-backs to respondents who were not
at home on the first call. Procedures should limit
the number of attempts to reach any one respondent
(three is a common limit).

--Neighborhood being surveyed. In some areas, costs
go up because residents refuse to open their doors to
strangers, pairs of interviewers are needed for security
reasons, housing or population listings are inadequate,
and so forth.

--Travel time and costs between interviews, and from
interview locations to the survey office.

--Financial payment to respondents. Most experienced
survey researchers frown on paying respondents for
participating, except as a last resort, but some ad-
vocate it. (pp. 20-21)

Weiss and Hatry (1971) emphasized that costs with the personal in-

terview method relate directly to the sample size and the number and com-

plexity of questions. Answers that allow easy coding take little time,

whereas long narrative and open ended questions required the development

of careful classifications and cross checking of different coders' de-

cisions on classifications (p. 21).

Table 6 displays the costs estimated by Weiss and Hatry for con-

ducting 500 interviews. The table shows that costs could range from a

low of $11.87 per interview to a high of $39.80 per interview. Consider-

ing only interviewing costs, the cost per interview was $14.60. For 400

interviews their cost estimates were a low of $12.71 per interview to a

high of $44.13 per interview. For 1000 interviews, cost estimates ranged

from a low of $10.90 per interview to a high of $33.00 per interview.


Table 6

Cost Estimates For Five Hundred Interviews

Low Moderate

1. Overall planning and direction

2. Preparing listing

3. Selecting sample, preparing maps,
assigning households to interviewers

4. Developing interview questionnaire
(including conferences and 20 pre-
test interviews)

5. Printing interview questionnaires,
training materials, manuals, supplies

6. Interviewing costs
(a) Recruitment
(b) Training (20 interviewers for 20
hours, plus trainer's salary)
(c) Interviewing time (including
travel, call-backs, recopying)
(d) Field supervision
(e) Travel expenses

7. Editing, coding, punching, tabulating

8. Telephoning

9. Mailing

10. Analyzing data and report writing

11. Reproducing report
Average cost per interview

$ 250





















(but not













Note. From An Introduction to Sample Surveys

by C. H. Weiss & H. P. Hatry. Washington, D. C.:

p. 42.

for Government Managers

Urban Institute, 1971,


Response Rate

Krueck (1975) in his follow-up method comparison study reported

that his data indicated the mailed questionnaire and personal interview

methods were approximately equal in effectiveness, eliciting response

rates of about 50%.

Parten (1950), however, stated that the most striking feature of

the personal interview method was the high percentage of acceptable re-

turns obtained, especially in well organized and carefully planned

studies (p. 73). She reported that in studies of the general public,

most surveys could depend on at least 80% response. Though Parten stated

the response rate was usually high, Denton et al., (1977) reported the

use of personal interviews reduced the number of individuals who could

be surveyed due to the time and expense involved in conducting personal

interviews (p. 7). Therefore, according to Denton et al., (1977), the

personal interview method may have a higher response rate, but the size

of the sample or population was usually smaller than other methods of


Baugh conducted a study in 1976 where he used both mail question-

naires and personal interviews. His response rate for the mail question-

naire section of the study was 55%, but he had a 100% response rate for

the personal interview. Mayer (1974) stressed that the personal inter-

view has the highest potential response rate.

Length of Time

A follow-up study consisting of personal interviews usually takes

a longer period of time to complete than either a telephone survey or

mail questionnaire. Cox (1966) disagreed with the majority of researchers

when he stated, "given a tight schedule, a researcher would expect to

turn to the personal interview method, usually at greater cost" (p. 392).

Denton et al., (1977) stated the more popular view when he reported the

personal interview to be time consuming. Jackson and Rothney (1961) per-

formed a comparison of the mail questionnaire and the personal interview

follow-up methods and found the mail questionnaire had the advantage of

saving time over the personal interview.

Dillman (1978) reported that the time involved in a personal inter-

view study varied depending on the sample size and geographical disper-

sion (p. 68). He further stated that the time required was closer to

that required for mail questionnaire studies than telephone surveys

(p. 68).

Krueck (1975) reported that the average time for conducting a fol-

low-up personal interview in his study was 20 minutes. He calculated

12 hours of clerical time was spent in planning and assimilating various

pieces of information relating to the personal interview. Totally,

116.5 manhours were required for the completion of approximately 80 in-

terviews (p. 24). Weiss and Hatry (1971) reported that the personal

interview took time for recruiting and selecting interviews as well as

training before the interviews could begin (p. 20).

Parten (1950) reported that the time required to cover a large pop-

ulation with personal interviews may make the method undesirable. She

also pointed out that the personal interview usually takes more time

than the telephone interview.

Basically, the time required for a personal interview, as for all

methods, depended on the number of responses desired. In a very large

sample, using a mail questionnaire, Parten (1950) reported that a per-

sonal interview may be required eventually to check for nonresponse

bias in the mail questionnaire (p. 96).

Mayer (1974) reported that the personal interview survey had the

longest and most complicated scheduling requirements. According to Mayer,

"The scheduling and timing requirements of personal interview surveys

were the chief reasons why this method of data collection was the slow-

est" (pp. 2-84 2-85).

Personnel Required

As McKinney and Oglesby (1971) reported, the personal interview

method of follow-up was not only expensive, but required a great amount

of staff time. Whether it was staff that was presently employed, or

staff that must be hired and trained, this method of follow-up was a time

consuming process.

The main reason personal interviews required such a large staff was

that all interviews must have a staff member present, or it was not a

personal interview. Parten (1950) reported that the organization required

for selecting, training, and supervising a group of interviewers working

in the field was more complex than that needed for surveys conducted by

other methods of follow-up (p. 81). Interviews of working people required

an even larger staff since those, interviews had to be conducted at cer-

tain hours in the evening.

Dillman (1978) reported that personnel requirements for the personal

interview were the most difficult to meet. "The availability of people

with interviewing and supervisory skills is as low as the cost is high"

(p. 66). Supervisory networks should be extensive for control. There-

fore, the size of the study would have a direct effect on the number of

people required to perform a personal interview study. In most cases,

the personal interview study required a larger staff than either the

mail questionnaire or the telephone survey (p. 66).

Quality of Data

The quality of data produced through personal interviews had been

both praised and criticized. McKinney and Oglesby (1971) suggested that

probably the most desirable kind of follow-up was the personal interview,

because it provided the opportunity for the respondent to ask questions

for clarification as well as for the interviewer to ask questions for

deeper insight into the problems of occupational preparation as perceived

by the respondent (p. 7).

Gordon (1975) stated that the personal interview provided more op-

portunity to motivate the respondent to supply accurate and complete in-

formation immediately (p. 76). He also stressed the personal interview

provided more opportunity to guide the respondent in his/her interpre-

tation of the question, allowed greater flexibility in questioning the

respondent, allowed greater control over the interview situation, and

provided a greater opportunity for evaluation of the validity of the

information through observation of the respondent's nonverbal responses

(pp. 76-77).

Parten (1950) reported that the personal interview data was more

correct than that secured by other methods since the interviewer could

clarify seemingly inaccurate answers by explaining the questions to

the respondent (p. 9). Interviewers could also be trained to spot

falsified responses or could collect supplementary information about the

respondent if the responses indicated a need for more data (p. 80).

Another advantage of data collection with the personal interview was

that visual cues could be utilized. However, the danger of interviewer

bias had to be considered as well.

Parker, Wright, and Clark (1957) performed a study where college

students were interviewed concerning their scholarships, and the inter-

views were recorded. A "check interviewer" listened to the recordings

and was asked both what the student said, and what they interpreted the

student as saying. An interesting result of their study was that the

two interviewers, the one who performed the interview in person and the

one who listened to a recording, did not always agree on what the stu-

dent actually said. Overall, the results from this study indicated

that the personal interview does not yield 100% reliable data. However,

they asked, "Can we afford the time and expense to train interviewers

when questionnaires or similar techniques can only be slightly less re-

liable?" (p. 220).

Herbert Hyman (1950) was concerned with the effect of the inter-

viewer on the quality of data gathered during a personal interview. A

two year study was done at the National Opinion Research Center to

isolate, measure and control the interviewer effects on the respondent.

It was thought that the respondents answers were affected by the per-

sonality or characteristics of the interviewer. One of the findings

of the study was that interviewers would report differently even if the

response was the same. Hyman used a recorded interview and found dif-

ferences between interviewer reports of the same responses, suggesting

differences between interviewer reports could cause inconsistencies

without including the aspects of respondents' answers. Next, the study

viewed the respondent and found respondents gave different responses

regardless of whether an interviewer was present or not. Therefore, some

of the inconsistencies came only from respondents. These two possible

biases, and the interaction that may exist between them concerned most

users of the personal interview method of follow-up.

The Hyman (1950) study also reported the other view; some respondents

might not be affected by interviewer bias at all. The study reported

that a few respondents could not even recall what the interviewers looked

like, and for such respondents, the interviewer respondent interaction

was minimal. However, according to Hyman, a series of other studies

showed that survey results for specialized attitudes are affected by

the disparities or similarities in the group membership of interviewers

and respondents. He reported that there was documentation to show that

responses varied with the disparity between the interviewer's and re-

spondent's sex, class, color, religion, and other group membership

factors. This suggested that respondents perceived something about the

interviewer immediately from his or her appearance or behavior, inter-

preted this for some reason in a certain way, and in turn, altered

their behavior. On the other side, the interviewer perceived the re-

spondent and subsequently altered his or her method of questioning,

probing, and recording. Interviewers were found to have certain ex-

pectations about how the respondent would respond and these expecta-

tions affected how he or she would probe for answers and record them.

To minimize interviewer effect, interviewers were usually required

to ask questions exactly as they were written, so all questions would

be asked with the same wording. Training of interviewers was given to

eliminate interviewer bias as much as possible, but it was still a

recognized problem with the personal interview method (Dillman, 1978,

p. 63).

Parker, Wright, and Clark (1957) reported on several studies which

supported the reliability of personal interviews. They reported that

Anderson (1954) found a 0.85 level of consistency in ratings when a

personal interview was used. Levin (1954) concluded that short inter-

views (about one hour in length as opposed to two or two and a half

hours) produced more reliable data than longer interviews.

In regard to completeness of data collected by the personal inter-

view method, Krueck (1975), in his comparison study, found that the

personal interview method had a much lower frequency of no response to

individual questions than did the mail questionnaire method.

Jackson and Rothney (1961) performed a study which found that per-

sonal interviews elicited significantly more complete answers than the

mail questionnaire. The personal interview also elicited a greater

number of responses than the mail questionnaire.

The personal interview method was found to produce more socially

desirable answers than either the telephone survey or the mail question-

naire. Hochstim (1967) concluded that generally respondents were most

honest in mail surveys, least honest in personal interviews, and in-

termediately honest in the telephone surveys. Dillman (1978) supported

the idea that personal interviews have the highest probability for

producing socially desirable answers as opposed to accurate answers.

Dillman (1978) stated "The greatest advantage of the personal interview--

the physical presence of the interviewer--may at tliaes be its greatest

drawback" (p. 63).


Parten (1950) listed the advantages of the personal interview as


1. The personal interview usually yields a high percentage
of response,for most people are willing to cooperate.

2. It can be made to yield an almost perfect sample of the
general population because practically everyone can be
reached by and can respond to this approach.

3. The information secured is likely to be more correct
than that secured by other techniques once the interviewer
can clear up seemingly inaccurate answers by explaining
the questions to the informant. If the latter deliberately
falsifies replies, the interviewer may be trained to spot
such cases and use special devices to get the truth.

4. The interviewer can collect supplementary information
about the informant's personal characteristics and erviron-
ment which is invaluable in interpreting results and eval-
uating the representativeness of the persons surveyed.

5. Scoring and test devices can be used, the interviewer
acting as experimenter.

6. Visual material to which the informant is to react
can be presented.

7. Return visits to complete items on the schedule or to
correct mistakes can usually be made without annoying the
informant. Thus greater numbers of usable returns are as-
sured than when other methods are employed.

8. The interviewer may catch the informant off guard and
thus secure more spontaneous reactions than would be the
case if a written form were mailed out for the informant
to mull over.

9. The interviewer can usually control which person or
persons answer the questions, whereas in mail surveys
several members of the household may confer before the
questions are answered. Group discussions can be held with
the personal interview method if desired.

10. The personal interview may take long enough to allow
the informant to become oriented to the topic under in-
vestigation. Thus recall of relevant material is facili-

11. Questions about which the informant is likely to be
sensitive can be carefully sandwiched in by the interviewer.
By observing the informant's reactions, the investigator

can change the subject if necessary or explain the
survey problem further if it appears that the inter-
viewee is about the rebel. in other words, a delicate
situation can usually be handled more effectively by
a personal interview than by other survey techniques.

12. More of the informant's time can be taken for
the survey than would be the case if the interviewer
were not present to elicit and record the information.

13. The language of the survey can be adapted to the
ability or educational level of the person interviewed.
Therefore it is comparatively easy to avoid misinter-
pretations or misleading questions.

Parten (1950) listed the disadvantages of the personal interview

as follows:

1. The transportation costs and the time period to
cover addresses in a large area may make the personal
interview method unfeasible.

2. The human equation may distort the returns. If an
interviewer has a certain economic bias, for example,
he may unconsciously ask the questions so as to secure
confirmation of his views. In opinion studies especially,
such biases may operate. To prevent such coloring of ques-
tions, most opinion surveyors instruct their interviewers
to ask the question exactly as printed on the schedule.

3. Unless the interviewers are properly trained and
supervised, the data recorded may be inaccurate and in-
complete. A few poor enumerators may make a much higher
percentage of returns unusable than if the informants
filled out and mailed the interview form to survey head-

4. The organization required for selecting, training,
and supervising a field staff is more complex than that
needed for surveys conducted by other methods.

5. It is usually claimed that costs per interview are
higher when field investigators are employed than when
telephone or mail surveys are used. This may not be true
if the area to be covered is not too great. If the gen-
eral public in a community is to be surveyed, the costs
of securing a representative sample by telephone or mail
inquiries will probably equal or exceed the cost by the
personal interview method, since in the end personal
follow-up will be necessary to round out the sample.

6. The personal interview usually takes more time than
the telephone interview providing the persons who can be
reached by telephone are a representative sample of the
type of population to be covered by the survey. However,
for a sample of the general public a telephone inquiry is
not a substitute for a personal interview. The lowest in-
come groups often do not have telephones.

7. If the interview is conducted in the home during the
day, the majority of the informants will be housewives.
If a response is to be obtained from a male member of
the household, most of the field work will have to be
done in the evening or on weekends. Since only an hour
or two can be used for evening interviewing, the personal
interview method requires a large staff for studies re-
quiring contacts with the working population. (pp. 79-82)

Computerized Data Retrieval System

The computerized data retrieval system was developed by the Florida

Inter-Institutional Research Council (IRC) as an alternative method of

obtaining information concerning community college graduates (Miner,

1977, p. 2). According to Norris Miner (1977), the computerized data

retrieval system used computer input data containing the graduate's

social security number (ssn) and program of study completed. These data

were matched by ssn with the third quarter of 1976 Department of Com-

merce files. The files contained employment data on all persons in

the state who were eligible for unemployment compensation. For every

match in ssn with graduate files, the number of weeks worked, total

wages, and the code for the employer were obtained. The graduate files

were then matched by ssn with the State University System (SUS) stu-

dent master file for term 1 of 1976-77. When a match occurred within

the SUS data tape, the university and program of study were obtained.

Member colleges of the IRC decided to compare two alternative

methods of follow-up by using both the mail questionnaire method, and
the computerized data retrieval method.

According to Miner (1977), a questionnaire was mailed to 443 grad-

uates of Seminole Community College. Then the computerized data re-

trieval system was performed with a match in ssn being considered a

response. Table 7 presents the results of the two studies.

The results showed the improved response rate that the computerized

data retrieval system supplied and also showed the more in-depth infor-

mation supplied by the questionnaire. The computerized data retrieval

system will be discussed using the five variables: costs, response rate,

length of time, personnel required, and the quality of data.


Since the costs were related to personnel costs, the costs for the

computerized data retrieval system were lower than the other methods of

follow-up. Miner (1977) reported that the computerized data retrieval

system was less expensive than mailing questionnaires and processing the

responses. Preparing the program, gathering the data tape, and using

computer time were the major expenses involved. The largest cost was

the initial cost of writing the program. After that, occasional up-

dating would allow the program to be functional. The cost of running

the computer program was not high, therefore the overall cost was low.

Also, since little human time was involved, this method was less ex-

pensive than other methods.

Response Rate

The response rate for the computerized data retrieval system in

Miner's study was 82.8% compared to the same study done with a mail

questionnaire response rate of 56%. The computerized data retrieval

Table 7

A Comparison of a Mail Questionnaire Study and a Computerized
Data Retrieval System Study


Data Retrieval

Completion rate 56% 82.8%

Response rate

Employed 12.9% 42.8%

Advanced education 81.5% 57.2%

Other 5.5% 3.3%

Note. From "The Problem with the Placement Study," by N. Miner,

Sanford, Florida: Seminole Community College, 1977, p. 6.

system had several constraints in regard to response. Any potential

respondent working or attending school out-of-state would be lost to

the study. Also, any potential respondent enrolled in a private school

would be lost to the study since neither would be on the two data tapes.

A follow-up of nonrespondents was conducted and Miner (1977) re-

ported that eleven nonrespondents were continuing their education with

seven delayed entrants into the SUS. Two were in private colleges and

two were in out-of-state colleges. Another five were found to be em-


The results also demonstrated a response bias in the mail question-

naire. It was found that graduates who continued their education were

more likely to responded to questionnaires.

The computerized data retrieval system had a built-in response bias,

the loss of responses from out-of-state graduates and private school stu-

dents. A "point in time" problem was apparent with the computerized data

retrieval system also. Some respondents were lost because they delayed

one or more school terms before entering the SUS. A longitudinal study

utilizing the computerized data retrieval system would eliminate part

of this problem.

Length of Time

The length of time needed to prepare a questionnaire, stuff the en-

velopes, mail them, wait for returns, and analyzed them, made the mailed

questionnaire much longer to perform than the computerized data retrieval

system. The computerized data retrieval system only waited on the pre-

paration of the data tapes once the program was initially written. It

could then be used innumerable times. The actual time needed to run the

program was minuscule once the results had been collected. This made.

the computerized data retrieval system require the shortest length of

time to perform an actual follow-up procedure.

Personnel Required

The only essential personnel for a computerized data retrieval

follow-up study was a person to perform the computer programming aspects.

The initial program had to be written and tested. Once it was operational,

then a computer operator could conduct a run of the program when neces-

sary, and the results would be ready for reporting.

Quality of Data

The data used in the computerized data retrieval system were data

supplied by college records and data supplied by the Department of Com-

merce. Miner (1977) reported that this eliminated any errors introduced

through self-reported data. Therefore the follow-up study results were

based on data that were as accurate as the college records and the govern-

ment employment records.

In regard to completeness of response, the computerized data retrie-

val system reported only the employer occupational code rather than the

occupational code of the respondent. Therefore, as Miner stated in his

report, the computerized data retrieval system did not provide direct in-

formation on related and nonrelated employment to program of study. This

imposed a lack of completeness with regard to employment relating to pro-

gram of study. The employer code, in some, but not all instances, sup-

plied sufficient information to determine the area of respondent employ-

ment, and whether or not the area was related to the programs of study at

the college.



Comparing different information collecting survey techniques has been

recognized as a difficult task. Dillman (1978) stated:

The question of which method is best cannot be answered
in abstract terms. Although each method has certain
strengths and weaknesses, they do not apply equally, or
sometimes at all, to every survey situation. Thus, until
the attributes of each method are considered in relation
to the study topic, the population to be surveyed, and
the precise survey objectives, the question of which is
best cannot be answered. (p. 39)

Table 8 is a replication of a table from Dillman's book Mail and

Telephone Surveys (1978). This book offers a comparison of mail question-

naire, telephone survey, and personal interview methods of conducting a

survey. The table lists the ratings of performance characteristics for

each of the three traditional survey methods as high, medium, or low.

This allows the researchers to find the performance characteristics im-

portant for a particular study and observe which survey method is high

or low in those important characteristics. Each survey will have some

characteristics which are more important than others. The survey method

chosen should have a high rating in as many of the important performance

characteristics as possible.

With the recognition of the difficulty of comparing follow-up methods,

this section will present a comparison of four methods of follow-up

based on the five criteria previously derived from the Review of Re-

lated Literature. An analysis of the advantages and disadvantages, and

Table 8

Performance Characteristics Rating for Personal Interviews
(Face-to-Face), Mail Questionnaires, and Telephone Interviews

Performance Face-to-Face Mail Ques- Telephone
Characteristics Interviews tionnaires Interviews

I. Obtaining a Represen-
tative Sample.

A. Known opportunity
for all members of
population to be in-
cluded in the sample.

1. Completely listed
populations. High High High

2. Populations which High Medium Medium
are not completely
listed (e.g., house-
hold occupants).

B. Control over selec- High Medium High
tion of respondents
within sampling units.

C. Likelihood that selec- Medium High High
ted respondents will
be located.

D. Insensitivity to Medium Low Low
substitution of re-
spondents and house-

E. Response rates.

1. Hetergeneous High Medium High
samples (e.g.,
general public).

2. Homogeneous, spec- High High High
ialized samples
(e.g., agency di-
rectors, ministers,

Full Text
xml version 1.0 encoding UTF-8
REPORT xmlns http:www.fcla.edudlsmddaitss xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.fcla.edudlsmddaitssdaitssReport.xsd
INGEST IEID EFFGSA094_HFTA3R INGEST_TIME 2012-03-09T00:03:32Z PACKAGE AA00003898_00001