This item is only available as the following downloads:
A METHODOLOGICAL STUDY OF VARIOUS
FOLLOW-UP PROCEDURES FOR COMMUNITY COLLEGE
PENNY PROCTOR NORIEGA
A DISSERTATION PRESENTED TO THE GRADUATE COUNCIL OF
THE UNIVERSITY OF FLORIDA
IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE
DEGREE OF DOCTOR OF PHILOSOPHY
UNIVERSITY OF FLORIDA
I would like to thank my committee, Dr. Stanley Su, Dr. James Hale,
and Dr. John Nickens, for their help and support during the writing of
this dissertation. I would like to extend special thanks to my chair-
man, Dr. John Nickens, for his continued encouragement throughout my
Personal thanks go to Dr. Debbie Hasselo and Dr. J. Nick Ehringer
for their assistance and exchange of moral support as we worked toward
dissertation completion. My appreciation also goes to Dr. Rosanne Gmuer
for her encouragement, understanding, and confidence. I would also like
to thank Carole Alonso for her diligence in typing this dissertation in
spite of the obstacles.
My gratitude also extends to my parents, Robert and Ann Proctor,
and my brother, Gary, for their sustaining belief in my abilities. Much
love and appreciation goes to my husband, Brian, for his advice, assis-
tance, and cooperation throughout this project.
TABLE OF CONTENTS
ACKNOWLEDGEMENTS . . ... ii
LIST OF TABLES . . ... v
ABSTRACT . .. . vi
CHAPTER 1 INTRODUCTION. . . .. 1
Statement of the Problem. . . 6
Delimitations . ... 6
Justification of the Study .............. 7
Assumptions. . . ... .. 8
Definitions of Terms . . 9
Procedures . ... ... ... 11
Organization of the Research Report. . .. 11
-CHAPTER 2 REVIEW OF RELATED LITERATURE. . ... 13
Introduction . . ... 13
Mail Questionnaire . ... 17
Telephone Survey . .... 58
Personal Interview .................. 72
Computerized Data Retrieval System . 86
CHAPTER 3 RESULTS . ... 91
Mail Questionnaire . . 95
Telephone Survey . .... 99
Personal Interview .. .. .. .. .. .101
Computerized Data Retrieval System . .104
Vocational Education Follow-up Studies ... .105
Follow-up Study of Vocational Program Graduates. .106
Follow-up Study of Vocational Graduates' Employers .. .120
CHAPTER 4 CONCLUSIONS . . .130
Comparisons. . .... .130
Evaluation .. .. ... ..... .. ..136
Mail Questionnaire . . .. 137
Telephone Survey . . .. .138
Personal Interview . .. .139
Computerized Data Retrieval System ........ .139
Further Research . .. .. 140
APPENDIX COMMON CORE OF QUESTIONS . ... ..145
REFERENCES . . .. .. .49
BIOGRAPHICAL SKETCH. . . ..155
LIST OF TABLES
Table 1 Cost Estimates for Mail Questionnaires .. 22
Table 2 Comparison of Major Study Costs Using Different
Monetary Inducements. . 24
Table 3 Costs for Mail Questionnaires and Personal
Interviews . . .. 26
Table 4 Christison's Cost Effectiveness Study Results. .. 27
Table 5 Cost Estimates for Telephone Interviews. ... 62
Table 6 Cost Estimates for Five Hundred Interviews 76
Table 7 A Comparison of a Mail Questionnaire and a
Computerized Data Retrieval System Study 88
Table 8 Performance Characteristic Rating for Personal
Interviews (Face-to-Face), Mail Questionnaires,
and Telephone Interviews . ... 92
Table 9 Analysis of the Four Methods of Follow-up Based
on Five Criteria . . 96
Table 10 Comparison of Cost for Different Sample Sizes. 107
Table 11 Ratings of the Four Methods of Follow-up Across
the Five Qualitative Criteria for Vocational
Graduates Follow-up. . 119
Table 12 Ratings of the Four Methods of Follow-up Across
the Five Qualitative Criteria for the Employer
Follow-up. . . 128
Abstract of Dissertation Presented to the Graduate Council
of the University of Florida in Partial Fulfillment of
the Requirements for the Degree of
Doctor of Philosophy
A METHODOLOGICAL STUDY OF VARIOUS
FOLLOW-UP PROCEDURES FOR COMMUNITY COLLEGE
Penny Proctor Noriega
Chairman: John M. Nickens
Major Department: Educational Administrative and Supervision
The problem of this study was to compare and evaluate selected
methods of conducting follow-up studies of graduates of community college
vocational education. Four methods were chosen for in-depth analysis:
mail questionnaires, telephone surveys, personal interviews, and computer-
ized data retrieval systems. Related literature was searched for criteria
on which to compare the methods. Five criteria were chosen: costs, re-
sponse rate, length of time, personnel required, and quality of data.
The four methods were then analyzed based on the five criteria. The rela-
tionships between the four methods of follow-up were discovered and reported.
Interrelationships between the five criteria and the four methods of follow-
up were also found. The advantages and disadvantages of each method were
analyzed and the subsequent strengths and weaknesses of the methods explored.
Consideration was given to each method's ability to collect follow-up data.
Theanalysis produced suggestions for optimal use of each method.
Specific methodological analyses were performed and recommendations
made for two common vocational education follow-up needs, studies of
vocational education employed graduates and employers of these graduates.
The results indicated that the mail questionnaire method of follow-
up was advantageous in obtaining follow-up information concerning soci-
ally sensitive subjects. This method was also found to be the most ef-
fective in costs and time when dealing with large sample sizes. A major
disadvantage of this method was low response rates. Various techniques
for improving response rates were reported in the Review of Related
The telephone survey was found to be advantageous when performing
a small, spontaneous study. Costs, time limits, personnel requirements,
and sample size were found to be interrelated in the telephone survey.
The telephone survey was found to be an acceptable compromise between the
mail questionnaire and the personal interview.
The personal interview was found to produce high costs but also pro-
cured the highest quality data in relation to complexity and depth of
answers. Sample size, geographic dispersion, costs, personnel required,
and length of time were found to be interrelated in the personal inter-
The computerized data retrieval system was found to have many ad-
vantages: low costs, short time period, and few personnel requirements.
However, the type of data that could be collected was limited, and the
geographic location of respondents was restricted.
Overall results indicated that for the purpose of follow-up voca-
tional graduates, either a mail questionnaire or a telephone survey
would be preferable. The computerized data retrieval system could be
utilized for limited data or as a validation procedure. For a follow-up
of employers of graduates of vocational education, the personal inter-
view method was found to be preferable to the other three methods for
in-depth evaluation. If more general information were desired, the tele-
phone survey and the mail questionnaire could be utilized.
A table was developed from the analysis listing the four methods of
follow-up and the five criteria. The strengths and weaknesses of each
method were reported for each criterion including sample size and geo-
graphic dispersion relationships.
During the 1970's an increasing emphasis on accountability was
imposed on educators by legislators and the general public. "Social
unrest and dissatisfaction with the social institutions--of which
the educational system is only one--have led to the politics of con-
frontation by which results, rather than promises, are demanded"
(Morphet, Johns, & Reller, 1974, p. 542). Legislation intended to ac-
complish accountability has become common as legislators attempt to
determine the results which are being achieved in relationship to
moneys appropriated for education (Morphet et al., 1974).
Educational accountability is not easily explained or measured.
Williams and Snyder (1974) explained accountability in the following
Accountability includes evaluation of educational processes
and outcomes in terms of stated objectives and communication
of findings to educators and the public. The findings from
evaluation projects, of course, also provide a base for modify-
ing educational objectives and programs. (p. 40)
Morphet et al. (1974, p. 545) stated that one of the necessary condi-
tions in accountability is that "goals and objectives must be stated
inmeasureable terms." Gleazer (1973, p. 110), when president of the
American Association of Junior Colleges (AAJC), stated that accounta-
bility is available to vocational programs in the forthright terms of
getting jobs for the people trained, preferably jobs related to their
training. Brantner (1975) stated that it is how these vocational edu-
cation graduates fare in the world of work that is the proper measure
of accountability. Kaiserchat (1972) reported that the design of
follow-up studies may well be able to answer any existing questions of
Follow-up, usually considered one aspect of an evaluation system,
is designed to obtain data, information, or opinions about or from
former participants in the educational process (Headrick, 1977, p. 2).
Data frequently sought include length of time between graduation and
employment, field of employment, type of work performed, salary, and
promotion history. Student evaluations of their programs and training
received while attending the college, and employer opinions on the
level of training exhibited by their employees who are graduates from
the college can also be collected. Information collected through fol-
low-up studies is used to measure the effectiveness of courses, and is
essential to systemized evaluation of vocational programs (Headrick,
1977, p. 2).
Employability is an element of vocational education accountabil-
ity. Many community college leaders are required to supply much of the
information needed to answer the questions raised by accountability.
Unfortunately, the more information desired, the more difficult and
complex becomes the process of obtaining the information. The trade-
off in follow-up studies is between the amount of information collected,
and the cost and time that can be afforded by an institution for the
acquisition process. The more information desired, the more complex
and difficult the information acquisition techniques, necessitating
higher costs in institutional resources. In many instances, the acqui-
sition of information deemed necessary is an impossible task due to the
lack of cooperation on the part of the population or sample being fol-
Gold and Morris (1977) stated the following observation after
working with follow-up studies. "It is almost universally agreed that
the task of obtaining follow-up information that is reliable, valid,
and useful is a difficult one" (p. 7).
Some of the major limitations associated with the traditional
methods of follow-up have been high costs, extended length of time for
completion, extensive use of college personnel, low response rate to
questionnaires, and uncertain reliability of information (Baugh, 1976;
Krueck, 1975; Miner, 1977; Paul, 1976). Some of these limitations
are found wholly or partially in every follow-up study.
In 1973, the Virginia Department of Community Colleges began a
national review of follow-up studies made by community college person-
nel of former vocational education students. It was found that the
great majority of community college leaders were not satisfied with the
follow-up methods they found available (Williams & Snyder, 1974, p. 40).
Krisham Paul (1976) reported that researchers "still have not developed
adequate measures which would be acceptable to most administrators of
vocational education" (p. 1). Therefore, there is an obvious need for
better and more feasible methods of follow-up.
The traditional methods of follow-up have been the mail question-
naire and the personal interview. A mail questionnaire study involves
mailing a list of written questions to potential respondents for them
to answer and return by mail. A personal interview study involves a
person acting as an interviewer, usually going to the potential re-
spondents, verbally asking the questions, and recording the answers on
an answer sheet. Jackson and Rothney (1961) made the following state-
ment concerning the need for validation studies on the traditional
research methods employed for follow-up:
The merits, weaknesses, and biases of both the mail question-
naire and the interview have been discussed frequently, but
there has been little research designed to assess the compara-
tive contributions to the two procedures for follow-up purposes.
Krisham Paul (1976) agreed that data collection is primarily geared
to questionnaire surveys and interviews, and pointed out they both suf-
fer from the problems of nonresponse and inadequate sampling.
The methods of follow-up have been expanded to include two re-
cently employed methods. The first is the telephone survey about which
little comparative research has been conducted. The second is the com-
puterized data retrieval system, first utilized by the Florida Commun-
ity/Junior College Inter-institutional Research Council (IRC) in 1975.
The telephone survey consists of interviewers calling on the tele-
phone and verbally asking prospective respondents a series of questions.
The respondents answer the questions verbally and the interviewer fills
in the answers on an answer sheet. This eliminates travel expenses
but can bias the sample since not every prospective respondent has a
telephone (Weiss & Hatry, 1971, p. 23).
The computerized data retrieval system utilized a new data source
for education, the Department of Commerce data file. This method was
used by the IRC to supply information for State of Florida government
reports. The IRC obtained a data tape of employment information from
the Florida Department of Commerce located in Tallahassee, Florida.
The data tape contained pertinent information about salary, weeks
worked, and industry code, all classified by social security number
The IRC next obtained data tapes from its community college mem-
bers and the universities of the state of Florida. These data tapes
had student information also classified by ssn. The data tapes were
then merged by matching ssn. Colleges who sent the ssn of their voca-
tional programs' graduates on their data tapes had the numbers matched
to the numbers on the Department of Commerce data tape, and the data
corresponding to the matched ssn was extracted from the Department of
Commerce tape. The data were then organized and analyzed to perform a
vocational education follow-up study.
The research validating the various methods of follow-up is ex-
tremely limited. Only a small portion of the research in the area of
follow-up surveys has dealt with comparisons of methodology and proce-
dures of follow-up methods, and those studies appear to have serious
limitations for utilization (Krueck, 1975, p. 4). Therefore, there
appears to be a need for a study which will compare the various meth-
odologies of follow-up, including techniques used to overcome their
limitations, to evaluate the usefulness of each method for the benefit
of those who wish to employ follow-up studies for the betterment of
Statement of the Problem
The problem of this study was to compare and evaluate selected
methods of conducting follow-up studies of graduates of vocational edu-
cation. These methods included the mail questionnaire, the telephone
survey, the personal interview, and the computerized data retrieval
system. This study used five criteria to compare the four methodolo-
gies previously listed. The advantages and disadvantages of each
method were identified and evaluated for appropriate use in conduct-
ing follow-up information research. The following five criteria formed
the basis of comparison:
1. Cost. The monetary expenditures associated with the methods
used to perform a follow-up study.
2. Response rate. The ratio of completed responses to the total
number of possible responses, expressed in percentages.
3. Length of time. Estimates of length of time involved in the
methods used to perform follow-up studies data collection.
4. Personnel required. The amount of people and their necessary
skills and training required to perform the methods of follow-up
studies data collection.
5. Quality of data. The accuracy and completeness of the data
The review of related literature for this study was limited to
studies reported in the Educational Resources Information Center (ERIC)
and the Education Index for the years 1970 to 1978, and available pub-
lications in the libraries of the University of Florida and the Univer-
sity of South Florida.
It was not the intent of this study to review the basic funda-
mentals of follow-up methodologies. Rather, the most recent strategies
used in follow-up studies were reviewed to discover the best use of
each strategy. Thus, the details of instrument development, prepara-
tion, and evaluation procedures were not studied.
Justification of the Study
The addition of efficient and more complete methods of research
is an advancement in scientific methods. New and better methods of
performing research are always in demand. The gathering of reliable
and valid information is a prevalent problem associated with vocational
education follow-up studies. In addition to being complex, acquiring
reliable and valid information has also been expensive.
Follow-up studies are a type of survey research which are used to
obtain information from or about former students of educational institu-
tions. According to Berdie and Anderson (1978) there is a need for
systematic study of survey research methods so they can be used cor-
rectly and efficiently. This study systematically compared various
methods of follow-up and thereby supplemented the literature regarding
the utility of follow-up methods for acquiring information essential
to community college vocational programs. This study synthesized in-
formation concerning the traditional methods of follow-up and the more
recently applied methods. They were evaluated and compared to discover
the extent to which the more recent methods eliminated or minimized
some of the problems associated with traditional methods of follow-up.
Besides conducting a comparison among follow-up methods, this
study added to the literature about two recent developments in follow-up
methodologies. One recent method about which limited information is
available is the telephone survey. The telephone survey has been em-
ployed in a limited capacity and has experienced limited exposure in
the literature. Comparisons among the telephone survey and the other
methods in the literature are rare.
Another recently employed method with limited literature is the
computerized data retrieval system. This method includes a new source
of data for educational researchers, the state Department of Commerce
data tapes. Since the computerized data retrieval system has only re-
cently been utilized, very little research has been reported regarding
its usefulness in educational follow-up studies. This study evaluated
the extent of the usefulness of the computerized data retrieval system
and its potential as a method of follow-up for community college voca-
These new follow-up methods could provide information vital to
community college vocational programs in the areas of government report-
ing, self studies, and program improvement. Improved methods of infor-
mation collected could also assist in the advancement of knowledge
concerning educational benefits accruing to all students.
An important assumption for this study was that the five criteria
chosen by this researcher from a review of the literature were valid
for evaluation and comparison of follow-up methods for community col-
lege vocational education use.
Definition of Terms
Accountability. The concept that education should be evaluated in
terms of stated objectives and goals.
Community college. A two year institution of higher education
with programs designed to meet the post secondary education needs of a
Computerized data retrieval system. The process of using a compu-
ter program to locate necessary stored data, matching related data sets
from several sources, and organizing the data into usable information.
Data tape. A magnetic tape that has data stored in the form of
magnetic charges, usually containing a multitude of individual records.
Florida Department of Commerce tape. A data tape on which the
Florida Department of Commerce has stored employment data of all em-
ployed persons in the state of Florida.
Follow-up study. A study conducted to observe the subsequent ex-
periences of students after terminating their enrollment in an educa-
Graduate. A recipient of either an Associate of Science degree,
Associate of Arts degree, or a certificate of completion from a commun-
Interviewer bias. The respondent's responses are affected by some
characteristic or attitude of the interviewer.
Mail questionnaire. A questionnaire mailed out to a population
or a sample of a population with a list of questions for people to an-
swer and mail back.
Monetary incentive or inducement. Offering respondents money to
respond to a mail questionnaire.
Personal interview. An interview conducted on a one-to-one corre-
spondence with the interviewer in the presence of the person being
interviewed. The interviewer asks the questions and records the an-
Response. A completed questionnaire or interview. Also, a match
in the computerized data retrieval system.
Response bias. Those who do not respond to a questionnaire or
interview may differ in nonrandom ways from those who do respond, there-
fore biasing the results.
Telephone survey. An interview conducted over the telephone
rather than in the presence of the person being interviewed. An inter-
viewer calls and asks a series of questions. The respondents answer
the questionnaire verbally over the phone, and the interviewer records
Sample. The utilization of only a portion of a population to
study the entire population. The sample must be selected in such a way
as to meet certain criteria, in order to obtain results generalizable
to the total population.
State University System. The collection of nine public universi-
ties of the state of Florida.
Vocational education program. An educational program designed to
prepare the student for direct entrance into employment, usually at the
semi-professional or technical level. The program at a community col-
lege is typically one or two years in length.
The procedures for this study involved three major phases. The
first phase was a review of the related literature of follow-up studies.
The second phase was data organization. The third phase was an evalu-
ation of the various methods of follow-up.
Review of Related Literature
The review of related literature involved the study of four methods
of follow-up: the mail questionnaire, the telephone survey, the per-
sonal interview, and the computerized data retrieval system. A brief
description of each method was reported. Information concerning costs,
response rate, length of time, personnel required, and quality of data,
relating to the four methods was obtained and reported. Suggestions
for improvement of follow-up studies presented in the literature were
Once the review of literature was completed, the required informa-
tion concerning the five evaluative criteria was organized to facili-
tate comparison among the various follow-up methods.
The costs, response rate, length of time, personnel required, and
quality of data were compared across the four follow-up methods. Each
method was then evaluated in terms of strengths and weaknesses. Recom-
mendations were made regarding the most beneficial use of each method
for vocational education follow-up.
Organization of the Research Report
This study is reported in four chapters. The first chapter in-
cludes the background and rationale of the study, statement of the pro-
blem, delimitations, justification of the study, assumptions, defini-
tions of terms, and procedures. Chapter two presents the review of the
related literature, which includes a qualitative review of the four
methods of follow-up and a review of literature concerning the five
evaluative criteria to be associated with each method. Chapter three
continues a discussion of the four methods of follow-up and a compari-
son of the four follow-up methods relative to the five evaluative cri-
teria. Chapter four draws conclusions for follow-up studies as well as
implications for further research.
REVIEW OF RELATED LITERATURE
A review of the literature demonstrated that educational surveys
and follow-up studies in particular are a recent development in higher
education. "The American survey of higher education, which is largely
a product of the twentieth century, is only one section of the broader
field of educational surveys in general" (Eells, 1937, p. 15). This
statement was made by Walter Crosby Eells, professor of education at
Stanford University, in his 1937 book, Surveys of American Higher Edu-
cation. The book was written as a comprehensive study and evaluation
of the survey reports that had been published up to that time. At no
point did the author mention follow-up studies, for at that time, no
Junior and community colleges were created early in the twentieth
century and began to expand to include vocational education at the post
secondary level during the 1920's and through the 1940's (Hillway, 1958,
p. 6). It was in this area of education that the scope of evaluation
was first broadened to include the follow-up of program completors
(Headrick, 1977, p. 2). The follow-up study has continued to develop
and expand along with vocational education and junior and community
colleges, whose growth increased tremendously during the years 1960
through 1975 (Jennings, 1975, p. 29). With the era of accountability
in the 1970's, follow-up studies were increasingly used for justifica-
tion for public funds spent on vocational programs (Headrick, 1977,
p. 3; Williams & Snyder, 1974, p. 40).
Accountability includes evaluation of educational processes and
outcomes in terms of stated educational objectives, and the communica-
tion of these findings to educators and the public (Williams & Snyder,
1974, p. 40). Because vocational education programs are relatively
new in higher education, it is considerably more difficult to develop
an accountability strategy for them than for traditional college pro-
grams, which evaluate themselves in terms of degrees produced and aca-
demic honors (Williams & Snyder, 1974, p. 40).
The need to gather accountability information for vocational edu-
cation programs was filled by the follow-up study. Originally, follow-
up procedures were employed in vocational education as a segregated
process, with little or no regard for integration with formal evaluation
or management plans (Headrick, 1977, p. 2). This restrictive approach
toward follow-up procedures has been in rapid decline as educators
become increasingly aware of the need for comprehensive management and
systemized evaluation programs (Headrick, 1977, p. 2).
McKinney and Oglesby (1971) stated a very general definition which
covers all types of follow-up studies. They stated that a follow-up
study was "a procedure for accumulating pertinent data from or about
individuals after they have had similar or comparable experiences" (p. 7).
There are various reasons for performing a follow-up study and collect-
ing pertinent information. Headrick (1977) listed four purposes for
performing a follow-up study:
1. To improve programs;
2. To provide a basis for accountability;
3. To aid in planning; and
4. To aid in decision making. (p. 3)
The Southern Regional Education Board (SREB) agreed with Headrick
that a college conducts a follow-up study for a variety of reasons.
Basically, however, according to the SREB, the main reason a college
or university conducts a follow-up study is to assess how well it has
met its objectives (SREB, 1978, p. 1). How well the objectives are
being met, which is the effectiveness of an educational program, is
important, but Brantner (1975) pointed out that before these can be
measured, the results must be observable.
The first objective of the follow-up study is to report the re-
sults of the educational programs by observing how the graduates fare
in the world of work. Once the results have been reported, then the
evaluation can be performed on the information gathered. Evaluation
should be done only on true and accurate results. Therein lies the
importance of the follow-up study, to gather reliable, accurate, and
pertinent information about the results of educational programs, and
evaluate and report on these results.
Follow-up studies for vocational programs are usually quite clear
on the type of information they wish to gather because vocational edu-
cation is well defined. Vocational education is any program whose major
goal is that the student shall attain gainful employment after a course
of study requiring no longer than two years for completion (Cohen, 1971,
p. 137). With the major goal of vocational education so clearly de-
fined as gainful employment, most vocational follow-up studies concen-
trate on employment information of their graduates.
Krisham Paul (1976) listed three questions for which most voca-
tional follow-up studies might seek answers.
1. What happens to the youths and adults who, after completion
of their training, try to find work rather than continue
2. Are there significant differences between the job-related
experiences of persons who complete vocational education
programs and those who do not?
3. Do the vocational education programs justify their costs?
Most of the vocational follow-up studies that are reported in the
literature have focused on the first question concerning the experiences
of graduates who seek employment. Information on comparison of voca-
tional graduates to nongraduates is difficult to collect and the ques-
tion of vocational programs justifying their costs involves a value
The SREB stated that traditional follow-up procedures make the
assumption that the graduate's own perception of how well the institu-
tion has fulfilled its objectives is one way of assessing the attain-
ment of program objectives (SREB, 1978, p. 2). Other methods of assess-
ment that can be used are scores on graduate record exams and competency
tests. Nevertheless, the major focus of most vocational programs'
follow-up studies stresses the opinion of the graduates and concentrates
on gathering information from them (SREB, 1978, p. 2).
There are three methods of follow-up customarily used and all of
them concentrate on gathering information from graduates. These are
the mail questionnaire, the telephone survey, and the personal inter-
view (Denton, Ash, Brown, Dowell, Gutcher, Hoyle & Swinney, 1977).
These methods all depend on a question and answer format. The questions
depend on the type of information desired and the purpose of the follow-
up study. The Southern Regional Education Board has published a manual
of procedures and common core questions for follow-up studies of voca-
tional programs of two year institutions (1978). The manual includes
criteria for the selection of follow-up questions and recommends that
an institution add questions that are tailored to its own needs while
using the suggested core questions for inter-institutional comparisons
The most frequently used method of follow-up is the mail question-
naire. It will be explored and reported on first, followed by the
telephone survey and the personal interview. Last to be reported on
will be the computerized data retrieval system.
Champion and Sear (1969) reported that the mail questionnaire has
been used for more than 40 years (p. 335). It has been used extensively
in many areas of social research and has been the most frequent method
employed with educational follow-up studies (McKinney & Oglesby, 1971,
The mail questionnaire procedure involves the mailing of a written
list of questions, called a questionnaire, to potential respondents.
The respondents then answer the questions by writing on the question-
naire, and mail it back.
The mail questionnaire has many advantages over the other survey
methods. It is generally agreed that the mail questionnaire is the
least expensive method of conducting a survey of a large sample or
population (Berdie & Anderson, 1974, p. 17). Compared with the second
most popular method of data collection, the personal interview, the
mail questionnaire has the following three major advantages:
1. If its questions aim at situations or decisions which con-
cern the entire household, rather than one individual,
then the mail questionnaire will offer the advantage of
opportunity for consultation within the family; the personal
interviewer rarely has a chance to obtain a "family inter-
2. More important, unavailability is sometimes a serious handi-
cap for the personal interviewer.
3. Third, its impersonality often prompts different shading to
the replies than those obtained when the respondent is part
of the psychological situation of being questioned by another
person. For some types of questions, the mail responses are
probably more specifically candid. (Franzen & Lazarsfeld,
1945, p. 294).
Benson (1946) stated that "in mail surveys there is no chance of
an interviewer intentionally or unintentionally, biasing the answers"
(p. 234). These are reasons why the mail questionnaire has become the
most widely employed method of follow-up. But, as with the other
methods of information acquisition, the mail questionnaire has its dis-
The mail questionnaire survey procedure will be reviewed using the
five variables previously chosen to compare and evaluate the various
follow-up methods. These variables are costs, response rate, length
of time, personnel required, and quality of data. A review of related
literature found a substantial amount of written material concerning
certain variables, while other variables were scarcely mentioned. There-
fore, information on each variable needed for later comparison is re-
ported, with those variables being of major interest to past researchers
having a greater quantity of research to report.
Costs for mail questionnaires were difficult to isolate and compare
across studies. Different studies included different types of costs, in-
curred in different years, and utilizing different techniques. The va-
riety of methods utilized to increase response rate made comparison of
costs across studies very complicated. Therefore, costs will first be
reported for various individual questionnaire studies, followed by com-
parison studies and cost effectiveness studies.
The SREB reported that costs of follow-up surveys vary widely, de-
pending primarily on whether or not staff costs were included. The
National Center for Higher Education Management Systems (NCHEMS) esti-
mated materials and mailing costs range from 17q to 26t per student con-
tact (as contrasted with student response) for nonprofit bulk and first
class rate mailing, respectively. This included only one mailing (SREB,
1978, p. 23).
The University of North Carolina (Chapel Hill) reported costs of
35.6t and 41.5 per student contact for surveys performed in 1976 and
1977, respectively. These amounts included only one mailing, first
class, as well as staff costs in processing, survey preparation, and
mailing. They did not include costs for analyzing responses and pre-
paring a report (SREB, 1978, pp. 23-24).
The University of Illinois reported costs of $1.86 and $1.50 per
student contact of 1972 and 1973 graduates, respectively. These costs
included three mailings and costs for staff, computer time, evaluation
and reporting (SREB, 1978, p. 24).
Navarro College, a two year institution in Texas, monitored follow-
up questionnaires and costs for six institutions while helping in the
development of the Texas Information System. The college reported that
staff costs accounted for 70% of total follow-up costs. For mailed
follow-up questionnaires, with response processed by computer, the per
student cost was calculated to be $3.27, including 70% staff costs.
Three and four mailings were included in the follow-up studies (SREB,
1978, p. 24).
Student contact was based on the number of questionnaires delivered.
Response rate refers to the percentage of questionnaires returned, a more
practical figure to use when evaluating costs. Increasing response rate
is important to researchers and the cost rise according to the attempts
and techniques used to increase the response rate. Therefore, costs
should be associated with response rate to show a more accurate indica-
tion of how much can be spent to increase the return of questionnaires.
Hochstim and Athanasopoulos (1970) reported a mail questionnaire
study performed in 1965. The total costs for the study were $37,000
for 5,630 returned questionnaires, a 70% response rate. The cost per
response was calculated by dividing the total cost by the number of
returned questionnaires to obtain a cost of $6.55 per questionnaire re-
turned. Since researchers are clearly interested in returned question-
naires, cost related to returned questionnaires is a more meaningful
figure than cost per student contact. The latter does not take into
consideration the response rate of the study.
Weiss and Hatry in their book, An Introduction to Sample Surveys for
Government Managers (1971), listed some cost ranges based on survey ex-
periences, and included other cost information that could be synthesized.
They broke down the costs to the various components of a mail question-
naire and listed them as seen in Table 1. These cost estimates were
based on 1971 cost figures. Both a second mailing of the questionnaire
and telephone calls to nonrespondents were used to increase response rate.
Also included was a sampling of nonrespondents to test for response bias,
since just over 50% response rate was achieved. Their estimate for the
cost of the mailed questionnaire was $8.10 per response.
Krueck (1975) reported a study where he compared the results of
three alternative methods of follow-up with the purpose of examining
the rate of response, costs, and nature of the response. Krueck divided
the costs into two sections, fixed costs and treatment costs. Fixed
costs were costs that applied commonly to all three treatments such as
printing and processing. Treatment costs varied depending on the pro-
In Krueck's study, the mail questionnaires' initial response rate
was 37.4% of all mail questionnaires. A telephone call to remind po-
tential respondents increased the response to 42.7%. A second tele-
phone reminder brought an additional 7.3% for a total of 50.6% response.
A total of 164 questionnaires was initially mailed out at a cost of
16t each, including a self-addressed, stamped return envelope. Seven-
teen additional questionnaires were mailed after telephone reminder
Cost Estimates For Mail Questionnaire
1. Overall planning and direction
2. Preparing listing
3. Selecting sample, addressing labels
4. Developing questionnaire (including confer-
ences and 30 pretest interviews)
5. Printing questionnaires, cover letter, return
a. Telephone follow-ups to nonreturners
b. Interviews with 50 nonreturners (to check
whether they differ from questionnaire re-
(1) by telephone (approximately 30)
(2) in person (approximately 20)
7. Editing, coding, punching, tabulating about
1,000 returned questionnaires
9. Mailing, including one mail follow-up to non-
returners (approximately 1,500)
10. Analyzing data and report writing
11. Reproducing report
Average cost of the estimated
For 2,000 ques-
Note. From An Introduction to Sample Surveys for Government Mana-
gers, by C. H. Weiss & H. P. Hatry. Washington, D. C.; Urban Institute,
1971, p. 46.
calls. Thus, $27.60 was expended for the mailing of questionnaires.
The work involved 46 manhours for a total cost of $165.60. Therefore,
the cost per response was approximately $2.00. Krueck (1975) found
this procedure to be the least expensive of the three procedures.
Dillman (1972) performed large mail surveys in 1970 and 1971. The
costs were approximately the same both years. Direct costs for postage,
paper and printing of the questionnaires, envelopes, and cover letters,
amounted to 68U per completed questionnaire. Labor costs were figured
at the rate of $2.00 per hour and added to direct costs bringing the
total cost per response to $1.46 (p. 257).
Wotruba (1966) investigated the effects of monetary inducement and
no inducement on the percentage and completeness of responses to a
mailed questionnaire. He performed a pretest to ascertain how many
questionnaries needed to be mailed with 25t included, a promise of 50,
or no monetary inducement, to obtain a return of 750 questionnaires.
The pretest results demonstrated that with a 250 monetary inducement,
1,875 questionnaires would have to be sent out to obtain 750 completed
questionnaires. With a promise of 50t, 3,750 questionnaires would
need to be sent out, and if no monetary inducement was included, 4,167
questionnaires would need to be sent out. Wotruba's results showed
that the approach using the 25t inducement would incur the least ex-
pense when completeness of responses (all questions answered) was im-
portant. If completeness was not important, the approach with no
inducement was least expensive. Table 2 presents a comparison of costs
between the three inducement alternatives. Second mailings to compen-
sate for incomplete questionnaires were included.
L L 000 0 o C n L LO -W C0 n LI
E o0 o oCo iLn n L o Co
Lim cr L tl 0 a lO Cr- X 00rO -O- X 0C) 0
3 C) 0 o CM in m- C (v i cn oo
S .0 co m Or- LO i-
C <30 0 .0 4-
9- i-X 0
w 0 *+)
4 ( -0: -C C
E *r- Um U" LO LO m o 0M r., r- r~% r% Ln LO 3
S a) a) 4
l0 000 C0o f% CMMM / 0LO0 0n
L Ll 0 o -f- CM Ul *
Io U- r -- C% M o ,- I- r- CM un
Sf 0 CM N" C. a C
-- IC < r*. C ..- i-
O C ON CO )
S r->. In ^
a) +41 Ln COOOC 0 0 00 .0-U- C O* a)
4- c ri. cMLno- CD r-, t.Or-LOl cnd- sl-
4- a .
*r I OLC) CO ;-CJ0; X O k.Ln X 3 cc; Xm x CV m
0M tru 0 Ln co Oi a% C\j c C:
N c 00 dLn cO 00 0) C
mCT) a- CM r o
a) c CM -. -*
0 ) 0 -
0C 0. r
4-) a) U) (U
(/ L/ + a4 -)
C) a) La) )
L C r- a
So -+j 0= 0
C u in 2a) n D en
.. in t- -a C
4- -f 0a) 3S-
0 oo h- c* r E 2 S.
C O R. QS-- L La )
o U- (D a 4.3 4.3 |
( C O -0) 0 CC aa)
*i-cu r 5n *SL E a (V re i- s- "
s. Cd) y-.r- r-. ca 0) -fyt--.. c Q. 'U '.0
r_ C 4C) + *- C:-P U) 0 E 413 to
S0 C4-J 0
0 Cda a) 41 m.'4.J 0
0 f. Ca) (a "a r +- a
4J 4- +J3 o m 4- +3 -0i = S
+.p_-r_ ) c O: C i C: E S-
Sto -r- .V) 4-)- W CL S- S- 0 'U
a)2 E cU 4- E Li ea 2 00 a)c
U 0t- ) 4-Ca) r4- (D ca a) C5 4-4- LL. in
S- 0 L S-3L O C f S- S- U a
CU+4 +3 J QUJ 4 3) 4 + 3 3 c)
>> 0 ( cu c Ul cC c ct 0n m w )o
a S- 0S OU C 0 0 4 C
u co U 4-0 00 0 O
+J C, .. *r *. Z +4
VU)) a-a4- 01 4) a a)
(D 4-) (a U .- +j > o
CLu Ln 4+ L.3 (A 4+) 44
Xa)0O O a0 0 00 0
LUI = ) I-Q-CL I-c- CS)
Ronald Baugh (1976) performed a follow-up cost study which com-
pared the cost of mail questionnaires to personal interviews, as well
as manual versus computer analysis of follow-up results. The results
of Baugh's study can be seen in Table 3. A comparison showed the per-
sonal interview costs to be 3 1/2 times more expensive. The cost for
a mail questionnaire with computer processing was $8.20 per response
and for manual processing was $4.85 per response.
Baugh (1976) concluded that there must be some optimum number of
responses where manual processing is less expensive than computer pro-
cessing, however; above the optimum number of responses, computer pro-
cessing would be less expensive than manual (p. 10). He also reported
that mailing bulk rate did not save money due to the use of personnel
to sort and bundle the bulk mail.
Christison (1977) performed a cost effectiveness study for follow-
up questionnaires with various methods to increase response rate. He
concluded that a reminder phone call, a second copy of the cover letter
and questionnaire, and prior contact by phone before mailing the in-
strument were all cost effective. High cost and high returns were as-
sociated with offering several dollars as monetary reward. Least
expensive, but with low return, was a single mailing of the cover let-
ter and questionnaire with no further contacts. The costs associated
with Christison's study are listed in Table 4.
Erdos (1974) stated that low overall costs result in more flexi-
bility per dollar spent for the mail questionnaire. The combined costs
of mailing, postage, and incentives offered are usually less than hir-
ing staff for personal contacts. Because of this difference in expense,
Costs for Mail Questionnaires and Personal Interviews
Total Cost per
Computer processing 271 271 28.672 28.672
Manual processing 0 0 0 0
Note. From "Follow-up Cost Study" by R. C. Baugh, Forth Worth,
Texas, Tarrant County Junior College, 1976, p. 26.
0o C co co UL co r
Mv Ln IC Ll) k0 ko rO
CM (> o o o
SLY LO 0 0 0 0C
00 C0 0 0 0 0
co (NI 1.0 C a 0 <
; (v, C;00
LO uO) L L LO n n
Cej C(N C.J C"j C\J C\M C\j
one could afford to produce a large number of completed questionnaires
by using the mail method (p. 2-91).
When looking at costs of mail questionnaires in relation to the per-
sonal interview, most researchers agreed that the mail questionnaire costs
less than the personal interview (Benson, 1946, p. 234). Berdie and
Anderson (1974) concurred with the above statement. They stated, "the
expense of printing questionnaires and distributing them to large numbers
of people is considerably less than that of interviewing similar numbers
of people" (p. 17). Gordon (1975) agreed, summarizing the advantage of
the mail questionnaire as being its economy (p. 77).
Kerlinger (1973) stated that the mail questionnaire has serious draw-
backs, the primary defect being lack of response, or not enough question-
naires being completed and returned. Most researchers criticized the
mail questionnaire basically for the low response frequently encountered
and for the ease of misinterpretation. Mildred Parten (1950) went so far
as to describe mail questionnaires as almost invariably open to question
(p. 400). As stated by Hackler and Bourgette (1973) the problem with re-
sponse rates was that those who did not respond might differ in nonrandom
ways from those who did respond (p. 276). This brings response bias into
the questionnaire results. Investigators who have supported the notion
of bias produced by nonresponse were listed by Leslie (1972) as Rothney
and Mooren, 1952, Bennett and Hill, 1964, Anastasiow, 1964, Norman, 1948,
Suchman and McCandless, 1940, and Pucel, Nelson, and Wheeler, 1971. Leslie
(1972) questioned these researchers and their common view that low response
necessarily biases the results.
According to Leslie (1972), the studies that have dominated the
theory of response bias are based primarily on the late 1940's and
early 1950's studies. He concluded:
The evidence of bias in these studies, however, was in most
cases the demonstration that demographic traits of the re-
spondents, differed from non--or late respondents. Differ-
ences between respondents and non--or late respondents, ac-
cording to answers to substantive questions posed, were not
usually shown. (p. 327)
Leslie (1972) developed the following generalization to counter-
act the prevalent view that response bias severely limited the useful-
ness of surveys and questionnaires which did not have 100% return:
1. There is ample evidence that response rate bias may occur
in mail surveys. However, much of the available evidence re-
veals only differences between respondents and nonrespondents
or late respondents in terms of such independent variables as
sex, geography, age, etc. It is often assumed that these dif-
ferences lead to differences between respondents and nonrespon-
dents on the dependent variables, i.e., the questions under study.
2. When populations surveyed are homogeneous (having a common
group identity), minor differences on independent variables be-
tween respondents and nonrespondents or late respondents may oc-
cur, but differences as to dependent variables are unlikely.
Leslie's (1972) conclusion was that researchers surveying issues
directly related to homogeneous groups should not be overly concerned
about the percentage of questionnaire returns, just that enough re-
sponses are gained to meet statistical assumptions. Benson (1946)
said something similar to Leslie's conclusion. Benson stated:
If the segment of the population to be surveyed is homogeneous,
the differences in tastes or opinions within that group are
much more restricted. Therefore, in a mail survey, results
for returns, from a small proportion of such a group would tend
to be more like the whole than is the case when tastes or opin-
ions within the group to be sampled are very different. (p. 237)
Many researchers still consider response bias the primary weakness
of the mail questionnaire. Wallace (1954) stated:
There is present in the response to any mail questionnaire an
important number of persons who reply to most of the
mail questionnaires they receive. These persons have higher
education and, very probably, certain other distinctive charac-
teristics. (p. 51)
Berdie and Anderson (1974) reported that many researchers have
found that nonrespondents are usually the least educated members of the
sample (p. 52). Baur (1947-48) compared early to late respondents and
found that the less educated and the married respondents were underre-
presented in the early returns, and overrepresented in the last returns.
Hochstim and Athanasopoulos (1970) performed a study on mail ques-
tionnaire response bias by using personal interviews for follow-up on non-
respondents. They found a slight response bias (p. 74). Personal con-
tact to retrieve the questionnaires by personally interviewing the non-
respondents tended to improve estimates associated with low socioeconomic
studies, since this stratum was least likely to respond by mail. It
also introduced some not-at-home bias characteristic of personal inter-
view surveys. This caused the final sample to be slightly overweighted
with large families and women.
Hochstim and Athanasopoulos (1970) found the largest difference
between the mailed response and total response after personal contact
to nonresponses occurred in two categories:
1. Group activities -- people who reported no organizational
membership and those who indicated no political activity were
less likely than others to respond by mail.
2. Socioeconomic characteristics -- a similar tendency was
shown by people with low education, job security, and with-
out provision for medical care. (p. 77)
Hochstim and Anthanasopoulos (1970) made the following tentative
Surveys conducted entirely by mail can achieve a good repre-
sentation of the universe. When parameters are not known,
careful evaluation should be made of differences in response
to substantive data by various socioeconomic and other rele-
vant groups in the sample to see whether a bias in the total
return would exist if such groups were over -- or underre-
ported. (p. 81)
Other studies have suggested that response bias is not slight but
can be substantial. Weiss and Hatry (1971) reported that people with
extreme views are more apt to respond to questionnaires than people
with moderate views. They also reported that older and less educated
people as well as women respond less frequently to mail questionnaires.
These differences between respondents and nonrespondents would certainly
cause a response bias. Most researchers still regard the low response
rate of questionnaires as the major drawback of the mail questionnaire
in survey and follow-up research.
To avoid the problems of response bias, the Advertising Research
Foundation recommended an 80% or better response rate for mail question-
naire surveys (Erdos, 1974, p. 2-102). They stated that this would
bring the rate of nonresponse into line with the rate of substitution
in well-conducted personal interview studies (p. 2-102). To be confi-
dent of an unbaised representation, Kerlinger (1973) stated that "every
effort should be made to obtain returns of at least 80 to 90 percent or
more, and lacking such returns, to learn something of the characteristics
of the nonrespondents" (p. 414).
The SREB suggested a check for nonresponse bias when response is
not as high as desired. The SREB reported several ways in which
nonresponse bias may be determined without tremendous expenditure of re-
The most direct check is to select a limited random sample
among the non-respondents for telephone interviews in order
to obtain responses from some non-respondents and then to
determine whether their responses follow the same pattern as
from the respondents. Often in such telephone interviewing,
the length of call can be reduced by asking selected questions
instead of the entire series of the survey form. A statistical
test (such as the chi-square test) is then used to determine
whether there is a significant difference between the survey
responses from respondents and non-respondents. A "sequential
non-respondent sampling" telephone technique, that takes the
place of a statistical test, has been used in Maryland in con-
junction with a state-wide survey of community college students.
This sequential sample technique essentially sets an arbitrary
range of tolerance variation around the results obtained on
any one question from the respondents. Cumulative results
from the responses to a question from telephoned non-respon-
dents are computed until this response result stabilizes (per-
haps after 30-50 calls). If the stabilized cumulative response
result from the non-respondents falls within the variation
tolerance previously established, then the respondents' and
non-respondents' responses are considered essentially the
same. (p. 28)
Another way for checking for nonresponse bias was to use demographic
If the demographic characteristics for respondents and the
total population in the survey are essentially the same, then
the inference can be made that the responses are representa-
tive of the total population, at least on the basis of their
demographic characteristics. In order to check for demo-
graphic differences between respondents and the total popu-
lation, demographic averages on various characteristics are
computed for the respondents, and then compared to those
for the total population. A chi-square test is applied to
check whether the resulting differences are significant.
If the demographic characteristics of the respondents and the total
population differ, further checks should be made using substantive
questions compared by sex, race, grade point average, or other vari-
The last way to check for response bias suggested by the SREB was
to check for differences between early and late respondents.
The inference is sometimes made that since non-respondents
might have more in common with late respondents than with
early respondents, therefore, if the responses from late
respondents significantly differ from those given by early
respondents, this might indicate non-respondents' bias. In
order to use this method to check for non-response bias it
is necessary to number responses sequentially as they arrive.
Survey results might then be compared for those graduates
that were among the first one-third to reply. If responses
are not significant different, the assumption is made that
the non-respondents would also not be significantly differ-
ent. (pp. 30-31)
Baur (1947-48) performed a study using a questionnaire to survey
veterans on their plans for education and training after leaving the
service. To analyze the amount of response bias, his sample was
divided into five groups according to the number of days that elapsed
between the original mailing date and receipt of the questionnaire.
Baur found that the greatest bias was introduced by differences in
interest in the subject of the questionnaire. Those without definite
plans for education or training were underrepresented in the early
returns and overrepresented in the late returns. Therefore, he con-
cluded that the smaller the proportion of returns in a mail questicn-
naire on a subject of interest to only a part of the sample, the
greater the bias attributed to differences in the subject. Baur also
found that the less educated and the married respondents were under-
estimated in the early returns and overrepresented in the late returns.
Nielsen, Moos, and Lee (1978) performed an extensive mail question-
naire follow-up study on college students from their freshman to senior
year. They found significant differences between early and late re-
spondents on nearly a dozen variables. They stated that their results
indicated that researchers cannot account for follow-up nonresponse
bias by making statistical adjustments according to data available at
initial testing. They suggested more study needs to be done to develop
categories of nonresponders who may be motivated to cooperate by differ-
ent types of follow-up techniques.
Many researchers with low response rates preferred to make extra
efforts to have more questionnaires returned to increase the accuracy
of their study, but these extra efforts increased the costs of follow-
up studies (Weiss & Hatry, 1971, p. 23). Hochstim and Athanasopoulos
(1970) performed a mail questionnaire study in which the initially re-
turned questionnaire cost $6.55 per questionnaire. A personal inter-
view follow-up of nonrespondents to the mailed questionnaire increased
the rate of response, but the additional questionnaires filled out
during personal interviews cost $13.64 each.
Bower and Renkiewicz (1977) reported that extra efforts did in-
crease response rates in mail questionnaires and encouraged researchers
to use them. Researchers have investigated a wide range of variables
in trying to increase the response rate for mail questionnaires. Among
the variables were monetary incentive, first class versus third class
mailing, stamped versus business reply envelopes, advance letters, per-
sonal versus nonpersonal wording of the cover letter, timing of the
questionnaire and follow-up mailings, length of the questionnaire, re-
peated mailings, and intrinsic interest.
Monetary incentive. Hackler and Bourgette (1973) applied a theory of
dissonance to mail questionnaire response. They theorized that sending
a dollar to potential respondents along with a questionnaire would
cause the respondents to feel cheap if they kept the dollar and did
not return the questionnaire, therefore causing internal dissonance.
According to Hackler and Bourgette, the dollar was not a great enough
reward to justify the mildly improper behavior of not returning the
questionnaire. Therefore, to reduce the dissonance felt by the re-
spondents by not returning the questionnaire or the dollar, the respond-
ents would return the questionnaire.
Hackler and Bourgette (1973) sent one group a dollar with the
questionnaire and the other group received the questionnaire without
the dollar. They expected those who received the dollar would exper-
ience dissonance, and to reduce their dissonance, would return the
questionnaire. Their results confirmed their expectations. The dollar
group returned 71% of the questionnaires compared with 39% return with
the nondollar group. When the nondollar group was later sent a dollar,
their response increased.
Huck and Gleason (1974) also performed an experiment utilizing
monetary incentive to increase response rate. They used four differ-
ent groups. The first group received a quarter on the first mail out,
the second group received a quarter on the second mail out, the third
group received a quarter on the third mail out, and the fourth group
was the control group, receiving no quarter with three mail outs. Huck
and Gleason's results were that on the first mail out the group receiv-
ing the quarter had a return of 78% with the other three groups only
receiving a return of 50%. The difference was significant. After the
second mailing, with the second group receiving the quarter, both the
first and second groups were significantly higher in responses than the
third and fourth groups. The same was true for the third mail out.
Therefore, Huck and Gleason recommended sending a monetary incentive
with the second mail out to get the highest response for the least cost.
Wotruba (1966) performed a study to investigate the effects of two
types of monetary reward or inducements, and no inducement, on the per-
centage and completeness of response to a mail questionnaire. His re-
sults were that sending a monetary inducement with the questionnaire
produced a better return of the completed questionnaire. There was no
significant difference between the promise to send a monetary induce-
ment and no inducement at all. Wotruba concluded that the value of an
immediate inducement seemed to be stronger than the monetary value of
a delayed inducement, as long as the immediate inducement was not un-
reasonably small. He also concluded that when completeness was consid-
ered, the desired number of responses was obtained with least overall
cost using the immediate inducement. If completeness of the question-
naire was disregarded, the no inducement approach was least expensive.
Kephart and Bressler (1958) reported that the inclusion of a
quarter with a questionnaire could augment or even create another re-
sponse bias among the receivers. They suggested that before researchers
include a monetary inducement into their survey, they should consider
that it is quite possible that people who are influenced by a quarter
or by any other type of inducement may differ from those who are not
Wiseman (1973) performed a study involving four variables. His
results were that a 10t monetary incentive and a post card reminder
improved response rate by 8%. Wiseman also offered to return survey
results but found this had a slightly negative effect on response. The
overall results were that respondents given a 10 monetary incentive,
a post card reminder, and a stamped return envelope demonstrated the
First class versus third class mailing. Gullahorn and Gullahorn
(1963) noted that first class mailing of a questionnaire elicited a
somewhat higher return than third class mailing (p. 294). Warwick and
Lininger (1975) agreed that higher returns were noted when question-
naires were sent out first class mail rather than third class mail
(p. 131). The SREB supported this finding: "First class mail is more
personal and also has the advantage that the post office will forward
such mail to a forwarding address, which is not done for bulk rate
mailing" (p. 22). But, the SREB stated, bulk rates cost considerably
less than first class rates. The SREB made the following recommenda-
On a bulk rate mailing envelope, stamp the message, "Return
Postage Guaranteed." If the addressee is not at the address,
the letter will be returned to the sender with the reason for
nondelivery. The sender pays a 45 charge for this service.
Re-address the letter and send it out first class this time,
and now the post office will forward it to a forwarding
address, if the post office has one. (pp. 22-23)
Return envelopes. In regard to return envelopes, Gullahorn and
Gullahorn (1963) reported that more questionnaires with stamped return
envelopes were returned than with business reply envelopes. In Wise-
man's 1973 study, the stamped return envelopes produced a 13% increase
in response rate. Champion and Sear (1969) found that response rates
were greater when regular stamped envelopes and special delivery postage
were used. Selgas (1977), using cover letters, stamped addressed
return envelopes, and two mailings obtained a 70% response rate with
his mail questionnaire. Warwick and Lininger (1975) also reported that
stamped, addressed, return envelopes, rather than conventional business
reply envelopes, were one of the more effective devices to increase
response rate (p. 131).
Advance letter. McKinney and Oglesby (1971) suggested the use of
an advance letter to be mailed out before the questionnaire. This ad-
vance letter or "alert card" informed the student that an important
questionnaire was soon to arrive and their cooperation in filling it
out and returning it was vital.
Ford (1967) performed a study observing the effects of an advance
letter on response rates, quickness of return, completeness of the
responses, and the accuracy of the responses. His study demonstrated
a significant increase in the rate of return with the use of an advance
letter, but did not find a significant effect with the other three
factors. Erdos (1974) also reported that the use of an advance notice
sent prior to the questionnaire improved returns of the questionnaire
Parsons and Medford (1972) stated that "previous studies of ad-
vance notice in mail questionnaire surveys have shown significant im-
provement in response rates" (p. 258). They suggested that when the
sample is drawn from a fairly homogeneous population, the advance
notice does not always improve the response rate. They conducted two
experiments where one group of a homogeneous sample received advanced
notice of the upcoming questionnaire and the other group of the sample
received no advance notice. They found in both experiments there was
no significant increase in response with the advance notice group.
They suggested their findings indicated that advance notice was not
necessary when the sample was from a fairly homogeneous population.
Cover letter. An important part of most mail questionnaires was
the accompanying cover letter. Berdie and Anderson (1974) stated that
the cover letter should be short and to the point (p. 59). According
to the SREB (1978), the letter set the tone for the message from the
institution to the students. The president's signature on the cover
letter increased its importance in the view of the student. The cover
letter should also stress the confidentiality of the student's re-
sponse and emphasize the value of the individual student's evaluation
of the institution. Francel (1966), a professional survey conductor,
stated the most important feature of a cover letter should be its
friendly, personal tone. He also reported an action phrase, such as
"Won't you do it now while you're thinking about it" and stressing the
importance of the individual respondent's opinion would have a positive
effect on the response rate.
Various studies have found that a personally worded and typed
cover letter did elicit a higher response rate. Edwin Carpenter (1974-
75) did a study involving different levels of personalization. He
came to the conclusion that reducing personalization would reduce re-
sponse. If the response to the questionnaire was high, in tnis case
70% or better, then the difference in response rate would not be
substantial, but at lower levels of response rate, the difference would
Arnold Linsky's (1965) experiments consisted of four factors and
their effect on response rate. He found that personalization and empha-
sizing the importance of the respondent (the person who receives the
questionnaire) had a significant positive effect on the response rate.
Emphasizing social utility and an appeal to help the researchers had no
effect on the response rate. Champion and Sear (1969) confirmed Linsky's
results in that they found that response rate would be greater when the
respondent's importance to the study was emphasized.
Frazier and Bird (1958) tested the use of a handwritten postscript
for increasing response rate. The postscript read "P.S. We need your
help in this report. Could you please send it in promptly?" They found
a 27% increase in response with the addition of the handwritten postscript
Other researchers suggested that personalizing a cover letter does
not have as much effect on response as it has been given credit. Andreasen
(1970) stated that most researchers assume the effect of personalization
is positive on response rate. He reported that many cover letters are
careful to emphasize anonymity for respondents. A personalized cover
letter could negate the advantage of anonymity of a mail questionnaire.
Andreasen did a study using a questionnaire sent out to lottery winners
under the assumption that they would prefer anonymity because of the
harassment from sales agents. He varied three levels of personalized
cover letters with the questionnaire. His findings were inconclusive,
though the most personalized, handwritten form of correspondence was
somewhat less effective in generating response. He found it took a
significant dose of personalization to secure a moderately significant
effect on response rate. Therefore, he concluded that the costs of per-
sonalization usually are not justified by their benefits and adds that
in some studies, and for some respondents, the expenditures for person-
alization may have the opposite effect than the one desired.
Timing. Timing the questionnaire arrival could also have an ef-
fect on the response rate. The SREB (1978) reported that a questionnaire
timed to reach students around the middle of November had the advantage
of perhaps finding them at their parents' homes, which might correspond
to the permanent address of the student on college records. "Some insti-
tutions have found the response rate increased when a mailing was timed
for the Thanksgiving holiday" (p. 19).
Reminder postcards have frequently been employed to increase re-
sponse rate. Nichols and Meyer (1966) performed a study with various
time schedules for sending out the reminder postcards. These postcards
included a "thank you" to those who had promptly returned the question-
naire and a request to those who had yet to return the questionnaire
to do so. Nichols and Meyer found that a thank you and reminder post-
card, sent three days after the questionnaire mailing was the most ef-
fective time for increased response. The best overall response came
from using both the three day postcard reminder and a later (45 days)
second mailing of the questionnaire.
Length of the questionnaire. The general opinion on the length
of the questionnaire has been the shorter, the better (Erdos, 1974;
Filion, 1975; Parten, 1950; Robin, 1965). This assumption was based on
the theory that respondents would be more willing to fill out a
questionnaire that would take less of their time than a longer, more
time consuming one. According to Erdos (1974) a questionnaire that be-
came long, six pages or more, should be administered by a well trained
interviewer who can maintain the respondent's interest (p. 2-92). He
further stated that when the mail questionnaire becomes too long, the
percentage of response falls off sharply.
Sletto's (1940) findings suggested the length of the questionnaire
was not an important variable in questionnaire response rate. Clausen
and Ford (1947) further challenged the widespread belief that a shorter
questionnaire would elicit a higher response rate. They showed that a
multi-subject questionnaire obtained a larger response in its initial
mailing than a shorter, single subject questionnaire. They also re-
ported that one or two pages of supplementary questions did not affect
responses to a three to six page questionnaire.
Champion and Sear (1969) varied the length of a questionnaire in
their follow-up study after they noted that other researchers reported
that generally shorter questionnaires were returned more frequently.
They performed a study utilizing questionnaires of three different
lengths. Their results were that the longer questionnaires were returned
most frequently, both in the initial and the second mailing. These
results differed considerably from accepted beliefs about mail ques-
tionnaires. The SREB stated that there is a consensus among experts
on surveying, that shorter questionnaires yield higher response rates
than longer questionnaires (p. 12). The Champion and Sear (1969)
results and the Clausen and Ford (1947) results indicated that the
issue of questionnaire length was more complex than first realized and
deserves further study.
Dillman (1972) took his questionnaire and had it and the cover
letter photographically reduced and put in booklet form to make the
questionnaire appear smaller. He performed two studies, one in 1970
and one in 1971, and obtained a response rate of 75% for each study
(p. 255). The appearance and form of the questionnaire should also be
considered for size and respondent's interpretation of the size,
according to Dillman (1972). Erdos (1974) stated that the layout of
a questionnaire should be designed to avoid a crowded appearance (p. 2-
Repeated mailing. The SREB (1978) reported that all institutions
have found that repeated mailings of questionnaires did improve re-
The added cost of extra mailing, however, must be weighed
against the improvement in response rates. A minimum of
two mailings is universally recommended by those with ex-
perience. Responses continue to rise after two mailings,
but at diminishing rates. (SREB, 1978, pp. 19-20)
Some studies use a reminder postcard as the second mail out, but the
SREB suggested that each mailing include a questionnaire. A suggested
time limit for returning the questionnaire should also be included, with
ten days to two weeks being suggested by the SREB. Selgas (1977) in his
follow-up study, included two mailings of the questionnaire with a
cover letter and a stamped addressed return envelope. He separated the
mailings by one month and obtained a 70% response rate to deliverable
questionnaires. Cox (1966) reported that previous recommendations were
to schedule one month for a two wave mailing. But, if projected time
was critical, then a two wave mailing could be conducted in a two week
period without too much loss of response since a majority of the returns
arrive soon after the mailing.
McKinney and Oglesby (1971) provided a list of recommendations for
performing follow-up studies which stressed repeated mailings. Once the
reason for the follow-up questionnaire was determined, the population to
be studied should be chosen. The next most important part of any follow-
up study was selecting the information gathering instrument. A useful
procedure for developing a follow-up questionnaire can be found in McKin-
ney and Oglesby's 1971 article, as well as in the Follow-up Surveys of
College Graduates by the SREB (1978). McKinney and Oglesby list recom-
mended steps for performing a follow-up study. They started with what
they call an "alert card," which was an advance letter informing the stu-
dent of the upcoming questionnaire and its importance. The student was
urged to complete and return the questionnaire. Following is the order
suggested by McKinney and Oglesby for performing a follow-up study using
a mail questionnaire. They suggested the steps be performed at one week
First mailing "Alert card."
Second mailing Follow-up instrument, cover letter,
and return evelope, stamped and ad-
Third mailing First thank you reminder card.
Fourth mailing Second request follow-up instrument
(second cover letter, and return en-
velope stamped and addressed).
Fifth mailing Second thank you reminder card. (p. 16)
Interest. Warwick and Lininger (1975) reported from their own obser-
vations that the greater the intrinsic interest of the subject matter to
the respondent's personal interest, the better the chances of a high re-
sponse rate (p. 132). Berdie and Anderson (1974) reported that offering
the respondent the results of the study can sometimes stimulate interest
and response (pp. 58-59).
Baur (1947-48) reported that interest in the subject of the ques-
tionnaire can increase response bias. He found that when comparing
early to late respondents, those interested in the subject returned
their questionnaires early while those not as interested returned theirs
Monetary incentive, first class versus third class mailing, stamped
versus business reply envelopes, advance letter, personal versus non-
personal wording of the cover letter, timing of questionnaire and fol-
low-up mailings, length of the questionnaire, repeated mailings, and
intrinsic interest are all different tactics for increasing response
rate. Berdie and Anderson suggested the use of varying tactics for dif-
ferent samples and studies to maximize response rates (1974, p. 53).
Rossmannand Astin (1974) performed a study where they compared the
different tactics and techniques for increasing response rate to the
costs of performing those techniques. In their study of cost effective-
ness, they investigated the cost in comparison to the response rate of
first class versus non-profit outgoing postage, business reply versus
live stamp return, matched insertion in envelopes versus window envelopes,
auto typed letters versus personalized letters, one mailing versus two
mailings (with and without a reminder postcard), and types of non-profit
The results of theRossmann and Astin study (1974) generally held
that the least expensive methods of performing a mail questionnaire
follow-up were the most cost effective. In comparing first class post-
age to non-profit postage, the results showed that using first class
postage caused an increase in response rate that varied from 0.2% to 3%.
The cost per response, however, favored non-profit postage with the sav-
ings varying from 8 to 18U. In the comparison of business reply versus
live stamp return, they found that live stamp increased response rate by
no more than 2%, but costs were considerably more, varying from 144 to
234 higher per response.
Their matched insertion versus window envelopes comparison found the
additional cost of matched insertion was approximately $15.00 per thou-
sand questionnaires. The results favored the window envelope with re-
sponse rate increasing slightly for window envelopes and cost per response
between the two techniques being similar.
The use of auto typed letters, with a personally addressed question-
naire included in the second wave,resulted in the highest rate of return,
but also had the highest cost. The 3% increased response rate hardly
seemed to justify the increased cost of 59t per response, according to
Rossmann and Astin (1974).
A postcard reminder, but no second mailing of the questionnaire de-
creased costs, but the response rate decreased 10% below the use of a
second mailing of the questionnaire. The 10% difference in response rate
was deemed, by Rossmann and Astin (1974), worth the extra cost for the
second mailing of the questionnaire. No postcard reminder with no second
mailing of the questionnaire had the lowest cost, but also the lowest re-
sponse rate. It was observed that a postcard reminder between the first
and second mailing made little difference in the results. Therefore, a
second mailing of the questionnaire, without a postcard reminder, seemed
to provide the maximum benefit to response rate without unnecessary addi-
tions to costs.
Rossmann and Astin (1974) also used three types of nonprofit per-
mits: printed, precanceled stamps, and meter. The data showed the
difference among the three methods was small, with a slightly higher
response rate produced with precanceled stamps.
The overall results of the Rossmann and Astin (1974) study were
that the use of nonprofit outgoing postage, window envelopes, and
business reply return postage resulted in considerably lower costs per
contact and per respondent, and did not lead to significant lower re-
sponse rates. The use of a second mailing of the questionnaire proved
to be an effective technique for increasing the response rate while
the use of reminder postcards and various types of nonprofit postage
had little impact.
Length of Time
Dillman (1978) reported that mail questionnaires are usually
locked into a definite time schedule of printing, folding, stuffing,
and mailing dates; and the different methods for increasing response
rate can affect the length of time involved in a study (p. 67). Exact
time limits are not available. Parten (1950) reported that mailing
costs are low compared with the transportation, as well as time costs,
associated with studies which require a field staff. The expensive
and time consuming task of training a staff of investigators is absent
with the mail questionnaire (p. 94). Time involved in performing a
follow-up study is associated with costs since the longer a staff mem-
ber is on the payroll, the higher the costs. The lower costs of mail
questionnaires could be associated with less time involved needed to
perform a follow-up study.
Jackson and Rothney (1961) stated the economic advantage of the
mail questionnaire was in saving cost and time (p. 571). Kephart and
Bressler (1958) stated the reason the mail questionnaire has come to
be favored by social scientists is for the expediency of the study
(p. 123). According to Kephart and Bressler, information can be pro-
duced more quickly by the mail questionnaire than by interview methods
Berdie and Anderson (1974) stated the use of mail questionnaires
could facilitate collecting data from an extremely large sample in a
short period of time (pp. 17-18). However, they continued, "with small
sample sizes, questionnaire administration may be more time consuming
than interviewing each subject" (p. 18).
The length of time for the development of a mail questionnaire
does vary. The development of an instrument and planning the follow-
up study depends on the situation and people involved, and will not be
dealt with here. The time involved with the actual performance of the
study will be considered the length of time of the follow-up study.
The start of a mail questionnaire follow-up study shall be when
the first mail out of the questionnaire takes place. According to the
National Opinion Research Center, six weeks to three months are re-
quired from the start of a follow-up study until the data are ready for
final form. Two to three weeks are required before the last completed
questionnaire is received, and from one week to a month is needed to
tabulate figures and compute percentages (Parten, 1950, p. 61).
Nichols and Meyer (1966) studied the effect of a third day post-
card reminder and a 16th day postcard reminder on response rate.
By the 16th day, 58% of the group receiving the third day postcard re-
minder had returned their questionnaire, but only 37% of the group not
receiving the third day postcard had returned their questionnaire.
After 45 days, there was no difference between the two groups. The re-
sults suggested that a third day postcard elicited the highest rate of
return in the shortest period of time.
Dillman (1972) included a one week postcard, a three week letter,
and a seven week mailing of a second questionnaire in his follow-up
study. The response time after this final mailing would extend the
length of this study to approximately ten weeks, after which would
come the evaluation. Dillman (1978) listed eight weeks as the time
needed to conduct the implementation of his total design method of
conducting mail questionnaires (p. 67).
According to some authors, the time involved with the actual
questionnaire distribution and collection can be shortened. Stanton
(1939) noted that 90% of the total possible returns to a mail ques-
tionnaire would be returned within three weeks following the first
mailing of the questionnaire. Francel (1966) reported that it is well
known that in mail questionnaires, the majority of replies tend to
come in during the first week after mailing. The remaining come in
at a rapidly diminishing rate over the succeeding three or four weeks
Boyd and Westfall (1964) reported that a minimum of two weeks was
usually necessary before the bulk of the returns from one mailing were
in. Scott (1961) reported that most researchers seem to assume that
the best policy was to wait until returns have almost stopped before
sending the second follow-up (p. 166).
Cox (1966) performed a follow-up study to investigate response
patterns with mail questionnaires. He reported that the approach of
waiting for the returns to slack off could cause unnecessary rejection
of mail surveys as a research method when time was very short (p. 392).
He reported that previous studies of response patterns to mail ques-
tionnaires have resulted in recommendations that one month be used for
a two wave questionnaire (p. 396).
Clausen and Ford (1940) reported that mailing intervals may be cut
down by perhaps one half. A second mailing can be made before the
first returns have finished coming in. The drawback is that some ques-
tionnaires will go needlessly to those who are in the process of re-
The data from Robinson and Agisim's study (1951) showed that re-
sponse is low during the first six days, but there is an increase in
cumulative returns on the seventh day. This seems to be a standard
response pattern. According to Cox (1966), these data suggest that it
would be unwise to employ a mailing interval of less than seven days.
Cumulative returns in the Robinson and Agisim study exceeded 90% on
the 14th day. Cox (1966) concluded that the optimum mailing interval
therefore should be between the 7th and 14th day after the previous
Gray (1957) found that response patterns for the second mailing
of a questionnaire are almost identical to the pattern of the first
mailing. Therefore, according to Cox (1966), the same mailing inter-
val can be used for all mailings. Cox went further to point out that
if there were no significant differences between the replies to the
first and second mailings, there would be no need to mail out the
questionnaire a third time (p. 395). He stated that the total time re-
quired to conduct a two wave questionnaire in the event of a short time
frame, would be approximately two weeks.
Erdos (1974) supported the two weeks time period. He stated "by
sending out a single mailing by air, a nationwide survey of a large
sample distribution all over the United States can be completed in two
weeks" (p. 2-91).
Mooren and Rothney (1956) performed an experiment to observe if
personalized forms of mail questionnaires would produce faster responses
than general nonpersonal forms. They found no significant difference
among the personal and general groups of their experiment in terms of
speed of response (p. 412).
In spite of researchers' attempts to decrease the length of time
required to perform a mail survey, the time still depends on various
factors, such as size of the study, the response rate desired, and the
pattern of response. Researchers have condensed the time to a minimum
of two weeks for a two wave questionnaire (Cox, 1966).
The personnel involved with a follow-up study depends on the size
of the study and the resources available. Many educational institu-
tions use their everyday staff to perform a follow-up study. Basically,
a mail questionnaire requires clerical help in the areas of typing,
printing, and putting the follow-up instrument in the mail envelope.
When the questionnaires are mailed back, they must be organized and
tabulated, using clerical help or keypunchers. A programmer may be
needed to program the statistical analysis before the results are ready
to be evaluated and reported.
Parten (1950) reported that mail questionnaire studies are less
expensive than other methods of follow-up because there is no large
staff to train. More staff may be needed if there are plans to collect
data from the people who do not answer the questionnaire.
Dillman's (1972) follow-up studies consisted of mailing out 4,500
questionnaires each. He used 1,215 labor hours for each study to per-
form the data collection process. In this instance, Dillman used a
substantial amount of labor but was performing a follow-up study on a
relatively large sample.
Franzen and Lazarsfeld (1945) reported one of the advantages of the
mail questionnaire was that it required few personnel. They stated
"the mail can carry a questionnaire into any territory and at the same
time can make the collecting of information a one-man job" (p. 293).
Benson (1946) also reported that widely scattered geographic coverage
can be obtained more easily and less expensively through mail question-
naires than through personal interviews. Erdos (1974) stated that mail
questionnaires can be conducted from one office with built in checks
and controls for every phase of the operation (p. 2-91). Dillman (1978)
reported that the personnel requirements for the mail questionnaire
were far less than those required for either personal interviews or
telephone surveys with equal sample size, and the skills required were
mostly clerical (p. 67).
As Franzen and Lazarsfeld stated, a mail questionnaire can be per-
formed by one person, even though most studies do employ some clerical
help. The mail questionnaire can get by on a small staff or a large
staff, depending on the size of the sample, the budget, and the labor
Krueck (1975) disagreed with the prevalent opinion that question-
naires required fewer human resources and manhours than personal con-
tact methods. He found in his follow-up methods comparison study, that
12.5 clerical manhours were required for labeling and stuffing envelopes,
checking returned questionnaires, and assimilating information obtained
through the use of telephone reminder calls (p. 22). He stated that
his finding "should dispel any belief that fewer human resources are re-
quired in a mailed questionnaire unless one considers repeated mailings"
Quality of Data
The quality of the data had two main aspects in this study. First
to be considered were the results of the study. Were the results based
on true and accurate data? Second to be considered were the data. Were
the data from which the results are taken complete? For the results of
any follow-up study to be useful, they must be accurate, and the data
must be complete. Otherwise, any conclusions drawn from the results will
not be dependable.
Accuracy of information which is returned on a questionnaire can be
difficult to verify. Parten (1950) supported questionnaire information
because she believed the respondent may answer questions more frankly
by mail since anonymity is frequently used. She also stated that since
the questionnaires were standardized, that eliminated possible personal
interviewer suggestiveness or bias (p. 94). On the negative side, Parten
(1950) suggested that since the respondent completes the questionnaire
without the aid of a personal interviewer, misinterpretation of ques-
tions is possible (p. 95).
Franzen and Lazarsfeld (1945) reported an item in favor of mail
questionnaires in the area of accuracy. They compared the mail ques-
tionnaire'to the personal interview on the basis of respondents educa-
tional level. They found that apparently, when not faced by an inter-
viewer, incomplete college and high school records were more willingly
divulged (p. 311). From this finding, Franzen and Lazarsfeld concluded
the mail questionnaire elicited more accurate responses in the area of
reporting educational level. Benson (1946) agreed that mail question-
naires elicited more accurate data. He stated that it is conceivable
that on certain subjects answers might be more reliable than those
that could be obtained by an interview. Hochstim (1967) also concluded
that respondents were more honest in mail questionnaires than the
other methods of gathering data. Knudson, Pope, and Irish (1967)
stated that research comparing questionnaires to interviews were rare,
though a few studies have indicated that the questionnaire may be the
more successful in gaining valid information.
Although the above researchers have contended that the question-
naire elicits accurate data, other researchers emphasized that it is
difficult to check the accuracy of the responses on a mail question-
naire. Parten (1950) stated, "checks on the honesty and reliability
of returns are difficult to devise when the personal interviewer does
not see and size up the informant" (p. 96).
Obtaining completed questionnaires has been another problem with
mail questionnaires. Hackler and Bourgette (1973) reported that mail
questionnaires are usually more appropriate for collecting sensitive
information (p. 276). But many researchers also felt that personal
interviews, rather than mail questionnaires, elicited more complete
answers. Jackson and Rothney (1961) performed a study utilizing both
mail questionnaires and interviews. They concluded "the interviews
elicited significantly more complete answers than the mail question-
naires" (p. 571). They also concluded the interview method drew forth
more extensive responses to questions than the mail questionnaire.
Krueck (1975) explored the differences in the nature of responses
elicited by each method of conducting a follow-up study. He found
questions concerning satisfaction with instruction, counseling, and open-
ended questions had a much higher frequency of nonresponse on a mail
questionnaire. He also reported that the mail questionnaire elicited
a greater number of negative responses when compared to the personal
forms of data collection (p. 19).
Wotruba (1966) found that sending a monetary inducement with the
questionnaire produced not only a better return, but also produced more
completed questions. He concluded that when completeness of the ques-
tionnaire information was vital, immediate monetary inducement was bene-
Mooren and Rothney (1956) performed a study to discover whether per-
sonalization would produce more complete responses. They found no signi-
ficant difference between personalized and general forms of questicn-
naires in terms of quantity of response (p. 412). Parten (1950)
summed up the advantages of the mail questionnaire as follows:
1. If mailed questionnaires are used, it is possible to cover
a wider geographical area and to reach a much larger popula-
tion with given funds than could be accomplished by personal
interviews with each informant. This lower cost applies pri-
marily if personal follow-ups are not made.
2. Mailing costs are relatively low compared with the trans-
portation and costs for a field staff.
3. The expensive and time-consuming task of training a staff
of investigators is eliminated. This assumes, of course, that
a large staff will not be needed to collect the schedule data
from the people who do not answer the questionnaire.
4. The informant may answer questions more frankly by mail
since anonymity is assured. On the other hand, some respond-
dents may hesitate to put their ideas in writing for fear
that their schedules may be identified even though unsigned.
Actually, it is questionable whether anonymity is either an
advantage or a disadvantage.
5. The questionnaire may reach groups who are more or less
protected from solicitors and investigators. In high-rent
apartment houses or private homes where servants protect the
occupants from solicitors and other doorbell ringers, for
example, it is often difficult for investigators to gain ad-
mittance. Of course, the mere fact that the mail is received
does not guarantee that it will not be filed in the waste-
basket by a secretary or even by the addressee as soon as he
glances at the heading.
6. Personal antagonism to investigators which may lead to a
refusal to give the desired information is avoided.
7. If time is not an important consideration and if the
sample extends over a wide area, the cost of securing practi-
cally complete returns is probably lower than in the personal
8. The questions are standardized, whereas in the personal
interview the investigator may alter them or suggest answers.
9. The questionnaire can be answered at the convenience of
the respondent. This gives him time to deliberate on each
point, and if necessary, to look up information needed to fill
in the items. However, he may consult other members of his
household, so his reply may be more representative of the
Family's point of view than of his own.
10. It is claimed that the mail questionnaire brings many
more returns from the man of the house than does the
telephone or personal interview method.
11. Where the persons to be reached are located in
widely scattered areas of cities and are a mobil ele-
ment of the population, it may be easier to locate
them by mail (registered or special delivery) than by
other methods. (pp. 94-95)
Parten (1950) also listed the disadvantages of the mail out question-
1. The people who return questionnaires are not repre-
sentative of the groups to whom the schedules are sent.
This limitation is sufficiently great to outweigh almost
all the advantages listed above.
2. The returns from mailed questionnaires sent to the
general public are usually very low, often ranging from
about 10 to 20 percent. The percentage of returns varies
greatly, however, with different schedules and informants.
One survey of M.D.'s in New York State received about 50
percent returns without follow-ups. By continued effort
the returns may be increased considerably over the first
responses. The techniques for increasing mail returns
are dealt with in Chapter XI (in the Parten Book).
3. Since the informant fills in the data on the question-
naire without the assistance of an investigator, he may
misinterpret questions, omit essential items, or send in
material which cannot be put in form for tabulation, thus
making it necessary to discard many of the questionnaires.
4. The questions used must be simple and practically
self-explanatory, since no training can be given the infor-
mant on their meaning and on how to fill out the schedule.
5. In most studies, the questionnaire must be relatively
brief if high returns are to be obtained.
6. If the sample is to be unbiased, it is necessary to
supplement the mailed returns with information obtained by
personal interviews with the nonrespondents.
7. Checks on the honesty and reliability of returns are
difficult to devise when the personal interviewer does not
see and size up the informant.
8. it is practically impossible to return unsatisfactory
or incomplete schedules to the informant for correction.
9. Because most people would rather talk than write,
questionnaires must be made very interesting to induce
10. An up-to-date address list of potential survey in-
formants is difficult to find.
11. Mail returns from the last third of the respondents
come in slowly; hence the mail survey must be spread over
a relatively long period, if a high percentage of returns
is to be secured.
12. Many questions which might antagonize the respondent
cannot be included on the mail questionnaires but can be
asked in personal interviews when the informant gradually
can be led around to the subject. (pp. 95-96)
Telephone surveys consist of interviews taking place over the tele-
phone rather than in person. Dillman (1978) reported that research litera-
ture on telephone surveys was as thin as that for the mail questionnaire
was thick (p. 9). The telephone survey was used in the Literary Digest
debacle of 1936. The Digest pollsters predicted a landslide victory for
Landon over Roosevelt in the presidential race. Using the telephone
caused a clear social class bias since only 35% of households in the
United States had telephones (p. 9). Therefore, the poll predicted in-
correctly and the use of the telephone as a survey method was generally
used only as a complement to other survey methods (p. 9).
As the percentage of households having telephones increased, so
did the feasibility of the telephone survey. Dillman (1978) reported
that the use of the telephone survey had mushroomed in the 1970's
(p. 10). Payne (1974) reported that interviews by telephone were satis-
factory, efficient, and economical for a wide range of survey purposes
(p. 2-105). Warwick and Lininger (1975) agreed. They stated that
researchers conducting follow-up studies have turned increasingly to the
telephone survey (p. 130). They reported the telephone survey had many
of the advantages of the personal interview, yet it was considerably
less expensive. It overcame the problem of mounting refusal rates in
some areas where strangers coming into the hume were unacceptable.
Some respondents who would not allow an interviewer in their home were
perfectly willing to talk with the same individual on the telephone
Warwick and Lininger (1975) also reported the limitations they
associated with the telephone survey. In their opinion, the greatest
drawback was the difficulty of establishing the same rapport that was
possible in a personal interview situation. With the telephone sur-
vey, there was a lack of communication through facial expression, ap-
pearance and sheer physical presence. Other limitations reported by
Warwick and Lininger included the respondent's ability to hang up.
As a result, telephone surveys were usually shorter and less demanding
than personal interviews. The asking of sensitive questions was also
limited because it increased the risk that the respondent would hang
In Payne's (1974) opinion, the main drawback of the telephone
survey, at least historically, was the fact that not everyone had a
telephone (p. 2-105). With well over 95% of all households obtaining
telephones, the chance of response bias due to lower income groups
not possessing a telephone was reduced (Dillman, 1978, p. 9). But
those in higher income groups who decided to have unlisted telephone
numbers could again induce response bias if the telephone directory
was the information pool from which the sample was drawn (Payne, 1974,
p. 2-107). Also the problem existed where higher income homes had
more telephones, and possibly a better than average chance of being in-
cluded in the survey. An indirect sampling technique which employed
the local telephone exchange and random numbers for the last four
digits could be utilized (Payne, 1974, p. 2-119). For further suggest-
ions on sampling techniques to be used in telephone surveys, see
Payne, "Data Collection Techniques: Telephone Surveys," in the Hand-
book of Marketing Research (Payne, 1974). For the purpose of this
paper, it will be assumed that sampling is not a problem, that the re-
searcher has access to the telephone numbers needed to perform a study.
McKinney and Oglesby (1971) also reported on the effectiveness of
sampling when used for a telephone survey. They stated that while the
telephone interview is- limited to the amount of information which can
be obtained, it may be the only workable procedure for some segments
of the population (p. 9).
The telephone survey technique will be reviewed using the five
variables previously chosen to compare and evaluate the various follow-
up methods. The five variables are: costs, response rate, length of
time, personnel required, and quality of data.
Denton et al., (1977) reported that a telephone survey required
a large monetary investment to locate and actually contact potential
respondents by phone. Comparing the telephone survey to the personal
interview, Parten (1950) reported that the telephone survey could pro-
vide a tremendous savings in transportation costs (p. 86). Weiss and
Hatry (1971) agreed that by utilizing the telephone survey, travel was
eliminated and costs reduced (p. 23). They estimated that for 500
telephone interviews, the average cost per interview would be $17.00
(see Table 5). They included detailed costs for preparation. Consider-
ing only actual interviewing costs, the costs for the study were $9.33
Thomas Krueck (1975) performed a comparison of three methods of
conducting a follow-up study. He reported that as conducted in his
study, the telephone survey required no immediate out-of-pocket outlay
of funds (p. 22). The costs were involved with the manpower needed to
make the telephone calls. He employed research assistants and paid
them $5.00 an hour. He calculated that 55 hours and 12 minutes were
devoted to telephone calling, producing a total cost of $276.00. The
response to the telephone survey was 108 out of 164 possible responses,
or a 66% response rate. Therefore, the cost per response was approx-
Payne (1974) agreed that an advantage of the telephone survey
over the personal interview was in the area of cost savings. He stated:
Generally, the shorter the actual interview time relative
to other time factors, the greater the economy of the
telephone method seldom do the data collection
costs of the telephone survey exceed two-thirds of the com-
parable costs of the personal survey. (p. 2-110)
Payne continued on to state that differences in costs between
telephone and mail surveys depend on the response rates.
Depending primarily on mail response rates, one or the other
may prove more economical, since the mailing cost per return
for a 10 percent mail response is essentially five times
that for a 50 percent mail response. (p. 2-110)
Cost Estimates For Telephone Interviews
For 500 tele-
1. Overall planning and direction
2. Preparing listing $ 500
3. Selecting sample 300
4. Developing interview questionnaire (including
conferences, 20 pretest personal interviews,
10 pretest telephone interviews) 600
5. Reproducing interview questionnaires, training
materials, manuals, supplies 650
6. Interviewing costs
a. Recruitment 200
b. Training (20 interviewers for 15 hours,
plus salary of trainer) 1,000
c. Interviewing time 1,900
d. Supervision 800
e. Supplementary in-person interviews for
50 persons without telephones or not answering 1,000
7. Editing, coding, punching, tabulating 800
8. Telephoning 500
9. Mailing 10
10. Analyzing data and report writing 150
11. Reproducing report 100
Average cost per interview $17.00
Note. From An Introduction to Sample Surveys for Government Man-
agers, by C. H. Weiss & H. P. Hatry. Washington, D. C.: Urban Insti-
tute, 1977, p. 47.
According to Parten (1950) the response rate on the telephone sur-
vey depended on whether or not recalls were made on "no answers" and
"busy signals" (p. 89). She reported that only about 1 1/2% refused
the calls, and about 1/3 were incomplete due to no answers or busy
signals. Therefore, approximately 2/3 of the calls made resulted in
completed interviews on the first call.
The response rate to a telephone survey was believed to depend on
the length of the survey, but studies listed by Dillman (1978, p. 55)
found only 2% to 4% of the respondents refused to complete a telephone
survey lasting from 20 to 30 minutes. Dillman (1978) reported that it
was not known how long interviews could be before cut offs become a
major problem, but the evidence clearly suggested that once people
were on the telephone, the length of the interview did not appear to
affect the response rate (p. 55).
Michael Quantz (1977) performed a telephone survey of community
college career students who graduated between 1973 and 1976. Telephone
interviewers, people actually in the program the respondents had gradu-
ated from, attempted 228 contacts. If the graduate was not available,
the desired information was obtained from a relative or co-worker. The
interviewers completed 191 of 228 possible contacts. The response rate
for this study was 84%.
Krueck's 1975 study performed a comparison of the three methods
of conducting a follow-up study. He reported that the telephone survey
methods had a higher response rate than the personal interview or the
mail questionnaire. The response rate in his study was approximately
66%. Following were the reasons listed by Krueck for any lack of re-
sponse to the telephone survey: 11 were either a wrong number or a
disconnected telephone, 19 had moved out of the local telephone dis-
trict, 14 could not be followed-up, and 1 refused to respond (p. 17).
Warwick and Lininger (1975) reported that perhaps the greatest
success with the telephone survey was seen when it was used as a
follow-up to a personal interview. A study using a personal interview
with multiple follow-up using a telephone call: obtained a response
rate of 97.5% (p. 131).
Length of Time
Parten (1950) stated that the telephone survey brought about a
savings in time as well as in costs, and it could be used when a study
needed to be conducted quickly and inexpensively. Parten also noted a
large portion of a population of a community could be covered in a
short time due to the fact a few telephone callers could make several
hundred calls per day (p. 86).
In Krueck's (1975) study a total of 303 phone calls were made to
obtain the response of 108 former students. The actual interviews
taken over the telephone required approximately 15 minutes each for a
total of 27 hours. The preliminary calls averaged 4 minutes each for
a total of 13 hours. Therefore, the actual number of minutes required
to obtain each response was 22 minutes (p. 23).
Payne (1974) stressed the need for interesting questions on inter-
esting subjects in telephone surveys (p. 2-108). It was much easier to
hang up when bored, than to ask an interviewer to leave. Although it
was once thought that telephone interviews must be brief, telephone
interviews have gone on for as long as 25 to 30 minutes (Payne, 1975,
p. 2-108). Payne gave the warning that, "a careful look might better
be taken at the questionnaire to judge whether its inordinate length is
justified" (p. 2-108).
The time involved to perform a follow-up telephone survey varied
not only according to the length of the interview, but with the number
of people placing the calls, and the number of recalls made. Parten
(1950) reported that the telephone survey was still the quickest of
the three main follow-up research techniques (p. 91). Parten also re-
ported that if the interview itself was brief, then an interviewer
could complete about 30 calls per hour (p. 91).
Payne (1974) reported that more interviews could be done by tele-
phone than in person within a limited time period, and more hours of
the day could be productive (p. 2-110). Dillman (1978) stated that as
a general rule, "telephone surveys are the fastest to complete" (p. 67).
Denton et al., (1977) alternately suggested that the telephone
survey could involve a great deal of time if it was necessary to locate
the potential respondents' phone numbers. In a situation which would
require substantial amounts of time to locate respondents and gather
telephone numbers, then the telephone survey would not necessarily be
the quickest method of a follow-up study.
The central people involved in a telephone survey are the people
who make the telephone calls. Almost all the other factors, such as
cost, time involved, and even response rate, depended on the people
making the telephone calls. According to Parten (1950, p. 90), if good
interviewers were employed, the number of refusals would be low. She also
reported that it was easy to train and supervise interviewers since they
could work all in one room directly beside the supervisor (p. 91).
In Quantz's (1977) study, the telephone interviewers were faculty
members from the different program areas. They contacted the student
who had graduated from their particular program. No special training
Dillman (1978) reported that personnel requirements for telephone
surveys were less than for personal interviews, but more than for mail
questionnaires (p. 66). Close supervision meant the skills and train-
ing needed could be developed as on the job training. In college and
university settings, students with flexible schedules and training in
research methodology could be used.
The size of the sample or population determines the number of
telephone interviewers required. Other people involved with the follow-
up study, such as statistical and keypunch operators, varied according
to the size of the sample, not the method of follow-up employed. With
the telephone survey, one or two telephone interviewers could investi-
gate a large portion of the population in a short time because several
hundred calls could be completed per day (Parten, 1950, p. 86).
Dillman (1978) reported that with a telephone survey:
Relatively few interviewers can conduct a large number of
interviews in a short period of time, in sharp contrast
to the rate of two or three a day often expected for face
to face interviews. (p. 66)
Quality of Data
To discover the quality of data, two considerations must be taken
into account; the accuracy and truth of the data, and the completeness
of the data.
Already mentioned was the belief that a telephone survey must be
short, nondemanding and void of sensitive questions to get respondents
to respond with accurate data (Warwick and Lininger, 1975). This limited
the type of data that could be collected and therefore could reduce the
ability of the researchers to accurately ascertain information on certain
Payne (1974) disagreed. He reported that respondents may reply with
greater candor to personal questions asked on the telephone than to the
same questions asked in person (pp. 2-122).
A major problem connected with the telephone survey for collecting
accurate and true data was improper sampling. Weiss and Hatry (1971)
reported that since not everyone had a telephone, there was sampling
bias with studies utilizing telephones. Lower income people have fewer
telephones and some people have unlisted telephone numbers. According
to Weiss and Hatry (1971), a recent investigation showed that people
with unlisted telephone numbers were primarily middle income and dispro-
portionately black (pp. 23-24). Therefore, researchers employing the
telephone survey should consider the intended respondents, to ascertain
whether or not a telephone survey will produce sampling bias.
Another problem with telephone surveys, according to Warwick and
Lininger (1975), was that telephone surveys were sometimes used for mag-
azines and other lead-in sales presentations as well as other types of
phone solicitation (p. 131). Parten (1950) stated that if the respondent
was annoyed by the phone call, he or she may deliberately give misinfor-
mation (p. 87). Without: the interviewer actually facing the respondent
to observe his or her reactions, the interviewer must accept the answers
without question (Parten, 1950, p. 87).
According to Parten (1950), other limitations that existed when
gathering data using a telephone survey was the lack of privacy on party
lines, housewives answered the phone more often than other members of
the family, and misinformation was hard to check in short telephone calls
One problem Payne (1974) reported was that open-ended questions might
not be answered in as great detail over the telephone as in person
(p. 2-108). Although the answers could be shorter, much of the extra
verbiage was repetition or elaboration. Therefore, telephone surveys
usually provided complete answers even if less words were used.
When considering completeness of data, Parten (1950) reported that
the telephone interview must be short, so only a few brief items could
be investigated (p. 87). Therefore, certain topics which would require
extensive information would not be able to gain complete data from a
Krueck (1975) found that negative comments were inhibited in the
two forms of personal contact, the telephone survey and the personal in-
terview. He also found a lack of consistency in negative answers when
comparing mail questionnaires to the telephone survey. He suggested
further research was needed with regard to the effect on personal in-
volvement in willingness to report in a negative manner (p. 19).
Hochstim (1967) reported that the personal interview elicited the
highest number of socially desirable answers, the mail questionnaire
elicited the least number of socially desirable answers, the telephone
survey fell between the two. Dillman (1978) stated that it seemed that
the transfer of the interviewer from the scene to the telephone decreased
the probability of socially desirable responses (p. 62).
Payne (1974) reported that the telephone interview was midway be-
tween the zero interviewer effect of the mail questionnaire, and the pro-
nounced interviewer effect in the personal interview. According to
Payne (1974), as little as possible of interviewer bias should go into
the recorded responses. To accomplish this, questions should be asked
in a standardized manner (p. 2-111). Dillman (1978) reported that the
distortion from interviewer bias would be easier to control in a tele-
phone interview than a personal interview with the opportunity for closer
supervision (p. 69). He also concluded that the telephone interview was
the best method in terms of the ability to control contamination from
other people (p. 65).
In a telephone interview, completeness of data depended on a com-
pleted phone interview. To accomplish this, researchers nave suggested
short interviews, avoiding sensitive questions, and employing well-
trained and competent interviewers to "sell" the respondent on the im-
portance of the interview (Parten, 1950, pp. 86-93; Warwick & Lininger,
1975, pp. 130-131).
Parten listed the following advantages of the telephone survey as
a summary of the follow-up technique:
1. The telephone interview is the quickest of the survey
techniques. Interviewers can complete about thirty calls
per hour if the calls are brief. It is especially adapted
to surveys of radio programs where a great many interviews
must be made during a given program.
2. The refusal rate is usually low among people who are
reached by phone.
3. The coincidental method can be used thus eliminating
the memory factor.
4. It is easy to train and supervise interviewers since
they can work in one room directly beside the supervisor.
5. The approach and questions are easy to standardize
from one interviewer to another.
6. The cost per completed interview is low for the
7. The geographic distribution of the sample can be
easily controlled. An address listing of numbers is
usually available and can be used for drawing the sample.
8. For studies of middle- and high-income groups, the
telephone interview may be satisfactory because most
of them have phones.
9. Interviews may be scattered over a wide area within
a city without adding to the cost.
10. As compared with a mail questionnaire, the telephone
survey is preferable because it usually costs less per
return. Returns are higher on first solicitation, and
they can be more effectively controlled from the point of
neighborhood distribution. (pp. 91-92)
Parten listed the following disadvantages of the telephone survey:
1. As a sample of the general population, telephone
subscribers are not representative. So unless the
telephone interview is supplemented by a method that
covers nonsubscribers, it should not be used. Less
hang half of all homes in towns over 2500 had telephones.
2. Detailed data cannot be gathered by this method
because the informants soon become annoyed or impatient.
If the schedule is too lengthy, the informant may either
hand up or give unreliable answers.
3. When observation of the situation is an important
element, the telephone interview is not useful. If the
interviewer is supposed to evaluate the answers as to
trustworthiness, he has very little to go on in a short
4. Information about the respondent must be limited to
one or two facts. Such items as age, nationality, in-
come, etc., are difficult to secure by telephone.
5. Attitude scales must be used with caution. Also,
opinions are less likely to be given freely since the
informant cannot be certain of the credentials of the
6. Since rural telephone ownership was low, the tele-
phone interview is not useful in such areas. Also,
because rural rates are higher than urban, the cost of
telephone inquiries is greater than in cities.
7. The brevity of the introduction and questions does
not give the informant much time to orient himself to
the subject matter of the survey. Reactions requiring
careful thought such as criticisms of various products,
suggestions as to new uses of products, appeals, etc. --
should not be obtained by this technique.
8. The telephone situation neither encourages the re-
spondent to amplify his replies nor gives the inter-
viewer much time to jot down the comments. A face-to-
face interview is more conducive to a considered response.
9. The task of checking on no-answers, wrong numbers,
busy signals, etc., is time-consuming but must be done
if the sample is to be representative of telephone
10. It is difficult to secure privacy on party lines.
11. The time may come when the telephone technique will
be used by so many groups tnat informants will develop
an antagonism to all telephone inquiries.
12. The surveyor must be careful not to antagonize in-
formants by phoning too early or too late in the day.
One well-known survey agency makes it a policy never to
call before 8:30 A.M. or after 10:30 P.M.
13. Misinformation is hard to detect and check in
short inquiries. (pp. 92-93)
In his 1975 study, Krueck stated the results of his study supported the
idea that more attention should be paid to telephone interviewing. It
had produced a higher response rate than the mail questionnaire or the
personal interview with only slightly higher costs than the mail ques-
tionnaire and much lower costs than the personal interview.
According to Warwick and Lininger (1975), a personal interview is
a form of verbal interaction designed to obtain information satisfying
the objectives of a particular study. "The interviewing process is
decidedly interactive, involving communication between the interviewer
and respondent" (Warwick & Lininger, 1975, p. 182). The standardized
interview, which was. designed to collect precisely the same categories
of information from a number of respondents so that the answer of all
respondents could be classified and compared, is the type of interview
which will be referred to in this paper as a "personal interview" (Gordon,
1975, p. 60). Three purposes for the personal interview were suggested
by Froelich and Darley (1952). These were supplementing information,
verifying information, and observation of characteristics or mannerisms.
Mayer (1974) reported that personal interviews were certainly the
most common way of gathering data in marketing research. He described
the personal interview as the most flexible method of data collection.
According to Mayer:
The interviewer's physical presence enables the researcher
to use various forms of interaction between the interviewer
and the respondent. The most common interaction is that
the interviewer asks questions of the respondent, following
a structured format or questionnaire, and records the re-
spondent's answers on the questionnaire. (p. 2-82)
Warwick and Lininger (1975) reported "the success of the personal
interview depends on the quantity and quality of the information ex-
changed between the interviewer and the respondent" (p. 182). They
stated four interacting parties or conditions affect the flow of infor-
mation; the interviewer, the respondent, the topic, and the immediate
interviewing situation. The interviewer, facing the respondent, must
communicate the question, motivate the respondent to cooperate, probe
for additional information, and record the information gathered from
the respondent. The respondent must provide answers which are complete
and truthful. The topic should stimulate the interest of the respondent
without arousing anxiety, hostility, or fear. The interviewing situa-
tion must consider such conditions as the time and place of the conver-
sation, the presence of any "third parties," and the similarities be-
tween the interviewer and the respondent (p. 183).
Mayer (1974) reported that the personal interview assumes the least
about the respondent (p. 2-85). The human interaction element brought
variability to the personal interview follow-up method. The personal
interview will be reviewed using the five variables previously chosen to
compare and evaluate the various follow-up methods. These five variables
are: costs, response rate, length of time, personnel required, and
quality of data.
It was generally agreed that conducting personal interviews was an
expensive method of gathering follow-up information. Since an inter-
viewer must be physically present, personal interviews were more expen-
sive than other methods of follow-up (Mayer, 1974, p. 2-85). Sudman
(1967) reported that from 30% to 40% of the interviewer's time was spent
interviewing. Mayer (1974) reported:
The interviewer spends time in learning the specific re-
quirements of the survey, traveling to the designated areas,
locating the required households, identifying the desired
respondent, asking questions, recording the answers, and sub-
mitting the completed work. Many of the attempts to contact
potential respondents may be unsuccessful. (p. 2-86)
Travel costs, including transportation expenses, interviewer time, and
the training of interviewers increased the cost of personal interviews.
Jackson and Rothney (1961) reported that for every dollar spent
for their mailed questionnaire study, approximately 60 dollars were
spent on the personal interview study. In this instance, the interview
method cost 60 times as much as the mail questionnaire method.
Krueck (1975) analyzed the costs in a comparison study of the three
methods of conducting follow-up methods. The personal interview method
included the travel which was converted to a dollar figure at a rate of 10
per mile. Krueck's study was fortunate in that 60% of the interviews
were conducted at the study sight, requiring no travel for the interviewer,
and thereby saving expenses. For the off location interviews requiring
travel, the average distance traveled one way was seven miles. Thus a
minimum of 284 miles were logged in conducting interviews for a cost of
$30.20. Clerical help and other manpower costs brought the total cost
of the personal interview study to $426.50. When the response rate was
considered, the cost per response was $5.86. Krueck concluded that the
personal interview follow-up method was the least cost effective method
when compared to the telephone survey and the mailed questionnaire.
Weiss and Hatry (1971) reported that in the early 1970's, profes-
sional interviewers were earning from $2.00 to $3.00 an hour. Interviewers
usually received pay not only for interviewing time, but for time spent
in training, reading instruction, traveling, locating and contacting re-
spondents, and preparing the data after the interview (p. 20). Further
costs associated with the personal interview method, according to Weiss
and Hatry (1971), were the time and money spent for recruiting and se-
lecting the interviewers, and for trainers and supervisors. Weiss and
Hatry listed the following factors as having an effect on interviewing
--Productivity (number of interviews completed per hour),
depending on the interviewers' ability, experience and
--Time spent for call-backs to respondents who were not
at home on the first call. Procedures should limit
the number of attempts to reach any one respondent
(three is a common limit).
--Neighborhood being surveyed. In some areas, costs
go up because residents refuse to open their doors to
strangers, pairs of interviewers are needed for security
reasons, housing or population listings are inadequate,
and so forth.
--Travel time and costs between interviews, and from
interview locations to the survey office.
--Financial payment to respondents. Most experienced
survey researchers frown on paying respondents for
participating, except as a last resort, but some ad-
vocate it. (pp. 20-21)
Weiss and Hatry (1971) emphasized that costs with the personal in-
terview method relate directly to the sample size and the number and com-
plexity of questions. Answers that allow easy coding take little time,
whereas long narrative and open ended questions required the development
of careful classifications and cross checking of different coders' de-
cisions on classifications (p. 21).
Table 6 displays the costs estimated by Weiss and Hatry for con-
ducting 500 interviews. The table shows that costs could range from a
low of $11.87 per interview to a high of $39.80 per interview. Consider-
ing only interviewing costs, the cost per interview was $14.60. For 400
interviews their cost estimates were a low of $12.71 per interview to a
high of $44.13 per interview. For 1000 interviews, cost estimates ranged
from a low of $10.90 per interview to a high of $33.00 per interview.
Cost Estimates For Five Hundred Interviews
1. Overall planning and direction
2. Preparing listing
3. Selecting sample, preparing maps,
assigning households to interviewers
4. Developing interview questionnaire
(including conferences and 20 pre-
5. Printing interview questionnaires,
training materials, manuals, supplies
6. Interviewing costs
(b) Training (20 interviewers for 20
hours, plus trainer's salary)
(c) Interviewing time (including
travel, call-backs, recopying)
(d) Field supervision
(e) Travel expenses
7. Editing, coding, punching, tabulating
10. Analyzing data and report writing
11. Reproducing report
Average cost per interview
Note. From An Introduction to Sample Surveys
by C. H. Weiss & H. P. Hatry. Washington, D. C.:
for Government Managers
Urban Institute, 1971,
Krueck (1975) in his follow-up method comparison study reported
that his data indicated the mailed questionnaire and personal interview
methods were approximately equal in effectiveness, eliciting response
rates of about 50%.
Parten (1950), however, stated that the most striking feature of
the personal interview method was the high percentage of acceptable re-
turns obtained, especially in well organized and carefully planned
studies (p. 73). She reported that in studies of the general public,
most surveys could depend on at least 80% response. Though Parten stated
the response rate was usually high, Denton et al., (1977) reported the
use of personal interviews reduced the number of individuals who could
be surveyed due to the time and expense involved in conducting personal
interviews (p. 7). Therefore, according to Denton et al., (1977), the
personal interview method may have a higher response rate, but the size
of the sample or population was usually smaller than other methods of
Baugh conducted a study in 1976 where he used both mail question-
naires and personal interviews. His response rate for the mail question-
naire section of the study was 55%, but he had a 100% response rate for
the personal interview. Mayer (1974) stressed that the personal inter-
view has the highest potential response rate.
Length of Time
A follow-up study consisting of personal interviews usually takes
a longer period of time to complete than either a telephone survey or
mail questionnaire. Cox (1966) disagreed with the majority of researchers
when he stated, "given a tight schedule, a researcher would expect to
turn to the personal interview method, usually at greater cost" (p. 392).
Denton et al., (1977) stated the more popular view when he reported the
personal interview to be time consuming. Jackson and Rothney (1961) per-
formed a comparison of the mail questionnaire and the personal interview
follow-up methods and found the mail questionnaire had the advantage of
saving time over the personal interview.
Dillman (1978) reported that the time involved in a personal inter-
view study varied depending on the sample size and geographical disper-
sion (p. 68). He further stated that the time required was closer to
that required for mail questionnaire studies than telephone surveys
Krueck (1975) reported that the average time for conducting a fol-
low-up personal interview in his study was 20 minutes. He calculated
12 hours of clerical time was spent in planning and assimilating various
pieces of information relating to the personal interview. Totally,
116.5 manhours were required for the completion of approximately 80 in-
terviews (p. 24). Weiss and Hatry (1971) reported that the personal
interview took time for recruiting and selecting interviews as well as
training before the interviews could begin (p. 20).
Parten (1950) reported that the time required to cover a large pop-
ulation with personal interviews may make the method undesirable. She
also pointed out that the personal interview usually takes more time
than the telephone interview.
Basically, the time required for a personal interview, as for all
methods, depended on the number of responses desired. In a very large
sample, using a mail questionnaire, Parten (1950) reported that a per-
sonal interview may be required eventually to check for nonresponse
bias in the mail questionnaire (p. 96).
Mayer (1974) reported that the personal interview survey had the
longest and most complicated scheduling requirements. According to Mayer,
"The scheduling and timing requirements of personal interview surveys
were the chief reasons why this method of data collection was the slow-
est" (pp. 2-84 2-85).
As McKinney and Oglesby (1971) reported, the personal interview
method of follow-up was not only expensive, but required a great amount
of staff time. Whether it was staff that was presently employed, or
staff that must be hired and trained, this method of follow-up was a time
The main reason personal interviews required such a large staff was
that all interviews must have a staff member present, or it was not a
personal interview. Parten (1950) reported that the organization required
for selecting, training, and supervising a group of interviewers working
in the field was more complex than that needed for surveys conducted by
other methods of follow-up (p. 81). Interviews of working people required
an even larger staff since those, interviews had to be conducted at cer-
tain hours in the evening.
Dillman (1978) reported that personnel requirements for the personal
interview were the most difficult to meet. "The availability of people
with interviewing and supervisory skills is as low as the cost is high"
(p. 66). Supervisory networks should be extensive for control. There-
fore, the size of the study would have a direct effect on the number of
people required to perform a personal interview study. In most cases,
the personal interview study required a larger staff than either the
mail questionnaire or the telephone survey (p. 66).
Quality of Data
The quality of data produced through personal interviews had been
both praised and criticized. McKinney and Oglesby (1971) suggested that
probably the most desirable kind of follow-up was the personal interview,
because it provided the opportunity for the respondent to ask questions
for clarification as well as for the interviewer to ask questions for
deeper insight into the problems of occupational preparation as perceived
by the respondent (p. 7).
Gordon (1975) stated that the personal interview provided more op-
portunity to motivate the respondent to supply accurate and complete in-
formation immediately (p. 76). He also stressed the personal interview
provided more opportunity to guide the respondent in his/her interpre-
tation of the question, allowed greater flexibility in questioning the
respondent, allowed greater control over the interview situation, and
provided a greater opportunity for evaluation of the validity of the
information through observation of the respondent's nonverbal responses
Parten (1950) reported that the personal interview data was more
correct than that secured by other methods since the interviewer could
clarify seemingly inaccurate answers by explaining the questions to
the respondent (p. 9). Interviewers could also be trained to spot
falsified responses or could collect supplementary information about the
respondent if the responses indicated a need for more data (p. 80).
Another advantage of data collection with the personal interview was
that visual cues could be utilized. However, the danger of interviewer
bias had to be considered as well.
Parker, Wright, and Clark (1957) performed a study where college
students were interviewed concerning their scholarships, and the inter-
views were recorded. A "check interviewer" listened to the recordings
and was asked both what the student said, and what they interpreted the
student as saying. An interesting result of their study was that the
two interviewers, the one who performed the interview in person and the
one who listened to a recording, did not always agree on what the stu-
dent actually said. Overall, the results from this study indicated
that the personal interview does not yield 100% reliable data. However,
they asked, "Can we afford the time and expense to train interviewers
when questionnaires or similar techniques can only be slightly less re-
liable?" (p. 220).
Herbert Hyman (1950) was concerned with the effect of the inter-
viewer on the quality of data gathered during a personal interview. A
two year study was done at the National Opinion Research Center to
isolate, measure and control the interviewer effects on the respondent.
It was thought that the respondents answers were affected by the per-
sonality or characteristics of the interviewer. One of the findings
of the study was that interviewers would report differently even if the
response was the same. Hyman used a recorded interview and found dif-
ferences between interviewer reports of the same responses, suggesting
differences between interviewer reports could cause inconsistencies
without including the aspects of respondents' answers. Next, the study
viewed the respondent and found respondents gave different responses
regardless of whether an interviewer was present or not. Therefore, some
of the inconsistencies came only from respondents. These two possible
biases, and the interaction that may exist between them concerned most
users of the personal interview method of follow-up.
The Hyman (1950) study also reported the other view; some respondents
might not be affected by interviewer bias at all. The study reported
that a few respondents could not even recall what the interviewers looked
like, and for such respondents, the interviewer respondent interaction
was minimal. However, according to Hyman, a series of other studies
showed that survey results for specialized attitudes are affected by
the disparities or similarities in the group membership of interviewers
and respondents. He reported that there was documentation to show that
responses varied with the disparity between the interviewer's and re-
spondent's sex, class, color, religion, and other group membership
factors. This suggested that respondents perceived something about the
interviewer immediately from his or her appearance or behavior, inter-
preted this for some reason in a certain way, and in turn, altered
their behavior. On the other side, the interviewer perceived the re-
spondent and subsequently altered his or her method of questioning,
probing, and recording. Interviewers were found to have certain ex-
pectations about how the respondent would respond and these expecta-
tions affected how he or she would probe for answers and record them.
To minimize interviewer effect, interviewers were usually required
to ask questions exactly as they were written, so all questions would
be asked with the same wording. Training of interviewers was given to
eliminate interviewer bias as much as possible, but it was still a
recognized problem with the personal interview method (Dillman, 1978,
Parker, Wright, and Clark (1957) reported on several studies which
supported the reliability of personal interviews. They reported that
Anderson (1954) found a 0.85 level of consistency in ratings when a
personal interview was used. Levin (1954) concluded that short inter-
views (about one hour in length as opposed to two or two and a half
hours) produced more reliable data than longer interviews.
In regard to completeness of data collected by the personal inter-
view method, Krueck (1975), in his comparison study, found that the
personal interview method had a much lower frequency of no response to
individual questions than did the mail questionnaire method.
Jackson and Rothney (1961) performed a study which found that per-
sonal interviews elicited significantly more complete answers than the
mail questionnaire. The personal interview also elicited a greater
number of responses than the mail questionnaire.
The personal interview method was found to produce more socially
desirable answers than either the telephone survey or the mail question-
naire. Hochstim (1967) concluded that generally respondents were most
honest in mail surveys, least honest in personal interviews, and in-
termediately honest in the telephone surveys. Dillman (1978) supported
the idea that personal interviews have the highest probability for
producing socially desirable answers as opposed to accurate answers.
Dillman (1978) stated "The greatest advantage of the personal interview--
the physical presence of the interviewer--may at tliaes be its greatest
drawback" (p. 63).
Parten (1950) listed the advantages of the personal interview as
1. The personal interview usually yields a high percentage
of response,for most people are willing to cooperate.
2. It can be made to yield an almost perfect sample of the
general population because practically everyone can be
reached by and can respond to this approach.
3. The information secured is likely to be more correct
than that secured by other techniques once the interviewer
can clear up seemingly inaccurate answers by explaining
the questions to the informant. If the latter deliberately
falsifies replies, the interviewer may be trained to spot
such cases and use special devices to get the truth.
4. The interviewer can collect supplementary information
about the informant's personal characteristics and erviron-
ment which is invaluable in interpreting results and eval-
uating the representativeness of the persons surveyed.
5. Scoring and test devices can be used, the interviewer
acting as experimenter.
6. Visual material to which the informant is to react
can be presented.
7. Return visits to complete items on the schedule or to
correct mistakes can usually be made without annoying the
informant. Thus greater numbers of usable returns are as-
sured than when other methods are employed.
8. The interviewer may catch the informant off guard and
thus secure more spontaneous reactions than would be the
case if a written form were mailed out for the informant
to mull over.
9. The interviewer can usually control which person or
persons answer the questions, whereas in mail surveys
several members of the household may confer before the
questions are answered. Group discussions can be held with
the personal interview method if desired.
10. The personal interview may take long enough to allow
the informant to become oriented to the topic under in-
vestigation. Thus recall of relevant material is facili-
11. Questions about which the informant is likely to be
sensitive can be carefully sandwiched in by the interviewer.
By observing the informant's reactions, the investigator
can change the subject if necessary or explain the
survey problem further if it appears that the inter-
viewee is about the rebel. in other words, a delicate
situation can usually be handled more effectively by
a personal interview than by other survey techniques.
12. More of the informant's time can be taken for
the survey than would be the case if the interviewer
were not present to elicit and record the information.
13. The language of the survey can be adapted to the
ability or educational level of the person interviewed.
Therefore it is comparatively easy to avoid misinter-
pretations or misleading questions.
Parten (1950) listed the disadvantages of the personal interview
1. The transportation costs and the time period to
cover addresses in a large area may make the personal
interview method unfeasible.
2. The human equation may distort the returns. If an
interviewer has a certain economic bias, for example,
he may unconsciously ask the questions so as to secure
confirmation of his views. In opinion studies especially,
such biases may operate. To prevent such coloring of ques-
tions, most opinion surveyors instruct their interviewers
to ask the question exactly as printed on the schedule.
3. Unless the interviewers are properly trained and
supervised, the data recorded may be inaccurate and in-
complete. A few poor enumerators may make a much higher
percentage of returns unusable than if the informants
filled out and mailed the interview form to survey head-
4. The organization required for selecting, training,
and supervising a field staff is more complex than that
needed for surveys conducted by other methods.
5. It is usually claimed that costs per interview are
higher when field investigators are employed than when
telephone or mail surveys are used. This may not be true
if the area to be covered is not too great. If the gen-
eral public in a community is to be surveyed, the costs
of securing a representative sample by telephone or mail
inquiries will probably equal or exceed the cost by the
personal interview method, since in the end personal
follow-up will be necessary to round out the sample.
6. The personal interview usually takes more time than
the telephone interview providing the persons who can be
reached by telephone are a representative sample of the
type of population to be covered by the survey. However,
for a sample of the general public a telephone inquiry is
not a substitute for a personal interview. The lowest in-
come groups often do not have telephones.
7. If the interview is conducted in the home during the
day, the majority of the informants will be housewives.
If a response is to be obtained from a male member of
the household, most of the field work will have to be
done in the evening or on weekends. Since only an hour
or two can be used for evening interviewing, the personal
interview method requires a large staff for studies re-
quiring contacts with the working population. (pp. 79-82)
Computerized Data Retrieval System
The computerized data retrieval system was developed by the Florida
Inter-Institutional Research Council (IRC) as an alternative method of
obtaining information concerning community college graduates (Miner,
1977, p. 2). According to Norris Miner (1977), the computerized data
retrieval system used computer input data containing the graduate's
social security number (ssn) and program of study completed. These data
were matched by ssn with the third quarter of 1976 Department of Com-
merce files. The files contained employment data on all persons in
the state who were eligible for unemployment compensation. For every
match in ssn with graduate files, the number of weeks worked, total
wages, and the code for the employer were obtained. The graduate files
were then matched by ssn with the State University System (SUS) stu-
dent master file for term 1 of 1976-77. When a match occurred within
the SUS data tape, the university and program of study were obtained.
Member colleges of the IRC decided to compare two alternative
methods of follow-up by using both the mail questionnaire method, and
the computerized data retrieval method.
According to Miner (1977), a questionnaire was mailed to 443 grad-
uates of Seminole Community College. Then the computerized data re-
trieval system was performed with a match in ssn being considered a
response. Table 7 presents the results of the two studies.
The results showed the improved response rate that the computerized
data retrieval system supplied and also showed the more in-depth infor-
mation supplied by the questionnaire. The computerized data retrieval
system will be discussed using the five variables: costs, response rate,
length of time, personnel required, and the quality of data.
Since the costs were related to personnel costs, the costs for the
computerized data retrieval system were lower than the other methods of
follow-up. Miner (1977) reported that the computerized data retrieval
system was less expensive than mailing questionnaires and processing the
responses. Preparing the program, gathering the data tape, and using
computer time were the major expenses involved. The largest cost was
the initial cost of writing the program. After that, occasional up-
dating would allow the program to be functional. The cost of running
the computer program was not high, therefore the overall cost was low.
Also, since little human time was involved, this method was less ex-
pensive than other methods.
The response rate for the computerized data retrieval system in
Miner's study was 82.8% compared to the same study done with a mail
questionnaire response rate of 56%. The computerized data retrieval
A Comparison of a Mail Questionnaire Study and a Computerized
Data Retrieval System Study
Completion rate 56% 82.8%
Employed 12.9% 42.8%
Advanced education 81.5% 57.2%
Other 5.5% 3.3%
Note. From "The Problem with the Placement Study," by N. Miner,
Sanford, Florida: Seminole Community College, 1977, p. 6.
system had several constraints in regard to response. Any potential
respondent working or attending school out-of-state would be lost to
the study. Also, any potential respondent enrolled in a private school
would be lost to the study since neither would be on the two data tapes.
A follow-up of nonrespondents was conducted and Miner (1977) re-
ported that eleven nonrespondents were continuing their education with
seven delayed entrants into the SUS. Two were in private colleges and
two were in out-of-state colleges. Another five were found to be em-
The results also demonstrated a response bias in the mail question-
naire. It was found that graduates who continued their education were
more likely to responded to questionnaires.
The computerized data retrieval system had a built-in response bias,
the loss of responses from out-of-state graduates and private school stu-
dents. A "point in time" problem was apparent with the computerized data
retrieval system also. Some respondents were lost because they delayed
one or more school terms before entering the SUS. A longitudinal study
utilizing the computerized data retrieval system would eliminate part
of this problem.
Length of Time
The length of time needed to prepare a questionnaire, stuff the en-
velopes, mail them, wait for returns, and analyzed them, made the mailed
questionnaire much longer to perform than the computerized data retrieval
system. The computerized data retrieval system only waited on the pre-
paration of the data tapes once the program was initially written. It
could then be used innumerable times. The actual time needed to run the
program was minuscule once the results had been collected. This made.
the computerized data retrieval system require the shortest length of
time to perform an actual follow-up procedure.
The only essential personnel for a computerized data retrieval
follow-up study was a person to perform the computer programming aspects.
The initial program had to be written and tested. Once it was operational,
then a computer operator could conduct a run of the program when neces-
sary, and the results would be ready for reporting.
Quality of Data
The data used in the computerized data retrieval system were data
supplied by college records and data supplied by the Department of Com-
merce. Miner (1977) reported that this eliminated any errors introduced
through self-reported data. Therefore the follow-up study results were
based on data that were as accurate as the college records and the govern-
ment employment records.
In regard to completeness of response, the computerized data retrie-
val system reported only the employer occupational code rather than the
occupational code of the respondent. Therefore, as Miner stated in his
report, the computerized data retrieval system did not provide direct in-
formation on related and nonrelated employment to program of study. This
imposed a lack of completeness with regard to employment relating to pro-
gram of study. The employer code, in some, but not all instances, sup-
plied sufficient information to determine the area of respondent employ-
ment, and whether or not the area was related to the programs of study at
Comparing different information collecting survey techniques has been
recognized as a difficult task. Dillman (1978) stated:
The question of which method is best cannot be answered
in abstract terms. Although each method has certain
strengths and weaknesses, they do not apply equally, or
sometimes at all, to every survey situation. Thus, until
the attributes of each method are considered in relation
to the study topic, the population to be surveyed, and
the precise survey objectives, the question of which is
best cannot be answered. (p. 39)
Table 8 is a replication of a table from Dillman's book Mail and
Telephone Surveys (1978). This book offers a comparison of mail question-
naire, telephone survey, and personal interview methods of conducting a
survey. The table lists the ratings of performance characteristics for
each of the three traditional survey methods as high, medium, or low.
This allows the researchers to find the performance characteristics im-
portant for a particular study and observe which survey method is high
or low in those important characteristics. Each survey will have some
characteristics which are more important than others. The survey method
chosen should have a high rating in as many of the important performance
characteristics as possible.
With the recognition of the difficulty of comparing follow-up methods,
this section will present a comparison of four methods of follow-up
based on the five criteria previously derived from the Review of Re-
lated Literature. An analysis of the advantages and disadvantages, and
Performance Characteristics Rating for Personal Interviews
(Face-to-Face), Mail Questionnaires, and Telephone Interviews
Performance Face-to-Face Mail Ques- Telephone
Characteristics Interviews tionnaires Interviews
I. Obtaining a Represen-
A. Known opportunity
for all members of
population to be in-
cluded in the sample.
1. Completely listed
populations. High High High
2. Populations which High Medium Medium
are not completely
listed (e.g., house-
B. Control over selec- High Medium High
tion of respondents
within sampling units.
C. Likelihood that selec- Medium High High
ted respondents will
D. Insensitivity to Medium Low Low
substitution of re-
spondents and house-
E. Response rates.
1. Hetergeneous High Medium High
2. Homogeneous, spec- High High High
(e.g., agency di-
xml version 1.0 encoding UTF-8
REPORT xmlns http:www.fcla.edudlsmddaitss xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.fcla.edudlsmddaitssdaitssReport.xsd
INGEST IEID EFFGSA094_HFTA3R INGEST_TIME 2012-03-09T00:03:32Z PACKAGE AA00003898_00001
AGREEMENT_INFO ACCOUNT UF PROJECT UFDC