• TABLE OF CONTENTS
HIDE
 Front Cover
 Preface
 Main
 Back Matter
 Back Cover






Group Title: Gatekeeper series
Title: O.K., the data's lousy, but its all we've got
CITATION THUMBNAILS PAGE IMAGE ZOOMABLE
Full Citation
STANDARD VIEW MARC VIEW
Permanent Link: http://ufdc.ufl.edu/UF00089561/00001
 Material Information
Title: O.K., the data's lousy, but its all we've got being a critique of conventional methods
Physical Description: 19 p. : ill. ; 25 cm.
Language: English
Creator: Gill, Gerard J.
Publisher: International Institute for Environment and Development ( IIED )
Place of Publication: London
Publication Date: 1993
Copyright Date: 1993
 Subjects
Subject: Natural resources -- Management -- Developing countries   ( lcsh )
Natural resources -- Management -- Nepal   ( lcsh )
Agricultural economics   ( sigle )
Documentation, information science and librarianship   ( sigle )
Genre: bibliography   ( marcgt )
non-fiction   ( marcgt )
 Notes
Bibliography: Includes bibliographical references (p. 16-17).
General Note: IIED Gatekeeper series, number 38
Statement of Responsibility: Gerard J. Gill.
 Record Information
Bibliographic ID: UF00089561
Volume ID: VID00001
Source Institution: University of Florida
Holding Location: University of Florida
Rights Management: All rights reserved by the source institution and holding location.
Resource Identifier: oclc - 28865647

Table of Contents
    Front Cover
        Front Cover 1
        Front Cover 2
    Preface
        Page 1
        Page 2
    Main
        Page 3
        Page 4
        Page 5
        Page 6
        Page 7
        Page 8
        Page 9
        Page 10
        Page 11
        Page 12
        Page 13
        Page 14
        Page 15
        Page 16
        Page 17
    Back Matter
        Page 18
        Page 19
        Page 20
    Back Cover
        Page 21
        Page 22
Full Text
Published by the Sustainable Agriculture Programme of the
International Institute for Environment and Development


16f 7 S'




O.K., The Data's
Lousy, But Its All
We've Got
(Being a Critique of
Conventional Methods)




Gerard J. Gill






IIED
GATEKEEPER SERIES No. 38
INTERNATIONAL
INSTITUTE FOR
ENVIRONMENT AND
DEVELOPMENT

















































The Gatekeeper Series of the Sustainable Agriculture Programme is produced by the International
Institute for Environment and Development to highlight key topics in the field of sustainable
agriculture. The Series is aimed at policy makers, researchers, planners and extension workers
in government and non-government organizations worldwide. Each paper reviews a selected issue
of contemporary importance and draws preliminary conclusions of relevance to development
activities. References are provided to important sources and background material. The Swedish
International Development Authority and the Ford Foundation fund the series.

Gerard J. Gill is Program Leader, Policy Analysis in Agriculture and Related Resource
Management program (His Majesty's Government of Nepal, Ministry of Agriculture/Winrock
International), Winrock International Institute for Agricultural Development, PO Box 1312,
Kathmandu, Nepal.

1993


GATEKEEPER SERIES NO. SA38 1




































































2 GATEKEEPER SERIES NO. SA38








O.K., THE DATA'S LOUSY, BUT IT'S
ALL WE'VE GOT
(BEING A CRITIQUE OF
CONVENTIONAL METHODS)



Almost ten years ago Chambers' damning indictment of 'survey slavery' did enormous service in
focusing attention on this form of addiction and encouraging the subsequent development of
participatory learning methods (Chambers 1983). However, although our previous unswerving
allegiance to large-scale customized surveys has at last been seriously challenged, and although
as a result these have lost considerable ground in Third World rural research, participatory methods
are still a very long way from the point where they may be regarded as 'conventional'.

This can be said for two reasons. First, although the large-scale customized survey has to a large
extent been discredited, this is not the principal way in which outsiders purport to learn about the
rural poor and the environments they occupy. A more important source is the vast body of official
published statistics existing in all developing countries. This is certainly true in policy analysis,
which is the primary practical reason why outsiders should want to know about the rural poor:
namely as a prelude to effective action to alleviate their poverty and any adverse environmental
consequences flowing from it.

Such statistics are often generated as by-products of other government activities (such as land
registration or tax collection). Obviously by definition such data-generating exercises are not
customized. Alternatively, official statistics may be collected in order to provide information
about a variety of issues in which a government is interested (such as land holding size, input use,
output levels, population size, land use, and trends in all of these). Here collection is routine and
aims at providing general information, rather generating data designed to test previously
formulated hypotheses, as a truly customized scientific data collection exercise would do.
Nevertheless this latter body of official statistics is based on large-scale surveys for example
sample censuses of agriculture or household expenditure surveys and as such is subject to the
same criticisms Chambers levelled at the large-scale customized survey. Chambers' critique
notwithstanding, such official surveys seem as popular as ever with executives at national
statistical bureaux and with the donor agencies which support their efforts.

The second reason that participatory methods cannot be described as conventional is that
questionnaires are still the basic often the only data gathering instruments used in socio-
economic research.' Although the large scale customized survey may have lost favour, the basic
instrument on which it depended is still employed at practically all levels, from the macro sample
census down to the postgraduate student's small-sample dissertation research.


I Strictly speaking the term "questionnaire" applies only to forms that are filled in by the respondent, as in the case of
Britain's decennial population census. Where an enumerator is employed to ask the questions and fill in the answers as
with rural surveys in Third World countries the form is actually a "schedule". However usage has by now sanctioned the
use of "questionnaire" for both types of form, so this tern is used here to avoid confusion.


GATEKEEPER SERIES NO. SA38








The quotation used as a title for this monograph is taken from an international economic expert
recently speaking off the record.2 The implication that official statistics are the only means through
which one can learn about the rural poor and their interaction with the environments they inhabit
(the topic under discussion when the statement was made) is one that would be challenged by those
convinced of the logic and power of the participatory approach. However to address this particular
issue in the present forum would be preaching to the converted. What will be done instead is first
to illustrate just how lousy the data actually are and, more importantly, what fundamental mistakes
can be made when policy decisions are based on their analysis. The second objective is to take
a rather close look at the questionnaire and to suggest that its employment in rural areas of
developing countries is quite inappropriate, and is a primary reason for the pestiferous nature of
the data it purports to generate. It is my conviction that while those who have adopted the
participatory approach to rural research in developing countries are prepared to ditch the large-
scale socio-economic survey, some are still reluctant to part company with the questionnaire.

This monograph will focus primarily on the sustainable use of natural resources in developing
countries. It will concentrate especially on socio-economic aspects of the topic and ii: will be based
largely on the experience of Nepal. There is, however, little reason to believe that Nepal is in any
way unique with respect to the topics under discussion.



Official Statistics

Official statistics exercise a powerful attraction and not just among economists for a number
of compelling reasons. First, they are readily available: sometimes the raw data are even available
in machine-readable format. Second, they are relatively cheap and painless to acquire and use,
since no data collection exercise is required. Third, they usually have a very wide coverage
compared to all but the most expensive and time-consuming customized surveys and are therefore
particularly well-suited to macro level analysis. Finally, since they are both published and
government statistics, their use has implicit official sanction.

Unfortunately, as the opening quotation implies, in developing countries official statistics are
often most unsatisfactory. They are characterized by unreliability, gaps, over-aggregation,
inaccuracies, mutual inconsistencies and lack of timely reporting. These problems are especially
marked in the realm of natural resource management, since the relevant variables are often
unusually hard to quantify (sometimes even hard to define), the geographical areas in question
frequently remote, and the necessary data correspondingly difficult to collect and verify.

Economists are trained to deal in numbers. So what does a trained economist do when the numbers
that exist exhibit the above inadequacies and there is neither the time nor the resources needed to
conduct a purpose-designed study? Unfortunately too many of us do what the above-quoted
international expert does: use them regardless. It may seem obvious, but in view of what has just
been said it is perhaps worth stating the obvious: wrong data are worse than no data. Analysis based

2This is an actual quote, and the speaker, a full professor of economics at a reputable US university, is reasonably well-
known. However it would be unfair to single him out for identification, as his views are representative of a fairly wide cross
section of development experts.


GATEKEEPER SERIES NO. SA38








upon them gives a spurious impression of accuracy where none is justified, and this in turn lends
an unwarranted weight and cogency to policies based upon it. This would not happen if we frankly
acknowledged our ignorance. If policy formulation is empirically based, wrong data can lead to
wrong decisions.

The following example from Nepal will illustrate how serious the dangers really are:

"Continued population pressure on land resources in the hills and mountains has resulted
in expansion of farming onto marginal cultivable land, with ensuing environmental
degradation soil erosion, losses of soilfertility, a deterioration offorests andforest covers"
(NARC-ADB, 1991:15).

This is the received wisdom about the Nepalese hills: cultivated area has been expanding, is
expanding and will continue to expand in the foreseeable future. But what is the basis of such
statements? First, census figures support the assumption that the population of the hills is
increasing.' Since the increasing number of mouths has to be fed, and since productivity is not
generally rising, increased production has to come from area expansion. Right? And since all the
best areas are obviously already cultivated, the rest of the quotation follows. The assumption of
expanding cultivated area is backed up by official statistics, which purport to show a steadily rising
trend in this variable. This is the set of statistics that now will be critically examined.

Statistics on cultivated area in Nepal derive from two principal sources, the decennial National
Agricultural Census and the ongoing Cadastral Survey. Nowhere is the mutual inconsistency of
official statistics demonstrated more clearly than when one compares these two sets of figures.
This is done in Table 1. Note that in the hill districts of every one of the country's five Development
Regions the Cadastral Survey figures are at least double, and can be as much as eight times as high
as, those of the Agricultural Census, and how over the surveyed hill districts as a whole, the
cadastral figures are almost four times as high as the Census estimates.

Table 1: Estimates of Total Cultivated Area in Hill Districts of Nepal's five
Development Regions*
Development National Agricultural Cadastral Survey Ratio of Census Figure
Region Census (hectares) (hectares) to Cadastral Figure

Eastern 231,444 766,637 1 3.3
Central 308,658 669,365 1 2.2
Western 210,813 980,891 1 4.7
Midwestern 51,581 439,632 1 8.5
Far Western 28,003 205,297 1 7.3

Total 830,499 3,061,822 1 3.7
Includes only those 69% of the couintlr's hill districts which have been cadaslrcl// surveyed so fur.
The dat./for this table were collected and complied f/iom the Agricultural Census IIand Casastral Siuriv reports O
hy ,M Devika Taniang.

The question of whether the figures support the assumption or derive filom it is one lthat will not be addressed here. Perhaps
it is worth observing. nonetheless, that internal migration could in falt reduce the population of parl(icular regions of a
country even within the context of'overa// /popIultion growth. It is also worth pointing out that acci( depnc of *', -' I view
on hill population growth is not self'-serving as fir as thl' preset argtent is concerned,. my rcas would he stronger still
were this assumption rejected, or even questioned.


GATEKEEPER SERIES NO. SA38








Balogun (1989) has presented a detailed and persuasive analysis of the figures for the Western
Development Region in which he concluded that the apparent trend of increasing cultivated area
in the hills is actually the result of combining these two mutually-inconsistent sets of statistics.
He found that when a district's Cadastral Survey is completed this figure is substituted for the
previously-used Census figure in the official estimates, a process which can produce huge jumps
in estimated cultivated area at district level.

Figure 1 goes beyond the Western Development Region analysed by Balogun and includes one
hill district from each of the country's five such Regions and illustrates the orders of magnitude
that can be involved. For each district this graph shows: (a) the official figure on cultivated area
based on the most recent Agricultural Census; (b) the revised figure for that district whenever the
Cadastral Survey figure is substituted, and (c) the resulting quantum leap in the official estimate.
The figure for Khotang District may be somewhat extreme (although it is certainly not a statistical
'outlier'), with the estimate of cultivated area leaping from 7,955 to 157,187 hectares in a single
year! (Note the totally spurious level of precision: calculated to the nearest hectare, no less!) This
extreme case has been chosen deliberately, for even here, even with an almost 2,000% increase
in estimated cultivated area in a single year, the revised figure was rolled into the official estimates
without a word of explanation or warning.

When examined in such a disaggregated form, the figures, are, of course, literally unbelievable.
However, because the Cadastral Survey is proceeding only slowly with just one or two hill
districts being completed each year the resulting jumps in the estimates at district level are lost
in the overall regional or national figures. This in turn produces the apparently steadily increasing
- and 'statistically significant' trend in area under cultivation that was mentioned earlier. It is not
a real trend, of course, but a spurious one caused by the progressive substitution of high estimates
for low ones. The census figures can be assumed to be understatements because farmers have every
incentive to under-report in order to minimize their land taxes. The cadastral figures are over-
statements because the Survey represents an opportunity for 'land grabbing': individuals
competing to register public land, commons and other lands to which they have no real claim.4

Carson, a soil scientist with extensive and prolonged field experience in the hills of Nepal, argues
(on the basis of participatory methods backed up by rigorous pedology) that cultivated areas in
the hills, far from increasing, are actually declining, as a result of loss of organic matter, growing
soil acidification and the build-up of aluminium toxicity on the land (Carson 1992). This, he
argues, is forcing farmers either to abandon cultivation altogether, or to put the land under
increasingly long fallows. Outside of the flattest valley bottoms, hill agriculture in Nepal is
basically terraced agriculture. In his extensive travels throughout the Nepalese hills, he found
thousands of examples of abandoned terraces against only a few dozen new ones he observed being
constructed.

But if cultivated area is not expanding perhaps even shrinking how are the extra mouths fed?
The most likely answer is the one suggested by Tamang (1992) migration, either permanent or
seasonal. Seasonal migration is almost always overlooked in Census data, but it plays a crucial
role in many livelihood systems, including those of hill farmers throughout Asia. Tamang,
SMany informed observers report that the Cadastral Survey is actually a major cause of deforestation, a.;people cut down
public or communal forests and put the land under the plough in order to be able to claim private ownership.


GATEKEEPER SERIES NO. SA38










Figure 1: Changes in Official Estimates of Cultivated Areas in Five Hill Districts
Estimate of Cultivated Area ('000 ha)


f


1978


1980


1982


1984


1986


1988


1990


Year


Doti 4- Salyan -E- Syangha - Nuwakot -- Khotang

(Drawn from Official NAC and CS data)

investigating indigenous perceptions and systems of soil fertility management, conducted a
transect study across a stretch of about 400 km of the Nepalese hills. Her findings support those
of Carson, showing that hill farmers, far from extending into marginal areas, are compelled by
increasing need for, and decreasing supplies of, organic fertilizer to pull back from such lands,
reducing cultivated areas, intensifying nutrient application on the better or more accessible lands,
and relying increasingly upon earnings from seasonal migration to make up the difference between
local food production and nutritional needs.5 Other researchers with prolonged field experience
in Nepal's hill agriculture also report extensive slack-season out-migration of men and older boys

STamang, like Carson, used participatory methods to reach her conclusions. Her findings were based on a combination
of transects and spells of residence in both accessible and remote villages, during which she conducted semi-structured
interviews during thefarmers' (including women farmers' )free time. This is usually in the evenings (when the typical survey
enumerlator would long since have departed the scene).


GATEKEEPER SERIES NO. SA38


I!
I I I I I








from hill villages, and although little hard information is available about trends, these observers
are convinced they are positive.

As far as the thesis that cultivation is expanding into increasingly marginal lands is concerned, this
migration has two important and mutually-reinforcing implications. First, the work of clearing
new land for cultivation, and/or constructing new terraces can only be done in the slack season,
yet off-season migration removes the bulk of the labour force that could otherwise be expected
to do this heavy physical work. Those left behind are the old, the sick and children, none of whom
can do such work, and the women who have a multitude of other things to do. Second, the migrants
use part of their earnings to purchase supplies of basic foodstuffs in the Tarai (where food is
relatively cheap), bring these and other goods home with them on their return. Thus, assuming that
informed observers are correct in believing that migration trends are positive, hill families are not
increasingly but decreasingly dependent upon local production for food, and so do not need to
extend their cultivated area in order to survive.

The above argument certainly does not dispute that there are severe problems with hill soils in
Nepal, but it does suggest that there is considerable confusion as to the nature of the problem, and
that many past policy initiatives based on official statistics may have been designed around an
entirely wrong set of perceptions.



Questionnaires and Data Collection

In his critique of 'survey slavery', Chambers argued that conventional socio-economic surveys
are characterized by over-long questionnaires and under-budgeted field work, under-training and
under-supervision of field staff, insufficient time for analysis and increasing pressure from
sponsors and donors for results, so that when the time comes to produce the report:

"Exhausted researchers ... stare at print-outs and tables. Under pressure for 'findings', they take
figures as facts. They have neither time nor inclination to reflect that these are aggregates of what
has emerged from fallible programming offallible punching offallible coding of responses which
are what investigators wrote down as their interpretation of their instructions as to how they were
to write down what they believed respondents said to them, which was only what respondents were
prepared to say to them in reply to the investigators' rendering of their understanding of a question
and the respondent's understanding of the way they asked it; always assuming that an interview
took place at all and that the answers were not more congenially compiled under a tree or in a
teashop or bar, without the tiresome complication of a respondent" (Chambers, 1983, p.53-4).

Other researchers have since added to the list of such statistical atrocities. I would add that in more
than twenty years experience in various parts of East Africa and South Asia in both conducting
economic research (much of it, it is freely and humbly admitted, based on questionnaire surveys),
and in administering socio-economic research award programmes, the same type of problem has
repeatedly emerged. There is no reason to believe that such problems are limited to economic
studies, but economics happens to be a field in which there is a reasonably widely-Laccepted body
of theory against which to evaluate the findings of empirical research.6 The results of such cross-
' This is an interesting, but necessary, role reversal, since it is the function ofempirical research to test theoretical constructs,
not the other way round!


GATEKEEPER SERIES NO. SA38








checking have been disappointing to say the least. Findings are often mutually inconsistent.
Coefficients that should, according to theory, be negative are positive, or negative when theory
suggests they should be positive. More frequently the same coefficient can vary between
significantly negative, through non-significant to significantly positive when there is no theoreti-
cal explanation of such differences. It would be a brave researcher indeed who would reject an
accepted body of theory on findings such as these!7

Chambers' devastating attack on 'survey slavery' is reminiscent of an earlier, and equally seminal,
assault on accepted wisdom about development, E. F. Schumacher's critique of inappropriate
technology transfer (Schumacher 1973). The "appropriate technology" debate has been around
for more than twenty years and has, at least in theory, gained widespread acceptance. The basic
premise is well-known: technologies designed in the developed countries to suit their conditions
should not be unthinkingly imported into developing countries where conditions are entirely
different. I believe that the same arguments may be applied to the questionnaire, which, like
capital-intensive methods of production, evolved in developed countries with developed country
conditions in view.

Table 2 reviews the differences between developed and developing countries with respect to the
conditions under which the questionnaire survey is designed and deployed. The identified
differences in the two "environments" are arranged in roughly ascending order of seriousness as
far as the problems of applying the questionnaire survey to natural resource management issues
are concerned. These differences and their effects are worth examining in some detail.

1. Design

The people who design questionnaire surveys in developing countries often do not have specific
training in questionnaire design, and therefore their questionnaires tend to be inefficient data-
gathering instruments. This is not an insoluble problem, as proper training of survey designers
would solve it. It is one of the few on Table 2 which is capable of relatively easy solution.



2. Language

If foreigners are involved in questionnaire design, translation is usually unavoidable. Even if only
nationals are involved, the problem remains if the sample includes more than one language group.
Not only is multi-lingualism relatively common in developing countries, but the larger the sample
and the wider its geographical coverage (and hence the more potentially useful the survey from
a policy analysis viewpoint), the more likely is this difficulty to arise. Translation is always a
potential source of misunderstanding and misinterpretation. At best -i.e. when the process is
rigorous and painstaking it imposes time delays and additional cost. At worst, when rigour is not
adhered to, it leads to differences in meaning between the various versions of the questionnaire.
A further complicating factor is that some developing country languages are non-written, so that
transliteration imposes still further complications, and these interact with translation problems.
7A colleague once informed me that in a similar situation, where the coefficients were "all over the place", he ran an analysis
of variance on a randomly selected subset of the data, using the enumerators' identification numbers as the independent
variable. He was alarmed, if not totally surprised, to find that the values of the F statistics were consistently so high as to
be "off the end of the scale"! He did not, for some reason, try to publish his findings.


GATEKEEPER SERIES NO. SA38










Table 2: Questionnaire Surveys in Developed and Developing Countries:
Divergences in Environment and Design

Developed Country Developing Country
1. Questionnaire generally designed by 1. Questionnaire often designed by persons
specialist with appropriate training with no specialist training in questiornaire
design
2. Questionnaire written in language in 2. Questionnaire normally written in another
which it will be administered language and translated, either beforehand or
during the interview
3. Respondents normally familiar with 3. Respondents unfamiliar with rationale
the general purpose of surveys behind surveys; often apprehensive as to use
of data
4. Restricted scope, simple issues 4. Complex issues; information often sensitive;
addressed, short questions; usually long questionnaires; wide scope; need for
"opinion type" surveys many "open-minded" questions
5. Built-in reliability checks 5. Often little scope to check reliability of
findings
6. Repeat surveys routine if trend 6. One-shot, cross-sectional; trend estimation
information required very difficult
7. Respondents tend either to give a 7. "Conspiracy of courtesy": tendency to give
flat refusal or else co-operate fully answers respondent thinks are wanted
8. Little if any systematic gender bias 8. Enumerators usually men; often severe
problems in interviewing women respondents

9. Literate respondents 9. Respondents either non-literate or
unrepresentative
10. Enumerators from roughly the same 10. Enumerators often from very different
socio-economic background as socio-economic background from
respondents respondents
11. Respondents can understand what 11. Non-literate respondents cannot correct
enumerator is writing; can correct any mistakes or misunderstandings
errors


3. Familiarity with Surveys

Surveys are by now well-known in developed countries and press reports make it clear that
information on even very sensitive issues like voting intentions is released only in highly
aggregated forms. This is certainly not the case in the rural areas of developing countries. The
population of such areas may have become used to (and often tired of) enumerators and their
questionnaires, but after the n'h survey they are often still no closer to knowing what it is all about
than they were when they met their first such researcher.



4. Scope

In developed countries the most successful, and the most frequent, use of questionnaires is in
'opinion' surveys, such as opinion polls and much of market research. Many people find it


GATEKEEPER SERIES NO. SA38








flattering to be asked their opinions, and so co-operation tends to be good. In developing countries
the situation can be very different. When complex issues are involved, a great deal of information
is required if the study is to be sufficiently comprehensive to be useful. Chambers (1983) has noted
the additional difficulties that arise when a multi-disciplinary team is required and all team
members have their own questions. The result is usually a long and cumbersome questionnaire,
many of whose questions are sensitive and difficult to ask. Even without the sensitivity issue, sheer
length alone will cause respondent fatigue and a tailing off of interest and therefore quality. When
deployed for this type of purpose, a data-gathering tool which already represents an inappropriate
technology is made even worse by using it for purposes far beyond its original scope.



5. Reliability Checks

The purposes for which questionnaire surveys are most commonly used in developed countries
lend themselves to automatic post-survey reliability checks. Pre-poll surveys of voting intentions,
for example, can obviously be checked against the actual election when it takes place. This has
led to successive methodological refinements that have made such surveys generally reliable
indicators. Similarly, in the case of market research, the survey findings must face the acid test
of how accurately they have predicted changes in consumer behaviour after product modification
(etc) arising from the study. This contrasts painfully with the developing country situation so
graphically depicted by Chambers (1983) in his analysis of 'survey slavery'.


6. Trends

In developed countries where trend estimates are required as in the case of opinion polls about
voting intentions a short, simple questionnaire is used and the survey repeated regularly (weekly,
fortnightly, monthly). Trends and patterns can then be computed with some degree of confidence.
With socio-economic surveys in the rural areas of developing countries, however, questionnaires
tend to be large, large-scale, expensive, and one-shot. Whatever their merits with respect to
collecting cross-sectional data, such surveys are particularly ill-suited to capturing trends.
Unfortunately, an understanding of trends is basic if the researcher is interested in sustainability
issues, since one is so often dealing with processes, like soil erosion, pasture degradation, forest
depletion, groundwater nitrification, and so forth.



7. "The Conspiracy of Courtesy"

In sharp contrast to the generally brash and direct societies of the West the rural populations of
developing countries tend often to be warm and welcoming towards strangers. The stranger is
easily looked on as a guest and the duties of a host are regarded as sacrosanct. Not understanding
the real purpose of the survey, the respondent, where not apprehensive about the use that will be
made of the information, tries to please his or her guest by giving what is assumed to be the required
answer. Very often the ill-trained enumerator makes this all too easy by prompting with suggested
answers.


GATEKEEPER SERIES NO. SA38








8. Gender Bias


Gender bias is certainly a significant socio-economic feature of industrialized society, but it does
not tend to manifest itself in forms that adversely affect the reliability of questionnaire surveys.
In developing countries, however particularly those of South Asia, where varying degrees of
female seclusion are commonplace it can be a major problem. Male enumerators are often simply
unable to interview women, while the number of potential women enumerators tends to be
relatively small. This in turn means that the findings of such surveys are unusually open to male
bias.


9. Literacy

Important as the above difficulties are, literacy is the real crux of the problem. It is a problem in
its own right and also lies at the root of the two remaining issues listed in Table 2. The literacy
barrier is the most important reason why questionnaire surveys represent an inappropriate
technology for the study of socio-economic issues in natural resource economics in developing
countries. Where the respondent is non-literate, the questionnaire itself is, and will always remain,
a barrier. The likely consequences or this barrier are described below (see "Feedback").


10. Enumerator-Respondent Differences

This is the opposite side of the "literacy" coin: while the respondent is usually not literate the
enumerator obviously has to be in order to use the questionnaire. Even if there were no class
differences to begin with, this distinction by itself, in the eyes of conventional wisdom, places the
enumerator in a higher socio-economic class than that of most respondents, with all the
communication barriers that this can entail.


11. Feedback

If, for a moment, we try to put ourselves in the shoes of the respondent (although, of course, many
of them will have none), we can perhaps see one of the major drawbacks of the questionnaire
survey. A stranger arrives in the village and asks if he can ask a few questions. The respondent,
out of courtesy, agrees to be interviewed. Little effort is made to develop rapport or even to explain
fully the purpose of the exercise (the enumerator has a tough quota to fulfil, but the respondent
does not know that). The stranger then produces a little board, and clipped to it, a wad of paper
covered in what to the respondent are unintelligible hieroglyphics. He then proceeds to ask
questions and write down the answers more hieroglyphics. The respondent has no idea of what
is being written down, whether his or her words have been understood or interpreted correctly. The
enumerator, being simply a data-gatherer, has no way of knowing and no responsibility to know
- whether the answers being given are correct or whether they make sense within the broader
framework of the survey. The interview complete, the enumerator departs and is probably never
seen again.


GATEKEEPER SERIES NO. SA38








Before leaving this aspect of the topic, one important qualification should be added. The above
problems show a powerful tendency to intensify with scale. At one extreme, the post-graduate
student, for example, having a high professional stake in the reliability of the study, having
personally designed the questionnaire, trained the enumerators and closely supervised their work
in the field, is likely to be able to compensate for many of the inherent defects of questionnaires
listed above. The same can be said of many well-supervised micro-level studies. At the other
extreme, however, where the survey is large-scale and multi-disciplinary, where there is a lengthy
'chain of command' between the person who conceived the survey in the first place and the
respondent at the other end, the drawbacks of this technique are likely to manifest themselves in
their most devastating form. Unfortunately policy analysis, particularly in the field of natural
resource management, needs macro- much more than micro-level data.



Questionnaires and Data Analysis

The above examination leads inevitably to the conclusion that questionnaire-based methods
represent a barrier to communication between the literate enumerator and the non-literate
respondent. These methods play to the former's strengths and the latter's weaknesses, and in so
doing frustrate the basic purpose of the exercise by generating 'data' that are frequently inaccurate
and misleading. This is extremely ironic, for, from the standpoint of the research team, a perceived
strength of the questionnaire is that its output is highly amenable to the very powerful and rigorous
techniques of modern statistical analysis. The questionnaire survey facilitates this by: (a) focusing
on quantitative (at the expense of qualitative) measurement, (b) generating a relatively large
number of individual interviews (in an effort to capture variation), and (c) aiming at standardi-
zation of questions across the entire sample of respondents, which in turn leads to concentration
on 'closed-ended' questions. Unfortunately, the undoubted rigour and accuracy of modem
statistical analysis is nullified by the above-mentioned drawbacks of the questionnaire as a data-
gathering instrument in the environment under examination. There is no way in which rigour in
analysis can compensate for an unknown and unknowable degree of inaccuracy in the measure-
ment of independent variables. Modem statistical analysis can handle sampling errors, but non-
sampling errors of a type arising either from the inherent difficulties summarized on the right hand
side of Table 2, or in the operational environment described by Chambers (quoted above), give
a totally different picture. Here the well-known computer scientists' aphorism applies: "Garbage
in, garbage out".



Missing the Boat

What has so far been said about questionnaires relates to their inability, in the circumstances just
described, to do what they are designed to do. The other side of the coin covers all the useful things
which could be done using more appropriate methods, but which cannot be confined within the
straightjacket of a questionnaire. Just as this instrument plays to the (largely illusory) strengths
of the researcher, it also plays to the weaknesses of the rural community. Among the latter's
strengths, the following are particularly relevant to an understanding of issues surrounding the
sustainable use of natural resources.


GATEKEEPER SERIES NO. SA38








1. Knowledge


Certainly during the first half of this century, but to a diminishing degree ever since, science and
technology have been viewed, even venerated, as the great shining hope of the human race.
Traditional outlooks were correspondingly dismissed as 'unscientific' and therefore irrelevant to
the modern world. The sense of disillusionment that has subsequently set in has nowhere been more
widespread than in the domain of the earth's environment and the sustainable use of its natural
resource base. Increasing perception of the unwelcome side-effects of economic development -
in such forms as air, soil and water pollution by industrial effluent, the non-sustainable mining of
natural resources to provide industrial raw materials, acid rain, ozone layer depletion, and possible
global warming has seen the blame for environmental degradation come increasingly to be laid
at the door of modern science- and technology-based industrialization.8

Alongside the development of this healthy scepticism probably as a corollary of it there has
grown an increasing awareness that 'pre-scientific man' is perhaps not quite the country bumpkin
we once took him (or her) for. Even representatives of the 'hard' sciences even theoretical
physicists are now beginning to query the philosophical underpinnings of classical science and,
belatedly, to see the wisdom in ancient sets of values and practices once dismissed as outdated and
irrelevant.9

Increasingly numerous studies of indigenous technical knowledge, not least in the sphere of
sustainable natural resource management, are bringing us towards a belated realization that many
unschooled but far from uneducated rural people possess an invaluable fund of knowledge about
the environments in which they live, and about the management of natural resources on which their
livelihoods depend."' If they appear to abuse these resources, it is probably poverty rather than
ignorance, that drives them to it.

This fund of knowledge, and the resource management systems, modes and mores built upon it,
represent invaluable sources of enlightenment for outsiders interested in learning about sustain-
able methods of natural resource management if only we can learn how to tap it. The questionnaire
survey is decidedly not the way to do so.

The questionnaire designer must (the ritual of protesting notwithstanding) determine in advance
what questions will, and, by default, which ones will not, be included in the printed form.
Unfortunately those who design these blunt instruments, themselves outsiders, do not normally
know in advance all of the questions that should be asked and even if they did, questions relevant
to one community or one farmer might be quite irrelevant to another. The questionnaire also
eliminates the possibility of capturing the unique and spontaneous insights which a single
informant or group of informants might offer. Perhaps the most serious limitation of the

" See especially Conway and Pretty (1991).
SCapra, himself a prominent theoretical physicist, has provided a stunning critique of the mechanistic world-view of
classical science. He notes how twentieth century physics, resting on its two great theoretical pillars, quantum theory (which
explores the sub-atomic world) and relativity theory (the realm of speeds approaching that of light) "now overcomes this
fragmentation and leads us back to the ideal of unity expressed in the early Greek and Eastern philosophies" (1984, p.0).
'"See,for example: Harwood (1979); Farrington and Martin (1988); Chambers et al. (1989); also the various publications
of the Center for Indigenous Knowledge in Agriculture and Rural Development (CIKARD) at Iowa Stcte University.
CIKARD's Indigenous Knowledge Documentation Unit has accessed hundreds of published and unpublished reports foom
individuals and institutions worldwide.


GATEKEEPER SERIES NO. SA38








questionnaire as an instrument for tapping into indigenous knowledge is that it concentrates on
What?" at the expense of Why?l2. The underlying implication of this approach is that if the
researcher knows What, then Why is either irrelevant, or can be deduced with perfect confidence
from answers to the What questions.

This is particularly true of questionnaires dominated by 'closed-ended' questions (the most
familiar type in economic research, especially large-sample macro-economic research). Typi-
cally, the only follow-up allowed is limited to such crude formats as: "If the answer to Question
6 is 'YES', ask Questions 7 to 11; otherwise go to Question 12." But since all of these questions
have had to be decided in advance, the type and level of knowledge that can -even assuming
accurate answers be tapped by them is both pre-determined and severely restricted. Even with
'open ended' questions even with those that ask 'why?', the answers are not normally allowed
to generate follow-up questions, i.e. new questions shaped by the answers to previous ones. If they
do, we are moving away from the questionnaire approach towards the 'semi-structured' interview,
a technique that is characteristic of participatory, rather than top-down, methods of attempting to
learn about and therefore from rural people.



2. Memory

There is an important price usually unrecognized and almost never acknowledged that those
of us who have been to school have had to pay for our education. That price is memory. Our literacy
enables us to write things down and look things up, and as a direct result our memories, like any
other underused faculty, tend to atrophy. Most of us would regard this as a small price to pay for
access to the vast fund of knowledge that exists in recorded sources as indeed it is but it is a
price nevertheless, and those who have not paid it have at least something to set on the credit side
of the ledger. Literate outsiders, not appreciating this, are often astounded by the degree of
accuracy and comprehensiveness with which non-literate people can remember things. Where
there are no written records of events and processes, this 'memory bank', like the 'knowledge
bank' mentioned earlier is invaluable. It is particularly important when we wish to examine trends.
But again, access to this memory bank is dependent upon the outsider's possessing the willingness,
ability, creativity, humility and sensitivity required to tap it. What price the questionnaire in these
circumstances?



3. Courtesy

This quality of many Third World rural societies was mentioned earlier as a factor that

" This is especially true of those that attempt to quantif'. Some typical examples: What is your landholding? How many
family members? How many goats? What is your level offornmal education ? How mnuclh of your land is irrigated? How many a
trees do you own? Moan such questions seek highly sensitive information and invite "second guessing" on the part of the
respondent and correspondingly evasive or misleading responses.
( (ie. those seeking explanations ofobserved or reported facts. processes, etc). For example: "Would you please explain
why you plough in that direction?" or "I don't understand this relationship, could you please explain further?" or "IflI
understood you correctly ...", etc. Such questions are based on the presupposition that if the respondent does something
then there is probably a good reason for it, and therefore good reason to try to understand this rationale. Obviously it would
be extremely difficult to confine such questions within the straightjacket of a normal eiumerator-administered question-
naire.


GATEKEEPER SERIES NO. SA38








paradoxically makes for inaccuracy and distortion in conventional data collection. The reverse
side of the coin is, of course, that this willingness to share information with outsiders can be made
productive instead of counter-productive if certain conditions are met. Above all, courtesy must
be met with courtesy and respect with respect. To say this is not to argue that the researcher should
abandon normal, healthy scientific scepticism when interviewing villagers. With the best will in
the world, distortions can creep in, for whatever reason, when language is used to communicate
ideas. But sensitivity, politeness, even humility, are essential when giving expression to such
scepticism.

Finally I would like to anticipate the possible protest that, in listing the above features of Third
World rural communities as strengths, I am in effect painting an idealized or romanticized picture.
Far from it. Such people are no better and no worse than the rest of us; they have their strengths
and their weaknesses, their saints and sinners, their geniuses and their dullards, just like any other
cross-section of humanity. However, they do also have the singular advantage of living in a
particular place (as often their families have lived for generations), earning a living from a
frequently hostile and unforgiving environment, faced with the ever-present prospect of paying
the price of failure in a way that few of us with formal qualifications and professional salaries are
ever called upon to do.



References

Balogun, P. K. (1989): A Review of Published Agricultural Statistics in the Western Development
Region of Nepal; Lumle Agricultural Centre, Kaski District, Nepal.

Capra, F. (1984): The Tao of Physics: An Exploration of the Parallels Between Modern Physics
and Eastern Mysticism (2nd Ed.); Bantam-New Age Books, New York.

Carson, B. (1992): The Land, the Farmer and the Future: A Soil Fertility Managenent Strategy
for Nepal; International Centre for Integrated Mountain Development (ICIMOD) Occasional
Paper No.21, Kathmandu.

Chambers, R. (1983): Rural Development: Putting the Last First; Longman Scientific and
Technical, London.

Chambers, R., A. Pacey and L.A. Thrupp (editors) (1989): Farmer First: Farmer Innovation and
Agricultural Research; Intermediate Technology Publications, London.

Conway, G.R., and J.N. Pretty (1991): Unwelcome Harvest: Agriculture and Pollution; Earthscan
Publications Ltd, London.

Farrington, J. and A. Martin (1988): Farmer Participation in Agricultural Research: A Review of
Concepts and Practices; Overseas Development Institute, London.

Gill, Gerard J. (1991): Seasonality and Agriculture in the Developing World: A Problem of the
Poor and Powerless; Cambridge University Press, Cambridge, England.


GATEKEEPER SERIES NO. SA38








Harwood, R. R. (1979): Small Farmer Development: Understanding and Improving Farming
Systems in the Humid Tropics; Westview Press, Boulder, Colorado, USA.

NARC-ADB (1991): Nepal Agricultural Research Study (A Study Team Report prepared for the
National Agricultural Research Council and the Asian Development Bank), Kathmandu.

Schumacher, E. F. (1973): Small is Beautiful: A Study of Economics as if People Mattered; Blond
and Briggs Ltd., London.

Tamang, D. (1992): Indigenous Soil Fertility Management in the Hills of Nepal: Lessons from an
East-West Transect; Research Report Paper No. 19, Winrock International, Kathmandu.


GATEKEEPER SERIES NO. SA38









GATEKEEPER SERIES


1. Pesticide Hazards in the Third World: New Evidence from the Philippines. 1987. J.A. McCracken and G.R.
Conway.

2. Cash Crops, Food Crops and Agricultural Sustainability. 1987. E.B. Barbier.

3. Trees as Savings and Security for the Rural Poor. 1993. Robert Chambers, Czech Conroy and Melissa Leach.
(1st edition, 1988)

4. Cancer Risk and Nitrogen Fertilisers: Evidence from Developing Countries. 1988. J.N. Pretty and G.R.
Conway.

5. The Blue-Baby Syndrome and Nitrogen Fertilisers: A High Risk in the Tropics? 1988. J.N. Pretty and G.R.
Conway.

6. Glossary of Selected Terms in Sustainable Agriculture. 1988. J.A. McCracken and J.N. Pretty.

7. Glossary of Selected Terms in Sustainable Economic Development. 1988. E.B. Barbier and J.A. McCracken.

8. Internal Resources for Sustainable Agriculture. 1988. C.A. Francis.

9. Wildlife Working for Sustainable Development. 1988. B. Dalal-Clayton.

10. Indigenous Knowledge for Sustainable Agriculture and Rural Development. 1988. D.M. Warren and K. Cashman.

11. Agriculture as a Global Polluter. 1989. Jules N. Pretty and G.R. Conway.

12. Evolution of Agricultural Research and Development Since 1950: Toward an Integrated Framework. 1989. Robert
E. Rhoades.

13. Crop-Livestock Interactions for Sustainable Agriculture. 1989. Wolfgang Bayer and Ann Waters-Bayer

14. Perspectives in Soil Erosion in Africa: Whose Problem? 1989. M. Fones-Sondell.

15. Sustainability in Agricultural Development Programmes: The Approach of USAID. 1989. Robert O. Blake.

16. Participation by Farmers, Researchers and Extension Workers in Soil Conservation. 1989. Sam Fujisaka.

17. Development Assistance and the Environment: Translating Intentions into Practice. 1989. Marianne Wenning.

18. Energy for Livelihoods: Putting People Back into Africa's Woodfuel Crisis. 1989. Robin Mearns and
Gerald Leach.

19. Crop Variety Mixtures in Marginal Environments. 1990. Janice Jiggins

20. Displaced Pastoralists and Transferred Wheat Technology in Tanzania. 1990. Charles Lane and Jules N. Pretty.

21. Teaching Threatens Sustainable Agriculture. 1990. Raymond I. Ison.

22. Microenvironments Unobserved. 1990. Robert Chambers.

23. Low Input Soil Restoration in Honduras: the Cantarranas Farmer-to-Farmer Extension Programme. 1990.
Roland Bunch.

24. Rural Common Property Resources: A Growing Crisis. 1991. N.S. Jodha

25. Participatory Education and Grassroots Development: The Case of Rural Appalachia. 1991.
John Gaventa and Helen Lewis

26. Farmer Organisations in Ecuador: Contributions to Farmer First Research and Development. 1991. A. Bebbington


GATEKEEPER SERIES NO. SA 38









27. Indigenous Soil and Water Conservation in Africa. 1991. Chris Reij

28. Tree Products in Agroecosystems: Economic and Policy Issues. 1991. J.E.M. Arnold

29. Designing Integrated Pest Management for Sustainable and Productive Futures. 1991. Michel P. Pimbert

30. Plants, Genes and People: Improving the Relevance of Plant Breeding. 1991. Angelique Hangerud and Michael
P. Collinson.

31. Local Institutions and Participation for Sustainable Development. 1992. Norman Uphoff.

32. The Information Drain: Obstacles to Research in Africa. 1992. Mamman Aminu Ibrahim.

33. Local Agro-Processing with Sustainable Technology: Sunflowerseed Oil in Tanzania. 1992. Eric Hyman.

34. Indigenous Soil and Water Conservation in India's Semi-Arid Tropics. 1992. John Kerr and N.K. Sanghi.

35. Prioritizing Institutional Development: A New Role for NGO Centres for Study and Development. 1992. Alan
Fowler.

36. Communities as Resource Management Institutions. 1993. Marshall W. Murphree.

37. Livestock, Nutrient Cycling and Sustainable Agriculture in the West African Sahel. 1993. J.M. Powell and T.O.
Williams.

38. O.K., the Data's Lousy, But It's All We've Got (Being a Critique of Conventional Methods). 1993. Gerard G.
Gill.

39. Homegarden Systems: Agricultural Characteristics and Challenges. 1993. Inge D. Hoogerbrugge and Louise
O. Fresco.

40. Opportunities for Expanding Water Harvesting in Sub-Saharan Africa: The Case of the Teras of Kassala. 1993.
Johan A. Van Dijk and Mohamed Hassan Ahmed.

41. Living in a Fragile Ecosystem: Indigenous Soil Management in the Hills of Nepal. 1993. Devika Tamang.




Copies of these papers are available from the Sustainable Agriculture Programme, IIED, London
(2.50 each inc. p and p).


GATEKEEPER SERIES NO. SA 38







The Sustainable Agriculture Programme


IIED


The Sustainable Agriculture Programme of IIED promotes
and supports the development of socially and environ-
mentally aware agriculture through research, training,
advocacy, networking and information dissemination.

The Programme emphasises close collaboration and con-
sultation with a wide range of institutions in the South.
Collaborative research projects are aimed at identifying
the constraints and potentials of the livelihood strategies
of the Third World poor who are affected by ecological,
economic and social change. These initiatives focus on
indigenous knowledge and resource management; par-
ticipatory planning and development; and agroecology
and resource conserving agriculture.

The refinement and application of Participatory Rural
Appraisal methods is an area of special emphasis. The
Programme is a leader in the training of individuals from
government and non-government organizations in the
application of these methods.

The Programme supports the exchange of field experi-
ences and research through a range of formal and informal
publications, including RRA Notes, aimed at practitioners
of Rapid and Participatory Rural Appraisal, and the Gate-
keeper Series, briefing papers aimed at policy makers. It
receives funding from the Swedish International Develop-
ment Authority, the Ford Foundation, and other diverse
sources




International Institute for
Environment and Development
3 Endsleigh Street,
London WC1H ODD, UK

Telephone: 071-388 2117
Fax: 071-388 2826
Telex: 317210 BUREAU G




University of Florida Home Page
© 2004 - 2010 University of Florida George A. Smathers Libraries.
All rights reserved.

Acceptable Use, Copyright, and Disclaimer Statement
Last updated October 10, 2010 - - mvs