A systematic tool for FSR/E project evaluation

Material Information

A systematic tool for FSR/E project evaluation
Series Title:
A systematic tool for FSR/E project evaluation
Zimet, David J.
Publication Date:

Record Information

Source Institution:
University of Florida
Holding Location:
University of Florida
Rights Management:
The University of Florida George A. Smathers Libraries respect the intellectual property rights of others and do not claim any copyright interest in this item. This item may be protected by copyright but is made available here under a claim of fair use (17 U.S.C. §107) for non-profit research and educational purposes. Users of this work have responsibility for determining copyright status prior to reusing, publishing or reproducing this item for purposes other than what is allowed by fair use or other copyright exemptions. Any reuse of this item in excess of fair use or other copyright exemptions requires permission of the copyright holder. The Smathers Libraries would like to learn more about this item and invite individuals or organizations to contact Digital Services ( with any additional information they can provide.
Resource Identifier:
183689542 ( OCLC )


This item has the following downloads:

Full Text

(Edited and final November 8, 1988)



David Zimet, Edwin C. French and Chris O. Andrewl


A strength of FSR/E methodology is that it encourages unique design of each project by

accounting for the environment (bio-physical and socio-economic variables) of the project area.

The uniqueness of the project, however, makes comparisons and, ultimately, evaluation

difficult. Technical recommendations are not usually repeated from one project to the next,

however basic FSR/E methodology is applied repeatedly. Analysis of project execution through

a systematic rating system, based on the FSR/E methodological phases can provide a means for

improving our understanding and implementation of the methodology to make it a more

efficient and effective process.

An evaluation team fielded by the Farming System Support Project was asked to evaluate

the methodology used in 13 FSR/E projects in Central America. All of the projects employed

a common process in application of a FSR/E methodology comprised of seven steps starting

with site selection and ending with institutional follow-up. Each step, except site selection,

was evaluated for each project, providing project composites and overall methodological

analyses. This evaluation approach proved a systematic tool which partitioned a complex

methodology into components, provided a way to focus and summarize much of the thought

process used during the evaluation, elicited a good deal of constructive comment from involved

'Assistant Professor Food and Resource Economics, Associate Professor Agronomy and
Professor Food and Resource Economics respectively, University of Florida.


. I

institutions, helped to pinpoint relative strengths and weaknesses in each of the 13 projects

and provided a valuable case example for other FSR/E project evaluations.

The Setting

In 1985, the Farming Systems Support Project (FSSP) evaluated the ROCAP-funded Small

Farm Production Systems Project (henceforth referred to as the Project) implemented by

CATIE.2 The FSSP assembled a team of two agronomists (E.C. French III and F. Poey), an

animal scientist (J. Conrad) and an agricultural economist (D. Zimet) who also served as team

leader to perform the evaluation. The team visited the Central American field sites of the

Project, the offices of CATIE and the offices of participating and interested institutions. Of

the Central American countries that were included in the Project only Nicaragua was excluded

from the evaluation.

This paper reports on the evaluation scheme used by the team to rate the methodology

used to implement the Project. The evaluation scheme provided an organized format for

comparing the thirteen FSR/E subprojects and facilitated the process of highlighting their

strengths and weaknesses. The evaluation scheme suggests some criteria and a rating system

which may have value for similar activities in other settings where multiproject comparison is


FSR/E Methodology

Farming systems research and extension (FSR/E) can be viewed as a process as well as a

methodology. The process entails a number of different steps or tasks that must take place

simultaneously and/or sequentially if a successful FSR/E project is implemented. Each phase

2FSSP was funded by the U.S. Agency for International Development and implemented by
University of Florida and 21 other universities and 4 consulting firms. ROCAP is Regional
Office for Central America and Panama of USAID. CATIE is Centro Agronomico Tropical de
Investigation y Ensenanza.

influences the outcome of the project and can be analyzed or evaluated independently as well

as interactively with the other phases.

In general, the phases of an FSR/E project proceed from site selection, to

characterization of the site and problem diagnosis, to design alternatives (based on the

permutations of bio-physical and socio-economic factors supported where possible with

experiment station findings), to on-farm testing (blending farmer/farm environment data with

experiment station research when needed), to validation (with more on-farm testing) and

transfer, and finally to institutional follow-up (e.g. extension and farmer loan backstopping).

A matrix of the rating process is presented by country in Table 1. In all cases site selection

was performed by the host countries, not CATIE, so this phase is not discussed. Institutional

follow-up is included and it is critical to the success of an FSR/E endeavor. Success of this

FSR/E experience depended upon involvement of the host country institution in the Project as

well as the quality of performance by the Project itself.

Project Methodology

The Small Farm Production Systems Project was a pioneer FSR/E effort. Through it and

its training component, CATIE played a key role in the development and dissemination of

FSR/E ideas in Central America. A critical aspect of the Project involved the development of

a methodology appropriate to a farming systems mode of research. Given that the Project was

a complex, multi-objective agricultural research effort operating through CATIE in the five

Central American countries and Panama, a complex multifaceted methodology resulted.

The methodology proposed by CATIE for development of technological alternatives in

specific areas was obtained by recursive experience in conjunction with national institutions in

the region and the experience of these institutions working with small farmers. The

conceptualization and structure imposed by the methodology provided for a synthesis of

investigative work done on farms. Where institutional memory provided continuity and where

capable individuals existed, project experiences were capitalized upon and a dynamic

technology generation process was observed. The methodology used for project implementation

was broadly structured to facilitate adaptation within the various ecological zones and to

conform with available resources of the national institutions and socioeconomic conditions in

the area of influence. National institutions, which in turn work toward the benefit of small

producers were to be the final users of the methodology.

Area selection, farmer/environment characterization are important to identification of

production constraints and producer problems. The process of designing alternatives, fielding

on-farm research and validating results is essential for development of viable technological

alternatives that will help resolve producer problems in a manner compatible with prevailing

circumstances. By focusing station research on specific questions, modification of technology,

when needed, can result in its expanded use in other areas.

Development of the methodology used in Project implementation occurred relatively early

in the life of the project. The evaluation team recognized the lag time required to

operationalize the newly developed methodology, considering it was put in place over all of

Central America. However, application of the methodology by CATIE varied, in several cases,

from that used widely by farming systems practitioners since 1981. In part this variation was

based on a difference in conceptual definition.

The Rating System

Important to the evaluation process was the development, by the evaluation team, of a

systematic means of comparing one country's subprojects to those of another. Comparisons

were made simply by assigning a numbered rating to each FSR/E methodological phase for

each project in each country. A four point scale was employed to rate each phase. Due to

team size (four team members) and a desire to avoid unnecessary deliberation that might have

resulted from a more precise scale, the scaling was kept simple. The four points utilized


1 = the step was not carried out;

2 = the step was performed poorly or in an incomplete fashion;

3 = the step was performed well, but was done to excess or required too much
time or too many resources;

4 = the step was done well.

Rating 3 was incorporated because it was believed that inefficient or ineffective resource use

historically had been a global problem in early "farming systems research" and that it is

incumbent upon FSR/E projects, both philosophically and conceptually, to use resources

efficiently. Table 1 exhibits the ratings for all thirteen subprojects for each FSR/E phase

with the exception of site selection. Site selection was excluded because it was performed

independent of the project staff.

The Evaluation

The evaluation was based on rating scores from each methodological phase. Each phase

in the FSR/E methodology is presented because of its importance relative to the other phases

and success of the overall FSR/E effort. As illustrated in Table 1 characterization, no matter

how performed (by rapid recognizance or by in-depth sample surveys), influences the type of

technology designed. Because of the interrelationship between preceding and succeeding

FSR/E phases, each should be evaluated within the context of the entire FSR/E methodology

application process.


Characterization is defined as the activity of describing bio-physical and socio-economic

variables which should include identification of the farmer/farm family production practices

found in the project area and specification of production and marketing problems and their


sources. This phase received a great deal of effort by the Project. If evaluated alone all of

the characterizations performed by individual projects would have received excellent scores

because considerable relevant information was collected and analyzed. Unfortunately, however,

much of the corollary information collected at each subproject site was not used in the

subsequent phase -- design of alternatives. Even with an information surplus, resulting from

over zealous surveying, it was necessary to collect additional information after the design

process was underway. Examining the characterization information gathered, the team believed

that all the characterization phase for subprojects warranted a "3" rating.

Design of Alternatives

Design of appropriate technology based alternatives depends upon the quality of the

characterization effort as well as the interpretation of the collected information. The design

of alternative technologies for all thirteen subprojects was facilitated by adequate level of

characterization data. This is reflected in the scoring. Only the swine and feed subproject in

Costa Rica received a design score other than "4". All the other designs could have lead to

meaningful farm trials and/or station research. In order to explain how the rating system

operated for this phase, two subprojects implemented in Costa Rica, the swine and feed

subproject and the maize subproject, are discussed. The maize subproject is presented first.

The search for alternative production methods of maize followed by other crops began

with a predecessor project and continued into the first half (until 1983) of the Small Farm

Production Systems Project. The search considered market conditions and producer

preferences. For example, cassava was dropped when the research people realized that

producers did not like the new variety of cassava being tested. In addition, the market price

for cassava declined, making cassava a less attractive production alternative.

Maize production alternatives concentrated on a combination of relatively slight changes

in cultural practices. Major changes were avoided. Alternative technologies included plant

spacing, fertilizer analysis and pesticides, but neither the use of entirely new inputs nor

entirely new cultural practices were proposed. The design phase for the maize subproject in

Costa Rica was given a "4" because it was based upon needs that were expressed in the

characterization, it developed out of sound previous experience but did not terminate with that

experience, and the design effort was sensitive to the needs of producers and the market.

The swine and feed subproject also originated from the needs expressed during the

characterization phase. Unlike the maize subproject, however, the swine and feed subproject

could not draw upon a store of previous information for the components critical to the

subproject. A complete management system could not be designed. A series of nutritional

experiments, were designed, however disregard for inherent limiting factors obtained from the

characterization phase resulted in a pursuit of inappropriate technology. Consequently, the

design phase of the Costa Rica swine and feed subproject scored a "2".

Back-up Experiment Station Research

Back-up research is intended to help solve problems that come to light either during the

characterization phase or during the on-farm testing phase. Station research is especially

useful if conducted with an applied end and is interactive with on-farm research. Of the

thirteen subprojects, the team assigned a well done rating ("4") to six subprojects for use of

back-up research. The two subprojects to be discussed are the Costa Rica maize subproject

with a well done rating and the Honduras dual-purpose cattle subproject which scored a low

"1" for not being carried out.

The support research conducted in Costa Rica concentrated on new maize varieties,

planting density and plant nutrition. Laboratory tests showed nitrogen to be the limiting

nutritional element in maize production. This was a critical finding because a maize-maize

rotation was one of the alternatives being recommended. Various fertilizer mixes and timing

of applications were tested on the research station. Research efforts focused on maintaining

fertilizer costs at parity with those identified during the characterization phase. The

fertilizer combinations that proved to be most successful from varietal and spacing experiments

were then examined further through on-farm trials.

In contrast there was virtually no back-up research conducted in support of the

Honduran cattle subproject. Instead, an attempt was made to move in toto the dairy

production module developed in Turrialba, Costa Rica, to Honduras and to adapt that module in

the field for dual purpose cattle. No producer saw fit to adopt the module in its entirety,

which may be the result of component research being performed. Thus, the team decided

that, in effect, no support research was performed and scored this phase of the Honduran

subproject as a "1".

On-Farm Testing

On-farm testing examines the appropriateness of technology and farmer managed

conditions rather than those of the experiment station. The production alternative is either

shown to need further modifications at the station level, minor modifications that can be

developed on-farm, or almost no modifications.

On-farm trials for the Costa Rican maize subproject started in 1980 and ended in 1983.

These trials began relatively early in the life of the subproject because the maize subproject

was a continuation of a previous project which had sponsored some on-farm trials, but

concentrated on station trials. Information from those previous station trials and the

concurrent station trials was adopted to early work of the maize subproject and extended over

successive years. The number of years and the total of forty-eight on-farm trials contributed

to the high rating ("4") received by the on-farm trial phase of the Costa Rica maize


The milk production subproject in El Salvador was less successful in the on-farm trial

phase. Only three farms for each of three modules, or a total of nine farms, were used for

on-farm trials. There were no station trials conducted specifically to support the farm trials.

In addition, there was almost no previous (or concurrent) station research upon which to base

the farm trials. Because of this void, the farm trials were, in essence, used to test

theoretical hypotheses relating to animal feeding and nutrition. Interaction between field staff

and producers was important. The on-farm trial phase of the Salvadorean milk subproject

rated a "2".

Validation (and Transfer)

Successful validation trials form the final FSR/E phase before the alternative is put into

the extension pipeline for transfer. If the trials are unsuccessful, indicated by producer

nonacceptance, the alternative should be modified with information from new experiment

station and/or on-farm trials.

The maize-maize rotation with plant density, herbicide, and fertilization recommendations

was extensively and intensively tested in validation trials. The validation trials started in

1982 and ended in 1984. There were two years of overlap with the experimental on-farm

trials during which period ninety-six validation trials were conducted. In 1984, a year with no

such overlap, thirty-six validation trials were conducted. Because of this overall effort the

team observed that at least 75% of the maize producers in the subproject region had adopted

the technical package. The maize subproject in Costa Rica scored a "4" in the validation and

transfer stage.

In Panama there were two rice subprojects. The separate locations were managed and

executed by separate field staff. Similar to the Costa Rican maize subproject, the Panama

rice subprojects benefitted from their predecessor projects. On-farm experimental trials were

executed successfully at both Panamanian locations. The locations were also similar in

implementation of on-farm validation trials which were managed by the technicians assigned to

each subproject, not the producers. Thus the eight validation trials run in Progreso and the

three in Guarumal served little purpose. This was reflected in responses by the producers who

"participated" in the validation trials but were not given management responsibility. They

were unsure as to what was done in the trials. Because there were relatively few validation

trials and because the producers did not participate to the point of understanding the

alternatives under consideration, both of the Panamian rice subprojects received a "2".

Institutional Follow-Up

The true outcome of an FSR/E (or any) project can only be determined after the direct

or specific project activities end. The continued participant use of the techniques developed

under the project and the wide-spread adoption by non-participants because the techniques

contribute to their well-being should be the critical criteria of success or failure. Often these

criteria cannot be met successfully without institutional support services such as credit,

extension and input availability. Thus, institutional follow-up can be very important to the

success or failure of a project. If a project has performed poorly, institutional follow-up

probably will not save it and in fact can be disastrous. Similarly, if commercial inputs are

available for an excellent project including well adapted alternative technologies, the effort

could succeed in the absence of follow-up by public institutions. However, support must

follow through from some institutional source or potentially good projects usually can not


The two Panamian rice subprojects received identical ratings until the institutional

follow-up stage. Because capable national field staff were available and because of

coordination with the agricultural bank, the subproject in Progreso scored a "4" for

institutional follow-up. In Guarumal, producers were ignorant of the fundamental aspects of

the recommendations and the remaining national field staff was weak. To exacerbate matters,

the research institution, extension agency and the agricultural bank were making different

recommendations regarding cultural practices and fertilization. Only the research institution,

because of its participation in the subproject, was making recommendations based upon the

subproject. Because the follow-up was of poor quality the Guarumal subproject scored a "2".

Projected Impacts

The evaluation team visited farm sites and spoke with institutional participants as the

subprojects were terminating. Given this situation, only predictions/extrapolations of

subproject impacts could be made. These predictions are presented in Table 2.

A comparison of Tables 1 and 2 reveals one case, the dual purpose cattle subproject in

Costa Rica, in which a poorly rated subproject had a reasonably good prognosis for success.

Such a positive prognosis was based on the fact that the participating producers also acted as

extension agents and the high level of sophistication of Costa Rican producers. Greater

success could have been projected had the project participants worked toward bringing

recommendations from the subproject in line with the farmer resource base. The other

subprojects which had great potential (a "3" in Table 2) all received excellent institutional

follow-up ratings.

The Costa Rica maize-maize subproject was the only one that scored a "2" in

institutional follow-up and yet received an excellent prognosis. It was the only subproject

which the team believed had already attained the ultimate goal of successful dissemination

which resulted from wide-spread on-farm experimental and validation trials.

Concluding Remarks

Cross project evaluation on a comparable and relatively objective scale provides an

opportunity to not only evaluate and score FSR/E projects on a relative basis, but also

facilitates the transfer of process oriented knowledge to improve the FSR/E methodology. Our

ability to absorb, comprehend and evaluate a range of project interventions involving diverse

enterprises in multi-national locations is mitigated without some common denominator which

can reduce a complex series of events into simple terms. Considering the site specific nature

of technology and methodology, this is particularly true.

This pragmatic attempt to systematize a simplified evaluation tool suggests neither that

the criteria nor the scale used in scoring necessarily would have broad use. The tool itself

should be adapted to specific evaluation needs and validated for application. A simple process

oriented tool, however, we believe will both improve evaluation efficiency and effectiveness.

As in any experience we have encountered limitations of the tool used for evaluating the

CATIE project. Our primary suggestion is that one should never take the results of an

analytical framework imposed on institutional processes as the final word. Good FSR/E

practitioners will apply subjective analysis to the case at hand in both adapting the tool and

considering its results.

Table 1. Evaluation Matrix.

FSR/E Methodological Phases
Charac. Design Back-up On-farm Validation Institutional
Subprojects of Exp. Station Testing and Follow-up
by Country Alter. Research Transfer


Dual Purpose 4 2 2 1 1
Swine and Feed 2 2 2 1 2
Maize-Maize 4 4 4 4 2

Milk 4 2 2 1 1
Maize-sorghum 4 4 4 2 4

Dual purpose 4 1 2 1 2
Rice 4 4 4 2 4
Maize-sorghum 4 4 4 2 4

Dual purpose 4 4 4 1 4
Vegetables 4 4 4 1 1

Rice (2 projects)a 4,4 2,2 4,4 2,2 4,2
Dual purpose 4 2 2 1 2

1. Not carried-out

2. Poor or scanty

3. Done to excess

4. Well done

aThe first value refers to the project Progreso, the second to the one in Guarumal.

Table 2. Impact projections of CATIE/ROCAP country projects based on their present status.
August, 1985.
Dual Maize- Maize- and
Purpose Sorghum Maize Rice Milk Feed Vegetables

Costa Rica 3 4 1

El Salvador 2 2

Honduras 1 2 3

Guatemala 3 1

Panama 2 3,1b

al = Little or no impact; 2 = Technology developed is adequate but has little potential;
3 = Technology developed is appropriate and has great potential; and 4 = Technology
developed appropriate and is moving out to farmers.

bThe values represent the situations at Progreso and Guarumal, respectively.