Impact evaluation has been the inevitable part of any socio- economic development literature since the middle of the 20th century. As the millennium development goals (MDGs) of the UN came into being, monitoring and evaluation (M&E) has become the fad of many an organisation. In fact good governance pre-supposes flaw less project implementation vis a vis ex- ante assumptions.
Now the question to be addressed is whether impact evaluations are carried out for a formality or are they done from the point o f view of learning from the mistakes, so that better project formulations are made possible in future. There are several types of evaluations. Some are known as impact evaluations. While a few others are known as concurrent evaluations, and yet others are known as end of the project evaluations, terminal evaluations, etc. Nonetheless, a terminology which has gained currency of late is M&E. It has become an integral jargon in any developmental discussions; leave alone planning exercises the world over.
The moot question to be answered by development practitioners is about the setting of evaluations. Are they demand driven or are they law bound? If they are demand driven, the future of impact evaluation is bright as well as promising.
Present Impasse:
It has become mandatory for all developmental initiatives to have an element of M&E built in to its body as part of good governance. Hence we could see ample of lip service rendered for the cause of M&E. It has become a fad with many development institutions since the advent of the MDGs to have an element of M&E. It is common knowledge that development banks, public enterprises and international organisations like ADB, UN, World Bank, African Development Bank and the like have separate M&E divisions in their organisations. Besides, at times one could see along with the M&E, the emergence of the institution of ombudsman. It is to settle the various disputes at different levels. But M&E is a misunderstood organisation in several places. Unfortunately at times, M&E is referred to as a “policing body” of the management to settle scores with the less privileged and disadvantaged. There is considerable mistrust among work mates in this count. Much depends on the awareness created at the corporate level and the flow of information as well. M&E as a management tool, based on a fair work culture must be understood at all levels. Regardless of the size of the organisation, M&E to a large extent is not demand driven. Though M&E is a misunderstood institution, it must be made user friendly and demand driven. For this conscious efforts are required at the corporate and government levels so that it percolates down as a normal institutional requirement like human resources, planning and the like.
Which Evaluation Methodology?
There is no one single evaluation methodology for all types of situations. At times one may choose a “before and after” methodology. At times it may be a “with and with out” project approach. Many have profitably used the “observation” methodology. At times photo evaluation methodology was used to ascertain the impact of an intervention with considerable discernment. Therefore, it is wrong to prescribe any single methodology as the panacea approach in impact evaluations. Many a time a combination of approaches may be required as well.
Critical Issues:
Now it is worth while to get acquainted with an empirical situation. Let us see a case study like situation. It was decided to conduct an impact assessment of an emergency irrigation project in Afghanistan by the M&E division. They have prepared seven sets of survey questionnaires meant for the beneficiary farmers, village elders, key project staff, water user associations, capacity development officials, control group of farmers and water distribution supervisors. Even training was imparted to the enumerators for three days both in the class room and in the field. Suddenly it was decided by the World Bank representatives who had descended over the project- a mission visit- to do away with these and instead asked to conduct a very quick study to save time and to be more precise as commonly used terminology in the M&E parlours, “white wash” exercise. This quick study means a one page questionnaire, 30 days for field work to cover 200 plus respondents, data analysis, report writing and submission. Now the M&E practitioners are caught up in a dilemma, “to be or not to be”. If they want to keep their jobs, the only answer is to say “yes sir”. And they said amen, particularly taking into account the present day world recession conditions.
In a normal situation, things would have been very different and to a large extent logical as well. Now let us examine, the normal way of a reasonably good approach to impact evaluation.
The initial hypotheses are that irrigation intervention may result in improved water supply to the existing command area and or may increase the irrigable land as well.
As the project itself is very complex, the questionnaires and the selection of the respondents assume added importance. In a way this would render the study findings representative with regard to irrigation in relation to its approach, efficiency, effectiveness, relevance and sustainability.
Based on the secondary data available, projects are selected randomly in each of the category namely small, medium and large.
Survey Methodology:
We make a decision as to how many strata should be used. The projects would be classified into three categories small, medium and large. Again from these, to keep the sample relatively small, meaningful and representative, they will be further classified into three major locations namely up stream ( Head) mid stream( Mid) and down stream( Tail).Of course, more strata and a larger sample mean getting more information at a higher cost in terms of money, time and man power.
But stratification ensures that the sample is representative and allows to reduce the sample size with out sacrificing the level of confidence in the conclusions arrived at the end of the study.
Strata for sample selection:
Regions constitute the first strata. They have different types of project interventions and agro- climatic conditions and socio- economic situations.
Time of the project intervention is yet another critical factor to be reckoned with in this process. At least one crop season must have completed after the project intervention to arrive at some meaningful impact indications and conclusions.
Similarly as already mentioned, type of intervention like small, medium and large is also taken into account while trying for stratification.
Issues in Impact Measurement:
If we compare the project impact in terms of “with and with out” project conditions, we may be able to arrive at the project impact. Generally once the intervention is effected, we can think of comparing the effect on the basis of “before and after”. But numerous other activities and exogenous factors could have affected the project area in its normal situation. “What would have happened” and “what is happening” with the project would have been the ideal approach to attribute the changes solely to the project intervention. But that is impossible in the normal way as the project intervention has been made. In practice, we mostly settle down for a “before and after” comparison with out getting into monetary comparisons. If we attempt comparisons in terms of money, it opens up a plethora of problems like the impact of inflation, forces of market interventions by controlled economies, etc.
There is another possible aberration in the before & after approach. If the yields are better after the intervention along with an increase in input prices and a decline in product prices, might inadvertently conclude that the intervention was a failure! If we take control groups- out side the intervention with more or less the same input price increase and produce price decline, the increased yields come to our rescue to attribute it as project impact. It is therefore advisable to concentrate on quantities instead of price based quantifications in the impact assessment exercises.
Control groups if selected from the potential future project intervention areas; it can serve as the base line as well.
Regarding the selection of the villages, they will come from the selected sub- project areas. If there is a conflict between two villages, then the one with maximum number of beneficiaries must be selected.
Household Selection:
They come from the selected sub project areas. They will be stratified as per their location in the irrigation system. If there are three levels of canals- main, distributory and water courses- these will be seen as strata. They will be dived into three categories- H, M & T each, taking the total length and divided into equal thirds.
Distributaries will be grouped into H, M & T sections, depending on the location of the off take along with main canal and one distributory will be selected at random.
The water courses at the selected distributory will be grouped into H, M & T sections and then water courses will be selected from each group.
The house holds operating land along the selected water course will be listed and grouped according to H, M & T locations. One house hold will be selected at random from each group. Similarly one household not benefited from irrigation (control group) will also be selected at random from each group.
Thus from a sub project, 18 house holds may be selected for interviewing, making the sample fairly representative.
Distribution of Households by Strata:
Final Selection: Head Middle Tail Total
Main canal 1 1 1 3
Control Group 1 1 1 3
Distributory 1 1 1 3
Control 1 1 1 3
Water course 1 1 1 3
Control 1 1 1 3
Total 6 6 6 18
Suppose we decide to select from each region 10 schemes (@ of 18 per scheme the total will be 180 per region. We have a total of 5 regions and the total sample will be (@180 per region) 900 farm households.
Alternatively if we select 5 schemes per region, the total may be 450 households.
As already mentioned, much depends on the time, manpower and money at our disposal.
Another approach: Water Courses (WC) will be dived into three categories- H, M & T each, taking the total length and divided into equal thirds.
The house holds operating land along the selected water course will be listed and grouped according to H, M & T locations. Three households will be selected at random from each group. Similarly three households not benefited from irrigation (control group) will also be selected at random from each group making the sample very representative.
The final sample selection comes as per the following details:
- Selection of the main canal
- Selection of the distributory
- Selection of the WC
From the randomly selected WC, five respondents each are selected from head, middle and tail sections making a total of 15 beneficiaries and similarly five each from each location making a total of fifteen for the control group. Thus the number of selected farm households will be 30 per water course.
If we have 10 schemes in each region, the number of selected farmer households may be 300, making a total of 300 * 5 = 1500 households for the study as the final sample.
Alternatively if we select 5 schemes per region, and in each scheme 30 households, the total may be 150 households. The final sample for all the five regions may be 750 households. Again, as already observed, much depends on the time, manpower and money at our disposal.
Supply Constrained vs. Demand Controlled:
It is amply clear that all these years, by and large, impact evaluation or for that matter M&E was a top down exercise, imposed by the donors of funds and services. It was the real string in the hands of those who controlled the purse, as their main motive was achieving the targets as set out in the programmes and project documents. This is what we call good governance. All the funding sources had insisted on setting aside a fixed amount to begin with, later on a certain percentage of the total plan out lay as M&E budget. Willy nilly the recipients agreed to such stipulations. However, they did not bother much about its truthful implementation. Thus it was more of a supply constrained provision.
It must give way to a demand driven situation. It is like a property owner taking an insurance policy. Initially it might have been due to the pressure of the insurance company. But later on when it becomes popular and as the benefits is well known, it is no more the company, but the potential beneficiaries rush to take the policy. This is known as demand driven system. In M&E also it has been, by and large, a story of a demand controlled institution. In a way a good M&E system would enable the project executing officials not to run around to reinvent the wheel. It is common knowledge that M&E basically means learning the lessons and move forward with out flaws while replicating similar project interventions elsewhere. But efforts are needed to institutionalise the demand driven M&E system in all sectors particularly taking into consideration the MDGs of the UN.
Myths of Evaluation:
There are very many closely held myths pertaining to evaluation as a whole. It was believed that for any impact evaluation worth its name at least 10 per cent of the population must constitute as the sample. Its logic was not explained by any one. If the very objective is to obtain a reasonable and representative sample, one should think in those lines, not to impose a huge sample size. It is to be studied and realistic conclusions are to be arrived at by M&E practitioners.
One can not say with any degree of confidence that one evaluation methodology is superior and that is the only accepted methodology for all occasions. One has to pick and choose the appropriate methodology depending on the situation.
Similarly the concept of control group was questioned by many an authority. The argument was particularly in agriculture no two situations are homogenous and as such the control groups are a kind of myth. In social sciences we are not measuring exactness, but only trends and suggestive indications. As such, control groups may be the only way to capture the impact of project intervention in an otherwise dynamic equilibrium. One should not loose sight that even with out the project intervention, some developments are taking place and several exogenous forces are in operation as well.
It must be understood that M&E normally tries to ensure trouble shooting while the project is being implemented and to learn the lessons for replicating similar interventions elsewhere. If this is understood clearly by policy makers and implementers alike, our understanding of M&E becomes transparently clear and easier. In fact, efforts are to be made for awareness creation at all levels for better understanding of M&E.
There is another myth associated with impact evaluation in its measurements. It is vehemently argued by one school of thought that the Economic Rate of Return (Internal Rate Return) must be calculated for every intervention. But another school of thought on impact evaluation argue that in an unequal and non- static milieu, this ERR approach is a myth and stupidity and can only fool those who are inadequately initiated into M&E. But some people do it while others throw it out of the window. The only commonly accepted approach is the comparison in terms of additional area brought under plough, area irrigated, new crops introduced, cropping intensity, increase in yields, etc. We need a clear consensus on this as well.
It is the considered opinion of many a practitioner of M&E that there is no one single evaluation methodology. It is also felt that insisting on any single methodology is not a very prudent approach either. We may have to suggest alternate approaches to M&E to be practical, keeping in mind the different emerging scenarios and the new challenges.
It is a truism that the number of M&E professionals in the areas of agriculture and rural development are very few. In fact, there is a great demand for such practitioners taking into consideration the number of such developmental interventions taking place the world over under the aegis of World Bank, European Union, Asian Development Bank, African Development Bank, etc.
It is high time that universities must take up M&E seriously in their curriculum. In fact the self taught professionals are not the only response to face this great challenge. As a matter of fact, during the days to come M&E would be something like every day need for policy makers and implementers alike both in private and public sectors. There is going to be a paradigm shift in favour of M&E during the days to come, taking into account the advent of good governance, social auditing, etc. As it is said, “the harvest is plentiful, but labourers are very few.” we should not be found wanting in responding to this thematic challenge.
The need of the hour is a holistic approach to M&E. It is to be understood that gone are the days of single window vision like blinkers to the horse in development. It must be focused as well. In any M&E system worth its name, there must be a good blend of practitioners from economics, agricultural economics, micro finance, sociology, anthropology, political science, environmental sciences, demography, statistics, etc. This would render both evaluation methodology and practice, participatory both at the practicing level and at the field level alike
END
Doc; Dr. KM George, 4th Dec 2008, Kabul, Afghansitan