The project monitoring system in Pakistan: an evaluation.

Sahibzada, Shamim A. ; Mahmood, Mir Annice

The primary function of this paper is to analyse and assess the functioning of the monitoring and ongoing evaluation system during project implementation in Pakistan. In other words, it is the appraisal of the effectiveness of the system which is aimed at. However, it should be mentioned here that the system of monitoring and ongoing evaluation which is examined in the paper is related to projects in the agriculture and rural development fields. This is because of lack of data in other areas.

Initially we shall examine some of the terminological concepts in monitoring and evaluation. This is followed by a description of the system as it exists in Pakistan at the national, provincial and project levels. The system's functioning at the project level is illustrated briefly with a number of case studies. The paper concludes with suggestions for policy-makers to improve the operational effectiveness of the monitoring system in Pakistan.

DEFINITION OF MONITORING AND (ONGOING) EVALUATION

Monitoring can be defined as "a process of measuring, recording, collecting, processing and communicating information to assist project management decision making" [4, p. 2]. "Ongoing evaluation is the analysis, by project management, of monitored information on a continuing basis, with a view to enabling it where necessary to adjust or redefine policies, objectives, institutional arrangements and resources affecting the project during implementation" [4, p.3]. Monitoring as seen by the World Bank "assesses whether project inputs are being delivered, are being used as intended and are having the initial effects as planned. Monitoring is an internal project activity, an essential part of good management practice and therefore an integral part of day-to-day management" [12, p.4].

Guido Deboeck states that "monitoring can be defined as the timely gathering of information on project inputs, outputs and complementary activities that are critical to the attainment of the objectives of the project. It utilizes baseline information collected during the design and preparation phase, and continues throughout the project's life-time when it compares actual inputs and outputs and activities with the expected or planned levels. It alerts project management and policy-makers to potential implementation problems requiring corrective action. It my also provide the necessary information for the instigation and preparation of ongoing evaluation" [3, p.2].

Monitoring and ongoing evaluation are interlinked. The latter refers to the process whereby project management continually examines the data flowing from a project's inputs and outputs making necessary adjustments to meet any deviations from the planned schedule. Concurrent evaluation is another term which can be used interchangeably with ongoing evaluation. Impact evaluation, or ex-post evaluation (which is not included in the scope of this paper) refers to the effects of the project after completion, that is after a time lapse of one to five years. Some other terms used in project monitoring--for example, inputs and outputs, effects and impact--are discussed below.

PROJECT MONITORING TERMINOLOGY

As projects involve the use of inputs, in monetary and non-monetary forms, to generate output, these have an effect and an impact on the target group. A monitoring system therefore keeps a record of project inputs, outputs, effects and impact on project beneficiaries [11]. These four objectives of a project are briefly described below.

Project inputs, which are the immediate object of a monitoring system, can be differentiated into three categories: (i) physical facilities, e.g. a health unit or an irrigation system; (ii) advice to project beneficiaries; and (iii) supply of goods and services, such as seed, fertilizer, and credit. The various types of inputs provided by a project give rise to output by those who benefit from the project. Such benefits, for example, are increased yields of crops due to greater availability of fertilizer, or increased school attendance or a more extensive use of health facilities, if a health unit has been provided, and so on.

Once outputs have been generated from a project, their effect on the target group can be seen in such form as, for example, increased crop production as a consequence of greater availability of fertilizer, an improvement in general health conditions as a result of the provision of basic health facilities, and so on. Not all project effects emerge in a short time. Some effects take longer to emerge because of the poor state of education of the target group and the low level of the peoples' awareness. Also, cultural factors may inhibit the adoption of new techniques and new ideas.

Project impact is the fourth concern of a monitoring system. The effects generated by project activities have an impact on the target population. Higher incomes may give rise to greater employment opportunities. Better health and education facilities lead to an improvement in the quality of life through improved nutrition, higher literacy rates, wider participation in community life, etc. Therefore, it can be seen that impact is a project's longer-term development objective as distinct from inputs and outputs which are of immediate concern.

It can be seen, therefore, that a monitoring and ongoing evaluation system has two major objectives. The first relates to the physical and financial progress of projects, indicating the degree of physical work completed and the amount of finances disbursed. The second relates to inputs and their utilization by those who benefit from the project. Thus, through a constant monitoring of inputs and outputs, it is possible to compare the actual position with the expected one.

For effective monitoring, however, there are three basic prerequisites [13]. The first is the responsiveness of management to the various flows of information from the field. The manager has to be competent to analyse the data and to take corrective action where necessary. The second prerequisite is a technical one which involves data collection, data processing, data analysis, and presentation of the results. Time is of importance here if the project manager is to take the requisite action to meet shortcomings in the implementation schedule. The third prerequisite is institutional and administrative arrangements within the country without which its effectiveness would be blunted.

MONITORING ARRANGEMENTS AND TECHNIQUES IN PAKISTAN

At the National Level

In this section we examine arrangements and techniques for project monitoring and evaluation at the national level. The second part of this section reviews the monitoring system at the provincial level, followed by a discussion of monitoring techniques and arrangements at the project level as adopted and established by autonomous organizations such as the Water and Power Development Authority (WAPDA), the Agricultural Development Bank of Pakistan (ADBP), etc., for their projects.

There is an Implementation and Progress Section in the Planning and Development Division. This section was created in the Project Wing in 1966 but has been in active operation since 1978, though with very inadequate staff. Two techniques are adopted for progress monitoring by this section. The first includes the filling up of special forms by the Project Directors. The data on these forms include the degree of progress in physical and financial terms, the difficulties being faced at the implementation stage and any deviations from the original plan along with any changes in the costing of the changed plans. These forms are known as PM-I, PM-II, and PM-III.

The second technique of progress monitoring is that of on-site visits by the relevant officials. Their reports are then circulated to various ministries and departments. The section also prepares special review reports for major development projects and in the last five years approximately 160 projects have been the subject of such reports. This is an annual average of 32. [2]. This small coverage is stated to have been due to the shortage of trained staff in the section [2]. Also, quarterly summaries of the rate of progress of various projects are presented in the meetings of the Executive Committee of the National Economic Council (ECNEC) along with the problem areas that have been identified. Directives are issued by ECNEC to the executing Ministries and Divisions to take remedial action, where necessary.

At the Provincial Level

At the provincial level, the Planning and Development Department of the Government of Baluchistan is in the process of developing an effective project monitoring system. The system is expected to operate for all sectors of the provincial economy: the industrial, agricultural, power and social sectors. Currently, monitoring work is primarily carried out by filling out the relevent forms every quarter or on a half-yearly basis.

The forms are of the standard type, indicating name of the project, the executing agency, the area to be served by the project, total cost, the year of commencement, annual financial phasing, benefits of the scheme, physical targets of the scheme, i.e. what has to be accomplished in physical terms (e.g. number of houses built or of canals constructed, or of health centres established, etc.), the expenditure actually incurred and the physical targets actually achieved. In some instances the expected date of completion is also mentioned. An important feature of these forms is that they also identify bottlenecks, if any, during implementation. For example, a scheme may be particularly well planned but when it comes to be executed, it is discovered that the relevant and qualified manpower is not available. Consequently, the scheme fails.

Apart from reporting, concerned officials often undertake unscheduled visits to inspect the pace of the development work. The data collected on the official forms relate to the physical and financial aspects of the project: how much work has been completed in physical terms (usually expressed in percentages) and how much money has been utilized in the project. For federally funded projects a similar system is also functioning with a Federal Inspector-General (only in Baluchistan) whose sole duty is to monitor the pace of development work of the federal projects, usually through on-site inspections which are followed by a written report.

The other three provinces also have arrangements for monitoring the progress of projects. In the Punjab there is a progress-monitoring section which has been functioning since 1974-75. The section is located in the Planning and Development Department. The responsibility of this section is to undertake monthly, quarterly, half-yearly and annual reviews of all the projects which are being executed within the framework of the Annual Development Programme (ADP). Regular monthly review meetings take place with the various administrative departments so that bottlenecks, if any, are identified and corrective action is taken [5].

In Sind also an institution has been in existence for monitoring progress since 1974-75. The Provincial Planning and Development Department is responsible for holding review meetings on a half-yearly basis whereas the Administrative Departments hold such meetings on a quarterly basis. These quarterly, half-yearly, and annual reviews are concerned mainly with the financial and physical progress of projects. However, there is no monitoring of each and every project as the staff that would be required for such purposes is not available [5].

In the NWFP, progress monitoring of projects in different sectors is done on a quarterly basis, where officials such as the Divisional Commissioner and Assistant Commissioner are required to monitor the progress of the development work. Their reports are submitted to the Planning and Development Department of the Provincial Government. In addition, officers of that department also visit important selected projects to review the pace of the development work [5].

Other forms of monitoring include the filling up of quarterly reports, spot inspections of selected projects by officers of the Planning and Development Department and visits of Inspection Teams, e.g. the Governor's Inspection Team, the Departmental Inspection Team, etc. There are also District Technical Review Committees which are headed by the respective Deputy Commissioners and whose responsibility it is to review the progress of projects in the Annual Development Programme [7]. Finally, there are review meetings held in the Planning and Development Department which consider the monthly review reports sent by the various administrative departments. Of particular importance is the mid-year review of all the projects included in the Annual Development Programme.

At the Project Level

At this level, monitoring arrangements exist only for foreign-aided projects. One such project is that of On-Farm Water Management, the monitoring arrangements for which are being made by the Water and Power Development Authority (WAPDA) whereas the sponsoring agency is the Ministry of Food, Agriculture and Cooperatives, Government of Pakistan. The project like other WAPDA projects has a very effective monitoring system.

The assignment to monitor and evaluate the project was given by the Ministry of Food, Agriculture and Cooperatives to WAPDA's Monitoring and Evaluation Directorate which has the relevant experience in this field. The Chief Engineer, Survey and Research Organization, Planning Division, WAPDA, has the overall administrative control of the project. The project itself is headed by a project director who supervises all the activities of the project. Three deputy directors also assist him at his headquarters. These deputy directors are specialists in the field of agricultural economics, agricultural engineering and statistics. Initially, two regional offices were established. These were located at Faisalabad and Hyderabad. A third regional office has just been established at Multan. The offices are headed by a Senior Research Officer. The area under the Faisalabad regional office covers the sample water courses which are situated in the NWFP and the Punjab whereas the Hyderabad office covers the sample sites which are located in the provinces of Sind and Baluchistan [8].

The project has been monitored through three stages; the pre-improvement stage, the ongoing stage and the post-improvement stage [8;9;10]. Data have been collected for all the three stages through administered questionnaire interviews and the recording of actual observations by officers in the field. Strict quality control of the data is ensured at the time of their collection. This is done by the Senior Research Officer in charge of the regional office who carries out field checks in areas under his jurisdiction. If there are any divergences or misunderstandings, the officer is expected to clear them up with his subordinates. Deputy Directors at the Head Office also undertake field trips and the Project Director also visits field areas to monitor the work and resolve any technical and administrative problems.

The second case study is the Fourth Agricultural Development Bank Project in Pakistan which is being monitored and evaluated by a Monitoring and Evaluation Cell of the Agricultural Development Bank of Pakistan (ADBP) [1]. This cell has been established because foreign agencies, such as the World Bank and International Fund for Agricultural Development, insist that the loans they give be monitored. In a way this ensures that credit is spent for the purpose it was given for. Also, the data received from monitoring the utilization of the loan helps in the efficient management of the projects. The monitoring cell is situated at the headquarters of the ADBP which is in Islamabad. Apart from baseline surveys which the cell conducts a system of regular reporting has also been instituted. Quarterly reports of physical and financial implementation are submitted to the relevant officials. The quarterly reports show Lending Operations, Recoveries, Status of Loan Applications, and Disbursements of IDA and IFAD Loans. The members of the cell will also conduct spot checks by visiting branches/regional offices.

Programme Monitoring and Evaluation Systems exist for Integrated Rural Development Projects. Daudzai, the third case study, is one such project run by the Pakistan Academy for Rural Development, Peshawar. Monitoring and evaluation of the activities of this integrated rural development project are done at the Academy, the markaz (project) level, and the village levels [6].

At the Academy level, where the Director is the administrative head, monitoring, which basically involves a comparison of project achievements with project targets, is carried out through regularly held monthly meetings, a system of quarterly reporting, annual reviews and monthly training conferences [6].

At the markaz (project) level also, similar monthly and annual meetings are held to review the progress of various activities, identify problems and provide guidelines for future course of action. At this level, extension education training sessions are also held, the major aim being to bring rural people into contact with extension workers [6].

At the village level, elders and other responsible workers meet regularly to discuss the implementation of production plans and development schemes [6].

CONCLUSIONS AND POLICY RECOMMENDATIONS

It will be premature to make general recommendations on Pakistan's project-monitoring system, especially in the agriculture and rural development sector, since experience is still limited, and to generalize from the three case studies used in this paper will be further disturbing because all the three are pilot projects and foreign-funded, and thus have reasonably effective monitoring systems.

However, our examination of the arrangements for project-monitoring systems at the federal, provincial and project levels enables us to reach a few conclusions and to make certain recommendations.

At the national level, the prominent shortcomings are as under:-

1. Some special review-reports for major development projects have been undertaken by the Implementation and Progress Section of the Planning and Development Division, but owing to very insufficient staff with the Section, a comprehensive coverage has not been possible.

2. The Project Monitoring Forms are biased towards financial monitoring and there is no standard criterion to be used for measuring physical progress, which is done in percentage term and can be misleading. No scheduling and controlling techniques like Programme Evaluation Review Techniques, Critical Path Method, and Bar Charts are used on a systematic basis while planning for implementation. This can be seen from the PM-I form which makes only an enquiry under Section 23 about the use of such techniques but does not make it a compulsory requirement.

At the provincial level, the arrangements are also extremely deficient. The weak points are as follows:

1. The provincial Planning and Development Departments lack trained staff for monitoring.

2. Monitoring arrangements for individual projects at the provincial level (as done by the Implementation and Progress Section of the Planning and Development Division) exist in theory but in actual practice these are almost non-existent.

3. At best, reviews are prepared on monthly, quarterly, half-yearly, or an annual basis. These reviews generally contain information by sectors which is collected from periodic review-reports or through monthly or quarterly review-meetings generally held at the provincial headquarters but without any visit to the project areas/sites.

4. The provincial governments (the NWFP for example) have assigned the task of monitoring the implementation of development projects to civil servants who lack the necessary expertise for monitoring and evaluation functions in a satisfactory manner.

At the project level, monitoring and evaluation systems exist for the implementation of foreign-aided development projects. Monitoring and evaluation cells/sections for such projects have been set up under pressures from International Financing Agencies. The system being adopted in the case-study projects mentioned above and discussed in this paper reflect such pressures.

We have also discussed very briefly the prerequisites of an effective monitoring system at the project level, but the Pakistani experience shows that this newly developed system of input and output monitoring is not being followed closely except in a few cases.

Whatever monitoring and evaluation (M&E) arrangements exist at the project level, these were constrained by late establishment which consequently affected the operation of M&E cells/units. They should have been considered at an earlier stage in the project cycle. A delay in establishing such units has been due to the lack of funds with Project Management for recruiting and training appropriate staff for the job.

There are institutional and financial constraints, too. Institutional constraints are reflected in the shortage of qualified staff to operate a monitoring and evaluation system. However, this does not hold true for all organizations. For example, WAPDA stated that their major problem was financial rather than of shortage of trained staff.

Four major recommendations can be made:

1. Since the organizational and institutional arrangements for many projects are very weak because they are considered very late in the project cycle, attention should be paid to assessing various organizational and institutional alternatives for project implementation at the appraisal stage.

2. It is observed that project monitoring and evaluation in Pakistan are carried out through a multiplicity of monthly, quarterly, half-yearly and annual reports which produce information in excess of requirements. This results in an unnecessary wastage of time, energy, and money in sorting out the relevant data for decision making. This is a design defect in the monitoring and evaluation system. Therefore, attention should be paid to this problem at the appraisal stage by limiting the number of reports and making them more objective in terms of input-and-output monitoring. The frequency of monitoring should also be decided at this time and a balance should be maintained between too many and two few reports, and the time interval between reports should be of reasonable duration.

3. Generally, no regular financial provisions are made in project budgets for monitoring and evaluation. It is recommended that a certain percentage of project costs should be allocated for this purpose, because experience in other countries shows that investment in monitoring and evaluation has proved to be highly profitable in terms of improved project performance.

4. Finally, the system can not be effectively operated if the requisite staff is not in position at the commencement of project implementation. Given the complexities of agricultural and rural development projects, there is need for a multidisciplinary team in the M&D cells/sections. Furthermore, it is essential that field staff be conversant with the latest techniques involved in the collection, processing and analysis of data. This requires training of personnel in the field of data handling.

Last but not the least, for effective monitoring, the project manager should be a highly competent person with sufficient authority and power to take timely action on the basis of the monitored data.

Comments on "The Project Monitoring System in Pakistan: An Evaluation"

I must compliment the authors on dealing with a relatively unresearched field in an imaginative manner. Despite the many obvious advantages of an effective monitoring system, the most interesting result of the paper that I found was that the monitoring system is almost non-existent at the Federal and Provincial levels, while at the project level it exists only for foreign-aided projects, probably under pressures from the donor agencies.

To begin with, it should be fruitful to identify the factors that may explain why effective monitoring systems have not become an integral part of development administration in Pakistan. Insufficient understanding by the policy-makers of the usefulness of monitoring as a management tool may partly explain their reluctance to build monitoring components into any development project. The policy-makers may resist the introduction because of the fears that the weaknesses of the programmes pointed out during monitoring and ongoing evaluation may have adverse implications for their professional careers which are probably a direct function of smooth running of the administrative machine under their command. Some programme managers oppose monitoring and evaluation as they fear that the weaknesses of the programmes highlighted by the monitoring exercise may provide gun power to the critics to effectively shoot down the programmes that they do not like. Effective monitoring and evaluation systems are very costly. In developing countries, where there is scarcity of technical skills and financial resources, whenever the choice is between allocating scarce resources between monitoring systems for development projects and programmes and new development projects, the choice is often made to fund the additional projects. Another reason for the lack, or rather slow development, of the effective monitoring system stems from the fact that each new system has to replace some existing set-up. Monitoring in Pakistan at present is being done in an informal manner through personal discussions, meetings at different levels, direct observation and evaluation of programme results obtained through informal channels of communication. Some resistance in replacing the existing informal monitoring system with formal, scientifically designed monitoring system is but natural. A twofold strategy may be helpful in overcoming this conflict. Studies should be carried out on the benefits gained from effective monitoring systems leading to appropriate management action. The costs of monitoring systems appear to be prohibitive while the benefits of such systems are not easily seen in an improved implementation of the development projects. The calculation of cost-benefit ratios for the monitoring system should provide useful pieces of information. The formal monitoring system may be introduced in stages; for some time, both the old informal system and the new monitoring system should run concurrently. The old system may be phased out when clear-cut, positive results emerge from the new monitoring set-up. The important point to note in this regard is that the superiority of the new system should be proved to the policy-makers, programme managers and the representatives of the intended beneficiaries of development projects and programmes. A consensus among the three actors should help in the installation of an effective monitoring system.

The decision to put in place a formal monitoring system presumes an agreement on the most suitable form of the organizational structure. The authors have not discussed this aspect, presumably because it is a vast area of study by itself. A number of pertinent questions, however, need to be raised and answered before some decision on the appropriate form of the monitoring set-up is taken. Should the monitoring set-up be an independent one or should it be a part of the project organization? Should the monitoring and evaluation set-ups be merged into one agency or should they be kept separate from each other and entrusted to two agencies? Should the monitoring set-up be centralized or decentralized to a project organization level? All these questions need a detailed empirical examination in the context of Pakistan before actual choices are made. Brief comments on each of the issues are given only to show the complex nature of the choices and pertinent considerations.

The creation of a separate organization for monitoring and ongoing evaluation may be justified as the staff implementing a project or a programme may hide the shortcomings and inflate the achievements of the programme being implemented by them. The programme staff, on the other hand, have abundant and easy access to data concerning both inputs and outputs of the programme and may be reluctant to pass on this information to another organization. This problem may be tackled if the separate organization is placed under the control of the head of the department implementing a project or a programme.

The issue concerning the relationship between the monitoring and evaluation agencies also needs a careful balancing of pros and cons of entrusting these functions to one agency or to two different agencies. Monitoring and ongoing evaluation overlap considerably. The information from monitoring exercise could easily be used by an evaluator when both functions are combined in one organization. On the other hand, combining the two agencies may mean some compromises regarding sound and impartial judgement of the evaluation staff.

The issue relating to the level at which monitoring needs to be carried out is also a complex one as a decision is required whether monitoring should be specific to each project or to major development programmes. Furthermore, it needs to be decided at what levels of implementation the monitoring set-ups need to be established. Project-specific monitoring can bring out specific difficulties requiring a corrective action on the part of the management. However, monitoring each and every project may be very expensive. Should we monitor all major projects and some selected small projects? Programme-level monitoring has its own uses but many require a large organization with usual defects associated with large bureaucracies. Moreover, a monitoring set-up needs to be established at each level of implementation. Each level has to have information which may enable it to take corrective action within its own area of competence. In the case of Pakistan, it means that effective monitoring set-ups should be established at the federal, provincial and local levels.

An identification of the factors inhibiting the use of an effective monitoring system and the specification of the organizational requirements for the monitoring set-up are useful pieces of information. Equally important issues are the identification of programme objectives, choice of critical variables requiring monitoring, and establishment of an appropriate reporting system. Programme objectives in official documents are often vague and do not clearly bring out the relationship between activities and goals. An agreement on the goals and the logical relationship these goals have with activities needs to be .ascertained through a careful scrutiny of all the project documents and a discussion with the concerned policy-makers. Various techniques, like project performance network chart, programme evaluation review techniques and critical path method, have been in use for a long time and have aided in the identification of the information needs for effective monitoring. For local-level projects, in which rural folks are involved, such techniques need to be somewhat amended. The problems in the application of such techniques to different projects and programmes in the context of Pakistan can only be identified after practical experience. It is my impression that such tools in Pakistan have been taught in project planning training courses but have not been widely used in practice. The authors confirm this when they say that the project documents do not contain the requisite information to enable a Project Manager to monitor the project.

The general conclusion that I reach is that the limitations of our knowledge in the area of project monitoring in Pakistan are immense. The fact that recommendations to install effective monitoring systems have been consistently ignored by intelligent and well-intentioned government officials confirms that there is far more ignorance than ill-will behind the undesirable policy actually chosen. There is thus an urgent need for more research in this area. Since development projects cannot, and should not, be frozen, general conclusions, even if they are only interim ones, have immense usefulness. From this point of view, the authors have done an immense service in drawing obvious but pertinent policy conclusions.

Sarfraz K. Qureshi

Chief of Research (Economics), Pakistan Institute of Development Economics, Islamabad

REFERENCES

[1.] Bukhari, Shaukat Ali. Report on Monitoring and Evaluation Arrangements for Fourth Agricultural Development Project in Pakistan. Islamabad: ADBP. February 1982.

[2.] Chaudhry, Shah Mohammad. Five Years of Project Monitoring in Pakistan, (1978-83). A Story of Trial and Success. Islamabad: Planning and Development Division, Implementation and Progress Section. 1983. (Report No. 13 of 1982-83)

[3.] Deboeck, Guido J. Basic Concepts and Issues on Monitoring and Evaluation of Rural Development Projects. Washington, D.C.: World Bank. 1978.(Working Paper)

[4.] FAO. Monitoring System for Agricultural and Rural Development Projects. Rome.1981.

[5.] Hussain, Mohammad. "Plan/Programme Implementation and Monitoring at National Level". Pakistan Administration. Vol. XVIII, No. 1. January-June 1981.

[6.] Khan, Minhajuddin. Participatory Project Management System in Daudzai IRD Markaz--Paper Presented at Workshop on Local Level Planning through Community Participation, March 1-19, 1983. Peshawar: Pakistan Academy for Rural Development. 1983.

[7.] Punjab Planning and Development Board. Planning Manual. Lahore. 1980.

[8.] Water and Power Development Authority. Monitoring and Evaluation of On-Farm Water Management Programme (Pre-Project Evaluation). Main Report. Lahore. 1981. (A study done for Ministry of Food, Agriculture and Cooperatives, Government of Pakistan)

[9.] Water and Power Development Authority. Monitoring and Evaluation of On-Farm Water Management Programme (Post-Project Evaluation). Semi-Annual Report. Lahore. December 1982. (A study done for Ministry of Food, Agriculture and Cooperatives, Government of Pakistan)

[10.] Water and Power Development Authority. Monitoring and Evaluation of On-Farm Water Management Programme (Post-Improvement H Evaluation). Semi-Annual Report. Lahore. June 1983. (A study done for Ministry of Food, Agriculture and Cooperatives, Government of Pakistan)

[11.] World Bank. Guidelines for the Design of Monitoring and Evaluation Systems for Agriculture and Rural Development Projects. Washington, D.C. November 1981.

[12.] World Bank. A Handbook on Monitoring and Evaluation of Agriculture and Rural Development Projects. Washington, D. C. November 1981.

[13.] World Bank. Monitoring Rural Development in East Asia. Washington, D.C. October 1980. (Staff Working Paper 439)

SHAMIM A. SAHIBZADA and MIR ANNICE MAHMOOD *

* Respectively Chief, Project Evaluation Section, and Research Economist, Pakistan Institute of Development Economics, Islamabad (Pakistan).