Assessing the performance of local government.

Stevens, Philip Andrew

We consider the measurement of performance in the public sector in general, focussing on local government and the provision of library services by English local authorities in particular. We will consider two methodologies that assess the performance of local authorities in terms of the efficiency with which they provide services and consider methods that allow us to account for exogenous influences on performance, such as the socio-economic profile of the population served by the authority. We find that although both methods' results appear similar, the implications for potential cost savings vary widely. Omitting to account for background factors leads to an overstatement of the level of inefficiency and hence the scope for reducing expenditure.

Keywords: local government; efficiency; SFA; DEA JEL classification: D20, H40, L32

I. Introduction

The performance of the public sector is always an issue that attracts interest. There are two reasons for this; one is its size and the other the services it provides. The public sector represents a sizable proportion of the economy, and it is getting bigger. According to OECD figures quoted in the Financial Times (22/03/05), the share of general government spending in GDP is forecast to rise from 37.5 per cent in 2000 to 45.2 per cent in 2006. Moreover, any inefficiency is likely to have a major impact upon the nation's welfare.

Education, policing and the like affect the whole of society and the vulnerable in particular. Crime affects the poor disproportionately, partly because the wealthy can move to safer areas or purchase security measures. The private sector often cannot be relied on to produce these services efficiently and/or equitably. Indeed, it is precisely because of this that many services are provided or at least financed by the public sector. The difficulties in the measurement of the output and performance of the public sector arise in great part for the very same reasons that the services are in the public sector in the first place.

In this paper we consider the measurement of performance in the public sector in general, focussing on local government and the provision of library services by English local authorities in particular. We will consider two methodologies that assess the performance of local authorities in terms of the efficiency with which they provide services and consider methods that allow us to account for exogenous influences on performance, such as the socio-economic profile of the population served by the authority.

In section 2 we outline the main issues surrounding assessing the performance of public sector organisations and relate these to local government. Section 3 discusses techniques of efficiency analysis. In section 4 we describe the data used in our empirical analysis. Our results are discussed in section 5 and section 6 provides some concluding comments.

2. Assessing public sector performance

There are three main problems in assessing public sector performance: identifying the outputs, the absence of prices, and the problem of attribution. It is useful to make an important distinction between activities, outputs and outcomes. Activities are the units of provision, for example the activities of the highways department of a local authority would include the number of checks carried out and the number of road repairs carried out. Outputs may require a bundle of activities; having a properly functioning road requires both that it is checked for problems and that these are fixed when required. Outcomes are the characteristics of outputs which affect utility, in our highways example it might be having free flowing traffic on the roads in an authority's region.

It is often difficult to identify what the outputs of public sector organisations are. There are some services where it is difficult to identify any of the outputs. The problem is that we tend to observe activities rather than the actual outputs. This has led to a concentration on processes (which can be easily counted) rather than the outputs the service was designed to provide. Thus in many cases the focus has become one step removed from the organisations' raisons d'etre.

Even if we can observe and measure the outputs of government, the services it provides, we seldom observe output prices. Thus it is difficult to aggregate the many services that organisations within the public sector provide up to an organisation-level indicator, much less a sector-wide measure of performance. In markets, prices carry important information about the value of the goods and services traded to the consumer--through their willingness to pay--and to the provider--since providers will not generally offer goods and services of which they cannot cover the costs of providing them.

The third problem is that of attribution. Many of the outcomes which government (either local or national) seeks to affect are influenced by other factors. Certain local authorities may find it difficult to provide waste collection services because they are sparsely populated. It is important to remember that what we are interested in is the marginal impact of the public sector. This accords with what is often called 'value added'.

The problems of identifying the outputs and the lack of prices in the public sector led to its output being measured by the amount of money spent for a long time. Thus in the public sector, if nowhere else, it could be said 'we must be doing better, we are spending more'. Productivity measurement was, therefore, impossible by definition.

Valuing local government outputs

There are two related reasons why is it so important to measure prices. The first is to allow us to aggregate outputs--to add up apples and oranges. In the private sector, the relative valuation of two products or services is simple since under certain assumptions the relative prices tell us all we need to know. The second use for prices is to account for quality change. If the quality of a good in the private sector improves over time, ceteris paribus, its price will increase. Thus relative prices allow us to compare different goods within and between years.

Products and services have a number of characteristics that are valued by consumers, which are reflected in their prices (Deaton and Muellbauer, 1980). For example, when we compare computers, we compare CPU speed, memory, and hard-disc size; when we compare houses, we consider the number of bedrooms, the floor-space, and the characteristics of the neighbourhood. Turning to local government, the characteristics of waste collection for example would include the frequency of collection, whether waste is collected from the home or the roadside, and aspects of recycling. The characteristics of library services might include the number, type and availability of items. In the hedonic pricing literature the implicit prices of the attributes of differentiated goods are derived by observing the joint variation of product prices and bundles of product characteristics. The estimated coefficients represent shadow prices of product attributes (i.e. the value of an additional unit of attribute holding the other attributes constant). The lack of prices for most publicly provided goods do not allow us to do this.

There are essentially five ways that one might overcome the lack of prices for public sector outputs. First, one might use prices from similar services operated in the private sector. Second, one might try to replicate the market by considering the impact of public services on outcomes. Third, we can use the costs of providing the services. Fourth, one could use value judgement weights. Finally, we could use the pattern of outputs or activities that are actually provided by authorities to impute shadow prices implicitly used by providers. Each of these methods has its strengths and weaknesses.

A common way to overcome the lack of prices in the analysis of the public sector is to ignore the problem. That is, one can use a method that does not require weights to be defined and allow the data to determine them. Thus we use the outcome of managerial and policy decisions by service providers implicitly to calculate the shadow prices of each output or activity that was used to determine levels of provision. This is the tacit assumption made in efficiency analysis using methods such as data envelopment analysis, although it is seldom declared so baldly (see Stone, 2002, for an honourable exception).

Local government in England

An important question--and one that goes beyond the scope of this paper--is why we have local government in the first place. In a large part, the division of labour between central and local government is a product of history rather than design (McLean, 2005). Prior to the Municipal Corporations Act 1835, some of the services provided by modern day local authorities were provided by Justices of the Peace. The current system of elected local government emerged in stages between 1983 and 1994.

Local authorities provide a wide range of services. These include, in descending order of cost: school education, personal social services, law and order, highways, fire services, libraries, waste collection and planning. In this paper we will consider one example, library services. This is because the available data describe a set of activities that closely resemble a set of outputs. We will consider the efficiency with which these services are provided using two common tools of analysis and examine the impact of factors that are beyond the control of the authorities.

3. Frontier analysis

Techniques of frontier analysis have become popular tools in the study of local government performance. (1) Frontier analysis determines which are the most efficient units in the sample and then compares the performance of all the others. In this paper we compare two commonly applied techniques for efficiency analysis--stochastic frontier analysis (SFA) and data envelopment analysis (DEA). Both have their pros and cons and so the most appropriate modelling strategy is likely to involve the use and comparison of both techniques. Comparing the results from the two methodologies has become common practice (Bjurek, Hjalmarsson and Forsund 1990; De Borger and Kerstens, 1996; Vitaliano, 1997, 1998), since it allows one to check the robustness of the resulting efficiency measures.

We shall also tackle the problem of attribution outlined above by considering the impact of exogenous factors (what are often called 'environmental', 'non-discretionary' or 'background variables') on the delivery of library services by English Local Authorities. These factors can be particularly important in determining the demand for and provision of local authority services. (2) Both DEA and SFA can be extended to include background variables, but there is no single, unambiguous way in which this might be done. These variables can be used directly in the determination of the frontier or they can be modelled as determinants of estimated efficiency itself. This can be done as one exercise or by using a second step regression of the efficiency scores. This two-step method, however, has been criticised for producing biased estimates (Wang and Schmidt 2002).

Data Envelopment Analysis

Data Envelopment Analysis is a linear programming methodology that has been widely used in the study of local authority performance, for example in local education authorities (Bessent and Bessent, 1980; Jesson and Mayston, 1989), and health authorities (Salinas-Jimenez and Smith, 1996). DEA identifies the frontier using mathematical linear programming.

Stochastic Frontier Analysis

Stochastic Frontier Analysis takes a statistical approach to determining the unknown frontier required for calculating efficiency. SFA allows for the existence of inefficiency by decomposing the residual into two components. The first is a random error component, common to most statistical regression analysis, whereas the second is an efficiency component, which is assumed to be distributed in a given way and independent of the random error term. SFA uses these assumptions about the distribution of the error components to differentiate between random errors and inefficiency.

Choosing the appropriate technique

One much touted advantage of DEA over SFA is that it does not impose a particular functional form on the data. Another is that it can also handle multiple input/ output technologies easily. This is a useful feature in the analysis of local authority performance since authorities, or departments thereof, usually provide more than one service and, as we have seen, there are considerable difficulties in converting them into a common metric, like the value of sales. This advantage may have been overemphasised since we can consider the cost (or expenditure) dual to the production frontier, which poses no problem for SFA.

There are, however, some disadvantages with the DEA methodology, which must be taken into account when one is undertaking the analysis. First, a key problem with DEA is its heavy reliance on the accuracy of the sample data; there is no allowance for stochastic errors. All deviations from the frontier are treated as inefficiencies. Indeed, the calculation of the frontier assumes that the DMUs operating at the frontier in particular are not measured with error. If this is not the case, it is possible that the frontier is defined largely by authorities whose outputs are overstated and/or expenditure is understated due to misreporting or other random influences on the data generation. This will affect the efficiency scores of all firms for whom these authorities define the relevant section of the frontier. (3)

In some circumstances a DMU may achieve a high efficiency score merely by being different (in its input or output mix) from other units. When the number of DMUs under consideration is small it is more likely that this problem will occur. The relative efficiency score achieved by each DMU can be sensitive to the number of inputs and outputs specified (Sexton et al., 1986; Nunamaker, 1985). Consider the extreme example of an authority that concentrates on one particular output to the exclusion of others, and is the only authority to do so. It will automatically be deemed to be efficient. This authority will be the most efficient at using this input or producing this output, as there are no others with which to compare it (Smith and Mayston, 1987). In general, the more input and output variables are included in the model, the higher will be the number of DMUs with an efficiency score equal to unity (Nunamaker, 1985). (4) This does not pose a problem for SFA since one can use standard statistical tests to evaluate the performance of the model. However, SFA crucially relies on the independence of the efficiency scores from the frontier to decompose the error, which in certain circumstances may not hold.

Accounting for background factors

The performance of a local authority in providing a particular service depends partly on the ability of staff to provide the services and the management to allocate resources efficiently and partly on the particular environment where the authority operates. For example, in areas with a high proportion of individuals for whom English is not the first language, teachers must devote time and resources to make up for any deficiency. Whilst these background factors can influence the distance of an authority from the minimum cost frontier, since they have no control over them, the inefficiency that they cause cannot be considered to be the 'fault' of the authority. In this context it is useful therefore to consider the inefficiency of an authority net of these factors. Since DEA is a linear programming methodology and SFA an econometric one, the methods whereby each one accommodates background variables are necessarily different.

Accounting for background variables in DEA

Two ways of introducing background variables have been considered in the existing literature. The first is to introduce such variables directly in the computation of the DEA efficiency scores. In the DEA analysis the linear programme will determine how much the discretionary inputs can be reduced while keeping the exogenous (nondiscretionary) ones fixed (Banker and Morey 1986). (5)

The second method used to introduce background variables is a two-stage approach. In the first stage the DEA frontier is derived, without accounting for the environmental effects. In the second stage a truncated regression model for the predicted technical efficiencies is formulated and the impact of the environmental variables is evaluated. A large number of studies have used the two-stage method with DEA (Bjurek et al. 1990; Chilingerian, 1995; Vitaliano, 1997, 1998; Worthington, 1999). However, this procedure will give biased results because the model estimated at the first step is misspecified (Wang and Schmidt, 2002). Moreover, it is curious that one has used a linear programming method to obtain the production technology and a statistical method to decompose the calculated efficiency scores.

Theory is relatively quiet on 'where background variables ought to go'. This is due in large part to the concentration of researchers on those variables over which economic agents have some control. Empirical researchers have included these variables to control for 'other influences', but it is only with the arrival of frontier analysis, which explicitly allows organisations to operate within the production possibilities set or feasible cost set, that the issue of the channels through which these factors exert their influence has been forced to the surface once more. Methods of resolving this problem have so far tended to be empirical rather than theoretical (Coelli et al., 1999; Stevens, 2004).

Accounting for background variables in SFA

There are also two ways in which one can account for the influence of background or environmental factors. The first is to include these influences in the frontier itself. The second method is to include them as determinants of the efficiency terms themselves. Note that some SFA studies have also investigated the determinants of cost inefficiency by regressing the predicted values of efficiency obtained from the estimation of a stochastic cost frontier on a set of background variables. An example of this in the context of the study of efficiency in library services is Vitaliano (1997). However, as Kumbhakar, Ghosh and McGulkin (1991) and Reifschneider and Stevenson (1991) have noted, there is a significant problem with this approach. In the first stage, the efficiency terms are assumed to be independently and identically distributed, but in the second stage they are assumed to be a function of these firm-specific factors, implying that they are not identically distributed, unless all the coefficients of the factors are simultaneously equal to zero. (6) Kumbhakar, Gosh and McGulkin (1991), Reifschneider and Stevenson (1991) and Huang and Liu (1994) presented models to overcome this problem by estimating both the frontier and efficiency terms in one stage. In this study we will use the results from Stevens (2005a) who utilises the model of Battese and Coelli (1995).

4. Data

Our data on expenditure and activities come from the 2000/1 Best Value Performance Indicators and Standard Spending Assessment data provided by the Department for Transport, Local Government and the Regions (local government is now the responsibility of the Office of the Deputy Prime Minister). These figures were initially presented as per head of population, so they were converted back into real terms using the resident population at midyear on 30 June 1998. The variables in table 1 are fairly standard in the analysis of library performance (Vitaliano, 1998; Worthington, 1999). (7) The activities [Q.sub.1A] to [Q.sub.3] provide a fairly comprehensive measure of the output of local authorities as providers of library services. One might perhaps argue that the number of books and recordings available in the council's libraries could be disaggregated to allow account to be taken of differences by type of item. There are four types of LA that provide library services: London Boroughs, Metropolitan Authorities, Unitary Councils and County Councils. By far the largest in terms of expenditure are the County Councils, with the London Boroughs and Metropolitan Authorities being of similar size. (8) In our estimation we will include a dummy variable to identify differences in costs by local authority type. Whilst it is possible for different types of authority to be inherently more or less costly than others we do not in this paper consider this as part of the exogenous determinants of efficiency, but rather interpret any significant difference as a topic worthy of further investigation.

The background variables that we will take into account in our analysis are presented in table 2. The first factor is population density, KIDDENS, measured by the population per hectare, since this is likely to affect the ability of the authority to concentrate provision in a small number of central libraries. Conversely, sparsely populated local authorities may have to provide mobile library services to cover large areas. We include two socio-economic indicators: the percentage of the population who are income deprived, as defined by the Income Domain of the Index of Deprivation (Noble et al, 2000), INCIID); and the percentage of the population for whom English is a second language. The next two variables account for differences in the age profile of the population, the percentage of the population who are aged five to fifteen years of age, YOUNGPOP, and the percentage who are aged over sixty-five years, OLDPOP. We also include dummy variables for LA type (the baseline LA type is Unitary Authority).

5. Results

First we outline the results using the SFA models, next the DEA models and finally we compare the predictions of efficiency with each other. Before we continue, it is important to note that the efficiency scores themselves represent the ratio of observed expenditure to the minimum feasible expenditure required to produce the set of outputs Q. Therefore they are bounded below by one, which signifies that the authority lies on the minimum cost frontier.

Stochastic Frontier Analysis

The results of the stochastic frontier analysis are presented in table 3. (9) In column (1) we present the results of the specification excluding background variables. Column (2) presents the results for the stochastic frontier with the background variables included in the frontier. Column (3) present the results for the models including the background variables in the efficiency determinants.

In all specifications the coefficients on the four output measures have the expected positive sign, although the coefficient on In[Q.sub.1A] is insignificantly different from zero in the column (1). This is the specification which excludes the background variables in either the frontier or the set of efficiency determinants and leads us to believe that this insignificance 1s due to misspecification. Neglecting to account for the influence of these environmental factors might lead one to conclude that the cost of running libraries is unrelated to the number of books issued!

Turning to the impact of the background variables, we see that when they are included the size and significance of the coefficients on the authority type dummies are reduced. This suggests that much of the apparent variation between authority types is due to the different environments in which they operate.

The direction and significance of the effect of the background variables are consistent across specifications. The only exception is the effect of population density. The results for model (2) suggest that there is no significant direct relationship between population density and expenditure on library services. The results for model (3) suggest that there may be an indirect one: libraries serving densely populated areas have higher levels of inefficiency. Costs are lower in authorities with a large young population (under 16) as well as a large portion of over 65s. Authorities serving in more deprived areas tend to have higher costs ceteris paribus. These results do not depend on whether we consider their influence to be a direct one (influencing costs via the cost frontier itself) or indirect (influencing costs via their effect on inefficiency). We find no significant effect of the proportion of the population for whom English is the second language on costs. One explanation for this is the fact that this variable is highly correlated with income deprivation.

We can examine the effect of our model specification on the estimated efficiency scores themselves by considering table 4. For model (3) we calculate both gross and net efficiency scores, i.e. both including and excluding the effect of the environmental factors, with an asterisk denoting that the scores are net of environmental factors. (10) Thus the figures for model (1) and model (3) are fairly comparable, since none of the three have the environmental effects removed; also, Models (2) and (3)* both take account of these factors.

As one would expect, the efficiency scores for model (1) and the gross efficiency scores for model (3) are the closest to each other, although there is less variation in the scores for model (1). These three models have similar means, with the average authority having costs just over 20 per cent above the minimum possible for their set of outputs. All three models suggest that some authorities are spending twice as much as they could be.

The scores for Model (2) and the net efficiency scores (3) * are on average much lower than those for the scores excluding the effects of the environment. Our results suggest that, once one accounts for the influence of environmental factors, authorities are, on average, spending between 3 and 10 per cent more than the feasible minimum to provide their library services. Note that the conclusions one draws about the potential for cost savings is quite different if one considers the effect of environmental factors to be on the technology of production (the minimum cost frontier) or on the efficiency with which authorities provide a given set of outputs.

Data Envelopment Analysis

We calculate six DEA models. The first three are the DEA analogue to model (1) above. That is, they include expenditure as the sole input and the four outputs of library services ([Q.sub.1A]-[Q.sub.3]). Models are calculated for the constant returns to scale model (model (5c)), the input-orientated variable returns to scale model (model (5vi)) and the output orientated variable returns to scale model (model (5vo)). The second three models include background variables as non-discretionary inputs (models (6c), (6vi) and (6vo)). Again we calculate constant returns to scale, input- and output-orientated variable returns to scale models. We use the scores obtained in models (5c), (5vi) and (5vo) as the basis for a second stage analysis of the effects of the background variables using a truncated (tobit) regression.

The results of the tobit analysis of the determinants of the gross efficiency scores for models (5c) (5vi) and (5vo) are presented in table 5. The pseudo-[R.sup.2] suggests that around half of the variance of the efficiency scores can be explained by the background variables. The signs, significance and relative sizes of the coefficients are similar to those for both of the stochastic frontier models with background variables included, which is reassuring.

The efficiency scores generated by the DEA are described in table 5. The rows denoted with an asterisk represent the background-adjusted efficiency scores based on the results of the tobit estimation. These are rescaled so that the minimum predicted value now lies on the frontier. What is most apparent from our analysis is the fact that the efficiency scores from the DEA are very different from those predicted by the SFA. The mean efficiency scores are much higher, and the standard deviations are also generally greater.

The descriptive statistics only tell part of the story. The pertinent question is whether the scores bear any relation to each other. Since we have seen that the results will be misleading if we exclude the influence of external factors, we concentrate on the models that include the background variables. There is a statistically significant correlation between all of the efficiency scores. This at least shows that we are not producing white noise and that our models do indeed tell us something about the performance of these local authorities. The two SFA models have very similar predictions to each other, suggesting that the way we choose to model the influence of background variables has a minimal influence on our results. Since we only have one input, it is little surprise to discover that the constant return to scale and input-orientated variable returns to scale DEA models are very highly correlated. However, the degree of correlation between the two methods of accounting for background variables in DEA is much lower. What is often considered important from a governance perspective is the ranking of organisations. They often appear to be used by the general public to assess performance, often more so than the indicators on which rankings are based. Such rankings are used to divide organisations into high and low performers and so it is less reassuring to discover that the (Spearman) rank correlation between the efficiency scores obtained from the different methods of analysis are lower than the (Pearson) correlations.

6. Discussion

We have considered the effect of using two alternative methodologies--SFA and DEA--that are very common in the analysis of efficiency in the private sector and have become increasingly so in the public sector. We have found that the predicted efficiencies we obtain from each of these methods bear some resemblance to each other in that they are correlated with each other, but are quite dissimilar in the level of inefficiency--and hence the potential for cost savings--that they imply. We have also found that if we ignore the circumstances in which the local authority operates, we will overestimate considerably the level of inefficiency in the authorities and the sector as a whole.

In order to put the implications of our results in terms of the potential for cost savings in some context, consider table 8. It is clear that the choice of method has a considerable impact on the scope for reducing local authority spending. The DEA methods imply much greater potential cost savings than the SFA estimates. Using the SFA method without accounting for the influence of exogenous factors, we would predict that local authorities could reduce their spending by 20 per cent whilst still maintaining the same level of service. However, much of this is due to the effect of the background variables, in particular the age structure and income of the population the authority services, and also possibly its density. The estimated effect of these variables depends on whether one considers them to be direct cost drivers or determinants of efficiency. This assumption has only a minor influence on the rankings of authorities but a larger one on the opportunity for reducing expenditure. Once we account for the environment in which the authority operates, the scope for cost savings may fall as low as 3 per cent.

The results of applying the DEA methodologies have very different implications for the reduction of local government expenditure. They suggest that, even once one accounts for the influence of the characteristics of the authorities' population, there is scope for expenditure to be reduced by over one-fifth.

Library services are an area of local authority provision where we can perhaps more closely observe the true outputs of local government because the measured activities represent the outputs fairly well. This is not to say there are not important aspects of library services that are excluded. Part of the service that one would expect from a library is aid in choosing and finding the appropriate items. In the private sector, people would be willing to pay a premium to libraries that increased the likelihood of them obtaining the book or recording they required. In the absence of such a measure, it is possible that what appears to be inefficiency could merely be certain authorities providing more of this kind of service. Indeed, one could imagine that at libraries that do not offer this service, the number of items issued was inflated due to people either taking more than they needed because they were unsure if a particular item would meet their requirements or returning to replace items that did not do so. The marginal impact of library services on people's lives is to provide access to the appropriate media.

The problems caused by omitted variables are not unique to library services. An analogy to one of the problems outlined above is the case of hospital readmission. If patients are readmitted to hospital because the treatment they received was substandard, this would appear as an additional output.

In conclusion, frontier techniques such as DEA and SFA provide a useful tool for the analysis of service provision in local government and in the public sector more generally, but they do not provide a silver bullet. As with all empirical techniques, they are only as good as the analyst that uses them. It is important to be aware of precisely what it is that is being produced by the organisation and understand the impact of the omission of indicators of particular aspects.

On the surface such methods appear to overcome the problem of aggregation. However, the question still remains as to whether it is appropriate to consider the question 'what are the appropriate weights?' as being purely (or rather merely) a technical one. The call to 'let the data do the talking' is in many ways an admirable one, but it must be understood precisely what the data are telling us. The data cannot tell us what the weights ought to be, merely what the implicit weights were to create the outcomes observed. Because of this such methods cannot tell us the true performance of an organisation without the specification of proper weights.

In the linear programming literature, this requires the definition and imposition of weights in the linear program. In the econometric cost function paradigm, of which the stochastic frontier methodology is a part, the marginal costs of producing an output are taken to be exogenous, defined by the production technology. Moreover, the levels of output themselves are taken to be exogenous. What do these assumptions imply for the measures of efficiency we calculate?

The efficiency scores calculated by a stochastic cost frontier are made up of technical efficiency--getting the most outputs for a given set of inputs, or using the least inputs to produce a given set of outputs--and allocative efficiency--choosing the inputs and outputs in the correct proportions, given their relative prices. There are a number of problems here for the public sector performance analyst.

Consider the process of how public sector organisations operate. First they have a constituency; this defines the scale of the operation. The size and make-up of this constituency determines the potential as well as actual demand for their services. For local authorities, many of the characteristics of their constituency are exogenous factors, the remainder they can only control in the longer term and are more correctly considered quasi-fixed as adjustment is not costless. Local authorities cannot move to other areas, or refuse to serve parts of their community. They may have some influence on geographical boundaries, but these are determined ultimately by central government. Over the longer term, the authority may influence the socio-economic makeup of its constituency, through its policy decisions, although there are boundaries to the extent to which a local authority's education strategy for example can influence the incomes of its population; much of any increase is likely to be captured by other authorities as well educated people move to more affluent areas.

All this suggests that the activities and outputs are to some extent exogenous. Nevertheless, authorities still have some control over both the quantity and the quality of the services they provide. The minimum leaving age and the population determine the number of pupils taught in compulsory school, but not how many exams they pass or the knowledge that they gain. Before we can consider the calculation of performance in the public sector to be a purely technical matter, what is needed is a technique that can better account for the level of control organisations have over their inputs and outputs and the environment in which they operate, and a set of weights that reflects the impact of these outputs consumers.

Table 1. Inputs and outputs

Variable Mean Std Dev

[Q.sub.1A] Number of books issued by the
 authority's libraries 2,678,115 2,335,234
[Q.sub.1B] Number of other items issued by
 the authority's libraries 217,276 196,061
[Q.sub.2] Number of visits to public
 libraries 2,004,709 1,624,734
[Q.sub.3] Number of books and recordings
 available in the council's
 libraries 389,163 264,880
E Net expenditure on libraries 4,217,533 2,817,699

Local authority types
LONDON London Borough 0.22 0.41
METRO Metropolitan Authority 0.25 0.44
UNITARY Unitary Authority 0.30 0.46
COUNTY County Council 0.23 0.42

Variable Min Max

[Q.sub.1A] Number of books issued by the
 authority's libraries 272,322 13,600,000
[Q.sub.1B] Number of other items issued by
 the authority's libraries 12,849 1,100,455
[Q.sub.2] Number of visits to public
 libraries 170,960 9,735,791
[Q.sub.3] Number of books and recordings
 available in the council's 33,193 1,665,005
 libraries
E Net expenditure on libraries 372,971 16,200,000

Local authority types
LONDON London Borough 0 1
METRO Metropolitan Authority 0 1
UNITARY Unitary Authority 0 1
COUNTY County Council 0 1

Table 2. Background variables

Variable London Metropolitan
 Borough Authority

KIDDENS Population per hectare 61.34 20.05
 (31.60) (9.65)
INCIID % of the population who 0.248 0.296
 are income deprived (0.106) (0.063)
YOUNGPOP % population aged 5-15 0.137 0.148
 (0.019) (0.008)
OLDPOP % population aged over 65 0.129 0.155
 (0.018) (0.013)
E2L % population with English 0.244 0.063
 as a second language (0.138) (0.067)

Variable Unitary County
 Authority Council

KIDDENS Population per hectare 18.6 2.555
 (15.61) (1.43)
INCIID % of the population who 0.232 0.168
 are income deprived (0.082) (0.036)
YOUNGPOP % population aged 5-15 0.146 0.138
 (0.012) (0.005)
OLDPOP % population aged over 65 0.151 0.171
 (0.029) (0.024)
E2L % population with English 0.053 0.018
 as a second language (0.081) (0.018)

Variable Total

KIDDENS Population per hectare 24.49
 (26.980)
INCIID % of the population who 0.237
 are income deprived (0.087)
YOUNGPOP % population aged 5-15 0.143
 (0.013)
OLDPOP % population aged over 65 0.152
 (0.026)
E2L % population with English 0.089
 as a second language (0.119)

Table 3. Results--stochastic frontier analysis

 (1) (2) (3)

CONSTANT 2.622 *** 3.87 *** 1.710 ***
 (0.661) (0.597) (0.483)
ln[Q.sub.1A] 0.166 0.375 *** 0.467 ***
 (0.133) (0.088) (0.079)
ln[Q.sub.1B] 0.093 ** 0.080 ** 0.069 **
 (0.045) (0.036) (0.032)
ln[Q.sub.2] 0.366 *** 0.231 *** 0.198 ***
 (0.098) (0.078) (0.075)
ln[Q.sub.3] 0.277 *** 0.183 *** 0.216 ***
 (0.056) (0.052) (0.046)
LONDON 0.172 *** 0.104 * 0.124 *
 (0.061) (0.061) (0.063)
METRO 0.110 ** 0.081 * 0.032
 (0.052) (0.045) (0.061)
COUNTY -0.070 0.150 -0.115 *
 (0.081) (0.092) (0.066)

Background variables Frontier Efficiency
 determinants

LNKIDDENS 0.032 0.201 **
 (0.025) (0.094)
YOUNGPOP -6.060 *** -7.259 ***
 (1.713) (2.631)
OLDPOP -3.868 *** -7.965 *
 (0.869) (2.376)
E2L 0.074 -0.486
 (0.221) (0.386)
INCIID 1.664 *** 2.397 ***
 (0.308) (0.548)
CONSTANT 1.046 *
 (0.616)
[gamma] 0.915 *** 0.752 *** 0.611 ***
 (0.080) (0.159) (0.163)
Log likelihood 19.268 66.787 62.109
Observations 139 139 139

Notes: Standard errors in parentheses. * significant at 10%;
** significant at 5%; *** significant at 1%. [gamma] is an indicator
of (although not exactly equal to) the proportion of the overall
variance explained by inefficiency.

Table 4. Descriptive statistics of efficiency scores (SFA)

 Min Max Mean s.d.

Model (1) 1.038 2.200 1.203 0.182
Model (2) 1.034 1.434 1.096 0.055
Model (3) 1.017 2.753 1.223 0.343
Model (3) * 1.012 1.245 1.028 0.025

Note: Model (3) * represents the net efficiency scores of model (3).

Table 5. Results: tobit estimation of DEA scores
(Model 5)

 CRS VRS-II VRS-10

LKIDDENS -0.001 -0.003 -0.009
 (0.045) (0.045) (0.041)
YOUNGPOP -11.572 *** -12.189 *** -11.117 ***
 (3.279) (3.227) (2.944)
OLDPOP -7.502 *** -7.043 *** -6.771 ***
 (1.714) (1.686) (1.538)
INCIID 3.145 *** 3.292 *** 2.916 ***
 (0.543) (0.541) (0.493)
E2L -0.069 -0.151 -0.192
 (0.436) (0.428) (0.390)
LONDON 0.145 0.253 0.266 *
 (0.157) (0.159) (0.144)
METROPOLITAN 0.145 0.221 * 0.200 *
 (0.122) (0.124) (0.113)
UNITARY 0.016 0.083 0.133
 (0.107) (0.110) (0.100)
CONSTANT 3.504 *** 3.367 *** 3.240 ***
 (0.612) (0.601) (0.548)
Pseudo [R.sup.2] 0.482 0.506 0.556
Log likelihood -53.043) -53.005 -42.154
Observations 139 139 139

Notes: Standard errors in parentheses. * significant at 10%;
** significant at 5%; *** significant at 1%.

Table 6. Descriptive statistics of efficiency scores (DEA)

 Min Max Mean s.d. Frontier
 authorities

Model (5c) 1 4.039 1.536 0.473 8
Model (5vi) 1 3.805 1.482 0.457 17
Model (5vo) 1 3.611 1.452 0.412 17
Model (5c) * 1 2.391 1.478 0.302 1
Model (5vi) * 1 2.096 1.326 0.236 1
Model (5vo) * 1 2.397 1.458 0.312 1
Model (6c) 1 4.039 1.412 0.496 42
Model (6vi) 1 3.805 1.345 0.460 54
Model (6vo) 1 2.649 1.157 0.214 54

Notes: Models marked with a * represent the net efficiency scores of
the respective models, based on the estimation presented in table 5
rescaled so that the minimum = 1.

Table 7. Correlations between efficiency scores

 Model (3) * Model (5c) * Model (5vi) *

Pearson correlations
Model (2) 0.875 0.815 0.787
Model (3) * 1 0.766 0.756
Model (5c) * 1 0.957
Model (5vi) * 1
Model (5vo) *
Model (6c)
Model (6vi)
Spearman rank
 correlations 1
Model (2) 0.894 0.570 0.509
Model (3) * 1 0.625 0.565
Model (5c) * 1 0.910
Model (5vi) * 1
Model (5vo) *
Model (6c)
Model (6vi)

 Model (5vo) * Model (6c) Model (6vi)

Pearson correlations
Model (2) 0.543 0.562 0.544
Model (3) * 0.580 0.567 0.555
Model (5c) * 0.781 0.603 0.600
Model (5vi) * 0.830 0.590 0.586
Model (5vo) * 1 0.377 0.378
Model (6c) 1 0.997
Model (6vi) 1
Spearman rank
 correlations
Model (2) 0.438 0.356 0.345
Model (3) * 0.475 0.578 0.574
Model (5c) * 0.853 0.499 0.507
Model (5vi) * 0.945 0.487 0.493
Model (5vo) * 1 0.388 0.394
Model (6c) 1 0.996
Model (6vi) 1

 Model (6vo)

Pearson correlations
Model (2) 0.539
Model (3) * 0.554
Model (5c) * 0.591
Model (5vi) * 0.574
Model (5vo) * 0.371
Model (6c) 0.993
Model (6vi) 0.996
Spearman rank
 correlations
Model (2) 0.352
Model (3) * 0.586
Model (5c) * 0.494
Model (5vi) * 0.469
Model (5vo) * 0.370
Model (6c) 0.990
Model (6vi) 0.994

Note: All correlations significant at the 1% level.

Table 8. Implied potential cost savings

 Potential cost savings

Model (1) 21%
Model (2) 9%
Model (3) * 3%
Model (5c) 30%
Model (5vi) 26%
Model (5c) * 29%
Model (5vi) * 21%
Model (6c) 22%
Model (6vi) 20%

NOTES

(1) For example, Ganley and Cubbin (1992) and O'Mahony et al. (2001); see Worthington and Dollery (2000) for an extensive review of earlier work on the topic.

(2) For example, it is not appropriate to compare the raw cost and outcome of social services in a deprived inner city area with the social services provided in a rural area, where the population may be older and more dispersed (Smith and Mayston, 1987).

(3) For example, consider three authorities using the same level expenditure who would, if it were not for random influences, be producing the same outputs. However, because of these unmeasured random influences (say different weather conditions) one is producing a little more of one output and another authority is producing a little less. This would mean that the frontier would be defined by the authority 'overproducing' and the other two authorities would appear inefficient and the efficiency scores for those authorities for whom these frontier authorities would be examples of best practice would be inflated. Moreover, because of the peculiar way in which DEA constructs the frontier, this may even make authorities elsewhere appear inefficient who are not so.

(4) It has been suggested, as a general rule of thumb, that it is advisable to ensure that the number of DMUs is at least three times the combined number of inputs and outputs (Banker et al., 1989). Smith (1997), using a Monte Carlo simulation framework, finds that a larger sample gets closer to the true efficiency. Also, as we have seen, efficiency scores increase with an increasing number of inputs and the more so with small samples. Hence large sample and parsimonious models can reduce the impact of the various problematic issues underlined above.

(5) Some studies have treated the environmental factors as normal inputs and outputs and they have been proportionally increased/decreased in the linear programme (Ganley and Cubbin, 1992; Smith and Mayston, 1987). However, the methodology described by Banker and Morey (1986) is more suitable to account for those factors that the local authority cannot modify.

(6) For more on this subject see Coelli, Rao and Battese (1998), chapter 8.

(7) In a previous version of this paper we tried to use user satisfaction measures, provided by the 2001 BVPI (Best Value Performance Indicators). The inclusion of such indicators would have provided an evaluation of the cost effectiveness of the local authority, rather than the most restrictive measure of cost efficiency. However, the impact of these indicators was never statistically significant and they were therefore dropped from the analysis.

(8) More information on how these indicators vary by authority type is presented in Stevens (2005b).

(9) These results are discussed in more detail in Stevens (2005a) along with comparison with further specifications of the SFA model.

(10) Note that the local authority type is not included in the set of background variables in the calculation of the net efficiency scores.

REFERENCES

Aigner, D.J., Lovell, C.A.K. and Schmidt, P. (1977), 'Formulation and estimation of stochastic frontier production function models', Journal of Econometrics, 6, pp. 21-37.

Banker, R., Charnes, A., Swarts, J., Cooper, W.W. and Thomas, D.A. (1989), 'An introduction to data envelopment analysis with some of its models and their uses,' Research in Governmental and Nonprofit Accounting, 5, pp. 125-63.

Banker, R.D. and Morey, R.C. (1986), 'Efficiency analysis for exogenously fixed inputs and outputs', Operations Research, 34(4), pp. 513-21.

Battese, G.E. and Coelli, T.J. (1995), 'A model for technical efficiency effects in a stochastic frontier production function for panel data', Empirical Economics, 20, pp. 325-32.

Bessent, A. and Bessent, W. (1980), 'Determining the comparative efficiency of schools through data envelopment analysis', Educational Administration Quarterly, 16(2), pp. 57-75.

Bjurek, H., Hjalmarsson, L. and Forsund, F.R. (1990), 'Deterministic parametric and nonparametric estimation of efficiency in service production: a comparison', Journal of Econometrics, 46, pp. 213-27.

Chilingerian, J.A. (1995), 'Evaluating physician efficiency in hospitals: a multivariate analysis of best practices', European Journal of Operational Research, 80, pp. 548-74.

Coelli, T.J., Rao, D.S.P. and Battese, G.E. (1998), An Introduction to Efficiency and Productivity Analysis, Boston, Kluwer Academic Publishers.

Coelli, T., Perelman, S. and Romano, E. (1999). 'Accounting for environmental influences in stochastic frontier models: with application to international airlines', Journal of Productivity Analysis, II, pp. 251-73.

Deaton, A. and Muellbauer, J. (1980), Economics and Consumer Behaviour, Cambridge, Cambridge University Press.

De Borger, B. and Kerstens, K. (1996), 'Cost efficiency of Belgian local governments: a comparative analysis of FDH, DEA, and econometric approaches', Regional Science and Urban Economics, 26(2), pp. 145-70.

Ganley, J.A. and Cubbin, J.S. (1992), Public Sector Efficiency Measurement: Applications of Data Envelopment Analysis, Amsterdam, North-Holland.

Huang, C.J. and Liu, J. (1994), 'Estimation of a non-neutral stochastic frontier production function', Journal of Productivity Analysis, 5, pp. 171-80.

Jesson, D. and Mayston, D. (1989), 'Measuring performance in authorities and schools', in Education and Training UK, Newbury, Berkshire, pp. 12-21.

Kumbhakar, S.C., Ghosh, S. and McGuckin, J.T. (1991), 'A generalized production frontier approach for estimating determinants of inefficiency in U.S. dairy farms', Journal of Business and Economic Statistics, 9, pp. 249-86.

McLean, I. (2005), 'Correlates of success and local government', OPSR/ESRC Public Services Programme Seminar, Oxford, 26 April.

Noble, M., Smith, G., Penhale, B., Wright, G., Dibben, C., Owen, T. and Lloyd, M. (2000), 'Measuring multiple deprivation at the small area level: the indices of deprivation 2000', Department of the Environment, Transport and the Regions, Regeneration Series, December.

Nunamaker, T. (1985), 'Using data envelopment analysis to measure the efficiency of non-profit organizations: a critical evaluation', Managerial and Decision Economics, 6(1), pp. 50-8

O'Mahony, M., Stevens, P.A., Vecchi, M. and Weale, M.R. (2001), 'Measuring the cost effectiveness of local authorities', report to the Department for Transport, Local Government and the Regions.

Reifschneider, D. and Stevenson, R. (1991), 'Systematic departures from the frontier: a framework for the analysis of firm inefficiency', International Economic Review, 32, pp. 715-23.

Salinas-Jimenez, J. and Smith, P. (I 996), 'Data envelopment analysis applied to quality in primary health care', Annals of Operations Research, 67, pp. 141-61.

Sexton, T.R., Silkman, R.H. and Hogan, A.J. (1986), 'Data envelopment analysis: critique and extensions', New Directions for Program Evaluation, 32, pp. 73-105.

Smith, P. (1997), 'Model misspecification in data envelopment analysis', Annals of Operations Research, 73, pp. 233-52.

Smith, P. and Mayston, D. (1987), 'Measuring efficiency in the public sector', Omega, 15(3), pp. 181-9.

Stevens, P.A. (2004), 'Accounting for background variables in stochastic frontier analysis', NIESR Discussion Paper Number 239.

--(2005a), 'The provision of library services by English local authorities', NIESR Discussion Paper Number 259.

--(2005b), 'What has the public sector ever done for us?', Presentation as part of the Economic and Social Research Council's Social Science Week, http://www.niesr.ac.uk/ pdf/ssw230605ppt.pdf.

Stone, M. (2002), 'How not to measure the efficiency of public services (and how one might)', Journal of the Royal Statistical Society, Series A, 165, pp. 405-34.

Vitaliano, D.F. (I 997), 'X-inefficiency in the public sector: the case of libraries', Public Finance Review, 25.6, pp. 629-43.

--(1998), 'Assessing public library efficiency using data envelopment analysis', Annals of Public and Cooperative Economics, 69(I), pp. 107-22.

Wang, H-J. and Schmidt, (2002), 'One-step and two-step estimation of the effects of exogenous variables on technical efficiency levels', Journal of Productivity Analysis, 18(2), pp. 129-44.

Worthington, A.C. (1999), 'Performance indicators and efficiency measurement in public libraries', Australian Economic Review, 32, pp. 31-42.

Worthington, A.C. and Dollery, B.E. (2000), 'Efficiency measurement in the local public sector: econometric and mathematical programming frontier techniques', Discussion Papers in Economics, Finance and International Competitiveness No. 78 , Queensland University of Technology, 1-21.

Philip Andrew Stevens, National Institute of Economic and Social Research. e-mail: [email protected]. This paper originates in work undertaken at the Institute on local authority cost effectiveness on behalf of the DETR, DTLR and ODPM. I would like thank Mary O'Mahony, Hiroko Plant, Michela Vecchi, Martin Weale, Willem de Boer and participants at the NIESR conference on 'Productivity and Performance in the Provision of Public Services' at the British Academy for help and comments. All mistakes remain the author's own.