Policy evaluation in a time of austerity: introduction.

Bryson, Alex ; Dorsett, Richard ; Portes, Jonathan 等

As Phil Davies (pp. R41-52) notes, the last government made a series of high-profile commitments to the idea of evidence-based policy. Central to this philosophy was a culture of evaluation whereby potential and actual policies were scrutinised and appraised in order to build and refine a knowledge base of 'what works'. While this was far from consistently applied, and the results of evaluations were not always comfortable reading for the government (for example, the very mixed results from the National Evaluation of Sure Start), on the whole there was a presumption that major social policy initiatives should be subject to independent evaluation.

The approach of the new government appears somewhat different. Although the Treasury has maintained its public commitment to evidence-based policymaking, research and evaluation budgets have been severely squeezed. Faced with a choice between service delivery and policy evaluation, it may be tempting to regard evaluation as a luxury that government can ill afford, at least in the short term.

We reflect briefly on why such a response may be shortsighted and offer some thoughts on how the approach to policy evaluation may adapt to these more straitened times. This is a theme that is taken up in a number of places by the papers in this issue.

The key reason why evaluation continues to be relevant in times of financial austerity is that it provides the evidence needed for informed decision-making. This is especially relevant when resources are scarce, a point made forcefully by Rachel Glennerster (pp. R4-14) in her review of policy evaluation in developing countries. Evaluation evidence, including the assessment of the costs and benefits of policy, offers the most direct insight to inform such decisions, as Haroon Chowdry (pp. R53-64) makes clear. Policies found through rigorous analysis to be both effective and cost-effective are strong candidates for investment. Furthermore, broader explorations--including qualitative analyses such as the one discussed by Heather Rolfe (pp. R65-76)--can provide a deeper understanding of the way policy is implemented and how and why a policy seems (not) to work, sometimes suggesting ways of adapting policies to better achieve their aims.

Evaluation can be costly but it is often a means of saving money. The recently-completed Employment Retention and Advancement (ERA) demonstration (Hendra et al., 2011) provides an edifying example of how a thorough evaluation can help guide policymakers away from wasteful spending. Davies and Chowdry outline the features of the intervention. The evaluation revealed that impacts for the first two years were large and significant but subsequently disappeared. Overall, the intervention failed the cost-benefit test for lone parents. The evidence cautions strongly against national roll-out of the policy (Chowdry). Evaluations of labour market programmes typically confine themselves to a shorter period over which to observe impacts. Had the ERA evaluation only observed outcomes for two years, say, the conclusion would have been very different, with significant financial implications.

Properly conducted, evaluations offer an objective assessment of the appropriateness of policy, thus enabling governments to deploy available funds in the most cost-effective manner. When the Work and Pensions Committee asked the employment minister, the Rt Hon Chris Grayling MP, why the Department for Work and Pensions (DWP) had decided not to commission an independent evaluation of the Future Jobs Fund (FJF) he cautioned of the "... danger in financially straitened times that we spend vast amounts of money evaluating things that we are no longer going to do." (1) The decision to discontinue the FJF was not based on evaluation evidence and, consequently, it is impossible to establish its cost-effectiveness. One argument for ending the FJF was that, from June 2011, the Work Programme has provided a unified mechanism for helping individuals move into sustained employment. Unfortunately, little allowance has been made for a robust quantitative evaluation of this new policy. The way it was introduced--nationwide and with no trialling--means that attempts to estimate its effectiveness face very difficult challenges. Given that both the FJF and the Work Programme involve the expenditure of billions of pounds of public money, this may be a false economy.

Convincing and impartial evaluations often require the involvement of external researchers. This can be achieved in various ways. Over the past decade, a number of commissioning departments have set up framework agreements as a means of channelling such input. Under this arrangement, external contractors compete to become framework members who can then be invited--on either a competitive or non-competitive basis--to submit proposals for discrete evaluation (or other) projects. Alternatively, departments can provide support for external centres. Stephen Machin and Sandra McNally (pp. R15-25) report on the contribution of the Centre for the Economics of Education (jointly funded, until recently, by the Department for Education and the Department for Business, Innovation and Skills) to the evaluation of schools policies in England. The Low Pay Commission (LPC) is an independent statutory body set up under the National Minimum Wage Act 1998. Tim Butcher (pp. R26-40) reflects on how the LPC's programme of evaluation research has influenced the evolution of the National Minimum Wage, while Justin van de Ven (pp. 77-89) reports on analyses exploring the potential impact of a new pension scheme for those on modest incomes, which was funded under the LPC's research programme. A recent interesting model for providing funding is the Education Endowment Foundation, a charity founded by the Sutton Trust and the Impetus Trust and responsible for using a large grant from the Department for Education to commission projects that will identify and evaluate ways of improving achievement levels among disadvantaged children.

Of course, the funds available to support evaluation are limited and there is a separate point about whether and how there might be scope to deliver robust evaluation evidence at lower cost. There are no universal laws here and each policy or pilot must be considered in its own right. However, it is important to keep in mind opportunities for efficiencies in this regard.

The ERA evaluation mentioned above was perhaps as comprehensive an evaluation as it is possible to imagine. The intervention itself was costly (involving substantial financial payments, training of special advisers and time-consuming delivery of support and guidance) but the evaluation involved a lengthy design phase, close monitoring of implementation, detailed manipulation of administrative data, large longitudinal survey data collection, extensive qualitative interviewing with recipients and providers of the service, careful estimation of the impact of the programme and, lastly, a cost-benefit analysis (which itself involved a substantial cost-collection exercise). Comprehensive and expensive evaluations such as this are unlikely to be commissioned in the foreseeable future. However, smaller evaluations can also provide high-quality evidence. As Rachel Glennerster points out, even random assignment evaluations, which are generally regarded as being more credible than alternative approaches, can be afforded in certain circumstances (often, it is not the randomisation itself which is costly). A good example is Dorsett and Smeaton (2008) which randomly assigned to older benefit claimants the requirement to participate fully in an active labour market programme. By tracking employment and benefit outcomes using administrative data, compelling evidence of programme effectiveness was provided. The point here is that the nature of the intervention was such that a much more limited evaluation was feasible and could provide reliable impact estimates at a fraction of the cost of the ERA demonstration. Whether such a pared-down approach is appropriate to other evaluations depends on a number of factors, including whether the programme is new (as opposed to an existing programme being extended to a new client group) and whether the outcomes of interest exist in readily-available administrative data. Certain conditions still must be met, of course. Rolfe reports here on a random assignment evaluation that was less successful.

One legitimate criticism often heard from politicians and senior policymakers is that evaluation evidence, while interesting, comes too late to influence policy as directly as it might. There is thus a premium on researchers working with policymakers to ensure timely feedback. Of course, certain types of evidence simply cannot be produced on short order. The ERA demonstration is again a good example; estimating longer-term impacts required observing what happens to people over a five-year period. However, some types of evidence can be provided more quickly. Qualitative evidence, for instance, can provide early insights and, importantly, highlight practical issues that have a clear relevance to the effectiveness of policy. Furthermore, in some cases (such as van de Ven in this issue), it may be possible to use economic models to estimate in advance what the effect of a particular intervention may be. Such an approach requires that the policy change is amenable to economic modelling. Changes to the tax/transfer system, for instance, involve an amendment to individuals' incentives to work that can be explored ex ante (see, for example, Brewer et al., 2006, for such an evaluation of Working Families' Tax Credit). However, there is an unavoidable fact that good evaluation evidence often takes time to materialise. While ministers may be impatient to see results, the quality and usefulness of the evidence base in the UK would be improved were this key point borne in mind, not least in the commissioning of research.

The academic and research community is interested in the effectiveness of government policies and wants to carry out evaluation studies even without government funding. Such studies are of clear public benefit and should be of interest to government. However, the data required to carry out evaluation studies are often subject to rigid access restrictions. This is entirely appropriate given the potential for disclosure when working with individual-level data. However, while issues of data security and anonymity remain paramount, this should not be an argument for stasis. Recently, there have been some improvements in facilitating safe access to individual-level datasets for research purposes. We very much welcome this and hope that it is the beginning of a trend that would extend to ever more government-held datasets. This offers a means of democratising research and allowing the data to achieve their potential as a public good. Taken together, the papers in this edition make a compelling case for robust, independent evaluation being at the heart of government efforts to deploy policies that improve citizens' welfare.

doi: 10.1177/002795011221900101

REFERENCES

Brewer, M., Duncan, A., Shephard, A. and Suarez, M. (2006), 'Did working families' tax credit work? The impact of in-work support on labour supply in Great Britain', Labor Economics December, 13(6), pp. 699-720.

Dorsett, R. and Smeaton, D. (2008), Mandating Intensive Activity Period for Jobseekers aged 50+: Final Report of the Quantitative Evaluation, DWP Research Report 500.

Hendra, R., Riccio, J., Dorsett, R., Greenberg, D., Knight, G., Phillips, J., Robins, P., Vegeris, S. and Walter, J., with Hill, A., Ray, K. and Smith, J. (2011), Breaking the Low-pay, No-pay cycle: Final Evidence from the UK Employment Retention and Advancement (ERA) Demonstration, DWP Research Report 765.

NOTE

(1) House of Commons Work and Pensions Committee, Youth Unemployment and the Future Jobs Fund, First report of 2010/-11, HC 472.

Alex Bryson, Richard Dorsett and Jonathan Portes *

* National Institute of Economic and Social Research. E-mail: [email protected]