Prove it! States offer many incentives to promote economic development, yet seldom evaluate their effectiveness.

Renault, Catherine S. ; Poole, Kenneth E.

States are ratcheting up their efforts to spur economic development and create jobs now that the effects of the Great Recession are fading. In some states that means becoming more business friendly. Arizona, Ohio and Wisconsin replaced their economic development agencies with public-private partnerships. Alabama, Georgia, Kansas and Nevada added new business-driven strategic planning to their economic development efforts.

For other states, it's creating new--or reorienting existing--incentives to attract new investments. In 2012, states added more than 180 new incentives and business assistance programs to some 1,700 already existing programs. But whether they will be (or have been) effective is unknown, and little information is collected to find out.

Policymakers often assume that economic development incentives result in job creation. But too few try to understand how well the incentives are working until something forces the issue.

Lawmakers Want Answers

For state legislators who want to create jobs in the midst of a long-term budget squeeze, all expenditures---even the most laudable of economic development efforts--should be under scrutiny. These lawmakers want answers, greater accountability and better information to evaluate whether investments in economic development actually do create jobs. Many say they're willing to make investments, but only if those investments generate a proven, adequate return. Most state economic development agencies report that they do, but many legislators question the veracity and quality of the available data.

Few states have all the necessary mechanisms for beginning the credible, rigorous evaluation of their economic development investments required to answer these questions. And, there's little incentive to create them, because data are seldom used to drive policymaking or allocate resources in any significant way.

[ILLUSTRATION OMITTED]

If an economic development program is shuttered, it's typically because of ideology or the need to cut budgets, not for lack of performance. And then it's often simply replaced with new iterations of the same incentive. Rarely are the results analyzed and used to adjust programs or policy.

Efforts Exist

Some evaluations are being done. Incoming governors often ask for audits to find economic development programs that are wasteful, being abused or not performing as expected. In Wisconsin, for example, an audit led to the reorganization and privatization of the state's economic development efforts.

Numerous academics have evaluated specific programs. For instance, Younghong Wu, a professor at the University of Illinois at Chicago, found that the existence of a state research and development tax credit has a positive and significant effect on the number of high-technology establishments in a state. For many years, independent evaluators have conducted regular studies of technology-based economic development programs.

Maine's annual evaluations of the state's entire portfolio of investments in research and development have consistently found a 10- to 14-fold return on investment for the state. Similarly, the Ben Franklin Partnership in Pennsylvania has performed annual assessments of state investments in start-up companies and found that new state tax revenues received represented a 3.36:1 return to the state.

When the political environment drives the need to investigate or defend a program, states are more apt to sponsor evaluations. For example, the alleged malfeasance by one film office director in Iowa and long-simmering concerns among academics and legislators alike about the value of film tax credits spurred a spate of film tax credit studies in other states, including New Jersey and New Mexico.

Pennsylvania recently produced an assessment that focused narrowly on how well its incentive programs aided manufacturing, an important part of the state's economic base. Connecticut, as part of a tax reform effort in 2010, required an assessment of its tax credits and abatements because of lawmakers' concerns about the value of many tax expenditures. In this environment, program managers tend to view evaluations as a lose-lose situation-more an attempt to identify missteps rather than an effort to guide program improvements.

Often, evaluations are not even considered in making policy changes. The 2010 Special Council of Tax Reform and Fairness for Georgians, for example, was one of several attempts in states to revise their tax codes, including eliminating some or all economic development tax credits and incentives. Yet, few studies or evaluations were conducted as part of these proceedings.

The Hindrances

The Pew Charitable Trusts recently reviewed more than 600 documents from state agencies or legislative committees, and conducted 175 interviews, focusing on evaluation efforts completed between 2007 and 2011. They concluded that half the states have yet to take "basic steps" to evaluate the effectiveness or efficacy of their tax incentives. Pew concludes that evaluations of tax incentives could shape policy choices, measure their economic impact and help determine whether the credits are achieving the state's goals.

States face several challenges, however, one being a lack of evaluation capacity. The number of program evaluation staff varies widely among the states, but only 55 percent of those are professionals who can perform significant evaluation studies. In the past three years, 84 percent of these offices have had their budgets reduced, with an average loss of eight positions. Their role has expanded to include program audits and other financial reporting, in addition to evaluation.

Another limitation is a failure to see the big picture. Individual economic development programs are created to fix one problem or to take advantage of a specific opportunity, but are often not recognized as part of an overall economic development strategy. So states rarely take the opportunity to look comprehensively at their economic development programs--as a portfolio of state investments that recognizes different programs have different economic purposes.

Focusing on evaluating individual programs, rather than on the entire economic strategy, is problematic in many ways. Business firms can and do use more than one service or program, so results can be double-counted or overstated when evaluations are conducted separately.

State economic development efforts can and should have several goals, including short-term and long-term ones. Measuring and comparing individual program elements often pit programs with different goals against one another, so there is a constant push and pull between efforts to recruit new firms and keep existing ones (often a short-term strategy) and innovation-based development (which tends to focus on the longer-term). Tension also exists between programs aimed at supporting entrepreneurship (long-term) versus those assisting existing businesses (short-term).

Taking a comprehensive look at a state's entire economic development activity is rare, and not part of an ongoing process of continuous improvement or accountability. But it's not unheard of. Both Arizona and Maine undertook a portfolio approach in their recent evaluations. Maine's effort was an extension of the state's long-standing research and development evaluation, while Arizona's was prompted by a new governor in a state that had rarely needed economic development before the Great Recession.

Another concern is that the limited range of research methods typically used to assess states' economic development plans do not really help us understand whether the public investment resulted in a firm taking an action it may not have done otherwise. Predominantly, these evaluations rely on surveys of program users, asking participating firms about their job retention and creation activities, the new revenues generated, or about how much other investments were leveraged as a result of the assistance they received.

Researchers also have used case studies describing a firm's activities or actions in great detail. Both these approaches are biased to the extent that they rely on the firm receiving help or the service provider to express the value of a program rather than on more sophisticated methodologies to assess cause and effect. As a result, it often is difficult to prove that the outcome observed is a direct result of the assistance received.

In contrast, academic studies tend to use more sophisticated methods capable of teasing out thorny questions of cause and effect. But economic development evaluators must be trained on how to interpret these more complex methods, or they can have trouble understanding the results and even more trouble explaining these rigorous evaluations to stakeholders and legislators. When an evaluation method is too complex, stakeholders often challenge the research, second-guessing the researcher's assumptions, the data used and especially the conclusions. Furthermore, these rigorous studies require more time, money and expertise than are readily available to most states.

So, a wide chasm exists between what economic development practitioners are doing, what scholars think they should do, and what politicians and other stakeholders believe could be done--both in terms of economic development incentives and evaluation methodologies.

Look for the Good

Changing the motivation for evaluation from punishment to continuous improvement is another challenge. Will policymakers and practitioners use the data and analysis from an evaluation to improve the program or reward success? Is the primary purpose to hype program successes? Or is the evaluation being done largely to eliminate the program or justify preconceived criticisms that undermine the willingness of all involved to participate in any further evaluations?

Most managers see little good resulting from an evaluation, so they fall back on a lack of data and limited resources as a rationale for why the information on their effectiveness is so scant. Advocacy groups and external stakeholders with differing interests can confuse this situation even further.

For example, programs such as the Kansas Technology Enterprise Corporation, the Ben Franklin Partnership of Pennsylvania, the Grow Florida economic gardening initiative, and many others were eliminated or pared down, despite the assertion from the economic development or academic community that they represent "best practices."

As evidence-based decision making becomes an increasingly important goal of legislators, economic developers cannot ignore the call for greater accountability. To make evaluation a more integral part of all economic development practices, evaluations need to balance scholarly design with ease of use and application. It won't be easy, but despite its many challenges, rigorous evaluation is not only desirable, but necessary.

New Initiative to Help States

The Pew Charitable Trusts and Center for Regional Economic Competitiveness are launching a program in 2014 called the Business Incentives Policy and Practice Initiative. Pew and CREC staff will work with states, including legislators, over the next year and a half to:

* Identify effective ways to adopt and manage economic development incentive policies and practices,

* Improve the participating states' ability to collect and analyze data from evaluations of incentive investments and report the findings,

* Develop national standards and best practices that can become roadmaps to guide other states that are adopting or adapting economic incentives.

For more information, contact Ken Poole at [email protected].

The Recommendations

States often view program evaluation as something that should be done only when absolutely necessary. But effective evaluation is a process, not an event. Successful evaluations rely on four key foundations.

1. Policy needs to be based on clear and measurable goals. Many economic development programs are created to address specific problems that are ill-defined or poorly documented. Clear, well-articulated and measurable goals in the enabling legislation for new programs make it easier to decide whether progress is being made.

2. Evaluation should be an integral element of any economic development effort. Data collection takes time and consideration. Funding should be sufficient for rigorous, third-party evaluations, since program managers may not have the necessary training or objectivity to adequately perform the task.

3. Objective, appropriate analysis should guide the decision on whether a program is achieving certain performance goals and also whether those goals are relevant to state policy priorities.

4. Data collection should be integrated into all program design, including gathering information on the most influential metrics so policymakers can determine if the goals of the program are being met. Interactions with firms should be treated as opportunities for collecting well-crafted and complete data.

Dr. Kenneth Poole is chief executive officer for the Center for Regional Economic Competitiveness and executive director of the Council for Community and Economic Research. Dr. Catherine Renault is the principal and owner of Innovation Policyworks and an assistant research professor at the University of Maine.