Good news for New Orleans: early evidence shows reforms lifting student achievement.

N. Harris, Douglas

What happened to the New Orleans public schools following the tragic levee breeches after Hurricane Katrina is truly unprecedented. Within the span of one year, all public-school employees were fired, the teacher contract expired and was not replaced, and most attendance zones were eliminated. The state took control of almost all public schools and began holding them to relatively strict standards of academic achievement. Over time, the state turned all the schools under its authority over to charter management organizations (CMOs) that, in turn, dramatically reshaped the teacher workforce.

A few states and districts nationally have experimented with one or two of these reforms; many states have increased the number of charter schools, for example. But no city had gone as far on any one of these dimensions or considered trying all of them at once. New Orleans essentially erased its traditional school district and started over. In the process, the city has provided the first direct test of an alternative to the system that has dominated American public education for more than a century.

Dozens of districts around the country are citing the New Orleans experience to justify their own reforms. In addition to being hailed by Democratic president Barack Obama and Louisiana's Republican governor, Bobby Jindal, parliamentary delegations from at least two countries have visited the city to learn about its schools.

[ILLUSTRATION OMITTED]

[FIGURE 1 OMITTED]

The unprecedented nature of the reforms and level of national and international attention by themselves make the New Orleans experience a worthy topic of analysis and debate. But also consider that the underlying principles are what many reformers have dreamed about for decades--that schools would be freed from most district and union contract rules and allowed to innovate. They would be held accountable not for compliance but for results.

There is clearly a lot of hype. The question is, are the reforms living up to it? Specifically, how did the reforms affect school practices and student learning? My colleagues and I at the Education Research Alliance for New Orleans (ERA-New Orleans) at Tulane University have carried out a series of studies to answer these and other questions. Our work is motivated by the sheer scale of the Katrina tragedy and the goal of supporting students, educators, and city leaders in their efforts to make the city's schools part of the city's revitalization effort. The rest of the country wants to know how well the New Orleans school reforms have worked. But the residents of New Orleans deserve to know. Here's what we can tell them so far.

Before the Storm

Assessing the effects of this policy experiment involves comparing the effectiveness of New Orleans schools before and after the reforms. As in most districts, before Hurricane Katrina, an elected board set New Orleans district policies and selected superintendents, who hired principals to run schools. Principals hired teachers, who worked under a union contract. Students were assigned to schools based mainly on attendance zones.

The New Orleans public school district was highly dysfunctional. In 2003, a private investigator found that the district system, which had about 8,000 employees, inappropriately provided checks to nearly 4,000 people and health insurance to 2,000 people. In 2004, the Federal Bureau of Investigation (FBI) issued indictments against 11 people for criminal offenses against the district related to financial mismanagement. Eight superintendents served between 1998 and 2005, lasting on average just 11 months.

This dysfunction, combined with the socioeconomic background of city residents--83 percent of students were eligible for free or reduced-price lunch--contributed to poor academic results. In the 2004-05 school year, Orleans Parish public schools ranked 67th out of 68 Louisiana districts in math and reading test scores. The graduation rate was 56 percent, at least 10 percentage points below the state average.

As a result, some reforms were already under way when Katrina hit in August 2005. The state-run Recovery School District (RSD) had already been created to take over low-performing New Orleans schools. The state had appointed an emergency financial manager to handle the district's finances. There were some signs of improvement in student outcomes just before the storm, but, as we will see, these were relatively modest compared with what came next.

A Massive Experiment

After Katrina, state leaders quickly moved almost all public schools under the umbrella of the RSD, leaving the higher-performing ones under the Orleans Parish School Board (OPSB). Gradually, the RSD turned schools over to charter operators, and the teacher workforce shifted toward alternatively prepared teachers from Teach for America and other programs. So new was the system that a new name was required--longtime education reformer Paul Hill called it the "portfolio" model.

Researchers often refer to such sudden changes as "natural experiments" and study them using a technique called "difference-in-differences." The idea is to first take the difference between outcomes before and after the policy, in the place where it was implemented--the treatment group. This first difference is insufficient, however, because other factors may have affected the treatment group at the same time. This calls for making the same before-and-after comparison in a group that is identical, except for being unaffected by the treatment. Subtracting these two--taking the difference of the two differences between the treatment and comparison groups--yields a credible estimate of the policy effect.

[ILLUSTRATION OMITTED]

[ILLUSTRATION OMITTED]

We have carried out two difference-in-differences strategies:

1) Returnees only. We study only those students who returned to New Orleans after Hurricane Katrina. The advantage of this approach is that it compares the same students over time. One disadvantage is that it omits nonreturnees. Also, we can only study returnees over a short period of time--after 2009, they no longer have measurable outcomes to study.

2) Different cohorts. We consider the achievement growth of different cohorts of students before and after the reforms--for example, students in 3rd grade in 2005 and students in 3rd grade in 2012. The advantages here are that we can include both returnees and nonreturnees, and we can use this strategy to study longer-term effects. But the students are no longer the same.

In both strategies, the New Orleans data set includes all publicly funded schools in the city, including those governed by the district (OPSB), since all public schools were influenced by the reforms. The main comparison group includes other districts in Louisiana that were affected by Hurricane Katrina, and by Hurricane Rita, which came soon afterward. This helps account for at least some of the trauma and disruption caused by the storms, the quality of schools students attended in other regions while their local schools were closed, and any changes in the state tests and state education policies that affected both groups.

Effects on Average Achievement

Figure 1 shows the scores for each cohort, separately for New Orleans and the matched comparison group. The scores cover grades 3 through 8, are averaged across subjects, and are standardized so that zero refers to the statewide mean. The first thing to notice is that before the reforms, students in New Orleans performed far below the Louisiana average, at about the 30th percentile statewide. Students from the comparison districts also lagged behind the rest of the state, but by a lesser amount. The New Orleans students and the comparison group were moving in parallel before the reforms, however, suggesting that our matching process produced a comparison group that is more appropriate than the state as a whole.

The performance of New Orleans students shot upward after the reforms. In contrast, the comparison group largely continued its prior trajectory. Between 2005 and 2012, the performance gap between New Orleans and the comparison group closed and eventually reversed, indicating a positive effect of the reforms of about 0.4 standard deviations, enough to improve a typical student's performance by 15 percentile points.

The estimates we obtain when we focus just on returnees are smaller and often not statistically significant, although the discrepancies are predictable: first, the returnees were probably more negatively affected by trauma and disruption; second, creating a new school system from scratch takes time, so we would expect any effects to be larger in later years; and third, the effects of the reforms seem more positive in early elementary grades, and the returnees were generally in middle school when they returned. Even so, the combination of analyses suggests effects of at least 0.2 standard deviations, or enough to improve a typical student's performance by 8 percentage points.

But there is still the possibility that what appear to be reform effects are actually the result of other factors.

Addressing Additional Concerns

The goal of any analysis like this is to rule out explanations for the changes in outcomes other than the reforms themselves. Our main comparisons deal with many potential problems, such as changes in state tests and policies. Here we consider in more depth four specific factors that could bias the estimated effects on achievement: population change, interim school effects, hurricane-related trauma and disruption, and test-based accountability distortions.

Population change. Hurricane Katrina forced almost everyone to leave the city. Some returned and some did not. The most heavily flooded neighborhoods were (not coincidentally) those where family incomes were lowest, and people in these neighborhoods returned at much lower rates than people who lived in other parts of the city. Given the strong correlation between poverty and student outcomes, this could mean that higher test scores shown in Figure 1 are driven not by the reforms but by schools serving more-advantaged students.

Observers have pointed out that the share of the student population eligible for free or reduced-price lunch (FRL) actually increased slightly in New Orleans after the storm. But there are many reasons not to trust FRL data. For example, they reflect crude yes/no measures and are unlikely to capture extreme poverty of the sort common in New Orleans. Also, what really matters here is not whether poverty increased in New Orleans, but whether poverty increased more than in the comparison group. Therefore, in addition, we gathered data from the U.S. Census, which measures changes in income and the percentages of the population with various levels of education. We also carried out the difference-in-differences analysis in these demographic measures to understand the changes in New Orleans relative to the matched comparison group of hurricane-affected districts, and then simulated the effect of changes in family background characteristics on test scores using data from the federal Early Childhood Longitudinal Study.

[ILLUSTRATION OMITTED]

We also examined pre-Katrina characteristics to see whether the returnees were different from nonreturnees and found that returnees did have slightly higher scores. In fact, we come to the same conclusion in both analyses: the expected increase in student outcomes after the hurricanes due to population change is no more than 0.02 to 0.06 standard deviations, or about 10 percent of the difference-in-differences estimates in Figure 1.

Interim school effects. Some of the changes in student learning may reflect neither the prestorm nor poststorm quality of New Orleans schools, but the performance of schools that students briefly attended outside the city after the evacuation. Other research on these students by Dartmouth economist Bruce Sacerdote suggests that New Orleans evacuees experienced larger improvements in school quality than evacuees from other districts.

Trauma and disruption. Any benefit of having good interim schools might be offset by the trauma and disruption of the storm itself and its aftermath. The majority of New Orleans returnees probably knew someone among the nearly 2,000 people who died in the Katrina aftermath. Also, almost all students experienced significant disruption, moving to unfamiliar neighborhoods and schools for extended periods. Reports of post-traumatic stress disorder remain common.

It is difficult to isolate trauma and interim school effects, but we can estimate the combination of the two. A study by the RAND Corporation of students from Louisiana districts affected by the hurricane suggests that these two factors had a short-term net negative effect on evacuees' performance of 0.03 to 0.06 standard deviations. Our analysis suggests that the negative influence is even larger for New Orleans students, most likely because of the more extensive destruction in the city compared with most other areas along the state's coast. Thus, at least in the years just after the reforms, the factors pushing student outcomes down were at least as large as the population changes pushing them up.

Test-based accountability distortions. One key part of the New Orleans reforms was the idea that the state would shut down schools within three to five years if they did not generate a high enough School Performance Score, a measure based on test scores and graduation rates. Prior research suggests that such intensive test-based accountability can lead to behaviors, such as teaching to the test, that increase scores without improvements in underlying learning or through reduced learning in nontested subjects.

To address this problem, we estimate effects separately by subject, recognizing that the stakes attached to math and language scores were roughly double the stakes for science and social studies scores during the period under analysis. Also, the state's social promotion policy raises the stakes for students in grades 4 and 8. We find no evidence that the size of effects varied systematically with the stakes attached to the subjects or grades. However, it is hard to rule out other potential test-based accountability distortions with our data.

As further evidence, we considered descriptive information on nontest outcomes. State government reports indicate that, relative to the state as a whole, the New Orleans high school graduation rate and college entry rate (among high school graduates) rose 10 and 14 percentage points, respectively.

So, in theory, there are many challenges to estimating the effects of the New Orleans package of school reforms. The combined effect of these alternative factors on long-term achievement gains appears small, however, especially when compared with our initial estimate of the reform effects.

There is a clear pattern across these methods. The estimates are consistently within the same range, and even the lower end of that range suggests large positive effects.

Equity of Outcomes

Public schools exist to ensure that all children have an opportunity to succeed in life. Thus we consider not only the average effects of the reform package, but also whether the most-disadvantaged students benefited.

We first define equity in terms of how New Orleans, as an urban district, performed relative to districts serving more-advantaged students. Both before and after the reforms, at least 80 percent of New Orleans students were minority or eligible for FRL. It is therefore noteworthy that the reforms brought the city's students near to the state average on a wide range of academic outcomes (see Figure 1).

It is also important to consider the distribution of effects within the city, and here the results are more mixed. All major subgroups of students--African American, low-income, special education, and English Language Learners (ELL)--were at least as well-off after the reforms, in terms of achievement. Critics of charter schools express concern about possible increases in racial isolation (some would say "segregation"). Among all of the various subgroups we considered, only Hispanic students seem to have experienced increases in isolation.

There have also been concerns about schools unfairly targeting low-income and African American students in disciplinary decisions. While we have not yet studied whether any student groups have been specifically targeted, we can say that the number of suspensions and expulsions has dropped since the reforms, for African American students and others alike.

There are a few less-positive signs, however. In our analysis of what families look for when choosing schools, we found that the lowest-income families place less weight on the School Performance Score than other families. Their circumstances may lead them to focus more on practical considerations such as distance to school and extended hours (to avoid extra child-care costs). Similarly, in our analysis of student mobility, we see that low-scoring students are less likely than high-scoring students to migrate toward schools with high scores. Finally, up until a few years ago, principals reported cherry-picking students by, for example, counseling out students deemed poor fits and holding invitation-only events to attract certain students.

Given the large improvements in average outcomes in a district that is almost entirely low-income and minority, and the mixed evidence on other equity indicators, it would be hard to say the outcomes from the New Orleans reforms are inequitable relative to what came before them. That said, they were highly inequitable to start with, and there is clearly room for improvement.

What Really Changed?

To help improve the schools going forward, it is important to know how school practices and other intermediate outcomes changed. In a series of 15 ongoing studies, my collaborators at ERA-New Orleans and I have examined four main components of the reforms: choice and competition, teachers and leaders, charters and CMOs, and test-based accountability.

Some of the reform effect may be driven by parental choice and competition. The supply of schools in New Orleans appears highly differentiated. Some schools specialize in math and science, others in the arts. Some schools offer language immersion programs, while other schools have fairly traditional curricula. Some schools have selective admissions, while others are open enrollment or seek diverse student bodies. We also find that New Orleans families diverge in their schooling preferences, so having this degree of differentiation in schooling options is likely to help match what families want with what schools offer (see "The New Orleans OneApp," features, and "Many Options in New Orleans Choice System," research, Fall 2015).

[ILLUSTRATION OMITTED]

It is still unclear, however, whether these changes in the market have contributed to the improvements in student outcomes. Even supporters of the reform efforts sometimes bristle when I use the word "market" and "competition" to describe the new system. Instead, they point to two other parts of the reform package: the authority of the state to close schools and the authority schools have over their teaching staffs.

Sixteen New Orleans schools have been completely closed and another 30 have been taken over in some fashion by either the RSD or OPSB--a large number in a city that has only about 90 public schools in total. Consistent with written state policies, we find that the School Performance Score is the strongest measurable driver of closure and renewal decisions. Moreover, in finding CMOs to open new schools and take over old ones, the RSD has preferred those with a track record of academic success.

[ILLUSTRATION OMITTED]

School leaders in New Orleans talk frequently about how critical flexibility in personnel management is to their overall school success. Free of state and local mandates and constraints from union contracts, leaders reopening schools after the storm could hire anyone they wanted, including uncertified teachers, and dismiss teachers relatively easily. As CMOs took over, more of the teacher workforce came from alternative preparation programs such as Teach for America and The New Teacher Project. Consistent with some other studies, analyses commissioned by the state suggest that graduates of these programs contribute more to student achievement than graduates of traditional preparation programs.

The combination of policies had two types of effects on the teacher workforce. First, the percentages of teachers with regular certification and with 20 or more years of experience dropped by about 20 points each. Also, due to both the short-term commitments of some alternatively certified teachers and school autonomy over personnel, the teacher turnover rate nearly doubled. The fact that such large improvements in student learning could be achieved with these common metrics going in the "wrong direction" reinforces a common finding in education research: teacher credentials and turnover are not always good barometers of effectiveness.

Finally, we turn to a topic that is not typically thought of as part of the reform package but may be an essential component: costs and resources. Our analysis suggests that from 2004-05 to 2011-12, the same years covered by our achievement analysis, total public schooling expenditures per student increased by $1,000 in New Orleans relative to other districts in the state. Some of the increase probably reflects one-time start-up costs of new schools, and we are working to understand what share falls in that category. Regardless, there is wide agreement that the reforms did not come cheap.

None of this really tells us exactly which of the factors drove the improvements in student outcomes--no doubt they are interconnected--but it does provide some indication of how schools and families responded to the policy shift.

[ILLUSTRATION OMITTED]

Implications for New Orleans

These findings have important implications for the New Orleans public schools, the many other urban districts pursuing the portfolio approach, and for the state and federal policies--especially test-based and market-based accountability --from which the New Orleans reforms emerged.

For New Orleans, the news on average student outcomes is quite positive by just about any measure. The reforms seem to have moved the average student up by 0.2 to 0.4 standard deviations and boosted rates of high school graduation and college entry. We are not aware of any other districts that have made such large improvements in such a short time.

The effects are also large compared with other completely different strategies for school improvement, such as class-size reduction and intensive preschool. This seems true even after we account for the higher costs. While it might seem hard to compare such different strategies, the heart of the larger school-reform debate is between systemic reforms like the portfolio model and resource-oriented strategies.

With the possible exception of distortions from test-based accountability, which are harder to identify, the reforms managed to avoid most of the side effects that many feared. But our findings also suggest areas of potential improvement. While the reforms have been successful on some dimensions of equity, it seems necessary to do more to ensure that all groups within the city benefit. All types of public school systems struggle with providing equitable access to quality schools, and the New Orleans system is no exception.

Implications for the Nation

Unfortunately, the effects of even the most successful programs are often not replicated when tried elsewhere, and there are good reasons to think the conditions were especially ripe for success in New Orleans:

There was nowhere to go but up. Pre-Katrina, the New Orleans public school system was highly dysfunctional, and student test scores made it the second-lowest-ranked district in the second-lowest-ranked state in the country.

New Orleans is an attractive city for young educators. The national response to the hurricane aftermath was heartening, and for many young people, contributing to the rebuilding effort became a calling. Later, as the reform effort took hold, New Orleans also became the nation's epicenter of school reform, an ideal place for aspiring reform-minded educators. Because the city is smaller than many urban districts, school leaders could be very selective in choosing from the pool of educators who wanted to come and work there.

The effects might also be smaller, at least in the short run, if the reforms were adopted on a statewide basis, because the reform is dependent on a specific supply of teachers. It seems difficult enough attracting effective teachers and leaders to work long hours at modest salaries in New Orleans; doing it throughout Louisiana is unrealistic without a major change in the educator labor market. Nonetheless, it would be a mistake to dismiss the relevance of the New Orleans experience for others. It is relevant precisely because it is so unusual. The city's reforms force us to question basic assumptions about what K-12 publicly funded education can and should look like.

There is more to the debate than we can cover here, including fundamental philosophical issues about whose objectives and values should count in making schooling decisions. But there is also wide agreement that the academic outcomes considered here are important, so learning how much the reforms contribute to changes in academic measures should also be a key part of the conversation. Better understanding of all the elements of the reforms is something we owe to the city, its children, and everyone who suffered and perished in this terrible tragedy.

by DOUGLAS N. HARRIS

Douglas N. Harris is professor of economics at Tulane University and founder and director of the Education Research Alliance for New Orleans. The research cited here is coauthored with others on the ERA-New Orleans research staff (Paula Arce-Trigatti, Nathan Barrett, Lindsay Bell Weixler, Christian Buerger, Matthew Larsen, Jane Arnold Tincove, Whitney Ruble, Robert Santillano, and Jon Valant) and members of the ERA-New Orleans National Research Team (Huriya Jabbar, Jennifer Jennings, Spiro Maroulis, Katharine Strunk, Patrick Wolf, and Ron Zimmer). All errors are the author's.