NEPC Resources on School Reform and Restructuring
NEPC Review: The Academic and Behavioral Consequences of Discipline Policy Reform: Evidence from Philadelphia (Thomas B. Fordham Institute, December 2017)
A report from the Fordham Institute investigated the impact of a reform in the School District of Philadelphia that eliminated suspensions for certain low-level misbehaviors. The report considered whether the policy change was associated with any of the following: (a) district-wide out-of-school suspension rates, (b) academic and behavioral outcomes for students (looking separately at students who had a record of prior suspensions and those with no prior suspensions), and (c) racial disparities in suspensions. While the report concluded that the reform was a failure, the actual results were mixed, with the positive trends for students who were earlier suspended being much stronger in magnitude than evidence of negative outcomes for students who were not. A strength of the report is the use of advanced statistical methods and a longitudinal dataset to answer the questions of interest. However, the report is plagued by logical fallacies, overly simplified interpretations of findings, and inflammatory language. Moreover, the report uses misleading causal (“consequences”) language in the title and to describe study results, even though the study design is limited by unmeasured confounding factors and inappropriate comparison groups. Thus, while the analyses upon which the report is based have some technical merits, the narrative seems more of an attempt to advance a political agenda opposed to the reform studied than to improve understanding of complex policy issues.
NEPC Review: School District Reform in Newark (National Bureau of Economic Research, October 2017)
In 2010, Facebook CEO Mark Zuckerberg announced, with great fanfare, a $100 million challenge grant for the support of a series of reforms in Newark, NJ schools. The two reports reviewed here are the first attempt at a comprehensive assessment of the impact of the Zuckerberg donation on student achievement. As such, the reports, which were written by a team of economists, have garnered significant attention. The reviewers evaluate the reports and their potential contribution to education policy research.
NEPC Review: Lights Off: Practice and Impact of Closing Low-Performing Schools (Center for Research on Education Outcomes (CREDO), August 2017)
This report provides an extensive analysis based on the most comprehensive dataset ever assembled for school closure research, including 1,522 low-performing schools that were closed across 26 states between 2006 and 2013. The report finds that even when holding constant academic performance, schools were more likely to be closed if they enrolled higher proportions of minority and low-income students. It also finds test score declines, relative to the comparison group, for two groups of students displaced by closures: those who transferred to schools with a prior record of relatively lower test-score performance and those who transferred to schools with equivalent past test-score performance. The slightly less than half of students who transferred to higher performing schools showed academic improvement relative to their matched peers. In general, although the reviewers found this to be a careful and rigorous study, they see a few missed opportunities. First, the report’s focus on some tenuous analyses (involving pre-closure transfers) obscures its most important findings – disproportionality in school closures and inadequate numbers of higher quality receiving schools, leading to performance declines for most. Second, the reviewers are concerned about statistical modeling choices and matching challenges that may threaten the validity of subgroup analyses (charter school students). Finally, the reviewers would have liked to see the report acknowledge the inescapable moral dimensions of school closure: The communities most likely to be negatively affected are unlikely to have participated in closure decisions.
NEPC Review: Leveraging ESSA to Support Quality-School Growth (Thomas B. Fordham Institute and Education Cities, March 2017)
A report offers a how-to guide for reform advocates interested in removing communities’ democratic control over their schools. The report explains how these reformers can influence states to use the Every Student Succeeds Act (ESSA) Title I school improvement funds to support a specific set of reforms: charter schools, state-initiated turnarounds, and appointment of an individual with plenipotentiary authority over districts or schools. While the report acknowledges that the research evidence on the effectiveness of these reforms as school improvement strategies is limited, it uses a few exceptional cases to explain how advocates seeking to influence the development of state ESSA plans can advance them anyway. As this review explains, support for the effectiveness of these approaches is simply too limited to present them as promising school improvement strategies. The report omits research that evaluates the models relative to other school reform initiatives, and it fails to take into account the opportunity costs of pursuing one set of policies over another. It also relies on test-score outcomes as the sole measure of success, thus ignoring other impacts these strategies may have on students and their local communities or the local school systems where they occur. For these reasons, policymakers, educators and state education administrators should be wary of relying on this report to guide them as they develop their state improvement plans and consider potential strategies for assisting low-performing schools and districts.
Review of Reimagining Learning: A Big Bet on the Future of American Education
Philanthropic involvement in K-12 education is growing, and it increasingly shapes the direction of reforms pursued throughout the country. A recent report from the NewSchools Venture Fund offers a thought experiment on how philanthropists can make a “big bet” over the next decade on innovative schools—a broad category that generally includes schools with a high degree of education technology use and so-called personalized approaches to learning that likely utilize digital platforms. Unfortunately, the report fails to provide a meaningful examination of research or a thorough basis for its recommendations. This critique focuses on six key concerns regarding the report: it fails to consider human capital constraints or to sufficiently consider obstacles confronting classroom technology usage, it overlooks equity concerns and past problems with dependence on external professional services, and it ignores both the potential for disruptive reform churn and the danger of philanthropic efforts altering public education systems in undemocratic ways. For these reasons, the report’s usefulness to policy and practice is limited.
NEPC Review: Measures of Last Resort: Assessing Strategies for State-Initiated Turnarounds (Center on Reinventing Public Education, November 2016)
The stated goal of this report is to strengthen the evidence base on state-initiated turnarounds and to provide guidance to help states use turnaround strategies more effectively. The report draws on multiple sources of information to develop a conceptual framework and profile of state-initiated turnaround strategies, to array the evidence on the effectiveness of turnaround initiatives, and to identify key elements of a successful turnaround strategy. However, given multiple methodological limitations, the report fails to elevate either the research base or the policy discourse. Specifically, the methods used to carry out the original research (e.g., analysis of state policies, interviews with stakeholders, and illustrative cases) are neither explained nor justified. Likewise, the methods employed in the eight evaluations selected to assess the effectiveness of turnaround approaches are not described, and the evidence base produced by these evaluations is not sufficient to support the sweeping claims made in the report. Equally important, the report neglects to consider relevant research on the specific mechanisms (e.g., school reconstitution, intensive professional development, private management systems) that states use when they employ the broad turnaround strategies discussed in the report. As a result of these problems, the report does not enhance the evidence base or provide the substantive guidance state policymakers require to make informed decisions about the use of various school turnaround strategies.
NEPC Review: A 21st Century School System in the Mile-High City (Progressive Policy Institute, May 2016)
A report published by the Progressive Policy Institute calls for aggressively closing more public schools and expanding charter schools and charter networks. It highlights reforms adopted by Denver Public Schools, notably a “portfolio model” of school governance, and argues that these reforms positively impacted student test scores. However, causality cannot be determined, and the report did not attempt to isolate the effect of a multitude of reforms—including charters, performance pay, and a new performance framework—from larger complex forces shaping student demographics in the city. Written in a reportorial voice, the only data presented are in the form of simple charts. The lack of conventional statistical analyses thwarts the reader’s understanding. The report also characterizes the reform’s adoption as a “political success” born of a healthily contentious electoral process. In doing so, it downplays the role of outside forces and moneyed groups that influenced the form of reforms, and it disregards missed opportunities for meaningful engagement with community stakeholders. Finally, while the report acknowledges the district’s failure to close achievement gaps and admits limitations with the evaluation system, it never explains how a successful reform could generate a widening gap in performance between student groups by race and class.
Update: David Osborne, the report’s author, has posted a response to the review. The response can be found at: http://www.progressivepolicy.org/blog/response-national-education-policy-center/
Terrenda White’s rejoinder to the author's response to her review is posted immediately below the review.
Review of Measuring School Turnaround Success
Measuring School Turnaround Success contends that the lack of a shared definition for turnaround success makes it hard for reformers to learn from each other and determine true success. The report claims to develop a model for defining turnaround success with measures, metrics, and cut scores that reach beyond student achievement tests, to include indicators of engagement by students, parents and teachers, teacher and leader effectiveness, and short-term learning outcomes. Unfortunately, given the dearth of research evidence and sound methodological techniques incorporated into its analysis, as well as the omission of several rigorous, peer-reviewed studies that contradict the majority of its proposals, the report does not meet a minimal standard of evidence to support its claims. Ironically, the report focuses largely on standardized test scores, despite its stated intentions. The result is a report that distracts attention and potential resources from schools’ other goals, including civic, social, emotional, and broader academic ones. In the end, the report puts forth yet another proposal, funded by public dollars, that encourages state leaders to continue over-relying on flawed, test-centered strategies. Policymakers and practitioners looking for guidance on measuring turnaround success will not find worthwhile recommendations. Instead, they will encounter several unsubstantiated ones and others that are contradicted by solid peer-reviewed research.
NEPC Review: Dramatic Action, Dramatic Improvement: The Research on School Turnaround (Center for American Progress, March 2015)
Dramatic Action, Dramatic Improvement: The Research on School Turnaround advocates for implementing the most effective, research-based methods for turning around low-performing schools through the federal School Improvement Grant (SIG) program. It argues that the available body of research points to five dramatic actions that are necessary to bring about dramatic school improvement. Unfortunately, the rationale for its assertions is narrow, incoherent, and misleading. This limitation stems from the report’s unsystematic review of literature, resulting in its failure to incorporate lessons from large bodies of research on high-stakes accountability, school improvement, and the emerging evidence on school closures and federally funded turnarounds—all of which reveal that the federal SIG program’s turnaround policies are based on unwarranted claims and are contradicted by the empirical evidence. Consequently, the report’s recommendations are unsupported by rigorous research. Like the SIG program itself, the report promotes seemingly bold school changes that appear seductive on the surface. But when compared to the real evidence on school turnarounds, their evidence-based foundation vanishes. In the end, schools, districts, and states that follow the report’s advice stand to reproduce the unequal conditions that have led, in part, to their need for dramatic turnaround in the first place.
NEPC Review: New York City's Children First: Lessons in School Reform (Center for American Progress, March 2014)
New York City’s Children First: Lessons in School Reform summarizes elementary and secondary level education policy reforms in New York City during Mayor Michael Bloomberg’s tenure. Education policy changes in New York City during this time frame, from 2002 through 2013, are collectively known as “Children First” reforms. The report reviews the elements of these reforms and analyzes their effectiveness both collectively and individually. The New York City school system experienced dramatic changes during this era, and the report does a very nice job of synthesizing important events and facts into a single narrative. The report occasionally goes too far in classifying various policies as successes or failures. In particular, the report overhypes research examining the success of small high schools and of charter schools; a more balanced interpretation of this research literature should lead to a far more neutral tone concerning the success of these schools. To its credit, the report also discusses important questions of systemic governance and policy implementation. Management and accountability systems set up during Children First likely had a tremendous impact on how other reform components were implemented and are also likely to continue to affect how education policies are implemented under the new mayoral regime.
Review of Report Card on American Education
The 18th edition of the American Legislative Exchange Council’s (ALEC) Report Card on American Education: Ranking State K-12 Performance, Progress, and Reform draws on ratings from market-oriented advocacy groups to grade states in areas such as support for charter schools, availability of vouchers, and permissiveness for homeschooling. The authors contend that these grades are based on “high quality” research demonstrating that the policies for which they award high grades will improve education for all students. This review finds that, contrary to these claims, ALEC’s grades draw selectively from these advocacy groups to make claims that are not supported in the wider, peer-reviewed literature. In fact, the research ALEC highlights is quite shoddy and is unsuitable for supporting its recommendations. The authors’ claims of “a growing body of research” lacks citations; their grading system contradicts the testing data that they report; and their data on alternative teacher research is simply wrong. Overall, ALEC’s Report Card is grounded less in research than in ideological tenets, as reflected in the high grades it assigns to states with unproven and even disproven market-based policies. The report’s purpose appears to be more about shifting control of education to private interests than in improving education.
Review of Mayoral Governance and Student Achievement
In Mayoral Governance and Student Achievement: How Mayor-Led Districts Are Improving School and Student Performance, published by the Center for American Progress, the authors seek to bring fiscal and student achievement data to the debate around mayoral control. The fiscal analyses of mayoral-led cities are problematic due to inappropriate comparisons and a lack of reliable and valid evidence supporting the assertion that mayoral control has an influence on the amount or the distribution of resources. Throughout the discussion of student achievement, the report highlights positive findings in a few districts, but offers limited discussion of mayor-led cities where such gains were not found and of other cities in the country that saw strong gains without mayoral control. These issues call into question whether “mayoral control” is appropriately credited with the improvements identified in the report. The paper does not provide or explain the statistical methods or provide the findings essential to supporting the authors’ claims. Nevertheless, this report offers useful information about the context for shifts to mayoral control in different cities and the challenges that may arise in such governance changes. The limitations, however, preclude relying on either the report’s findings or its recommendations in making policy decisions.
What Does It Take to Scale Up Innovations?
NEPC Review: Parent Power: Grass-Roots Activism and K-12 Education Reform (American Enterprise Institute, July 2012)
The authors of this American Enterprise Institute report interviewed 28 leaders and practitioners of four national educational reform organizations to catalogue opportunities for and barriers to “parent power.” The report unevenly reflects the competing conceptions of “parent power” underlying the national debate on education reform. One conception, embraced uncritically by the authors and the new wave of well-funded national advocacy organizations, sees parents primarily as “consumers” of educational services who seek better choices in a more privatized education marketplace. An alternative, dismissed and overlooked by the authors but embraced by a long tradition of community organizers and public education advocates, views parents as the citizen owners-managers of a public education system that is a central institution of democratic civic life. These competing visions arise from sharply different histories and politics and give rise to dramatically different prescriptions for reform. The report suffers from an inadequate and slanted literature review; highly selective sampling; a serious lack of objectivity; disturbing characterizations of urban parents as “ignorant,” under-engaged and resistant to change; and a failure to contend with empirical evidence that challenges their views on “what parents want.” Its failure to adequately examine and document the full range of “grass-roots activism,” organizing, and history reflects both its blinders and its narrow political objective: to provide a briefing paper for the side it has chosen in what it calls “the fight.”
NEPC Review: The Louisiana Recovery School District: Lessons for the Buckeye State (Thomas B. Fordham Institute, January 2012)
In The Louisiana Recovery School District: Lessons for the Buckeye State, the Thomas B. Fordham Institute criticizes local urban governance structures and presents the decentralized, charter-school-driven Recovery School District (RSD) in New Orleans as a successful model for fiscal and academic performance. Absent from the review is any consideration of the chronic under-funding and racial history of New Orleans public schools before Hurricane Katrina, and no evidence is provided that a conversion to charter schools would remedy these problems. The report also misreads the achievement data to assert the success of the RSD, when the claimed gains may be simply a function of shifting test standards. The report also touts the replacement of senior teachers with new and non-traditionally prepared teachers, but provides no evidence of the efficacy of this practice. Additionally, the report claims public support for the reforms, but other indicators—never addressed in the report—reveal serious concerns over access, equity, performance, and accountability. Ultimately, the report is a polemic advocating the removal of public governance and the replacement of public schools with privately operated charter networks. It is thin on data and thick on claims, and should be read with great caution by policymakers in Ohio and elsewhere.
NEPC Review: What’s Trust Got to Do With It? (Public Agenda, December 2011)
This report offers recommendations for building community support for federal school turnaround approaches, particularly in communities that oppose these approaches. Parents, the report concludes, want improvement but cherish their local schools and distrust the turnaround options mandated from above by higher levels of government. Thus, community members rise up in anger when their school faces closure, conversion to a charter school, breaking up, or forced replacement of staff. Arguing that this resistance is due in large part to parents not understanding how bad their schools are, the report proposes that by engaging the public constructively and using eight communication strategies, parents will react more positively towards imposed turnaround approaches. The report does not address the body of research that shows school turnarounds to be generally unsuccessful. Further, even though parents in the study raise concerns that their schools are under-resourced and face significant social problems, the report fails to address these issues. By diverting attention from the real problems correctly identified by the parents and by possibly disrupting ongoing reforms, this communication strategy holds little promise for actually improving education and could prove harmful.
How Many Decades Before 'Reform' Becomes 'Status Quo'?
Review of Organizing Schools to Improve Student Achievement
This report carefully reviews high-quality empirical evidence from the last several years on the test score effects of three approaches to modifying the organization of schools: (1) starting schools later in the morning, (2) favoring K-8 grade configuration instead of junior high or middle school configurations, and (3) increasing teacher specialization by grade and subject. It estimates the earnings benefits of each intervention and, for interventions (1) and (2), compares monetary benefits to costs. The report concludes that benefits outweigh costs, although the rough cost estimates suggest that better data are required to draw definite conclusions. The report’s main conclusion is that organizational reforms deserve a more prominent place in education debates, and that individual school districts should carefully consider them alongside more popular reform options. The review points to a few shortcomings but concludes that the report’s analyses are solid and helpful and that the results are presented carefully and cautiously.
NEPC Review: Do High Flyers Maintain Their Altitude? (Thomas B. Fordham Institute, September 2011)
The research report reviewed here concludes that many initially high-achieving students are falling further and further behind over the course of their years in school. The report intends to raise the alarm and to advocate for improved programs for these students. It is, however, a false alarm due to biased methodology and misleading arguments. The report’s norm-referenced framework guarantees “losers” as well as “winners,” regardless of any true improvement made by the students. Also, the “regression to the mean” effect produces a false illusion of a tradeoff of over-progress by low achievers at the cost of under-progress for high achievers. Finally, its prescription for stronger school accountability for high-achieving students under NCLB does not follow research-based guidance on how to improve student learning. Other research, including that conducted by this reviewer, finds that students who are high achievers and low achievers make approximately equal academic progress in reading and math, while the achievement gaps remain large. Moreover, a substantial part of the variation in student progress is attributable to teacher and school effects beyond students’ initial status and background characteristics.
Two authors of the report published a written reponse on October 30, 2011. That response is available here, and Dr. Lee's brief reply is published and downloadable below.
NEPC Review: Charting New Territory: Tapping Charter Schools to Turn Around the Nation's Dropout Factories (Center for American Progress, June 2011)
Charting New Territory: Tapping Charter Schools to Turn Around the Nation’s Dropout Factories argues for a more prominent role for charter operators in turning around perennially low-performing high schools. However, the report’s ultimate findings and conclusions are out of proportion to the strength of the research evidence on school turnarounds, charter schools, and charter management organizations, as well as to the data on which the analysis is based. As such, its recommendations are of little utility in guiding policy on charter management organizations as tools for turning around struggling schools. Further, the report fails to justify the general practice of converting chronically low-performing schools to charter status. Instead, the report reproduces familiar arguments for market-based school reforms that are grounded in little empirical evidence and that use our nation’s neediest schools—those serving primarily poor children and children of color—as laboratories for educational experiments, notwithstanding existing evidence that the experiments will not succeed. In doing so, the report distracts policymakers and practitioners from more fundamental and worthwhile questions about the types of policies that can secure the necessary conditions for all students to succeed.
Suggested Citation: Trujillo, T. (2011). Review of “Charting New Territory: Tapping Charter Schools to Turn Around the Nation’s Dropout Factories.” Boulder, CO: National Education Policy Center. Retrieved [date] from http://nepc.colorado.edu/thinktank/review-charting-new-territory
Review of Yearning to Break Free: Ohio Superintendents Speak Out
The report, Yearning to Break Free: Ohio Superintendents Speak Out, describes findings of a survey of 246 Ohio school superintendents about critical issues facing the state’s educational system. In particular, the intent of the study was to examine how superintendents might do more with fewer resources. The authors conclude that Ohio districts need increased managerial authority and control over state revenue streams and other funds; transformed collective bargaining rights; and repeal of automatic increases in teacher salaries. Such efforts, they argue, are imperative to improve the educational opportunities of all children in the state. But the authors do not provide evidence to support this latter claim. The combined effects of non-representative sampling, loaded or inappropriately worded items, and the conflating of opinion and fact make the report’s conclusions problematic. Myriad factors contribute to student achievement, including home and community effects, campus resources (material and non-material), as well as teacher competence which are not examined or considered. And, despite the reported finding that superintendents prefer greater autonomy in personnel and school policies over increased funding, the majority of superintendents also contend that they would see a trade-off of more autonomy with a decrease in funding as undesirable. While the report’s main thrust is to justify flat or reduced spending, the report lacks sufficient rigor to make it useful to guide policy or practice.
Suggested Citation: Horn, C. & Dworkin, G. (2011). Review of “Yearning to Break Free: Ohio Superintendents Speak Out.” Boulder, CO: National Education Policy Center. Retrieved [date] from http://nepc.colorado.edu/thinktank/review-ohio-superintendents
Note that Steve Farkas, an author of the Fordham report, replied to the review, available at http://www.educationgadfly.net/flypaper/2011/05/critics-of-fordham%E2%8…
Horn and Dworkin responded to this reply, available at http://nepc.colorado.edu/sites/default/files/NEPC-Yearning-Farkas-Respo…
NEPC Review: Reform With Results for New Jersey Schools (December 2010)
A report published by the Lexington Institute presents findings on the effectiveness of New Jersey’s Abbott v. Burke court decisions from the late 1990s through 2009. The report argues that the reforms ordered by the state’s supreme court failed to significantly increase student achievement despite what it terms as dramatic increases in spending. Based on these findings, the report argues that the increases in spending in these urban districts and their continued dismal student achievement rates make New Jersey, and particularly Newark, ideal for instituting a number of reforms. These advocated reforms include parental empowerment, increases in the number of charter schools, changes in teacher union contracts, and the enactment of private school choice policies (e.g., vouchers). The report cannot stand as a research document and provides little or no empirical evidence to support its critiques of Abbott or its recommendations for reform. It omits important parts of the existing research literature, such as NAEP data showing New Jersey as high performing and as closing the achievement gap. The Lexington report contains no methodology to speak of. Overall, the report has little or no use for informing educational reform in Newark, New Jersey or nationally.
Suggested Citation: Sadovnik, A.R. (2011). Review of “Reform with Results for New Jersey Schools.” Boulder, CO: National Education Policy Center. Retrieved [date] from http://NEPC.org/thinktank/review-reform-with-results
Due Diligence and the Evaluation of Teachers
NEPC Review: Students in N.J. Charter Schools Outperform Those in Local School Districts, Data Shows (January 2011)
Bruce Baker responds to a Star-Ledger article reporting higher test scores for charter schools than for district public schools. His review of student population differences in Newark schools and reanalysis of 2009 test results provide the basis of his argument that because N.J. charter schools tend to serve different populations than district public schools and to have high rates of attrition, the comparisons of test scores of the two sets of schools do not provide meaningful input into the policy debate about school reform.