A report from the Cato Institute opens with Horace Mann’s well-known conviction that public schools are the bedrock of a democratic society – a public good that should be made available to all. Yet the report, Is Public Schooling a Public Good? An Analysis of Schooling Externalities, improperly conflates the civic and economic definitions of a public good. Although the report begins with Mann’s vision of the role of public schools as building a better society, it then misleadingly shifts the analysis to the economic value of public schools as a market-based “good” like steel or corn. The report relies on a false equivalence of the civic and economic definitions of a public good to advance a proposal for de-funding public schools and introducing a nationwide education savings account (voucher) program. While there is extensive research on the educational purposes of schooling, the Cato report’s limited review of this literature consistently misrepresents the meaning, scope and implications of this literature. The result is a portrayal of public schools as “agents of harm” for what appears to be an ideologically driven thought experiment. Even for those who might be in favor of vouchers, the report’s imbalance, flawed logic and limited research base render the report of no use to policymakers.

A study published by the American Enterprise Institute seeks to illuminate how students in career and technical education (CTE) programs demonstrate higher high school graduation rates, overall educational attainment, and earnings than students in academic programs whose previous test scores are similar. Although those students who take many CTE courses by 12th grade tend to have significantly lower test scores, this study finds that these students have higher noncognitive skills (e.g., attendance and homework time). The most remarkable feature of this study is the broad array of indicators it compiles, including how much effort students exhibit on a routine task (e.g., a long and boring survey in school), and teacher reports of student effort. Using such data, the study suggests that CTE may improve attainments by improving noncognitive skills. The key implication is that, instead of the narrow policy focus on academic skills, educators need to consider how to improve students’ other skills to improve education and job outcomes. While this implication is reasonably drawn from the study, educators need a clearer interpretation of these “noncognitive skills” and whether they are persistent attributes or highly changeable behaviors. Overall, however, the study presents a strong empirical analysis of a strong dataset and should prove useful for policymakers.

NCTQ released its 2018 review of U.S. teacher preparation programs. Employing open-records requests and online searches, the report ranks 567 graduate teacher preparation programs, 129 alternative route programs, and 18 residencies on practice, knowledge and admissions. The report seeks to determine if the teacher preparation programs are aligned with NCTQ’s standards. Such alignment, the report insists, will produce teachers “not only ready to achieve individual successes, but also to start a broader movement toward increased student learning and proficiency.” However, the report determines that most programs are not aligned with its standards. Accordingly, it finds “severe structural problems with both graduate and alternative route programs that should make anyone considering them cautious.” However, the report has multiple logical, conceptual, and methodological flaws. Its rationale includes widely critiqued assumptions about the nature of teaching, learning, and teacher credentials. Its methodology, which employs a highly questionable documents-only evaluation system, is a maze of inconsistencies, ambiguities, and contradictions. Further, the report ignores accumulating evidence that there is little relationship between graduates’ classroom performance and NCTQ’s ratings. Finally, the report fails to substantively account for broad shifts in the field of teacher education that are nuanced, hybridized, and dynamic. Regrettably, the report exacerbates the dysfunctional dichotomy between university programs and alternative routes and offers little guidance for policymakers, practitioners, or the general public.

A report from NCTQ begins with nine goals purportedly based on the “best available research evidence” about teacher quality. Yet neither this report nor its companion, which describes the original development of the goals, cites any research evidence. The report also uses the terms “teacher quality” and “teacher effectiveness” (on raising test scores) interchangeably. The report assumes reader buy-in to its goals, to its focus on test scores, and to its assumption that “great teachers” have an “outsize impact” on students’ learning and lives. Grounded in these assumptions, the report highlights examples of “leading state work” in 37 policy areas related to teacher quality, aiming to hold up these state policies as exemplars for other state policymakers to replicate. Despite its intentions, the report has multiple flaws that undermine its validity and usefulness. It offers no explanation about how the 37 best practices were selected in the first place and no justification for its selection of “leading” policy work, some of which has occurred in states that have consistently been low performers on national assessments. In addition, the report offers no evidence to support its approach and makes no references to the nuanced and complex research literature in this area. The report focuses primarily on human capital policies that explicitly target the qualifications and evaluation of the teacher workforce. This ignores the growing consensus that many other factors matter in the production of students’ learning, including supports that help teachers succeed, school contexts and cultures, state and regional labor markets, teachers’ relationship-building capacities, and the social organization of teachers’ work. In the end, the report is of limited use.

A report released by the University of Arkansas Department of Education Reform contends that charter schools produce more achievement per dollar invested, as compared to public schools. This newest report is focused on city-level analyses in eight US cities (Atlanta, Boston, Denver, Houston, Indianapolis, New York City, San Antonio, and Washington D.C.) and uses cost effectiveness and Return on Investment (ROI) ratios. It concludes that charter schools deliver a weighted average of an additional 4.34 NAEP reading points and 4.73 NAEP math points per $1000 invested. The report also argues that that charter schools offer an advantage of $1.77 in lifetime earnings for each dollar invested, representing a ROI benefit of 38%. However, there are a variety of methodological choices made by the authors that threaten the validity of the results. For example, the report uses revenues rather than actual expenditures – despite well-established critiques of this approach. The report also fails to account for the non-comparability of the student populations in charter and comparison public schools. Three other problems also undercut the report’s claims. First, even though the think tank’s earlier productivity report included a caveat saying that causal claims would not be appropriate, the new report omits that caution. Second, the report’s lack of specificity plagues the accuracy and validity of its calculations; e.g., using state-level data in city-level analyses and completely excluding race and gender. Finally, the authors again fail to reconcile their report with the extensive literature of contrary findings.

Reports from Bellwether and New America each review the current state of early childhood education (ECE) teacher preparation programs at the community college and university levels. Both raise awareness about the challenges facing the ECE workforce and educational institutions in dealing with the recent policy push to increase the required educational credentials for pre-K teachers. Specifically, these policies elevate the requirement from the Child Development Associate certificate to an associate’s degree or to a bachelor’s degree with early childhood specialization. The reports also illustrate innovative practices that some states and colleges are implementing to improve the quality, flexibility, mobility, and diversity of ECE teachers. They end with policy recommendations for states, colleges, and ECE leaders. The methods consist of simple literature reviews (which generally rely on think tank white papers, as opposed to peer-reviewed journal publications). The reports also incorporate informal interviews with ECE stakeholders and leaders, but the methodology and selection procedures used are unclear. Notwithstanding these limitations, the analyses appear sound and the conclusions and recommendations made are generally reasonable, supported by the relatively limited data available. The longer report from Bellwether on the role of community colleges appears useful for policymakers and the public, due to its depth and effectiveness in presentation. The report from New America is too short and rather awkward in presentation, which limits its effectiveness.

A report from the American Enterprise Institute (AEI), Do Impacts on Test Scores Even Matter? Lessons from Long-Run Outcomes in School Choice Research, examines whether student achievement scores on math and English language arts tests align with “long-run” attainment outcomes such as high school graduation rates, college enrollment, and college graduation rates. Drawing on a systematic review of the literature, it concludes that the impacts of school choice programs on test scores are not well connected to such attainment outcomes, which are presented as more positive. This review considers two issues with the report: consistency and evidence. Regarding consistency, the report’s suggestion that achievement scores should play a smaller role in determining the efficacy of school choice models represents a stunning effort to move the goalposts in search of new justifications for supporting their preferred policies. After decades of pro-school-choice research and advocacy promoting test score comparisons with public schools as the primary measurement for evaluating school choice models (e.g., charters and school vouchers), the AEI report now suggests that less attention be given to these learning outcomes. Regarding evidence, the AEI report is riddled with numerous internal inconsistencies in its discussion and treatment of a set of studies that were selected by questionable methods. In view of the 180-degree turn based on questionable evidence, the report — despite the authors’ assertions — is of little use to policymakers.

A report, Charter Schools and the Achievement Gap, finds that, though charter schools on average perform no better than traditional public schools, urban “no-excuses” charter schools—which often use intensive discipline to enforce order—demonstrate promising results. It recommends that these schools and their practices be widely replicated within and outside of the charter school sector. We find three major flaws with this conclusion. First, the report’s recommendations are based solely on the academic success of these schools and fail to address the controversy over their use of harsh disciplinary methods. No-excuses disciplinary practices can contribute to high rates of exclusionary discipline (e.g., suspensions that push students out of school) and may not support a broad definition of student success. Second, the recommendation that schools replicate no-excuses practices begs the question of what exactly should be replicated. It does not confront the lack of research identifying which school practices are effective for improving student achievement. Third, the report does not address many of the underlying factors that would allow no-excuses schools and their practices to successfully replicate, such as additional resources, committed teachers, and students and families willing and able to abide by these schools’ stringent practices. Thus, while the report is nuanced in its review of charter school impacts, it lacks this same care in drawing its conclusions—greatly decreasing the usefulness of the report.

This report offers political advice to school superintendents. The authors draw solely on a collection of anecdotes from superintendents of large urban districts to offer a narrative replete with maxims, aphorisms, and pithy advice. Superintending is, in their view, a continuous string of almost unavoidable conflicts, so the reader is led through internal groups (central office, school board, unions, teachers and principals) and external groups (foundations, businesses and state politicians) and advised on how to cultivate support and deal with opponents. There is no research design, literature review, or systematic data collection. The report relies heavily on the authors’ experiences, yet neither author appears to have worked in a K-12 district or served as a superintendent. Moreover, the advice that follows is often contrary to contemporary professional practices. The report improperly generalizes from more politicized large urban districts and does not consider how the favored approaches may limit vision, flexibility and effectiveness. In the end, the report perpetuates antiquated methods and does not advance our knowledge.

This review examines a report evaluating four years of implementation of the 2010 Teacher Incentive Fund (TIF) to determine whether pay-for-performance (PFP) bonuses to teachers and principals improve student achievement. The report notes that the goal of PFP is to motivate improved educator performance and to attract and retain more effective teachers, thereby increasing student achievement. The report found that PFP marginally increased student test scores in reading and math by year two and that it was associated with slightly higher classroom observation ratings; it also found that classroom observation ratings were not associated with student test scores and that PFP had no effect on filling teacher vacancies. Problematic issues in the report involve combining non-comparable sites, employing different outcome measures, and generating a misleading cost-effectiveness analysis. Although several thorough analyses are presented, the data do not support the report’s suggestion that PFP holds promise to address achievement disparities. Overall, because of the report’s incorporation of a polyglot of outcome measures of questionable validity, this review finds that the report lacks utility for policymakers.

A report from the Fordham Institute investigated the impact of a reform in the School District of Philadelphia that eliminated suspensions for certain low-level misbehaviors. The report considered whether the policy change was associated with any of the following: (a) district-wide out-of-school suspension rates, (b) academic and behavioral outcomes for students (looking separately at students who had a record of prior suspensions and those with no prior suspensions), and (c) racial disparities in suspensions. While the report concluded that the reform was a failure, the actual results were mixed, with the positive trends for students who were earlier suspended being much stronger in magnitude than evidence of negative outcomes for students who were not. A strength of the report is the use of advanced statistical methods and a longitudinal dataset to answer the questions of interest. However, the report is plagued by logical fallacies, overly simplified interpretations of findings, and inflammatory language. Moreover, the report uses misleading causal (“consequences”) language in the title and to describe study results, even though the study design is limited by unmeasured confounding factors and inappropriate comparison groups. Thus, while the analyses upon which the report is based have some technical merits, the narrative seems more of an attempt to advance a political agenda opposed to the reform studied than to improve understanding of complex policy issues.

Balancing Act explores the racial imbalance of schools and neighborhoods across the nation, finding that the racial composition of schools often reflects the surrounding neighborhoods. It suggests that racially concentrated schools are the result of residential segregation and how school district boundaries are drawn to separate populations. The authors also posit that charter schools, which are freed from the constraints of traditional boundaries, can interrupt the school-housing relationship. Yet the report finds that charter schools are, on average, more racially imbalanced than other public schools. The report deserves credit for tackling the complicated relationship between school and housing segregation, and for bringing renewed attention to issues that are often considered separately. Still, a shaky grounding in the research and law, and in the methods that inform school desegregation, limits the report’s usefulness when it comes to policy and practice. Moreover, crucial methodological decisions, like how neighborhoods are defined or why metro areas are treated the same as rural ones, either are not fully explained or lack a research-based rationale. This weakness undermines the utility of the newly created database—which the authors view as a vital contribution of the report. In short, Balancing Act represents a missed opportunity to accurately explore the nexus between school and housing segregation.

A report from New America argues that online programs, appropriately structured, have the potential to professionalize many early childhood (EC) teachers. The report contends that online bachelor’s programs offer more flexible and financially accessible pathways for pre-K teachers to obtain a degree, thereby propelling these teachers towards higher quality practice and higher salaries. The report presents a perspective on what a professionalized pre-K workforce should entail, recognizing the significant knowledge and skills required of these teachers, and equating raising pre-K teacher credential standards with professionalization, and therefore, quality. While the report draws attention to some of the unique considerations necessary for EC workforce policy, it underplays a number of critical issues in professionalizing the field. It lacks adequate treatment of the EC workforce’s economic realities that place them at the poverty line, regardless of credential or degree. It presents testimonials rather than evidence of outcomes for online programs. It makes pre-K teachers responsible for closing the pay gap and for upgrading their low professional esteem. Finally, though recognizing the importance of high-quality EC programs, it leaves the term under-analyzed at best. Unfortunately, without a nuanced discussion of these issues, the report’s usefulness is limited.

The policy brief Student Centered State Funding: A How-to Guide for State Policymakers attempts to illustrate and explain to state policymakers how to design a state school funding system whereby all state and local dollars may flow freely, by parental choice, following children to local district, charter or private schools. In particular, the brief seeks to provide state policy guidance on setting specific funding levels for each child. These “money follows the child” approaches provide a system that eases the process of funding an assortment of school choice programs. Unfortunately, this colorful multi-page infographic suffers from three major shortcomings as a guide for state policymakers: First, the brief advances the false dichotomy that advocates for state and district school finance systems to focus on funding the child, not funding the essential institutions that serve those children. This dichotomy wrongly promotes the idea that there is no benefit to children of equitably and adequately financing educational institutions. Second, the brief is based on overly simplistic, frequently misrepresented, and often outright incorrect versions of the status quo, including overbroad mischaracterizations of how schools are currently financed. Third, the details of the brief’s proposals and espoused benefits are entirely speculative and unsubstantiated, in some cases simply made up and in other cases supported only by insular and circular self-citation to previous work that itself cites no strong empirical support. For these reasons and others, the report is of absolutely no value to policymakers for informing the design of state school finance systems or school district resource allocation formulas.

In 2010, Facebook CEO Mark Zuckerberg announced, with great fanfare, a $100 million challenge grant for the support of a series of reforms in Newark, NJ schools. The two reports reviewed here are the first attempt at a comprehensive assessment of the impact of the Zuckerberg donation on student achievement. As such, the reports, which were written by a team of economists, have garnered significant attention. The reviewers evaluate the reports and their potential contribution to education policy research.

The Every Students Succeeds Act (ESSA) directs states and districts to identify equity gaps in students’ access to excellent educators and transformative school leaders. States are encouraged to use Title II funds strategically in order to identify and remedy these gaps. A new report from The Education Trust draws on ESSA documents and state teacher equity plans to provide guidance to state leaders, including some sound advice—but with significant omissions. The report does not engage with thorny issues around alternative pathways into teaching, and it largely skirts issues around incentives for supporting teacher recruitment and retention in hard-to-staff schools. The report also does not consider what attracts teachers into the profession and into particular school environments. Likewise, the report fails to draw on the explicit remedies sought by ESSA to link high-quality leadership with strong teacher recruitment and retention. Instead, the report casts the teacher equity problem primarily in terms of labor supply shortages and treats teachers like interchangeable widgets. Relying heavily on advocacy sources, it misses an opportunity to unpack the root causes of the teacher retention problem, particularly the corrosive impact of past federal and state policies on the teaching profession. The report does not help state leaders understand how they might build incentives and cultures that draw strong teachers into high-need schools, and they will thus be left with an incomplete and insufficient set of tools for ensuring that all students have equitable access to excellent educators.

In this report, a school choice advocacy group presents results from its survey of K-12 parents within and across the public and private sectors. They report that parents are highly satisfied with voucher and tax credit scholarship programs and suggest that the findings support the expansion of school choice programs. However, these and other findings are consistent with research showing general parental satisfaction with their children’s schools. One underemphasized finding is that substantial proportions of public school families—the largest constituency of K-12 parents in Indiana—did not participate in private school choice programs because they are happy with their current schools and want to support public schools. The survey and analysis fall short in four ways. First, three incompatible data collection methods were used to collect small samples of non-representative groups of Indiana parents. Second, the statistical analyses are too weak to draw clear conclusions. Third, while organized like a conventional research study, the report appears to be designed to advance an agenda rather than provide substantive answers to important policy questions. Finally, the report provides little new information about parents’ experiences with their children’s schools. Thus, the report does not add to our knowledge about school choice in Indiana or provide much useful information about public support for school choice programs.

This report compares average rates of frequent teacher absence (more than 10 days) for teachers with and without union or union-like contracts in traditional public schools and charter schools. The study’s rationale is that such absences substantively harm students and cost taxpayers billions of dollars. It finds that teachers contractually allowed more absences are frequently absent more often than teachers allowed fewer absences. Based on these averages, the report concludes that the contracts result in non-beneficial, or uncalled-for absences, rather than absences for legitimate reasons, and it recommends that contracts be made less generous. However, the report lacks support for its major claims, ignores known discrepancies in data, uses cited resources in highly selective ways, ignores large bodies of contradictory research, and draws unwarranted conclusions. In addition, the report’s idiosyncratic use of the term “chronic absenteeism” misrepresents the data and, along with its use of graphics, appears intended to create a national alarmist picture unwarranted either by the data or by other research. Accordingly, the report appears to be an effort to generate numbers and charts useful in discrediting teachers as irresponsible shirkers.

This report provides an extensive analysis based on the most comprehensive dataset ever assembled for school closure research, including 1,522 low-performing schools that were closed across 26 states between 2006 and 2013. The report finds that even when holding constant academic performance, schools were more likely to be closed if they enrolled higher proportions of minority and low-income students. It also finds test score declines, relative to the comparison group, for two groups of students displaced by closures: those who transferred to schools with a prior record of relatively lower test-score performance and those who transferred to schools with equivalent past test-score performance. The slightly less than half of students who transferred to higher performing schools showed academic improvement relative to their matched peers. In general, although the reviewers found this to be a careful and rigorous study, they see a few missed opportunities. First, the report’s focus on some tenuous analyses (involving pre-closure transfers) obscures its most important findings – disproportionality in school closures and inadequate numbers of higher quality receiving schools, leading to performance declines for most. Second, the reviewers are concerned about statistical modeling choices and matching challenges that may threaten the validity of subgroup analyses (charter school students). Finally, the reviewers would have liked to see the report acknowledge the inescapable moral dimensions of school closure: The communities most likely to be negatively affected are unlikely to have participated in closure decisions.

A report by researchers at the University of Arkansas concludes that charter schools in New York City are not fairly funded, in comparison to district schools. The report asserts that this inequity is especially big for charter schools that are not co-located in public schools; and describes expenditure patterns across schools that show no clear differences between charter schools and district schools. This review of the report raises several concerns. Perhaps most importantly, the report simply does not attempt a rigorous comparison across schools. It assumes that any differences in student characteristics across charter and district schools are trivial, therefore concluding that raw, unadjusted funding amounts are sufficient for assessing fairness. It does not undertake any sensitivity testing to identify the precision of its estimates, nor does it investigate in detail what the optimal amount of funding should be for charter schools that are not co-located in public school buildings. Finally, New York City has significantly reformed its funding regulations for charter schools since the data upon which the report is based were collected, rendering the report’s estimates policy-irrelevant.

The Center for Research on Education Outcomes (CREDO) assessed the impact of different types of charter school-operating organizations on student outcomes in 24 states, plus New York City and Washington, D.C. The study finds that students in charter schools display slightly greater gains in performance than their peers in traditional public schools, especially students in charter schools operated by certain types of organizations. CREDO’s distinctions between organization types are, however, arbitrary and unsupported by other research in the field. This raises concerns about the practical utility of the CREDO findings. In addition, CREDO researchers made several dubious methodological decisions that threaten the validity of the study. A number of the problems flagged by the reviewers of this study have been raised in reviews of prior CREDO studies. Specifically, CREDO studies tend to over-interpret small effect sizes; fail to justify the statistical assumptions underlying the group comparisons made; and not take into account or acknowledge either the large body of charter school research beyond their own work or the limitations inherent in the research approach they have taken. Because these problems have gone unaddressed in the present study, and because the CREDO researchers have compounded them by creating a confusing and illogical charter organization classification system, the report is of limited if any value in policymaking. Readers should review the report with care.

This report asserts that tax credit scholarship programs, that distribute scholarships to students via Scholarship Tuition Organizations (STOs), have saved state treasuries between $1.7 and $3.4 billion dollars since 1998. The report argues that these programs are able to realize fiscal savings as a result of students leaving public schools and entering private schools (defined as “switchers”). The report claims that the percentage of students leaving public schools, coupled with the offset of variable per-student costs that districts no longer need to expend, have resulted in the sizable financial savings for state governments. This review questions the method used to estimate the percentage of switcher students across these various programs, and examines how the report determines variable cost fluctuations for each student that leaves public schooling. Since no STO programs require officials to track data on which students transfer out of public schooling into private, these lax accountability standards have led the report author to estimate fiscal savings using conjecture. Instead of following students, they interpreted broad population changes to STOs. Consequently, the results of this report do not provide an acceptable causal conclusion for policymakers. Suggestions for more extensive accounting procedures along with more nuanced methodologies for calculating true variable student costs are discussed.

A report from EdChoice documents two staffing trends in public schools. After a temporary pause during the Great Recession, school staffing in the U.S. resumed an upward trajectory, and hiring tilted toward non-teaching personnel as compared to teachers. The report concludes that staffing growth outpaces enrollment growth and that there has been no corresponding increase in student performance. The report then recommends increasing teacher pay, at the expense of non-teaching staff, and school choice. While the data on staffing trends are obtained from NCES publications, the report’s discussion of inputs, outcomes, and policy approaches is poorly grounded, leading to unsupported conclusions and policy prescriptions. It does not examine why there has been a staffing surge or whether it reflects a valid use of personnel. It evaluates the effectiveness of school staffing changes using concurrent achievement and finance measures, and it fails to acknowledge that educational outcomes have steadily improved, even though any benefits of staffing increases will be lagged and only show up gradually over time. The report presents no logical relationship between staffing trends and the solutions proposed. As a result, the report is irrelevant and is devoid of any important policy implications.

Two reports contend that the introduction of school choice can promote economic development in economically distressed urban areas. The first report, published by EdChoice, presents a case study of a charter school that has, according to the report, contributed to the economic development of the city of Santa Ana, California. The second report, published by the American Enterprise Institute, presents a proposal for a hypothetical voucher-like program that, if implemented, would purportedly spur economic development in high-poverty neighborhoods by luring higher income families into those neighborhoods. This review explains that both reports overlook significant bodies of relevant research literature and make unsupported claims that rely on flawed logic and data. The EdChoice report fails to collect and analyze data related to the report’s causal assertion that economic development in Santa Ana resulted from the establishment of the charter school. The American Enterprise Institute report’s claims about the benefits of the proposed program to publicly fund private schooling are unsupported by existing research. We conclude that these reports offer little useful guidance for policy or practice.

While inequalities in school funding resulting from state and local policies have long been a source of concern to education researchers and policymakers, a report from the Center for American Progress examines a source of educational inequality that receives less attention: private fundraising by parents. It focuses on the 50 Parent-Teacher Associations (PTAs) that raised the most money in 2013-2014, with two main findings. First, the PTAs raising large amounts were located in schools and districts with low rates of student poverty. Second, while a PTA in a high-poverty community may raise only a few hundred dollars, PTAs in this sample raised hundreds of thousands of dollars each year. Using case studies, the report considers district regulation of private fundraising. This review concludes that the report’s findings about the scope and beneficiaries of private fundraising are credible and important—showing the impact successful PTAs can have. However, the focus on a small number of schools and districts, a lack of attention to school and community context, and problems with the case study design limit the report’s overall relevance. In addition, it is important to note that most funding inequalities arise at the state level; funds raised by parents represent only a minute portion of overall school spending. Nevertheless, the report’s recommendations, especially in support of equity grants, will be useful to district-level policymakers.

A report offers a how-to guide for reform advocates interested in removing communities’ democratic control over their schools. The report explains how these reformers can influence states to use the Every Student Succeeds Act (ESSA) Title I school improvement funds to support a specific set of reforms: charter schools, state-initiated turnarounds, and appointment of an individual with plenipotentiary authority over districts or schools. While the report acknowledges that the research evidence on the effectiveness of these reforms as school improvement strategies is limited, it uses a few exceptional cases to explain how advocates seeking to influence the development of state ESSA plans can advance them anyway. As this review explains, support for the effectiveness of these approaches is simply too limited to present them as promising school improvement strategies. The report omits research that evaluates the models relative to other school reform initiatives, and it fails to take into account the opportunity costs of pursuing one set of policies over another. It also relies on test-score outcomes as the sole measure of success, thus ignoring other impacts these strategies may have on students and their local communities or the local school systems where they occur. For these reasons, policymakers, educators and state education administrators should be wary of relying on this report to guide them as they develop their state improvement plans and consider potential strategies for assisting low-performing schools and districts.

A report from the Department of Education Reform at the University of Arkansas examines the association between out-of-school suspensions and student test scores. Using dynamic and multilevel regression modeling of six years of school records from all K-12 public schools in Arkansas, the paper purports to estimate a causal relationship between exclusionary discipline and academic performance. It concludes, in contrast to prior work, that the number of days of suspension a student receives has a positive relationship to math and language arts test scores. However, the outcomes (end-of-year test scores) are measured at least a full year after the hypothesized causal factor (days suspended), and there is no control for days suspended in the year the test was taken. Consequently, the results do not reflect missed instructional time for the tested material or other associated mechanisms through which suspension might adversely affect test scores. The analyses also control for a large number of infractions that are strongly related to days of out-of-school suspension, which may produce biased results. The findings also have weak face validity, in light of the weight of evidence suggesting that exclusionary discipline and school absences have adverse effects on test scores, GPA, grade retention, and dropping out. For these reasons, the reviewers caution that this paper is not useful for policymakers.

A response from the authors is below the review.

The reviewers of “Understanding A Vicious Cycle: Do Out-Of-School Suspensions Impact Student Test Scores?” respond to criticisms of their review in two parts, Part I by Daniel Losen and Part II by Brea Louise Perry.

A common argument leveled against charter schools is that they attract the most motivated and intelligent students from already struggling public schools. Marcus Winters seeks to examine this claim, known as “cream-skimming,” by comparing the performance of New York City’s (NYC) charter middle schools with a set of traditional selective public middle schools, which admit students on the basis of prior performance. Findings indicate that, controlling for student characteristics, charter schools perform no differently in English Language Arts (ELA) and significantly better in math than selective schools. Based on this, the report concludes that the success of NYC charter schools cannot be explained by cream-skimming. While on its face this conclusion may seem logical, the report suffers from two primary flaws. First, it assumes that selective school applicants are higher performing and more motivated than charter school applicants. This is unlikely to be the case because all students are required to apply to traditional middle schools in NYC, while applying to a charter school requires navigating an additional application process. Second, the report relies on a single year of data to make comparisons of ill-defined and inappropriate outcomes—an approach that does not address either the question of cream-skimming or charter school success. As an evaluation of cream-skimming in charter schools, this report misses the mark.

This recent Heritage Foundation report argues that Title IX does not require schools to protect students from discrimination on the basis of gender identity. It criticizes the Obama Administration for issuing guidance and enforcing Title IX to protect transgender students, and it urges the Trump Administration and courts to keep gender identity protections out of federal laws. Oddly, transgender youth are at the center of, and yet somehow go wholly unexamined in, this report on gender identity policy. The report never acknowledges or addresses (a) legal opinions that gender identity discrimination is a form of sex discrimination; (b) the near-consensus within the medical, scientific, and educational communities concerning how transgender students should be treated; and (c) other research or literature shedding light on the appropriate care for and education of transgender youth. Additionally, the report erroneously asserts that transgender-inclusive policies will embolden men to enter women’s facilities to assault or abuse them. What is entirely missing from this report — and what policymakers and educators urgently need — is guidance in an area that may be new or unfamiliar to them. Fortunately, many states and districts have adopted positive gender-identity-related laws, policies, and practices that answer questions and serve as useful guidance for other jurisdictions about how to successfully integrate transgender students in schools.

The report reviewed here compares Wisconsin student test score performance for the 2015-16 school year across public schools, charter schools and private schools participating in one of the state’s voucher programs. Comparing a single year’s test scores across school sectors that serve different student populations is inherently problematic. The report uses linear regression models to attempt to adjust for these differences and make what the authors claim are “apples to apples” comparisons. Based on these comparisons, the report concludes that charter schools and private schools participating in the voucher programs are more effective than traditional public schools. Unfortunately, the limited nature of available data undermines any such causal conclusions. The inadequate and small number of school-level variables included in the regression models are not able to control for important confounding variables, most notably prior student achievement. Further, the use of aggregate percent proficient metrics masks variation in performance across grade levels and makes the results sensitive to the (arbitrary) location of the proficiency cut scores. The report’s description of methods and results also includes some troubling inconsistencies. Thus, while the report does present important descriptive statistics about test score performance in Wisconsin, it cannot provide answers for those interested in determining which schools or school choice policies in Wisconsin are most effective.

A report by the American Enterprise Institute (AEI), Differences by Design?, compares differences in approaches and demographics between and among charter school models and local “traditional public schools.” Using three national data sets, the report effectively captures the national universe of charter schools. It empirically demonstrates that cream-skimming occurs and that charters segregate by income, special education, race and ethnicity, in that different demographic groups attend different types of charter schools. Charter schools, the authors contend, provide differentiated and “innovative schooling options” through varied academic models that cater to, and ultimately reflect, parental choices for their children. The resulting de facto segregation is presented as a benign byproduct of beneficial choices differentially associated with different racial and ethnic groups. They contend this is “in line with a properly functioning charter sector.” Unfortunately, the report does not demonstrate familiarity with the research on parent decision-making or with the extensive research suggesting that charter schools are not particularly innovative in the curricular or instructional options. Despite what the report claims, traditional public schools do, in fact, offer various academic model specializations like the ones offered by the charter schools. Ultimately, the report’s dismissive characterization of de facto segregation in charters, as a benign byproduct of parental choice, is at odds with the purpose and aims of equitable public education.  

Bellwether Education Partners’ report, A New Agenda, calls for a “rational” and “rigorous” research agenda for teacher education. Although the report’s rationale is not fully explicated, it asserts that programs are “blindly swinging from one popular reform to the next” and that decades of input- and outcome-based research has failed to improve teacher education. Instead, the report calls for “rapid cycle evaluations.” Regrettably, this depiction of past research includes mischaracterizations and also omits a wide swath of relevant literature about teacher education. Further, the report does not adequately explain what “rapid cycle evaluations” would entail or how they would work to improve teacher preparation program design, nor does the report offer a research foundation for this approach. The report also fails to recognize the socio-political context of teacher education, wherein programs are often left scrambling to meet competing accountability expectations. A New Agenda leaves practical questions unanswered, muddies the waters about promising research avenues, and ignores important bodies of literature in teacher education. Ultimately it represents a missed opportunity to offer guidance to either policymakers or institutions. 

Based on a review of GPA and SAT/ACT requirements at 221 institutions in 25 states, a new report from the National Council on Teacher Quality (NCTQ) recommends that states, institutions of higher education, and the Council for the Accreditation of Educator Preparation (CAEP) maintain or establish a higher bar for entry into teacher preparation programs. The NCTQ report suggests that boosting teacher candidate entry requirements in ways they advocate would significantly improve teacher quality in the U.S.. Yet the report does not provide the needed supports for its assertions or recommendations. In addition, the report makes multiple unsupported and unfounded claims about the impact on teacher diversity of raising admissions requirements for teacher candidates, about public perceptions of teaching and teacher education, and about attracting more academically able teacher candidates. Each claim is based on one or two cherry-picked citations while ignoring the substantial body of research that either provides conflicting evidence or shows that the issues are much more complex and nuanced than the report suggests. Ultimately the report offers little guidance for policymakers or institutions. 

Philanthropic involvement in K-12 education is growing, and it increasingly shapes the direction of reforms pursued throughout the country. A recent report from the NewSchools Venture Fund offers a thought experiment on how philanthropists can make a “big bet” over the next decade on innovative schools—a broad category that generally includes schools with a high degree of education technology use and so-called personalized approaches to learning that likely utilize digital platforms. Unfortunately, the report fails to provide a meaningful examination of research or a thorough basis for its recommendations. This critique focuses on six key concerns regarding the report: it fails to consider human capital constraints or to sufficiently consider obstacles confronting classroom technology usage, it overlooks equity concerns and past problems with dependence on external professional services, and it ignores both the potential for disruptive reform churn and the danger of philanthropic efforts altering public education systems in undemocratic ways. For these reasons, the report’s usefulness to policy and practice is limited.

Although teacher evaluation reform has rapidly expanded across the U.S. in recent years, the newly enacted Every Student Succeeds Act (ESSA) seems to mark a federal deprioritization of teacher evaluation reform. This may in turn prompt states to consider changes to their evaluation systems. A new report from Bellwether Education Partners, For Good Measure? Teacher Evaluation Policy in the ESSA Era, argues for the maintenance of key elements of high-stakes teacher evaluation, including the use of student outcomes to evaluate teachers and a heavy focus on accountability. It urges policymakers to move slowly in revising their systems and to invest in management, capacity, and strategies to capture lessons learned. While the report raises several good questions with regard to the future of teacher evaluation, it overstates the likelihood that ESSA will result in widespread changes to evaluation systems, it ignores the literature regarding substantial technical challenges and unintended consequences of growth measures, and it dismisses the ideological and political debates surrounding teacher accountability. The unsubstantiated claims and dogged defense of student growth metrics provides little fresh or worthwhile new directions to policymakers seeking a nuanced and research-based discussion of teacher evaluation reform in the ESSA era. 

The stated goal of this report is to strengthen the evidence base on state-initiated turnarounds and to provide guidance to help states use turnaround strategies more effectively. The report draws on multiple sources of information to develop a conceptual framework and profile of state-initiated turnaround strategies, to array the evidence on the effectiveness of turnaround initiatives, and to identify key elements of a successful turnaround strategy. However, given multiple methodological limitations, the report fails to elevate either the research base or the policy discourse. Specifically, the methods used to carry out the original research (e.g., analysis of state policies, interviews with stakeholders, and illustrative cases) are neither explained nor justified. Likewise, the methods employed in the eight evaluations selected to assess the effectiveness of turnaround approaches are not described, and the evidence base produced by these evaluations is not sufficient to support the sweeping claims made in the report. Equally important, the report neglects to consider relevant research on the specific mechanisms (e.g., school reconstitution, intensive professional development, private management systems) that states use when they employ the broad turnaround strategies discussed in the report. As a result of these problems, the report does not enhance the evidence base or provide the substantive guidance state policymakers require to make informed decisions about the use of various school turnaround strategies. 

Two recent reports from the Fordham Institute address the question of the impact of state accountability systems on “high achievers,” referred to in the reports as “students who have already crossed the proficiency threshold.” Both reports argue that this group is being neglected educationally, and they advocate for accountability systems to be redesigned to attend to the needs of high-achieving students. Both reports also recommend that states use a “performance index,” as opposed to proficiency rates, to measure school achievement. This review, however, concludes that: 1) the reports’ central assumptions about high-achieving students are problematic; 2) growth measures are not an effective means for directing attention to high-achieving students; 3) narrow, high-stakes forms of assessment may negatively impact the education provided to these students; and, 4) further stratifying educational settings and reallocating resources toward “high-achieving” students has troublesome implications for the democratic goals of education. Implementation of the reports’ recommendations may in fact result in a furthering of the inequitable educational opportunities that ESSA was designed to reduce.

Better Evidence, Better Choices, Better Schools, a recent report from the Center for American Progress and the Knowledge Alliance, focuses on the evidence-based research provisions in the Every Student Succeeds Act (ESSA). ESSA provisions ask districts and schools to consider various sources of evidence, make judgments about the strength and reliability of that evidence, and use that evidence to justify their choices of interventions. Contrasting these new standards with the scientifically based research requirements featured in previous federal legislation, the report aims to provide guidance for the state and local education agencies now tasked with implementing evidence-based school improvement practices. The guidance consists of eight recommendations for implementing evidence-based reform strategies: defining roles, using evidence clearinghouses, supporting robust decision-making, ensuring high-quality service providers, promoting communication, partnering with intermediate organizations, facilitating effective implementation and promoting continuous learning. While helpful general pieces of advice, these recommendations remain relatively vague and are grounded in neither the existing research literature nor the empirical study featured in the report. The nebulous idea of evidence-based policy is easy to agree about, but skims over more challenging questions. For instance, what counts as evidence, and who is able (and authorized) to determine what kinds of evidence are most relevant to local contexts? While this report raises a number of important issues, it leaves these more difficult questions unexplored.

Noting the nation’s renewed attention to remedying school segregation, Segregation, Race, and Charter Schools presents evidence about the extent of school segregation and its relationship with improving student achievement for students of color. The report argues that school segregation has remained flat for decades and also argues that the nation would be wise to instead attend to improving the quality of schools that students of color and low-income students attend. It points to some urban charter schools as exemplars of this latter approach. However, this review finds that the report omits significant research directly related to the topic and includes other studies that are less relevant. Moreover, the report draws questionable conclusions from studies that are included—conclusions that are not reflective of the research consensus. The report’s selective interpretation of existing research leads to two erroneous conclusions about improving educational outcomes for students of color: (1) that focusing on school integration is relatively unimportant; and (2) that attending to school quality via school choice, rather than addressing the complex array of policies to combat racial segregation, should instead be pursued. In fact, because most forms of school choice further segregation, the report’s recommendation will likely only further segregation and inequality for students.

Two new papers from researchers at the University of Arkansas predict the budgetary consequences of terminating the Louisiana Scholarship Program (LSP), a voucher program that funds over 7,100 Louisiana students to attend private schools. Using an economic model, the papers offer several different scenarios and then conclude that terminating the LSP would increase the costs statewide and do so in almost all districts in the state. The papers’ findings are reasonable but do not make a fully convincing case that the state will incur extra expenditures without the LSP. There may be savings or additional expenditures, depending on several key parameters which have not been precisely estimated. Puzzlingly, the reports’ findings of extra costs run counter to those of the state’s Legislative Fiscal Office, but the reports do not mention this contrary evidence. In any case, the net fiscal effect of terminating the LSP is unlikely to be large; in the context of a $9 billion state expenditure, the change would likely be less than $10 million.

A report published by the Progressive Policy Institute calls for aggressively closing more public schools and expanding charter schools and charter networks. It highlights reforms adopted by Denver Public Schools, notably a “portfolio model” of school governance, and argues that these reforms positively impacted student test scores. However, causality cannot be determined, and the report did not attempt to isolate the effect of a multitude of reforms—including charters, performance pay, and a new performance framework—from larger complex forces shaping student demographics in the city. Written in a reportorial voice, the only data presented are in the form of simple charts. The lack of conventional statistical analyses thwarts the reader’s understanding. The report also characterizes the reform’s adoption as a “political success” born of a healthily contentious electoral process. In doing so, it downplays the role of outside forces and moneyed groups that influenced the form of reforms, and it disregards missed opportunities for meaningful engagement with community stakeholders. Finally, while the report acknowledges the district’s failure to close achievement gaps and admits limitations with the evaluation system, it never explains how a successful reform could generate a widening gap in performance between student groups by race and class.

Update: David Osborne, the report’s author, has posted a response to the review. The response can be found at: http://www.progressivepolicy.org/blog/response-national-education-policy-center/

Terrenda White’s rejoinder to the author's response to her review is posted immediately below the review.

Measuring School Turnaround Success contends that the lack of a shared definition for turnaround success makes it hard for reformers to learn from each other and determine true success. The report claims to develop a model for defining turnaround success with measures, metrics, and cut scores that reach beyond student achievement tests, to include indicators of engagement by students, parents and teachers, teacher and leader effectiveness, and short-term learning outcomes. Unfortunately, given the dearth of research evidence and sound methodological techniques incorporated into its analysis, as well as the omission of several rigorous, peer-reviewed studies that contradict the majority of its proposals, the report does not meet a minimal standard of evidence to support its claims. Ironically, the report focuses largely on standardized test scores, despite its stated intentions. The result is a report that distracts attention and potential resources from schools’ other goals, including civic, social, emotional, and broader academic ones. In the end, the report puts forth yet another proposal, funded by public dollars, that encourages state leaders to continue over-relying on flawed, test-centered strategies. Policymakers and practitioners looking for guidance on measuring turnaround success will not find worthwhile recommendations. Instead, they will encounter several unsubstantiated ones and others that are contradicted by solid peer-reviewed research.

The authors of this report created “efficiency scores” for Milwaukee schools and, on the basis of those scores, draw conclusions about the relative efficiency of Milwaukee’s charter and traditional public schools. A first set of scores was generated by dividing test scores in math and science by per-pupil funding. In a second analysis, “efficiency scores” were estimated for school types using a regression model that also accounted for selected student demographic characteristics. Several major problems arise from the authors’ approach: First, test scores do not comprehensively represent the purposes of schools. Second, threats to the validity of the author’s assumption that there is uniform financial accounting across schools and types are not addressed. Third, the analytic description of the study was incomplete, making interpretation difficult. Fourth, the report makes strong but unmeasured claims about the superior “efficiency” of charter schools based on their having greater autonomy. Finally, the authors did not appear to adjust for selection effects – effects that would prove fatal to their conclusions.  As a result of its manifold flaws, and because its conclusions are not supported by the evidence presented, the report is of little if any use to policymakers.

Update: Will Flanders and CJ Szafir, the report's authors, have posted a response to the review. The response can be found immediately below Cobb's original review of their report and at: http://www.will-law.org/will-blog-will-responds-criticism-colorado-group/.

Casey Cobb's rejoinder to the authors' response to his review is posted immediately below the authors' response.

Two reports claim to offer empirical support for the efficacy of voucher programs that allow parents to use taxpayer dollars to send their children to private schools. One report (A Win-Win Solution) is the latest in a series from the Friedman Foundation for Educational Choice. The Friedman report reviews studies purporting to show positive impacts from voucher programs in the US. The other report (The Participant Effects of Private School Vouchers across the Globe) is from the Department of Education Reform at the University of Arkansas. The authors of the Arkansas report conducted a limited meta-analysis of US and international studies of voucher programs. The two reports share a positive view of the impacts of vouchers, and both focus on randomized studies of those effects. Both reports are marred by a number of serious problems and errors, including misrepresentations of the research literature, a failure to acknowledge the limitations of their approaches, not addressing the shortcomings of the theoretical underpinnings of vouchers, and the use of methods that bias the selections of the studies they utilize. The Friedman report is a rudimentary “vote-counting “analysis of an extremely narrow set of 18 studies using a biased counting system. The Arkansas meta-analysis aspires to be “global,” but despite identifying over 9,000 potential studies for the analysis, ultimately uses only 19, almost half of which were conducted by the Arkansas authors or their associates. Moreover, the “global” meta-analysis only encompasses three countries (one of which is consistently misspelled). Together, their manifold serious flaws undercut the trustworthiness and usefulness of these reports.

A report from Michigan’s Mackinac Center asserts that there is little or no relationship between student achievement and marginal increases to what the report characterizes as the already “high” levels of spending in that state. Yet the report never substantiates its assertion that present spending levels are high, on average, or uniformly high across all children, districts, or schools statewide. The report discounts a significant body of peer-reviewed research that specifically shows positive effects of previous Michigan school finance reforms, including positive effects on state assessments and educational attainment, concentrated on those students who attended, before those reforms, the lowest funded schools or lower performing schools. Additionally, while the report argues that increased spending on schools as they presently exist would necessarily be inefficient and ineffective, this contention is undermined by the lack of evidence for more efficient alternatives and by existing research validating the value of traditional resources. Both a major national study and a Michigan-specific study show funding increases as efficacious when allocated primarily toward traditional investments (increased teacher salaries and smaller class sizes). Finally, the empirical analysis included in the report lacks depth and rigor when compared to four other studies—three of which were peer-reviewed—each of which find positive effects of prior school finance reforms in Michigan.

This report uses state-level data from the National Assessment of Educational Progress to describe a positive association between tracking in eighth grade and the proportion of students passing AP exams in high school. The relationship is moderately strong and holds true for White, Black, and Hispanic students. The report suggests that the separate learning environments for high achievers created by tracking are important for providing students (including students of color) with the skills and knowledge to succeed with the most demanding coursework offered in high schools. The findings are based on correlations and cannot establish a causal relationship, nor can they identify what mechanisms might be at work. However, they are consistent with prior research that has frequently (although not always) identified benefits of tracking for high-achieving students. A key weakness is that the report neglects to consider how tracking is likely to affect lower-achieving students. Tracking is often implemented in ways that hinder the learning of students assigned to low tracks. Because disadvantaged and minority students are disproportionally assigned to low tracks, the report’s conclusion that tracking could be “a potential tool for promoting equity” is dubious.

Relative to earlier research, this study from the Center on Reinventing Public Education provides a more in-depth analysis of policy features across the 27 states that allow online charter schools. It presents a well-organized description of policy features and includes a set of policy recommendations that generally, but not always, follow well from the study’s evidence. Because the findings are largely negative, this report may be seized on by groups that are broadly critical of charter schools. But the report is published by an organization that often advocates for charter school growth, and the authors’ discussion of findings suggests that the charter school establishment is advocating for separating online charter schools from brick-and-mortar charter schools and governing them with a separate policy framework. While there is some justification for such an approach, online charter issues are not completely distinct from issues that arise from other charters. Overall, the detailed analyses of policy environments and the summary of problems in the online charter school sector included in this report should be useful to policymakers who are willing and able to pursue more restrictive oversight and increase accountability for online charter schools.

In a report, University of Arkansas researchers studied the relationship between school vouchers and crime. Using data from the Milwaukee Parental Choice Program (MPCP), they compared crimes processed through the Wisconsin courts for program participants and a matched sample of public school students. Based on a series of comparisons, it finds that some groups of MPCP students are less likely to commit crimes as adults. This result is plausible: education and crime are often found to be negatively correlated and the MPCP has generated some, albeit modest and mixed, benefits. However, the study’s title should not imply that voucher programs are a "get-out-of-jail" card, and the evidence in the study is not causal. One concern is that the paper employs a matching method that omits some important factors that explain school choice and crime. Also, the results are highly variable, with most of the association between MPCP participation and measures of adult crime showing statistically insignificant results. Indeed, a valid interpretation of the paper is that vouchers and crime are not correlated. Conversely, for subgroups and estimation approaches that do yield statistically significant associations, the MPCP effects appear to be too extreme. Even assuming that vouchers do reduce adult crime, it remains unclear by what mechanisms vouchers might do so.

As part of an ongoing series of reports by the National Council on Teacher Quality (NCTQ), Learning About Learning: What Every New Teacher Needs to Know makes broad claims about teacher education based on a limited analysis of textbooks and syllabi. The report argues that teacher education materials, specifically educational psychology and methods textbooks, are a waste of funds and do not adequately focus on what the report identifies as six essential strategies. These inadequacies, the report contends, result in ill-prepared teacher candidates lacking in “research-proven instructional strategies” (p. vi). The report offers recommendations for textbook publishers, teacher education programs, and state departments of education. However, it is not grounded in a comprehensive examination of the literature on teaching methods, and it fails to validate the evaluative criteria it employs in selecting programs, textbooks, and syllabi. The single source it relies on to justify its “six essential strategies” provides limited support for NCTQ’s claims. This primary source concludes, with only one exception, that the evidence supporting each of the six strategies is only moderate or weak. Limiting the analysis to one source that provides only tepid support renders the report of little value for improving teacher preparation, selecting textbooks, or guiding educational policy.

The Manhattan Institute's SchoolGrades.org evaluates and assigns grades, using reading and math test scores, to U.S. schools and compares schools across their respective states and to other countries. They apparently use a four-step process: (1) average two state test scores; (2) “norm” these results to the NAEP exam; (3) make an adjustment to this national “normed” measure using free and reduced price lunch data to account for SES; and (4) “norm” these results to the international PISA exam. The claim is that this process allows a parent to compare a local school to schools in their state and to other countries like South Korea and Lithuania. But the unsubstantiated norming chain is too tenuous and the results are overly extrapolated to be of any useful value. The website does not explain how international scores are “normed” (equated) to the national standard they developed or how letter grades were determined, nor does it explain how free and reduced price lunch counts are used to make socioeconomic adjustments. While there is considerable equating research available, none is cited. Further, the reliance on aggregated test scores is far too narrow a base to serve as a useful evaluation of schools. Thus, the website’s approach to evaluating schools fails on technical grounds and, just as importantly, it fails to understand and consider the broader purposes of education in a democratic society.

This Center for American Progress report examines whether states’ adoption of standards-based policies predicts low-income students’ NAEP achievement trends in fourth and eighth grade math and reading throughout the 2003-2013 decade. The report claims to analyze changes across five separate two-year intervals, but it only reports findings for 2009-2011, with no explanation of why or any documentation of the representativeness of that single interval. The reported finding for the selected interval is that state adoption of standards policies positively predicts fourth (but not eighth) grade math NAEP and eighth (but not fourth) grade reading NAEP. Even these selected positive results are statistically significant only at the generally unacceptable 0.10 level of significance. The report includes effect sizes but nothing about the percent of the variance explained in their model. In short, the report does not adequately describe variables or analytic methods or completely report findings, and the data and methods used do not allow for any causal findings. They use state standards adoption across grade levels and subject areas as well as selected accountability mandates as predictors but fail to assess their quality or fidelity of implementation. Yet based on these very problematic and limited analyses, the authors conclude that their analysis “strongly supports the potential of the Common Core to drive improvements in educational outcomes.” The study simply does not support this conclusion or the set of recommendations that follow. 

A report investigates the enrollment and achievement of students with disabilities and students with English language learning (ELL) needs in oversubscribed charter schools in Boston. The report concludes that Boston charters and Boston Public Schools enroll similar numbers of both special populations and that charter attendance has a positive statistically significant effect for those who enter Boston’s charter school lottery and then enroll after being offered a seat. This review finds that econometric models used to estimate the effects are appropriate, but also more limited than the report would suggest. The report finds some interesting patterns that deserve further study; however the effects cannot be generalized to charter schools outside Boston or even to most students inside Boston. The study also offers no context to compare the size of reported gains and it does not adequately examine how or why the reported test score gains are realized; for example, it does not account for peer effects or spending differences. Ultimately, while this report takes an important step in studying how oversubscribed charters may affect the academic achievement of special needs students, a closer examination is needed in order to accurately inform those making education policy. 

An evaluation report from RAND focused on school-wide initiatives funded by the Bill & Melinda Gates Foundation to promote teaching approaches touted as personalized learning. These reforms generally rely on digital technology and encompass a range of strategies, such as developing learner profiles with individualized goals, and using data to provide personalized learning paths in which students have choice, get individualized support, and engage in learning outside school. The research, which includes many high-quality elements, suggests that some of the studied approaches are associated with higher scores on a common assessment (the MAP). Broad conclusions about the efficacy of technology-based personalized learning, however, are not warranted by the research. Limitations include a sample of treatment schools that is unrepresentative of the general population of schools, the lack of a threshold in the study for what qualified as implementing “personalized learning” in the treatment schools, and the reality that disruptive strategies such as competency-based progression, which require the largest departures from current practice, were rarely implemented in the studied schools.

This report from the Center for American Progress offers 10 recommendations for improving the public perceptions of and experiences of classroom teachers. While elements of these recommendations would likely be beneficial, they also include policy changes that would increase surveillance of teachers, reduce teachers’ job security, evaluate teachers by students’ test scores, and create merit pay systems that would likely have the opposite effect. For evidence, the report relies too heavily on popular rhetoric, sound bites, opinion articles, and advocacy publications to advance a policy agenda that in many ways could do further harm to the teaching profession. However, many of the report’s recommendations do align with policy reforms currently being proposed for the Higher Education Act and included in the Elementary and Secondary Education Act reauthorizations and are therefore important to read critically and consider carefully. In advancing evaluation of teachers by test scores, the report goes against the cautions and guidelines recently released by the American Statistical Association and the American Educational Research Association. Other than a review of contemporary issues, the report offers little of substance to advance the teaching profession.

The Integration Anomaly explores a “puzzling divergence” between changes in metropolitan residential and school segregation. Based on a review of existing literature, it argues that the best way to address rising school segregation is to decouple school assignment from neighborhoods through universal school choice. The report concludes with suggestions for how to structure school choice to achieve integration. On the surface, the report provides clear-cut, useful recommendations for addressing persistent school segregation. Yet the analysis of the empirical relationship between school and residential segregation relies on flawed methodological decisions concerning how to define segregation and divergent trends over time. Those problematic definitions, in turn, yield biased results and prompt the reader to incorrectly assume that housing integration policies will have little bearing on school segregation. Moreover, the report’s review of the literature on school choice is haphazard and incomplete, drawing conclusions beyond what the research supports. Perhaps most importantly, The Integration Anomaly ignores a growing body of literature finding that the very type of unregulated school choice it proposes has, in many instances, exacerbated racial segregation. The report presents arguments and solutions largely driven by ideology, not evidence, offering little value for policymakers or educators meaningfully engaged in the critical search for strategies to reduce school segregation.

A response from report author Benjamin Scafidi, and a further response from review authors Genevieve Siegel-Hawley and Erica Frankenberg follow.

A recent NBER working paper examines Florida’s policy to retain many low-scoring third graders. The report concludes that third-grade retention has immediate positive effects on the following year’s test results, but these effects fade over the next six years, with no effect on graduation. The regression discontinuity methods used to estimate the effects, comparing students immediately above and below the law’s cut-score, are generally good for making causal claims. But there is a serious shortcoming in the design—namely, the law requires that students below the cut-score receive intensive extra services intended to raise their subsequent achievement, and this applies to those retained and those promoted. This means the researchers do not know if these positive outcomes for those below the cut-score were due to the greater likelihood of retention or to the assurance of additional services. Also, two-thirds of students who fall below the cut-off score are nonetheless promoted because they fall into exception categories. Finally, the researchers exacerbate outcome differences by using an Instrumental Variable approach, which attributes the entire above/below-threshold difference to just retained students, effectively making the outcome difference appear more than three times as large. Because the policy stipulates that promoted students below the threshold also receive extra services that promoted students above the threshold do not receive, the IV approach is inappropriate. Even setting aside these problems, the study has extremely limited generalizability, restricted to students at or very near the threshold and directly affected by the policy. These and other problems call into serious question any causal claims of the longer-term effects and any policy utility for the report.

A recent Center for American Progress report, The Hidden Value of Curriculum Reform, draws bold conclusions about the high payoff of better textbooks. It finds that textbooks are rarely chosen based on evidence of effectiveness and true alignment with standards. It also finds that elementary mathematics textbook prices vary little, regardless of quality or whether or not a state recommends particular texts for adoption. While these elements of the report have merit, it then overreaches. Based on a single prior study of 1st and 2nd-grade math curricula in some high-poverty schools, it draws general conclusions about the Return on Investment (ROI) for good versus weak textbooks, ignoring key findings within the original study and also ignoring other research showing that curricular effects vary, depending on context and implementation—that is, a good book is no guarantee of benefits. The report then compares its estimated ROI for textbooks with another study’s reported ROI for other interventions, while ignoring nuances of that second study’s calculations. Overall, Hidden Value is timely and provides important insights about the need for evidence-based curriculum selection. However, its highly optimistic claims about curricular ROI could reduce support for other worthy interventions and spur simplistic comparisons of textbooks that lead to naïve conclusions about which curriculum is “best.”

A recent report published by a new project at Brookings called “Evidence Speaks” claims that advocates exaggerate unmet need as well as the cost of universal pre-kindergarten (UPK). It estimates that 69 percent of all four-year-olds already attend preschool and that universal access tops out at 80 percent enrollment. To close this modest gap, the report proposes a means-tested subsidy for half-day preschool that fully funds only those in poverty. The estimated cost is $2 to $4 billion per year. Unfortunately, the report vastly underestimates unmet need and costs; both estimates are based on serious factual errors and unfounded assumptions. Access to high-quality preschool (as opposed to attendance in any preschool classroom) for four-year-olds is under 25 percent, not 69 percent. High-quality UPK could enroll more than 90 percent, not 80 percent, of children. Enrollment rates go up with better policy design. Also, the Brookings plan would leave one in five children in low-income families with no access to any preschool education and, since the plan ignores dosage and quality, would leave many more in weak or ineffective programs. The report’s conclusions regarding needs and the costs to meet them are invalid and misleading, and it should not be used as a basis for policymaking.

A recent report published by two organizations, Public Impact and New Schools for New Orleans, reviews the ten years of education reform in post-Katrina New Orleans, and the creation of a “portfolio model” of school governance. While the report acknowledges some of the critiques of these reforms, it exaggerates improvements and inaccurately downplays the ways that the reforms created and, in some cases, exacerbated inequities. In addition, the report erroneously presents the reforms as the result of a logical and apolitical process that simply recasts the role of government in public education, changing from the manager or operator of schools to “a regulator of educational outcomes and equity,” with operation turned over to a variety of non-profit organizations. Finally, the report overstates its claims of post-Katrina academic gains in New Orleans, evidence for which is scant and has significant limitations, particularly due to the repeated changes in test-score standards across the decade. For these reasons, the report does little to accurately inform the lay public or scholars about the current state and future of public education in New Orleans or the viability of “portfolio” districts. 

The Mirage: Confronting the Hard Truth About Our Quest for Teacher Development argues for fundamental changes in the way public school districts think about teacher growth. Using original data collected from teachers and administrators in three public districts and one charter network, the authors contend that although the public districts invest heavily in teacher professional development, what is offered is often a poor fit to teacher needs and ultimately ineffective as a means to improving teacher evaluation scores. Many descriptive portions of this study are strong – including evidence that public district teachers are doubtful about the utility of their own professional development and that public district offerings often lack coherence. The per-teacher cost estimate for professional development, however, was calculated by including teacher salary increases that result from professional development credits and master’s degrees, a choice at odds with much of the prior research on this topic. As well, the analysis comparing growth in teacher evaluation scores to teachers’ professional development experiences suffers from a number of issues, including a mis-match between the behaviors rewarded by teacher evaluation and the professional development features explored in this study. That said, readers who rely more on the report’s empirical evidence and less on its hyperbolic statements will profit from reading this report. 

Program variation has long been touted as a charter school advantage, so a recent AEI report rates the diversity of charter school programs in 17 major cities. Examining charter school websites, the report finds the schools evenly split between “Specialized” schools (e.g., no excuses, STEM, or Arts) and “General” schools. It finds small to moderate correlations between city demographics and certain types of charter schools but also finds that specialized schools tend to morph into homogenized general schools. The report has several weaknesses. It claims superior diversity for charter schools but doesn’t empirically compare charter offerings with those of traditional public schools, which typically include many diverse options. Similarly, the introduction claims charter schools are hamstrung by red tape, but this is not addressed in the report. Also, as acknowledged in the report, coding schools based on website descriptions is an error-prone enterprise, yet no check of the accuracy of the data is provided. As the correlations between charter schools and city demographics are based on only 17 cases (cities are the unit of analysis), this is too weak a base to support the report’s conclusions. Fundamentally, the report does not make the case for its major claims, and thus has only minimal utility.

A recent report found a general increase in test scores of students displaced from closed urban schools in Ohio, who were disproportionately African American and low-income. Students displaced from closed charter schools showed gains in math but not reading, relative to students from non-closed schools; students displaced from district schools showed gains in math and reading. Gains associated with closure were greater for students who transferred to “higher-performing schools”—those with higher test scores. Overall achievement growth in receiving schools, however, decreased in the year that they accommodated displaced students. Although the finding that displaced students showed improvement in test scores is encouraging, several factors limit the study’s policy implications. The report itself cautions that the potential for test-score gains depends on the availability of higher-performing schools for displaced students, a condition often unmet. Forty percent of students in closed schools transferred to schools that were not higher performing. Also, because demographic data were not reported about the receiving schools, an alternative explanation—that displaced students benefited from transferring to schools with less economic and racial segregation—was not explored. School closure also raises moral and political questions about democratic decision-making and community voice.

This report asserts that more stringent accountability measures for schools (i.e., high academic standards for public school students) along with benchmarks for inclusion in state testing have improved the quality of education for students with disabilities. It compares 2000 to 2013 NAEP and NCES national-level data and finds increased test scores, decreased dropout rates, and increased graduation rates for students with disabilities, as well as improved outcomes for Black and Hispanic students with disabilities. While student outcomes have improved for students with disabilities, they cannot be causally connected with NCLB or NCLB-type reforms. This report is based on simple descriptive comparisons and assumes its interpretations and conclusions without any foundation. While an expansive research literature is available, none was used in this report. Further, aggregating data across the nation over 14 years obscures a multitude of possible other interpretations as well as hides regional, temporal, governmental and state variations. Consequently, the report does little to advance public policy for students with disabilities.

A recent report discusses three commonly held propositions about education’s economic power: 1) education is the critical factor in creating economic prosperity; 2) college degrees increase earning power; and 3) increasing educational attainment will narrow income inequality. The report endorses the first two propositions but finds the third inaccurate, concluding that a significant increase in educational attainment is not likely to significantly decrease wage inequality. The use of an empirically based simulation to project what would happen if an additional 10 percent of the population suddenly received college degrees is illuminating. However, the analysis has important limitations. There is little evidence provided to show that increasing educational attainment is, as the authors contend, “the most effective and direct way” to improve economic prosperity. The data are drawn only from males and no attention is paid to how income gains differ across race, field of study, labor-market conditions, and institutional reputation. Critically, no analysis compares education with other approaches to economic problems. Claiming that the primary solution to important economic problems is to improve “human capital,” the report perpetuates a problematic myth that undervalues alternative approaches to poverty and economic insecurity. Indeed the knowledge society narrative, assuming that everything depends upon more education, may itself be flawed. 

Dramatic Action, Dramatic Improvement: The Research on School Turnaround advocates for implementing the most effective, research-based methods for turning around low-performing schools through the federal School Improvement Grant (SIG) program. It argues that the available body of research points to five dramatic actions that are necessary to bring about dramatic school improvement. Unfortunately, the rationale for its assertions is narrow, incoherent, and misleading. This limitation stems from the report’s unsystematic review of literature, resulting in its failure to incorporate lessons from large bodies of research on high-stakes accountability, school improvement, and the emerging evidence on school closures and federally funded turnarounds—all of which reveal that the federal SIG program’s turnaround policies are based on unwarranted claims and are contradicted by the empirical evidence. Consequently, the report’s recommendations are unsupported by rigorous research. Like the SIG program itself, the report promotes seemingly bold school changes that appear seductive on the surface. But when compared to the real evidence on school turnarounds, their evidence-based foundation vanishes. In the end, schools, districts, and states that follow the report’s advice stand to reproduce the unequal conditions that have led, in part, to their need for dramatic turnaround in the first place.

To investigate claims that New York City charter schools push out low-performing students in order to inflate academic achievement scores, this report uses six years of student-level data that allows for tracking student movements over time in both district and charter schools. It relies on a regression equation that includes “low scorers” (defined in several ways), whether a school is a charter or a traditional public school (TPS), and an interaction between those two characteristics in order to estimate the probability of a student exiting. The report concludes that charter schools have exit rates similar to TPSs, and there is no significant interaction between low-performing students and charter schools; and thus, there is no charter push-out effect for low-achieving students. The main problem is that the research design does not address its primary push-out question. The brief report has little detail and does not examine a host of other relevant factors. Dichotomous test scores are a proxy for low-achievement, reasons for disenrollment are not addressed, mid-year vs. end-of-year mobility is not parsed, cumulative rates of attrition are not examined, a possible data discrepancy between the two sectors in grades 5 and 6 is not considered, and 5% of the student population is missing. These unresolved issues are particularly regrettable in light of the rich dataset. While the report’s central question is important, this report fails to provide policymakers with new or definitive guidance.

Following up on a previous study, researchers sought to investigate whether the effect on reading and math scores of being in a charter school was different in urban areas compared with other areas and to explore what might contribute to such differences. Overall, the study finds a small positive effect of being in a charter school on both math and reading scores and finds that this effect is slightly stronger in urban environments. There are significant reasons to exercise caution, however. The study’s “virtual twin” technique is insufficiently documented, and it remains unclear and puzzling why the researchers use this approach rather than the more accepted approach of propensity score matching. Consequently, the study may not adequately control for the possibility that families selecting a charter school may be very different from those who do not. Other choices in the analysis and reporting, such as the apparent systematic exclusion of many lower-scoring students from the analyses, the estimation of growth, and the use of “days of learning” as a metric, are also insufficiently justified. Even setting aside such concerns over analytic methods, the actual effect sizes reported are very small, explaining well under a tenth of one percent of the variance in test scores. To call such an effect “substantial” strains credulity.

This report uses two methodological innovations to measure the impact of advocacy groups on education reform policy, Surveys with Placebo (SwP) and Critical Path Analysis (CPA). SwP is designed to measure the influence of advocacy groups and CPA to identify which tactics were successful in influencing reform. The report finds them to be effective methods for achieving these goals. It finds that coordination of advocacy groups strengthens their impact and that the perceived impact of advocacy groups tracks closely with policy outcomes. While we agree that the SwP and CPA may be useful in education policy research, these methods are more limited than the report acknowledges. Moreover, the research is a small case study of three states, with a low response rate for the SwP and CPA based on advocacy groups’ self-reported tactics. The report also fails to give sufficient information on the responses to the SwP and the selection of the advocacy groups studied to assess the usefulness of methods and validity of the report’s conclusions. Finally, there is not a strong connection between the evidence presented in the report and its conclusions. We therefore caution against adoption of the methods or reliance on the conclusions presented in this report without significant further research.

Critics of typical “single-salary” pay scales for teachers argue they under-compensate great teachers and over-compensate inept teachers. As a result, many pay-for-performance plans have been tried, with mixed results. This report proposes yet another variable compensation plan: paying the top quartile of teachers in a district a bonus for accepting up to three additional students into their existing classes. Without evidence, the report posits that having more students work with more effective teachers would offset any potential sacrifice in student learning. It projects significant district savings because larger classes would allow substantial reductions in faculty. The report does not address the well-documented problem of identifying high-performing teachers; it misrepresents the known effect of class size on student learning; and it ignores what is known about teacher pay, attitudes and job satisfaction. Projected outcomes are based on average class sizes, which obscure the potential impact on thousands of teachers and students in already overcrowded classrooms. Moreover, its proposed bonuses obscure the fact that teacher salaries overall are too low to recruit and retain sufficient numbers of talented faculty, especially in high-needs schools, and would likely remain so even under this more-work-for-more-pay model. Rather than a practical response to known issues with single-salary pay scales, the proposal seems primarily a scheme to reduce the teaching force. The report is superficial and misleading, and its proposal has no value as a nationwide model.

The irony of a recent report from the National Alliance for Public Charter Schools is that it purports to “separate fact from fiction” about charter schools. Unfortunately, in addressing 21 “myths,” it embraces fiction whenever useful to push advocacy goals, thus perpetuating its own myths and fictions about charter schools. Since it relies overwhelmingly on other advocacy documents, it does not give a balanced or thorough examination of any of the 21 “myths.” But the exercise provides a useful opportunity for this review to walk through the various claims and succinctly address each. Among the areas addressed are the financial equality of charter schools, lower teacher qualifications, student selection demographics, academic outcomes, segregation, and innovation. While the NAPCS report itself may provide only sound-bite fodder for advocates, we hope that the two documents combined—report plus review—offer an overview of issues that does advance comprehensive understanding.

Please note: This review was replaced on March 3, 2015, due to a substantive change in the content of Endnote 24, which is described in the endnote itself.

In a recent report, The Texas Economy and School Choice, written by Arthur Laffer for the Texas Association of Business and the Texas Public Policy Foundation, Laffer evaluated the effect of the proposed Taxpayer Savings Grant Program (TSGP)—essentially a universal voucher program designed to provide school choice to every student in Texas. Laffer concludes that by raising graduation rates, improving education achievement, and thus increasing human capital, the TSGP would substantially raise wages and income for working families, thereby improving economic growth in Texas. In this review, we highlight two profoundly problematic areas of the report. First, Laffer’s assertions about the educational benefits of choice represent a severe overreach with and misapplication of the available research. Second, the author’s economic estimations are over-generalized and heavily biased towards those families who already have the wealth to choose and relocate. The report applies simplistic economic logic to education and fails to consider all but an extremely narrow and inappropriate slice of research on education, making the report unsuitable as a basis for public policy decisions. 

This report presents the findings of research on parent organizing within what the report calls Education Reform Advocacy Organizations (ERAO), such as Stand for Children and Parent Revolution. The ERAO agendas focus on standards, test-based accountability, teacher tenure reform, and parent choice. The report recognizes that ERAOs are often criticized as astroturf organizations that mobilize parents behind their agendas instead of building authentic parent power; it then presents some evidence that many groups are committed to long-lasting parent engagement, and it reports on the community organizing practices they use to build parent leadership. Unfortunately, the report’s presentation of research methods is so weak that the research cannot be relied upon without a better idea of the rigor or lack of rigor of its approach. The findings may be valid for the groups studied, but the selection is biased towards ERAOs that work to build sustainable forms of parent participation; it is unclear if those groups are representative of the broader field of ERAOs. While the report suggests some community organizing strategies can be used to advance the ERAO version of education reform, this approach undermines an understanding of community organizing as a democratic practice through which organizations and agendas emerge out of the concerns and through the actions of indigenous community leaders working to build a more inclusive public democracy.

The working paper reviewed here seeks to assess the extent to which “No Excuses” charter schools raise student achievement in English language arts and math and thereby close the achievement gap. The paper defines such schools as having: a) high academic standards, b) strict disciplinary codes, c) extended instructional time, and d) targeted supports for low-performing students. From their meta-analysis of 10 quasi-experimental studies , the authors concluded students who attended No Excuses charter schools had average achievement gains of 0.16 standard deviations in English language arts and 0.25 in mathematics. While conceding that charter schools with lotteries and No Excuses charter schools are not representative of all charter schools, the authors did not address whether or how students who apply to lottery charter schools might not be representative of all charter school students. They also did not address the possible relevance of student attrition for the individual studies’ findings and their own analysis. As a result, the claim that No Excuses schools can close the achievement gap substantially overstates their findings. Moreover, the report’s relatively small sample of schools concentrated in Northeast Coast cities suggests the current research base is too limited to draw conclusions about the effectiveness of No Excuses charter schools.

On December 3, 2014, the U.S. Department of Education released a draft of proposed new Teacher Preparation Regulations under Title II of the Higher Education Act with a call for public comments within 60 days. The proposal enumerates federally mandated but state-enforced regulations of all teacher preparation programs. Specifically, it requires states to assess and rate every teacher preparation program every year with four Performance Assessment Levels (exceptional, effective, at-risk, and low-performing), and states must provide technical assistance to “low-performing” programs. “Low-performing” institutions and programs that do not show improvement may lose state approval, state funding, and federal student financial aid. This review considers the evidentiary support for the proposed regulations and identifies seven concerns: (1) an underestimation of what could be a quite high and unnecessary cost and burden; (2) an unfounded attribution of educational inequities to individual teachers rather than to root systemic causes; (3) an improperly narrow definition of teacher classroom readiness; (4) a reliance on scientifically discredited processes of test-based accountability and value-added measures for data analysis; (5) inaccurate causal explanations that will put into place a disincentive for teachers to work in high-needs schools; (6) a restriction on the accessibility of federal student financial aid and thus a limiting of pathways into the teaching profession; and (7) an unwarranted, narrow, and harmful view of the very purposes of education.

The two reports reviewed here advocate for school choice as a mechanism for improving educational outcomes. The ConnCAN report bases its claims on largely anecdotal evidence, making many unsubstantiated claims. It is merely a choice advocacy document and is therefore of little value to serious policy reform conversations or academic researchers. The report from the American Enterprise Institute is more interesting. It is grounded in solid scholarship and data from existing choice research as well as research conducted by its author. It makes useful suggestions regarding how choice might be more effectively implemented. However, like the ConnCAN report, it too ultimately makes unsubstantiated claims about the power of choice to reform schools and to improve educational outcomes. The AEI report provides a useful overview and analysis of the choice literature and is valuable for those seeking a deeper understanding of school choice, although it overreaches in its claims regarding the power of choice and of the market to reform schools and produce better educational outcomes.

A new report from the Center for American Progress estimates substantial economic benefits from closing achievement gaps. These gains result from higher economic growth, which the report suggests would arise from higher levels of student achievement and specifically from higher achievement by minority students. Overall, the report estimates that if Black and Hispanic high school math scores converged to equal those of White high school students, the size of the U.S. economy would increase by $20 trillion over the period from 2014 to 2050. Federal and state/local tax revenues would also increase, by $4 trillion and $3 trillion respectively, over this period. Thus, the report makes an economic case for sizeable public investments to close achievement gaps. However, although there are likely to be economic gains from closing these gaps, the report does not include much detail concerning specific calculations and does not check the accuracy of its estimates. Moreover, these estimates rely on a single study, and that study has limitations: it looks across countries rather than at the U.S. economy, and it implies a very powerful role for cognitive skills (test scores) over behaviors. A general proposition—that reducing educational gaps makes sense on both efficiency and equity grounds—is plausible. But the report does not provide enough detail for readers to see how big the efficiency gains are, and readers are asked to accept that closing achievement gaps—rather than raising graduation rates or enhancing socio-emotional skills—will yield the biggest economic pay-off.

This report argues that federal Title I funding needs to be reformed because, as currently structured, it does not address funding inequities between Title I and non-Title I schools, and because current regulations governing the program negate its effectiveness. The report proposes reforming Title I so that funding is portable, that is, it follows the child. That change would, among other things, facilitate school choice policies. To support its arguments, the report relies on two strategies: (1) it cites literature that supports its position while ignoring conflicting evidence, and (2) it uses rhetoric rather than evidence. The report ignores the complexity of Title I and of federal education funding generally. It also provides no analysis of the factors contributing to the alleged Title I problems. For these reasons and others, the report is little more than a polemic, using an eclectic assortment of disconnected facts and figures about Title I funding to promote choice and voucher policies. The report provides no evidence that its recommendations will improve academic outcomes and does not consider the adverse impact its recommended policies would have on improving educational opportunities, which is what Title I is designed to do.

A new report compares the performance of Florida Virtual School (FLVS) students with students in traditional brick-and-mortar schools and  concludes the FLVS students perform about the same or somewhat better on state tests and at a lower cost. The report claims to be the first empirical study of K-12 student performance in virtual education. This is not correct, and the report in fact confirms the findings and repeats the methodological flaws and limitations of previous research. The report’s findings fail to account for the potential bias of student selectivity in the FLVS sample, the potential impact of regression effects, differential mortality in the two groups, and the fact that the virtual environment is simply a delivery medium. Given the limitations of research such as this new study, researchers have moved beyond simply investigating whether one medium is better than the other and begun—and need to continue—investigating under what conditions K-12 online and blended learning can be effectively designed, delivered, and supported.

This report attempts to examine whether charter schools have a positive effect on student achievement. From a review of 52 studies that the authors considered superior (40 of them used in an earlier report),it finds charters are serving students well, particularly in math. This conclusion is overstated; the actual results are not positive in reading and are not significant in high school math; for elementary and middle school math, effect sizes are very small, ranging from 0.03 to 0.08 s.d. The report does a solid job describing the methodological limitations of the studies reviewed, then seemingly forgets those limits in the analysis. For example, the authors include lottery-based studies, considering them  akin to random assignment, but lotteries only exist in charter schools that are much more popular than the comparison public schools from which  students are drawn. This limits the study’s usefulness in broad comparisons of all charters versus public schools. The report also seeks to examine whether the effects of charter schools have changed over time. Despite finding no change, the authors inexplicably assert that there is a positive trend. Claims of positive effects when they are not statistically significant, exaggeration of the magnitude of effects, reliance on simple vote-counts from a selected sample of studies, and unwarranted extrapolation of the available evidence to assert the effectiveness of charter schools further render the report of little value for informing policy and practice.

A new report that scores and ranks national education systems based on their efficiency has been receiving considerable media attention on both sides of the Atlantic. Efficiency is measured based on test scores, and resource use is analyzed in terms of teacher wages and pupil-teacher ratios. Looking across the 30 countries, the model predicts that, in order to get a 5% increase in PISA scores, teacher wages would have to go up by 14% or class sizes would have to go down by 13 students per class. But the optimal wages and class sizes for any given country may sometimes demand an increase or decrease in one or the other factor. For Switzerland, for example, the optimal teacher salary would require wages to be cut by almost half; for Indonesia, wages would have to be increased more than three-fold. For four countries, the optimal class size is estimated at fewer than two students per teacher. These extreme findings are due, in large part, to weaknesses in each of the study’s three key elements: the output measure is questionable, the input measures are unclear, and the econometric method by which they are correlated does not have a straightforward economic interpretation. The report may satisfy an apparent keenness for reports that rank countries— and especially for reports that castigate low-rank countries. But it does not advance our understanding of how to make the provision of education more efficient.

This report argues that a particular charter school preschool model is successful and that, therefore, charter approaches should be used to expand preschool access, recommending that states develop large numbers of charter preschools. But the report fails to make the case that the model is unusually effective or that charter status is critical to any success it does have.While the AppleTree model may well be as effective as the Pioneer authors suggest, this report lacks rigorous evidence regarding the model’s development, implementation, cost, and effectiveness. The report uses a pre-test, post-test design to argue the program is effective, but lacks a comparison group that could show if the test results are impressive or disappointing. It also provides no evidence that the children served are comparable to children served by other preschool programs. Sample sizes, attrition, and statistical methods are unreported, and no statistical tests of significance appear to have been conducted. Preschool models with rigorous evidence of high levels of effectiveness have been developed and are currently implemented by public school systems where adequate funding has been made available. We will not know whether AppleTree can add to the preschool policy debates without more rigorous evaluation of the program and its effects.

These two reports focus on the need for more effective principals to improve school qual­ity. Each includes a sensible discussion of workplace conditions affecting the principal pipeline and contributing to leadership effectiveness. Great Principals at Scale recom­mends better school and district alignment of goals and strategies, along with district-pro­vided support structures and greater local autonomy for principals. Lacking Leaders examines the hiring practices in five school districts and identifies ineffective hiring prac­tices as contributing to a pipeline problem; it argues that attracting the best candidates will require additional remuneration, greater autonomy over staffing decisions, and in­creased district-level collaboration. The headline policy recommendation is to increase salaries $100,000 above current levels to attract more effective principals into the pipe­line. No research in the report, however, justifies the size of the salary recommendation or demonstrates salary as the most important factor influencing principal recruitment, selec­tion, or retention. Also, while both reports focus on the principal as the primary source of leadership in schools, neither considers other important sources of leadership. Both re­ports suggest leadership and management skills found in many organizational settings outside of education are easily or directly transferable to education. This suggestion, how­ever, underestimates the human context of teaching and learning. These and other limita­tions undermine the usefulness of the reports’ resulting recommendations.

This report claims superiority of charter schools in producing achievement per dollar invested. The findings are cast as cost-effectiveness ratios, where effects are measured by National Assessment of Educational Progress (NAEP) points and costs are measured as “revenues received.” The report concludes that charter schools deliver an additional 17 NAEP points per $1000 in math and 16 points per $1000 in reading. All analyses are undertaken with data for 21 states and the District of Columbia. Because an earlier review by Baker pointed to serious flaws in the “cost” part of the ratio, this review will focus on achievement. The effects of charter versus traditional public schools are estimated by comparing state averages of both sectors without attempting to equate them on demographic variables like poverty (free lunch eligibility) or special-needs status. Not reported is the fact that the demographic differences between the two sectors are highly correlated with the estimates of differential effects; the sector with the higher percentage of poor pupils scores lower on the NAEP test. This failure alone renders the report and its recommendations indefensible. Furthermore, the assessment of expenditures in the two sectors rests on non-comparable data across states and questionable data within states. These weaknesses leave little evidence on which to base any valid conclusions. Reports of this type can only be viewed as advocacy research, in large part because they fail to reconcile their findings with the extensive literature of contrary findings.

RAND’s recent evaluation attempts to determine the New Leaders principal preparation program’s impact on student test scores, concluding that New Leader principals are slightly more effective, albeit only for certain grade levels, subject areas, and districts. But the study’s overall conclusion is problematic for at least eight reasons. First, the effect sizes are quite small—less than 2 percentile points in the lower grades, where the data allow for more reliable analyses—and the study’s results are more mixed than its bottom-line conclusion would suggest. Indeed, most of the results find no statistically significant impact of New Leader principals on student test scores, and there were nearly as many negative findings as positive ones. Also, the study fails to acknowledge the extant research on the limitations of the study’s value-added model (VAM) with school fixed effects approach. Additionally, problems include a failure to control for the same variables in each VAM, for peer effects, or for the scores of student stayers and leavers. The study also failed to discuss the potential impact of changes in the way student achievement was measured in the districts or to explore how the distribution of school-level scores might have influenced the results. Yet in the face of these problems and limitations, this study implies—incorrectly—that it can offer a valid causal determination that the New Leaders principals had a small positive effect on student test scores.

Based on the presumed success of school choice programs, Expanding the Education Universe: A Fifty-State Strategy for Course Choice seeks to take choice a step further. Each student would design a personal program of online and off-line courses chosen from a marketplace of curricula developed by for-profit and not-for-profit vendors, as well as school districts or other public entities. Such course choice would, the report contends, alleviate transportation problems, provide greater options, and circumvent the restricted offerings even in choice schools. The proposal is presented in the form of a “guide” to addressing practical policy issues and implementation problems. However, the report assumes, without solid evidence, that course choice, electronic educational provisions, and the like are viable, effective, and proven methods. No direct research is presented, and relevant related research that might support the efficacy of the method is not included. Accordingly, the piece rests entirely on assumptions and assertions. Given the lack of supporting evidence and detail, policymakers and the public have little basis for assessing the benefits and liabilities of a program that potentially has enormous financial costs and educational quality implications for public education.

This sincere and well-written but methodologically and politically unsophisticated report argues states should step aside from any direct involvement in the reform business and hand it over to an “ecosystem of nonprofit organizations.” The report makes five assertions about State Education Agencies (SEAs): they suffer from a lack of human resources; their procurement practices are cumbersome and time-consuming; they suffer from antiquated rulemaking; they are undermined by statewide politics; and they suffer from “institutional sclerosis.” These claims set the stage for the report’s basic recommendation: “The SEA should not attempt to implement the nuts and bolts of school improvement, but instead create an environment in which a variety of other organizations can fill the void.” In place of expanding the authority of the SEAs, the report suggests a“4Cs” model of operation: control, contract, cleave and create. Drawing on secondary materials, the report’s claims about the failures of the SEAs are strong but unsubstantiated by data independent from advocacy. Privatizing educational reform is an idea whose time has not come, and most likely never will, because it’s an abstraction based on a model of American education disconnected from the democratic ethos that animates public education. Public education is a public good; it is the loom by which citizens together weave the social contract. 

New York City’s Children First: Lessons in School Reform summarizes elementary and secondary level education policy reforms in New York City during Mayor Michael Bloomberg’s tenure. Education policy changes in New York City during this time frame, from 2002 through 2013, are collectively known as “Children First” reforms. The report reviews the elements of these reforms and analyzes their effectiveness both collectively and individually. The New York City school system experienced dramatic changes during this era, and the report does a very nice job of synthesizing important events and facts into a single narrative. The report occasionally goes too far in classifying various policies as successes or failures. In particular, the report overhypes research examining the success of small high schools and of charter schools; a more balanced interpretation of this research literature should lead to a far more neutral tone concerning the success of these schools. To its credit, the report also discusses important questions of systemic governance and policy implementation. Management and accountability systems set up during Children First likely had a tremendous impact on how other reform components were implemented and are also likely to continue to affect how education policies are implemented under the new mayoral regime.

This review addresses two different proposals for reforming teacher training, neither of which is grounded in research. Further, neither provides useful evidence that the proposed programs have been, or promise to be, effective. TNTP’s Fast Start initiative essentially replaces teacher preparation with a five-week pre-service program followed by a closely monitored internship. As proof that compressing teacher preparation into basic survival training is effective, the authors report three weak correlations between the performance of program participants and TNTP’s certification evaluation rubric. The report concludes with three self-evident aphorisms: practice improves teaching, teachers who master teaching skills do better, and inadequate performers should be weeded out. Unfortunately, the TNTP report fails to show its policy prescription is effective or superior to other approaches. New America Foundation’s Time to Improve proposes a federal regulatory approach, rating teacher preparation programs based primarily on the k-12 test scores of the pupils their graduates go on to teach. It doesn’t address why policymakers should favor extending the use of high-stakes student test scores to teacher preparation programs in light of the long string of uncontrolled intervening factors that invalidate this approach. In light of these weaknesses, neither proposal provides useful guidance for teacher preparation policy.

The University of Arkansas Center for Education Reform’s report on charter school funding inequities proclaims large and growing inequities between school district and charter school revenues, even after accounting for differences in student needs. But the report displays complete lack of understanding of intergovernmental fiscal relationships, which results in the blatantly erroneous assignment of “revenues” between charters and district schools. A district’s expenditure can be a charter’s revenue, since charter funding is in most states and districts received by pass-through from district funding, and districts often retain responsibility for direct provision of services to charter school students—a reality that the report entirely ignores when applying its resource-comparison framework. In addition, the report suffers from alarmingly vague documentation regarding data sources and methodologies, and it constructs entirely inappropriate comparisons of student population characteristics. Simply put, the findings and conclusions of the study are not valid or useful.

The Effect of Co-locations on Student Achievement in NYC Public Schools argues that co-locations of charter schools with traditional public schools have no statistical impact on traditional public school student achievement in New York City. However, the report omits important details about its analyses, which leaves readers unable to judge the validity of its methods and ultimate claims. Also, the report does not build on existing research or background knowledge on co-locations or related topics, and it expressly neglects to consider important outcomes related to students’ socio-emotional development, safety, health, and broader academic experiences, thus perpetuating an overly narrow focus on standardized test scores as the ultimate outcome of schooling. The report ultimately serves more as a marketing tool for the continued growth of charter schools in New York City than as a carefully presented research study. As a result, it does little to help policymakers and practitioners evaluate the effects of co-location on students’ educational experiences and outcomes, both of which are inextricably linked with their opportunities for and access to high-quality conditions for teaching and learning.

Can the quality of teachers be measured the way that a person’s weight or height is measured? Some economists have tried, but the “value-added” they have attempted to measure has proven elusive. The results have not been consistent over tests or over time. Nevertheless, a two-part report by Raj Chetty and his colleagues claims that higher value-added scores for teachers lead to greater economic success for their students later in life. This review of the methods of Chetty et al. focuses on their most important result: that teacher value-added affects income in adulthood. Five key problems with the research emerge. First, their own results show that the calculation of teacher value-added is unreliable. Second, their own research also generated a result that contradicts their main claim—but the report pushed that inconvenient result aside. Third, the trumpeted result is based on an erroneous calculation. Fourth, the report incorrectly assumes that the (miscalculated) result holds across students’ lifetimes despite the authors’ own research indicating otherwise. Fifth, the report cites studies as support for the authors’ methodology, even though they don’t provide that support. Despite widespread references to this study in policy circles, the shortcomings and shaky extrapolations make this report misleading and unreliable for determining educational policy.

A response from the authors is below. Additional information on the study can be found at:
http://obs.rc.fas.harvard.edu/chetty/value_added.html

A further response from Moshe Adler follows below.

The Economic Benefits of New York City’s Public School Reforms, 2002-2013 attempts to estimate the economic impact of school reforms implemented during the tenure of Mayor Michael Bloomberg. The report focuses on two types of effects: direct effects on the earnings of students graduating under the reforms (who might not otherwise have done so), and indirect effects of higher graduation rates and charter school availability on residential property values. The aggregate impact on earnings and property values is estimated to exceed $74 billion. While such estimates are always an exercise in some level of speculation, this report relies on highly inappropriate assumptions to reach its conclusions. Specifically, it attributes all gains in high school completion and college enrollment to the reforms, applies national statistics on earnings and college completion to the marginal graduate in NYC, and extrapolates cross-sectional associations between graduation rates and home prices at the zip code level as the causal effect of higher graduation rates. Without taking away from the real educational and economic gains that many students experienced during this period, this seriously flawed analysis should be taken by no one as a credible estimate of its economic impact.

Updating Career and Technical Education for the 21st Century makes the claim that schools are failing to unleash the potential of Career and Technical Education (CTE). According to the report, CTE enables students to graduate from high school with the skills, knowledge and abilities required to meet the needs of a rapidly changing economy and the demands of the workplace. While the report claims to highlight the “most effective models underway across the United States” and discuss program elements best suited for replication, the showcased program descriptions provide limited evidence regarding effectiveness, and the report neglects to identify how programs and practices could be replicated. Further, while the report makes reference to programs that have the potential of bridging CTE and academic curricula, it fails to detail the importance of this strategy or to describe these efforts or their potential impact. Rather, the report reinforces the harmful existing mindset that views CTE and college preparatory curricula as being on different sides of the educational spectrum. 

Bellwether Education Partners published A New Frontier: Utilizing Charter Schooling to Strengthen Rural Education in 2014. According to the publication, nine of the nation’s 10 most rurally populated states have no charter schools (in 8 of the 9, they are not permitted); a major purpose of the document is to argue for expanding charter schools into these states. While it is presented in a fashion similar to scholarly research, serious omissions and distortions make New Frontier little more than a political lobbying document targeting rural regions (even the most urbanized states have rural regions). Especially problematic are the inadequate support or explanation for New Frontier’s premises and its presentation of superficial and misleading use of research, particularly rural education research. In the end, it is little more than an advocacy document with premises that predetermine its recommendations: how to establish more charter schools in rural regions. Missing research and slanted representations render the document useless as a source of objective information. New Frontier is propaganda—neither a thoughtful inquiry nor an honest report. 

The new Fordham report, Pluck & Tenacity, examines the impact of school vouchers on five private schools in Ohio. While the journalist who authored the report is primarily interested in the effect on this small set of schools, we focus here on an underlying assumption asserted in the executive summary of the report: that because of vouchers, “school outcomes will improve.” As presented in this report, this assumption about the beneficial impacts of vouchers is a case-study in how to engage in slanted selection and interpretation of research evidence. As we show in this review, the totality of three endnotes used in the report reflect not just an incomplete picture of the research literature on vouchers, but an extreme case of cherry-picking sources to support a contested policy agenda. Moreover, even with the few sources cited to put voucher outcomes in a favorable light, the report cherry-picks the findings that suit Fordham’s agenda, while ignoring the findings from those very same sources that do not support—and even contradict—the premise. Thus, the report is grounded in a twice-skewed and intellectually dishonest view of the research on vouchers and their academic outcomes. The subsequent journalistic celebration of five schools in Ohio then continues this unsystematic treatment of evidence, amounting to little more than cheerleading for vouchers.

This report examines the extent to which the Texas education system is efficient. It emphasizes x-efficiency, a more extensive concept than productive efficiency that includes incentives, information, and adaptability. Applying the concept of x-efficiency, the authors argue that in key areas—teacher training, teacher evaluation, teacher pay-setting, and use of instructional materials—the Texas education system is unlikely to be efficient or cannot demonstrate efficiency. However, because x-efficiency is a more comprehensive concept it requires more evidence and analysis in order to draw conclusions. As the evidence and analysis presented by the authors is insufficient, it cannot be determined whether the Texas education system is x-efficient or x-inefficient. Thus, the authors have in no way proved that the Texas education system is inefficient. That would require a more comprehensive and rigorous analysis than is provided here. Either way, such a determination is academic if there are no alternative ways to make the system more efficient. This report does not provide any alternatives. Its usefulness for policymakers and education professionals is therefore limited to the prescription that x-efficiency is important and should be addressed more rigorously. 

The recently released report Should Charter Schools Pay Rent? from the Manhattan Institute reflects concerns of charter advocates and operators regarding potential policy changes under New York City’s new mayor, Bill de Blasio. The report asserts that charging charter schools rent would cause many to run budget deficits large enough to result in staffing reductions, likely leading to a reduction in high-performing charter schools and thus, “fewer good schools” overall. The report presents a handful of poorly documented tables and graphs listing potential budget deficits, speculative layoffs, and average proficiency rates of co-located and non-co-located charter schools, few if any of which actually validate the author’s conclusions regarding the impact of charging rent on the growth of “good schools.” But the central problem with the report is that the author assumes that there exists no possible downside when resources are transferred from city schools to charter schools. The assumption is that providing these subsidies benefits charters and harms no one and that not providing these subsidies harms charters and benefits no one. The policy brief entirely ignores the broader and more complex policy questions of what it takes to manage a balanced and equitable portfolio of schooling options.

The Teacher Transfer Incentive project was designed to establish whether excellent teachers in large districts would transfer to low-performing schools for an additional stipend, whether they would remain at those schools, and whether they would have a positive impact. Results of this extensive and well-executed study are not encouraging. Of 1,500 teachers actively encouraged to transfer, 5% actually did. While 90% of transfer teachers stayed the full two years required to collect their full stipend, only 60% planned to stay for a third year, the same rate as for teachers not receiving the incentive. Elementary school students seem to have benefitted modestly from these teachers, but middle school students appear not to have benefitted. The results suggest that a financial incentive would have to remain in place longer to continue high teacher retention rates, making the program much more expensive. The study was too short to draw conclusions about long-term impact, and the use of only test scores as an outcome measure further limits policy usefulness. Finally, a transfer policy that implies winners and losers raises ethical questions beyond the reach of this particular study.

Fixing Classroom Observations: How Common Core Will Change the Way We Look at Teaching is an advocacy document. It asserts that current classroom observation rubrics are not aligned with Common Core standards and have too many cumbersome criteria; thus, observers are overloaded, give too many high ratings, and seldom give productive feedback. To remedy these problems, the report proposes two “must-have” changes to observation rubrics: (1) pay more attention to lesson content; and (2) pare observation rubrics down to make them more focused and clear. These “must haves” may or may not address some problems of classroom observations, but there is good reason to conclude that they won’t provide much benefit. The report includes no research-informed argument to support its claim that new observation rubrics improve implementation of new teacher evaluation systems by fixing inadequate observer training, insufficient monitoring of rater calibration, and lack of time or low skills in providing instructional feedback. Tools that help observers focus on lesson content may guide substantive improvements, but the report does not offer a strong rationale for doing so. Streamlined instruments and curriculum orientation may also hold some promise, but are unlikely to seriously address core problems surrounding teacher evaluations.

This report from a think tank called Public Impact begins with two unsupported premises: that only one in four teachers is good enough to help close achievement gaps, and that current efforts to recruit and retain excellent teachers are inadequate. To allow existing excellent teachers to reach more students and to develop excellence in their colleagues, it proposes a model for restructuring teaching. Hierarchically arranged teaching teams would rely on fewer teachers but more paraprofessionals, more digital instruction, longer work hours, and some larger classes. Teacher salaries would increase. However, while the report targets teacher excellence, it offers no specific means of identifying and assessing that quality. In addition, the report does not take into account relevant research literature in key areas, including teacher assessment, multiple influences on student achievement, digital instruction, teacher burnout, and teacher attrition. Overall, the proposal is based on unsupported assumptions, assertions and projections—wishes and beliefs that if the approach were put into practice, it would somehow play out to the benefit of students. Lacking an empirical base, the report is not a useful guide for policy.